testeiya 0.10.1 → 0.10.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (225) hide show
  1. package/README.md +7 -19
  2. package/dist/prompt/clis.d.ts +7 -0
  3. package/dist/prompt/pipeline-report.d.ts +1 -0
  4. package/dist/prompt/report.d.ts +10 -0
  5. package/dist/prompt/system-prompt.d.ts +22 -0
  6. package/dist/prompt/testomat.io.d.ts +19 -0
  7. package/dist/prompt/thread.d.ts +73 -0
  8. package/dist/src/args.d.ts +39 -0
  9. package/dist/src/attribution-extension.d.ts +3 -0
  10. package/dist/src/checkpoint.d.ts +30 -0
  11. package/dist/src/cli.d.ts +2 -0
  12. package/dist/src/cli.js +27 -32
  13. package/dist/src/cli.js.map +1 -1
  14. package/dist/src/doctor.d.ts +11 -0
  15. package/dist/src/env.d.ts +30 -0
  16. package/dist/src/exec.d.ts +11 -0
  17. package/dist/src/git.d.ts +41 -0
  18. package/dist/src/langfuse-extension.d.ts +7 -0
  19. package/dist/src/langfuse.d.ts +67 -0
  20. package/dist/src/mcp.d.ts +19 -0
  21. package/dist/src/model.d.ts +20 -0
  22. package/dist/src/models.d.ts +1 -0
  23. package/dist/src/output.d.ts +84 -0
  24. package/dist/src/pipeline/command-stage.d.ts +12 -0
  25. package/dist/src/pipeline/crash-guard.d.ts +12 -0
  26. package/dist/src/pipeline/crash-guard.js +21 -0
  27. package/dist/src/pipeline/crash-guard.js.map +1 -0
  28. package/dist/src/pipeline/diagram.d.ts +24 -0
  29. package/dist/src/pipeline/gate.d.ts +21 -0
  30. package/dist/src/pipeline/launcher.d.ts +33 -0
  31. package/dist/src/pipeline/launcher.js +16 -3
  32. package/dist/src/pipeline/launcher.js.map +1 -1
  33. package/dist/src/pipeline/log.d.ts +29 -0
  34. package/dist/src/pipeline/log.js +9 -0
  35. package/dist/src/pipeline/log.js.map +1 -1
  36. package/dist/src/pipeline/overlay.d.ts +29 -0
  37. package/dist/src/pipeline/pipeline-file.d.ts +44 -0
  38. package/dist/src/pipeline/pipeline.d.ts +78 -0
  39. package/dist/src/pipeline/pipeline.js +22 -0
  40. package/dist/src/pipeline/pipeline.js.map +1 -1
  41. package/dist/src/pipeline/report.d.ts +26 -0
  42. package/dist/src/pipeline/report.js +13 -0
  43. package/dist/src/pipeline/report.js.map +1 -1
  44. package/dist/src/pipeline/spec.d.ts +63 -0
  45. package/dist/src/pipeline/stage-env.d.ts +13 -0
  46. package/dist/src/pipeline/stage-env.js +31 -0
  47. package/dist/src/pipeline/stage-env.js.map +1 -0
  48. package/dist/src/pipeline/stage.d.ts +71 -0
  49. package/dist/src/pipeline/stage.js.map +1 -1
  50. package/dist/src/pipeline/state.d.ts +63 -0
  51. package/dist/src/pipeline/summary-stage.d.ts +24 -0
  52. package/dist/src/pipeline/task-stage.d.ts +24 -0
  53. package/dist/src/pipeline/task-stage.js +86 -2
  54. package/dist/src/pipeline/task-stage.js.map +1 -1
  55. package/dist/src/pipeline/variables.d.ts +20 -0
  56. package/dist/src/result.d.ts +6 -0
  57. package/dist/src/run.d.ts +69 -0
  58. package/dist/src/run.js +116 -60
  59. package/dist/src/run.js.map +1 -1
  60. package/dist/src/session.d.ts +43 -0
  61. package/dist/src/sessions.d.ts +23 -0
  62. package/dist/src/skills.d.ts +29 -0
  63. package/package.json +11 -2
  64. package/pipelines/api-testing.yml +1 -1
  65. package/scripts/vendor-skills.js +62 -0
  66. package/skills/codeceptjs/ci-fix-tests/SKILL.md +0 -119
  67. package/skills/codeceptjs/codeceptjs-auth/SKILL.md +0 -141
  68. package/skills/codeceptjs/codeceptjs-exploration/SKILL.md +0 -100
  69. package/skills/codeceptjs/codeceptjs-fundamentals/SKILL.md +0 -196
  70. package/skills/codeceptjs/codeceptjs-run-analysis/SKILL.md +0 -83
  71. package/skills/codeceptjs/debugging-codeceptjs-tests/SKILL.md +0 -126
  72. package/skills/codeceptjs/lint-codeceptjs-tests/SKILL.md +0 -104
  73. package/skills/codeceptjs/migrate-codeceptjs-4/SKILL.md +0 -109
  74. package/skills/codeceptjs/migrate-cypress-to-codeceptjs/SKILL.md +0 -295
  75. package/skills/codeceptjs/migrate-protractor-to-codeceptjs/SKILL.md +0 -339
  76. package/skills/codeceptjs/migrate-selenium-java-to-codeceptjs/SKILL.md +0 -459
  77. package/skills/codeceptjs/migrate-testcafe-to-codeceptjs/SKILL.md +0 -338
  78. package/skills/codeceptjs/refactoring-codeceptjs-tests/SKILL.md +0 -78
  79. package/skills/codeceptjs/writing-codeceptjs-tests/SKILL.md +0 -102
  80. package/skills/playwright/playwright-best-practices/SKILL.md +0 -303
  81. package/skills/playwright/playwright-best-practices/advanced/authentication-flows.md +0 -360
  82. package/skills/playwright/playwright-best-practices/advanced/authentication.md +0 -871
  83. package/skills/playwright/playwright-best-practices/advanced/clock-mocking.md +0 -364
  84. package/skills/playwright/playwright-best-practices/advanced/mobile-testing.md +0 -409
  85. package/skills/playwright/playwright-best-practices/advanced/multi-context.md +0 -288
  86. package/skills/playwright/playwright-best-practices/advanced/multi-user.md +0 -393
  87. package/skills/playwright/playwright-best-practices/advanced/network-advanced.md +0 -452
  88. package/skills/playwright/playwright-best-practices/advanced/third-party.md +0 -464
  89. package/skills/playwright/playwright-best-practices/architecture/pom-vs-fixtures.md +0 -363
  90. package/skills/playwright/playwright-best-practices/architecture/test-architecture.md +0 -369
  91. package/skills/playwright/playwright-best-practices/architecture/when-to-mock.md +0 -383
  92. package/skills/playwright/playwright-best-practices/browser-apis/browser-apis.md +0 -391
  93. package/skills/playwright/playwright-best-practices/browser-apis/iframes.md +0 -403
  94. package/skills/playwright/playwright-best-practices/browser-apis/service-workers.md +0 -504
  95. package/skills/playwright/playwright-best-practices/browser-apis/websockets.md +0 -403
  96. package/skills/playwright/playwright-best-practices/core/annotations.md +0 -424
  97. package/skills/playwright/playwright-best-practices/core/assertions-waiting.md +0 -361
  98. package/skills/playwright/playwright-best-practices/core/configuration.md +0 -452
  99. package/skills/playwright/playwright-best-practices/core/fixtures-hooks.md +0 -417
  100. package/skills/playwright/playwright-best-practices/core/global-setup.md +0 -434
  101. package/skills/playwright/playwright-best-practices/core/locators.md +0 -242
  102. package/skills/playwright/playwright-best-practices/core/page-object-model.md +0 -315
  103. package/skills/playwright/playwright-best-practices/core/projects-dependencies.md +0 -453
  104. package/skills/playwright/playwright-best-practices/core/test-data.md +0 -492
  105. package/skills/playwright/playwright-best-practices/core/test-suite-structure.md +0 -361
  106. package/skills/playwright/playwright-best-practices/core/test-tags.md +0 -298
  107. package/skills/playwright/playwright-best-practices/debugging/console-errors.md +0 -420
  108. package/skills/playwright/playwright-best-practices/debugging/debugging.md +0 -504
  109. package/skills/playwright/playwright-best-practices/debugging/error-testing.md +0 -360
  110. package/skills/playwright/playwright-best-practices/debugging/flaky-tests.md +0 -496
  111. package/skills/playwright/playwright-best-practices/frameworks/angular.md +0 -530
  112. package/skills/playwright/playwright-best-practices/frameworks/nextjs.md +0 -469
  113. package/skills/playwright/playwright-best-practices/frameworks/react.md +0 -531
  114. package/skills/playwright/playwright-best-practices/frameworks/vue.md +0 -574
  115. package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/ci-cd.md +0 -468
  116. package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/docker.md +0 -283
  117. package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/github-actions.md +0 -546
  118. package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/gitlab.md +0 -397
  119. package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/other-providers.md +0 -521
  120. package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/parallel-sharding.md +0 -371
  121. package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/performance.md +0 -453
  122. package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/reporting.md +0 -424
  123. package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/test-coverage.md +0 -497
  124. package/skills/playwright/playwright-best-practices/testing-patterns/accessibility.md +0 -359
  125. package/skills/playwright/playwright-best-practices/testing-patterns/api-testing.md +0 -719
  126. package/skills/playwright/playwright-best-practices/testing-patterns/browser-extensions.md +0 -506
  127. package/skills/playwright/playwright-best-practices/testing-patterns/canvas-webgl.md +0 -493
  128. package/skills/playwright/playwright-best-practices/testing-patterns/component-testing.md +0 -500
  129. package/skills/playwright/playwright-best-practices/testing-patterns/drag-drop.md +0 -576
  130. package/skills/playwright/playwright-best-practices/testing-patterns/electron.md +0 -509
  131. package/skills/playwright/playwright-best-practices/testing-patterns/file-operations.md +0 -377
  132. package/skills/playwright/playwright-best-practices/testing-patterns/file-upload-download.md +0 -562
  133. package/skills/playwright/playwright-best-practices/testing-patterns/forms-validation.md +0 -561
  134. package/skills/playwright/playwright-best-practices/testing-patterns/graphql-testing.md +0 -331
  135. package/skills/playwright/playwright-best-practices/testing-patterns/i18n.md +0 -508
  136. package/skills/playwright/playwright-best-practices/testing-patterns/performance-testing.md +0 -476
  137. package/skills/playwright/playwright-best-practices/testing-patterns/security-testing.md +0 -430
  138. package/skills/playwright/playwright-best-practices/testing-patterns/visual-regression.md +0 -634
  139. package/skills/playwright/playwright-cli/SKILL.md +0 -489
  140. package/skills/playwright/playwright-cli/references/element-attributes.md +0 -23
  141. package/skills/playwright/playwright-cli/references/playwright-tests.md +0 -39
  142. package/skills/playwright/playwright-cli/references/pr-attachments.md +0 -60
  143. package/skills/playwright/playwright-cli/references/request-mocking.md +0 -87
  144. package/skills/playwright/playwright-cli/references/running-code.md +0 -245
  145. package/skills/playwright/playwright-cli/references/session-management.md +0 -227
  146. package/skills/playwright/playwright-cli/references/storage-state.md +0 -275
  147. package/skills/playwright/playwright-cli/references/test-generation.md +0 -433
  148. package/skills/playwright/playwright-cli/references/tracing.md +0 -139
  149. package/skills/playwright/playwright-cli/references/video-recording.md +0 -216
  150. package/skills/skills.lock.json +0 -89
  151. package/skills/skills.yaml +0 -35
  152. /package/skills/{testomatio/explorbot → explorbot}/explorbot-fundamentals/SKILL.md +0 -0
  153. /package/skills/{testomatio/explorbot → explorbot}/explorbot-fundamentals/references/no-install.md +0 -0
  154. /package/skills/{testomatio/explorbot → explorbot}/explorbot-plan/SKILL.md +0 -0
  155. /package/skills/{testomatio/explorbot → explorbot}/explorbot-setup/SKILL.md +0 -0
  156. /package/skills/{testomatio/explorbot → explorbot}/prima/SKILL.md +0 -0
  157. /package/skills/{testomatio/qa-process → qa-process}/qa-lead-strategy-advisor/SKILL.md +0 -0
  158. /package/skills/{testomatio/qa-process → qa-process}/qa-lead-strategy-advisor/references/output-format.md +0 -0
  159. /package/skills/{testomatio/qa-process → qa-process}/qa-review-pr/SKILL.md +0 -0
  160. /package/skills/{testomatio/qa-process → qa-process}/testing-workflow/SKILL.md +0 -0
  161. /package/skills/{testomatio/requirements → requirements}/catalog-features/SKILL.md +0 -0
  162. /package/skills/{testomatio/requirements → requirements}/catalog-features/references/layout.md +0 -0
  163. /package/skills/{testomatio/requirements → requirements}/detect-product-mismatches/SKILL.md +0 -0
  164. /package/skills/{testomatio/requirements → requirements}/detect-product-mismatches/references/features.md +0 -0
  165. /package/skills/{testomatio/requirements → requirements}/detect-product-mismatches/references/layout.md +0 -0
  166. /package/skills/{testomatio/requirements → requirements}/pull-request-diff-analyzer/SKILL.md +0 -0
  167. /package/skills/{testomatio/requirements → requirements}/qa-explain-behavior/SKILL.md +0 -0
  168. /package/skills/{testomatio/requirements → requirements}/qa-explain-behavior/references/answer-shapes.md +0 -0
  169. /package/skills/{testomatio/requirements → requirements}/qa-explain-behavior/references/examples.md +0 -0
  170. /package/skills/{testomatio/requirements → requirements}/qa-pr-requirements-analyzer/SKILL.md +0 -0
  171. /package/skills/{testomatio/requirements → requirements}/qa-pr-requirements-analyzer/references/summary-example.md +0 -0
  172. /package/skills/{testomatio/requirements → requirements}/qa-requirement-reviewer/SKILL.md +0 -0
  173. /package/skills/{testomatio/requirements → requirements}/qa-requirement-reviewer/references/requirements_reviewer_examples.md +0 -0
  174. /package/skills/{testomatio/requirements → requirements}/qa-thinking/SKILL.md +0 -0
  175. /package/skills/{testomatio/requirements → requirements}/wiki-from-code/SKILL.md +0 -0
  176. /package/skills/{testomatio/requirements → requirements}/wiki-from-code/references/behaviors.md +0 -0
  177. /package/skills/{testomatio/requirements → requirements}/wiki-from-code/references/layout.md +0 -0
  178. /package/skills/{testomatio/requirements → requirements}/write-user-story/SKILL.md +0 -0
  179. /package/skills/{testomatio/test-automation → test-automation}/automate-manual-test-cases/CLAUDE.md +0 -0
  180. /package/skills/{testomatio/test-automation → test-automation}/automate-manual-test-cases/SKILL.md +0 -0
  181. /package/skills/{testomatio/test-automation → test-automation}/automate-manual-test-cases/references/CODECEPTJS_BEST_PRACTICES.md +0 -0
  182. /package/skills/{testomatio/test-automation → test-automation}/automate-manual-test-cases/references/FINAL_SUMMARY_TEMPLATE.md +0 -0
  183. /package/skills/{testomatio/test-automation → test-automation}/automate-manual-test-cases/references/PLAYWRIGHT_BEST_PRACTICES.md +0 -0
  184. /package/skills/{testomatio/test-automation → test-automation}/automate-manual-test-cases/references/POM_BEST_PRACTICES.md +0 -0
  185. /package/skills/{testomatio/test-automation → test-automation}/automate-manual-test-cases/references/TEST_DATA_MANAGEMENT.md +0 -0
  186. /package/skills/{testomatio/test-automation → test-automation}/debug-fix-failed-flaky-autotests/SKILL.md +0 -0
  187. /package/skills/{testomatio/test-automation → test-automation}/debug-fix-failed-flaky-autotests/references/DEBUGGING_QUICK_REFERENCE.md +0 -0
  188. /package/skills/{testomatio/test-automation → test-automation}/epic-requirements-specification/SKILL.md +0 -0
  189. /package/skills/{testomatio/test-automation → test-automation}/qa-automation-test-consolidation/SKILL.md +0 -0
  190. /package/skills/{testomatio/test-automation → test-automation}/qa-data-seeder/SKILL.md +0 -0
  191. /package/skills/{testomatio/test-automation → test-automation}/run-tests-with-testomatio-reporter/SKILL.md +0 -0
  192. /package/skills/{testomatio/test-automation → test-automation}/setup-change-aware-pr-testing/SKILL.md +0 -0
  193. /package/skills/{testomatio/test-automation → test-automation}/setup-ci-automation/SKILL.md +0 -0
  194. /package/skills/{testomatio/test-automation → test-automation}/testomat-allure-adapter/SKILL.md +0 -0
  195. /package/skills/{testomatio/test-management → test-management}/detect-duplicate-test-cases/SKILL.md +0 -0
  196. /package/skills/{testomatio/test-management → test-management}/detect-duplicate-test-cases/references/DUPLICATE_INSTRUCTIONS.md +0 -0
  197. /package/skills/{testomatio/test-management → test-management}/detect-useless-tests/SKILL.md +0 -0
  198. /package/skills/{testomatio/test-management → test-management}/improve-test-cases/SKILL.md +0 -0
  199. /package/skills/{testomatio/test-management → test-management}/improve-test-cases/references/TESTOMAT_MARKDOWN_EXAMPLE.md +0 -0
  200. /package/skills/{testomatio/test-management → test-management}/migrate-to-testomatio/SKILL.md +0 -0
  201. /package/skills/{testomatio/test-management → test-management}/migrate-to-testomatio/references/CSV_MIGRATION.md +0 -0
  202. /package/skills/{testomatio/test-management → test-management}/migrate-to-testomatio/references/CUSTOM_MIGRATION.md +0 -0
  203. /package/skills/{testomatio/test-management → test-management}/migrate-to-testomatio/references/TESTMO_MIGRATION.md +0 -0
  204. /package/skills/{testomatio/test-management → test-management}/migrate-to-testomatio/references/TESTRAIL_MIGRATION.md +0 -0
  205. /package/skills/{testomatio/test-management → test-management}/migrate-to-testomatio/references/XRAY_MIGRATION.md +0 -0
  206. /package/skills/{testomatio/test-management → test-management}/qa-e2e-tests-reporting/SKILL.md +0 -0
  207. /package/skills/{testomatio/test-management → test-management}/qa-e2e-tests-reporting/references/TESTOMATIO_ARTIFACTS.md +0 -0
  208. /package/skills/{testomatio/test-management → test-management}/qa-e2e-tests-reporting/references/TESTOMATIO_HTML_REPORT.md +0 -0
  209. /package/skills/{testomatio/test-management → test-management}/qa-e2e-tests-reporting/references/TESTOMATIO_REPORTERS_CONFIG.md +0 -0
  210. /package/skills/{testomatio/test-management → test-management}/qa-split-testing-levels-pyramid/SKILL.md +0 -0
  211. /package/skills/{testomatio/test-management → test-management}/qa-sprint-report-by-testomatio/SKILL.md +0 -0
  212. /package/skills/{testomatio/test-management → test-management}/qa-sprint-report-by-testomatio/references/qa-sprint-report.md +0 -0
  213. /package/skills/{testomatio/test-management → test-management}/qa-test-code-coverage/SKILL.md +0 -0
  214. /package/skills/{testomatio/test-management → test-management}/qa-test-code-coverage/references/COVERAGE_FILE_FORMAT.md +0 -0
  215. /package/skills/{testomatio/test-management → test-management}/qa-test-code-coverage/references/E2E_FRAMEWORKS.md +0 -0
  216. /package/skills/{testomatio/test-management → test-management}/qa-test-code-coverage/scripts/check-coverage.mjs +0 -0
  217. /package/skills/{testomatio/test-management → test-management}/qa-write-test-cases/SKILL.md +0 -0
  218. /package/skills/{testomatio/test-management → test-management}/qa-write-test-cases/references/test-case-format.md +0 -0
  219. /package/skills/{testomatio/test-management → test-management}/qa-write-test-cases/references/testomat-tms-guide.md +0 -0
  220. /package/skills/{testomatio/test-management → test-management}/qa-write-test-cases/references/writing-rule.md +0 -0
  221. /package/skills/{testomatio/test-management → test-management}/scan-automation-project/SKILL.md +0 -0
  222. /package/skills/{testomatio/test-management → test-management}/sync-test-cases-with-tms/SKILL.md +0 -0
  223. /package/skills/{testomatio/test-management → test-management}/sync-test-cases-with-tms/references/TESTOMATIO_CLI.md +0 -0
  224. /package/skills/{testomatio/test-management → test-management}/testomatio-mcp/SKILL.md +0 -0
  225. /package/skills/{testomatio/test-management → test-management}/testomatio-mcp/references/MCP_SETUP.md +0 -0
@@ -1,119 +0,0 @@
1
- ---
2
- name: ci-fix-tests
3
- description: "Use on CI in non-interactive mode after a failed CodeceptJS run — attempts safe fixes only (locator drift, missing waits), reruns failing scenarios, rolls back edits that didn't help, reports at `output/ci-fix.md`. No refactors, no config edits, no flake-masking. Trigger on \"ci fix tests\", \"auto-fix failing tests\", \"attempt repair on CI\", or as a CI step after a failed run."
4
- ---
5
-
6
- # Auto-fix CodeceptJS Tests on CI
7
-
8
- A non-interactive auto-repair loop. Reads the failed-tests baseline from `output/` (where `aiTrace` left trace artifacts), attempts a small set of safe fixes, reruns only the failing scenarios, compares, and writes a report at `output/ci-fix.md`. Rolls back any edit that didn't help.
9
-
10
- **Conservative by design.** CI is not the place to restructure tests, change abstractions, or fix anything that needs human judgement. The goal is "pass the run". Anything riskier is escalated through the report.
11
-
12
- This skill assumes a working **git** checkout — rollback uses `git checkout -- <file>` to restore originals.
13
-
14
- ## Workflow
15
-
16
- ### 1. Read the baseline failures
17
- Use the **codeceptjs-run-analysis** skill against the existing `output/trace_*/` artifacts to enumerate every failing scenario. Record:
18
-
19
- - the **set** of failing tests (file path + scenario name)
20
- - the failed step in each, and the locator / wait / assertion involved
21
- - a cluster signature per failure (so unique vs systemic is clear)
22
-
23
- Save this as the **baseline** — `count`, `failing_set`, `clusters`.
24
-
25
- ### 2. Pick safe fix candidates
26
- Only attempt fixes from this allowed list:
27
-
28
- - **Locator drift** — the failed locator no longer matches anything but a similar element exists. Use the **codeceptjs-exploration** skill (headless) to find candidates; pick a short locator scoped to a stable region — `I.click('Save', '.toolbar')` — in this order: visible text / accessible name → ARIA `{ role, name }` → `$name` (`customLocator`) → composed CSS. Replace the locator at the failing step only.
29
- - **Missing wait for a spinner / loader / modal** — the failed step's ARIA snapshot shows a spinner or skeleton present, or the target element appears later. Add a single matching `I.waitFor*` immediately before the failing step.
30
- - **`I.wait(N)` replacement** — when a hardcoded sleep is the only thing between a failing assertion and a passing one and the gating element is identifiable, replace the sleep with a specific `waitFor*`.
31
-
32
- **Do not attempt:**
33
- - Auth / credential / env-var fixes — those are CI configuration, not test code.
34
- - Refactoring (extracting page objects, custom helpers, splitting scenarios).
35
- - Renaming Scenarios or tags — CI's `--grep` and pipelines reference them.
36
- - Adding `retries: N` to a Scenario — masking flake is a human decision.
37
- - Anything outside the failing test file (no `codecept.conf.*`, no helper, no PO edits).
38
-
39
- If a failure doesn't match any allowed-fix pattern, **skip it** and record it as `unresolved` for the report.
40
-
41
- ### 3. Apply edits and rerun
42
- Track every modification: file path, line number, before/after snippet, fix kind. Then rerun **only the failing scenarios** under the same config CI used:
43
-
44
- ```bash
45
- npx codeceptjs run --grep '<scenario>|<scenario>|...'
46
- ```
47
-
48
- Find the right config via `package.json` scripts or the CI workflow file. Match the worker count if `run-workers` was used originally.
49
-
50
- ### 4. Compare against the baseline
51
- Use **codeceptjs-run-analysis** again to enumerate the failing set after the fix.
52
-
53
- | New failing-set vs baseline | Status |
54
- |---|---|
55
- | Empty (zero failures) | `SUCCESS_FULL` |
56
- | Strict subset of the baseline | `SUCCESS_PARTIAL` |
57
- | Identical set | `NO_IMPROVEMENT` |
58
- | Larger, or contains a test not in baseline | `REGRESSED` |
59
-
60
- ### 5. Decide: keep or roll back
61
-
62
- - `SUCCESS_FULL` / `SUCCESS_PARTIAL` → **keep** the edits.
63
- - `NO_IMPROVEMENT` / `REGRESSED` → **roll back every edit** with `git checkout -- <files>`. The on-disk state at the end of the skill must be identical to what it was at the start.
64
-
65
- A regression is treated the same as no improvement — the attempt didn't work, the working state is what was on disk before. Don't try to "undo only the bad ones" — the whole batch goes.
66
-
67
- ### 6. Write the report (always)
68
- Write `output/ci-fix.md` regardless of outcome — CI reads it to surface results.
69
-
70
- ## Report format
71
-
72
- ```markdown
73
- # CI Fix Report — <ISO timestamp>
74
-
75
- **Status:** SUCCESS_FULL | SUCCESS_PARTIAL | NO_IMPROVEMENT | REGRESSED
76
-
77
- ## Baseline
78
- Failing scenarios: N
79
- - `tests/foo_test.js` — "user can log in"
80
- - `tests/bar_test.js` — "checkout flow"
81
- - ...
82
-
83
- ## Attempted fixes
84
- - `tests/foo_test.js:42` — locator drift: `I.click('Save')` → `I.click('Save', '.toolbar')`
85
- - `tests/bar_test.js:15` — added `I.waitForInvisible('.spinner')` before checkout click
86
- - `tests/baz_test.js:7` — replaced `I.wait(3)` with `I.waitForVisible('.confirmation-dialog', 10)`
87
-
88
- ## Result
89
- Failing scenarios after fix: M (was N)
90
- - Resolved: `tests/foo_test.js` — "user can log in"
91
- - Still failing: `tests/bar_test.js` — "checkout flow"
92
- - New failures (REGRESSED only): `tests/baz_test.js` — "edit profile"
93
-
94
- ## Failure clusters (NO_IMPROVEMENT / REGRESSED only)
95
- - **Cluster A (3 scenarios)** — `Element not found: button.btn-primary`. Likely a deploy regression or design change.
96
- - **Cluster B (1 scenario)** — `Navigation timeout to /checkout`. Possible backend issue.
97
-
98
- ## Discarded changes (NO_IMPROVEMENT / REGRESSED only)
99
- - `tests/foo_test.js` — restored
100
- - `tests/bar_test.js` — restored
101
- - `tests/baz_test.js` — restored
102
- ```
103
-
104
- The first line of `Status:` is the machine-parseable signal. The rest is for the human reading the failed CI build.
105
-
106
- ## Things to avoid
107
-
108
- - Editing files outside the failing test files. Config, helpers, page objects, and CI workflows are off-limits in this mode.
109
- - Attempting more than one fix kind on the same scenario in one pass. Fix one thing, observe, then try another.
110
- - `retries: N` to mask flakiness — human decision.
111
- - Skipping rollback on `NO_IMPROVEMENT`. Even no-op edits can confuse the next run or the next reviewer.
112
- - Writing a partial or missing `output/ci-fix.md`. CI depends on it; the absence of a report is itself a failure mode.
113
- - Running the **whole** suite for the verification step — only the originally-failing scenarios, to keep CI time bounded.
114
-
115
- ## Related skills
116
-
117
- - `codeceptjs-run-analysis` — baseline, clustering, post-fix comparison
118
- - `codeceptjs-exploration` — replacement locators for drift
119
- - `codeceptjs-fundamentals` — config, helper, CI environment
@@ -1,141 +0,0 @@
1
- ---
2
- name: codeceptjs-auth
3
- description: >
4
- Use when a CodeceptJS test needs login, user roles, or session reuse — or
5
- when another CodeceptJS skill identifies authorization is required. Also on
6
- auth-related test failures. Trigger on login, sign-in, sign-up,
7
- authentication, sessions, "logged in", admin/editor/user roles.
8
- ---
9
-
10
- # CodeceptJS Auth Plugin
11
-
12
- The `auth` plugin logs each user in once, captures cookies (or localStorage/token via overrides), and restores the session for subsequent tests. Stale sessions trigger a fresh login automatically.
13
-
14
- If the project already has `auth` configured (fundamentals' discovery output), **reuse** its existing inject name and user keys — don't reconfigure.
15
-
16
- ## Decide first — ask the user
17
-
18
- Four answers shape the plugin. Don't guess any.
19
-
20
- 1. **Is a session needed at all?** Public flows (landing, signup) → skip the plugin.
21
- 2. **One user or many?** Default one; add more only when actually exercised.
22
- 3. **If many — what splits them?** Role, workspace/tenant, plan tier, sign-in provider, per-test fixture — real systems vary. Ask; use the answer to name `users.<key>` entries.
23
- 4. **What's the auth type?**
24
- - **Form** — default; canonical shape below
25
- - **OAuth / SSO** — click provider button, drive the IdP page (often separate origin)
26
- - **Magic link / passwordless** — UI flow rarely worth automating; prefer an API mint or reading the link from a test mailbox
27
- - **API token** — skip the form; write the token into `localStorage` via `executeScript`, or `I.setCookie(...)`
28
- - **2FA / OTP** — async `login`; fetch the code from a test mailbox / backdoor before submitting
29
-
30
- ## Rules
31
-
32
- 1. **Login flow never lives in the conf.** Put it in `steps_file.js` (if included) or a page object; conf only references it: `login: (I) => I.login()`.
33
- 2. **Credentials from env only** — `.env` loaded via `process.loadEnvFile()` (no dotenv dependency). Passwords wrapped with `secret(...)`. No literals anywhere — conf, steps file, test, git history.
34
- 3. **`.env` is gitignored; `.env.example` is committed** with names, no values. Gitignore `output/*_session.json` too.
35
-
36
- ## Canonical shape
37
-
38
- ```js
39
- // codecept.conf.js — first line of the file
40
- process.loadEnvFile()
41
-
42
- export const config = {
43
- include: { I: './steps_file.js' },
44
- plugins: {
45
- auth: {
46
- enabled: true,
47
- saveToFile: true,
48
- users: {
49
- admin: {
50
- login: (I) => I.login(),
51
- check: (I) => I.see('Welcome, User', '.navbar'),
52
- },
53
- },
54
- },
55
- },
56
- }
57
- ```
58
-
59
- ```js
60
- // steps_file.js
61
- import { secret } from 'codeceptjs'
62
- const { I } = inject()
63
-
64
- export default function () {
65
- return actor({
66
- login() {
67
- I.amOnPage('/login')
68
- I.fillField('Email', process.env.ADMIN_EMAIL)
69
- I.fillField('Password', secret(process.env.ADMIN_PASSWORD))
70
- I.click('Sign in')
71
- },
72
- })
73
- }
74
- ```
75
-
76
- ```sh
77
- # .env (gitignored) # .env.example (committed)
78
- USER_EMAIL=... USER_EMAIL=
79
- USER_PASSWORD=<secret> USER_PASSWORD=
80
- ```
81
-
82
- ## Pre-flight (before writing config)
83
-
84
- 1. **Read the real login page** — MCP `run_code` to `/login`, inspect the ARIA snapshot (`codeceptjs-exploration`). Field labels / `name` / `id` / submit control from the actual page, not guesses. Unclear authorization mechanism → ask the user.
85
- 2. **Pick a role-specific post-login marker** — something rendered only for *this* user (navbar username, `data-user-role`).
86
- 3. **Confirm session storage** — cookies (default) for server-rendered apps; localStorage/sessionStorage for SPAs. Verify after a manual login with `I.executeScript(() => Object.keys(localStorage))`. Cookie fetch/restore silently no-op against token storage.
87
-
88
- ## Verify
89
-
90
- Run a one-Scenario file that calls `login(<role>)` then asserts on the post-login marker:
91
-
92
- ```bash
93
- npx codeceptjs run --grep '<scenario>' --debug # real run, not dry — dry-run doesn't init plugins
94
- ```
95
-
96
- Enable `saveToFile: true` only after this round-trip succeeds — a bad saved session masks a broken `login`.
97
-
98
- Then wire into hooks/tests: `Before(({ login }) => login())` for suite-wide, or per-test as needed.
99
-
100
- ## Multi-role shape
101
-
102
- Only after question 3 is answered. Keys named after whatever splits users *in this system*; one matching actor method per key:
103
-
104
- ```js
105
- users: {
106
- admin: { login: (I) => I.loginAsAdmin() },
107
- workspaceB: { login: (I) => I.loginToWorkspaceB() },
108
- }
109
- ```
110
-
111
- Don't parameterise into a single `login(key)` — the plugin keys sessions by name, explicit methods read better. Switch mid-Scenario: `session('<key>')` opens a parallel browser context (fundamentals § Writing tests).
112
-
113
- ## Token / localStorage auth
114
-
115
- Override `fetch` / `restore` when sessions live outside cookies:
116
-
117
- ```js
118
- admin: {
119
- login: (I) => I.loginAsAdmin(),
120
- check: (I) => I.see('Admin', '.navbar'),
121
- fetch: (I) => I.executeScript(() => localStorage.getItem('session_id')),
122
- restore: (I, session) => {
123
- I.amOnPage('/')
124
- I.executeScript((s) => localStorage.setItem('session_id', s), session)
125
- },
126
- }
127
- ```
128
-
129
- `check(I, session)` receives whatever `fetch` returned — throw inside `check` to force fresh login (e.g. `/me` endpoint shows wrong user).
130
-
131
- ## Pitfalls
132
-
133
- - Credentials inlined in conf/test — always env-driven + `secret()`.
134
- - Forgetting to gitignore `.env` and `output/*_session.json` — both leak credentials.
135
-
136
- ## Related skills
137
-
138
- - `codeceptjs-fundamentals` — secrets rule, sessions, config mutation trap
139
- - `codeceptjs-exploration` — reading the live login page
140
- - `writing-codeceptjs-tests` / `refactoring-codeceptjs-tests` — invoke this skill when auth is identified
141
- - `debugging-codeceptjs-tests` — auth-related failure patterns
@@ -1,100 +0,0 @@
1
- ---
2
- name: codeceptjs-exploration
3
- description: >
4
- Use when an agent needs to learn what's on a page in CodeceptJS — read the
5
- ARIA tree, inspect candidate elements, pick or disambiguate a stable locator.
6
- Drives the live browser via MCP `run_code` / `snapshot`. Invoked by
7
- writing-codeceptjs-tests, debugging-codeceptjs-tests, and
8
- refactoring-codeceptjs-tests whenever page inspection is needed.
9
- ---
10
-
11
- # CodeceptJS Page Exploration
12
-
13
- Authoring a test, debugging a failure, and refactoring a stale locator share one task: open a page, find the right element, pick a stable locator. This is that playbook.
14
-
15
- ## Tools
16
-
17
- - **`run_code`** — runs CodeceptJS code, returns produced values, captures `console.*`, saves a final-state snapshot. For *do something and look at the result*.
18
- - **`snapshot`** — captures state without acting (URL, cookies, localStorage, HTML, ARIA, screenshot, console). For "what's on the page right now".
19
-
20
- Artifact sources, in preference order:
21
-
22
- 1. **ARIA snapshot** — structured, no styling noise, easy duplicate/accessibility-name scanning
23
- 2. **Screenshot** — visual confirmation; catches layout breaks ARIA can't show
24
- 3. **HTML** — only when ARIA lacks context (custom widgets without accessible names, attribute-driven behaviour)
25
-
26
- ## Inspect an element
27
-
28
- `I.grabWebElement(locator)` → one WebElement; `I.grabWebElements(locator)` → array. Same cross-helper API on Playwright / Puppeteer / WebDriver.
29
-
30
- | You want to … | Method |
31
- |---|---|
32
- | Confirm rendered / visible / enabled | `exists()`, `isVisible()`, `isEnabled()` |
33
- | Read text / value / attribute / property | `getText()`, `getValue()`, `getAttribute(n)`, `getProperty(n)` |
34
- | Position on page | `getBoundingBox()` — flags offscreen / zero-sized |
35
- | Rendered markup | `toOuterHTML()`, `toSimplifiedHTML(300)` (truncated, MCP-friendly) |
36
- | Stable selector for a fix | `toAbsoluteXPath()` |
37
- | Inside an iframe | `inIframe(async (body) => { ... })` |
38
- | Drill into children | `$(loc)`, `$$(loc)` |
39
- | Browser-side function | `evaluate(fn, ...args)` |
40
-
41
- ## Discover candidates when the obvious locator misses
42
-
43
- When `Edit` matches nothing, the control may say "Change", carry `aria-label="Edit user"`, or live in `.btn-edit`. Cast a wide net with a permissive XPath via `I.grabWebElements`, then disambiguate.
44
-
45
- OR together in the XPath:
46
-
47
- - visible text — `text()` (or `.` for descendants)
48
- - attributes — `@class`, `@aria-label`, `@title`, `@data-action`, `@id`
49
- - **synonyms** — edit/change/modify; delete/remove/trash; submit/send/save
50
-
51
- Case-insensitive via `translate(...)`:
52
-
53
- ```
54
- //*[contains(translate(., 'EDIT', 'edit'), 'edit')
55
- or contains(translate(@class, 'EDIT', 'edit'), 'edit')
56
- or contains(translate(@aria-label, 'EDIT', 'edit'), 'edit')
57
- or contains(translate(., 'CHANGE', 'change'), 'change')]
58
- ```
59
-
60
- Then iterate `toSimplifiedHTML(150)` over the results, pick the right candidate, commit a stable locator from its discriminating attribute or text.
61
-
62
- ## Pick a stable locator
63
-
64
- Two decisions in order: **which region scopes the lookup** (context), **what identifies the element inside it**. Region first keeps the identifier short and semantic — the discriminator found during disambiguation belongs in the context argument:
65
-
66
- ```js
67
- I.click('Edit user', '.user-row') // ✅ region + what the user sees
68
- I.click('#user-row-42 button.edit') // ❌ same element, brittle, unreadable
69
- ```
70
-
71
- Stable regions: landmarks (`nav`, `main`, `{ role: 'dialog' }`), app-shell containers (`.sidebar`, `.toolbar`, `.modal`), rows/cards identified by data via `locate(...)`. Identifier priority (full rationale: `codeceptjs-fundamentals` § Locators):
72
-
73
- 1. Visible label / accessible name — plain string already matches `aria-label`; don't expand to `{ css: '[aria-label="..."]' }`
74
- 2. ARIA role when ambiguous within context or role is part of the check
75
- 3. `$name` via `customLocator` when team test attributes exist
76
- 4. Composed CSS, still scoped: `I.click('button.edit', '#user-row-42')`
77
- 5. `toAbsoluteXPath()` — last resort; flag the team to add a `data-testid`
78
-
79
- **Never commit an unverified locator** — confirm via `run_code` (`I.seeElement(loc, context)` or `grabWebElement(loc)`) that it matches exactly one element.
80
-
81
- ## Common patterns
82
-
83
- - Strict mode 2+ matches → `grabWebElements('Save')` + `toSimplifiedHTML(200)` each, find discriminator, pass as context: `I.click('Save', '.modal')`
84
- - Button rendered but doesn't act → `grabWebElement('Submit')` + `isEnabled()` + `getBoundingBox()` — disabled? offscreen? zero-sized?
85
- - Wrong row in a list → `grabWebElements('.user-row')`, `getText()` per row to identify, `getAttribute('data-id')` for stable hook
86
- - Inside iframe → `(await I.grabWebElement('iframe.editor')).inIframe(async (body) => body.$('button'))`
87
-
88
- ## Things to avoid
89
-
90
- - Choosing a locator without seeing candidates first.
91
- - Committing `toAbsoluteXPath()` when a semantic locator is available.
92
- - Committing unscoped locators where a context keeps them short.
93
- - Ignoring the screenshot — "exists in HTML" ≠ "user can see it".
94
- - `usePlaywrightTo` / `useWebDriverTo` when WebElement methods cover it.
95
-
96
- ## Related skills
97
-
98
- - `codeceptjs-fundamentals` — locator priority, await rule
99
- - `writing-codeceptjs-tests` — invokes this during Mode B exploration
100
- - `debugging-codeceptjs-tests` — invokes this for live inspection; offline variant via `codeceptq`
@@ -1,196 +0,0 @@
1
- ---
2
- name: codeceptjs-fundamentals
3
- description: >
4
- Run first when working with any CodeceptJS 4 project — before writing,
5
- debugging, refactoring, or migrating tests. Teaches the framework's
6
- non-obvious rules and runs four-step discovery (`check` → read config →
7
- `list` → `dry-run`) reporting which helpers, plugins, page objects, custom
8
- actions, and tests are active. Other CodeceptJS skills depend on this output.
9
- ---
10
-
11
- # CodeceptJS Fundamentals
12
-
13
- Two jobs, in order: learn the rules below, then discover what *this* project has configured.
14
-
15
- ## Gate
16
-
17
- - CodeceptJS 4 is **ESM/TypeScript only**; tests, configs, page objects, helpers use `import`/`export`.
18
- - No `"type": "module"` in package.json → add it before anything else.
19
- - TypeScript: config `codecept.conf.ts`, TS loader entry in `require: [...]`.
20
- - Project on 3.x or CommonJS (`require()`, removed plugins like `autoLogin`, helper `Nightmare`) → stop, recommend `migrate-codeceptjs-4`. Don't patch files piecemeal — migration is whole-project.
21
-
22
- ## Main rule
23
-
24
- - Tests are written from the user's perspective: a linear scenario of actions, readable as prose.
25
- - Good: `I.click('Login')`, `I.fillField('Email', ...)`, `I.see('Welcome')`
26
- - **Tests are declarative, helpers imperative — recommended layering:**
27
- - Scenario shows *what* the user does via `I.*`; implementation details live below
28
- - Low-level access (`this.helpers['Playwright'].page`, fetch, filesystem) works fine inside a Scenario, but it's recommended to push it into a helper and expose one `I.*` action instead
29
- - Keep tests short: repeated sequences → actor method, page object, or step object.
30
- - Prefer semantic locators over selectors so tests survive markup churn.
31
-
32
- ## Where things go (recommended placement)
33
-
34
- - Site-wide actions (`login`, dropdowns, rich text editors) → **actor file** (custom steps)
35
- - Page/screen actions + locators → **page object**; SPA screen = one page object
36
- - Site-wide widgets (nav, modals, datepickers) → **page fragments / component objects**
37
- - Low-level driver access (DB connections, email, filesystem, complex mouse) → **helper**
38
- - Data creation/cleanup via API → **data objects** (REST/GraphQL helper + `_after()` cleanup), or `ApiDataFactory` (`I.have(...)`)
39
- - Don't overengineer: no page object until an abstraction is reused across tests.
40
-
41
- ## Shortcuts
42
-
43
- - Login needed → `autoLogin` plugin or actor method, not inline steps per test
44
- - Test data → create via API before the test, not through the UI
45
- - Long test → break into several; long tests are fragile and hard to follow
46
- - Optional UI element / conditional flow → `await tryTo(...)` instead of `if (await I.grab...)` — keeps scenarios linear
47
-
48
- ## Architecture
49
-
50
- - **Config**: `codecept.conf.{js,ts,mjs,cjs}` at repo root; multiple files selected via `--config <file>`.
51
- - **Helpers execute; `I` delegates.** Every `I.<method>` is routed to whichever active helper implements it (Playwright, WebDriver, Puppeteer, Appium share one API surface). Active helpers = keys under `helpers`. Tests call the actor, never the engine — backends stay swappable.
52
- - **DI container** — why it exists:
53
- - Everything shared (actor, helpers, page objects, support objects) registers under one container; `include` maps names → modules
54
- - Classes are auto-instantiated by the container — no `new`, no manual wiring
55
- - **`inject()` returns lazy proxies**: destructuring at module top resolves at call time, so circular page-object references work where plain `import` would give `undefined`
56
- - Access: destructure in Scenario signature (`Scenario('...', ({ I, loginPage }) => ...)`) or `const { I } = inject()` once per file
57
- - **Custom helpers** extend `Helper`, register under `helpers`, add new `I.*` methods.
58
- - **`I` does not exist inside a helper.** Compose via `this.helpers['<HelperName>']` (e.g. `this.helpers['Playwright'].page`, `this.helpers['REST'].sendGetRequest(...)`).
59
- - **Plugins** are event listeners on lifecycle events (`suite.*`, `test.*`, `step.*`, `hook.*`, `multiple.*`). Full list: `node_modules/codeceptjs/lib/event.js`. Register under `plugins` with `enabled: true`.
60
-
61
- ## Config mutation trap
62
-
63
- - `@codeceptjs/configure` mutates resolved config at load time (`setHeadlessWhen`, `setBrowser`, ...). Static values can lie — grep for its import before trusting `show:` / `browser:` fields.
64
- - `setCommonPlugins()`: **enables** `retryFailedStep` + `screenshot`; **registers** (off until `-p`) `pause`, `browser`, `aiTrace`, `heal`.
65
-
66
- ## Plugins worth knowing
67
-
68
- - `retryFailedStep` — retries transient step failures
69
- - `screenshot` — screenshots on failure; `slides: true` → `output/records.html` slideshow
70
- - `pageInfo` — dumps URL/HTML/console on failure
71
- - `auth` — session reuse for login (see `codeceptjs-auth` skill)
72
- - `aiTrace` — per-step screenshots/HTML/ARIA/console for AI debugging
73
- - `pause` — interactive pause
74
- - `heal` — AI-suggested fixes for broken steps (off in `--debug`)
75
- - `screencast` — video of the run
76
- - `customLocator` — maps `$name` prefix to team's test attribute (`data-testid`, `data-qa`)
77
- - `browser` — CLI-only override of browser helper config (see below)
78
-
79
- Note: `tryTo`, `retryTo`, `eachElement` are not plugins in 4.x — import them from `codeceptjs/effects`.
80
-
81
- ## Plugins from CLI
82
-
83
- Any plugin can be enabled/reconfigured per-run with `-p <plugin>`, args chained with `:`:
84
-
85
- ```sh
86
- npx codeceptjs run -p aiTrace # enable for this run
87
- npx codeceptjs run -p screenshot:on=step # reconfigure inline
88
- npx codeceptjs run -p pause:on=file:path=tests/login_test.js;line=43
89
- ```
90
-
91
- - `screenshot`, `pause`, `aiTrace`, `heal` share an `on=` trigger: `fail` (default except aiTrace) | `step` | `test` | `file:path=...;line=N` | `url:pattern=<glob>`
92
- - `browser` plugin overrides without touching config — CI matrix legs, one-off env variants:
93
- - `-p browser:hide` / `-p browser:show` / `-p browser:browser=firefox` / `-p browser:windowSize=1280x800`
94
- - Requires `@codeceptjs/configure`
95
-
96
- ## Effects (`codeceptjs/effects`)
97
-
98
- Flow-control functions imported from `codeceptjs/effects`. In 4.x these are no longer plugins/globals.
99
-
100
- - **`tryTo(() => ...)`** — runs steps that may fail without stopping the test; returns `boolean`.
101
- - **Prefer `tryTo` over `if`:** scenarios should stay linear — instead of branching on a grabbed value to decide whether a UI state exists, attempt the optional steps and branch on the boolean result:
102
- ```js
103
- const banner = await tryTo(() => { I.see('Cookie banner'); I.click('Accept cookies') })
104
- if (!banner) I.say('No cookie banner')
105
- ```
106
- - Auto-retries are disabled inside `tryTo` blocks.
107
- - **`retryTo(() => ..., maxTries, pollInterval = 200)`** — retries a step block until it succeeds (flaky elements, animations); callback receives the current attempt count.
108
- - **`hopeThat(() => ...)`** — soft assertions (see Assertions); end with `hopeThat.noErrors()`.
109
- - **`within(locator | { frame }, fn)`** — scopes resolution to subtree or iframe; can return values (`await`). Prefer the context parameter of individual actions (`I.click('Save', '.toolbar')`) when possible — reserve `within` for genuinely scoped blocks.
110
-
111
- All effects return Promises — `await` them.
112
-
113
- ## Element-based API (`codeceptjs/els`)
114
-
115
- Hybrid style: mix `I.*` with direct element access. Import `{ element, eachElement, expectElement, expectAnyElement, expectAllElements } from 'codeceptjs/els'`.
116
-
117
- - `element(locator, async el => { ... })` — scoped access to one element; chain `el.$(locator)` into children without re-querying
118
- - `eachElement(locator, async (el, index) => ...)` — iterate collections
119
- - `expectElement` / `expectAnyElement` / `expectAllElements(locator, fn)` — custom conditions
120
- - Elements are `WebElement` wrappers — same API on all helpers: `getText()`, `getAttribute()`, `isVisible()`, `isEnabled()`, `getBoundingBox()`, `exists()`, `$$()`
121
- - Optional purpose string improves debug logs: `element('verify discount applied', '.price', ...)`
122
- - Use when built-ins don't cover it: collections, layout checks (`getBoundingBox`), per-element loops, chaining ops on one element. Prefer `I.*` for readability otherwise.
123
-
124
- ## Writing tests
125
-
126
- - Structure: one `Feature(...)` per file, one or more `Scenario(...)` inside. No nested suites, no multiple Features per file.
127
- - Hooks: `Before`, `After`, `BeforeSuite`, `AfterSuite`, `Fail(...)`.
128
- - Page object lifecycle hooks: `_before()` (lazy, once per test, on first use), `_after()` (skipped if unused), `_beforeSuite()`, `_afterSuite()`.
129
- - **`await` required for**: `grab*` methods, imported functions, page-object methods containing async ops (elsewhere: unhandled rejections). Never for plain action steps — the recorder chains them.
130
- - Secrets: `I.fillField('Password', secret(process.env.PASSWORD))` — masks logs, traces, AI prompts. Import from `codeceptjs`.
131
- - Sessions: `session(name, fn)` — parallel browser context for multi-user Scenarios (chat, multi-tenant).
132
-
133
- ## Locators
134
-
135
- - **ARIA locators are strongest** — resilient to CSS refactors, describe what the user sees:
136
- - `I.click({ role: 'button', name: 'Save' })`
137
- - Actions accept plain strings (visible text, label, placeholder, `name`, `aria-label`) or objects (`{ css }`, `{ xpath }`, `{ id }`).
138
- - Plain string already matches `aria-label` — no `'aria-label=...'` prefix needed.
139
- - **Pass context as last argument** — scoped semantic locator beats long unscoped one:
140
- - `I.click('Save', '.toolbar')` not `I.click('#toolbar .btn-save')`
141
- - Avoid style-based class names (`.bg-green`); prefer semantic ones (`.btn-save`).
142
- - `data-testid`/`data-qa` apps → enable `customLocator`, write `$name`.
143
- - No semantic name fits → `locate(...)` builder (`.withClass`, `.withText`, `.inside`, `.and`): `locate('.button').withText('Click me')`.
144
-
145
- ## Waiting
146
-
147
- - Action steps auto-wait for existence + interactability. Add explicit `waitFor*` only when the condition isn't tied to an interaction (modal after network call, spinner hiding).
148
- - Avoid `I.wait(N)` — last resort.
149
-
150
- ## Assertions
151
-
152
- Built-in browser assertions come first: `I.see`, `I.seeTextEquals`, `I.seeElement`, `I.seeInField`, `I.seeNumberOfElements`, `I.seeInCurrentUrl` (+ `dontSee*` counterparts). Clear failures, recorder-integrated. `see` matches *visible* text; hidden DOM content needs `seeInSource` / `seeElementInDOM`.
153
-
154
- For what built-ins don't cover, in order of preference:
155
-
156
- 1. **Reusable custom assertion** in a helper — `I.seeTableIsOrdered('Price', 'desc')`; name positives `see*`, negatives `dontSee*`; use `codeceptjs/assertions` inside, never raw `throw new Error()`
157
- 2. **ExpectHelper** (`@codeceptjs/expect-helper`) — chai matchers on `I`: `I.expectEqual`, `I.expectDeepEqualExcluding`, `I.expectMatchesPattern`, `I.expectJsonSchema`; appears in step log like other steps
158
- 3. **`codeceptjs/assertions`** directly — dependency-free factories: `equals(subject).assert(actual, expected)` / `.negate(...)`; failure messages match `I.see` formatting
159
- 4. **Any library** on grabbed data (`grab*` always needs `await`) — chai/jest/`node:assert`; fails the test but won't show as a step
160
-
161
- Soft assertions: `hopeThat(() => I.see(...))` from `codeceptjs/effects` — logs each failure and continues; end with `hopeThat.noErrors()` to fail if any were recorded.
162
-
163
- ## Parallel runs
164
-
165
- - `run-workers <N>` — splits Scenarios across worker threads
166
- - `run-multiple <profile>` — profiles via `multiple` block in config (browsers, viewports)
167
-
168
- ## Config organization (recommended)
169
-
170
- - Multiple config files per environment (`codecept.conf.js`, `codecept.ci.conf.js`, ...); share parts via modules in a `config/` dir
171
- - `.env` files + `dotenv` for secrets/env-specific values
172
- - Bulk-register page objects/components by spreading exported maps into `include`
173
- - Pass data from config/bootstrap into tests via `codeceptjs.container.append({ testUser })` — injectable by name
174
-
175
- ## Discover this project
176
-
177
- In order; skipping steps produces wrong guesses:
178
-
179
- 1. **Verify setup loads**: `npx codeceptjs check -c <config>` — validates everything; output doubles as inventory. Fix failures before continuing.
180
- 2. **Read the active config**: helpers (+ browser/baseURL/viewport/env-driven values), plugins (incl. anything `setCommonPlugins()` injects), AI provider + required env var, env selection mechanism, page objects from `include`, custom helpers.
181
- 3. **List actions**: `npx codeceptjs list -c <config>` (`--docs` adds JSDoc; `--action <name>` for one). The actual `I.*` surface differs from built-ins when custom helpers exist — always check before suggesting a method.
182
- 4. **List tests**: `npx codeceptjs dry-run -c <config>` — `--steps` shows queued actions, `--grep` filters, `--numbers` gives per-test step indices matching MCP `pauseAt`.
183
- - ⚠ `dry-run --grep` and `run --grep` do **not** select the same set (4.1.0): `run --grep` matches `Feature` + `Scenario`, `dry-run --grep` matches the Scenario title only. `dry-run --grep 'Dialogs'` lists 0 tests where `run --grep 'Dialogs'` executes all 11. Never size a run from a dry-run's grep, and target a whole Feature by file path (`run tests/foo_test.ts`) when the selection must be exact.
184
-
185
- Gherkin projects: `npx codeceptjs gherkin:steps -c <config>`.
186
-
187
- Reference docs live under `node_modules/codeceptjs/docs/` — read them instead of guessing APIs.
188
-
189
- ## Report
190
-
191
- Short prose summary. Must include:
192
-
193
- - Env-driven values flagged as env-driven (`process.env.BROWSER || 'chromium'`, not just `'chromium'`)
194
- - Conflicts flagged (static `show: true` vs `setHeadlessWhen(CI)`; `auth` configured but credential env vars missing)
195
- - No config at root and no `--config` referenced → recommend `npx codeceptjs init .`, stop
196
- - 3.x/CommonJS detected → recommend `migrate-codeceptjs-4`, stop
@@ -1,83 +0,0 @@
1
- ---
2
- name: codeceptjs-run-analysis
3
- description: >
4
- Use after running CodeceptJS tests with the `aiTrace` plugin enabled and the
5
- results need review — verify a fix held, investigate a single failure, cluster
6
- errors across a CI fail-storm, diagnose flakiness across reruns. Invoked by
7
- other CodeceptJS skills whenever a run has happened. Trigger on "what failed",
8
- "analyse the run", "cluster these errors", "is it flaky", "did the fix hold".
9
- ---
10
-
11
- # CodeceptJS Run Analysis
12
-
13
- After `npx codeceptjs run`, trace artifacts land in `output/`. This skill reads them efficiently — right step, right file, right slice of a giant HTML snapshot — via bash tools rather than re-running through MCP.
14
-
15
- No single end goal: pick the use case matching the situation. The foundations apply to all of them.
16
-
17
- ## Foundations
18
-
19
- - **aiTrace must be on.** Everything leans on `output/trace_<TestName>_<hash>/trace.md`. Confirm in the active config (run `codeceptjs-fundamentals` if unknown). Without it there are only screenshots and `pageInfo` dumps — suggest enabling and re-running before deep analysis.
20
- - `run_step_by_step` is interactive only; `aiTrace` is the sole source of per-step files. Ad-hoc `run_code` / `snapshot` still produce single-shot bundles under `output/trace_run_code_*` / `output/snapshot_*`.
21
- - **Locate traces**: reruns create new dirs — when unclear, most recent wins (`ls -dt output/trace_*`).
22
- - **Read trace.md first** — it's the index linking each step to its artifacts.
23
- - Focus on the failed step; none marked → last step in the trace.
24
- - Multiple failures marked → the **first** is usually the cause; the rest cascade.
25
-
26
- ### Artifact order for the focus step (`NNNN_<step>`)
27
-
28
- | Artifact | When |
29
- |---|---|
30
- | `NNNN_*_aria.txt` | First read — lean, structured, easy to scan for duplicates |
31
- | `NNNN_*_screenshot.png` | Visual confirmation — layout, animation, "rendered but wrong" |
32
- | `NNNN_*_console.json` | JS errors, 4xx/5xx, deprecation warnings explaining vanished elements |
33
- | `NNNN_*_storage.json` | Cookies + localStorage at this step — first stop when auth suspected |
34
- | `NNNN_*_page.html` | Last resort, and only via `grep` |
35
-
36
- ### Never read big files whole
37
-
38
- - HTML snapshots: search with `grep` (text, class, aria-label, `data-*`) — line numbers + context flags keep matches readable.
39
- - `console.json`: filter with `jq`, don't read every entry.
40
- - Scanning many files: `grep -l` for filenames only.
41
-
42
- ## Use cases
43
-
44
- ### Verify a fix held
45
- Locate latest trace → read trace.md → confirm no FAILED markers. Glance at `console.json` for warnings worth fixing while you're there.
46
-
47
- ### Cluster errors across a CI batch
48
- Extract failing-step lines from every `trace.md`, group by signature (`grep` + `sort` + `uniq -c`), rank by frequency.
49
- - Same error in many tests = **systemic** (env var, auth, base URL, deploy regression) — fix root cause once, rerun the batch.
50
- - Different errors per test = local — triage one at a time.
51
- Start with the most frequent root cause; rerun to see how many tests came back with it.
52
-
53
- ### Diagnose flakiness
54
- Rerun the same test 5–10 times; compare which step failed in each trace.
55
- - Different step each run → timing, environment, external service
56
- - Same step, different state → missing/wrong wait — `diff` the ARIA snapshots of that step between runs
57
- - `console.json` differs between runs → transient backend/network errors
58
- - Bounding box differs → layout reflow or late-loading content
59
-
60
- ### Investigate a single failure
61
- Failed-or-last step → ARIA + screenshot first, console second, HTML last (grep only). Form a hypothesis. Trace not enough / page needs live poking → hand off to `debugging-codeceptjs-tests`.
62
-
63
- ## After analysis
64
-
65
- - Systemic cause → fix root once (env, auth, deploy), not per test
66
- - Locator drift → `codeceptjs-exploration`
67
- - Timing/wait issue → Waiting guidance from fundamentals/writing skills; replace `I.wait(N)` with specific `waitFor*`
68
- - Resists static analysis → `debugging-codeceptjs-tests` (live MCP loop)
69
- - Clean pass → done, but glance at `console.json` anyway
70
-
71
- ## Things to avoid
72
-
73
- - Reading large HTML or `console.json` whole — `grep` / `jq`.
74
- - Stopping at the last marked failure instead of the first.
75
- - Triaging individual failures before clustering.
76
- - Flakiness conclusions from a single run — needs 5+ reruns.
77
- - Deleting `output/` mid-investigation.
78
-
79
- ## Related skills
80
-
81
- - `codeceptjs-fundamentals` — what's configured, aiTrace `-p` overrides
82
- - `codeceptjs-exploration` — locator drift fixes
83
- - `debugging-codeceptjs-tests` — live loop + offline locator resolution via `codeceptq`