testeiya 0.3.9 → 0.3.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +312 -27
- package/dist/prompt/index.js +6 -2
- package/dist/prompt/index.js.map +1 -1
- package/dist/prompt/print.js +8 -0
- package/dist/prompt/print.js.map +1 -1
- package/dist/prompt/system-prompt.js +14 -4
- package/dist/prompt/system-prompt.js.map +1 -1
- package/dist/prompt/testomatio.js +3 -3
- package/dist/prompt/tools.js +15 -7
- package/dist/prompt/tools.js.map +1 -1
- package/dist/src/args.js +147 -47
- package/dist/src/args.js.map +1 -1
- package/dist/src/cli.js +50 -14
- package/dist/src/cli.js.map +1 -1
- package/dist/src/doctor.js +186 -0
- package/dist/src/doctor.js.map +1 -0
- package/dist/src/env.js +29 -5
- package/dist/src/env.js.map +1 -1
- package/dist/src/mcp.js +3 -3
- package/dist/src/mcp.js.map +1 -1
- package/dist/src/model.js +15 -10
- package/dist/src/model.js.map +1 -1
- package/dist/src/models.js +38 -0
- package/dist/src/models.js.map +1 -0
- package/dist/src/output.js +185 -0
- package/dist/src/output.js.map +1 -0
- package/dist/src/run.js +116 -24
- package/dist/src/run.js.map +1 -1
- package/dist/src/session.js +38 -23
- package/dist/src/session.js.map +1 -1
- package/dist/src/sessions.js +61 -0
- package/dist/src/sessions.js.map +1 -0
- package/package.json +7 -3
- package/prompt/index.ts +11 -2
- package/prompt/print.ts +9 -0
- package/prompt/system-prompt.ts +18 -4
- package/prompt/testomatio.ts +3 -3
- package/prompt/tools.ts +15 -7
- package/skills/codeceptjs/ci-fix-tests/SKILL.md +119 -0
- package/skills/codeceptjs/codeceptjs-auth/SKILL.md +141 -0
- package/skills/codeceptjs/codeceptjs-exploration/SKILL.md +100 -0
- package/skills/codeceptjs/codeceptjs-fundamentals/SKILL.md +196 -0
- package/skills/codeceptjs/codeceptjs-run-analysis/SKILL.md +83 -0
- package/skills/codeceptjs/debugging-codeceptjs-tests/SKILL.md +126 -0
- package/skills/codeceptjs/migrate-codeceptjs-4/SKILL.md +109 -0
- package/skills/codeceptjs/migrate-cypress-to-codeceptjs/SKILL.md +295 -0
- package/skills/codeceptjs/migrate-protractor-to-codeceptjs/SKILL.md +339 -0
- package/skills/codeceptjs/migrate-selenium-java-to-codeceptjs/SKILL.md +459 -0
- package/skills/codeceptjs/migrate-testcafe-to-codeceptjs/SKILL.md +338 -0
- package/skills/codeceptjs/refactoring-codeceptjs-tests/SKILL.md +78 -0
- package/skills/codeceptjs/writing-codeceptjs-tests/SKILL.md +102 -0
- package/skills/playwright/playwright-best-practices/SKILL.md +303 -0
- package/skills/playwright/playwright-best-practices/advanced/authentication-flows.md +360 -0
- package/skills/playwright/playwright-best-practices/advanced/authentication.md +871 -0
- package/skills/playwright/playwright-best-practices/advanced/clock-mocking.md +364 -0
- package/skills/playwright/playwright-best-practices/advanced/mobile-testing.md +409 -0
- package/skills/playwright/playwright-best-practices/advanced/multi-context.md +288 -0
- package/skills/playwright/playwright-best-practices/advanced/multi-user.md +393 -0
- package/skills/playwright/playwright-best-practices/advanced/network-advanced.md +452 -0
- package/skills/playwright/playwright-best-practices/advanced/third-party.md +464 -0
- package/skills/playwright/playwright-best-practices/architecture/pom-vs-fixtures.md +363 -0
- package/skills/playwright/playwright-best-practices/architecture/test-architecture.md +369 -0
- package/skills/playwright/playwright-best-practices/architecture/when-to-mock.md +383 -0
- package/skills/playwright/playwright-best-practices/browser-apis/browser-apis.md +391 -0
- package/skills/playwright/playwright-best-practices/browser-apis/iframes.md +403 -0
- package/skills/playwright/playwright-best-practices/browser-apis/service-workers.md +504 -0
- package/skills/playwright/playwright-best-practices/browser-apis/websockets.md +403 -0
- package/skills/playwright/playwright-best-practices/core/annotations.md +424 -0
- package/skills/playwright/playwright-best-practices/core/assertions-waiting.md +361 -0
- package/skills/playwright/playwright-best-practices/core/configuration.md +452 -0
- package/skills/playwright/playwright-best-practices/core/fixtures-hooks.md +417 -0
- package/skills/playwright/playwright-best-practices/core/global-setup.md +434 -0
- package/skills/playwright/playwright-best-practices/core/locators.md +242 -0
- package/skills/playwright/playwright-best-practices/core/page-object-model.md +315 -0
- package/skills/playwright/playwright-best-practices/core/projects-dependencies.md +453 -0
- package/skills/playwright/playwright-best-practices/core/test-data.md +492 -0
- package/skills/playwright/playwright-best-practices/core/test-suite-structure.md +361 -0
- package/skills/playwright/playwright-best-practices/core/test-tags.md +298 -0
- package/skills/playwright/playwright-best-practices/debugging/console-errors.md +420 -0
- package/skills/playwright/playwright-best-practices/debugging/debugging.md +504 -0
- package/skills/playwright/playwright-best-practices/debugging/error-testing.md +360 -0
- package/skills/playwright/playwright-best-practices/debugging/flaky-tests.md +496 -0
- package/skills/playwright/playwright-best-practices/frameworks/angular.md +530 -0
- package/skills/playwright/playwright-best-practices/frameworks/nextjs.md +469 -0
- package/skills/playwright/playwright-best-practices/frameworks/react.md +531 -0
- package/skills/playwright/playwright-best-practices/frameworks/vue.md +574 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/ci-cd.md +468 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/docker.md +283 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/github-actions.md +546 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/gitlab.md +397 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/other-providers.md +521 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/parallel-sharding.md +371 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/performance.md +453 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/reporting.md +424 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/test-coverage.md +497 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/accessibility.md +359 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/api-testing.md +719 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/browser-extensions.md +506 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/canvas-webgl.md +493 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/component-testing.md +500 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/drag-drop.md +576 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/electron.md +509 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/file-operations.md +377 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/file-upload-download.md +562 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/forms-validation.md +561 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/graphql-testing.md +331 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/i18n.md +508 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/performance-testing.md +476 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/security-testing.md +430 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/visual-regression.md +634 -0
- package/skills/playwright/playwright-cli/SKILL.md +420 -0
- package/skills/playwright/playwright-cli/references/element-attributes.md +23 -0
- package/skills/playwright/playwright-cli/references/playwright-tests.md +39 -0
- package/skills/playwright/playwright-cli/references/request-mocking.md +87 -0
- package/skills/playwright/playwright-cli/references/running-code.md +241 -0
- package/skills/playwright/playwright-cli/references/session-management.md +225 -0
- package/skills/playwright/playwright-cli/references/storage-state.md +275 -0
- package/skills/playwright/playwright-cli/references/test-generation.md +433 -0
- package/skills/playwright/playwright-cli/references/tracing.md +139 -0
- package/skills/playwright/playwright-cli/references/video-recording.md +143 -0
- package/skills/skills.lock.json +39 -36
- package/skills/testomatio/explorbot/explorbot-fundamentals/SKILL.md +76 -0
- package/skills/testomatio/explorbot/explorbot-fundamentals/references/no-install.md +36 -0
- package/skills/testomatio/explorbot/explorbot-plan/SKILL.md +108 -0
- package/skills/testomatio/explorbot/explorbot-setup/SKILL.md +86 -0
- package/skills/testomatio/qa-process/qa-explain-behavior/SKILL.md +111 -0
- package/skills/testomatio/qa-process/qa-explain-behavior/references/answer-shapes.md +116 -0
- package/skills/testomatio/qa-process/qa-explain-behavior/references/examples.md +157 -0
- package/skills/testomatio/qa-process/qa-lead-strategy-advisor/SKILL.md +91 -0
- package/skills/testomatio/qa-process/qa-lead-strategy-advisor/references/output-format.md +71 -0
- package/skills/testomatio/qa-process/testing-workflow/SKILL.md +227 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/CLAUDE.md +56 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/SKILL.md +242 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/references/CODECEPTJS_BEST_PRACTICES.md +182 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/references/FINAL_SUMMARY_TEMPLATE.md +22 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/references/PLAYWRIGHT_BEST_PRACTICES.md +90 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/references/POM_BEST_PRACTICES.md +53 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/references/TEST_DATA_MANAGEMENT.md +52 -0
- package/skills/testomatio/test-automation/debug-fix-failed-flaky-autotests/SKILL.md +107 -0
- package/skills/testomatio/test-automation/debug-fix-failed-flaky-autotests/references/DEBUGGING_QUICK_REFERENCE.md +113 -0
- package/skills/testomatio/test-automation/qa-automation-test-consolidation/SKILL.md +29 -0
- package/skills/testomatio/test-automation/qa-data-seeder/SKILL.md +83 -0
- package/skills/testomatio/test-automation/run-tests-with-testomatio-reporter/SKILL.md +153 -0
- package/skills/testomatio/test-automation/setup-change-aware-pr-testing/SKILL.md +190 -0
- package/skills/testomatio/test-automation/setup-ci-automation/SKILL.md +76 -0
- package/skills/testomatio/test-automation/testomat-allure-adapter/SKILL.md +289 -0
- package/skills/testomatio/test-management/detect-duplicate-test-cases/SKILL.md +86 -0
- package/skills/testomatio/test-management/detect-duplicate-test-cases/references/DUPLICATE_INSTRUCTIONS.md +54 -0
- package/skills/testomatio/test-management/improve-test-cases/SKILL.md +114 -0
- package/skills/testomatio/test-management/improve-test-cases/references/TESTOMAT_MARKDOWN_EXAMPLE.md +66 -0
- package/skills/testomatio/test-management/pull-request-diff-analyzer/SKILL.md +141 -0
- package/skills/testomatio/test-management/qa-e2e-tests-reporting/SKILL.md +354 -0
- package/skills/testomatio/test-management/qa-e2e-tests-reporting/references/TESTOMATIO_ARTIFACTS.md +82 -0
- package/skills/testomatio/test-management/qa-e2e-tests-reporting/references/TESTOMATIO_HTML_REPORT.md +37 -0
- package/skills/testomatio/test-management/qa-e2e-tests-reporting/references/TESTOMATIO_REPORTERS_CONFIG.md +118 -0
- package/skills/testomatio/test-management/qa-pr-requirements-analyzer/SKILL.md +149 -0
- package/skills/testomatio/test-management/qa-pr-requirements-analyzer/references/summary-example.md +54 -0
- package/skills/testomatio/test-management/qa-requirement-reviewer/SKILL.md +126 -0
- package/skills/testomatio/test-management/qa-requirement-reviewer/references/requirements_reviewer_examples.md +128 -0
- package/skills/testomatio/test-management/qa-split-testing-levels-pyramid/SKILL.md +55 -0
- package/skills/testomatio/test-management/qa-sprint-report-by-testomatio/SKILL.md +220 -0
- package/skills/testomatio/test-management/qa-sprint-report-by-testomatio/references/qa-sprint-report.md +126 -0
- package/skills/testomatio/test-management/qa-test-code-coverage/SKILL.md +174 -0
- package/skills/testomatio/test-management/qa-test-code-coverage/references/COVERAGE_FILE_FORMAT.md +138 -0
- package/skills/testomatio/test-management/qa-test-code-coverage/references/E2E_FRAMEWORKS.md +65 -0
- package/skills/testomatio/test-management/qa-test-code-coverage/scripts/check-coverage.mjs +40 -0
- package/skills/testomatio/test-management/qa-thinking/SKILL.md +36 -0
- package/skills/testomatio/test-management/qa-write-test-cases/SKILL.md +267 -0
- package/skills/testomatio/test-management/qa-write-test-cases/references/test-case-format.md +387 -0
- package/skills/testomatio/test-management/qa-write-test-cases/references/testomat-tms-guide.md +72 -0
- package/skills/testomatio/test-management/qa-write-test-cases/references/writing-rule.md +159 -0
- package/skills/testomatio/test-management/scan-automation-project/SKILL.md +152 -0
- package/skills/testomatio/test-management/sync-test-cases-with-tms/SKILL.md +107 -0
- package/skills/testomatio/test-management/sync-test-cases-with-tms/references/TESTOMATIO_CLI.md +140 -0
- package/skills/testomatio/test-management/testomatio-mcp/SKILL.md +195 -0
- package/skills/testomatio/test-management/testomatio-mcp/references/MCP_SETUP.md +177 -0
package/dist/src/session.js
CHANGED
|
@@ -1,35 +1,56 @@
|
|
|
1
1
|
import { join } from "node:path";
|
|
2
|
-
import { createAgentSession, DefaultResourceLoader, ModelRuntime,
|
|
2
|
+
import { createAgentSession, DefaultResourceLoader, ModelRuntime, SettingsManager, } from "@earendil-works/pi-coding-agent";
|
|
3
3
|
import { buildSystemPrompt } from "../prompt/index.js";
|
|
4
|
-
import { PI_STATE_DIR, TESTEIYA_HOME } from "./env.js";
|
|
4
|
+
import { PACKAGE_ROOT, PI_STATE_DIR, TESTEIYA_HOME } from "./env.js";
|
|
5
5
|
import { hasMcp, tmsAccess } from "./mcp.js";
|
|
6
6
|
import { applyEnvKeys, resolveModel } from "./model.js";
|
|
7
7
|
import { createSetResultTool } from "./result.js";
|
|
8
|
-
|
|
9
|
-
//
|
|
10
|
-
|
|
8
|
+
import { sessionModel } from "./sessions.js";
|
|
9
|
+
// From the package root, never this file's own directory: `../skills` would
|
|
10
|
+
// resolve to dist/skills, and the filter below would silently drop everything.
|
|
11
|
+
export const BUNDLED_SKILLS_DIR = join(PACKAGE_ROOT, "skills");
|
|
11
12
|
const MCP_EXTENSION = join(import.meta.dirname, "mcp-extension.js");
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
// environment rather than from anything we pass in.
|
|
15
|
-
process.env.PI_CODING_AGENT_DIR = PI_STATE_DIR;
|
|
13
|
+
/** The one runtime factory, so every command reads the same auth. */
|
|
14
|
+
export async function createRuntime() {
|
|
16
15
|
const runtime = await ModelRuntime.create({
|
|
17
16
|
authPath: join(TESTEIYA_HOME, "auth.json"),
|
|
18
17
|
modelsPath: null,
|
|
19
18
|
});
|
|
20
19
|
await applyEnvKeys(runtime);
|
|
21
|
-
|
|
20
|
+
return runtime;
|
|
21
|
+
}
|
|
22
|
+
/**
|
|
23
|
+
* The loader every command shares, so `doctor` reports the skills a run would
|
|
24
|
+
* actually load. A filter, not a loader: pi also discovers skills from the
|
|
25
|
+
* checkout it is pointed at (`.agents/skills`, `.pi/skills`, and the user's
|
|
26
|
+
* own), and an arbitrary CI clone must not be able to hand the model its own.
|
|
27
|
+
*/
|
|
28
|
+
export function createLoader(options) {
|
|
29
|
+
return new DefaultResourceLoader({
|
|
30
|
+
cwd: options.cwd,
|
|
31
|
+
agentDir: PI_STATE_DIR,
|
|
32
|
+
settingsManager: options.settingsManager,
|
|
33
|
+
additionalSkillPaths: [BUNDLED_SKILLS_DIR],
|
|
34
|
+
additionalExtensionPaths: options.extensionPaths,
|
|
35
|
+
systemPromptOverride: options.systemPrompt,
|
|
36
|
+
skillsOverride: (current) => ({
|
|
37
|
+
skills: current.skills.filter((s) => s.baseDir.startsWith(BUNDLED_SKILLS_DIR)),
|
|
38
|
+
diagnostics: current.diagnostics,
|
|
39
|
+
}),
|
|
40
|
+
});
|
|
41
|
+
}
|
|
42
|
+
export async function createTesteiyaSession(options) {
|
|
43
|
+
const runtime = await createRuntime();
|
|
44
|
+
const model = resolveModel(runtime, options.model, sessionModel(options.sessionManager));
|
|
22
45
|
const settingsManager = SettingsManager.inMemory();
|
|
23
46
|
const extensionPaths = [];
|
|
24
47
|
if (hasMcp())
|
|
25
48
|
extensionPaths.push(MCP_EXTENSION);
|
|
26
|
-
const loader =
|
|
49
|
+
const loader = createLoader({
|
|
27
50
|
cwd: options.cwd,
|
|
28
|
-
agentDir: PI_STATE_DIR,
|
|
29
51
|
settingsManager,
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
systemPromptOverride: () => buildSystemPrompt({
|
|
52
|
+
extensionPaths,
|
|
53
|
+
systemPrompt: () => buildSystemPrompt({
|
|
33
54
|
cwd: options.cwd,
|
|
34
55
|
mode: "print",
|
|
35
56
|
tms: tmsAccess(),
|
|
@@ -37,13 +58,7 @@ export async function createTesteiyaSession(options) {
|
|
|
37
58
|
connection: options.connection,
|
|
38
59
|
backendUrl: options.backendUrl,
|
|
39
60
|
outputFile: options.outputFile,
|
|
40
|
-
|
|
41
|
-
// A filter, not a loader: pi also discovers skills from the checkout it is
|
|
42
|
-
// pointed at (`.agents/skills`, `.pi/skills`, and the user's own), and an
|
|
43
|
-
// arbitrary CI clone must not be able to hand the model its own skills.
|
|
44
|
-
skillsOverride: (current) => ({
|
|
45
|
-
skills: current.skills.filter((s) => s.baseDir.startsWith(BUNDLED_SKILLS_DIR)),
|
|
46
|
-
diagnostics: current.diagnostics,
|
|
61
|
+
brief: options.brief,
|
|
47
62
|
}),
|
|
48
63
|
});
|
|
49
64
|
await loader.reload();
|
|
@@ -54,7 +69,7 @@ export async function createTesteiyaSession(options) {
|
|
|
54
69
|
modelRuntime: runtime,
|
|
55
70
|
resourceLoader: loader,
|
|
56
71
|
settingsManager,
|
|
57
|
-
sessionManager:
|
|
72
|
+
sessionManager: options.sessionManager,
|
|
58
73
|
customTools: [createSetResultTool(options.result)],
|
|
59
74
|
// Extensions start their runtime on this event. Without it the MCP adapter
|
|
60
75
|
// registers its tools but never connects, and every call answers
|
package/dist/src/session.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"session.js","sourceRoot":"","sources":["../../src/session.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EACL,kBAAkB,EAClB,qBAAqB,EACrB,YAAY,EACZ,
|
|
1
|
+
{"version":3,"file":"session.js","sourceRoot":"","sources":["../../src/session.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EACL,kBAAkB,EAClB,qBAAqB,EACrB,YAAY,EACZ,eAAe,GAGhB,MAAM,iCAAiC,CAAC;AACzC,OAAO,EAAE,iBAAiB,EAAE,MAAM,oBAAoB,CAAC;AACvD,OAAO,EAAE,YAAY,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACrE,OAAO,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC;AAC7C,OAAO,EAAE,YAAY,EAAE,YAAY,EAAE,MAAM,YAAY,CAAC;AACxD,OAAO,EAAE,mBAAmB,EAAkB,MAAM,aAAa,CAAC;AAClE,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAE7C,4EAA4E;AAC5E,+EAA+E;AAC/E,MAAM,CAAC,MAAM,kBAAkB,GAAG,IAAI,CAAC,YAAY,EAAE,QAAQ,CAAC,CAAC;AAE/D,MAAM,aAAa,GAAG,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,OAAO,EAAE,kBAAkB,CAAC,CAAC;AAEpE,qEAAqE;AACrE,MAAM,CAAC,KAAK,UAAU,aAAa;IACjC,MAAM,OAAO,GAAG,MAAM,YAAY,CAAC,MAAM,CAAC;QACxC,QAAQ,EAAE,IAAI,CAAC,aAAa,EAAE,WAAW,CAAC;QAC1C,UAAU,EAAE,IAAI;KACjB,CAAC,CAAC;IACH,MAAM,YAAY,CAAC,OAAO,CAAC,CAAC;IAC5B,OAAO,OAAO,CAAC;AACjB,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,YAAY,CAAC,OAAsB;IACjD,OAAO,IAAI,qBAAqB,CAAC;QAC/B,GAAG,EAAE,OAAO,CAAC,GAAG;QAChB,QAAQ,EAAE,YAAY;QACtB,eAAe,EAAE,OAAO,CAAC,eAAe;QACxC,oBAAoB,EAAE,CAAC,kBAAkB,CAAC;QAC1C,wBAAwB,EAAE,OAAO,CAAC,cAAc;QAChD,oBAAoB,EAAE,OAAO,CAAC,YAAY;QAC1C,cAAc,EAAE,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC;YAC5B,MAAM,EAAE,OAAO,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,UAAU,CAAC,kBAAkB,CAAC,CAAC;YAC9E,WAAW,EAAE,OAAO,CAAC,WAAW;SACjC,CAAC;KACH,CAAC,CAAC;AACL,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,qBAAqB,CAAC,OAAuB;IACjE,MAAM,OAAO,GAAG,MAAM,aAAa,EAAE,CAAC;IACtC,MAAM,KAAK,GAAG,YAAY,CAAC,OAAO,EAAE,OAAO,CAAC,KAAK,EAAE,YAAY,CAAC,OAAO,CAAC,cAAc,CAAC,CAAC,CAAC;IAEzF,MAAM,eAAe,GAAG,eAAe,CAAC,QAAQ,EAAE,CAAC;IAEnD,MAAM,cAAc,GAAa,EAAE,CAAC;IACpC,IAAI,MAAM,EAAE;QAAE,cAAc,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC;IAEjD,MAAM,MAAM,GAAG,YAAY,CAAC;QAC1B,GAAG,EAAE,OAAO,CAAC,GAAG;QAChB,eAAe;QACf,cAAc;QACd,YAAY,EAAE,GAAG,EAAE,CACjB,iBAAiB,CAAC;YAChB,GAAG,EAAE,OAAO,CAAC,GAAG;YAChB,IAAI,EAAE,OAAO;YACb,GAAG,EAAE,SAAS,EAAE;YAChB,MAAM,EAAE,OAAO,CAAC,MAAM;YACtB,UAAU,EAAE,OAAO,CAAC,UAAU;YAC9B,UAAU,EAAE,OAAO,CAAC,UAAU;YAC9B,UAAU,EAAE,OAAO,CAAC,UAAU;YAC9B,KAAK,EAAE,OAAO,CAAC,KAAK;SACrB,CAAC;KACL,CAAC,CAAC;IACH,MAAM,MAAM,CAAC,MAAM,EAAE,CAAC;IAEtB,MAAM,EAAE,OAAO,EAAE,GAAG,MAAM,kBAAkB,CAAC;QAC3C,GAAG,EAAE,OAAO,CAAC,GAAG;QAChB,QAAQ,EAAE,YAAY;QACtB,KAAK;QACL,YAAY,EAAE,OAAO;QACrB,cAAc,EAAE,MAAM;QACtB,eAAe;QACf,cAAc,EAAE,OAAO,CAAC,cAAc;QACtC,WAAW,EAAE,CAAC,mBAAmB,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC;QAClD,2EAA2E;QAC3E,iEAAiE;QACjE,yBAAyB;QACzB,iBAAiB,EAAE,EAAE,IAAI,EAAE,eAAe,EAAE,MAAM,EAAE,SAAS,EAAE;KAChE,CAAC,CAAC;IAEH,2EAA2E;IAC3E,wEAAwE;IACxE,6EAA6E;IAC7E,qDAAqD;IACrD,MAAM,OAAO,CAAC,cAAc,CAAC;QAC3B,IAAI,EAAE,OAAO;QACb,OAAO,EAAE,CAAC,GAAG,EAAE,EAAE;YACf,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,sBAAsB,GAAG,CAAC,aAAa,MAAM,GAAG,CAAC,KAAK,IAAI,CAAC,CAAC;QACnF,CAAC;KACF,CAAC,CAAC;IAEH,MAAM,MAAM,GAAG,MAAM,CAAC,SAAS,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC;IAChD,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,GAAG,KAAK,CAAC,QAAQ,IAAI,KAAK,CAAC,EAAE,EAAE,EAAE,MAAM,EAAE,CAAC;AACrE,CAAC"}
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
import { buildSessionContext, SessionManager } from "@earendil-works/pi-coding-agent";
|
|
2
|
+
/**
|
|
3
|
+
* Sessions are pi's, not ours: it stores, lists and reopens them under
|
|
4
|
+
* PI_STATE_DIR. `--no-session` keeps a run entirely in memory.
|
|
5
|
+
*/
|
|
6
|
+
export async function openSessionManager(cwd, args) {
|
|
7
|
+
if (args.noSession)
|
|
8
|
+
return SessionManager.inMemory(cwd);
|
|
9
|
+
if (args.resume) {
|
|
10
|
+
const sessions = await SessionManager.list(cwd);
|
|
11
|
+
const matched = sessions.filter((s) => s.id.startsWith(args.resume ?? ""));
|
|
12
|
+
if (matched.length === 0)
|
|
13
|
+
return `no session "${args.resume}" in this folder`;
|
|
14
|
+
if (matched.length > 1) {
|
|
15
|
+
const ids = matched.map((s) => shortId(s.id)).join(", ");
|
|
16
|
+
return `"${args.resume}" matches ${matched.length} sessions: ${ids}`;
|
|
17
|
+
}
|
|
18
|
+
return SessionManager.open(matched[0].path);
|
|
19
|
+
}
|
|
20
|
+
if (args.continueLast)
|
|
21
|
+
return SessionManager.continueRecent(cwd);
|
|
22
|
+
return SessionManager.create(cwd);
|
|
23
|
+
}
|
|
24
|
+
/** The model a resumed session was last running, so `-c` needs no --model. */
|
|
25
|
+
export function sessionModel(manager) {
|
|
26
|
+
const entries = manager.getEntries();
|
|
27
|
+
if (entries.length === 0)
|
|
28
|
+
return null;
|
|
29
|
+
return buildSessionContext(entries, manager.getLeafId()).model;
|
|
30
|
+
}
|
|
31
|
+
export async function runSessions(cwd, json) {
|
|
32
|
+
const sessions = await SessionManager.list(cwd);
|
|
33
|
+
if (json) {
|
|
34
|
+
const rows = sessions.map((s) => ({
|
|
35
|
+
id: s.id,
|
|
36
|
+
name: s.name ?? null,
|
|
37
|
+
modified: s.modified.toISOString(),
|
|
38
|
+
messages: s.messageCount,
|
|
39
|
+
first: s.firstMessage,
|
|
40
|
+
}));
|
|
41
|
+
process.stdout.write(`${JSON.stringify({ sessions: rows }, null, 2)}\n`);
|
|
42
|
+
return 0;
|
|
43
|
+
}
|
|
44
|
+
if (sessions.length === 0) {
|
|
45
|
+
process.stdout.write(" no saved sessions in this folder\n");
|
|
46
|
+
return 0;
|
|
47
|
+
}
|
|
48
|
+
process.stdout.write("\n");
|
|
49
|
+
for (const session of sessions) {
|
|
50
|
+
const label = session.name ?? session.firstMessage.slice(0, 60);
|
|
51
|
+
const when = session.modified.toISOString().slice(0, 16).replace("T", " ");
|
|
52
|
+
process.stdout.write(` ${shortId(session.id)} ${when} ${session.messageCount} msg ${label}\n`);
|
|
53
|
+
}
|
|
54
|
+
process.stdout.write("\n Continue one with: testeiya task \"...\" --resume <id>\n\n");
|
|
55
|
+
return 0;
|
|
56
|
+
}
|
|
57
|
+
/** Long enough to clear the UUIDv7 timestamp prefix, which repeats per minute. */
|
|
58
|
+
export function shortId(id) {
|
|
59
|
+
return id.slice(0, 13);
|
|
60
|
+
}
|
|
61
|
+
//# sourceMappingURL=sessions.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"sessions.js","sourceRoot":"","sources":["../../src/sessions.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,mBAAmB,EAAE,cAAc,EAAE,MAAM,iCAAiC,CAAC;AAEtF;;;GAGG;AACH,MAAM,CAAC,KAAK,UAAU,kBAAkB,CACtC,GAAW,EACX,IAAiB;IAEjB,IAAI,IAAI,CAAC,SAAS;QAAE,OAAO,cAAc,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC;IAExD,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;QAChB,MAAM,QAAQ,GAAG,MAAM,cAAc,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAChD,MAAM,OAAO,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,UAAU,CAAC,IAAI,CAAC,MAAM,IAAI,EAAE,CAAC,CAAC,CAAC;QAC3E,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,eAAe,IAAI,CAAC,MAAM,kBAAkB,CAAC;QAC9E,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YACvB,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YACzD,OAAO,IAAI,IAAI,CAAC,MAAM,aAAa,OAAO,CAAC,MAAM,cAAc,GAAG,EAAE,CAAC;QACvE,CAAC;QACD,OAAO,cAAc,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAE,CAAC,IAAI,CAAC,CAAC;IAC/C,CAAC;IAED,IAAI,IAAI,CAAC,YAAY;QAAE,OAAO,cAAc,CAAC,cAAc,CAAC,GAAG,CAAC,CAAC;IACjE,OAAO,cAAc,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;AACpC,CAAC;AAED,8EAA8E;AAC9E,MAAM,UAAU,YAAY,CAAC,OAAuB;IAClD,MAAM,OAAO,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC;IACrC,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,IAAI,CAAC;IACtC,OAAO,mBAAmB,CAAC,OAAO,EAAE,OAAO,CAAC,SAAS,EAAE,CAAC,CAAC,KAAK,CAAC;AACjE,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,WAAW,CAAC,GAAW,EAAE,IAAc;IAC3D,MAAM,QAAQ,GAAG,MAAM,cAAc,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAEhD,IAAI,IAAI,EAAE,CAAC;QACT,MAAM,IAAI,GAAG,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;YAChC,EAAE,EAAE,CAAC,CAAC,EAAE;YACR,IAAI,EAAE,CAAC,CAAC,IAAI,IAAI,IAAI;YACpB,QAAQ,EAAE,CAAC,CAAC,QAAQ,CAAC,WAAW,EAAE;YAClC,QAAQ,EAAE,CAAC,CAAC,YAAY;YACxB,KAAK,EAAE,CAAC,CAAC,YAAY;SACtB,CAAC,CAAC,CAAC;QACJ,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,EAAE,QAAQ,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;QACzE,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,QAAQ,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC1B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,sCAAsC,CAAC,CAAC;QAC7D,OAAO,CAAC,CAAC;IACX,CAAC;IAED,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAC3B,KAAK,MAAM,OAAO,IAAI,QAAQ,EAAE,CAAC;QAC/B,MAAM,KAAK,GAAG,OAAO,CAAC,IAAI,IAAI,OAAO,CAAC,YAAY,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QAChE,MAAM,IAAI,GAAG,OAAO,CAAC,QAAQ,CAAC,WAAW,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,OAAO,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC;QAC3E,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,KAAK,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC,KAAK,IAAI,KAAK,OAAO,CAAC,YAAY,SAAS,KAAK,IAAI,CAAC,CAAC;IACrG,CAAC;IACD,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,iEAAiE,CAAC,CAAC;IACxF,OAAO,CAAC,CAAC;AACX,CAAC;AAED,kFAAkF;AAClF,MAAM,UAAU,OAAO,CAAC,EAAU;IAChC,OAAO,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;AACzB,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "testeiya",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.10",
|
|
4
4
|
"description": "AI testing agent — QA-focused coding agent for manual and automated tests",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -28,10 +28,12 @@
|
|
|
28
28
|
},
|
|
29
29
|
"scripts": {
|
|
30
30
|
"start": "npm run build && node dist/src/cli.js",
|
|
31
|
+
"dev": "./testeiya",
|
|
31
32
|
"build": "tsc -p tsconfig.json && npm run build:vendor",
|
|
32
33
|
"build:vendor": "esbuild scripts/vendor-entry.ts --bundle --format=esm --platform=node --packages=external --outfile=dist/vendor/mcp.js",
|
|
33
34
|
"typecheck": "tsc -p tsconfig.json --noEmit",
|
|
34
|
-
"prepublishOnly": "npm run build"
|
|
35
|
+
"prepublishOnly": "npm run build",
|
|
36
|
+
"test": "npm run build && node --test test/*.test.js"
|
|
35
37
|
},
|
|
36
38
|
"keywords": [
|
|
37
39
|
"testing",
|
|
@@ -54,6 +56,8 @@
|
|
|
54
56
|
"devDependencies": {
|
|
55
57
|
"@types/node": "^22.19.19",
|
|
56
58
|
"esbuild": "^0.28.1",
|
|
57
|
-
"
|
|
59
|
+
"tsx": "^4.23.12",
|
|
60
|
+
"typescript": "^5.9.3",
|
|
61
|
+
"yaml": "^2.9.0"
|
|
58
62
|
}
|
|
59
63
|
}
|
package/prompt/index.ts
CHANGED
|
@@ -6,7 +6,7 @@ import {
|
|
|
6
6
|
projectSettings,
|
|
7
7
|
type TmsAccess,
|
|
8
8
|
} from "./testomatio.js";
|
|
9
|
-
import { nonInteractive, reportOutput } from "./print.js";
|
|
9
|
+
import { briefAnswer, nonInteractive, reportOutput } from "./print.js";
|
|
10
10
|
import {
|
|
11
11
|
contextPromptSection,
|
|
12
12
|
type ContextEntry,
|
|
@@ -28,6 +28,8 @@ export function buildSystemPrompt(options?: SystemPromptOptions): string {
|
|
|
28
28
|
interactive,
|
|
29
29
|
toolBullets: options?.toolBullets,
|
|
30
30
|
rules: options?.rules,
|
|
31
|
+
connectedClis: options?.connectedClis,
|
|
32
|
+
supportedMcps: options?.supportedMcps,
|
|
31
33
|
}),
|
|
32
34
|
];
|
|
33
35
|
|
|
@@ -62,7 +64,8 @@ export function buildSystemPrompt(options?: SystemPromptOptions): string {
|
|
|
62
64
|
parts.push(testomatioNotConnected());
|
|
63
65
|
}
|
|
64
66
|
|
|
65
|
-
// The
|
|
67
|
+
// The answer contract goes last so it is the final instruction the model reads.
|
|
68
|
+
if (options?.brief) parts.push(briefAnswer);
|
|
66
69
|
if (options?.outputFile) parts.push(reportOutput(options.outputFile));
|
|
67
70
|
|
|
68
71
|
return parts.join("\n\n");
|
|
@@ -85,9 +88,15 @@ export interface SystemPromptOptions {
|
|
|
85
88
|
tms?: TmsAccess;
|
|
86
89
|
/** Absolute path the agent must write its final report to (`--output`). */
|
|
87
90
|
outputFile?: string;
|
|
91
|
+
/** Answer a question instead of doing a task and reporting (`testeiya ask`). */
|
|
92
|
+
brief?: boolean;
|
|
88
93
|
projectInfo?: TestomatioProjectInfo | null;
|
|
89
94
|
/** User-added context (linked folders, cloned repos, uploaded docs). */
|
|
90
95
|
contextEntries?: ContextEntry[];
|
|
91
96
|
/** Non-empty predefined `.testeiya` context folders (manual-tests, code, …). */
|
|
92
97
|
contextFolders?: ContextFolder[];
|
|
98
|
+
/** CLI tools the user has connected and signed in (e.g. `gh`, `acli`). */
|
|
99
|
+
connectedClis?: string[];
|
|
100
|
+
/** MCP servers Testeiya supports as connections (the connection catalog). */
|
|
101
|
+
supportedMcps?: string[];
|
|
93
102
|
}
|
package/prompt/print.ts
CHANGED
|
@@ -21,3 +21,12 @@ export function reportOutput(path: string): string {
|
|
|
21
21
|
</final-report>
|
|
22
22
|
`;
|
|
23
23
|
}
|
|
24
|
+
|
|
25
|
+
export const briefAnswer = dedent`
|
|
26
|
+
<answer>
|
|
27
|
+
* You were asked a question, not given a task. Answer it.
|
|
28
|
+
* Lead with the answer in one line, then the evidence you checked.
|
|
29
|
+
* A few sentences. No report file, no headings, no plan.
|
|
30
|
+
* Say plainly when what you found does not settle the question.
|
|
31
|
+
</answer>
|
|
32
|
+
`;
|
package/prompt/system-prompt.ts
CHANGED
|
@@ -9,6 +9,8 @@ export function getSystemPrompt(cwd?: string, options?: PromptSurface): string {
|
|
|
9
9
|
if (!interactive) missingSecretAction = "report it as a blocker in your output";
|
|
10
10
|
let extraRules = "";
|
|
11
11
|
for (const rule of options?.rules ?? []) extraRules += ` * ${rule}\n`;
|
|
12
|
+
let cliList = (options?.connectedClis ?? []).join(", ") || "none connected yet";
|
|
13
|
+
let mcpList = (options?.supportedMcps ?? []).join(", ") || "none connected yet";
|
|
12
14
|
return dedent`
|
|
13
15
|
<role>
|
|
14
16
|
You are Testeiya, an AI agent that helps with QA tasks.
|
|
@@ -87,6 +89,16 @@ export function getSystemPrompt(cwd?: string, options?: PromptSurface): string {
|
|
|
87
89
|
|
|
88
90
|
${tools({ extra: options?.toolBullets })}
|
|
89
91
|
|
|
92
|
+
<connections>
|
|
93
|
+
Testeiya connects external tools for you — CLI tools and MCP servers. This session has:
|
|
94
|
+
|
|
95
|
+
* **Connected CLIs:** ${cliList}
|
|
96
|
+
* **Supported MCP servers:** ${mcpList}
|
|
97
|
+
|
|
98
|
+
* **Missing Connection — Ask, Never Improvise:** When a task needs a tool that is not in the lists above, STOP and ask the user to connect it in Testeiya (Settings → Connections). Never reach the service sideways: no raw REST/GraphQL calls against its API, no scraping credentials from dotfiles or env dumps, no installing binaries on your own.
|
|
99
|
+
* If a task needs both a CLI and an MCP server, ask once for both and say that Testeiya supports them both as connections — the user installs them side by side there.
|
|
100
|
+
</connections>
|
|
101
|
+
|
|
90
102
|
<goals>
|
|
91
103
|
You help in variety of tasks related to software testing, including writing test cases, analyzing test results, and providing feedback.
|
|
92
104
|
You have skills to perform QA tasks on user demand.
|
|
@@ -98,10 +110,8 @@ export function getSystemPrompt(cwd?: string, options?: PromptSurface): string {
|
|
|
98
110
|
* Assisting with manual test execution
|
|
99
111
|
* Manage tests with TMS Testomat.io
|
|
100
112
|
* Setting up CI pipelines for continuous testing
|
|
101
|
-
* Analyzing requirements
|
|
102
|
-
*
|
|
103
|
-
* Launching automated exploratory tests via explorbot
|
|
104
|
-
* Analyzing test results and providing feedback
|
|
113
|
+
* Analyzing requirements and issues
|
|
114
|
+
* Analyzing test results
|
|
105
115
|
|
|
106
116
|
This is what you can do.
|
|
107
117
|
</goals>
|
|
@@ -148,6 +158,10 @@ export interface PromptSurface {
|
|
|
148
158
|
toolBullets?: string[];
|
|
149
159
|
/** Extra `<rules>` bullets, for rules that only hold in this harness. */
|
|
150
160
|
rules?: string[];
|
|
161
|
+
/** CLI tools the user has connected and signed in (e.g. `gh`, `acli`). */
|
|
162
|
+
connectedClis?: string[];
|
|
163
|
+
/** MCP servers Testeiya supports as connections (the connection catalog). */
|
|
164
|
+
supportedMcps?: string[];
|
|
151
165
|
}
|
|
152
166
|
|
|
153
167
|
/*
|
package/prompt/testomatio.ts
CHANGED
|
@@ -35,7 +35,7 @@ const tmsViaDirectTools = dedent`
|
|
|
35
35
|
| Runs, testruns, plans, labels, issues, analytics | MCP \`testomatio-*\` tools |
|
|
36
36
|
| Creating/updating tests or suites | edit the markdown file, then \`npx check-tests push\` |
|
|
37
37
|
| Creating/updating runs, linking issues | MCP create/update tools |
|
|
38
|
-
| Launching automated or mixed Run remotely on CI | Use npx @testomatio/reporter run --remote <profile>, raise error if no CI profiles configured
|
|
38
|
+
| Launching automated or mixed Run remotely on CI | Use npx -y @testomatio/reporter@latest run --remote <profile>, raise error if no CI profiles configured
|
|
39
39
|
| Running automated Run | Use local test runner with testomatio reporter attached to report to current project
|
|
40
40
|
|
|
41
41
|
---
|
|
@@ -73,7 +73,7 @@ const tmsViaProxy = dedent`
|
|
|
73
73
|
| Runs, testruns, plans, labels, issues, analytics | \`mcp({ search: … })\`, then \`mcp({ tool: … })\` |
|
|
74
74
|
| Creating/updating tests or suites | edit the markdown file, then \`npx check-tests push\` |
|
|
75
75
|
| Creating/updating runs, linking issues | the matching \`mcp\` create/update operation |
|
|
76
|
-
| Launching automated or mixed Run remotely on CI | Use npx @testomatio/reporter run --remote <profile>, raise error if no CI profiles configured |
|
|
76
|
+
| Launching automated or mixed Run remotely on CI | Use npx -y @testomatio/reporter@latest run --remote <profile>, raise error if no CI profiles configured |
|
|
77
77
|
| Running automated Run | Use local test runner with testomatio reporter attached to report to current project |
|
|
78
78
|
|
|
79
79
|
---
|
|
@@ -106,7 +106,7 @@ const tmsViaCli = dedent`
|
|
|
106
106
|
| Suite hierarchy / structure / which files exist | \`ls\` / \`find\` |
|
|
107
107
|
| Runs, testruns, plans, labels, issues, analytics | \`curl\` against \`/api/v2\` |
|
|
108
108
|
| Creating/updating tests or suites | edit the markdown file, then \`npx check-tests push\` |
|
|
109
|
-
| Launching automated or mixed Run remotely on CI | Use npx @testomatio/reporter run --remote <profile>, raise error if no CI profiles configured |
|
|
109
|
+
| Launching automated or mixed Run remotely on CI | Use npx -y @testomatio/reporter@latest run --remote <profile>, raise error if no CI profiles configured |
|
|
110
110
|
| Running automated Run | Use local test runner with testomatio reporter attached to report to current project |
|
|
111
111
|
|
|
112
112
|
---
|
package/prompt/tools.ts
CHANGED
|
@@ -11,11 +11,19 @@ import dedent from 'dedent';
|
|
|
11
11
|
*/
|
|
12
12
|
export function tools(options?: { extra?: string[] }): string {
|
|
13
13
|
const extra = (options?.extra ?? []).join('');
|
|
14
|
-
let
|
|
15
|
-
if (
|
|
16
|
-
|
|
14
|
+
let routing = ' - GitHub: issues and projects through MCP.';
|
|
15
|
+
if (onPath('gh')) {
|
|
16
|
+
routing +=
|
|
17
17
|
'\n - Pull requests and repositories: `gh` in `bash`. When a PR question needs issue detail, query the issue through MCP — never infer it from a PR title.';
|
|
18
18
|
}
|
|
19
|
+
if (onPath('acli')) {
|
|
20
|
+
routing +=
|
|
21
|
+
'\n - Jira in bulk (many issues at once), boards, sprints and filters: `acli` in `bash`, per the `atlassian` skill. Confluence and attachments have no `acli` command and stay on MCP.';
|
|
22
|
+
}
|
|
23
|
+
if (onPath('glab')) {
|
|
24
|
+
routing +=
|
|
25
|
+
'\n - GitLab releases, labels, milestones, snippets, CI artifacts, and any endpoint with no MCP tool: `glab` in `bash`. Issues, merge requests, pipelines and code search stay on MCP, which covers them well.';
|
|
26
|
+
}
|
|
19
27
|
return dedent`
|
|
20
28
|
<available-tools>
|
|
21
29
|
You have these tools available:
|
|
@@ -27,7 +35,7 @@ ${extra} * **Internal Skills:** For QA-related tasks, check available skills fi
|
|
|
27
35
|
* **External Integrations:** Use MCP tools when they provide superior data or specialized integrations.
|
|
28
36
|
- Primary MCP (Testomat.io): Use Testomat.io MCP tools to extend functionality.
|
|
29
37
|
- Secondary MCP (Jira, GitHub, etc.): Invoke only when user explicitly asks or when remote context exploration is required.
|
|
30
|
-
${
|
|
38
|
+
${routing}
|
|
31
39
|
|
|
32
40
|
<tool-governance>
|
|
33
41
|
* **Prioritize Specificity:** Use \`read\` instead of \`cat\`, and \`edit\` instead of \`sed\`. Dedicated tools provide better error tracking and safety.
|
|
@@ -39,11 +47,11 @@ ${github}
|
|
|
39
47
|
`;
|
|
40
48
|
}
|
|
41
49
|
|
|
42
|
-
/** Is
|
|
43
|
-
function
|
|
50
|
+
/** Is this CLI on PATH? Never tell the agent to use a binary this machine lacks. */
|
|
51
|
+
function onPath(bin: string): boolean {
|
|
44
52
|
for (const dir of (process.env.PATH ?? '').split(delimiter)) {
|
|
45
53
|
if (!dir) continue;
|
|
46
|
-
if (existsSync(join(dir,
|
|
54
|
+
if (existsSync(join(dir, bin)) || existsSync(join(dir, `${bin}.exe`))) return true;
|
|
47
55
|
}
|
|
48
56
|
return false;
|
|
49
57
|
}
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ci-fix-tests
|
|
3
|
+
description: "Use on CI in non-interactive mode after a failed CodeceptJS run — attempts safe fixes only (locator drift, missing waits), reruns failing scenarios, rolls back edits that didn't help, reports at `output/ci-fix.md`. No refactors, no config edits, no flake-masking. Trigger on \"ci fix tests\", \"auto-fix failing tests\", \"attempt repair on CI\", or as a CI step after a failed run."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Auto-fix CodeceptJS Tests on CI
|
|
7
|
+
|
|
8
|
+
A non-interactive auto-repair loop. Reads the failed-tests baseline from `output/` (where `aiTrace` left trace artifacts), attempts a small set of safe fixes, reruns only the failing scenarios, compares, and writes a report at `output/ci-fix.md`. Rolls back any edit that didn't help.
|
|
9
|
+
|
|
10
|
+
**Conservative by design.** CI is not the place to restructure tests, change abstractions, or fix anything that needs human judgement. The goal is "pass the run". Anything riskier is escalated through the report.
|
|
11
|
+
|
|
12
|
+
This skill assumes a working **git** checkout — rollback uses `git checkout -- <file>` to restore originals.
|
|
13
|
+
|
|
14
|
+
## Workflow
|
|
15
|
+
|
|
16
|
+
### 1. Read the baseline failures
|
|
17
|
+
Use the **codeceptjs-run-analysis** skill against the existing `output/trace_*/` artifacts to enumerate every failing scenario. Record:
|
|
18
|
+
|
|
19
|
+
- the **set** of failing tests (file path + scenario name)
|
|
20
|
+
- the failed step in each, and the locator / wait / assertion involved
|
|
21
|
+
- a cluster signature per failure (so unique vs systemic is clear)
|
|
22
|
+
|
|
23
|
+
Save this as the **baseline** — `count`, `failing_set`, `clusters`.
|
|
24
|
+
|
|
25
|
+
### 2. Pick safe fix candidates
|
|
26
|
+
Only attempt fixes from this allowed list:
|
|
27
|
+
|
|
28
|
+
- **Locator drift** — the failed locator no longer matches anything but a similar element exists. Use the **codeceptjs-exploration** skill (headless) to find candidates; pick a short locator scoped to a stable region — `I.click('Save', '.toolbar')` — in this order: visible text / accessible name → ARIA `{ role, name }` → `$name` (`customLocator`) → composed CSS. Replace the locator at the failing step only.
|
|
29
|
+
- **Missing wait for a spinner / loader / modal** — the failed step's ARIA snapshot shows a spinner or skeleton present, or the target element appears later. Add a single matching `I.waitFor*` immediately before the failing step.
|
|
30
|
+
- **`I.wait(N)` replacement** — when a hardcoded sleep is the only thing between a failing assertion and a passing one and the gating element is identifiable, replace the sleep with a specific `waitFor*`.
|
|
31
|
+
|
|
32
|
+
**Do not attempt:**
|
|
33
|
+
- Auth / credential / env-var fixes — those are CI configuration, not test code.
|
|
34
|
+
- Refactoring (extracting page objects, custom helpers, splitting scenarios).
|
|
35
|
+
- Renaming Scenarios or tags — CI's `--grep` and pipelines reference them.
|
|
36
|
+
- Adding `retries: N` to a Scenario — masking flake is a human decision.
|
|
37
|
+
- Anything outside the failing test file (no `codecept.conf.*`, no helper, no PO edits).
|
|
38
|
+
|
|
39
|
+
If a failure doesn't match any allowed-fix pattern, **skip it** and record it as `unresolved` for the report.
|
|
40
|
+
|
|
41
|
+
### 3. Apply edits and rerun
|
|
42
|
+
Track every modification: file path, line number, before/after snippet, fix kind. Then rerun **only the failing scenarios** under the same config CI used:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
npx codeceptjs run --grep '<scenario>|<scenario>|...'
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Find the right config via `package.json` scripts or the CI workflow file. Match the worker count if `run-workers` was used originally.
|
|
49
|
+
|
|
50
|
+
### 4. Compare against the baseline
|
|
51
|
+
Use **codeceptjs-run-analysis** again to enumerate the failing set after the fix.
|
|
52
|
+
|
|
53
|
+
| New failing-set vs baseline | Status |
|
|
54
|
+
|---|---|
|
|
55
|
+
| Empty (zero failures) | `SUCCESS_FULL` |
|
|
56
|
+
| Strict subset of the baseline | `SUCCESS_PARTIAL` |
|
|
57
|
+
| Identical set | `NO_IMPROVEMENT` |
|
|
58
|
+
| Larger, or contains a test not in baseline | `REGRESSED` |
|
|
59
|
+
|
|
60
|
+
### 5. Decide: keep or roll back
|
|
61
|
+
|
|
62
|
+
- `SUCCESS_FULL` / `SUCCESS_PARTIAL` → **keep** the edits.
|
|
63
|
+
- `NO_IMPROVEMENT` / `REGRESSED` → **roll back every edit** with `git checkout -- <files>`. The on-disk state at the end of the skill must be identical to what it was at the start.
|
|
64
|
+
|
|
65
|
+
A regression is treated the same as no improvement — the attempt didn't work, the working state is what was on disk before. Don't try to "undo only the bad ones" — the whole batch goes.
|
|
66
|
+
|
|
67
|
+
### 6. Write the report (always)
|
|
68
|
+
Write `output/ci-fix.md` regardless of outcome — CI reads it to surface results.
|
|
69
|
+
|
|
70
|
+
## Report format
|
|
71
|
+
|
|
72
|
+
```markdown
|
|
73
|
+
# CI Fix Report — <ISO timestamp>
|
|
74
|
+
|
|
75
|
+
**Status:** SUCCESS_FULL | SUCCESS_PARTIAL | NO_IMPROVEMENT | REGRESSED
|
|
76
|
+
|
|
77
|
+
## Baseline
|
|
78
|
+
Failing scenarios: N
|
|
79
|
+
- `tests/foo_test.js` — "user can log in"
|
|
80
|
+
- `tests/bar_test.js` — "checkout flow"
|
|
81
|
+
- ...
|
|
82
|
+
|
|
83
|
+
## Attempted fixes
|
|
84
|
+
- `tests/foo_test.js:42` — locator drift: `I.click('Save')` → `I.click('Save', '.toolbar')`
|
|
85
|
+
- `tests/bar_test.js:15` — added `I.waitForInvisible('.spinner')` before checkout click
|
|
86
|
+
- `tests/baz_test.js:7` — replaced `I.wait(3)` with `I.waitForVisible('.confirmation-dialog', 10)`
|
|
87
|
+
|
|
88
|
+
## Result
|
|
89
|
+
Failing scenarios after fix: M (was N)
|
|
90
|
+
- Resolved: `tests/foo_test.js` — "user can log in"
|
|
91
|
+
- Still failing: `tests/bar_test.js` — "checkout flow"
|
|
92
|
+
- New failures (REGRESSED only): `tests/baz_test.js` — "edit profile"
|
|
93
|
+
|
|
94
|
+
## Failure clusters (NO_IMPROVEMENT / REGRESSED only)
|
|
95
|
+
- **Cluster A (3 scenarios)** — `Element not found: button.btn-primary`. Likely a deploy regression or design change.
|
|
96
|
+
- **Cluster B (1 scenario)** — `Navigation timeout to /checkout`. Possible backend issue.
|
|
97
|
+
|
|
98
|
+
## Discarded changes (NO_IMPROVEMENT / REGRESSED only)
|
|
99
|
+
- `tests/foo_test.js` — restored
|
|
100
|
+
- `tests/bar_test.js` — restored
|
|
101
|
+
- `tests/baz_test.js` — restored
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
The first line of `Status:` is the machine-parseable signal. The rest is for the human reading the failed CI build.
|
|
105
|
+
|
|
106
|
+
## Things to avoid
|
|
107
|
+
|
|
108
|
+
- Editing files outside the failing test files. Config, helpers, page objects, and CI workflows are off-limits in this mode.
|
|
109
|
+
- Attempting more than one fix kind on the same scenario in one pass. Fix one thing, observe, then try another.
|
|
110
|
+
- `retries: N` to mask flakiness — human decision.
|
|
111
|
+
- Skipping rollback on `NO_IMPROVEMENT`. Even no-op edits can confuse the next run or the next reviewer.
|
|
112
|
+
- Writing a partial or missing `output/ci-fix.md`. CI depends on it; the absence of a report is itself a failure mode.
|
|
113
|
+
- Running the **whole** suite for the verification step — only the originally-failing scenarios, to keep CI time bounded.
|
|
114
|
+
|
|
115
|
+
## Related skills
|
|
116
|
+
|
|
117
|
+
- `codeceptjs-run-analysis` — baseline, clustering, post-fix comparison
|
|
118
|
+
- `codeceptjs-exploration` — replacement locators for drift
|
|
119
|
+
- `codeceptjs-fundamentals` — config, helper, CI environment
|