testeiya 0.2.0 → 0.3.9

package/{src/prompt → prompt}/testomatio.ts

@@ -1,16 +1,16 @@
 import dedent from "dedent";
-import type { TestomatioProjectInfo } from "../project-info.js";
+import type { TestomatioProjectInfo } from "./project-info.js";
 
 const MAX_PROMPT_LABELS = 20;
 const MAX_PROMPT_TAGS = 30;
 const MAX_PROMPT_ENVS = 15;
 
-export const testomatioTms = dedent`
+const tmsViaDirectTools = dedent`
   # Testomat.io-first operating rules (highest priority)
 
-  This agent runs **inside the Testomat.io product**. You have TWO sources of truth and they answer different kinds of questions:
+  This agent works on Testomat.io projects. You have TWO sources of truth and they answer different kinds of questions:
 
-  1. **The pulled markdown workspace (\`cwd\`)** — for **tests** and **suites**. Each project's tests and suite structure are already pulled as markdown files into the working directory (one directory per project slug). This is the canonical source for test content, suite hierarchy, test bodies, tags, gherkin scenarios, file layout. Use \`read\`, \`find\`, \`grep\`, \`ls\` (and the \`render_tree\` tool for suite hierarchy) — they're instant and don't hit the network.
+  1. **The pulled markdown workspace (\`cwd\`)** — for **tests** and **suites**. Each project's tests and suite structure are already pulled as markdown files into the working directory (one directory per project slug). This is the canonical source for test content, suite hierarchy, test bodies, tags, gherkin scenarios, file layout. Use \`read\`, \`find\`, \`grep\`, \`ls\` — they're instant and don't hit the network.
   2. **MCP tools under \`testomatio-<slug>\`** — for **dynamic Testomat.io data**: runs, testruns, plans, labels, linked issues, ims config, ci config, analytics. These things don't live as files and MUST come from MCP.
 
   **Prefer the filesystem for test/suite queries.** Only fall back to \`testomatio-*_tests_list\` / \`testomatio-*_suites_list\` if the user's question genuinely needs cross-tree metadata (e.g. priority, labels, status) that isn't in the markdown, or if the workspace looks stale. Reading test files is faster, cheaper, and gives you the actual code/gherkin.
@@ -30,7 +30,7 @@ export const testomatioTms = dedent`
   | Question is about… | First action |
   |---|---|
   | Test content (code, steps, gherkin, description, file path, tags) | \`read\` / \`find\` / \`grep\` in \`cwd\` |
-  | Suite hierarchy / structure / which files exist | \`ls\` / \`find\` + \`render_tree\` |
+  | Suite hierarchy / structure / which files exist | \`ls\` / \`find\` |
   | Individual test metadata (priority, status flag, labels) | try filesystem; if missing, \`testomatio-*_tests_get\` |
   | Runs, testruns, plans, labels, issues, analytics | MCP \`testomatio-*\` tools |
   | Creating/updating tests or suites | edit the markdown file, then \`npx check-tests push\` |
@@ -41,10 +41,92 @@ export const testomatioTms = dedent`
   ---
 `;
 
+const tmsViaProxy = dedent`
+  # Testomat.io-first operating rules (highest priority)
+
+  This agent works on a Testomat.io project. You have TWO sources of truth and they answer different kinds of questions:
+
+  1. **The pulled markdown workspace (\`cwd\`)** — for **tests** and **suites**. Test content, suite hierarchy, bodies, tags and gherkin scenarios are files. Use \`read\`, \`find\`, \`grep\`, \`ls\` — they are instant and hit no network.
+  2. **The \`mcp\` tool** — for **dynamic Testomat.io data**: runs, testruns, plans, labels, linked issues, ims config, ci config, analytics. These do not live as files and MUST come from there.
+
+  Testomat.io is reached through **one \`mcp\` tool**, not a tool per operation. Call \`mcp({ search: "<what you need>" })\` to find the right operation, then \`mcp({ tool: "<name>", ... })\` to run it. The handful of reads used most often are also registered as their own tools — use those directly when they fit.
+
+  **Prefer the filesystem for test/suite queries.** Reach for a listing only when the question needs cross-tree metadata (priority, labels, status) the markdown does not carry, or when the workspace looks stale.
+
+  **Never** ask the user for the **Testomat.io** API token — it's configured. (This does NOT cover secrets the *app under test* needs to run — its own env vars/tokens. If a test run is blocked by one of those, report it.)
+
+  **Statuses and counts are live** — runs change them at any time. Fresh query results supersede numbers from earlier in the conversation; when they differ, the data changed — report the new snapshot.
+
+  ## Banned moves
+
+  - Do NOT search for an operation as your first action for a test/suite question — **read the filesystem first** (\`find\`, \`ls\`, \`grep\`, \`read\`). Fall back to a listing only if the metadata you need isn't in the markdown.
+  - Do NOT ask the user for the **Testomat.io** API token — it's already configured. (Secrets the app under test needs to run are a different thing — report those if a run is blocked.)
+  - Do NOT call \`bash\` / \`find\` / \`grep\` to answer questions about **runs, testruns, plans, labels, linked issues, analytics** — those don't exist as files.
+
+  ## Filesystem vs. Testomat.io cheat sheet
+
+  | Question is about… | First action |
+  |---|---|
+  | Test content (code, steps, gherkin, description, file path, tags) | \`read\` / \`find\` / \`grep\` in \`cwd\` |
+  | Suite hierarchy / structure / which files exist | \`ls\` / \`find\` |
+  | Individual test metadata (priority, status flag, labels) | try filesystem; if missing, \`mcp({ search: "get test" })\` |
+  | Runs, testruns, plans, labels, issues, analytics | \`mcp({ search: … })\`, then \`mcp({ tool: … })\` |
+  | Creating/updating tests or suites | edit the markdown file, then \`npx check-tests push\` |
+  | Creating/updating runs, linking issues | the matching \`mcp\` create/update operation |
+  | Launching automated or mixed Run remotely on CI | Use npx @testomatio/reporter run --remote <profile>, raise error if no CI profiles configured |
+  | Running automated Run | Use local test runner with testomatio reporter attached to report to current project |
+
+  ---
+`;
+
+const tmsViaCli = dedent`
+  # Testomat.io-first operating rules (highest priority)
+
+  This agent works on a Testomat.io project through the **shell only** — there are no Testomat.io tools in this session.
+
+  1. **The pulled markdown workspace (\`cwd\`)** — the canonical source for **tests** and **suites**: content, hierarchy, bodies, tags, gherkin scenarios. Use \`read\`, \`find\`, \`grep\`, \`ls\`.
+  2. **\`npx check-tests\`** — moves test cases between the markdown and Testomat.io (\`pull\`, \`push\`). It reads its credentials from the environment.
+  3. **The REST API through \`bash\`** — for dynamic data (runs, testruns, plans, labels, linked issues, analytics): \`curl -H "Authorization: $TESTOMATIO" "$TESTOMATIO_URL/api/v2/..."\`.
+
+  **Never** ask the user for the **Testomat.io** API token — it's configured as \`TESTOMATIO\`. (This does NOT cover secrets the *app under test* needs to run — its own env vars/tokens. If a test run is blocked by one of those, report it.)
+
+  **Statuses and counts are live** — runs change them at any time. Fresh query results supersede numbers from earlier in the conversation.
+
+  ## Banned moves
+
+  - Do NOT shell out for test content that is already a file in \`cwd\` — read the file.
+  - Do NOT invent REST paths or parameters. Read the workspace first; when a call is genuinely needed, keep to documented \`/api/v2\` endpoints.
+  - Do NOT ask the user for the **Testomat.io** API token.
+
+  ## Filesystem vs. shell cheat sheet
+
+  | Question is about… | First action |
+  |---|---|
+  | Test content (code, steps, gherkin, description, file path, tags) | \`read\` / \`find\` / \`grep\` in \`cwd\` |
+  | Suite hierarchy / structure / which files exist | \`ls\` / \`find\` |
+  | Runs, testruns, plans, labels, issues, analytics | \`curl\` against \`/api/v2\` |
+  | Creating/updating tests or suites | edit the markdown file, then \`npx check-tests push\` |
+  | Launching automated or mixed Run remotely on CI | Use npx @testomatio/reporter run --remote <profile>, raise error if no CI profiles configured |
+  | Running automated Run | Use local test runner with testomatio reporter attached to report to current project |
+
+  ---
+`;
+
+/**
+ * How the agent reaches Testomat.io in this session. The rules are the same;
+ * the way to act on them is not, so the harness picks the matching wording.
+ */
+export function testomatioTms(access: TmsAccess): string {
+  if (access === "mcp-proxy") return tmsViaProxy;
+  if (access === "cli-only") return tmsViaCli;
+  return tmsViaDirectTools;
+}
+
 export function testomatioConnection(
   tokenSlugs: string[],
   backendUrl?: string,
-  connection?: { projectId?: string; title?: string }
+  connection?: { projectId?: string; title?: string },
+  access: TmsAccess = "mcp-direct"
 ): string {
   let urlSuffix = "";
   if (backendUrl) urlSuffix = ` (\`${backendUrl}\`)`;
@@ -60,7 +142,7 @@ export function testomatioConnection(
 
       The project API key is **already set** as \`TESTOMATIO\` in the environment of every \`bash\` command you run, along with \`TESTOMATIO_URL\`${urlSuffix}. **Never ask the user for a Testomat.io token or API key** — it is already in scope.
 
-      Push or pull manual test cases with \`npx check-tests\` (or other \`npx @testomatio/*\` commands) — they pick the credentials up from the environment. If \`testomatio-*\` MCP tools are available, prefer them for structured actions (list/create/update tests, runs, labels, suites, etc.).
+      Push or pull manual test cases with \`npx check-tests\` (or other \`npx @testomatio/*\` commands) — they pick the credentials up from the environment. ${envTokenActions(access)}
 
     `;
   }
@@ -89,7 +171,7 @@ export function testomatioConnection(
     Projects pulled into this working directory (one dir per project):
     ${projects}
 
-    The **Testomat.io MCP server** is also auto-connected, one instance per project (tool prefix: \`testomatio-<slug>\`). Prefer MCP tools for structured actions (list/create/update tests, runs, labels, suites, etc.) — they are typed, safer, and faster than shell calls. Fall back to \`bash\` + \`check-tests\` only for things the MCP does not cover (e.g. pulling full markdown files).
+    ${perProjectActions(access)}
 
     When you need to push changes back, run \`npx check-tests push\` (or similar \`npx @testomatio/*\` commands) directly — credentials are already in scope.
 
@@ -161,6 +243,28 @@ export function projectSettings(info: TestomatioProjectInfo): string {
   `;
 }
 
+// How to act on a connection whose token comes from the environment.
+function envTokenActions(access: TmsAccess): string {
+  if (access === "mcp-proxy") {
+    return "For structured actions (list/create/update tests, runs, labels, suites, etc.) prefer the `mcp` tool: `mcp({ search })` to find an operation, `mcp({ tool })` to run it.";
+  }
+  if (access === "cli-only") {
+    return "For runs, plans, labels and analytics use `curl` against `/api/v2` — there are no Testomat.io tools in this session.";
+  }
+  return "If `testomatio-*` MCP tools are available, prefer them for structured actions (list/create/update tests, runs, labels, suites, etc.).";
+}
+
+// The same, where the harness runs one MCP server per pulled project.
+function perProjectActions(access: TmsAccess): string {
+  if (access === "mcp-proxy") {
+    return "The **Testomat.io MCP server** is also connected, reached through the `mcp` tool (`mcp({ search })` to find an operation, `mcp({ tool })` to run it). Prefer it for structured actions (list/create/update tests, runs, labels, suites, etc.) — those are typed, safer, and faster than shell calls. Fall back to `bash` + `check-tests` only for things it does not cover (e.g. pulling full markdown files).";
+  }
+  if (access === "cli-only") {
+    return "There are no Testomat.io tools in this session: use `npx check-tests` for test cases and `curl` against `/api/v2` for runs, plans and labels.";
+  }
+  return "The **Testomat.io MCP server** is also auto-connected, one instance per project (tool prefix: `testomatio-<slug>`). Prefer MCP tools for structured actions (list/create/update tests, runs, labels, suites, etc.) — they are typed, safer, and faster than shell calls. Fall back to `bash` + `check-tests` only for things the MCP does not cover (e.g. pulling full markdown files).";
+}
+
 function capped(shown: string[], total: number): string {
   let suffix = "";
   if (total > shown.length) {
@@ -168,3 +272,10 @@ function capped(shown: string[], total: number): string {
   }
   return shown.join(", ") + suffix;
 }
+
+/**
+ * `mcp-direct` — every Testomat.io operation is its own tool, prefixed
+ * `testomatio-<slug>`. `mcp-proxy` — one `mcp` tool with search and call.
+ * `cli-only` — no tools at all: `check-tests` and REST through the shell.
+ */
+export type TmsAccess = "mcp-direct" | "mcp-proxy" | "cli-only";

package/{src/prompt → prompt}/tools.ts

@@ -1,21 +1,33 @@
+import { existsSync } from 'node:fs';
+import { delimiter, join } from 'node:path';
+
 import dedent from 'dedent';
 
-export const tools = dedent`
+/**
+ * The tools every harness gives the agent. Anything a particular harness adds —
+ * a question tool, a browser it can drive for a watching user — arrives as an
+ * extra bullet from that harness, since the agent must not be told about a tool
+ * it does not have.
+ */
+export function tools(options?: { extra?: string[] }): string {
+  const extra = (options?.extra ?? []).join('');
+  let github = '    - GitHub: issues and projects through MCP.';
+  if (hasGh()) {
+    github +=
+      '\n    - Pull requests and repositories: `gh` in `bash`. When a PR question needs issue detail, query the issue through MCP — never infer it from a PR title.';
+  }
+  return dedent`
 <available-tools>
   You have these tools available:
   * **Read operations (use freely):** \`read\`, \`grep\`, \`ls\`.
     - Use these aggressively to understand the system under test before proposing changes.
   * **Write operations (restricted):** \`write\`, \`edit\`, \`bash\`.
     - Requires explicit user confirmation for destructive or shared-state actions.
-  * **User interaction:** \`ask\`.
-    - Use for clarifying requirements or providing multi-choice options.
-    - **Ask questions** in a way that leads to actionable answers. Avoid open-ended questions that require interpretation.
-  * **Browser automation (use \`playwright-cli\`):** For any browser task — launching a browser, navigating, clicking, filling forms, taking screenshots, recording video — drive the \`playwright-cli\` command-line tool through \`bash\` and follow the \`playwright-cli\` skill. Always prefer it over the built-in \`browser\` tool. Launch browsers headed so the user can watch: \`playwright-cli open --headed <url>\`.
-    - A message may include a \`<browser_state>\` notice — that means a browser is already open at that URL. Operate on the running session (snapshot, click, navigate) instead of launching a new one.
-  * **Internal Skills:** For QA-related tasks, check available skills first and prefer using them when applicable.
+${extra}  * **Internal Skills:** For QA-related tasks, check available skills first and prefer using them when applicable.
   * **External Integrations:** Use MCP tools when they provide superior data or specialized integrations.
     - Primary MCP (Testomat.io): Use Testomat.io MCP tools to extend functionality.
     - Secondary MCP (Jira, GitHub, etc.): Invoke only when user explicitly asks or when remote context exploration is required.
+${github}
 
   <tool-governance>
     * **Prioritize Specificity:** Use \`read\` instead of \`cat\`, and \`edit\` instead of \`sed\`. Dedicated tools provide better error tracking and safety.
@@ -24,4 +36,14 @@ export const tools = dedent`
     * **Analysis-First:** Use your own reasoning to guide tool selection. Do not wait for step-by-step instructions if discovery tools can provide the answer.
   </tool-governance>
 </available-tools>
-`;
+`;
+}
+
+/** Is the `gh` CLI on PATH? Never tell the agent to use a binary this machine lacks. */
+function hasGh(): boolean {
+  for (const dir of (process.env.PATH ?? '').split(delimiter)) {
+    if (!dir) continue;
+    if (existsSync(join(dir, 'gh')) || existsSync(join(dir, 'gh.exe'))) return true;
+  }
+  return false;
+}

package/prompt/vocab.ts

@@ -0,0 +1,7 @@
+/**
+ * The per-project config directory name, as the agent knows it. Part of the
+ * vocabulary the system prompt teaches — every path the prompt names below it
+ * (`.testeiya/manual-tests`, `.testeiya/project-info.json`, …) is built from
+ * this constant, so the prompt and the harness can never disagree about it.
+ */
+export const TESTEIYA_DIR_NAME = ".testeiya";

package/{skills.lock.json → skills/skills.lock.json}

@@ -2,8 +2,8 @@
   "sources": [
     {
       "source": "testomatio/skills",
-      "ref": "feat/setup-pr-testing-skill",
-      "sha": "99a6808422ff9e3a6bd46edab04a32d37a6ed028",
+      "ref": null,
+      "sha": "fa1cf38572773bae6c6ffe417392587b5d8087e2",
       "folder": "testomatio",
       "skills": [
         "testomatio-mcp",
@@ -12,14 +12,18 @@
         "qa-write-test-cases",
         "qa-thinking",
         "qa-test-code-coverage",
+        "qa-sprint-report-by-testomatio",
+        "qa-split-testing-levels-pyramid",
         "qa-requirement-reviewer",
         "qa-pr-requirements-analyzer",
         "qa-e2e-tests-reporting",
         "pull-request-diff-analyzer",
         "improve-test-cases",
         "detect-duplicate-test-cases",
-        "setup-pr-testing",
+        "setup-ci-automation",
+        "setup-change-aware-pr-testing",
         "run-tests-with-testomatio-reporter",
+        "qa-data-seeder",
         "qa-automation-test-consolidation",
         "debug-fix-failed-flaky-autotests",
         "automate-manual-test-cases",
@@ -53,8 +57,8 @@
     {
       "source": "currents-dev/playwright-best-practices-skill",
       "ref": null,
-      "sha": "ef329e7e65149918e1ff0eed2cf7d2e6e6f9eb5b",
-      "folder": "playwright-best-practices-skill",
+      "sha": "283d5cbc5d11aac1abda058b16ad22c317d54dc0",
+      "folder": "playwright",
       "skills": [
         "playwright-best-practices"
       ]
@@ -62,8 +66,8 @@
     {
       "source": "microsoft/playwright-cli/tree/main/skills/playwright-cli",
       "ref": null,
-      "sha": "793cfb32572733cbcb401e6f28d05a7a914ce408",
-      "folder": "playwright-cli",
+      "sha": "4f9d85a9472fc6d0c9a91ac62f3f3ecc24c16de2",
+      "folder": "playwright",
       "skills": [
         "playwright-cli"
       ]

package/skills/skills.yaml

@@ -0,0 +1,34 @@
+# External skills for Testeiya. ONE rule: every line is a GitHub repo.
+#
+#   owner/repo                        a skill repo (one skill, or many)
+#   owner/repo/tree/<ref>/<subdir>    pin a branch/tag/commit, or point at a subfolder
+#
+# Each source owns one vendor folder next to this file — the repo name, or the
+# owner when the repo is just "skills" (testomatio/skills -> testomatio/).
+# Updates replace only that folder; a folder this file does not own is treated
+# as internal and never touched. First-party skills are not kept here.
+#
+# Vendored folders are gitignored: they belong to their authors, under their own
+# licences. Send patches for them upstream, to the repo listed below.
+#
+# The folder is the category, and the repo decides it:
+#   - repo with a Claude-plugin marketplace (.claude-plugin/marketplace.json)
+#     -> one category subfolder per plugin  (testomatio/ -> test-management / test-automation / explorbot)
+#   - repo without one -> skills sit right under the vendor folder (the vendor is the category)
+#
+# Name the folder here when repos from different owners belong under one
+# category — the sources then share it, each skill in its own subfolder:
+#
+#   - source: owner/repo
+#     folder: playwright
+#
+# Edit the list, then run:  bunosh skills:update [vendor]      See the result:  bunosh skills:list
+
+- testomatio/skills
+- codeceptjs/skills
+
+# One Playwright category, two upstreams.
+- source: currents-dev/playwright-best-practices-skill
+  folder: playwright
+- source: microsoft/playwright-cli/tree/main/skills/playwright-cli
+  folder: playwright