testeiya 0.8.1 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/PIPELINES.md +153 -19
- package/README.md +29 -14
- package/dist/prompt/pipeline-report.js +23 -0
- package/dist/prompt/pipeline-report.js.map +1 -0
- package/dist/prompt/report.js +1 -1
- package/dist/prompt/testomat.io.js +2 -2
- package/dist/prompt/testomat.io.js.map +1 -1
- package/dist/src/args.js +19 -1
- package/dist/src/args.js.map +1 -1
- package/dist/src/cli.js +4 -2
- package/dist/src/cli.js.map +1 -1
- package/dist/src/doctor.js +8 -10
- package/dist/src/doctor.js.map +1 -1
- package/dist/src/langfuse-extension.js +2 -0
- package/dist/src/langfuse-extension.js.map +1 -1
- package/dist/src/mcp.js +20 -90
- package/dist/src/mcp.js.map +1 -1
- package/dist/src/pipeline/command-stage.js +2 -2
- package/dist/src/pipeline/command-stage.js.map +1 -1
- package/dist/src/pipeline/log.js +5 -0
- package/dist/src/pipeline/log.js.map +1 -1
- package/dist/src/pipeline/overlay.js +217 -0
- package/dist/src/pipeline/overlay.js.map +1 -0
- package/dist/src/pipeline/pipeline-file.js +67 -5
- package/dist/src/pipeline/pipeline-file.js.map +1 -1
- package/dist/src/pipeline/pipeline.js +48 -29
- package/dist/src/pipeline/pipeline.js.map +1 -1
- package/dist/src/pipeline/stage.js +49 -13
- package/dist/src/pipeline/stage.js.map +1 -1
- package/dist/src/pipeline/state.js +3 -1
- package/dist/src/pipeline/state.js.map +1 -1
- package/dist/src/pipeline/summary-stage.js +99 -0
- package/dist/src/pipeline/summary-stage.js.map +1 -0
- package/dist/src/pipeline/task-stage.js +25 -10
- package/dist/src/pipeline/task-stage.js.map +1 -1
- package/dist/src/pipeline/variables.js +34 -2
- package/dist/src/pipeline/variables.js.map +1 -1
- package/dist/src/session.js +19 -14
- package/dist/src/session.js.map +1 -1
- package/package.json +6 -9
- package/pipelines/api-testing.yml +9 -6
- package/pipelines/issue-test-cases.yml +14 -13
- package/pipelines/pr-review.yml +13 -9
- package/pipelines/regression.yml +6 -3
- package/pipelines/web-testing.yml +13 -12
- package/prompt/pipeline-report.ts +23 -0
- package/prompt/report.ts +1 -1
- package/prompt/testomat.io.ts +4 -4
- package/skills/codeceptjs/lint-codeceptjs-tests/SKILL.md +104 -0
- package/skills/skills.lock.json +5 -2
- package/skills/testomatio/qa-process/testing-workflow/SKILL.md +12 -7
- package/skills/testomatio/requirements/catalog-features/SKILL.md +132 -0
- package/skills/testomatio/requirements/catalog-features/references/layout.md +43 -0
- package/skills/testomatio/requirements/detect-product-mismatches/SKILL.md +146 -0
- package/skills/testomatio/requirements/detect-product-mismatches/references/features.md +84 -0
- package/skills/testomatio/requirements/detect-product-mismatches/references/layout.md +93 -0
- package/skills/testomatio/requirements/wiki-from-code/SKILL.md +58 -71
- package/skills/testomatio/requirements/wiki-from-code/references/behaviors.md +42 -0
- package/skills/testomatio/requirements/wiki-from-code/references/layout.md +70 -71
- package/dist/src/mcp-extension.js +0 -9
- package/dist/src/mcp-extension.js.map +0 -1
- package/dist/vendor/mcp.js +0 -14801
- package/mcp-tools.json +0 -1949
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: lint-codeceptjs-tests
|
|
3
|
+
description: >
|
|
4
|
+
Use when a CodeceptJS 4 suite needs a static quality gate — before committing
|
|
5
|
+
tests, reviewing a PR that touches tests, hardening CI, or on direct asks:
|
|
6
|
+
"lint the tests", "find I.wait / setTimeout sleeps", "leftover .only or
|
|
7
|
+
pause()", "missing await on grab", "passwords printed in logs", "why are
|
|
8
|
+
tests slow/flaky". Runs `@codeceptjs/lint` — an AST-based linter that needs
|
|
9
|
+
no CodeceptJS install and no browser.
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
# Lint CodeceptJS 4 Tests
|
|
13
|
+
|
|
14
|
+
`@codeceptjs/lint` reads tests, page objects and helpers as an AST and reports the smells that make suites slow, flaky or leaky — fixed sleeps, `if` branching, raw `fetch`, missing `await` on grabbers, plain-text credentials, leftover `.only`/`pause()`. One finding per line; exit codes `0` / `1` (errors found) / `2` (bad input, unparsable file) drop straight into CI.
|
|
15
|
+
|
|
16
|
+
## Commands
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
npx @codeceptjs/lint # files from codecept.conf.js in cwd
|
|
20
|
+
npx @codeceptjs/lint tests/ pages/login.js # explicit paths (no config needed)
|
|
21
|
+
npx @codeceptjs/lint -r no-pause -r no-only # only these rules (or --rules a,b)
|
|
22
|
+
npx @codeceptjs/lint rules # list rules + defaults
|
|
23
|
+
npx @codeceptjs/lint --json # file, line, column, rule, level, message, source
|
|
24
|
+
npx @codeceptjs/lint -c e2e/codecept.conf.js # config in another directory
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Without paths it reads `codecept.conf.js|ts|cjs|mjs` and lints everything matched by `tests`, local `include` files (page objects, steps file) and helpers loaded via `require`. TypeScript is parsed with Node's type stripping (Node 22.13+); `enum`/`namespace` report as parse errors. A TS config importing `./x.js` for an `x.ts` file (tsx convention) loads when the project has `tsx` installed.
|
|
28
|
+
|
|
29
|
+
## Rules and the fix for each
|
|
30
|
+
|
|
31
|
+
| Rule | Default | Detects → fix |
|
|
32
|
+
|---|---|---|
|
|
33
|
+
| `no-fixed-wait` | error | `I.wait(5)` → `I.waitForElement` / `I.waitForText` / `I.see` |
|
|
34
|
+
| `no-sleep` | error | bare and `window.`/`globalThis.` `setTimeout`, incl. `await new Promise(r => setTimeout(r, ms))` and `timers/promises`, outside helpers → condition wait |
|
|
35
|
+
| `no-if-in-test` | error | `if` inside a Scenario/hook → `tryTo()` from `codeceptjs/effects` |
|
|
36
|
+
| `no-fetch` | error | `fetch()` (incl. `window.fetch`, `globalThis.fetch`) outside helpers → REST helper `I.sendGetRequest` / `I.sendPostRequest` |
|
|
37
|
+
| `no-console` | warn | `console.*` outside helpers → `I.say()` (lands in the report) |
|
|
38
|
+
| `no-only` | error on CI, warn locally | `Scenario.only`, `Feature.only`, `Data(...).only` → remove before commit |
|
|
39
|
+
| `no-pause` | error on CI, warn locally | `pause()`, `this.pause()` → remove before commit |
|
|
40
|
+
| `no-skip` | warn | `Scenario.skip`, `Feature.skip`, `xScenario`, `xFeature` → intentional? suppress with a comment explaining why |
|
|
41
|
+
| `secret-credentials` | error | credential-ish `process.env.X` / `process.env['X']` or a password/token/secret/api-key locator passed to `I.*` → wrap with `secret()` |
|
|
42
|
+
| `await-grab` | error | `I.grab*()` assigned, destructured or passed without `await` (`await Promise.all([...])` and `.then()` chains are fine) → add `await` |
|
|
43
|
+
| `no-actor-in-helper` | error | `I` inside a class extending `Helper` → `this.helpers['Name']` |
|
|
44
|
+
| `raw-browser-in-test` | warn | `I.use*To`, `I.executeScript` in a Scenario → move into a helper or page object |
|
|
45
|
+
|
|
46
|
+
"On CI" = the `CI` env var is set — local debugging stubs (`pause()`, `.only`) stay warnings so they don't fail the lint while investigating.
|
|
47
|
+
|
|
48
|
+
## Configuring per project
|
|
49
|
+
|
|
50
|
+
Levels live in the config, next to the tests they govern:
|
|
51
|
+
|
|
52
|
+
```js
|
|
53
|
+
export const config = {
|
|
54
|
+
tests: './*_test.js',
|
|
55
|
+
lint: {
|
|
56
|
+
rules: { 'no-if-in-test': 'warn', 'raw-browser-in-test': 'off' },
|
|
57
|
+
},
|
|
58
|
+
}
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Levels: `error`, `warn`, `off`. For a single deliberate case, suppress inline — rule id required, never disable a rule project-wide for one line:
|
|
62
|
+
|
|
63
|
+
```js
|
|
64
|
+
I.wait(1) // codeceptjs-lint-disable-line no-fixed-wait
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Workflow
|
|
68
|
+
|
|
69
|
+
1. Run `npx @codeceptjs/lint` at the project root; if no config loads (secrets, tsx-less TS config), pass paths explicitly.
|
|
70
|
+
2. Fix errors first (they fail CI), warnings by judgment. Triage by cost: sleeps and missing `await` are the flakiness drivers; `secret()` fixes are one-line.
|
|
71
|
+
3. For bulk cleanup, feed `--json` output to drive file-by-file edits — the `source` field holds the offending line (truncated to 120 chars).
|
|
72
|
+
4. Add the gate so it stays clean:
|
|
73
|
+
|
|
74
|
+
```yaml
|
|
75
|
+
# CI, before codeceptjs run
|
|
76
|
+
- run: npx @codeceptjs/lint
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
For agent-driven work, a Claude Code `PreToolUse` hook blocks edits that *add* a new error (pre-existing errors and warnings never block; a failing hook allows the edit):
|
|
80
|
+
|
|
81
|
+
```json
|
|
82
|
+
{
|
|
83
|
+
"hooks": {
|
|
84
|
+
"PreToolUse": [
|
|
85
|
+
{ "matcher": "Write|Edit|MultiEdit", "hooks": [{ "type": "command", "command": "npx @codeceptjs/lint --hook claude" }] }
|
|
86
|
+
]
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
## Things to avoid
|
|
92
|
+
|
|
93
|
+
- "Fixing" `await-grab` by deleting the `await` usage — the grab result is a promise, the fix is `await`, not removal.
|
|
94
|
+
- Replacing `I.wait` with a longer `I.wait`.
|
|
95
|
+
- Suppressing `no-if-in-test` when the branch is load-bearing — that's `tryTo()` or a helper's job.
|
|
96
|
+
- Adding the CI step before triaging existing findings — the suite goes red on day one.
|
|
97
|
+
- Skipping the hook's config scope check — it only lints files the config covers.
|
|
98
|
+
|
|
99
|
+
## Related skills
|
|
100
|
+
|
|
101
|
+
- `debugging-codeceptjs-tests` — `I.wait(N)` as a timing probe is fine mid-debug, lint-clean it before committing
|
|
102
|
+
- `refactoring-codeceptjs-tests` — bulk cleanup of a lint-red suite (extract helpers, tame locators)
|
|
103
|
+
- `writing-codeceptjs-tests` — write clean tests from the start
|
|
104
|
+
- `ci-fix-tests` — run the lint gate first so CI fixes don't reintroduce smells
|
package/skills/skills.lock.json
CHANGED
|
@@ -3,12 +3,14 @@
|
|
|
3
3
|
{
|
|
4
4
|
"source": "testomatio/skills",
|
|
5
5
|
"ref": null,
|
|
6
|
-
"sha": "
|
|
6
|
+
"sha": "e1cf03227adfcb553c94ea5ee08c0eebc397e875",
|
|
7
7
|
"folder": "testomatio",
|
|
8
8
|
"skills": [
|
|
9
9
|
"automate-manual-test-cases",
|
|
10
|
+
"catalog-features",
|
|
10
11
|
"debug-fix-failed-flaky-autotests",
|
|
11
12
|
"detect-duplicate-test-cases",
|
|
13
|
+
"detect-product-mismatches",
|
|
12
14
|
"detect-useless-tests",
|
|
13
15
|
"epic-requirements-specification",
|
|
14
16
|
"explorbot-fundamentals",
|
|
@@ -46,7 +48,7 @@
|
|
|
46
48
|
{
|
|
47
49
|
"source": "codeceptjs/skills",
|
|
48
50
|
"ref": null,
|
|
49
|
-
"sha": "
|
|
51
|
+
"sha": "c9a82e3e237c908ba53ec47d2c841c680b88e03a",
|
|
50
52
|
"folder": "codeceptjs",
|
|
51
53
|
"skills": [
|
|
52
54
|
"ci-fix-tests",
|
|
@@ -55,6 +57,7 @@
|
|
|
55
57
|
"codeceptjs-fundamentals",
|
|
56
58
|
"codeceptjs-run-analysis",
|
|
57
59
|
"debugging-codeceptjs-tests",
|
|
60
|
+
"lint-codeceptjs-tests",
|
|
58
61
|
"migrate-codeceptjs-4",
|
|
59
62
|
"migrate-cypress-to-codeceptjs",
|
|
60
63
|
"migrate-protractor-to-codeceptjs",
|
|
@@ -17,7 +17,9 @@ Orchestrates the test case lifecycle by routing requests to specialized skills a
|
|
|
17
17
|
| `qa-thinking` | Analyze a feature as QA — edge cases, negative flows, abuses, risk scenarios |
|
|
18
18
|
| `qa-split-testing-levels-pyramid` | Apply the test pyramid — assign scenarios to testing levels, coverage split |
|
|
19
19
|
| `write-user-story` | Write user stories and acceptance criteria (the requirements) |
|
|
20
|
-
| `wiki-from-code`
|
|
20
|
+
| `wiki-from-code` | Build a whole-product wiki from code, or update it after large changes |
|
|
21
|
+
| `catalog-features` | Catalog features of one source: code, issue tracker, requirements, user docs |
|
|
22
|
+
| `detect-product-mismatches` | Compare code, issues, requirements, user docs with each other — mismatches |
|
|
21
23
|
| `qa-requirement-reviewer` | Review requirements for ambiguity, gaps, and testability |
|
|
22
24
|
| `qa-write-test-cases` | Generate new test cases and checklists from requirements |
|
|
23
25
|
| `improve-test-cases` | Improve existing test cases quality |
|
|
@@ -38,7 +40,9 @@ Orchestrates the test case lifecycle by routing requests to specialized skills a
|
|
|
38
40
|
- Flows are examples, not exhaustive. Combine or extend them when a request spans several tasks.
|
|
39
41
|
- When suggesting next steps, take into account the flows, context, user request, and results of previous steps.
|
|
40
42
|
- **Write / draft user stories** (requirements, spec, acceptance criteria) → route to the `write-user-story` skill. Review of existing requirements → `qa-requirement-reviewer`.
|
|
41
|
-
- **Wiki / spec from code** (build or
|
|
43
|
+
- **Wiki / spec from code** (build or update the whole-product wiki, actual behavior, implicit requirements) → route to the `wiki-from-code` skill.
|
|
44
|
+
- **Catalog features of a source** (codebase, Jira, Confluence, Notion, user docs, etc.; import or sync into the wiki) → route to the `catalog-features` skill.
|
|
45
|
+
- **Compare sources** (code, issues, requirements, user docs: not implemented, differs, unspecified, undocumented) → route to the `detect-product-mismatches` skill.
|
|
42
46
|
- **Behavior questions** ("what happens when…", "can a user…", "is X supported") ask what the product does rather than for an artifact → route to the `qa-explain-behavior` skill first, then continue with the flow the answer points to.
|
|
43
47
|
- **Strategic intent** ("where do I start", "improve our QA process", "QA maturity review") → route to the `qa-lead-strategy-advisor` skill instead. It owns the high-level roadmap and delegates execution back here.
|
|
44
48
|
|
|
@@ -47,14 +51,15 @@ Orchestrates the test case lifecycle by routing requests to specialized skills a
|
|
|
47
51
|
### Wiki from Code Flow
|
|
48
52
|
|
|
49
53
|
```
|
|
50
|
-
User: asks to build/
|
|
54
|
+
User: asks to build/update a wiki, requirements, or spec from the codebase
|
|
51
55
|
=>
|
|
52
|
-
Use `wiki-from-code` skill to
|
|
56
|
+
Use `wiki-from-code` skill to build or update `wiki/` from implemented code
|
|
53
57
|
=>
|
|
54
58
|
After the wiki is written, suggest next actions:
|
|
55
|
-
1.
|
|
56
|
-
2.
|
|
57
|
-
3.
|
|
59
|
+
1. 🔍 Cross-check with issues, requirements, user docs (with `detect-product-mismatches` skill)
|
|
60
|
+
2. 🧠 Risk scenarios from a feature (with `qa-thinking` skill)
|
|
61
|
+
3. 📝 Test cases from a feature (with `qa-write-test-cases` skill)
|
|
62
|
+
4. 📘 User stories from a feature (with `write-user-story` skill)
|
|
58
63
|
```
|
|
59
64
|
|
|
60
65
|
### Test Generation Flow
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: catalog-features
|
|
3
|
+
description: >-
|
|
4
|
+
Builds a catalog of product features from one source: a tree that mirrors the source, each feature as a key, title, and link to the source. Code, issues, requirements, docs, etc. are supported.
|
|
5
|
+
Use when asked to catalog/list features, existing functionality, issues, requirements, specs, PRDs, user docs or similar sources.
|
|
6
|
+
metadata:
|
|
7
|
+
author: Testomat.io
|
|
8
|
+
version: 1.0.0
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Catalog features
|
|
12
|
+
|
|
13
|
+
Catalog of **the source's features**: a tree that mirrors the source, each feature with its key, title, and link. Its behaviors (description, requirements, acceptance criteria, etc.) stay at the source.
|
|
14
|
+
|
|
15
|
+
Output folder: given by the calling skill or the user. Default: `features/.catalog/`. Write only inside it.
|
|
16
|
+
|
|
17
|
+
## Modes
|
|
18
|
+
|
|
19
|
+
`build` from scratch; `update` from an existing catalog. When `update`, use info from the catalog README.
|
|
20
|
+
|
|
21
|
+
## Source roles
|
|
22
|
+
|
|
23
|
+
| Role | Description |
|
|
24
|
+
| ---------- | ----------------------------------------------------------- |
|
|
25
|
+
| Actual | Actual behavior of the product. Code, configs, etc. |
|
|
26
|
+
| Intended | What we attempted to implement. Requirements, tickets, etc. |
|
|
27
|
+
| Documented | What the product says it does. User docs, etc. |
|
|
28
|
+
|
|
29
|
+
## Rules
|
|
30
|
+
|
|
31
|
+
- **Read-only on the source.** Do NOT edit the source even when the tools offer it.
|
|
32
|
+
- One catalog per source.
|
|
33
|
+
|
|
34
|
+
## Sources
|
|
35
|
+
|
|
36
|
+
### Code
|
|
37
|
+
|
|
38
|
+
- Tree: areas → features, named from what the product does for its users, not from the stack or code folders, classes, files, or functions.
|
|
39
|
+
- Feature: one per feature, not per file; same feature in two places → one feature. Link: repo paths of the main implementation (route, screen, job), not every file.
|
|
40
|
+
- Scope: features users can reach now. Not stubs, routes users cannot reach, or features behind disabled flags; comments, docs, and tests do not make a feature.
|
|
41
|
+
- `update`: `git diff --name-status -M {baseline}..HEAD`; take into account renamed, moved, deleted files.
|
|
42
|
+
- Role: Actual.
|
|
43
|
+
- History: No.
|
|
44
|
+
|
|
45
|
+
### Issue tracker
|
|
46
|
+
|
|
47
|
+
- Tree: the tracker's hierarchy: initiatives, epics, parent issues, issues, sub-tasks, etc.
|
|
48
|
+
- Scope: a query (JQL, `gh` search, Linear filter, etc.); ask the user for it and store it. "Done" issues only (typical successfully resolved statuses); ask the user when in doubt.
|
|
49
|
+
- Role: Intended.
|
|
50
|
+
- History: Yes (unless user specified). Because different issues may be related to the same feature. Need to store date of creation and resolution to determine which issue overrides the other.
|
|
51
|
+
|
|
52
|
+
### Requirements
|
|
53
|
+
|
|
54
|
+
- Word, PDF: extract the text first.
|
|
55
|
+
- Tree: area by heading or other logical structure.
|
|
56
|
+
- Scope: ask user for the scope and other crucial filters. Take into account that there could be drafts, deprecated pages, copies. Process them accordingly.
|
|
57
|
+
- Role: Intended.
|
|
58
|
+
- History: No.
|
|
59
|
+
|
|
60
|
+
### User docs
|
|
61
|
+
|
|
62
|
+
- Tree: keep the original structure with required hierarchy levels.
|
|
63
|
+
- Role: Documented.
|
|
64
|
+
- Scope: ask user.
|
|
65
|
+
- History: No.
|
|
66
|
+
|
|
67
|
+
### Other
|
|
68
|
+
|
|
69
|
+
Any source that describes the product. Agree on role, coverage, history with the user before syncing.
|
|
70
|
+
|
|
71
|
+
## Steps
|
|
72
|
+
|
|
73
|
+
```
|
|
74
|
+
- [ ] 1. Source and domain
|
|
75
|
+
- [ ] 2. Inventory
|
|
76
|
+
- [ ] 3. Write the tree
|
|
77
|
+
- [ ] 4. Validate
|
|
78
|
+
- [ ] 5. Report
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Read [references/layout.md](references/layout.md) before writing files.
|
|
82
|
+
|
|
83
|
+
## 1. Source and domain
|
|
84
|
+
|
|
85
|
+
Ask the user in one message, unless given:
|
|
86
|
+
|
|
87
|
+
- Type and access: repository, MCP, CLI, URL, export, or file path. No access → stop and ask for an MCP connection or an export.
|
|
88
|
+
- Scope as a concrete, repeatable filter: repo paths, tracker query (JQL, project, label), root page or space, docs root.
|
|
89
|
+
- Domain: what the product does and for whom, key terms. Not given → infer it from the source.
|
|
90
|
+
|
|
91
|
+
Name area and features in the domain terms.
|
|
92
|
+
|
|
93
|
+
Several sources in one run → one output folder each; define all of them first, then run steps 2–4 per source.
|
|
94
|
+
|
|
95
|
+
## 2. Inventory
|
|
96
|
+
|
|
97
|
+
List every feature in scope with key or URL, title, status. Navigate all the pages.
|
|
98
|
+
|
|
99
|
+
When `update`: take into account renamed, moved, deleted files. Process them accordingly.
|
|
100
|
+
|
|
101
|
+
If source is code create areas and features based on features and logic, not just files, classes, functions.
|
|
102
|
+
|
|
103
|
+
## 3. Write the tree
|
|
104
|
+
|
|
105
|
+
Catalog README and area READMEs, as [references/layout.md](references/layout.md) describes.
|
|
106
|
+
|
|
107
|
+
## 4. Validate
|
|
108
|
+
|
|
109
|
+
```markdown
|
|
110
|
+
Structure
|
|
111
|
+
- [ ] Catalog README has every fact and the counts; indexes every area
|
|
112
|
+
- [ ] Every area README lists its subareas and features; the tree mirrors the source
|
|
113
|
+
|
|
114
|
+
Content
|
|
115
|
+
- [ ] Every feature yml is according to the `references/layout.md`
|
|
116
|
+
- [ ] Code: areas and features named from the product, not the stack
|
|
117
|
+
- [ ] Update: features gone from scope removed
|
|
118
|
+
- [ ] Only existing keys and urls are used in the catalog
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
## 5. Report
|
|
122
|
+
|
|
123
|
+
Where to start. Features synced, skipped by reason.
|
|
124
|
+
|
|
125
|
+
To a calling skill: features added, changed, and removed.
|
|
126
|
+
|
|
127
|
+
## Next actions
|
|
128
|
+
|
|
129
|
+
- `detect-product-mismatches`
|
|
130
|
+
- Feature pages from the code → `wiki-from-code`
|
|
131
|
+
- `qa-requirement-reviewer`
|
|
132
|
+
- Test cases from requirements → `qa-write-test-cases`
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Catalog layout
|
|
2
|
+
|
|
3
|
+
The tree mirrors how the source groups its content: code areas, epics, page tree, docs nav.
|
|
4
|
+
|
|
5
|
+
```
|
|
6
|
+
features/.catalog/ # output folder
|
|
7
|
+
README.md # source facts, counts, areas, ungrouped features
|
|
8
|
+
{area}/ # code area, epic, parent page, nav section; any depth
|
|
9
|
+
README.md # the area's own link/path, subarea if any, its features
|
|
10
|
+
{feature}.yaml # feature page
|
|
11
|
+
{subarea}/ # subarea (optional)
|
|
12
|
+
README.md # the subarea own link/path, name, its features; breadcrumbs
|
|
13
|
+
{feature}.yaml # feature page; breadcrumbs; feature info (below)
|
|
14
|
+
...
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Formatting
|
|
18
|
+
|
|
19
|
+
Every `.md` should include info about its entity and child entities list (if any). Relative links between catalog pages.
|
|
20
|
+
|
|
21
|
+
README.md in catalog root:
|
|
22
|
+
|
|
23
|
+
| Fact | Value |
|
|
24
|
+
| ------- | ----------------------------------------------------------------------------------------------------- |
|
|
25
|
+
| Type | Code, issue tracker, requirements, user docs or any other |
|
|
26
|
+
| Role | Actual, Intended, or Documented (SKILL.md › Source roles) |
|
|
27
|
+
| Scope | Defined by user (or inferred) |
|
|
28
|
+
| History | Important for sources like issue tracker |
|
|
29
|
+
| Access | MCP, CLI, URL, files |
|
|
30
|
+
| Date | Date of this sync; code: baseline commit and its date; issue tracker: date of creation and resolution |
|
|
31
|
+
| Domain | What the product does and for whom; infer if not given |
|
|
32
|
+
|
|
33
|
+
## Feature YAML
|
|
34
|
+
|
|
35
|
+
Name: {feature-name}.yaml
|
|
36
|
+
|
|
37
|
+
Feature YAML should include:
|
|
38
|
+
|
|
39
|
+
- Key [optional]. Do not invent. Use only if the source provides it.
|
|
40
|
+
- Title: short description of the feature.
|
|
41
|
+
- Links/headings: link(s) to sources that describe the feature (url, repo path, etc.); headings in case source is a document.
|
|
42
|
+
|
|
43
|
+
No description, requirements or other details.
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: detect-product-mismatches
|
|
3
|
+
description: >-
|
|
4
|
+
Compares what different sources say the product does: code (actual behavior), issue tracker and requirements (intended behavior), user docs (documented behavior), and any other source (test cases, release notes etc.).
|
|
5
|
+
Checks every feature in every source and records mismatches: contradictions and differences, not implemented, unspecified, undocumented.
|
|
6
|
+
Use when the user asks to compare, cross-check, align, validate requirements, specs, wiki, tickets, issues, user docs, or code, codebase against each other, to find mismatches, contradictions, differences, or outdated docs or requirements, or to find what is not implemented, not documented, or not specified.
|
|
7
|
+
metadata:
|
|
8
|
+
author: Testomat.io
|
|
9
|
+
version: 1.0.0
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
# Detect product mismatches
|
|
13
|
+
|
|
14
|
+
Checks each feature in every source and records where the sources disagree.
|
|
15
|
+
|
|
16
|
+
Output folder:
|
|
17
|
+
- given by calling skill or the user
|
|
18
|
+
- all paths are relative inside the output folder
|
|
19
|
+
- default: `mismatches/`
|
|
20
|
+
|
|
21
|
+
> `wiki/mismatches/` is the output of `wiki-from-code`: mismatches inside a single source, the code. Don't use it as the output folder and don't use its results.
|
|
22
|
+
|
|
23
|
+
## Modes
|
|
24
|
+
|
|
25
|
+
`build` (from scratch) or `update` (for changes since the last check).
|
|
26
|
+
|
|
27
|
+
## Sources
|
|
28
|
+
|
|
29
|
+
Type, role, scope, and history of each source come from its catalog README (`catalog-features` › Source roles, Sources).
|
|
30
|
+
|
|
31
|
+
| Source | Role |
|
|
32
|
+
| ------------- | --------------------- |
|
|
33
|
+
| Code | Actual |
|
|
34
|
+
| Requirements | Intended |
|
|
35
|
+
| Issue tracker | Intended |
|
|
36
|
+
| User docs | Documented |
|
|
37
|
+
| Other | As its catalog README |
|
|
38
|
+
|
|
39
|
+
Per feature, what each source says is rewritten as behaviors in one shape ([features.md](references/features.md) › Behaviors), so sources compare line by line:
|
|
40
|
+
|
|
41
|
+
- Two sources say different things about the same behavior → Differs.
|
|
42
|
+
- Another source has a behavior the code lacks → Not implemented. Code is the product: it must do what the other sources say.
|
|
43
|
+
- Several intended sources: a feature is specified when any of them has it.
|
|
44
|
+
- `History: Yes` (issue tracker by default): the newest issue wins (resolution date, else creation date); older contradicting issues are superseded, not mismatches.
|
|
45
|
+
|
|
46
|
+
**Not a mismatch:**
|
|
47
|
+
|
|
48
|
+
- Same behavior in other words.
|
|
49
|
+
- **Silence on a detail.** A coarser source agrees with a finer one unless it contradicts it (`Customer can save a card.` agrees with `Customer can save up to 5 cards.`).
|
|
50
|
+
- A behavior only the code has, inside a feature the other sources have.
|
|
51
|
+
- Superseded issues; behaviors of another version; content outside a catalog Scope (open issues, drafts, deprecated pages).
|
|
52
|
+
- Can't tell (vague behavior, unreadable content) → open question.
|
|
53
|
+
|
|
54
|
+
## Likely cause
|
|
55
|
+
|
|
56
|
+
From dates read at the source: issue resolution date (else creation date), page last version, docs "last updated" (else `git log -1 --format=%cs -- {path}`); code: `git log -1 --format=%cs -- {paths}` for the feature's code paths. The same dates decide the newest issue for `History: Yes`.
|
|
57
|
+
|
|
58
|
+
- Code differs from the newest intended behavior, and nothing newer explains it → Bug.
|
|
59
|
+
- Requirement older than the issue or code change that set the current behavior → Stale requirement.
|
|
60
|
+
- User docs older than that change → Stale docs.
|
|
61
|
+
- Otherwise → Decide (product owner).
|
|
62
|
+
|
|
63
|
+
## Steps
|
|
64
|
+
|
|
65
|
+
```
|
|
66
|
+
- [ ] 1. Sources and access
|
|
67
|
+
- [ ] 2. Catalogs: create missing, update stale
|
|
68
|
+
- [ ] 3. Check each feature (one subagent each)
|
|
69
|
+
- [ ] 4. Verify (subagents)
|
|
70
|
+
- [ ] 5. Merge
|
|
71
|
+
- [ ] 6. Validate
|
|
72
|
+
- [ ] 7. Report
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Read [references/features.md](references/features.md) and [references/layout.md](references/layout.md) before writing files.
|
|
76
|
+
|
|
77
|
+
## Rules
|
|
78
|
+
|
|
79
|
+
- **Read-only on sources.** Never edit issues, pages, or code.
|
|
80
|
+
- Catalogs and `features/` change only by running their skills.
|
|
81
|
+
- **Report, don't decide.** A mismatch page shows what each source says; its Solution only suggests a fix.
|
|
82
|
+
- Compare behaviors, not wording: different values, conditions, roles, labels, or flows differ.
|
|
83
|
+
- Write only paths, keys, and URLs that exist in the catalogs, the sources, or the repo. Do not invent them.
|
|
84
|
+
|
|
85
|
+
## 1. Sources and access
|
|
86
|
+
|
|
87
|
+
1. List the sources to compare: given by the user, else ask; offer the ones already cataloged.
|
|
88
|
+
2. Check access: one real read per source (MCP tool, `gh`, git, URL, file). No access → stop and ask for an MCP connection or an export. A catalog alone is not enough: behaviors are read at the source.
|
|
89
|
+
|
|
90
|
+
## 2. Catalogs
|
|
91
|
+
|
|
92
|
+
Every listed source needs a catalog. Create the missing ones, update the stale ones, before any check:
|
|
93
|
+
|
|
94
|
+
Use `catalog-features` skill to build the catalogs for each source. Use `wiki-from-code` also if **code is the source**.
|
|
95
|
+
|
|
96
|
+
## 3. Check each feature
|
|
97
|
+
|
|
98
|
+
Features: every feature from catalogs in this order: code, requirements, user docs, issue tracker, other.
|
|
99
|
+
|
|
100
|
+
One subagent per feature, in parallel. Each gets: the feature (its feature YAML), the compared sources (name, type, role, scope, history, access, catalog path), the version, the Domain from the catalog READMEs, the source table and rules above, [references/features.md](references/features.md), [references/layout.md](references/layout.md).
|
|
101
|
+
|
|
102
|
+
Subagent:
|
|
103
|
+
|
|
104
|
+
1. Per source: finds the feature in the catalog, else searches the source ([features.md](references/features.md) › Find).
|
|
105
|
+
2. Reads every matching feature at its links and states its behaviors ([features.md](references/features.md) › Read, Behaviors).
|
|
106
|
+
3. Aligns the same behavior across sources.
|
|
107
|
+
4. Returns one mismatch per disagreement: its kind (Differs or Not implemented) and every section of a mismatch page ([layout.md](references/layout.md) › Page), with code evidence as `path:lines`.
|
|
108
|
+
|
|
109
|
+
Subagent writes nothing. Returns: matched catalog features per source (or `not found` + where searched), mismatches, sources silent on the whole feature, open questions, catalog errors.
|
|
110
|
+
|
|
111
|
+
## 4. Verify
|
|
112
|
+
|
|
113
|
+
One fresh subagent per batch of mismatches, with the same inputs as step 3. It re-reads each mismatch at its Links and returns it as confirmed, corrected, or dropped (with why). Can't confirm → open question.
|
|
114
|
+
|
|
115
|
+
## 5. Merge
|
|
116
|
+
|
|
117
|
+
1. Write the confirmed mismatch pages ([layout.md](references/layout.md) › Page). Same disagreement from several features → one mismatch page, every disagreeing source on it.
|
|
118
|
+
2. `update`: remove resolved mismatches.
|
|
119
|
+
3. In `mismatches/README.md`, write the mismatches list ([layout.md](references/layout.md) › README).
|
|
120
|
+
4. New `sources/` or `mismatches/` → add a link and purpose for it to the root `README.md`.
|
|
121
|
+
|
|
122
|
+
## 6. Validate
|
|
123
|
+
|
|
124
|
+
```markdown
|
|
125
|
+
Structure
|
|
126
|
+
- [ ] Every feature of the compared catalogs was checked, or listed as an open question
|
|
127
|
+
- [ ] Mismatches list states compared sources (catalog Date or baseline), version, check date; indexes every page and missing feature
|
|
128
|
+
|
|
129
|
+
Content
|
|
130
|
+
- [ ] Each page has every section of [layout.md](references/layout.md) › Page and one row per compared source; silent sources marked silent with where searched
|
|
131
|
+
- [ ] Difference opens with a one-sentence summary; each bullet is clear alone: sources, condition, what the user gets; the title doesn't say which side is right
|
|
132
|
+
- [ ] Every page was confirmed in Verify; nothing from Not a mismatch is a page
|
|
133
|
+
- [ ] Every behavior on the pages passes the check in [references/features.md](references/features.md)
|
|
134
|
+
- [ ] update: resolved pages removed
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
## 7. Report
|
|
138
|
+
|
|
139
|
+
Where to start. Mismatches by highest user impact first. Missing features per source. Catalog errors (→ `catalog-features`; code catalog with `features/` → `wiki-from-code`). Open questions.
|
|
140
|
+
|
|
141
|
+
## Next actions
|
|
142
|
+
|
|
143
|
+
- Update a catalog → `catalog-features`; feature pages from code → `wiki-from-code`
|
|
144
|
+
- User stories for not-implemented behavior → `write-user-story`
|
|
145
|
+
- Review contradicting requirements → `qa-requirement-reviewer`
|
|
146
|
+
- Test cases for differing behavior → `qa-write-test-cases`
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# Features
|
|
2
|
+
|
|
3
|
+
A feature is something the product does for its users that they would name: "Pay with saved card", "Export report to CSV". In a catalog, one feature YAML is one feature of its source, in the area where the source puts it. Another source may name it differently or split it differently: e.g. one epic covers three code features, one docs page covers two.
|
|
4
|
+
|
|
5
|
+
A feature is made of behaviors: what someone who uses it (end user, admin, API client, integrated system) can do, see, or get, and when. Sources are compared behavior by behavior, so every source's behaviors are written the same way.
|
|
6
|
+
|
|
7
|
+
Behavior is similar to requirement or acceptance criteria, but those terms describe intended behavior; code gives actual behavior, user docs documented behavior (`catalog-features` › Source roles). "Behavior" fits every role: implemented, planned, or documented.
|
|
8
|
+
|
|
9
|
+
## Find
|
|
10
|
+
|
|
11
|
+
Per source:
|
|
12
|
+
|
|
13
|
+
1. Catalog: key, title, the area it sits in. Links in the feature YAML to other sources help.
|
|
14
|
+
2. Not in the catalog → search the source, within the catalog's Scope:
|
|
15
|
+
|
|
16
|
+
| Source | Search |
|
|
17
|
+
| ------------- | -------------------------------------------------------------------------------------- |
|
|
18
|
+
| Code | Domain terms, UI labels, messages, routes; start next to related features' code |
|
|
19
|
+
| Issue tracker | Text search inside the catalog's Scope query (JQL `text ~ "term"`, `gh search issues`) |
|
|
20
|
+
| Requirements | Search under the root page or space (CQL `text ~ "term"`); headings of long documents |
|
|
21
|
+
| User docs | Docs site search or nav; `grep` in the docs repo |
|
|
22
|
+
| Other | The source's own search |
|
|
23
|
+
|
|
24
|
+
- Found at the source, missing from the catalog → use it and report a catalog error.
|
|
25
|
+
- Found, but outside the catalog's Scope (open issue, draft, deprecated page or copy, other version) → not a behavior. Mention it on the mismatch page when it explains one.
|
|
26
|
+
- Not found → the source is silent on the feature. Record where you searched.
|
|
27
|
+
|
|
28
|
+
## Read
|
|
29
|
+
|
|
30
|
+
Read every matching feature at its links (documents: under its headings). Behaviors come from:
|
|
31
|
+
|
|
32
|
+
| Source | Behaviors from | Not behaviors |
|
|
33
|
+
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
34
|
+
| Code | The code at the feature's `code` paths (else its links); Behaviors on its feature page in `features/`, when present, as a start | Claims (comments, docstrings, developer docs, API specs, configs, tests); stubs, routes users cannot reach; internal steps with no visible effect |
|
|
35
|
+
| Issue tracker | Stories and feature issues: acceptance criteria, then the description. Epics: their scope. Bugs: the expected result | A bug's reported actual result; tasks, spikes, chores, tech debt |
|
|
36
|
+
| Requirements | Functional requirements, business rules, acceptance criteria, flows; one behavior per table row and per meaningful matrix cell; exclusions as negative ones | Future phases, nice-to-have, MoSCoW Could and Won't; goals, rationale, personas, metrics, open questions |
|
|
37
|
+
| User docs | Guides, how-tos, limits, notes and warnings; badges (plan, edition, role, platform) as conditions; navigation steps | Screenshots and videos |
|
|
38
|
+
| Test cases | Expected results | Steps, test data |
|
|
39
|
+
|
|
40
|
+
- Code behind a flag or setting that can be turned on → a behavior with that condition (as `wiki-from-code`).
|
|
41
|
+
- Comments and later edits may change the body ("dropped", "moved to PROJ-456", "decided: 3 retries, not 5"): state the final behavior, never the comment.
|
|
42
|
+
- Acceptance criteria may sit in a custom field, a description heading, a checklist, or sub-tasks: find the convention in the first issues.
|
|
43
|
+
|
|
44
|
+
## Behaviors
|
|
45
|
+
|
|
46
|
+
### Shape
|
|
47
|
+
|
|
48
|
+
- One behavior per line, one sentence, present tense.
|
|
49
|
+
- No `should`, `shall`, `must`, `will`: state it as a fact. The source's role says whether it is actual, intended, or documented.
|
|
50
|
+
- Name who acts or what changes, in domain terms. Never "the system", "it", or "we".
|
|
51
|
+
- Outcome is what the actor can observe (screen, response, file, email, notification), not how the product gets there.
|
|
52
|
+
- Conditions stay in the behavior: role, platform, setting, state.
|
|
53
|
+
- Source key, when the behavior has one, goes first.
|
|
54
|
+
- Negative behaviors count: `Viewer cannot delete a project.`
|
|
55
|
+
- Split on "and" when each part can be checked on its own.
|
|
56
|
+
|
|
57
|
+
### Values and words
|
|
58
|
+
|
|
59
|
+
- Values as the source states them, with units. No value in the source → none in the behavior. Never round, convert, or fill one in; a vague behavior stays vague (open question).
|
|
60
|
+
- Code: exact values from the code that runs; a value read from config or environment: its default, marked `(configurable)` (as `wiki-from-code`).
|
|
61
|
+
- UI labels and messages verbatim, in quotes; navigation paths with `›`.
|
|
62
|
+
- Domain terms (prefer the user docs' terms), one per concept across sources and `features/`.
|
|
63
|
+
- No class, function, table, file, flag, or library names. Protocol details (endpoints, status codes, fields) only when the API is what users use.
|
|
64
|
+
- Keep the source's level of detail: a coarse behavior stays coarse. Never merge behaviors and drop values or conditions.
|
|
65
|
+
- Restate, never paste. Never people, comments, or customer data.
|
|
66
|
+
|
|
67
|
+
## Examples
|
|
68
|
+
|
|
69
|
+
| Source says | Behavior |
|
|
70
|
+
| --------------------------------------------------------------------- | ------------------------------------------------------------------- |
|
|
71
|
+
| The system shall allow users to store payment methods (max 5). | `Customer can save up to 5 cards.` |
|
|
72
|
+
| Users can save cards to pay faster. | `Customer can save a card.` (benefit dropped) |
|
|
73
|
+
| AC: if the card is declined, show "Payment failed. Try another card." | `Declined card: Customer sees "Payment failed. Try another card."` |
|
|
74
|
+
| Enterprise customers get SSO; Pro is coming soon. | `Enterprise plan: SSO sign-in is available.` (Pro: future, dropped) |
|
|
75
|
+
| Session times out after a while. | `Session expires after inactivity.` (no value invented) |
|
|
76
|
+
| Bug: saving the same card twice creates a duplicate; expected: one. | `Saving the same card again keeps one saved card.` |
|
|
77
|
+
|
|
78
|
+
## Check
|
|
79
|
+
|
|
80
|
+
- [ ] Every source either has the feature or records where it was searched
|
|
81
|
+
- [ ] One observable behavior per line, present tense, who or what named
|
|
82
|
+
- [ ] Values, labels, and conditions exactly as in the source; none added
|
|
83
|
+
- [ ] Domain terms, the same term for the same thing
|
|
84
|
+
- [ ] No implementation, plans, rationale, work items, people, or copied text
|