testeiya 0.8.1 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -0
- package/package.json +1 -1
- package/skills/codeceptjs/lint-codeceptjs-tests/SKILL.md +104 -0
- package/skills/skills.lock.json +5 -2
- package/skills/testomatio/qa-process/testing-workflow/SKILL.md +12 -7
- package/skills/testomatio/requirements/catalog-features/SKILL.md +132 -0
- package/skills/testomatio/requirements/catalog-features/references/layout.md +43 -0
- package/skills/testomatio/requirements/detect-product-mismatches/SKILL.md +146 -0
- package/skills/testomatio/requirements/detect-product-mismatches/references/features.md +84 -0
- package/skills/testomatio/requirements/detect-product-mismatches/references/layout.md +93 -0
- package/skills/testomatio/requirements/wiki-from-code/SKILL.md +58 -71
- package/skills/testomatio/requirements/wiki-from-code/references/behaviors.md +42 -0
- package/skills/testomatio/requirements/wiki-from-code/references/layout.md +70 -71
package/README.md
CHANGED
|
@@ -23,6 +23,7 @@ its own browser, and syncs with your Testomat.io project.
|
|
|
23
23
|
| OS | Download |
|
|
24
24
|
|---|---|
|
|
25
25
|
| macOS (Apple Silicon) | [`stable-macos-arm64-Testeiya.dmg`](https://github.com/testomatio/testeiya-app/releases/latest/download/stable-macos-arm64-Testeiya.dmg) |
|
|
26
|
+
| macOS (Intel) | [`stable-macos-x64-Testeiya.dmg`](https://github.com/testomatio/testeiya-app/releases/latest/download/stable-macos-x64-Testeiya.dmg) |
|
|
26
27
|
| Windows (x64) | [`stable-win-x64-Testeiya.zip`](https://github.com/testomatio/testeiya-app/releases/latest/download/stable-win-x64-Testeiya.zip) |
|
|
27
28
|
| Linux (x64) | [`Testeiya-x86_64.AppImage`](https://github.com/testomatio/testeiya-app/releases/latest/download/Testeiya-x86_64.AppImage) |
|
|
28
29
|
|
package/package.json
CHANGED
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: lint-codeceptjs-tests
|
|
3
|
+
description: >
|
|
4
|
+
Use when a CodeceptJS 4 suite needs a static quality gate — before committing
|
|
5
|
+
tests, reviewing a PR that touches tests, hardening CI, or on direct asks:
|
|
6
|
+
"lint the tests", "find I.wait / setTimeout sleeps", "leftover .only or
|
|
7
|
+
pause()", "missing await on grab", "passwords printed in logs", "why are
|
|
8
|
+
tests slow/flaky". Runs `@codeceptjs/lint` — an AST-based linter that needs
|
|
9
|
+
no CodeceptJS install and no browser.
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
# Lint CodeceptJS 4 Tests
|
|
13
|
+
|
|
14
|
+
`@codeceptjs/lint` reads tests, page objects and helpers as an AST and reports the smells that make suites slow, flaky or leaky — fixed sleeps, `if` branching, raw `fetch`, missing `await` on grabbers, plain-text credentials, leftover `.only`/`pause()`. One finding per line; exit codes `0` / `1` (errors found) / `2` (bad input, unparsable file) drop straight into CI.
|
|
15
|
+
|
|
16
|
+
## Commands
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
npx @codeceptjs/lint # files from codecept.conf.js in cwd
|
|
20
|
+
npx @codeceptjs/lint tests/ pages/login.js # explicit paths (no config needed)
|
|
21
|
+
npx @codeceptjs/lint -r no-pause -r no-only # only these rules (or --rules a,b)
|
|
22
|
+
npx @codeceptjs/lint rules # list rules + defaults
|
|
23
|
+
npx @codeceptjs/lint --json # file, line, column, rule, level, message, source
|
|
24
|
+
npx @codeceptjs/lint -c e2e/codecept.conf.js # config in another directory
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Without paths it reads `codecept.conf.js|ts|cjs|mjs` and lints everything matched by `tests`, local `include` files (page objects, steps file) and helpers loaded via `require`. TypeScript is parsed with Node's type stripping (Node 22.13+); `enum`/`namespace` report as parse errors. A TS config importing `./x.js` for an `x.ts` file (tsx convention) loads when the project has `tsx` installed.
|
|
28
|
+
|
|
29
|
+
## Rules and the fix for each
|
|
30
|
+
|
|
31
|
+
| Rule | Default | Detects → fix |
|
|
32
|
+
|---|---|---|
|
|
33
|
+
| `no-fixed-wait` | error | `I.wait(5)` → `I.waitForElement` / `I.waitForText` / `I.see` |
|
|
34
|
+
| `no-sleep` | error | bare and `window.`/`globalThis.` `setTimeout`, incl. `await new Promise(r => setTimeout(r, ms))` and `timers/promises`, outside helpers → condition wait |
|
|
35
|
+
| `no-if-in-test` | error | `if` inside a Scenario/hook → `tryTo()` from `codeceptjs/effects` |
|
|
36
|
+
| `no-fetch` | error | `fetch()` (incl. `window.fetch`, `globalThis.fetch`) outside helpers → REST helper `I.sendGetRequest` / `I.sendPostRequest` |
|
|
37
|
+
| `no-console` | warn | `console.*` outside helpers → `I.say()` (lands in the report) |
|
|
38
|
+
| `no-only` | error on CI, warn locally | `Scenario.only`, `Feature.only`, `Data(...).only` → remove before commit |
|
|
39
|
+
| `no-pause` | error on CI, warn locally | `pause()`, `this.pause()` → remove before commit |
|
|
40
|
+
| `no-skip` | warn | `Scenario.skip`, `Feature.skip`, `xScenario`, `xFeature` → intentional? suppress with a comment explaining why |
|
|
41
|
+
| `secret-credentials` | error | credential-ish `process.env.X` / `process.env['X']` or a password/token/secret/api-key locator passed to `I.*` → wrap with `secret()` |
|
|
42
|
+
| `await-grab` | error | `I.grab*()` assigned, destructured or passed without `await` (`await Promise.all([...])` and `.then()` chains are fine) → add `await` |
|
|
43
|
+
| `no-actor-in-helper` | error | `I` inside a class extending `Helper` → `this.helpers['Name']` |
|
|
44
|
+
| `raw-browser-in-test` | warn | `I.use*To`, `I.executeScript` in a Scenario → move into a helper or page object |
|
|
45
|
+
|
|
46
|
+
"On CI" = the `CI` env var is set — local debugging stubs (`pause()`, `.only`) stay warnings so they don't fail the lint while investigating.
|
|
47
|
+
|
|
48
|
+
## Configuring per project
|
|
49
|
+
|
|
50
|
+
Levels live in the config, next to the tests they govern:
|
|
51
|
+
|
|
52
|
+
```js
|
|
53
|
+
export const config = {
|
|
54
|
+
tests: './*_test.js',
|
|
55
|
+
lint: {
|
|
56
|
+
rules: { 'no-if-in-test': 'warn', 'raw-browser-in-test': 'off' },
|
|
57
|
+
},
|
|
58
|
+
}
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Levels: `error`, `warn`, `off`. For a single deliberate case, suppress inline — rule id required, never disable a rule project-wide for one line:
|
|
62
|
+
|
|
63
|
+
```js
|
|
64
|
+
I.wait(1) // codeceptjs-lint-disable-line no-fixed-wait
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Workflow
|
|
68
|
+
|
|
69
|
+
1. Run `npx @codeceptjs/lint` at the project root; if no config loads (secrets, tsx-less TS config), pass paths explicitly.
|
|
70
|
+
2. Fix errors first (they fail CI), warnings by judgment. Triage by cost: sleeps and missing `await` are the flakiness drivers; `secret()` fixes are one-line.
|
|
71
|
+
3. For bulk cleanup, feed `--json` output to drive file-by-file edits — the `source` field holds the offending line (truncated to 120 chars).
|
|
72
|
+
4. Add the gate so it stays clean:
|
|
73
|
+
|
|
74
|
+
```yaml
|
|
75
|
+
# CI, before codeceptjs run
|
|
76
|
+
- run: npx @codeceptjs/lint
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
For agent-driven work, a Claude Code `PreToolUse` hook blocks edits that *add* a new error (pre-existing errors and warnings never block; a failing hook allows the edit):
|
|
80
|
+
|
|
81
|
+
```json
|
|
82
|
+
{
|
|
83
|
+
"hooks": {
|
|
84
|
+
"PreToolUse": [
|
|
85
|
+
{ "matcher": "Write|Edit|MultiEdit", "hooks": [{ "type": "command", "command": "npx @codeceptjs/lint --hook claude" }] }
|
|
86
|
+
]
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
## Things to avoid
|
|
92
|
+
|
|
93
|
+
- "Fixing" `await-grab` by deleting the `await` usage — the grab result is a promise, the fix is `await`, not removal.
|
|
94
|
+
- Replacing `I.wait` with a longer `I.wait`.
|
|
95
|
+
- Suppressing `no-if-in-test` when the branch is load-bearing — that's `tryTo()` or a helper's job.
|
|
96
|
+
- Adding the CI step before triaging existing findings — the suite goes red on day one.
|
|
97
|
+
- Skipping the hook's config scope check — it only lints files the config covers.
|
|
98
|
+
|
|
99
|
+
## Related skills
|
|
100
|
+
|
|
101
|
+
- `debugging-codeceptjs-tests` — `I.wait(N)` as a timing probe is fine mid-debug, lint-clean it before committing
|
|
102
|
+
- `refactoring-codeceptjs-tests` — bulk cleanup of a lint-red suite (extract helpers, tame locators)
|
|
103
|
+
- `writing-codeceptjs-tests` — write clean tests from the start
|
|
104
|
+
- `ci-fix-tests` — run the lint gate first so CI fixes don't reintroduce smells
|
package/skills/skills.lock.json
CHANGED
|
@@ -3,12 +3,14 @@
|
|
|
3
3
|
{
|
|
4
4
|
"source": "testomatio/skills",
|
|
5
5
|
"ref": null,
|
|
6
|
-
"sha": "
|
|
6
|
+
"sha": "e1cf03227adfcb553c94ea5ee08c0eebc397e875",
|
|
7
7
|
"folder": "testomatio",
|
|
8
8
|
"skills": [
|
|
9
9
|
"automate-manual-test-cases",
|
|
10
|
+
"catalog-features",
|
|
10
11
|
"debug-fix-failed-flaky-autotests",
|
|
11
12
|
"detect-duplicate-test-cases",
|
|
13
|
+
"detect-product-mismatches",
|
|
12
14
|
"detect-useless-tests",
|
|
13
15
|
"epic-requirements-specification",
|
|
14
16
|
"explorbot-fundamentals",
|
|
@@ -46,7 +48,7 @@
|
|
|
46
48
|
{
|
|
47
49
|
"source": "codeceptjs/skills",
|
|
48
50
|
"ref": null,
|
|
49
|
-
"sha": "
|
|
51
|
+
"sha": "c9a82e3e237c908ba53ec47d2c841c680b88e03a",
|
|
50
52
|
"folder": "codeceptjs",
|
|
51
53
|
"skills": [
|
|
52
54
|
"ci-fix-tests",
|
|
@@ -55,6 +57,7 @@
|
|
|
55
57
|
"codeceptjs-fundamentals",
|
|
56
58
|
"codeceptjs-run-analysis",
|
|
57
59
|
"debugging-codeceptjs-tests",
|
|
60
|
+
"lint-codeceptjs-tests",
|
|
58
61
|
"migrate-codeceptjs-4",
|
|
59
62
|
"migrate-cypress-to-codeceptjs",
|
|
60
63
|
"migrate-protractor-to-codeceptjs",
|
|
@@ -17,7 +17,9 @@ Orchestrates the test case lifecycle by routing requests to specialized skills a
|
|
|
17
17
|
| `qa-thinking` | Analyze a feature as QA — edge cases, negative flows, abuses, risk scenarios |
|
|
18
18
|
| `qa-split-testing-levels-pyramid` | Apply the test pyramid — assign scenarios to testing levels, coverage split |
|
|
19
19
|
| `write-user-story` | Write user stories and acceptance criteria (the requirements) |
|
|
20
|
-
| `wiki-from-code`
|
|
20
|
+
| `wiki-from-code` | Build a whole-product wiki from code, or update it after large changes |
|
|
21
|
+
| `catalog-features` | Catalog features of one source: code, issue tracker, requirements, user docs |
|
|
22
|
+
| `detect-product-mismatches` | Compare code, issues, requirements, user docs with each other — mismatches |
|
|
21
23
|
| `qa-requirement-reviewer` | Review requirements for ambiguity, gaps, and testability |
|
|
22
24
|
| `qa-write-test-cases` | Generate new test cases and checklists from requirements |
|
|
23
25
|
| `improve-test-cases` | Improve existing test cases quality |
|
|
@@ -38,7 +40,9 @@ Orchestrates the test case lifecycle by routing requests to specialized skills a
|
|
|
38
40
|
- Flows are examples, not exhaustive. Combine or extend them when a request spans several tasks.
|
|
39
41
|
- When suggesting next steps, take into account the flows, context, user request, and results of previous steps.
|
|
40
42
|
- **Write / draft user stories** (requirements, spec, acceptance criteria) → route to the `write-user-story` skill. Review of existing requirements → `qa-requirement-reviewer`.
|
|
41
|
-
- **Wiki / spec from code** (build or
|
|
43
|
+
- **Wiki / spec from code** (build or update the whole-product wiki, actual behavior, implicit requirements) → route to the `wiki-from-code` skill.
|
|
44
|
+
- **Catalog features of a source** (codebase, Jira, Confluence, Notion, user docs, etc.; import or sync into the wiki) → route to the `catalog-features` skill.
|
|
45
|
+
- **Compare sources** (code, issues, requirements, user docs: not implemented, differs, unspecified, undocumented) → route to the `detect-product-mismatches` skill.
|
|
42
46
|
- **Behavior questions** ("what happens when…", "can a user…", "is X supported") ask what the product does rather than for an artifact → route to the `qa-explain-behavior` skill first, then continue with the flow the answer points to.
|
|
43
47
|
- **Strategic intent** ("where do I start", "improve our QA process", "QA maturity review") → route to the `qa-lead-strategy-advisor` skill instead. It owns the high-level roadmap and delegates execution back here.
|
|
44
48
|
|
|
@@ -47,14 +51,15 @@ Orchestrates the test case lifecycle by routing requests to specialized skills a
|
|
|
47
51
|
### Wiki from Code Flow
|
|
48
52
|
|
|
49
53
|
```
|
|
50
|
-
User: asks to build/
|
|
54
|
+
User: asks to build/update a wiki, requirements, or spec from the codebase
|
|
51
55
|
=>
|
|
52
|
-
Use `wiki-from-code` skill to
|
|
56
|
+
Use `wiki-from-code` skill to build or update `wiki/` from implemented code
|
|
53
57
|
=>
|
|
54
58
|
After the wiki is written, suggest next actions:
|
|
55
|
-
1.
|
|
56
|
-
2.
|
|
57
|
-
3.
|
|
59
|
+
1. 🔍 Cross-check with issues, requirements, user docs (with `detect-product-mismatches` skill)
|
|
60
|
+
2. 🧠 Risk scenarios from a feature (with `qa-thinking` skill)
|
|
61
|
+
3. 📝 Test cases from a feature (with `qa-write-test-cases` skill)
|
|
62
|
+
4. 📘 User stories from a feature (with `write-user-story` skill)
|
|
58
63
|
```
|
|
59
64
|
|
|
60
65
|
### Test Generation Flow
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: catalog-features
|
|
3
|
+
description: >-
|
|
4
|
+
Builds a catalog of product features from one source: a tree that mirrors the source, each feature as a key, title, and link to the source. Code, issues, requirements, docs, etc. are supported.
|
|
5
|
+
Use when asked to catalog/list features, existing functionality, issues, requirements, specs, PRDs, user docs or similar sources.
|
|
6
|
+
metadata:
|
|
7
|
+
author: Testomat.io
|
|
8
|
+
version: 1.0.0
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Catalog features
|
|
12
|
+
|
|
13
|
+
Catalog of **the source's features**: a tree that mirrors the source, each feature with its key, title, and link. Its behaviors (description, requirements, acceptance criteria, etc.) stay at the source.
|
|
14
|
+
|
|
15
|
+
Output folder: given by the calling skill or the user. Default: `features/.catalog/`. Write only inside it.
|
|
16
|
+
|
|
17
|
+
## Modes
|
|
18
|
+
|
|
19
|
+
`build` from scratch; `update` from an existing catalog. When `update`, use info from the catalog README.
|
|
20
|
+
|
|
21
|
+
## Source roles
|
|
22
|
+
|
|
23
|
+
| Role | Description |
|
|
24
|
+
| ---------- | ----------------------------------------------------------- |
|
|
25
|
+
| Actual | Actual behavior of the product. Code, configs, etc. |
|
|
26
|
+
| Intended | What we attempted to implement. Requirements, tickets, etc. |
|
|
27
|
+
| Documented | What the product says it does. User docs, etc. |
|
|
28
|
+
|
|
29
|
+
## Rules
|
|
30
|
+
|
|
31
|
+
- **Read-only on the source.** Do NOT edit the source even when the tools offer it.
|
|
32
|
+
- One catalog per source.
|
|
33
|
+
|
|
34
|
+
## Sources
|
|
35
|
+
|
|
36
|
+
### Code
|
|
37
|
+
|
|
38
|
+
- Tree: areas → features, named from what the product does for its users, not from the stack or code folders, classes, files, or functions.
|
|
39
|
+
- Feature: one per feature, not per file; same feature in two places → one feature. Link: repo paths of the main implementation (route, screen, job), not every file.
|
|
40
|
+
- Scope: features users can reach now. Not stubs, routes users cannot reach, or features behind disabled flags; comments, docs, and tests do not make a feature.
|
|
41
|
+
- `update`: `git diff --name-status -M {baseline}..HEAD`; take into account renamed, moved, deleted files.
|
|
42
|
+
- Role: Actual.
|
|
43
|
+
- History: No.
|
|
44
|
+
|
|
45
|
+
### Issue tracker
|
|
46
|
+
|
|
47
|
+
- Tree: the tracker's hierarchy: initiatives, epics, parent issues, issues, sub-tasks, etc.
|
|
48
|
+
- Scope: a query (JQL, `gh` search, Linear filter, etc.); ask the user for it and store it. "Done" issues only (typical successfully resolved statuses); ask the user when in doubt.
|
|
49
|
+
- Role: Intended.
|
|
50
|
+
- History: Yes (unless user specified). Because different issues may be related to the same feature. Need to store date of creation and resolution to determine which issue overrides the other.
|
|
51
|
+
|
|
52
|
+
### Requirements
|
|
53
|
+
|
|
54
|
+
- Word, PDF: extract the text first.
|
|
55
|
+
- Tree: area by heading or other logical structure.
|
|
56
|
+
- Scope: ask user for the scope and other crucial filters. Take into account that there could be drafts, deprecated pages, copies. Process them accordingly.
|
|
57
|
+
- Role: Intended.
|
|
58
|
+
- History: No.
|
|
59
|
+
|
|
60
|
+
### User docs
|
|
61
|
+
|
|
62
|
+
- Tree: keep the original structure with required hierarchy levels.
|
|
63
|
+
- Role: Documented.
|
|
64
|
+
- Scope: ask user.
|
|
65
|
+
- History: No.
|
|
66
|
+
|
|
67
|
+
### Other
|
|
68
|
+
|
|
69
|
+
Any source that describes the product. Agree on role, coverage, history with the user before syncing.
|
|
70
|
+
|
|
71
|
+
## Steps
|
|
72
|
+
|
|
73
|
+
```
|
|
74
|
+
- [ ] 1. Source and domain
|
|
75
|
+
- [ ] 2. Inventory
|
|
76
|
+
- [ ] 3. Write the tree
|
|
77
|
+
- [ ] 4. Validate
|
|
78
|
+
- [ ] 5. Report
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Read [references/layout.md](references/layout.md) before writing files.
|
|
82
|
+
|
|
83
|
+
## 1. Source and domain
|
|
84
|
+
|
|
85
|
+
Ask the user in one message, unless given:
|
|
86
|
+
|
|
87
|
+
- Type and access: repository, MCP, CLI, URL, export, or file path. No access → stop and ask for an MCP connection or an export.
|
|
88
|
+
- Scope as a concrete, repeatable filter: repo paths, tracker query (JQL, project, label), root page or space, docs root.
|
|
89
|
+
- Domain: what the product does and for whom, key terms. Not given → infer it from the source.
|
|
90
|
+
|
|
91
|
+
Name area and features in the domain terms.
|
|
92
|
+
|
|
93
|
+
Several sources in one run → one output folder each; define all of them first, then run steps 2–4 per source.
|
|
94
|
+
|
|
95
|
+
## 2. Inventory
|
|
96
|
+
|
|
97
|
+
List every feature in scope with key or URL, title, status. Navigate all the pages.
|
|
98
|
+
|
|
99
|
+
When `update`: take into account renamed, moved, deleted files. Process them accordingly.
|
|
100
|
+
|
|
101
|
+
If source is code create areas and features based on features and logic, not just files, classes, functions.
|
|
102
|
+
|
|
103
|
+
## 3. Write the tree
|
|
104
|
+
|
|
105
|
+
Catalog README and area READMEs, as [references/layout.md](references/layout.md) describes.
|
|
106
|
+
|
|
107
|
+
## 4. Validate
|
|
108
|
+
|
|
109
|
+
```markdown
|
|
110
|
+
Structure
|
|
111
|
+
- [ ] Catalog README has every fact and the counts; indexes every area
|
|
112
|
+
- [ ] Every area README lists its subareas and features; the tree mirrors the source
|
|
113
|
+
|
|
114
|
+
Content
|
|
115
|
+
- [ ] Every feature yml is according to the `references/layout.md`
|
|
116
|
+
- [ ] Code: areas and features named from the product, not the stack
|
|
117
|
+
- [ ] Update: features gone from scope removed
|
|
118
|
+
- [ ] Only existing keys and urls are used in the catalog
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
## 5. Report
|
|
122
|
+
|
|
123
|
+
Where to start. Features synced, skipped by reason.
|
|
124
|
+
|
|
125
|
+
To a calling skill: features added, changed, and removed.
|
|
126
|
+
|
|
127
|
+
## Next actions
|
|
128
|
+
|
|
129
|
+
- `detect-product-mismatches`
|
|
130
|
+
- Feature pages from the code → `wiki-from-code`
|
|
131
|
+
- `qa-requirement-reviewer`
|
|
132
|
+
- Test cases from requirements → `qa-write-test-cases`
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Catalog layout
|
|
2
|
+
|
|
3
|
+
The tree mirrors how the source groups its content: code areas, epics, page tree, docs nav.
|
|
4
|
+
|
|
5
|
+
```
|
|
6
|
+
features/.catalog/ # output folder
|
|
7
|
+
README.md # source facts, counts, areas, ungrouped features
|
|
8
|
+
{area}/ # code area, epic, parent page, nav section; any depth
|
|
9
|
+
README.md # the area's own link/path, subarea if any, its features
|
|
10
|
+
{feature}.yaml # feature page
|
|
11
|
+
{subarea}/ # subarea (optional)
|
|
12
|
+
README.md # the subarea own link/path, name, its features; breadcrumbs
|
|
13
|
+
{feature}.yaml # feature page; breadcrumbs; feature info (below)
|
|
14
|
+
...
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Formatting
|
|
18
|
+
|
|
19
|
+
Every `.md` should include info about its entity and child entities list (if any). Relative links between catalog pages.
|
|
20
|
+
|
|
21
|
+
README.md in catalog root:
|
|
22
|
+
|
|
23
|
+
| Fact | Value |
|
|
24
|
+
| ------- | ----------------------------------------------------------------------------------------------------- |
|
|
25
|
+
| Type | Code, issue tracker, requirements, user docs or any other |
|
|
26
|
+
| Role | Actual, Intended, or Documented (SKILL.md › Source roles) |
|
|
27
|
+
| Scope | Defined by user (or inferred) |
|
|
28
|
+
| History | Important for sources like issue tracker |
|
|
29
|
+
| Access | MCP, CLI, URL, files |
|
|
30
|
+
| Date | Date of this sync; code: baseline commit and its date; issue tracker: date of creation and resolution |
|
|
31
|
+
| Domain | What the product does and for whom; infer if not given |
|
|
32
|
+
|
|
33
|
+
## Feature YAML
|
|
34
|
+
|
|
35
|
+
Name: {feature-name}.yaml
|
|
36
|
+
|
|
37
|
+
Feature YAML should include:
|
|
38
|
+
|
|
39
|
+
- Key [optional]. Do not invent. Use only if the source provides it.
|
|
40
|
+
- Title: short description of the feature.
|
|
41
|
+
- Links/headings: link(s) to sources that describe the feature (url, repo path, etc.); headings in case source is a document.
|
|
42
|
+
|
|
43
|
+
No description, requirements or other details.
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: detect-product-mismatches
|
|
3
|
+
description: >-
|
|
4
|
+
Compares what different sources say the product does: code (actual behavior), issue tracker and requirements (intended behavior), user docs (documented behavior), and any other source (test cases, release notes etc.).
|
|
5
|
+
Checks every feature in every source and records mismatches: contradictions and differences, not implemented, unspecified, undocumented.
|
|
6
|
+
Use when the user asks to compare, cross-check, align, validate requirements, specs, wiki, tickets, issues, user docs, or code, codebase against each other, to find mismatches, contradictions, differences, or outdated docs or requirements, or to find what is not implemented, not documented, or not specified.
|
|
7
|
+
metadata:
|
|
8
|
+
author: Testomat.io
|
|
9
|
+
version: 1.0.0
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
# Detect product mismatches
|
|
13
|
+
|
|
14
|
+
Checks each feature in every source and records where the sources disagree.
|
|
15
|
+
|
|
16
|
+
Output folder:
|
|
17
|
+
- given by calling skill or the user
|
|
18
|
+
- all paths are relative inside the output folder
|
|
19
|
+
- default: `mismatches/`
|
|
20
|
+
|
|
21
|
+
> `wiki/mismatches/` is the output of `wiki-from-code`: mismatches inside a single source, the code. Don't use it as the output folder and don't use its results.
|
|
22
|
+
|
|
23
|
+
## Modes
|
|
24
|
+
|
|
25
|
+
`build` (from scratch) or `update` (for changes since the last check).
|
|
26
|
+
|
|
27
|
+
## Sources
|
|
28
|
+
|
|
29
|
+
Type, role, scope, and history of each source come from its catalog README (`catalog-features` › Source roles, Sources).
|
|
30
|
+
|
|
31
|
+
| Source | Role |
|
|
32
|
+
| ------------- | --------------------- |
|
|
33
|
+
| Code | Actual |
|
|
34
|
+
| Requirements | Intended |
|
|
35
|
+
| Issue tracker | Intended |
|
|
36
|
+
| User docs | Documented |
|
|
37
|
+
| Other | As its catalog README |
|
|
38
|
+
|
|
39
|
+
Per feature, what each source says is rewritten as behaviors in one shape ([features.md](references/features.md) › Behaviors), so sources compare line by line:
|
|
40
|
+
|
|
41
|
+
- Two sources say different things about the same behavior → Differs.
|
|
42
|
+
- Another source has a behavior the code lacks → Not implemented. Code is the product: it must do what the other sources say.
|
|
43
|
+
- Several intended sources: a feature is specified when any of them has it.
|
|
44
|
+
- `History: Yes` (issue tracker by default): the newest issue wins (resolution date, else creation date); older contradicting issues are superseded, not mismatches.
|
|
45
|
+
|
|
46
|
+
**Not a mismatch:**
|
|
47
|
+
|
|
48
|
+
- Same behavior in other words.
|
|
49
|
+
- **Silence on a detail.** A coarser source agrees with a finer one unless it contradicts it (`Customer can save a card.` agrees with `Customer can save up to 5 cards.`).
|
|
50
|
+
- A behavior only the code has, inside a feature the other sources have.
|
|
51
|
+
- Superseded issues; behaviors of another version; content outside a catalog Scope (open issues, drafts, deprecated pages).
|
|
52
|
+
- Can't tell (vague behavior, unreadable content) → open question.
|
|
53
|
+
|
|
54
|
+
## Likely cause
|
|
55
|
+
|
|
56
|
+
From dates read at the source: issue resolution date (else creation date), page last version, docs "last updated" (else `git log -1 --format=%cs -- {path}`); code: `git log -1 --format=%cs -- {paths}` for the feature's code paths. The same dates decide the newest issue for `History: Yes`.
|
|
57
|
+
|
|
58
|
+
- Code differs from the newest intended behavior, and nothing newer explains it → Bug.
|
|
59
|
+
- Requirement older than the issue or code change that set the current behavior → Stale requirement.
|
|
60
|
+
- User docs older than that change → Stale docs.
|
|
61
|
+
- Otherwise → Decide (product owner).
|
|
62
|
+
|
|
63
|
+
## Steps
|
|
64
|
+
|
|
65
|
+
```
|
|
66
|
+
- [ ] 1. Sources and access
|
|
67
|
+
- [ ] 2. Catalogs: create missing, update stale
|
|
68
|
+
- [ ] 3. Check each feature (one subagent each)
|
|
69
|
+
- [ ] 4. Verify (subagents)
|
|
70
|
+
- [ ] 5. Merge
|
|
71
|
+
- [ ] 6. Validate
|
|
72
|
+
- [ ] 7. Report
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Read [references/features.md](references/features.md) and [references/layout.md](references/layout.md) before writing files.
|
|
76
|
+
|
|
77
|
+
## Rules
|
|
78
|
+
|
|
79
|
+
- **Read-only on sources.** Never edit issues, pages, or code.
|
|
80
|
+
- Catalogs and `features/` change only by running their skills.
|
|
81
|
+
- **Report, don't decide.** A mismatch page shows what each source says; its Solution only suggests a fix.
|
|
82
|
+
- Compare behaviors, not wording: different values, conditions, roles, labels, or flows differ.
|
|
83
|
+
- Write only paths, keys, and URLs that exist in the catalogs, the sources, or the repo. Do not invent them.
|
|
84
|
+
|
|
85
|
+
## 1. Sources and access
|
|
86
|
+
|
|
87
|
+
1. List the sources to compare: given by the user, else ask; offer the ones already cataloged.
|
|
88
|
+
2. Check access: one real read per source (MCP tool, `gh`, git, URL, file). No access → stop and ask for an MCP connection or an export. A catalog alone is not enough: behaviors are read at the source.
|
|
89
|
+
|
|
90
|
+
## 2. Catalogs
|
|
91
|
+
|
|
92
|
+
Every listed source needs a catalog. Create the missing ones, update the stale ones, before any check:
|
|
93
|
+
|
|
94
|
+
Use `catalog-features` skill to build the catalogs for each source. Use `wiki-from-code` also if **code is the source**.
|
|
95
|
+
|
|
96
|
+
## 3. Check each feature
|
|
97
|
+
|
|
98
|
+
Features: every feature from catalogs in this order: code, requirements, user docs, issue tracker, other.
|
|
99
|
+
|
|
100
|
+
One subagent per feature, in parallel. Each gets: the feature (its feature YAML), the compared sources (name, type, role, scope, history, access, catalog path), the version, the Domain from the catalog READMEs, the source table and rules above, [references/features.md](references/features.md), [references/layout.md](references/layout.md).
|
|
101
|
+
|
|
102
|
+
Subagent:
|
|
103
|
+
|
|
104
|
+
1. Per source: finds the feature in the catalog, else searches the source ([features.md](references/features.md) › Find).
|
|
105
|
+
2. Reads every matching feature at its links and states its behaviors ([features.md](references/features.md) › Read, Behaviors).
|
|
106
|
+
3. Aligns the same behavior across sources.
|
|
107
|
+
4. Returns one mismatch per disagreement: its kind (Differs or Not implemented) and every section of a mismatch page ([layout.md](references/layout.md) › Page), with code evidence as `path:lines`.
|
|
108
|
+
|
|
109
|
+
Subagent writes nothing. Returns: matched catalog features per source (or `not found` + where searched), mismatches, sources silent on the whole feature, open questions, catalog errors.
|
|
110
|
+
|
|
111
|
+
## 4. Verify
|
|
112
|
+
|
|
113
|
+
One fresh subagent per batch of mismatches, with the same inputs as step 3. It re-reads each mismatch at its Links and returns it as confirmed, corrected, or dropped (with why). Can't confirm → open question.
|
|
114
|
+
|
|
115
|
+
## 5. Merge
|
|
116
|
+
|
|
117
|
+
1. Write the confirmed mismatch pages ([layout.md](references/layout.md) › Page). Same disagreement from several features → one mismatch page, every disagreeing source on it.
|
|
118
|
+
2. `update`: remove resolved mismatches.
|
|
119
|
+
3. In `mismatches/README.md`, write the mismatches list ([layout.md](references/layout.md) › README).
|
|
120
|
+
4. New `sources/` or `mismatches/` → add a link and purpose for it to the root `README.md`.
|
|
121
|
+
|
|
122
|
+
## 6. Validate
|
|
123
|
+
|
|
124
|
+
```markdown
|
|
125
|
+
Structure
|
|
126
|
+
- [ ] Every feature of the compared catalogs was checked, or listed as an open question
|
|
127
|
+
- [ ] Mismatches list states compared sources (catalog Date or baseline), version, check date; indexes every page and missing feature
|
|
128
|
+
|
|
129
|
+
Content
|
|
130
|
+
- [ ] Each page has every section of [layout.md](references/layout.md) › Page and one row per compared source; silent sources marked silent with where searched
|
|
131
|
+
- [ ] Difference opens with a one-sentence summary; each bullet is clear alone: sources, condition, what the user gets; the title doesn't say which side is right
|
|
132
|
+
- [ ] Every page was confirmed in Verify; nothing from Not a mismatch is a page
|
|
133
|
+
- [ ] Every behavior on the pages passes the check in [references/features.md](references/features.md)
|
|
134
|
+
- [ ] update: resolved pages removed
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
## 7. Report
|
|
138
|
+
|
|
139
|
+
Where to start. Mismatches by highest user impact first. Missing features per source. Catalog errors (→ `catalog-features`; code catalog with `features/` → `wiki-from-code`). Open questions.
|
|
140
|
+
|
|
141
|
+
## Next actions
|
|
142
|
+
|
|
143
|
+
- Update a catalog → `catalog-features`; feature pages from code → `wiki-from-code`
|
|
144
|
+
- User stories for not-implemented behavior → `write-user-story`
|
|
145
|
+
- Review contradicting requirements → `qa-requirement-reviewer`
|
|
146
|
+
- Test cases for differing behavior → `qa-write-test-cases`
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# Features
|
|
2
|
+
|
|
3
|
+
A feature is something the product does for its users that they would name: "Pay with saved card", "Export report to CSV". In a catalog, one feature YAML is one feature of its source, in the area where the source puts it. Another source may name it differently or split it differently: e.g. one epic covers three code features, one docs page covers two.
|
|
4
|
+
|
|
5
|
+
A feature is made of behaviors: what someone who uses it (end user, admin, API client, integrated system) can do, see, or get, and when. Sources are compared behavior by behavior, so every source's behaviors are written the same way.
|
|
6
|
+
|
|
7
|
+
Behavior is similar to requirement or acceptance criteria, but those terms describe intended behavior; code gives actual behavior, user docs documented behavior (`catalog-features` › Source roles). "Behavior" fits every role: implemented, planned, or documented.
|
|
8
|
+
|
|
9
|
+
## Find
|
|
10
|
+
|
|
11
|
+
Per source:
|
|
12
|
+
|
|
13
|
+
1. Catalog: key, title, the area it sits in. Links in the feature YAML to other sources help.
|
|
14
|
+
2. Not in the catalog → search the source, within the catalog's Scope:
|
|
15
|
+
|
|
16
|
+
| Source | Search |
|
|
17
|
+
| ------------- | -------------------------------------------------------------------------------------- |
|
|
18
|
+
| Code | Domain terms, UI labels, messages, routes; start next to related features' code |
|
|
19
|
+
| Issue tracker | Text search inside the catalog's Scope query (JQL `text ~ "term"`, `gh search issues`) |
|
|
20
|
+
| Requirements | Search under the root page or space (CQL `text ~ "term"`); headings of long documents |
|
|
21
|
+
| User docs | Docs site search or nav; `grep` in the docs repo |
|
|
22
|
+
| Other | The source's own search |
|
|
23
|
+
|
|
24
|
+
- Found at the source, missing from the catalog → use it and report a catalog error.
|
|
25
|
+
- Found, but outside the catalog's Scope (open issue, draft, deprecated page or copy, other version) → not a behavior. Mention it on the mismatch page when it explains one.
|
|
26
|
+
- Not found → the source is silent on the feature. Record where you searched.
|
|
27
|
+
|
|
28
|
+
## Read
|
|
29
|
+
|
|
30
|
+
Read every matching feature at its links (documents: under its headings). Behaviors come from:
|
|
31
|
+
|
|
32
|
+
| Source | Behaviors from | Not behaviors |
|
|
33
|
+
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
34
|
+
| Code | The code at the feature's `code` paths (else its links); Behaviors on its feature page in `features/`, when present, as a start | Claims (comments, docstrings, developer docs, API specs, configs, tests); stubs, routes users cannot reach; internal steps with no visible effect |
|
|
35
|
+
| Issue tracker | Stories and feature issues: acceptance criteria, then the description. Epics: their scope. Bugs: the expected result | A bug's reported actual result; tasks, spikes, chores, tech debt |
|
|
36
|
+
| Requirements | Functional requirements, business rules, acceptance criteria, flows; one behavior per table row and per meaningful matrix cell; exclusions as negative ones | Future phases, nice-to-have, MoSCoW Could and Won't; goals, rationale, personas, metrics, open questions |
|
|
37
|
+
| User docs | Guides, how-tos, limits, notes and warnings; badges (plan, edition, role, platform) as conditions; navigation steps | Screenshots and videos |
|
|
38
|
+
| Test cases | Expected results | Steps, test data |
|
|
39
|
+
|
|
40
|
+
- Code behind a flag or setting that can be turned on → a behavior with that condition (as `wiki-from-code`).
|
|
41
|
+
- Comments and later edits may change the body ("dropped", "moved to PROJ-456", "decided: 3 retries, not 5"): state the final behavior, never the comment.
|
|
42
|
+
- Acceptance criteria may sit in a custom field, a description heading, a checklist, or sub-tasks: find the convention in the first issues.
|
|
43
|
+
|
|
44
|
+
## Behaviors
|
|
45
|
+
|
|
46
|
+
### Shape
|
|
47
|
+
|
|
48
|
+
- One behavior per line, one sentence, present tense.
|
|
49
|
+
- No `should`, `shall`, `must`, `will`: state it as a fact. The source's role says whether it is actual, intended, or documented.
|
|
50
|
+
- Name who acts or what changes, in domain terms. Never "the system", "it", or "we".
|
|
51
|
+
- Outcome is what the actor can observe (screen, response, file, email, notification), not how the product gets there.
|
|
52
|
+
- Conditions stay in the behavior: role, platform, setting, state.
|
|
53
|
+
- Source key, when the behavior has one, goes first.
|
|
54
|
+
- Negative behaviors count: `Viewer cannot delete a project.`
|
|
55
|
+
- Split on "and" when each part can be checked on its own.
|
|
56
|
+
|
|
57
|
+
### Values and words
|
|
58
|
+
|
|
59
|
+
- Values as the source states them, with units. No value in the source → none in the behavior. Never round, convert, or fill one in; a vague behavior stays vague (open question).
|
|
60
|
+
- Code: exact values from the code that runs; a value read from config or environment: its default, marked `(configurable)` (as `wiki-from-code`).
|
|
61
|
+
- UI labels and messages verbatim, in quotes; navigation paths with `›`.
|
|
62
|
+
- Domain terms (prefer the user docs' terms), one per concept across sources and `features/`.
|
|
63
|
+
- No class, function, table, file, flag, or library names. Protocol details (endpoints, status codes, fields) only when the API is what users use.
|
|
64
|
+
- Keep the source's level of detail: a coarse behavior stays coarse. Never merge behaviors and drop values or conditions.
|
|
65
|
+
- Restate, never paste. Never people, comments, or customer data.
|
|
66
|
+
|
|
67
|
+
## Examples
|
|
68
|
+
|
|
69
|
+
| Source says | Behavior |
|
|
70
|
+
| --------------------------------------------------------------------- | ------------------------------------------------------------------- |
|
|
71
|
+
| The system shall allow users to store payment methods (max 5). | `Customer can save up to 5 cards.` |
|
|
72
|
+
| Users can save cards to pay faster. | `Customer can save a card.` (benefit dropped) |
|
|
73
|
+
| AC: if the card is declined, show "Payment failed. Try another card." | `Declined card: Customer sees "Payment failed. Try another card."` |
|
|
74
|
+
| Enterprise customers get SSO; Pro is coming soon. | `Enterprise plan: SSO sign-in is available.` (Pro: future, dropped) |
|
|
75
|
+
| Session times out after a while. | `Session expires after inactivity.` (no value invented) |
|
|
76
|
+
| Bug: saving the same card twice creates a duplicate; expected: one. | `Saving the same card again keeps one saved card.` |
|
|
77
|
+
|
|
78
|
+
## Check
|
|
79
|
+
|
|
80
|
+
- [ ] Every source either has the feature or records where it was searched
|
|
81
|
+
- [ ] One observable behavior per line, present tense, who or what named
|
|
82
|
+
- [ ] Values, labels, and conditions exactly as in the source; none added
|
|
83
|
+
- [ ] Domain terms, the same term for the same thing
|
|
84
|
+
- [ ] No implementation, plans, rationale, work items, people, or copied text
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# Mismatches layout
|
|
2
|
+
|
|
3
|
+
Reads catalogs (layout owned by `catalog-features`) and feature pages (owned by `wiki-from-code`); changes them only through those skills (SKILL.md › 2. Catalogs). Writes mismatch pages into `mismatches/`
|
|
4
|
+
|
|
5
|
+
```
|
|
6
|
+
mismatches/ # output folder
|
|
7
|
+
README.md # sitemap; one line each
|
|
8
|
+
sources/
|
|
9
|
+
README.md # every catalog here: name, type, role, link, Date
|
|
10
|
+
{name}/ # source name
|
|
11
|
+
features/ # feature pages (wiki-from-code), when present
|
|
12
|
+
.catalog/ # internal: `catalog-features` skill output
|
|
13
|
+
mismatches/
|
|
14
|
+
README.md # index
|
|
15
|
+
{mismatch}.md # one mismatch per disagreement, every compared source
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Formatting
|
|
19
|
+
|
|
20
|
+
Every page: `# Title`, then a breadcrumb that matches folders (links to the parent pages only), then the body. Mismatch pages: no breadcrumb. Relative links to pages in the output folder and to repo paths; source features as key + URL.
|
|
21
|
+
|
|
22
|
+
## Kinds
|
|
23
|
+
|
|
24
|
+
| Kind | Meaning | Where |
|
|
25
|
+
| ------------------- | ---------------------------------------------------------------------------- | ----------- |
|
|
26
|
+
| Differs | Sources disagree on one behavior: values, conditions, roles, labels, flow | Page |
|
|
27
|
+
| Not implemented | Behavior from another source that the code lacks | Page |
|
|
28
|
+
| Missing in {source} | Feature a source is silent on, within its catalog Scope (SKILL.md › Sources) | README list |
|
|
29
|
+
|
|
30
|
+
## README
|
|
31
|
+
|
|
32
|
+
`mismatches/README.md` missing → create it: `# Mismatches`, breadcrumb. Write the mismatches list.
|
|
33
|
+
|
|
34
|
+
1. Compared: one row per source: name, role, scope, history, catalog link, Date (code: baseline commit). Product version. Check date.
|
|
35
|
+
2. Differs, Not implemented: one list per kind, highest impact first: page link + impact level + sources + likely cause.
|
|
36
|
+
3. Missing features: one list per silent source (Not implemented, Unspecified, Undocumented, Missing in {source}): feature + where found (catalog feature links) + where searched.
|
|
37
|
+
4. Open questions: vague behaviors, unread content, or unclear Scope that blocks a comparison.
|
|
38
|
+
|
|
39
|
+
## Page
|
|
40
|
+
|
|
41
|
+
A reader learns what disagrees from the first lines, without opening the sources.
|
|
42
|
+
|
|
43
|
+
File name: follow existing file name pattern or kebab-case of the title by default.
|
|
44
|
+
|
|
45
|
+
- `# Title`: the behavior in question, as the user meets it.
|
|
46
|
+
- `Feature:` the feature area › title; link to its feature page in `features/`, when present.
|
|
47
|
+
- `## Difference`: first a one-sentence summary of how they differ, then one bullet per side, code first when compared; sources that agree grouped on one bullet. Each bullet: the sources (key or name), then their behavior. Write each side so it is clear alone: the condition, then what the user gets. Not implemented: the code bullet says what is missing and where it was searched.
|
|
48
|
+
- `## Impact`: level first: **High** (users blocked, data or money wrong), **Medium** (a workaround exists), **Low** (wording, cosmetic). Then who meets it, when, and what they see or cannot do.
|
|
49
|
+
- `## Likely cause`: SKILL.md › Likely cause, with the dates that decide it.
|
|
50
|
+
- `## Solution`: suggested fix, from the likely cause: Bug → change the code; Stale requirement → update the requirement; Stale docs → update the docs; Decide → the options for the product owner.
|
|
51
|
+
- `## Sources`: evidence, one row per compared source; Code row always, first, when code is compared.
|
|
52
|
+
- Other info required to explain the mismatch.
|
|
53
|
+
|
|
54
|
+
### Example
|
|
55
|
+
|
|
56
|
+
```markdown
|
|
57
|
+
# Expired cards at checkout
|
|
58
|
+
|
|
59
|
+
Feature: [Checkout › Pay with saved card](../sources/code/features/checkout/pay-with-saved-card.md)
|
|
60
|
+
|
|
61
|
+
## Difference
|
|
62
|
+
|
|
63
|
+
The code lists expired saved cards at checkout, greyed out; Confluence says they are not listed.
|
|
64
|
+
|
|
65
|
+
- **Code, PAY-112:** Expired saved cards are listed at checkout, greyed out and not selectable.
|
|
66
|
+
- **Confluence "Saved cards":** Expired saved cards are not listed at checkout.
|
|
67
|
+
|
|
68
|
+
## Impact
|
|
69
|
+
|
|
70
|
+
Low. A customer with an expired card sees it greyed out at checkout. Support answers based on Confluence say it should not appear.
|
|
71
|
+
|
|
72
|
+
## Likely cause
|
|
73
|
+
|
|
74
|
+
Stale requirement: PAY-112 (2026-08-28) changed the behavior after the Confluence page (2026-01-10).
|
|
75
|
+
|
|
76
|
+
## Solution
|
|
77
|
+
|
|
78
|
+
Update the Confluence page "Saved cards": expired saved cards are listed at checkout, greyed out and not selectable.
|
|
79
|
+
|
|
80
|
+
## Sources
|
|
81
|
+
|
|
82
|
+
| Source | Says | Link | Updated |
|
|
83
|
+
| ---------------------- | ----------------------------------------------------- | ------------------------------------------------ | ---------- |
|
|
84
|
+
| Code (actual) | Expired saved cards are listed, greyed out | `src/checkout/cards.ts:48-61` | 2026-08-30 |
|
|
85
|
+
| Jira (intended) | PAY-112: Expired saved cards are listed, not selectable | [PAY-112](https://example.atlassian.net/...) | 2026-08-28 |
|
|
86
|
+
| Confluence (intended) | Expired saved cards are not listed | [Saved cards](https://example.atlassian.net/...) | 2026-01-10 |
|
|
87
|
+
| User docs (documented) | silent | Searched: docs/payments/, site search "expired" | |
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
- `Says`: the source's behavior, as [features.md](features.md) › Behaviors describes; `silent` when it has none.
|
|
91
|
+
- `Link`: repo file: `path:lines`; issue: key + URL; page: title + URL, with the heading when the page is long; silent: where searched.
|
|
92
|
+
- `Updated`: the date read at the source (SKILL.md › Likely cause).
|
|
93
|
+
- Superseded issues of a source with history: omit, or mark `superseded by {key}` when it explains the mismatch.
|
|
@@ -1,124 +1,111 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: wiki-from-code
|
|
3
3
|
description: >-
|
|
4
|
-
Builds
|
|
5
|
-
|
|
6
|
-
Use when the user asks to build or
|
|
4
|
+
Builds a product wiki from the application source code or updates it after large code changes (release, refactor, many merged PRs).
|
|
5
|
+
Records mismatches between code and claims about it (comments, docstrings, developer docs, API specs, configs).
|
|
6
|
+
Use when the user asks to build or update the wiki/requirements/specification/docs from code.
|
|
7
7
|
metadata:
|
|
8
8
|
author: Testomat.io
|
|
9
|
-
version:
|
|
9
|
+
version: 2.0.0
|
|
10
10
|
---
|
|
11
11
|
|
|
12
12
|
# Wiki from code
|
|
13
13
|
|
|
14
|
-
Wiki of **how the product works now
|
|
14
|
+
Wiki of **how the product works now**: one page per feature, listing its behaviors, and mismatches between the code and other repo claims.
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
Output folder:
|
|
17
|
+
- default: `wiki/` (in the repo root)
|
|
18
|
+
- or given by calling skill or the user
|
|
19
|
+
- all paths are relative inside the output folder
|
|
20
|
+
- if there is `features-catalog/` dir → move it to `features/.catalog/`
|
|
17
21
|
|
|
18
|
-
**
|
|
19
|
-
|
|
20
|
-
Decide audience once:
|
|
21
|
-
|
|
22
|
-
| Audience | Human wiki |
|
|
23
|
-
| --- | --- |
|
|
24
|
-
| End users (default, preferred) | Behavior and acceptance only. No file paths, types, stack, or protocol internals. |
|
|
25
|
-
| Callers who are not end users (API, job, contract etc.) | Technical entry could be treated as requirements. |
|
|
22
|
+
**Mismatch** is a disagreement between code and a claim about it (comment, docstring, developer docs, API spec, config, infra, test). They could differ, be not implemented, not used, etc.
|
|
26
23
|
|
|
27
24
|
## Modes
|
|
28
25
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
|
32
|
-
|
|
|
33
|
-
| `bootstrap` | First wiki from this repo |
|
|
34
|
-
| `refresh` | Code moved; update implicit pages + catalog |
|
|
35
|
-
|
|
36
|
-
Default root: `wiki/` (or the user’s path).
|
|
37
|
-
|
|
38
|
-
Move from **overview** to **details**. Do not write feature pages before the tree exists.
|
|
26
|
+
| Mode | When |
|
|
27
|
+
| -------- | --------------------------------------------------------------------------------------- |
|
|
28
|
+
| `build` | No `features/` or baseline in the root `README.md`, no git, or the user asks to rebuild |
|
|
29
|
+
| `update` | `features/` and baseline exist: change only what the code changed since the baseline |
|
|
39
30
|
|
|
40
31
|
## Steps
|
|
41
32
|
|
|
42
33
|
```
|
|
43
|
-
- [ ] 1. Catalog
|
|
44
|
-
- [ ] 2.
|
|
45
|
-
- [ ] 3.
|
|
46
|
-
- [ ] 4.
|
|
34
|
+
- [ ] 1. Catalog
|
|
35
|
+
- [ ] 2. Indexes
|
|
36
|
+
- [ ] 3. Feature pages (one subagent per area)
|
|
37
|
+
- [ ] 4. Merge
|
|
47
38
|
- [ ] 5. Validate
|
|
48
39
|
- [ ] 6. Report
|
|
49
40
|
```
|
|
50
41
|
|
|
51
|
-
Read [references/layout.md](references/layout.md) before writing files.
|
|
42
|
+
Read [references/behaviors.md](references/behaviors.md) and [references/layout.md](references/layout.md) before writing files.
|
|
52
43
|
|
|
53
44
|
## Rules
|
|
54
45
|
|
|
55
|
-
-
|
|
56
|
-
-
|
|
57
|
-
-
|
|
58
|
-
-
|
|
59
|
-
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
The **catalog** is what exists in the product: first a shallow area → feature list, then YAML facts in `catalog/`. Code, test, doc, and issue tracker links live here (`sources`, `tests`, `wiki_path`, `issues`).
|
|
64
|
-
|
|
65
|
-
From **code** map the product into **areas**, each with a **feature list**. That list *is* the catalog of what exists in the product at this step. Do not draft Requirement/Acceptance or YAML yet.
|
|
46
|
+
- Source is the repository only.
|
|
47
|
+
- **Current-state code wins** over claims (comments, docstrings, developer docs, API specs, configs, tests, etc.). What they claim that the code lacks or does differently goes in `mismatches/` only, never as a "should" on a feature page.
|
|
48
|
+
- Write only paths that exist in the repo. Never guess or build them.
|
|
49
|
+
- Name areas and features from the product and user's perspective, NOT from the code/stack/files/classes/etc.
|
|
50
|
+
- Domain terms (prefer the docs' terms), one term per concept across the wiki.
|
|
51
|
+
- Update catalog file names and content as you get more info about the features.
|
|
52
|
+
- Treat and describe features as end-to-end flows or use cases, not just "actions" from code.
|
|
66
53
|
|
|
67
|
-
|
|
54
|
+
## 1. Catalog
|
|
68
55
|
|
|
69
|
-
|
|
56
|
+
Run `catalog-features` skill on the repo, same mode, output folder `features/.catalog/`.
|
|
70
57
|
|
|
71
|
-
|
|
58
|
+
`update`: get the baseline commit > `git diff --name-status -M {baseline}..HEAD` > process updated/removed files accordingly.
|
|
72
59
|
|
|
73
|
-
|
|
60
|
+
## 2. Indexes
|
|
74
61
|
|
|
75
|
-
Write
|
|
62
|
+
Write the `features/` indexes before any feature page.
|
|
76
63
|
|
|
77
|
-
|
|
64
|
+
[layout.md](references/layout.md)
|
|
78
65
|
|
|
79
|
-
|
|
66
|
+
## 3. Feature pages
|
|
80
67
|
|
|
81
|
-
|
|
68
|
+
One subagent per area, in parallel (`update`: affected areas only). Each subagent gets: the area, its subareas and catalog features (feature YAMLs), allowed code paths, the Domain from `features/.catalog/README.md`, the rules above, [references/behaviors.md](references/behaviors.md), [references/layout.md](references/layout.md). `update` also: the area's existing pages and mismatches, and its changed features.
|
|
82
69
|
|
|
83
|
-
|
|
70
|
+
Subagent, per feature, starting from its catalog links:
|
|
84
71
|
|
|
85
|
-
|
|
72
|
+
1. In its catalog feature YAML: adds implementation files to `code`, and adds `tests`.
|
|
73
|
+
2. Writes the feature page ([layout.md](references/layout.md) › Feature page): Description, then Behaviors ([behaviors.md](references/behaviors.md)).
|
|
74
|
+
3. Returns one mismatch per disagreement: its kind and every section of a mismatch page ([layout.md](references/layout.md) › Mismatches).
|
|
86
75
|
|
|
87
|
-
|
|
76
|
+
Subagent writes only inside its area, in `features/` and `features/.catalog/`. Catalog features found missing, misnamed, or misplaced → fix them in the area's catalog files. Anything outside the area (mismatch pages, catalog README, other areas) → return it.
|
|
88
77
|
|
|
89
|
-
|
|
78
|
+
Subagent returns: files added, changed, and removed; mismatches (`update`: also resolved ones); changes outside its area; open questions; catalog errors; terms the Domain lacks.
|
|
90
79
|
|
|
91
|
-
|
|
80
|
+
## 4. Merge
|
|
92
81
|
|
|
93
|
-
|
|
82
|
+
1. Apply the changes subagents returned outside their areas, including links to renamed or moved pages. Terms the Domain lacks → add to the Domain in `features/.catalog/README.md`.
|
|
83
|
+
2. Update every `README.md` in `features/` to list the feature pages that now exist. Write the root `README.md`: a link and purpose for each folder (not `.catalog/`), plus the baseline commit.
|
|
84
|
+
3. Write the mismatch pages subagents returned. Same disagreement from several areas → one mismatch page.
|
|
85
|
+
4. Two terms for one concept → keep one, fix the pages that use the other.
|
|
86
|
+
5. `update`: remove resolved mismatches.
|
|
87
|
+
6. In `mismatches/README.md`, write the mismatches list ([layout.md](references/layout.md) › Mismatches).
|
|
94
88
|
|
|
95
89
|
## 5. Validate
|
|
96
90
|
|
|
97
91
|
```markdown
|
|
98
92
|
Structure
|
|
99
|
-
- [ ]
|
|
100
|
-
- [ ] catalog
|
|
101
|
-
- [ ]
|
|
102
|
-
- [ ] Area (and group) folders named from the product, not from stack/code
|
|
103
|
-
- [ ] Feature pages use the layout heading order (Requirement, Acceptance); breadcrumbs match folder depth; links are relative
|
|
104
|
-
- [ ] Feature pages link to the paired catalog YAML (relative path)
|
|
105
|
-
- [ ] Feature pages are not mega-dumps (one cluster per page)
|
|
106
|
-
- [ ] Feature pages not overloaded with meta info (it is located in catalog YAML)
|
|
107
|
-
- [ ] Tracker links are in catalog YAML (`issues`); omitted when declined or none matched; no invented keys/URLs
|
|
93
|
+
- [ ] README.md is a sitemap with the baseline commit
|
|
94
|
+
- [ ] Every catalog feature has a feature page, and its feature YAML lists the rest of its implementation files in `code` and its tests in `tests`; `features/` mirrors the catalog's areas and subareas
|
|
95
|
+
- [ ] Every `links`, `code`, and `tests` path exists in the repo
|
|
108
96
|
|
|
109
97
|
Content
|
|
110
|
-
- [ ]
|
|
111
|
-
- [ ]
|
|
112
|
-
- [ ]
|
|
113
|
-
- [ ] mismatches/ covers disagreements (between code and other sources)
|
|
98
|
+
- [ ] Every feature page has a Description and at least one behavior under Behaviors; every behavior passes the check in [references/behaviors.md](references/behaviors.md)
|
|
99
|
+
- [ ] Every disagreement between code and claims has a mismatch page
|
|
100
|
+
- [ ] update: resolved mismatches removed; nothing left for deleted code
|
|
114
101
|
```
|
|
115
102
|
|
|
116
103
|
## 6. Report
|
|
117
104
|
|
|
118
|
-
Where to start
|
|
105
|
+
Where to start. Results (`update`: features added, changed, removed). Mismatches by kind, highest user impact first. Catalog errors. Open questions.
|
|
119
106
|
|
|
120
107
|
## Next actions
|
|
121
108
|
|
|
122
|
-
-
|
|
123
|
-
-
|
|
124
|
-
- Test cases for
|
|
109
|
+
- Compare with issues, requirements, user docs → `detect-product-mismatches`
|
|
110
|
+
- Explain a feature in QA language → `qa-explain-behavior`
|
|
111
|
+
- Test cases for a feature → `qa-write-test-cases`
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Behaviors
|
|
2
|
+
|
|
3
|
+
A behavior is one observable statement about how the product works now: what someone who uses it (end user, admin, API client, integrated system) can do, see, or get, and when. Feature pages list them under Behaviors.
|
|
4
|
+
|
|
5
|
+
## From code
|
|
6
|
+
|
|
7
|
+
- Code that runs now and users can reach. Behind a flag or setting that can be turned on → a behavior with that condition.
|
|
8
|
+
- Comments, docstrings, developer docs, API specs, configs, and tests are claims, not behaviors. Where they disagree with the code → `mismatches/`.
|
|
9
|
+
- Never a behavior: internal steps with no visible effect (caching, logging, internal retries), refactoring, helpers.
|
|
10
|
+
|
|
11
|
+
## Shape
|
|
12
|
+
|
|
13
|
+
- One behavior per bullet, one sentence, present tense.
|
|
14
|
+
- No `should`, `shall`, `must`, `will`: state it as a fact, present tense.
|
|
15
|
+
- Name who acts or what changes, in domain terms. Never "the system", "it", or "we".
|
|
16
|
+
- Outcome is what the actor can observe (screen, response, file, email, notification), not how the code gets there.
|
|
17
|
+
- Conditions stay in the behavior: role, platform, setting, state.
|
|
18
|
+
- Negative behaviors count: `Viewer cannot delete a project.`
|
|
19
|
+
- Split on "and" when each part can be checked on its own.
|
|
20
|
+
|
|
21
|
+
## Values and words
|
|
22
|
+
|
|
23
|
+
- Exact values from the code that runs: limits, defaults, timeouts, sizes, formats.
|
|
24
|
+
- A value read from config or environment: its default, marked `(configurable)`.
|
|
25
|
+
- UI labels and messages verbatim, in quotes; navigation paths with `›`.
|
|
26
|
+
- No class, function, table, file, flag, or library names. Protocol details (endpoints, status codes, fields) only when the API is what users use.
|
|
27
|
+
|
|
28
|
+
## Examples
|
|
29
|
+
|
|
30
|
+
| Code | Behavior |
|
|
31
|
+
| --------------------------------------------------- | ------------------------------------------------------------ |
|
|
32
|
+
| `if (cards.length >= 5) throw new CardLimitError()` | `Customer can save up to 5 cards.` |
|
|
33
|
+
| `IDLE_TIMEOUT = env.IDLE_TIMEOUT ?? 1800` (seconds) | `Session expires after 30 min of inactivity (configurable).` |
|
|
34
|
+
| `cache.set(user_id, cards)` | Not a behavior: no visible effect |
|
|
35
|
+
|
|
36
|
+
## Check
|
|
37
|
+
|
|
38
|
+
- [ ] Every reachable behavior in the feature's code is listed
|
|
39
|
+
- [ ] One observable behavior per bullet, present tense, who or what named
|
|
40
|
+
- [ ] Exact values from the code; config values marked `(configurable)`
|
|
41
|
+
- [ ] Domain terms; no stack names unless the API is what users use
|
|
42
|
+
- [ ] No internal steps; nothing taken from comments, docs, or tests
|
|
@@ -1,94 +1,93 @@
|
|
|
1
1
|
# Wiki layout
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
`features/` mirrors `features/.catalog/`: same areas and subareas, one feature page per catalog feature. Every folder has a README. Tree may be updated while the wiki is built.
|
|
4
4
|
|
|
5
|
-
Default tree is multiple levels under `capabilities/`. Add sub-folders (with required depth) when the project is complex enough that an area needs a named group. Each with a README.
|
|
6
5
|
|
|
7
6
|
```
|
|
8
|
-
|
|
9
|
-
README.md
|
|
10
|
-
|
|
11
|
-
README.md
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
{
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
{feature}.yaml
|
|
23
|
-
{group}/
|
|
24
|
-
README.md
|
|
25
|
-
{feature}.yaml
|
|
26
|
-
mismatches/ # optional — create when a disagreement exists
|
|
27
|
-
README.md
|
|
7
|
+
{output}/ # output folder
|
|
8
|
+
README.md # sitemap, baseline commit
|
|
9
|
+
features/
|
|
10
|
+
README.md # areas
|
|
11
|
+
.catalog/ # internal: code catalog (catalog-features)
|
|
12
|
+
{area}/ # any depth, as in the catalog
|
|
13
|
+
README.md # subareas and features
|
|
14
|
+
{feature}.md # feature page
|
|
15
|
+
{subarea}/ # as in the catalog
|
|
16
|
+
README.md # subareas and features
|
|
17
|
+
{feature}.md # feature page
|
|
18
|
+
mismatches/ # code vs claims in the repo
|
|
19
|
+
README.md # index
|
|
20
|
+
{mismatch}.md # one behavior, every source that disagrees
|
|
28
21
|
```
|
|
29
22
|
|
|
30
|
-
|
|
23
|
+
## Feature page
|
|
31
24
|
|
|
32
|
-
|
|
25
|
+
- Breadcrumb that matches folders.
|
|
26
|
+
- Feature description.
|
|
27
|
+
- `## Behaviors`: [behaviors.md](behaviors.md).
|
|
33
28
|
|
|
34
|
-
|
|
35
|
-
| --- | --- |
|
|
36
|
-
| Default | `capabilities/{area}/{feature}.md` |
|
|
37
|
-
| Extra depth | New group folder when the area (or group) is too crowded or a cluster needs a name. Any depth. |
|
|
38
|
-
| Split | New sibling `{feature}.md`, a new group folder, or a new `{area}/`. |
|
|
39
|
-
| Share | One page owns a shared rule; others link. Same behavior, two entries → one page. |
|
|
40
|
-
| Growth | New behavior → new leaf + catalog entry + one new index line. Do not thicken an existing leaf into a mega-page. |
|
|
41
|
-
| Overview first | Indexes exist before any leaf is written. |
|
|
29
|
+
## Catalog feature YAML
|
|
42
30
|
|
|
43
|
-
|
|
31
|
+
- No key (cause does not exist in code).
|
|
32
|
+
- `links` become `code` paths.
|
|
44
33
|
|
|
45
|
-
|
|
34
|
+
```yaml
|
|
35
|
+
# features/.catalog/checkout/pay-with-saved-card.yaml
|
|
36
|
+
title: Pay with saved card # catalog-feature
|
|
37
|
+
code: # implementation files
|
|
38
|
+
- src/api/cards.ts
|
|
39
|
+
tests: # tests that cover the feature
|
|
40
|
+
- test/checkout/cards.spec.ts
|
|
41
|
+
```
|
|
46
42
|
|
|
47
|
-
|
|
43
|
+
## Mismatches
|
|
48
44
|
|
|
49
|
-
|
|
50
|
-
| --- | --- |
|
|
51
|
-
| `wiki/README.md` | Roots (`capabilities/`, `catalog/`, `changes/` / `mismatches/` (if present)) |
|
|
52
|
-
| `capabilities/README.md` | Every area: link + purpose (markdown indexes only — not catalog YAML) |
|
|
53
|
-
| `capabilities/{area}/README.md` | Every child feature (or group): link + purpose |
|
|
54
|
-
| Extra group `README.md` | Every child feature (or deeper group): link + purpose |
|
|
55
|
-
| `catalog/README.md` | Every area: link to that area’s catalog README |
|
|
56
|
-
| `catalog/{area}/README.md` | Every child YAML: link + purpose (YAML indexes only — not feature pages) |
|
|
45
|
+
`mismatches/README.md` missing → create it: `# Mismatches`, breadcrumb. Write the list of mismatches, grouped by kind (not implemented, differs, unwired/not used, etc.), highest impact first: page link + impact level.
|
|
57
46
|
|
|
58
|
-
|
|
47
|
+
### Mismatch page
|
|
59
48
|
|
|
60
|
-
|
|
49
|
+
A reader learns what disagrees from the first lines, without opening the sources.
|
|
61
50
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
- key: PROJ-123
|
|
74
|
-
url: https://example.com/browse/PROJ-123
|
|
75
|
-
behaviors:
|
|
76
|
-
- One as-implemented fact (observable outcome).
|
|
77
|
-
```
|
|
51
|
+
File name: follow existing file name pattern or kebab-case of the title by default.
|
|
52
|
+
|
|
53
|
+
- `# Title`: the behavior in question, as the user meets it.
|
|
54
|
+
- `Feature:` the feature area › title, linked to its feature page.
|
|
55
|
+
- `## Difference`: first a one-sentence summary of how they differ, then one bullet for the code, first, then one per claim; claims that agree grouped on one bullet. Each bullet: the source, then its behavior. Write each side so it is clear alone: the condition, then what the user gets.
|
|
56
|
+
- `## Impact`: level first: **High** (users blocked, data or money wrong), **Medium** (a workaround exists), **Low** (wording, cosmetic). Then who meets it, when, and what they see or cannot do.
|
|
57
|
+
- `## Solution`: suggested fix: change the code, or correct the claim.
|
|
58
|
+
- `## Sources`: evidence, one row per source; Code row always, first. Code, Comment, Docstring, README, Developer docs, API spec, Config, Infra, Test, etc.
|
|
59
|
+
- Other info required to explain the mismatch.
|
|
60
|
+
|
|
61
|
+
#### Example
|
|
78
62
|
|
|
79
|
-
|
|
63
|
+
```markdown
|
|
64
|
+
# Upload retries after a network error
|
|
80
65
|
|
|
81
|
-
|
|
66
|
+
Feature: [Reports › Upload results](../features/reports/upload-results.md)
|
|
82
67
|
|
|
83
|
-
|
|
68
|
+
## Difference
|
|
84
69
|
|
|
85
|
-
|
|
70
|
+
The code retries a failed upload 3 times; the docstring and developer docs say 5.
|
|
86
71
|
|
|
87
|
-
|
|
72
|
+
- **Code:** A failed upload is retried 3 times, then the run shows "Upload failed".
|
|
73
|
+
- **Docstring, developer docs:** A failed upload is retried 5 times.
|
|
88
74
|
|
|
89
|
-
|
|
75
|
+
## Impact
|
|
76
|
+
|
|
77
|
+
Low. On a flaky network, "Upload failed" shows after 3 attempts, sooner than a developer reading the docs expects.
|
|
78
|
+
|
|
79
|
+
## Solution
|
|
80
|
+
|
|
81
|
+
Correct the docstring and developer docs: 3 retries.
|
|
82
|
+
|
|
83
|
+
## Sources
|
|
84
|
+
|
|
85
|
+
| Source | Says | Link | Updated |
|
|
86
|
+
| -------------- | ---------------------------- | --------------------------- | ---------- |
|
|
87
|
+
| Code (actual) | Retries 3 times, then fails | `src/upload/client.ts:40-52` | 2026-09-02 |
|
|
88
|
+
| Docstring | "Retries up to 5 times" | `src/upload/client.ts:31` | 2026-03-11 |
|
|
89
|
+
| Developer docs | "5 retries before giving up" | `docs/upload.md:18` | 2026-03-11 |
|
|
90
|
+
```
|
|
90
91
|
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
| Implemented | Always — current behavior based on code |
|
|
94
|
-
| Intended | Always — the non-wiki source that disagrees with those files (comment, doc, infra, issue, unused config, or stub) |
|
|
92
|
+
- `Link`: repo file: `path:lines`; outside the repo: URL, with the heading when the page is long.
|
|
93
|
+
- `Updated`: repo file: date the cited line last changed (`git blame --date=short -L {line},+1 -- {path}`), doc section without a line: the section's last change; URL: the page's "last updated" date, else empty.
|