@ia-qa/qa-discovery 0.1.2 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +52 -1
- package/TUTORIAL.md +433 -0
- package/dist/cli/args.js +1 -0
- package/dist/cli/args.js.map +1 -1
- package/dist/cli/coverage.d.ts +20 -0
- package/dist/cli/coverage.js +178 -0
- package/dist/cli/coverage.js.map +1 -0
- package/dist/cli/index.d.ts +1 -1
- package/dist/cli/index.js +23 -0
- package/dist/cli/index.js.map +1 -1
- package/dist/cli/scan.d.ts +3 -1
- package/dist/cli/scan.js +3 -2
- package/dist/cli/scan.js.map +1 -1
- package/dist/config.d.ts +13 -0
- package/dist/config.js.map +1 -1
- package/dist/coverage.d.ts +150 -0
- package/dist/coverage.js +263 -0
- package/dist/coverage.js.map +1 -0
- package/dist/coverageView.d.ts +28 -0
- package/dist/coverageView.js +154 -0
- package/dist/coverageView.js.map +1 -0
- package/dist/htmlReport.js +8 -1
- package/dist/htmlReport.js.map +1 -1
- package/dist/mcp/server.d.ts +28 -0
- package/dist/mcp/server.js +47 -0
- package/dist/mcp/server.js.map +1 -1
- package/dist/overview.d.ts +9 -0
- package/dist/overview.js +10 -1
- package/dist/overview.js.map +1 -1
- package/dist/overviewFile.js +23 -0
- package/dist/overviewFile.js.map +1 -1
- package/package.json +62 -61
package/README.md
CHANGED
|
@@ -6,7 +6,8 @@
|
|
|
6
6
|
|
|
7
7
|
It answers *what is there*. It does not yet tell you what to test, and it never returns a pass/fail verdict: this is reconnaissance, and pretending otherwise would be the most useful-looking lie a tool like this could tell.
|
|
8
8
|
|
|
9
|
-
**Where this is going.** Reconnaissance is step one of a longer goal: reproduce what a senior QA engineer does when handed an unfamiliar app — work out what it is *for*, where it would hurt most if it broke, and produce a prioritised test plan you can question line by line. Every stage after this one reads the capture below and must cite it, because a plan nobody can interrogate is a plan nobody can trust.
|
|
9
|
+
**Where this is going.** Reconnaissance is step one of a longer goal: reproduce what a senior QA engineer does when handed an unfamiliar app — work out what it is *for*, where it would hurt most if it broke, and produce a prioritised test plan you can question line by line. Every stage after this one reads the capture below and must cite it, because a plan nobody can interrogate is a plan nobody can trust. `ROADMAP.md`, shipped in this package, has the arc and what this will deliberately never become.
|
|
10
|
+
The tutorial is at [ia-qa.com/devtools/qa-discovery/tutorial](https://www.ia-qa.com/devtools/qa-discovery/tutorial).
|
|
10
11
|
|
|
11
12
|
```bash
|
|
12
13
|
npx @ia-qa/qa-discovery scan https://your-app.example.com --save --report --open
|
|
@@ -16,6 +17,29 @@ npx @ia-qa/qa-discovery scan https://your-app.example.com --save --report --open
|
|
|
16
17
|
|
|
17
18
|
▶ **[See the full report](https://www.ia-qa.com/media/discover-report-full.png)** — one self-contained HTML file, written by `--report`. That is `ia-qa.com` scanned by its own tool, and it found something: all 37 of its input fields carry no `data-testid`, no `id` and no `name`, so any test written against them can only use a positional selector.
|
|
18
19
|
|
|
20
|
+
## Which verb answers which question
|
|
21
|
+
|
|
22
|
+
| Your question | Verb |
|
|
23
|
+
|---|---|
|
|
24
|
+
| what is even in this app? | `scan` — pages, headings, forms, fields, real API calls |
|
|
25
|
+
| my app is behind a login | `login` — a visible browser, you log in, the session is reused |
|
|
26
|
+
| most of the inputs are behind a menu or a tab | `scan --deep` — opens them one level and captures what they reveal |
|
|
27
|
+
| **what does my test suite not test?** | **`coverage`** — the gap, ranked; needs `@ia-qa/self-healing` and one watched run |
|
|
28
|
+
| what changed since last time? | `history` |
|
|
29
|
+
| what is each entry point *for*, and how sensitive? | `ia-qa-discover-ai classify` (optional, your own key) |
|
|
30
|
+
|
|
31
|
+
`ia-qa-discover <verb> --help` for flags. Nothing here emits a verdict and nothing here gates
|
|
32
|
+
CI: a page count is not coverage, and a gap is a decision for a human.
|
|
33
|
+
|
|
34
|
+
**Same operations for an AI agent** — one contract served twice, never a simplified view for
|
|
35
|
+
one of them. `ia-qa-discover-mcp` exposes `scan_app`, `coverage_map`, `classify_app` and
|
|
36
|
+
`discovery_history`. `login` is deliberately not a tool: it waits for a person at a browser.
|
|
37
|
+
|
|
38
|
+
> **New here?** The [step-by-step tutorial](https://www.ia-qa.com/devtools/qa-discovery/tutorial)
|
|
39
|
+
> walks the whole thing from an app you have never seen, with no jargon — first scan, login
|
|
40
|
+
> wall, and "what does my suite not test" (same text as `TUTORIAL.md`, shipped in this
|
|
41
|
+
> package). This README is the short reference.
|
|
42
|
+
|
|
19
43
|
---
|
|
20
44
|
|
|
21
45
|
## What a real session looks like
|
|
@@ -211,6 +235,32 @@ It refuses under CI and without a TTY: it waits for a person twice over, so a pi
|
|
|
211
235
|
|
|
212
236
|
The file it produces holds live cookies — whoever has it is logged in as you. The path is announced before the browser opens, `.ia-qa-discovery/.gitignore` is written on the way out, and nothing here uploads it.
|
|
213
237
|
|
|
238
|
+
### `coverage`
|
|
239
|
+
|
|
240
|
+
**What your test suite does not test.** The one thing neither package can say alone: this one knows what exists, `@ia-qa/self-healing` knows where your suite actually went.
|
|
241
|
+
|
|
242
|
+
```bash
|
|
243
|
+
npx ia-qa-discover coverage
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
Reads local files only — no browser, no network, no argument. It compares the pages this scan found against the pages your suite was **observed** visiting, and ranks the gap by what each page takes as input: credentials, form submissions, required fields, API calls. The factors *are* the ranking — there is no score, because a single number invites tuning and hides that a page ranks high for a reason you may consider irrelevant.
|
|
247
|
+
|
|
248
|
+
It also reports **depth**: how many of each visited page's contracted elements your tests actually name. Named is not asserted — a test that clicks a button names it without checking anything, and nothing on disk can tell the two apart, so this never says "tested".
|
|
249
|
+
|
|
250
|
+
**It needs a suite that was watched running.** That means `@ia-qa/self-healing` installed, `ia-qa-heal map`, and one `ia-qa-heal run`. Without it you get a refusal that names the command, not a number: a contract means somebody configured a page, and only a watched run means a test went there. "No measurement" and "zero coverage" are different sentences, and only one of them would be true.
|
|
251
|
+
|
|
252
|
+
**No percentage, ever.** A `34%` reads as code coverage — a ratio over a denominator a compiler guarantees. Here the denominator is uncertain by construction: a login wall, an app that navigates without `<a href>`, the page cap, states nothing opened. Counts and page names only, always beside what the scan could not see.
|
|
253
|
+
|
|
254
|
+
Writes `.ia-qa-discovery/coverage-map.json` (committable, diffable) and refreshes `_overview.md`. `--report [file.html] [--open]` for the branded dossier, `--json` for the map on stdout.
|
|
255
|
+
|
|
256
|
+
**Closing a line.** A gap list that never shrinks stops being read. Declare pages you deliberately do not test in `outOfScope` in `config.json` — same glob dialect as self-healing's `volatile`:
|
|
257
|
+
|
|
258
|
+
```json
|
|
259
|
+
{ "outOfScope": ["legal-*", "styleguide"] }
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
They stay **counted and named** as out of scope, never silently dropped, and nothing is ever added to that list on its own.
|
|
263
|
+
|
|
214
264
|
### `history`
|
|
215
265
|
|
|
216
266
|
The trend no single scan can reconstruct: pages, forms, API surface, and what changed run over run.
|
|
@@ -338,6 +388,7 @@ A dependency-free JSON-RPC 2.0 stdio server exposing the **same** operations the
|
|
|
338
388
|
|
|
339
389
|
- `scan_app` — crawl and capture. Check `loginWall` and `hrefBlind` in the result before reporting coverage.
|
|
340
390
|
- `classify_app` — the optional BYOK layer below, for an agent. The key is read from an environment variable **named** in the call (`api_key_env`); a raw key is never a tool argument.
|
|
391
|
+
- `coverage_map` — what the suite does **not** test. Returns `measured: false` with a typed `reason` when no run was ever watched: report that as *no measurement*, never as "the suite covers nothing".
|
|
341
392
|
- `discovery_history` — the trend.
|
|
342
393
|
|
|
343
394
|
`login` is deliberately **not** a tool: it waits for a person at a browser, which an agent cannot be. Every path and URL an agent supplies is checked against the project root and the configured origin.
|
package/TUTORIAL.md
ADDED
|
@@ -0,0 +1,433 @@
|
|
|
1
|
+
# QA Discovery — the complete beginner's guide
|
|
2
|
+
|
|
3
|
+
You have been handed a web application. There is no documentation, no test plan, and possibly
|
|
4
|
+
nobody left who built it. Your job is to test it.
|
|
5
|
+
|
|
6
|
+
This guide walks the whole thing, with no jargon. By the end you will have a written record of
|
|
7
|
+
what the app contains, what it takes as input, what it calls — and, if you already have a test
|
|
8
|
+
suite, a ranked list of the pages nobody tests.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## Table of contents
|
|
13
|
+
|
|
14
|
+
1. [Is this for me?](#1-is-this-for-me)
|
|
15
|
+
2. [The 30-second mental model](#2-the-30-second-mental-model)
|
|
16
|
+
3. [Before you start](#3-before-you-start)
|
|
17
|
+
4. [Path A — I use an AI agent](#4-path-a--i-use-an-ai-agent)
|
|
18
|
+
5. [Path B — I use a terminal](#5-path-b--i-use-a-terminal)
|
|
19
|
+
6. [Reading what it wrote](#6-reading-what-it-wrote)
|
|
20
|
+
7. [The four ways a page count lies](#7-the-four-ways-a-page-count-lies)
|
|
21
|
+
8. [What my suite does not test](#8-what-my-suite-does-not-test)
|
|
22
|
+
9. [Doing this every week](#9-doing-this-every-week)
|
|
23
|
+
10. [Optional: what is each entry point *for*?](#10-optional-what-is-each-entry-point-for)
|
|
24
|
+
11. [Troubleshooting](#11-troubleshooting)
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## 1. Is this for me?
|
|
29
|
+
|
|
30
|
+
**Yes, if** you are looking at an app and cannot yet answer *"what is in it?"* — a new job, an
|
|
31
|
+
inherited project, an acquisition, an audit, a QA engineer parachuted onto a team.
|
|
32
|
+
|
|
33
|
+
**Yes, if** you have some tests and no idea how much of the app they miss.
|
|
34
|
+
|
|
35
|
+
**No, if** your question is *"my tests broke, which selector moved?"*. That is a different tool
|
|
36
|
+
in the same family: [`@ia-qa/self-healing`](https://www.npmjs.com/package/@ia-qa/self-healing).
|
|
37
|
+
They work together — section 8 is exactly where they meet.
|
|
38
|
+
|
|
39
|
+
**One thing this tool refuses to do:** give you a pass or a fail. It is reconnaissance. It
|
|
40
|
+
reports what it observed and, just as carefully, what it could **not** see. A tool like this
|
|
41
|
+
that pretended to a verdict would be telling the most useful-looking lie available to it.
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## 2. The 30-second mental model
|
|
46
|
+
|
|
47
|
+
Three ideas and you have it.
|
|
48
|
+
|
|
49
|
+
**It writes down what it sees, and nothing else.** A page's title, its headings, every input,
|
|
50
|
+
every API call the browser really made. If the app does not state something, it stays absent —
|
|
51
|
+
never guessed, never filled in with something plausible.
|
|
52
|
+
|
|
53
|
+
**Every number says what it does not measure.** "9 of 9 pages captured" is meaningless if six
|
|
54
|
+
of them were the login screen. So a count never travels alone: it comes with the login wall it
|
|
55
|
+
hit, the links it could not follow, the cap it stopped at.
|
|
56
|
+
|
|
57
|
+
**It runs on your machine.** A headless browser it brings itself, so `localhost`, a staging box
|
|
58
|
+
or an app behind a VPN all work. Nothing is uploaded. The one exception is an optional, separate
|
|
59
|
+
command in section 10, which uses *your* API key and announces exactly what it sends before it
|
|
60
|
+
sends it.
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## 3. Before you start
|
|
65
|
+
|
|
66
|
+
You need **Node.js 18 or newer**. That is all — the browser comes with the package.
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
node --version
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
### 💡 "I don't want to download another browser"
|
|
73
|
+
|
|
74
|
+
If you already run Playwright, you already have one and it will be reused. If you do not, the
|
|
75
|
+
first run downloads Chromium once (~130 MB) into a shared cache and never again.
|
|
76
|
+
|
|
77
|
+
---
|
|
78
|
+
|
|
79
|
+
## 4. Path A — I use an AI agent
|
|
80
|
+
|
|
81
|
+
If you drive Claude Code, Cursor, Windsurf, VS Code + Copilot or any MCP-capable agent, it can
|
|
82
|
+
run all of this for you.
|
|
83
|
+
|
|
84
|
+
### Step 4.1 — Add the server to your agent
|
|
85
|
+
|
|
86
|
+
Claude Code:
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
claude mcp add qa-discovery -- npx -y @ia-qa/qa-discovery ia-qa-discover-mcp
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Or, for any client that reads a JSON config:
|
|
93
|
+
|
|
94
|
+
```json
|
|
95
|
+
{
|
|
96
|
+
"mcpServers": {
|
|
97
|
+
"qa-discovery": {
|
|
98
|
+
"command": "npx",
|
|
99
|
+
"args": ["-y", "@ia-qa/qa-discovery", "ia-qa-discover-mcp"]
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
}
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
### Step 4.2 — Restart the agent, then ask
|
|
106
|
+
|
|
107
|
+
> Scan https://your-app.example.com and tell me what is in it.
|
|
108
|
+
|
|
109
|
+
The agent calls `scan_app`, gets the capture back, and reads it. Four tools are available to
|
|
110
|
+
it: `scan_app`, `coverage_map`, `classify_app` and `discovery_history` — the same operations
|
|
111
|
+
the terminal has, with the same guarantees.
|
|
112
|
+
|
|
113
|
+
### Step 4.3 — The one thing your agent cannot do
|
|
114
|
+
|
|
115
|
+
**It cannot log you in.** `login` opens a visible browser and waits for a human twice: once at
|
|
116
|
+
the browser, once at the keyboard. That is not something an agent can perform, so it is
|
|
117
|
+
deliberately not exposed as a tool. When the scan reports a login wall, the agent will ask
|
|
118
|
+
*you* to run this, in your own terminal:
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
npx @ia-qa/qa-discovery login
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Then ask it to scan again.
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## 5. Path B — I use a terminal
|
|
129
|
+
|
|
130
|
+
### Step 5.1 — Install
|
|
131
|
+
|
|
132
|
+
```bash
|
|
133
|
+
npm i -D @ia-qa/qa-discovery
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Or run it without installing anything:
|
|
137
|
+
|
|
138
|
+
```bash
|
|
139
|
+
npx @ia-qa/qa-discovery scan https://your-app.example.com --save
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### Step 5.2 — Your first scan
|
|
143
|
+
|
|
144
|
+
```bash
|
|
145
|
+
npx ia-qa-discover scan https://your-app.example.com --save --report --open
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
- `--save` remembers the URL, so later runs are just `ia-qa-discover scan`.
|
|
149
|
+
- `--report` writes a single self-contained HTML file you can send to anyone.
|
|
150
|
+
- `--open` opens it.
|
|
151
|
+
|
|
152
|
+
What you get on screen:
|
|
153
|
+
|
|
154
|
+
```console
|
|
155
|
+
Discovering pages from https://app.example.com…
|
|
156
|
+
🌐 Using bundled Chromium (shared Playwright cache)
|
|
157
|
+
visited / (2 links)
|
|
158
|
+
visited /terms (1 links)
|
|
159
|
+
Found 3 page(s). Capturing…
|
|
160
|
+
✔ home
|
|
161
|
+
✔ terms
|
|
162
|
+
✔ policy-cookies
|
|
163
|
+
|
|
164
|
+
✔ 3 of 3 page(s) captured
|
|
165
|
+
|
|
166
|
+
⚠ This app has a login (home), and no session was used.
|
|
167
|
+
What you have is the PUBLIC surface only — whatever is behind the login was never reached.
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
**Read that warning before you read the number.** "3 of 3" looks complete and is not.
|
|
171
|
+
|
|
172
|
+
### Step 5.3 — Behind a login
|
|
173
|
+
|
|
174
|
+
```bash
|
|
175
|
+
npx ia-qa-discover login
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
A visible browser opens. You log in **yourself** — SSO, MFA, a consent screen, a magic link, a
|
|
179
|
+
device check: all of them work, because the tool models none of them. Nothing is typed for you
|
|
180
|
+
and no credential is read or stored.
|
|
181
|
+
|
|
182
|
+
What is saved is the browser session, in `.ia-qa-discovery/session.json`. **That file holds live
|
|
183
|
+
cookies: whoever has it is logged in as you.** The tool says so before opening the browser and
|
|
184
|
+
writes a `.gitignore` next to it so a commit cannot carry it away.
|
|
185
|
+
|
|
186
|
+
Already have a session? An authenticated Playwright suite usually writes one in `globalSetup`:
|
|
187
|
+
|
|
188
|
+
```bash
|
|
189
|
+
npx ia-qa-discover scan --session playwright/.auth/user.json
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
Then scan again — this time it goes behind the wall.
|
|
193
|
+
|
|
194
|
+
### Step 5.4 — Most of the inputs are behind a click
|
|
195
|
+
|
|
196
|
+
A loaded page is *one state* of a page. Menus, tabs, accordions and dialogs hide the rest.
|
|
197
|
+
|
|
198
|
+
```bash
|
|
199
|
+
npx ia-qa-discover scan --deep
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
This opens each control one level down and captures what it reveals. Every field found this way
|
|
203
|
+
records **how to reach it** — the click path — because telling you to test a field without
|
|
204
|
+
telling you how to get to it is useless.
|
|
205
|
+
|
|
206
|
+
Two things keep this safe: every non-GET request is blocked for the whole walk, so a click
|
|
207
|
+
cannot change anything on the server, and URLs that act (`/logout`, `/delete`…) are never
|
|
208
|
+
followed.
|
|
209
|
+
|
|
210
|
+
It costs roughly twice the scan time. Even without it, the tool always tells you how many
|
|
211
|
+
controls *could* reveal more — a page with 64 clickable controls and 2 visible fields must not
|
|
212
|
+
report those 2 as its input surface.
|
|
213
|
+
|
|
214
|
+
---
|
|
215
|
+
|
|
216
|
+
## 6. Reading what it wrote
|
|
217
|
+
|
|
218
|
+
```
|
|
219
|
+
.ia-qa-discovery/
|
|
220
|
+
config.json # baseUrl, crawl settings, declared pages, out-of-scope pages
|
|
221
|
+
history.jsonl # one line per scan — the trend no single run can show
|
|
222
|
+
session.json # only if you ran `login`. Live cookies: gitignored automatically
|
|
223
|
+
capture/
|
|
224
|
+
surface.json # the machine-readable index
|
|
225
|
+
_overview.md # ← start here as a human
|
|
226
|
+
home.json # one file per page
|
|
227
|
+
…
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
**Open `_overview.md` first.** It is the same scan, written to be read — by you, or by an LLM
|
|
231
|
+
using it as context. It leads with what was *not* measured, then the pages, their forms and
|
|
232
|
+
fields, and the API calls.
|
|
233
|
+
|
|
234
|
+
Commit `capture/`. It is small, deterministic text, and committing it is what makes the next
|
|
235
|
+
section possible.
|
|
236
|
+
|
|
237
|
+
Three things a real scan found on a real app, each traceable to an observation:
|
|
238
|
+
|
|
239
|
+
- All 9 pages shared one `<title>` — the tab, the history and a screen reader cannot tell them
|
|
240
|
+
apart.
|
|
241
|
+
- `GET /api/integration/status` answered **401** and `/api/billing/payment-history` **404**, on
|
|
242
|
+
a valid session.
|
|
243
|
+
- Every field on the login form had no `id`, no `name`, no `data-testid` — so any test written
|
|
244
|
+
against it can only use a positional selector, which breaks the day a field is inserted above.
|
|
245
|
+
|
|
246
|
+
---
|
|
247
|
+
|
|
248
|
+
## 7. The four ways a page count lies
|
|
249
|
+
|
|
250
|
+
This is the most useful section in this guide. Each of these produced a confident, wrong number
|
|
251
|
+
before it was caught, and each is now stated out loud.
|
|
252
|
+
|
|
253
|
+
**1. The login wall.** Pages still showing a login form were captured — you measured the wall,
|
|
254
|
+
not the app. Worse: an app whose catch-all route renders the login page answers HTTP 200 at
|
|
255
|
+
*every* URL, so the count looks like a success. → run `login`.
|
|
256
|
+
|
|
257
|
+
**2. No `<a href>` to follow.** Discovery follows links. An app navigating with React Router's
|
|
258
|
+
`navigate()` or a button's `onClick` exposes none, so an 11-route app reports "1 of 1 found".
|
|
259
|
+
→ declare the routes you know in `config.json` → `pages`; they are scanned *in addition* to the
|
|
260
|
+
crawl.
|
|
261
|
+
|
|
262
|
+
**3. Inputs outside a `<form>`.** Modern apps wrap nothing. A scan that only walked `<form>`
|
|
263
|
+
tags reported "0 fields" on an app with 464 inputs. → handled automatically now; the fields are
|
|
264
|
+
listed with the heading they sit under.
|
|
265
|
+
|
|
266
|
+
**4. The page cap.** `--max` stops the run, and pages known-but-not-visited are counted and
|
|
267
|
+
named rather than silently dropped. → raise `--max`.
|
|
268
|
+
|
|
269
|
+
**The rule underneath all four:** every counter this tool prints must be able to say what it
|
|
270
|
+
does *not* measure. If you ever see a number here with no caveat beside it, that is a bug worth
|
|
271
|
+
reporting.
|
|
272
|
+
|
|
273
|
+
---
|
|
274
|
+
|
|
275
|
+
## 8. What my suite does not test
|
|
276
|
+
|
|
277
|
+
This is where the two packages meet, and it is the one question neither can answer alone.
|
|
278
|
+
|
|
279
|
+
Discovery knows **what exists**. `@ia-qa/self-healing` knows **where your suite actually went** —
|
|
280
|
+
because it can watch the suite run. Put the two together and you get the sentence that matters:
|
|
281
|
+
|
|
282
|
+
```console
|
|
283
|
+
$ npx ia-qa-discover coverage
|
|
284
|
+
|
|
285
|
+
📊 Your suite visits 4 of the 27 pages this scan found · 23 not tested
|
|
286
|
+
|
|
287
|
+
Not tested — ranked by what the scan observed on each page:
|
|
288
|
+
· login — takes credentials · submits a form · has required input · 2 inputs · 1 API call
|
|
289
|
+
· checkout — submits a form · 1 input · 1 API call
|
|
290
|
+
· profile — has required input · 4 inputs
|
|
291
|
+
|
|
292
|
+
This is not a verdict: a page nobody tests may be deliberately out of scope.
|
|
293
|
+
|
|
294
|
+
📐 Depth — your suite names 12 of 380 contracted elements. Named, not asserted.
|
|
295
|
+
|
|
296
|
+
What this does NOT measure:
|
|
297
|
+
⚠ 2 known page(s) were cut by the page cap (--max) and are not in these totals.
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
### What you need first
|
|
301
|
+
|
|
302
|
+
```bash
|
|
303
|
+
npm i -D @ia-qa/self-healing
|
|
304
|
+
npx ia-qa-heal init # declare your pages and your test command
|
|
305
|
+
npx ia-qa-heal map # capture the app's contracts
|
|
306
|
+
npx ia-qa-heal ingest # inventory what your tests name
|
|
307
|
+
npx ia-qa-heal run # runs your suite with capture on — this is the step that matters
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
That last command is the whole point: it runs your suite and records **which pages it actually
|
|
311
|
+
loaded**. Until that has happened, `coverage` shows you no number at all — it tells you there is
|
|
312
|
+
nothing to measure yet and names the command. A page having a contract only proves somebody
|
|
313
|
+
configured it; only a watched run proves a test went there.
|
|
314
|
+
|
|
315
|
+
### Three words to read carefully
|
|
316
|
+
|
|
317
|
+
**"Not tested" is not a verdict.** A page nobody tests may be deliberately out of scope. You
|
|
318
|
+
decide, and section 9 shows how to write that decision down.
|
|
319
|
+
|
|
320
|
+
**"Named" is not "asserted".** The depth line counts elements your tests *designate*. A test
|
|
321
|
+
that clicks a button names it without checking anything, and nothing on disk can tell the two
|
|
322
|
+
apart — so the tool never says "tested".
|
|
323
|
+
|
|
324
|
+
**There is no percentage, and there never will be.** A `34%` reads as code coverage, which is a
|
|
325
|
+
ratio over a denominator a compiler guarantees. Here the denominator is uncertain by
|
|
326
|
+
construction — the login wall, the links nobody could follow, the cap, the states nothing
|
|
327
|
+
opened. A percentage over that would be a comforting number with its caveats stripped off, which
|
|
328
|
+
is precisely how a team stops testing something. You get counts and page names.
|
|
329
|
+
|
|
330
|
+
`--report` writes the HTML version, `--json` the machine-readable map, and
|
|
331
|
+
`.ia-qa-discovery/coverage-map.json` is written every time — commit it and watch the gap move.
|
|
332
|
+
|
|
333
|
+
---
|
|
334
|
+
|
|
335
|
+
## 9. Doing this every week
|
|
336
|
+
|
|
337
|
+
A discovery is a perishable snapshot. The value is in re-running it.
|
|
338
|
+
|
|
339
|
+
```bash
|
|
340
|
+
npx ia-qa-discover scan # same command; an unchanged app must diff clean
|
|
341
|
+
npx ia-qa-discover history
|
|
342
|
+
```
|
|
343
|
+
|
|
344
|
+
```console
|
|
345
|
+
2026-09-04T12:19:39Z 9/9 pages +9 -1 ~0 300df55 (main)
|
|
346
|
+
2026-09-04T12:35:06Z 9/9 pages +0 -0 ~0 (no change) 300df55 (main)
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
### Closing a line you have decided about
|
|
350
|
+
|
|
351
|
+
A gap list that never shrinks stops being read. By the third week, 23 pages reported identically
|
|
352
|
+
every time is noise, and the two that started mattering are lost in it.
|
|
353
|
+
|
|
354
|
+
So write the decision down, in `.ia-qa-discovery/config.json`:
|
|
355
|
+
|
|
356
|
+
```json
|
|
357
|
+
{
|
|
358
|
+
"baseUrl": "https://app.example.com",
|
|
359
|
+
"outOfScope": ["legal-*", "styleguide"]
|
|
360
|
+
}
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
Those pages stay **counted and named** as out of scope. They are never silently dropped, and
|
|
364
|
+
nothing is ever added to that list on its own — a page leaves the gap because a person wrote it
|
|
365
|
+
down in a file a reviewer can see.
|
|
366
|
+
|
|
367
|
+
---
|
|
368
|
+
|
|
369
|
+
## 10. Optional: what is each entry point *for*?
|
|
370
|
+
|
|
371
|
+
Everything above is deterministic and local. This section is neither, and it is a separate
|
|
372
|
+
binary so that it cannot be run by accident.
|
|
373
|
+
|
|
374
|
+
```bash
|
|
375
|
+
npx ia-qa-discover-ai classify
|
|
376
|
+
```
|
|
377
|
+
|
|
378
|
+
It reads the capture you already have — **no browser, your app is never touched** — and asks a
|
|
379
|
+
model, with **your** API key, what each form and field is for and how sensitive it is.
|
|
380
|
+
|
|
381
|
+
**It is the only thing in this package that sends anything anywhere, and it says so before it
|
|
382
|
+
does**, listing exactly what goes: the page URL, title, description, up to 25 headings, the
|
|
383
|
+
observed API calls, and each field's name, type, label and selector. Read that notice before
|
|
384
|
+
running it on a private or authenticated app.
|
|
385
|
+
|
|
386
|
+
Two guarantees make the output usable:
|
|
387
|
+
|
|
388
|
+
- **Every claim cites the capture, and the citation is checked** before the line is kept. A
|
|
389
|
+
claim pointing at something that does not exist is dropped, and the drop is reported.
|
|
390
|
+
- **The model can refuse.** "Not enough evidence" is a real answer, listed separately — an LLM
|
|
391
|
+
asked to classify will otherwise always classify, and that is how confident nonsense ships.
|
|
392
|
+
|
|
393
|
+
The check proves the *premise*, never the *conclusion*: a model can cite a real password field
|
|
394
|
+
and still call the page a checkout. So the sentence and the confidence are always shown, never
|
|
395
|
+
the label alone.
|
|
396
|
+
|
|
397
|
+
Without a key, nothing changes: every command above behaves exactly as it does today.
|
|
398
|
+
|
|
399
|
+
---
|
|
400
|
+
|
|
401
|
+
## 11. Troubleshooting
|
|
402
|
+
|
|
403
|
+
**"Found 1 page" on an app I know has twenty.** Section 7, case 2 — the app navigates without
|
|
404
|
+
links. Declare the routes in `config.json` → `pages`.
|
|
405
|
+
|
|
406
|
+
**Everything is the login page.** Section 7, case 1. Run `login`, then scan again.
|
|
407
|
+
|
|
408
|
+
**`login` says it refuses.** It refuses without a terminal and under CI, because it waits for a
|
|
409
|
+
person twice. That is also why an AI agent cannot run it. For a pipeline, point `--session` at a
|
|
410
|
+
`storageState` your own setup writes.
|
|
411
|
+
|
|
412
|
+
**"Nothing to show yet" from `coverage`.** Your suite has never been watched running. Run
|
|
413
|
+
`npx ia-qa-heal run`. This is not zero coverage — it is no measurement, which is why no number
|
|
414
|
+
is shown.
|
|
415
|
+
|
|
416
|
+
**"No test suite is being tracked here."** `coverage` needs `@ia-qa/self-healing`. Without it,
|
|
417
|
+
scanning and classification work exactly as before; reach and depth are not computed at all.
|
|
418
|
+
|
|
419
|
+
**The scan is slow.** Lower `--max`, lower `--depth`, or drop `--deep`. A deep scan costs about
|
|
420
|
+
twice a normal one.
|
|
421
|
+
|
|
422
|
+
**A field has no name in the report.** The app labels it nowhere — no `<label>`, no
|
|
423
|
+
`aria-label`, no placeholder. That is a finding about the app, not about the scan: a screen
|
|
424
|
+
reader cannot announce it either.
|
|
425
|
+
|
|
426
|
+
---
|
|
427
|
+
|
|
428
|
+
## Where to go next
|
|
429
|
+
|
|
430
|
+
- [README.md](./README.md) — the short reference, every flag.
|
|
431
|
+
- [ROADMAP.md](./ROADMAP.md) — where this is going, and what it will deliberately never become.
|
|
432
|
+
- [`@ia-qa/self-healing`](https://www.npmjs.com/package/@ia-qa/self-healing) — once tests exist,
|
|
433
|
+
the tool that keeps them alive when selectors move.
|
package/dist/cli/args.js
CHANGED
package/dist/cli/args.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"args.js","sourceRoot":"","sources":["../../src/cli/args.ts"],"names":[],"mappings":";;;
|
|
1
|
+
{"version":3,"file":"args.js","sourceRoot":"","sources":["../../src/cli/args.ts"],"names":[],"mappings":";;;AAuBA,gDAQC;AA/BD,sGAAsG;AACzF,QAAA,WAAW,GAAsC;IAC5D,IAAI,EAAE;QACJ,SAAS;QACT,OAAO;QACP,aAAa;QACb,eAAe;QACf,WAAW;QACX,cAAc;QACd,cAAc;QACd,QAAQ;QACR,eAAe;QACf,qBAAqB;QACrB,QAAQ;QACR,QAAQ;QACR,UAAU;QACV,QAAQ;KACT;IACD,KAAK,EAAE,CAAC,OAAO,EAAE,WAAW,CAAC;IAC7B,OAAO,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC;IAC9B,QAAQ,EAAE,CAAC,QAAQ,EAAE,UAAU,EAAE,QAAQ,CAAC;CAC3C,CAAC;AAEF,SAAgB,kBAAkB,CAAC,OAAe,EAAE,IAAc;IAChE,MAAM,KAAK,GAAG,mBAAW,CAAC,OAAO,CAAC,CAAC;IACnC,IAAI,KAAK,KAAK,SAAS;QAAE,OAAO;IAChC,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;QACrB,IAAI,CAAC,CAAC,CAAC,UAAU,CAAC,GAAG,CAAC;YAAE,SAAS;QACjC,IAAI,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC;YAAE,SAAS;QAChC,MAAM,IAAI,KAAK,CAAC,iBAAiB,CAAC,yBAAyB,OAAO,kDAAkD,CAAC,CAAC;IACxH,CAAC;AACH,CAAC"}
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* `ia-qa-discover coverage` — what the suite tests, what the app has, and the gap.
|
|
3
|
+
*
|
|
4
|
+
* **A verb, not a flag on `scan`.** It reads local files only and opens no browser; making
|
|
5
|
+
* it a flag would force a full re-crawl of the application to read two files already on
|
|
6
|
+
* disk. It is also opt-in for a reason that matters more: `scan` must stay silent about the
|
|
7
|
+
* sibling package, because a cross-reference that fires on the ordinary run is advertising,
|
|
8
|
+
* and this package treats a message that fires on the good case as a defect. Here the user
|
|
9
|
+
* asked, so the premise of their own command is what is missing — that is an answer.
|
|
10
|
+
*
|
|
11
|
+
* **Plug-and-play has a precise test here**: in a project that already has `.ia-qa/` and
|
|
12
|
+
* `.ia-qa-discovery/`, the bare verb produces the map with no argument and no question.
|
|
13
|
+
*
|
|
14
|
+
* Exit code is 0 on a refusal as well as on a gap. A missing optional prerequisite is not a
|
|
15
|
+
* failure of anything, and a coverage gap is a decision for a human — gating on it would
|
|
16
|
+
* make this the verdict discovery has spent its whole design refusing to emit.
|
|
17
|
+
*/
|
|
18
|
+
export declare const COVERAGE_FILENAME = "coverage-map.json";
|
|
19
|
+
export declare function coverageMapPath(cwd?: string): string;
|
|
20
|
+
export declare function runCoverage(args: string[]): Promise<void>;
|