@ia-qa/qa-discovery 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +82 -2
- package/ROADMAP.md +26 -4
- package/TUTORIAL.md +85 -2
- package/dist/ai/classify.d.ts +13 -36
- package/dist/ai/classify.js +23 -166
- package/dist/ai/classify.js.map +1 -1
- package/dist/ai/model.d.ts +93 -0
- package/dist/ai/model.js +192 -0
- package/dist/ai/model.js.map +1 -0
- package/dist/ai/plan.d.ts +224 -0
- package/dist/ai/plan.js +0 -0
- package/dist/ai/plan.js.map +1 -0
- package/dist/cli/args.js +1 -0
- package/dist/cli/args.js.map +1 -1
- package/dist/cli/index.d.ts +1 -1
- package/dist/cli/index.js +13 -0
- package/dist/cli/index.js.map +1 -1
- package/dist/cli/skill.d.ts +20 -0
- package/dist/cli/skill.js +63 -0
- package/dist/cli/skill.js.map +1 -0
- package/dist/cli-ai/index.d.ts +3 -2
- package/dist/cli-ai/index.js +268 -4
- package/dist/cli-ai/index.js.map +1 -1
- package/dist/htmlReport.js +20 -4
- package/dist/htmlReport.js.map +1 -1
- package/dist/index.d.ts +6 -0
- package/dist/index.js +32 -1
- package/dist/index.js.map +1 -1
- package/dist/mcp/server.d.ts +49 -0
- package/dist/mcp/server.js +143 -20
- package/dist/mcp/server.js.map +1 -1
- package/dist/overview.d.ts +9 -0
- package/dist/overview.js +9 -4
- package/dist/overview.js.map +1 -1
- package/dist/overviewFile.js +5 -0
- package/dist/overviewFile.js.map +1 -1
- package/dist/planView.d.ts +141 -0
- package/dist/planView.js +345 -0
- package/dist/planView.js.map +1 -0
- package/package.json +3 -2
- package/skills/ia-qa-discover/SKILL.md +166 -0
package/README.md
CHANGED
|
@@ -27,13 +27,23 @@ npx -p @ia-qa/qa-discovery ia-qa-discover scan https://your-app.example.com --sa
|
|
|
27
27
|
| **what does my test suite not test?** | **`coverage`** — the gap, ranked; needs `@ia-qa/self-healing` and one watched run |
|
|
28
28
|
| what changed since last time? | `history` |
|
|
29
29
|
| what is each entry point *for*, and how sensitive? | `ia-qa-discover-ai classify` (optional, your own key) |
|
|
30
|
+
| what does somebody come to this page to **do**? | `ia-qa-discover-ai plan` (optional, your own key) |
|
|
31
|
+
|
|
32
|
+
**Driving this with an AI agent?** One command gives it the whole doctrine — the verb for
|
|
33
|
+
each question, what every refusal means, and what it must never do:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
ia-qa-discover skill --print # read it
|
|
37
|
+
ia-qa-discover skill --install # drop it in .claude/skills/, where agents actually look
|
|
38
|
+
```
|
|
30
39
|
|
|
31
40
|
`ia-qa-discover <verb> --help` for flags. Nothing here emits a verdict and nothing here gates
|
|
32
41
|
CI: a page count is not coverage, and a gap is a decision for a human.
|
|
33
42
|
|
|
34
43
|
**Same operations for an AI agent** — one contract served twice, never a simplified view for
|
|
35
|
-
one of them. `ia-qa-discover-mcp` exposes `scan_app`, `coverage_map`, `classify_app
|
|
36
|
-
`discovery_history`. `login` is deliberately not a tool: it waits for a
|
|
44
|
+
one of them. `ia-qa-discover-mcp` exposes `scan_app`, `coverage_map`, `classify_app`,
|
|
45
|
+
`plan_pages` and `discovery_history`. `login` is deliberately not a tool: it waits for a
|
|
46
|
+
person at a browser.
|
|
37
47
|
|
|
38
48
|
> **New here?** The [step-by-step tutorial](https://www.ia-qa.com/devtools/qa-discovery/tutorial)
|
|
39
49
|
> walks the whole thing from an app you have never seen, with no jargon — first scan, login
|
|
@@ -414,6 +424,7 @@ A dependency-free JSON-RPC 2.0 stdio server exposing the **same** operations the
|
|
|
414
424
|
|
|
415
425
|
- `scan_app` — crawl and capture. Check `loginWall` and `hrefBlind` in the result before reporting coverage.
|
|
416
426
|
- `classify_app` — the optional BYOK layer below, for an agent. The key is read from an environment variable **named** in the call (`api_key_env`); a raw key is never a tool argument.
|
|
427
|
+
- `plan_pages` — the other BYOK question: what somebody comes to each page **to do**. Same key handling, and it sends **less** — no selectors, no observed API calls. Defaults to the pages `coverage_map` says no test visits, in that tool's order. Ranks nothing, gates nothing; a `failed` page is a failed call, never a finding about the app.
|
|
417
428
|
- `coverage_map` — what the suite does **not** test. Returns `measured: false` with a typed `reason` when no run was ever watched: report that as *no measurement*, never as "the suite covers nothing".
|
|
418
429
|
- `discovery_history` — the trend.
|
|
419
430
|
|
|
@@ -428,8 +439,15 @@ Everything above is deterministic and never leaves your machine. This one binary
|
|
|
428
439
|
```bash
|
|
429
440
|
ia-qa-discover-ai classify --dry-run # see exactly what would be sent, and to whom
|
|
430
441
|
ia-qa-discover-ai classify
|
|
442
|
+
|
|
443
|
+
ia-qa-discover-ai plan --dry-run # the other question, and a smaller payload
|
|
444
|
+
ia-qa-discover-ai plan --report --open
|
|
431
445
|
```
|
|
432
446
|
|
|
447
|
+
Two verbs, two questions, and **`plan` sends strictly less than `classify`**: no selectors and
|
|
448
|
+
no observed API calls ever leave for it. Each announces its own payload, derived from the
|
|
449
|
+
function that builds it, so neither can promise less than it sends.
|
|
450
|
+
|
|
433
451
|
It reads the capture **offline** — it opens no browser and never touches your app again — and asks a model what each entry point is *for* (`authentication`, `payment`, `search`, `data-entry`…) and how sensitive what it handles is. The vocabulary is closed and versioned: a label outside it is rejected before anything else is checked.
|
|
434
452
|
|
|
435
453
|
**Every claim cites the capture, and every citation is resolved and checked.** A classification whose evidence does not exist, or does not say what it was claimed to say, is dropped before you see it, and the drop is reported. That proves the *premise*, never the conclusion — a model can cite a real password field and still be wrong about what the page is for — so read the `why` and the confidence, not the label alone.
|
|
@@ -441,6 +459,12 @@ Configure it in `.ia-qa-discovery/config.json` — the file holds the *name* of
|
|
|
441
459
|
```json
|
|
442
460
|
"ai": { "provider": "anthropic", "model": "claude-haiku-4-5",
|
|
443
461
|
"apiKey": { "source": "env", "key": "ANTHROPIC_API_KEY" } }
|
|
462
|
+
|
|
463
|
+
// or any OpenAI-compatible endpoint — DeepSeek, Groq, Mistral, OpenRouter, vLLM,
|
|
464
|
+
// or a model on your own machine, in which case nothing leaves it at all:
|
|
465
|
+
"ai": { "provider": "openai-compatible", "model": "deepseek-chat",
|
|
466
|
+
"baseUrl": "https://api.deepseek.com/v1",
|
|
467
|
+
"apiKey": { "source": "keychain", "key": "DEEPSEEK_KEY" } }
|
|
444
468
|
```
|
|
445
469
|
|
|
446
470
|
Before the first request it prints what leaves and where it goes:
|
|
@@ -467,6 +491,62 @@ rewrite — it is your file — so `scan` warns instead.
|
|
|
467
491
|
|
|
468
492
|
Read that list before pointing it at an authenticated or private app. The result lands in `.ia-qa-discovery/classification.json` **and** in `_overview.md`, which is rewritten so the reading appears next to the capture it was made from.
|
|
469
493
|
|
|
494
|
+
### `plan` — what somebody comes to this page to do
|
|
495
|
+
|
|
496
|
+
`classify` answers what an entry point *is*, against a closed vocabulary. `plan` answers a
|
|
497
|
+
different question, on a different axis: **what is a person trying to do here, and what can
|
|
498
|
+
they no longer do if it breaks** — in the words of whoever uses the app, not a developer's.
|
|
499
|
+
|
|
500
|
+
It exists because that axis is the one nothing else in this ecosystem can reach. The
|
|
501
|
+
deterministic half reports what an app *accepts* — fields, types, required, what moved.
|
|
502
|
+
Nothing in a DOM states that this form is how a locked-out customer gets back in. Neither
|
|
503
|
+
layer is a subset of the other, which is why one never ranks above the other.
|
|
504
|
+
|
|
505
|
+
```bash
|
|
506
|
+
ia-qa-discover-ai plan # the pages `coverage` says no test visits, in its order
|
|
507
|
+
ia-qa-discover-ai plan --all # every readable page, tested or not
|
|
508
|
+
ia-qa-discover-ai plan --report --open # the same branded dossier `scan --report` writes
|
|
509
|
+
```
|
|
510
|
+
|
|
511
|
+
By default it reads exactly the untested pages from `ia-qa-discover coverage`, **in the order
|
|
512
|
+
that verb ranked them** — so an interrupted run has read the ones that mattered most. With no
|
|
513
|
+
suite tracked here there is no untested list to narrow to, and it reads every page and says
|
|
514
|
+
so rather than refusing: a project with no tests is the one this package exists for.
|
|
515
|
+
|
|
516
|
+
Four properties, and each is tested:
|
|
517
|
+
|
|
518
|
+
- **It sends less than `classify`** — no selectors, no observed API calls. The most sensitive
|
|
519
|
+
half of a capture, and technical noise for a question that is not technical.
|
|
520
|
+
- **Every reading cites the capture** and is dropped if the citation does not resolve, exactly
|
|
521
|
+
like a classification. A path the prompt never offered is refused *before* resolution, so
|
|
522
|
+
citing an API call — which this verb does not send — cannot happen.
|
|
523
|
+
- **The shell is not a page's content.** Headings repeated across the app (a footer's `Legal`,
|
|
524
|
+
`Contact`, `Tools`) are dropped from the payload, using the same threshold `autoLayout`
|
|
525
|
+
uses. Measured before the change: a page whose whole interest was `🗂️ Environment Manager`
|
|
526
|
+
had its reading propped up by three footer links — citations that resolve, and support
|
|
527
|
+
nothing. A page whose *every* heading is shared keeps them all: the exclusion narrows the
|
|
528
|
+
evidence, it never makes a page unreadable.
|
|
529
|
+
- **At least one citation must carry the subject** — a level-1 or level-2 heading, a form, or
|
|
530
|
+
a field. A reading standing entirely on nav labels is dropped. The rule is skipped on a page
|
|
531
|
+
that offers no such anchor, because a rule nobody can satisfy would punish a page for its own
|
|
532
|
+
markup.
|
|
533
|
+
- **It ranks nothing and gates nothing.** The order is `coverage`'s measurement; this adds a
|
|
534
|
+
labelled line beside it, never a row to it.
|
|
535
|
+
- **The origin travels with the artifact.** `plan.json` carries the model and the timestamp,
|
|
536
|
+
one module renders the terminal, `_overview.md` and the HTML report, a page re-captured
|
|
537
|
+
after it was read is marked **stale**, and the report's footer stops saying "no LLM".
|
|
538
|
+
|
|
539
|
+
A model that cannot tell answers `unclear`: kept, labelled, never silently dropped — the
|
|
540
|
+
capture may genuinely not say, and a proposal nobody can anchor is still worth reading as
|
|
541
|
+
long as it is marked as one.
|
|
542
|
+
|
|
543
|
+
**Its honest limit**, measured on a real run of 8 pages: the reading is worth most where the
|
|
544
|
+
deterministic half is blind — a page with no form at all, whose headings name the stake — and
|
|
545
|
+
thins out on content pages, where "somebody comes here to read the articles" repeats the
|
|
546
|
+
capture back. Read the confidence and the citations, not the sentence alone. After the two
|
|
547
|
+
rules above, that same run produced **0 citations pointing at the footer or the nav**, against
|
|
548
|
+
three on a single page before them.
|
|
549
|
+
|
|
470
550
|
---
|
|
471
551
|
|
|
472
552
|
## Library
|
package/ROADMAP.md
CHANGED
|
@@ -84,7 +84,7 @@ be understood.
|
|
|
84
84
|
|
|
85
85
|
| Stage | The question it answers | Status |
|
|
86
86
|
|---|---|---|
|
|
87
|
-
| **Value and personas** (F0) | Who uses this app, for what, and where would it hurt most if it broke? |
|
|
87
|
+
| **Value and personas** (F0) | Who uses this app, for what, and where would it hurt most if it broke? | partly shipped — see `plan` below |
|
|
88
88
|
| **Surface reconnaissance** (F1) | What pages exist, what do they take as input, what do they call? | ✅ shipped (0.1.0) |
|
|
89
89
|
| **Component classification** (F2) | What is each entry point *for*, and how sensitive is what it handles? | in progress |
|
|
90
90
|
| **Zones and synthesis** (F2.5) | How do the pages group into functional areas, and what is the app as a whole? | not built |
|
|
@@ -218,9 +218,31 @@ navigation graph), then a grounded synthesis on top. Reconciles with the value s
|
|
|
218
218
|
|
|
219
219
|
## Value and personas (F0) — *the game changer, built with care*
|
|
220
220
|
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
221
|
+
**A first, deliberately narrow slice shipped: `ia-qa-discover-ai plan`.** It answers the
|
|
222
|
+
value question for **one page at a time** — what somebody comes here to do, and what they can
|
|
223
|
+
no longer do if it breaks — and nothing beyond that. No personas, no app-level framing, no
|
|
224
|
+
"moment of truth": those are the claims the validation protocol below exists to gate, and
|
|
225
|
+
none of them is made.
|
|
226
|
+
|
|
227
|
+
Why this slice was safe to ship first, in the terms this file uses everywhere else:
|
|
228
|
+
|
|
229
|
+
- It **produces no executable artifact**, so a wrong reading cannot become a broken test.
|
|
230
|
+
- Every reading **cites the capture** and is dropped when the citation does not resolve —
|
|
231
|
+
the same mechanism as classification, on a per-page claim small enough for that check to
|
|
232
|
+
mean something.
|
|
233
|
+
- It **ranks nothing**: the order is `coverage`'s measurement, and the model never adds a row
|
|
234
|
+
to it. That is the difference between commenting on a measurement and replacing one.
|
|
235
|
+
- It sends **less** than `classify` — no selectors, no observed API calls.
|
|
236
|
+
|
|
237
|
+
Measured limit, from a real 8-page run: the reading is worth most on a page the deterministic
|
|
238
|
+
half is blind to (no fields at all, the stake living in the headings) and is near-tautological
|
|
239
|
+
on content pages. That asymmetry is itself an argument for keeping this stage per-page until
|
|
240
|
+
the protocol below exists.
|
|
241
|
+
|
|
242
|
+
What remains unbuilt is the hard half — who the users are, and where the moment of truth is.
|
|
243
|
+
This is the stage that makes the output a senior's plan instead of a classified inventory, and
|
|
244
|
+
the one whose claims are hardest to check — so it is scheduled as **careful** work, not as
|
|
245
|
+
later work.
|
|
224
246
|
|
|
225
247
|
**It needs a validation protocol before it needs code.** Classification can be checked in seconds
|
|
226
248
|
against the capture; this cannot. Label a handful of real apps by hand, compare the stage's
|
package/TUTORIAL.md
CHANGED
|
@@ -30,7 +30,8 @@ suite, a ranked list of the pages nobody tests.
|
|
|
30
30
|
8. [What my suite does not test](#8-what-my-suite-does-not-test)
|
|
31
31
|
9. [Doing this every week](#9-doing-this-every-week)
|
|
32
32
|
10. [Optional: what is each entry point *for*?](#10-optional-what-is-each-entry-point-for)
|
|
33
|
-
11. [
|
|
33
|
+
11. [Optional: what does somebody come here to *do*?](#11-optional-what-does-somebody-come-here-to-do)
|
|
34
|
+
12. [Troubleshooting](#12-troubleshooting)
|
|
34
35
|
|
|
35
36
|
---
|
|
36
37
|
|
|
@@ -90,6 +91,20 @@ first run downloads Chromium once (~130 MB) into a shared cache and never again.
|
|
|
90
91
|
If you drive Claude Code, Cursor, Windsurf, VS Code + Copilot or any MCP-capable agent, it can
|
|
91
92
|
run all of this for you.
|
|
92
93
|
|
|
94
|
+
### Step 4.0 — Give it the instructions first
|
|
95
|
+
|
|
96
|
+
Before the server, one command. It costs nothing and removes most of the back-and-forth:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
npx -p @ia-qa/qa-discovery ia-qa-discover skill --install
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
That writes `.claude/skills/ia-qa-discover/SKILL.md` — which verb answers which question, what
|
|
103
|
+
each refusal means (`no-surface`, `no-healing`, `no-run` are three different sentences), what
|
|
104
|
+
it must never do, and how to set an API key up without ever holding one. Agents read
|
|
105
|
+
`.claude/skills/`, never `node_modules`, so the file has to land there to exist at all.
|
|
106
|
+
Use `--print` to read it yourself first, or `--user` to install it for every project.
|
|
107
|
+
|
|
93
108
|
### Step 4.1 — Add the server to your agent
|
|
94
109
|
|
|
95
110
|
Claude Code:
|
|
@@ -420,7 +435,75 @@ Without a key, nothing changes: every command above behaves exactly as it does t
|
|
|
420
435
|
|
|
421
436
|
---
|
|
422
437
|
|
|
423
|
-
## 11.
|
|
438
|
+
## 11. Optional: what does somebody come here to *do*?
|
|
439
|
+
|
|
440
|
+
Section 10 asks what an entry point **is**. This asks something a DOM cannot answer at all:
|
|
441
|
+
**what is a person trying to do on this page, and what can they no longer do if it breaks.**
|
|
442
|
+
|
|
443
|
+
```bash
|
|
444
|
+
npx ia-qa-discover-ai plan
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
With no flags it reads exactly the pages section 8 said no test visits, **in the order that
|
|
448
|
+
verb ranked them**. If you stop it halfway, the pages it already read are the ones that
|
|
449
|
+
mattered most. No suite tracked yet? It reads every page and tells you that is what it did —
|
|
450
|
+
you are the reader this package was written for.
|
|
451
|
+
|
|
452
|
+
What you get, per page:
|
|
453
|
+
|
|
454
|
+
```
|
|
455
|
+
· devtools-environments
|
|
456
|
+
A person comes here to switch API keys and model configurations for different
|
|
457
|
+
environments like development, production and staging. · If it breaks: that person
|
|
458
|
+
cannot change or manage the API keys and model configurations. (confidence 0.90)
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
That page has **zero form fields**. The deterministic half could only say `nothing observed
|
|
462
|
+
says it matters (2 API calls)` — it had nothing to observe. This is the gap the layer exists
|
|
463
|
+
for, and the honest flip side is that on a page of articles the same model writes "somebody
|
|
464
|
+
comes here to read the articles", which repeats the capture back at you. **Read the
|
|
465
|
+
confidence and the citations, not the sentence.**
|
|
466
|
+
|
|
467
|
+
### It sends less than `classify`, deliberately
|
|
468
|
+
|
|
469
|
+
No selectors and no observed API calls leave for this question — the most sensitive half of a
|
|
470
|
+
capture, and technical noise for a question about people. The notice it prints before the
|
|
471
|
+
first request is derived from the function that builds the payload, so it cannot promise less
|
|
472
|
+
than it sends.
|
|
473
|
+
|
|
474
|
+
**Your footer does not leave either.** Headings that appear across your whole app — `Legal`,
|
|
475
|
+
`Contact`, a nav title — are the shell, not this page's content, and they are dropped from
|
|
476
|
+
what goes. That started as a privacy-shaped decision and turned out to be a quality one: on a
|
|
477
|
+
real run, the reading of a page whose whole interest was `🗂️ Environment Manager` had been
|
|
478
|
+
propped up by three footer links. They resolve, so nothing rejected them, and they support
|
|
479
|
+
nothing. On the same run after the change: **zero citations pointing at a footer or a nav.**
|
|
480
|
+
|
|
481
|
+
And a reading has to stand on something: at least one citation must be a level-1 or level-2
|
|
482
|
+
heading, a form, or a field. One built entirely out of nav labels is dropped — unless the page
|
|
483
|
+
offers no such anchor at all, in which case the rule is skipped rather than punishing the page
|
|
484
|
+
for its own markup.
|
|
485
|
+
|
|
486
|
+
### Three things it will not do
|
|
487
|
+
|
|
488
|
+
- **It will not rank anything.** The order is the one `coverage` measured. A model reordering
|
|
489
|
+
a measured list has stopped commenting on a measurement and started replacing it.
|
|
490
|
+
- **It will not gate.** Nothing here has an exit code that fails a build.
|
|
491
|
+
- **It will not hide its origin.** `plan.json` carries the model and the time it ran;
|
|
492
|
+
`_overview.md` and the HTML report show the section labelled `model-read`; a page
|
|
493
|
+
re-captured *after* it was read is marked **stale**; and the report's footer stops claiming
|
|
494
|
+
the document is LLM-free, because it no longer is.
|
|
495
|
+
|
|
496
|
+
If the capture genuinely does not say, the model answers `unclear` — kept and labelled, never
|
|
497
|
+
quietly dropped.
|
|
498
|
+
|
|
499
|
+
```bash
|
|
500
|
+
npx ia-qa-discover-ai plan --all # the tested pages too
|
|
501
|
+
npx ia-qa-discover-ai plan --report --open # the branded dossier, in a browser
|
|
502
|
+
```
|
|
503
|
+
|
|
504
|
+
---
|
|
505
|
+
|
|
506
|
+
## 12. Troubleshooting
|
|
424
507
|
|
|
425
508
|
**"Found 1 page" on an app I know has twenty.** Section 7, case 2 — the app navigates without
|
|
426
509
|
links. Declare the routes in `config.json` → `pages`.
|
package/dist/ai/classify.d.ts
CHANGED
|
@@ -2,6 +2,7 @@ import type { AiProvider } from '@ia-qa/self-healing';
|
|
|
2
2
|
import type { PageCapture } from '../capture/page';
|
|
3
3
|
import { type ToolTrigger } from '../taxonomy';
|
|
4
4
|
import { type CaptureIndex, type Citation, type CitationVerdict } from '../citations';
|
|
5
|
+
import { providerError, type ModelOptions } from './model';
|
|
5
6
|
/**
|
|
6
7
|
* F2 — classify each entry point, offline, from F1's capture.
|
|
7
8
|
*
|
|
@@ -12,22 +13,18 @@ import { type CaptureIndex, type Citation, type CitationVerdict } from '../citat
|
|
|
12
13
|
* **On duplication, stated because this project punishes silent drift:** the
|
|
13
14
|
* provider/model catalogue is imported from `@ia-qa/self-healing` rather than
|
|
14
15
|
* copied — there must not be a third model list in this repo. The HTTP transport
|
|
15
|
-
*
|
|
16
|
-
*
|
|
17
|
-
*
|
|
16
|
+
* now lives in `./model.ts`, which is a second implementation of something
|
|
17
|
+
* self-healing keeps internal (`callModel`); it moved out of this file when a
|
|
18
|
+
* second BYOK verb arrived, so the cost stays at two copies instead of three.
|
|
19
|
+
* Any change to a provider's request shape owes an edit there as well as in
|
|
20
|
+
* self-healing.
|
|
18
21
|
*/
|
|
19
22
|
/** The document writes, and the two strings a later stage keys on. */
|
|
20
23
|
export declare const CLASSIFICATION_SCHEMA = "qa-discovery-classification@1";
|
|
21
24
|
export declare const CLASSIFICATION_FILENAME = "classification.json";
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
apiKey: string;
|
|
26
|
-
/** Injected in tests; defaults to the global fetch. */
|
|
27
|
-
fetchFn?: typeof fetch;
|
|
28
|
-
timeoutMs?: number;
|
|
29
|
-
temperature?: number;
|
|
30
|
-
}
|
|
25
|
+
/** The transport's options, named for this verb. One shape for every BYOK caller. */
|
|
26
|
+
export type ClassifyOptions = ModelOptions;
|
|
27
|
+
export { providerError };
|
|
31
28
|
/** One thing on a page that takes input, addressed by a path a citation can resolve. */
|
|
32
29
|
export interface EntryPoint {
|
|
33
30
|
/** e.g. `pages/checkout.json#forms[0]` or `pages/checkout.json#looseFields[2]` */
|
|
@@ -125,9 +122,9 @@ export declare function collect(raws: Partial<Classification>[], index: CaptureI
|
|
|
125
122
|
* What leaves this machine, said before it leaves — the one thing that makes
|
|
126
123
|
* "everything is local except the model call" a promise rather than a claim.
|
|
127
124
|
*
|
|
128
|
-
* It
|
|
129
|
-
* `buildRequest` sends to, so a new provider cannot be reachable
|
|
130
|
-
* notice still lists three. And it is derived from `buildPrompt` — every line
|
|
125
|
+
* It resolves the host through `egressHost` rather than naming one of its own: the notice names the
|
|
126
|
+
* very constant `buildRequest` sends to, so a new provider cannot be reachable
|
|
127
|
+
* while the notice still lists three. And it is derived from `buildPrompt` — every line
|
|
131
128
|
* below is a field that function actually puts in the payload; adding one there
|
|
132
129
|
* owes a line here.
|
|
133
130
|
*
|
|
@@ -146,27 +143,7 @@ export declare function collect(raws: Partial<Classification>[], index: CaptureI
|
|
|
146
143
|
* the token here. The word "full" is gone from that line for the same reason the
|
|
147
144
|
* allowlist is named: the two statements must not contradict each other.
|
|
148
145
|
*/
|
|
149
|
-
export declare function egressNotice(provider: AiProvider, model: string, pages: number, planned?: boolean): string;
|
|
150
|
-
/**
|
|
151
|
-
* What the provider itself said went wrong.
|
|
152
|
-
*
|
|
153
|
-
* Found by execution, not review: a Gemini key that is simply wrong comes back
|
|
154
|
-
* **400**, not 401 — so the previous message read `refused the call with HTTP
|
|
155
|
-
* 400`, said nothing about the key, and left the one person who could fix it in
|
|
156
|
-
* sixty seconds with nothing to go on. Mapping more status codes by hand is the
|
|
157
|
-
* wrong shape of fix (the list rots, and it is different per provider); the
|
|
158
|
-
* providers already answer the question in the body, and all three happen to
|
|
159
|
-
* use the same `{error:{message}}` envelope.
|
|
160
|
-
*
|
|
161
|
-
* Two guards. The body is untrusted text that lands in a terminal and in
|
|
162
|
-
* `classification.json`, so it is capped and flattened. And anything shaped like
|
|
163
|
-
* a credential is redacted before it is printed — a provider that echoes the
|
|
164
|
-
* request back would otherwise put the user's own key in a file they commit.
|
|
165
|
-
*/
|
|
166
|
-
export declare function providerError(res: {
|
|
167
|
-
status: number;
|
|
168
|
-
text: () => Promise<string>;
|
|
169
|
-
}): Promise<string>;
|
|
146
|
+
export declare function egressNotice(provider: AiProvider, model: string, pages: number, planned?: boolean, baseUrl?: string): string;
|
|
170
147
|
/**
|
|
171
148
|
* One page in, validated classifications out. Returns an empty result rather
|
|
172
149
|
* than throwing on a network failure, an unparseable reply or an empty page:
|
package/dist/ai/classify.js
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
-
exports.CLASSIFICATION_FILENAME = exports.CLASSIFICATION_SCHEMA = void 0;
|
|
3
|
+
exports.providerError = exports.CLASSIFICATION_FILENAME = exports.CLASSIFICATION_SCHEMA = void 0;
|
|
4
4
|
exports.entryPointsOf = entryPointsOf;
|
|
5
5
|
exports.offeredPaths = offeredPaths;
|
|
6
6
|
exports.buildPrompt = buildPrompt;
|
|
@@ -8,10 +8,11 @@ exports.parseClassifications = parseClassifications;
|
|
|
8
8
|
exports.validate = validate;
|
|
9
9
|
exports.collect = collect;
|
|
10
10
|
exports.egressNotice = egressNotice;
|
|
11
|
-
exports.providerError = providerError;
|
|
12
11
|
exports.classifyPage = classifyPage;
|
|
13
12
|
const taxonomy_1 = require("../taxonomy");
|
|
14
13
|
const citations_1 = require("../citations");
|
|
14
|
+
const model_1 = require("./model");
|
|
15
|
+
Object.defineProperty(exports, "providerError", { enumerable: true, get: function () { return model_1.providerError; } });
|
|
15
16
|
/**
|
|
16
17
|
* F2 — classify each entry point, offline, from F1's capture.
|
|
17
18
|
*
|
|
@@ -22,14 +23,15 @@ const citations_1 = require("../citations");
|
|
|
22
23
|
* **On duplication, stated because this project punishes silent drift:** the
|
|
23
24
|
* provider/model catalogue is imported from `@ia-qa/self-healing` rather than
|
|
24
25
|
* copied — there must not be a third model list in this repo. The HTTP transport
|
|
25
|
-
*
|
|
26
|
-
*
|
|
27
|
-
*
|
|
26
|
+
* now lives in `./model.ts`, which is a second implementation of something
|
|
27
|
+
* self-healing keeps internal (`callModel`); it moved out of this file when a
|
|
28
|
+
* second BYOK verb arrived, so the cost stays at two copies instead of three.
|
|
29
|
+
* Any change to a provider's request shape owes an edit there as well as in
|
|
30
|
+
* self-healing.
|
|
28
31
|
*/
|
|
29
32
|
/** The document writes, and the two strings a later stage keys on. */
|
|
30
33
|
exports.CLASSIFICATION_SCHEMA = 'qa-discovery-classification@1';
|
|
31
34
|
exports.CLASSIFICATION_FILENAME = 'classification.json';
|
|
32
|
-
const DEFAULTS = { timeoutMs: 30000, temperature: 0 };
|
|
33
35
|
/**
|
|
34
36
|
* The citable units of a page.
|
|
35
37
|
*
|
|
@@ -140,28 +142,7 @@ function buildPrompt(page, entryPoints) {
|
|
|
140
142
|
}
|
|
141
143
|
/** Pull the JSON object out of a model reply, tolerating fences and prose around it. */
|
|
142
144
|
function parseClassifications(text) {
|
|
143
|
-
|
|
144
|
-
return null;
|
|
145
|
-
const attempts = [text];
|
|
146
|
-
const fenced = /```(?:json)?\s*([\s\S]*?)```/i.exec(text);
|
|
147
|
-
if (fenced)
|
|
148
|
-
attempts.push(fenced[1]);
|
|
149
|
-
const first = text.indexOf('{');
|
|
150
|
-
const last = text.lastIndexOf('}');
|
|
151
|
-
if (first !== -1 && last > first)
|
|
152
|
-
attempts.push(text.slice(first, last + 1));
|
|
153
|
-
for (const candidate of attempts) {
|
|
154
|
-
try {
|
|
155
|
-
const parsed = JSON.parse(candidate.trim());
|
|
156
|
-
const list = Array.isArray(parsed) ? parsed : parsed?.classifications;
|
|
157
|
-
if (Array.isArray(list))
|
|
158
|
-
return list;
|
|
159
|
-
}
|
|
160
|
-
catch {
|
|
161
|
-
/* try the next shape */
|
|
162
|
-
}
|
|
163
|
-
}
|
|
164
|
-
return null;
|
|
145
|
+
return (0, model_1.parseJsonList)(text, 'classifications');
|
|
165
146
|
}
|
|
166
147
|
/**
|
|
167
148
|
* Validate one classification against the closed taxonomy, then against the
|
|
@@ -241,19 +222,14 @@ function collect(raws, index, offered) {
|
|
|
241
222
|
}
|
|
242
223
|
return { taxonomyVersion: taxonomy_1.TAXONOMY_VERSION, kept, refused, dropped };
|
|
243
224
|
}
|
|
244
|
-
// ──
|
|
245
|
-
const HOSTS = {
|
|
246
|
-
anthropic: 'api.anthropic.com',
|
|
247
|
-
openai: 'api.openai.com',
|
|
248
|
-
google: 'generativelanguage.googleapis.com',
|
|
249
|
-
};
|
|
225
|
+
// ── What leaves this machine ─────────────────────────────────────────────────
|
|
250
226
|
/**
|
|
251
227
|
* What leaves this machine, said before it leaves — the one thing that makes
|
|
252
228
|
* "everything is local except the model call" a promise rather than a claim.
|
|
253
229
|
*
|
|
254
|
-
* It
|
|
255
|
-
* `buildRequest` sends to, so a new provider cannot be reachable
|
|
256
|
-
* notice still lists three. And it is derived from `buildPrompt` — every line
|
|
230
|
+
* It resolves the host through `egressHost` rather than naming one of its own: the notice names the
|
|
231
|
+
* very constant `buildRequest` sends to, so a new provider cannot be reachable
|
|
232
|
+
* while the notice still lists three. And it is derived from `buildPrompt` — every line
|
|
257
233
|
* below is a field that function actually puts in the payload; adding one there
|
|
258
234
|
* owes a line here.
|
|
259
235
|
*
|
|
@@ -272,125 +248,21 @@ const HOSTS = {
|
|
|
272
248
|
* the token here. The word "full" is gone from that line for the same reason the
|
|
273
249
|
* allowlist is named: the two statements must not contradict each other.
|
|
274
250
|
*/
|
|
275
|
-
function egressNotice(provider, model, pages, planned = false) {
|
|
251
|
+
function egressNotice(provider, model, pages, planned = false, baseUrl) {
|
|
252
|
+
// The host it will REALLY reach. Throws when a configurable provider was given no URL —
|
|
253
|
+
// before anything is announced, because a notice printing `undefined` reads as a checked fact.
|
|
254
|
+
const host = (0, model_1.egressHost)(provider, baseUrl);
|
|
276
255
|
const n = `${pages} page${pages === 1 ? '' : 's'}`;
|
|
277
|
-
return (`\n 🌐 ${planned ? 'Would send' : 'Sending'} ${n} to ${
|
|
256
|
+
return (`\n 🌐 ${planned ? 'Would send' : 'Sending'} ${n} to ${host} (${provider} · ${model}) — your key, your account.\n` +
|
|
278
257
|
` Leaves this machine: each page's URL, title and description, up to 25 headings,\n` +
|
|
279
258
|
` every observed API call (method, normalised path, status), and each form field's name,\n` +
|
|
280
259
|
` type, label and selector. Headings and labels are live text from your app.\n` +
|
|
281
260
|
` Query values are stripped at capture EXCEPT an allowlist that includes \`q\` — a search\n` +
|
|
282
261
|
` term reaches the model as typed (\`capture.safeQueryParams\` in config.json narrows it).\n` +
|
|
283
262
|
` Does NOT leave: page HTML, screenshots, cookies or your session file, your test files,\n` +
|
|
284
|
-
` your API key (sent as a header to ${
|
|
263
|
+
` your API key (sent as a header to ${host} only).\n` +
|
|
285
264
|
` ${planned ? '' : 'Ctrl-C to stop. '}Deterministic \`ia-qa-discover\` sends nothing, ever.\n`);
|
|
286
265
|
}
|
|
287
|
-
function buildRequest(provider, model, apiKey, prompt, temperature) {
|
|
288
|
-
if (provider === 'anthropic') {
|
|
289
|
-
const body = { model, max_tokens: 4096, messages: [{ role: 'user', content: prompt }] };
|
|
290
|
-
if (temperature !== undefined)
|
|
291
|
-
body.temperature = temperature;
|
|
292
|
-
return {
|
|
293
|
-
url: `https://${HOSTS.anthropic}/v1/messages`,
|
|
294
|
-
headers: { 'content-type': 'application/json', 'x-api-key': apiKey, 'anthropic-version': '2023-06-01' },
|
|
295
|
-
body,
|
|
296
|
-
};
|
|
297
|
-
}
|
|
298
|
-
if (provider === 'openai') {
|
|
299
|
-
const body = { model, messages: [{ role: 'user', content: prompt }] };
|
|
300
|
-
if (temperature !== undefined)
|
|
301
|
-
body.temperature = temperature;
|
|
302
|
-
return {
|
|
303
|
-
url: `https://${HOSTS.openai}/v1/chat/completions`,
|
|
304
|
-
headers: { 'content-type': 'application/json', authorization: `Bearer ${apiKey}` },
|
|
305
|
-
body,
|
|
306
|
-
};
|
|
307
|
-
}
|
|
308
|
-
const body = { contents: [{ parts: [{ text: prompt }] }] };
|
|
309
|
-
if (temperature !== undefined)
|
|
310
|
-
body.generationConfig = { temperature };
|
|
311
|
-
return {
|
|
312
|
-
url: `https://${HOSTS.google}/v1beta/models/${encodeURIComponent(model)}:generateContent?key=${encodeURIComponent(apiKey)}`,
|
|
313
|
-
headers: { 'content-type': 'application/json' },
|
|
314
|
-
body,
|
|
315
|
-
};
|
|
316
|
-
}
|
|
317
|
-
function extractText(provider, data) {
|
|
318
|
-
if (provider === 'anthropic')
|
|
319
|
-
return data?.content?.[0]?.text ?? '';
|
|
320
|
-
if (provider === 'openai')
|
|
321
|
-
return data?.choices?.[0]?.message?.content ?? '';
|
|
322
|
-
return data?.candidates?.[0]?.content?.parts?.[0]?.text ?? '';
|
|
323
|
-
}
|
|
324
|
-
async function attempt(opts, prompt, temperature) {
|
|
325
|
-
const fetchFn = opts.fetchFn ?? globalThis.fetch;
|
|
326
|
-
if (typeof fetchFn !== 'function')
|
|
327
|
-
return null;
|
|
328
|
-
const req = buildRequest(opts.provider, opts.model, opts.apiKey, prompt, temperature);
|
|
329
|
-
const controller = new AbortController();
|
|
330
|
-
const timer = setTimeout(() => controller.abort(), opts.timeoutMs ?? DEFAULTS.timeoutMs);
|
|
331
|
-
try {
|
|
332
|
-
return await fetchFn(req.url, {
|
|
333
|
-
method: 'POST',
|
|
334
|
-
headers: req.headers,
|
|
335
|
-
body: JSON.stringify(req.body),
|
|
336
|
-
signal: controller.signal,
|
|
337
|
-
});
|
|
338
|
-
}
|
|
339
|
-
catch {
|
|
340
|
-
return null; // network error or timeout — never throws at the caller
|
|
341
|
-
}
|
|
342
|
-
finally {
|
|
343
|
-
clearTimeout(timer);
|
|
344
|
-
}
|
|
345
|
-
}
|
|
346
|
-
/**
|
|
347
|
-
* What the provider itself said went wrong.
|
|
348
|
-
*
|
|
349
|
-
* Found by execution, not review: a Gemini key that is simply wrong comes back
|
|
350
|
-
* **400**, not 401 — so the previous message read `refused the call with HTTP
|
|
351
|
-
* 400`, said nothing about the key, and left the one person who could fix it in
|
|
352
|
-
* sixty seconds with nothing to go on. Mapping more status codes by hand is the
|
|
353
|
-
* wrong shape of fix (the list rots, and it is different per provider); the
|
|
354
|
-
* providers already answer the question in the body, and all three happen to
|
|
355
|
-
* use the same `{error:{message}}` envelope.
|
|
356
|
-
*
|
|
357
|
-
* Two guards. The body is untrusted text that lands in a terminal and in
|
|
358
|
-
* `classification.json`, so it is capped and flattened. And anything shaped like
|
|
359
|
-
* a credential is redacted before it is printed — a provider that echoes the
|
|
360
|
-
* request back would otherwise put the user's own key in a file they commit.
|
|
361
|
-
*/
|
|
362
|
-
async function providerError(res) {
|
|
363
|
-
const status = `HTTP ${res.status}`;
|
|
364
|
-
// Kept alongside the provider's own words, not replaced by them: a 401 with
|
|
365
|
-
// an empty body still has exactly one likely cause, and an existing test went
|
|
366
|
-
// red the moment this hint was dropped — correctly.
|
|
367
|
-
const hint = res.status === 401 || res.status === 403 ? ' Check the API key.' : '';
|
|
368
|
-
let body = '';
|
|
369
|
-
try {
|
|
370
|
-
body = await res.text();
|
|
371
|
-
}
|
|
372
|
-
catch {
|
|
373
|
-
return `${status}.${hint}`;
|
|
374
|
-
}
|
|
375
|
-
let message = '';
|
|
376
|
-
try {
|
|
377
|
-
const parsed = JSON.parse(body);
|
|
378
|
-
message = parsed?.error?.message ?? parsed?.message ?? '';
|
|
379
|
-
}
|
|
380
|
-
catch {
|
|
381
|
-
message = body;
|
|
382
|
-
}
|
|
383
|
-
message = String(message).replace(/\s+/g, ' ').trim();
|
|
384
|
-
// sk-…, AIza…, ghp_…, and any long opaque run that could be a secret.
|
|
385
|
-
message = message.replace(/\b(?:sk-|AIza|ghp_|gsk_)[A-Za-z0-9_\-]{8,}/g, '[redacted]');
|
|
386
|
-
if (!message)
|
|
387
|
-
return `${status}.${hint}`;
|
|
388
|
-
if (message.length > 200)
|
|
389
|
-
message = message.slice(0, 200) + '…';
|
|
390
|
-
if (hint && !/[.!?…]$/.test(message))
|
|
391
|
-
message += '.';
|
|
392
|
-
return `${status} — ${message}${hint}`;
|
|
393
|
-
}
|
|
394
266
|
/**
|
|
395
267
|
* One page in, validated classifications out. Returns an empty result rather
|
|
396
268
|
* than throwing on a network failure, an unparseable reply or an empty page:
|
|
@@ -411,25 +283,10 @@ async function classifyPage(page, index, opts) {
|
|
|
411
283
|
return empty();
|
|
412
284
|
const prompt = buildPrompt(page, entryPoints);
|
|
413
285
|
const offered = offeredPaths(page, entryPoints);
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
let res = await attempt(opts, prompt, opts.temperature ?? DEFAULTS.temperature);
|
|
419
|
-
if (res && res.status === 400)
|
|
420
|
-
res = await attempt(opts, prompt, undefined);
|
|
421
|
-
if (!res)
|
|
422
|
-
return empty(`No reply from ${opts.provider} — network error or timeout.`);
|
|
423
|
-
if (!res.ok)
|
|
424
|
-
return empty(`${opts.provider} refused the call: ${await providerError(res)}`);
|
|
425
|
-
let text;
|
|
426
|
-
try {
|
|
427
|
-
text = extractText(opts.provider, await res.json());
|
|
428
|
-
}
|
|
429
|
-
catch {
|
|
430
|
-
return empty(`Could not read ${opts.provider}'s reply as JSON.`);
|
|
431
|
-
}
|
|
432
|
-
const raws = parseClassifications(text);
|
|
286
|
+
const call = await (0, model_1.callModel)(opts, prompt);
|
|
287
|
+
if (!call.ok)
|
|
288
|
+
return empty(call.reason);
|
|
289
|
+
const raws = parseClassifications(call.text);
|
|
433
290
|
if (!raws)
|
|
434
291
|
return empty(`${opts.provider} replied, but not with the JSON that was asked for.`);
|
|
435
292
|
return collect(raws, index, offered);
|