@ia-qa/qa-discovery 0.1.0

package/ROADMAP.md

@@ -0,0 +1,321 @@
+# @ia-qa/qa-discovery — roadmap
+
+## What this is for
+
+Hand a senior QA engineer an application they have never seen, with no documentation and
+no test plan. Watch what they do. They do **not** start by listing pages — they work out
+what the app is *for*, who uses it, and where it would hurt most if it broke. Then they
+decide what to test, and in what order, and they can tell you **why** for every line.
+
+That movement is what this package automates. The goal is not an inventory of an app; it
+is **the test plan a senior would write after a day with it** — prioritised, justified, and
+arriving in hours instead of days. Discovery is the judgement step that everything else in
+the ia-qa.com ecosystem assumes has already happened: `generate_test_cases` writes the
+scenario you hand it, the eval and security tools check the thing you point them at,
+`@ia-qa/self-healing` keeps a suite alive once it exists. All of them start *after*
+somebody decided what mattered. This is that somebody.
+
+Three consequences shape everything below.
+
+**"Why" is the deliverable, not the list.** A plan nobody can question is a plan nobody
+can trust or edit. So every claim traces to something observed in the app — that is what
+the governing rule below is for, and why an LLM never touches the capture itself.
+
+**It is used repeatedly, not once.** An app changes; a discovery is a perishable snapshot.
+The point is to re-run it and see what moved — which is why reconnaissance already diffs, keeps a
+history, and writes files meant to be committed. A tool that produces one impressive
+report and is never opened again has failed, however good the report.
+
+**A human and an agent are the same user.** Whatever the CLI does, `ia-qa-discover-mcp`
+exposes identically — `scan_app`, `classify_app`, `discovery_history`. There is no simplified
+view for one of them and a richer one for the other: one contract, served twice. The two
+exceptions are named rather than quiet, and both are the same exception: `login` waits for a
+person at a browser, which an agent cannot be.
+
+---
+
+How the pipeline grows past reconnaissance, and — just as important — what it will **not**
+become. One rule governs every phase:
+
+> **The LLM infers, the system decides.** Every claim a later stage makes carries two
+> things: a sentence a human reads, and a **path into the capture** that the system
+> resolves and checks before the claim is kept. A stage that cannot cite must be able to
+> say *"not enough evidence"* and stop. If a user configures no key, the tool behaves
+> exactly as it does today.
+>
+> **And the exact limit of that check, so nobody mistakes it for more:** the path proves
+> the **premise**, never the **conclusion**. A model can cite a password field that really
+> exists and still call the page a checkout. The machine can verify that what the model
+> leaned on is real; only a human can verify that the inference follows. That gap is the
+> whole reason the sentence must stay readable and the confidence must stay visible.
+
+The guardrails that make this safe, kept across all phases:
+
+1. **`scan` never calls an LLM.** Reconnaissance is deterministic, and stays the only thing that
+   touches the app. Everything inferential reads its output offline.
+2. **A separate binary** (`ia-qa-discover-ai`), like `ia-qa-heal-ai`. Absent key ⇒ the
+   AI layer is inert and the deterministic verbs behave identically.
+3. **Citations are checked, not trusted.** A classification naming
+   `pages/checkout.json#looseFields[2]` is only kept if that path exists and holds what
+   was claimed. An uncited claim is dropped, and the drop is reported.
+4. **A warning's register matches the claim's status.** A deterministic finding is a fact
+   about the app — stated flatly, once, no hedging ("2 of 2 fields cannot be located
+   durably"). An inference carries its doubt inside itself, so it does not also need a
+   klaxon on every row: the general caveat goes once at the top of the section, confidence
+   is shown per row, and a per-row alert is **reserved** for rows that earn it (below the
+   confidence floor, citation rejected, contradicted by another signal). Stamping "VERIFY
+   THIS" on 200 lines produces alarm fatigue, which is worse than no warning at all — the
+   three rows that needed attention drown in the 197 that did not.
+5. **A refusal path is mandatory.** An LLM asked to classify will always classify.
+   Without an "insufficient evidence" outcome, we ship confident nonsense — the exact
+   failure family reconnaissance already met four times (login wall, href blindness, fields outside
+   a form, non-unique selectors), each one a confident number that was wrong.
+6. **No verdict, still.** Nothing in this pipeline emits a CI gate. A test plan is a
+   proposal a human edits; the gate is `@ia-qa/self-healing`, downstream, once tests exist.
+7. **Zero bundled model.** Provider calls are a raw `fetch`, keys by reference, never on disk.
+
+---
+
+## The stages
+
+Seven of them. The `F` codes come from the original specification and are kept only so the
+two documents line up — the name is what matters, and nothing below should need the code to
+be understood.
+
+| Stage | The question it answers | Status |
+|---|---|---|
+| **Value and personas** (F0) | Who uses this app, for what, and where would it hurt most if it broke? | not built |
+| **Surface reconnaissance** (F1) | What pages exist, what do they take as input, what do they call? | ✅ shipped (0.1.0) |
+| **Component classification** (F2) | What is each entry point *for*, and how sensitive is what it handles? | in progress |
+| **Zones and synthesis** (F2.5) | How do the pages group into functional areas, and what is the app as a whole? | not built |
+| **Scenario generation** (F3) | What are the concrete test cases for this entry point? | not built |
+| **Risk prioritisation** (F4) | What should be tested first, and why? | not built |
+| **Export** (F5) | How does the plan reach Jira and the automation tooling? | not built |
+
+At run time they compose in that order — value first, export last. **The sections below are
+in _build_ order instead**, which is different and argued under Component classification: the
+order things get written in is driven by what unblocks what.
+
+## Surface reconnaissance (F1) — ✅ shipped in 0.1.0
+
+Crawl + capture: pages, meta, headings, forms and fields, observed API calls, and the
+coverage caveats that keep a page count from reading as coverage. Sitemap, crawl and
+declared pages as three merged sources; `login` for what is behind a wall; `--deep` for
+what is behind a click. Deterministic, local, no verdict.
+
+**What it measured on real apps, and what that changed:** an app with 11 routes and zero
+`<a href>` (a page count of "1 of 1 found" that meant one page); a catch-all route serving
+the login page at HTTP 200 on every URL; 464 source inputs against 26 `<form>` tags; one
+page with 64 clickable controls and 2 visible fields. Every one of those is now a stated
+caveat rather than a silent number.
+
+---
+
+## Component classification (F2) — **in progress**
+
+For each page and each entry point: what is this *for*, and how sensitive is what it
+handles? Input is the reconnaissance capture, offline. Output is a classification with citations.
+
+### Build order: classification is a quick win, value framing is a game changer
+
+The conceptual order is value-first: a senior reads what the app is *for* before judging what a
+given form is worth. The build order is the other way round, and the reason is not that classification
+matters more — it is that **the two are different kinds of work**.
+
+**Classification is a quick win, and its output is checkable in seconds.** "This entry point is
+authentication" → is there a password field, yes or no. That is what makes it the right
+place to *prove the citation architecture*: if the mechanism is wrong, it shows immediately.
+It also unlocks something deterministic straight away — an entry point classified
+`authentication` or `payment` can pull in security tools that already exist on the
+ia-qa.com MCP (`security_headers_check`, `cookie_security_audit`, `cors_test`,
+`ssl_certificate_check`, `secret_scan`) instead of asking an LLM to imagine those scenarios
+at the scenario stage.
+
+**Value framing is the game changer, and it deserves the design time.** "The moment of truth is
+checkout" cannot be verified mechanically — it needs domain judgement. So it needs
+something classification does not: **a validation protocol**. Label a handful of real apps by hand,
+compare, measure the agreement, and only then decide what the stage may assert. That is
+design work, not coding, and rushing it is how a game changer turns into a plausible-
+sounding paragraph nobody can check.
+
+**The risk of this order, written down so it is not discovered later:** on its own, this stage
+is only a labeller. It will look useful, it will demo well, and the value layer that makes
+this a *senior's* plan rather than a sophisticated inventory can quietly never get built.
+"Later" is how a game changer becomes a backlog item. Two things guard against it: it is
+scheduled as careful work, not optional work; and **the classification schema must leave a hole for
+the value map to fill** — a component's importance is a field the value stage supplies, never something
+the classifier invents from the DOM — so shipping it later completes the picture instead of reopening it.
+
+### Decided
+
+**Citation = a sentence *and* a path.** Not a choice between them: the sentence is what a
+human reads, the path is what the machine resolves. The claim is kept only if the path
+checks out, and the sentence is what makes the surviving claim auditable — see the limit
+stated in the governing rule (premise, not conclusion).
+
+```jsonc
+{
+  "entryPoint": "pages/home.json#forms[0]",
+  "intent": "authentication",
+  "confidence": 0.92,
+  "why": "The form takes an email and a password, and the page calls /auth/profile.",
+  "evidence": [
+    "pages/home.json#forms[0].fields[1]",   // type: "password"
+    "pages/home.json#apiCalls[3]"           // GET /auth/profile → 401
+  ]
+}
+```
+
+**Granularity = per entry point, not per page.** A page label is not actionable. Measured:
+`/llm-playground` on ia-qa.com holds 12 fields — a model selector, temperature sliders, a
+prompt box, and an API key field. One label for that page says nothing; per entry point,
+the API key field is a sensitive surface that should pull in `secret_scan` while the
+temperature slider is not. The grouping this needs already exists: reconnaissance stamps a `context` on
+every loose field (`main › Billing address`) for exactly this.
+
+**The vocabulary is closed and versioned** (`src/taxonomy.ts`, `qa-discovery-taxonomy@1`). The
+model picks an id or refuses; it cannot invent a label, because an invented vocabulary
+drifts between runs and two scans of one app stop being comparable — which kills the diff
+this package is built on.
+
+*The admission rule that keeps the list from bloating:* **an intent earns its place only if
+it changes what you would test.** Two labels leading to the same test plan are one label.
+Same discipline as `check`'s admission rule in self-healing; without it every plausible noun
+becomes a category and the classification discriminates nothing.
+
+*Two axes, not one* — the spec asks "what intention, **what data sensitivity**", and
+collapsing them makes both wrong. `admin` is not an intent (an admin page can be data entry,
+configuration or navigation); it is an access dimension. A test pins that, along with the
+rule that no tool mapping may name an MCP tool that does not exist.
+
+**Built so far** (the deterministic half, no LLM involved yet):
+- `src/citations.ts` — the verifier, written and tested *before* anything that produces
+  claims. A judge built after the accused tends to be shaped to acquit.
+- `src/taxonomy.ts` — the closed referentials, plus the intent/sensitivity → security-tool
+  mapping that is this stage's deterministic payoff.
+
+**On the refusal threshold — ship first, then set it from data.** "Empirical" does not mean
+a labelling protocol before any code; it means the number comes from looking at real output
+instead of picking one that sounds right. Two things make that safe here: the safety net is
+the **citation verifier**, not the threshold (invented evidence is rejected at any setting),
+and there is already a categorical refusal that needs no number at all — the model can
+answer `unknown`, a value in the taxonomy.
+
+So v0.1 **discards nothing on confidence**. Every classification is emitted with its score
+visible, because *a threshold applied on the first run hides exactly the data needed to set
+it*: ship at 0.7 and drop the rest, and you never learn whether 0.6 was fine or 0.8 was
+needed. The numeric floor is a second layer, added once the distribution over real captures
+says where it belongs.
+
+Still open:
+- **The prompt and the BYOK call**, mirroring `ia-qa-heal-ai`'s shape (raw `fetch`, three
+  providers, temperature 0).
+
+## Zones and app-level synthesis (F2.5)
+
+Cluster pages into functional zones (deterministic first: route structure and the
+navigation graph), then a grounded synthesis on top. Reconciles with the value stage once it exists.
+
+## Value and personas (F0) — *the game changer, built with care*
+
+Who uses this app, for what, and where the moment of truth is. This is the stage that makes
+the output a senior's plan instead of a classified inventory, and the one whose claims are
+hardest to check — so it is scheduled as **careful** work, not as later work.
+
+**It needs a validation protocol before it needs code.** Classification can be checked in seconds
+against the capture; this cannot. Label a handful of real apps by hand, compare the stage's
+framing to the labels, measure the agreement, and let that decide what this stage is allowed to
+assert and where its refusal threshold sits. Shipping a persuasive paragraph nobody can
+verify would be the uncalibrated-judge anti-pattern, one layer up.
+
+**One open question is now answered by evidence**: the original plan asked whether this stage needs
+access to the front-end bundle for its deterministic guardrails. Probably not — reconnaissance already
+captures every page's `<title>`, meta description and full heading outline, which is where
+the value proposition literally lives. Measured: one app's home title read
+*"139 LLM Testing Tools | Prompt Tests, RAG Audits, CI-Ready Assertions"*, and its method
+page carried 132 headings. That is the marketing content, verbatim, already on disk.
+
+## Risk prioritisation (F4)
+
+Deterministic scoring, visible and editable formula. **Part of it needs no LLM and could
+ship before classification**: reconnaissance already observes password fields (an auth surface), 4xx/5xx responses
+on a valid session, fields no test could locate durably, third-party origins, and forms
+with no submit control. Worth separating "risk signals reconnaissance already has" from "risk informed
+by the classifier's intent", so the first half is not blocked on the second.
+
+## Scenario generation (F3)
+
+**The audit found the assumed inputs do not exist.** There is no closed grid of test
+patterns anywhere in the ecosystem to derive from — `QA_AGENT_RECIPES` and
+`AUGMENTED_QA_PLAYBOOK` are editorial content, not structured data. So the intent→scenario
+grid is net-new. What *does* exist and should be reused rather than reimplemented:
+`generate_test_cases` (MCP) already turns a declared field constraint into valid / edge /
+invalid / pairwise cases — that is the field-level half, done.
+
+## Export (F5)
+
+**No standard test-suite format exists to target.** `jira_to_test_suite` returns whatever
+JSON the LLM produced, validated only for Gherkin syntax — so `test-plan.json` has to be
+designed, not adopted. The ADF/Gherkin rendering helpers in `atlassianHandler.js` are
+reusable for the Jira half.
+
+---
+
+---
+
+## Surfaces, not stages
+
+Two things that are not pipeline stages, deferred deliberately — with what would
+make them worth building, so the decision can be re-taken on evidence rather than
+on enthusiasm.
+
+### A local console (`ui`)
+
+`@ia-qa/self-healing` has one, and its console earns its place by being a **decision
+surface**: it serves the rows where the tool gave up and needs a human to choose,
+with the verbs as buttons.
+
+Discovery had no decisions to offer, which is why this was declined once already —
+a console would have been a prettier rendering of the HTML report. **Classification
+changes that argument**: refused entry points and dropped claims are exactly rows
+that need a person, and a risk ranking is something you want to sort and filter
+rather than read. The trigger is therefore risk prioritisation landing: once there
+is something to rank, there is something to explore. Building it before that would
+repeat the mistake the report already covers.
+
+Whatever gets built inherits self-healing's four containment rules without
+restatement: loopback only, a per-process token, Host pinning, and no new authority
+(actions spawn this package's own CLI with a fixed argv, never a string from the
+request).
+
+### A tutorial
+
+There is none, and that was the right call at three verbs: the README plus a real
+session transcript *was* the tutorial. **At three binaries and six verbs it is
+becoming the wrong call** — the loop is now scan → login → deep → classify, with a
+BYOK key and a taxonomy in the middle, and that is more than a README section can
+carry without becoming a manual.
+
+The trigger is scenario generation landing, because that is the point at which the
+output stops being "a description of your app" and becomes "a thing you act on" —
+which is what a tutorial has to teach. Note the shipping cost before starting: a
+`TUTORIAL.md` rendered by ia-qa.com owes a **site deploy as well as an npm publish**
+(see the README-image trap in the root CLAUDE.md).
+
+---
+
+## Explicitly out of scope (won't build)
+
+- **An LLM anywhere in `scan`.** The capture is the evidence. Contaminating it would leave nothing
+  for later stages to be checked against.
+- **A CI gate on any of this.** A test plan is a proposal. Gating on an LLM's classification
+  is the uncalibrated-judge anti-pattern the project rejects everywhere else.
+- **A hosted crawl.** Everything runs on the user's machine, on their app, with their key.
+  ia-qa.com ships tools, not a service that holds other people's applications.
+- **Autonomous test execution.** Discovery says what to test; running it is the Automation
+  skills' job, and self-healing's once tests exist.
+- **A second npm package for the AI layer.** It consumes the reconnaissance capture, so it cannot be
+  standalone; a third `bin` here gives the optional-add-on shape without a second release
+  pipeline.
+- **A `--junit` export.** There is no verdict, so it would be a gate that can never fail —
+  the most expensive thing this ecosystem can ship.

package/dist/ai/classify.d.ts

@@ -0,0 +1,170 @@
+import type { AiProvider } from '@ia-qa/self-healing';
+import type { PageCapture } from '../capture/page';
+import { type ToolTrigger } from '../taxonomy';
+import { type CaptureIndex, type Citation, type CitationVerdict } from '../citations';
+/**
+ * F2 — classify each entry point, offline, from F1's capture.
+ *
+ * The model never sees the app: it reads JSON that was captured deterministically,
+ * and every answer it gives must point back into that JSON. Nothing here opens a
+ * browser or touches the user's application.
+ *
+ * **On duplication, stated because this project punishes silent drift:** the
+ * provider/model catalogue is imported from `@ia-qa/self-healing` rather than
+ * copied — there must not be a third model list in this repo. The HTTP transport
+ * below *is* a second implementation, because self-healing does not export its
+ * own (`callModel` is internal). Any change to a provider's request shape owes an
+ * edit here as well as there.
+ */
+/** The document  writes, and the two strings a later stage keys on. */
+export declare const CLASSIFICATION_SCHEMA = "qa-discovery-classification@1";
+export declare const CLASSIFICATION_FILENAME = "classification.json";
+export interface ClassifyOptions {
+    provider: AiProvider;
+    model: string;
+    apiKey: string;
+    /** Injected in tests; defaults to the global fetch. */
+    fetchFn?: typeof fetch;
+    timeoutMs?: number;
+    temperature?: number;
+}
+/** One thing on a page that takes input, addressed by a path a citation can resolve. */
+export interface EntryPoint {
+    /** e.g. `pages/checkout.json#forms[0]` or `pages/checkout.json#looseFields[2]` */
+    path: string;
+    kind: 'form' | 'field';
+    page: string;
+    /** Everything the model is allowed to reason from, verbatim from the capture. */
+    detail: unknown;
+}
+export interface Classification {
+    entryPoint: string;
+    intent: string;
+    sensitivity: string;
+    /** The model's own number. v0.1 discards nothing on it — see ROADMAP. */
+    confidence: number;
+    /** The sentence a human reads. The citations are what the machine checks. */
+    why: string;
+    evidence: Citation[];
+    /** Filled in deterministically from the taxonomy, never by the model. */
+    securityTools?: ToolTrigger[];
+}
+export interface DroppedClassification {
+    classification: Partial<Classification>;
+    reason: string;
+    rejected: Array<Extract<CitationVerdict, {
+        ok: false;
+    }>>;
+}
+export interface ClassifyResult {
+    taxonomyVersion: string;
+    kept: Classification[];
+    dropped: DroppedClassification[];
+    /** Entry points the model declined to label. A refusal is a result, not a gap. */
+    refused: Classification[];
+    /**
+     * Set when the call did not succeed at all — network down, key rejected,
+     * reply unreadable.
+     *
+     * Without this, a failed call and a page with nothing to classify are the same
+     * empty result, and "0 classified" reads as a fact about the app instead of a
+     * fact about the request. Measured: an invalid key produced
+     * "0 classified · 0 refused · 0 dropped", exit 0, and a written file.
+     */
+    failed?: {
+        reason: string;
+    };
+}
+/**
+ * The citable units of a page.
+ *
+ * A `<form>` is one entry point. A loose field is its own, because grouping them
+ * would need a path that does not exist in the capture and a citation must
+ * resolve to something real. If that proves too granular in practice, grouping is
+ * a v0.2 question — answered by looking at real output, not guessed now.
+ */
+export declare function entryPointsOf(page: PageCapture): EntryPoint[];
+/**
+ * What the model is given. Deliberately narrow: the page's identity and headings
+ * for context, and the entry points verbatim. It is not handed the whole capture,
+ * because a model given everything cites the most quotable thing rather than the
+ * most relevant one.
+ */
+/**
+ * Every path the prompt offers, and therefore the only paths a citation may
+ * use. The model picks from a menu; it does not compose.
+ *
+ * This is the same guardrail `@ia-qa/self-healing` puts on its resolver — "the
+ * model chooses a live candidate by index, so it cannot emit a broken selector".
+ * Measured need: across three real runs the model cited `#page.description`,
+ * `#looseFields[0].detail` and a bare `#description`, all of which existed in the
+ * *prompt's* shape and none in the capture's. Resolving-and-rejecting caught
+ * them, but only after the fact; refusing a constructed path removes the whole
+ * class of error instead.
+ */
+export declare function offeredPaths(page: PageCapture, entryPoints: EntryPoint[]): Set<string>;
+export declare function buildPrompt(page: PageCapture, entryPoints: EntryPoint[]): string;
+/** Pull the JSON object out of a model reply, tolerating fences and prose around it. */
+export declare function parseClassifications(text: string): Partial<Classification>[] | null;
+/**
+ * Validate one classification against the closed taxonomy, then against the
+ * capture. Order matters: an off-list label is rejected before any citation is
+ * resolved, because a model inventing vocabulary is not a model whose evidence
+ * is worth checking.
+ */
+export declare function validate(raw: Partial<Classification>, index: CaptureIndex, offered?: Set<string>): {
+    ok: true;
+    value: Classification;
+} | {
+    ok: false;
+    dropped: DroppedClassification;
+};
+/** Sort validated classifications into kept / refused, and collect the drops. */
+export declare function collect(raws: Partial<Classification>[], index: CaptureIndex, offered?: Set<string>): ClassifyResult;
+/**
+ * What leaves this machine, said before it leaves — the one thing that makes
+ * "everything is local except the model call" a promise rather than a claim.
+ *
+ * It lives beside `HOSTS` on purpose: the notice names the very constant
+ * `buildRequest` sends to, so a new provider cannot be reachable while the
+ * notice still lists three. And it is derived from `buildPrompt` — every line
+ * below is a field that function actually puts in the payload; adding one there
+ * owes a line here.
+ *
+ * Written to stderr in every mode, `--json` included: it is a security
+ * announcement, not decoration, and stdout must stay one parseable document.
+ *
+ * The last clause is the one that took a check to get right. Query values are
+ * stripped at capture *except* an allowlist, and that allowlist contains `q` —
+ * a search term, i.e. real user input. Writing "query values are stripped" here
+ * would have been a false reassurance in the one place a reader cannot afford
+ * one, so the allowlist is named instead of summarised.
+ */
+export declare function egressNotice(provider: AiProvider, model: string, pages: number, planned?: boolean): string;
+/**
+ * What the provider itself said went wrong.
+ *
+ * Found by execution, not review: a Gemini key that is simply wrong comes back
+ * **400**, not 401 — so the previous message read `refused the call with HTTP
+ * 400`, said nothing about the key, and left the one person who could fix it in
+ * sixty seconds with nothing to go on. Mapping more status codes by hand is the
+ * wrong shape of fix (the list rots, and it is different per provider); the
+ * providers already answer the question in the body, and all three happen to
+ * use the same `{error:{message}}` envelope.
+ *
+ * Two guards. The body is untrusted text that lands in a terminal and in
+ * `classification.json`, so it is capped and flattened. And anything shaped like
+ * a credential is redacted before it is printed — a provider that echoes the
+ * request back would otherwise put the user's own key in a file they commit.
+ */
+export declare function providerError(res: {
+    status: number;
+    text: () => Promise<string>;
+}): Promise<string>;
+/**
+ * One page in, validated classifications out. Returns an empty result rather
+ * than throwing on a network failure, an unparseable reply or an empty page:
+ * this layer is optional by construction, and a broken key must never take down
+ * a deterministic scan that already succeeded.
+ */
+export declare function classifyPage(page: PageCapture, index: CaptureIndex, opts: ClassifyOptions): Promise<ClassifyResult>;