@chrono-meta/fh-gate 3.0.0 → 3.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/regression/probes_live.yaml +137 -0
- package/.claude/rules/.residency-patterns.defaults +7 -0
- package/.claude/rules/fh_4axis_gate.md +50 -1
- package/.claude-plugin/marketplace.json +8 -2
- package/AGENTS.md +27 -0
- package/CATALOG.md +17 -0
- package/CLAUDE.md +12 -2
- package/README.ja.md +51 -7
- package/README.ko.md +48 -7
- package/README.md +37 -5
- package/README.zh.md +45 -8
- package/docs/STANDARDS_ALIGNMENT.md +61 -0
- package/docs/USER_GUIDE.md +3 -0
- package/docs/USE_CASES.md +50 -0
- package/docs/model_tier_expectations.md +60 -0
- package/knowledge/shared/harness-core/fh_three_layer_canon.md +30 -10
- package/knowledge/shared/harness-core/field_verdict_crossfamily_gate.md +8 -0
- package/knowledge/shared/harness-core/harness_incubator_doctrine.md +9 -0
- package/knowledge/shared/harness-core/iso_ai_standards_crosswalk.md +139 -0
- package/knowledge/shared/harness-core/measurement-integrity-checklist.md +20 -0
- package/knowledge/shared/learnings/subagent_invocations_log.yaml +441 -4
- package/package.json +40 -2
- package/plugins/fh-commons/.claude-plugin/plugin.json +1 -1
- package/plugins/fh-commons/skills/preprep/README.md +4 -1
- package/plugins/fh-commons/skills/preprep/SKILL.md +99 -3
- package/plugins/fh-commons/skills/preprep/diagram_from_json.py +154 -0
- package/plugins/fh-commons/skills/preprep/fixtures/fixture_R3_negative.pptx +0 -0
- package/plugins/fh-commons/skills/preprep/fixtures/fixture_R3_positive.pptx +0 -0
- package/plugins/fh-commons/skills/preprep/fixtures/mk_slide_fixtures.py +179 -0
- package/plugins/fh-commons/skills/preprep/interslide_deps.py +98 -9
- package/plugins/fh-commons/skills/preprep/lane_adjacent_dup.py +4 -1
- package/plugins/fh-commons/skills/preprep/lane_diagram.py +95 -0
- package/plugins/fh-commons/skills/preprep/lane_geometry.py +181 -0
- package/plugins/fh-commons/skills/preprep/lane_promise.py +4 -1
- package/plugins/fh-commons/skills/preprep/lane_slide_refs.py +134 -0
- package/plugins/fh-commons/skills/preprep/lane_slide_relations.py +300 -0
- package/plugins/fh-commons/skills/preprep/preprep.py +129 -8
- package/plugins/fh-commons/skills/preprep/preprep_wire.py +210 -0
- package/plugins/fh-commons/skills/preprep/presentation_checklist.md +4 -0
- package/plugins/fh-commons/skills/preprep/surfaces.example.yaml +18 -0
- package/plugins/fh-commons/skills/preprep/test_preprep_lanes_rp.py +184 -0
- package/plugins/fh-meta/.claude-plugin/plugin.json +1 -1
- package/plugins/fh-meta/CHANGELOG.md +43 -1
- package/plugins/fh-meta/skills/auto-decorrelation/SKILL.md +87 -2
- package/plugins/fh-meta/skills/frontier-digest/SKILL.md +2 -2
- package/plugins/fh-meta/skills/frontier-digest/SKILL_detail.md +25 -5
- package/plugins/fh-meta/skills/hub-cc-pr-reviewer/SKILL.md +11 -0
- package/plugins/fh-meta/skills/hub-cc-pr-reviewer/SKILL_detail.md +3 -0
- package/plugins/fh-qp/.claude-plugin/plugin.json +22 -0
- package/plugins/fh-qp/README.md +71 -0
- package/plugins/fh-qp/fixtures/evidence_known_clean.txt +3 -0
- package/plugins/fh-qp/fixtures/evidence_known_dirty.txt +6 -0
- package/plugins/fh-qp/fixtures/reach_known_partial.tsv +4 -0
- package/plugins/fh-qp/fixtures/reach_known_reached.tsv +4 -0
- package/plugins/fh-qp/fixtures/reach_known_wall.tsv +3 -0
- package/plugins/fh-qp/fixtures/verdicts_known_bad_branch.tsv +2 -0
- package/plugins/fh-qp/fixtures/verdicts_known_bad_vacuous_machine.tsv +2 -0
- package/plugins/fh-qp/fixtures/verdicts_known_good.tsv +4 -0
- package/plugins/fh-qp/fixtures/verdicts_verify_only.tsv +3 -0
- package/plugins/fh-qp/qp_profile.example.yaml +29 -0
- package/plugins/fh-qp/scripts/qp_tools.sh +216 -0
- package/plugins/fh-qp/skills/qp/SKILL.md +65 -0
- package/plugins/fh-qp/skills/qp-plan/SKILL.md +42 -0
- package/plugins/fh-qp/skills/qp-regress/SKILL.md +45 -0
- package/plugins/fh-qp/skills/qp-run/SKILL.md +49 -0
- package/scripts/chamber_run.sh +14 -5
- package/scripts/com.forge-harness.live-eval.plist +84 -0
- package/scripts/compaction_probe.sh +9 -35
- package/scripts/directional_diff_gate.sh +14 -2
- package/scripts/frontier_digest_autopilot.sh +4 -1
- package/scripts/map_postprocess.py +90 -0
- package/scripts/outbound_query_guard.sh +131 -0
- package/scripts/outbound_query_hook.sh +373 -0
- package/scripts/package_coverage_check.sh +55 -16
- package/scripts/pipe_verdict_guard.sh +41 -1
- package/scripts/probe_live_eval.sh +240 -0
- package/scripts/probe_live_eval_lib.py +579 -0
- package/scripts/proposal_hook.sh +120 -17
- package/scripts/push_zone_check.sh +78 -0
- package/scripts/residency_closure_scan.py +252 -0
- package/scripts/selfcheck.sh +41 -1
- package/scripts/session_close_check.sh +100 -0
- package/scripts/sim_isolated_run.sh +98 -2
- package/scripts/test_action_yml_lanes.sh +97 -0
- package/scripts/test_fh_qp_lanes.sh +105 -0
- package/scripts/test_gate_two_verdicts_lanes.sh +139 -0
- package/scripts/test_map_postprocess_lanes.sh +143 -0
- package/scripts/test_marker_affected_lanes.sh +93 -0
- package/scripts/test_marker_crossfamily_lanes.sh +90 -6
- package/scripts/test_marker_oracle_lanes.sh +136 -0
- package/scripts/test_outbound_query_hook_lanes.sh +433 -0
- package/scripts/test_outbound_query_lanes.sh +87 -0
- package/scripts/test_pipe_verdict_guard_lanes.sh +26 -0
- package/scripts/test_preprep_diagram_lanes.sh +87 -0
- package/scripts/test_preprep_drift_anchor.sh +3 -3
- package/scripts/test_preprep_slide_refs_lanes.sh +169 -0
- package/scripts/test_probe_live_eval_lanes.sh +437 -0
- package/scripts/test_proposal_hook_lanes.sh +22 -1
- package/scripts/test_push_zone_lanes.sh +304 -0
- package/scripts/test_residency_closure_lanes.sh +70 -0
- package/scripts/test_sim_isolated_run_lanes.sh +119 -0
- package/scripts/test_utterance_intake_lanes.sh +414 -0
- package/scripts/test_worktree_reclaim_lanes.sh +70 -0
- package/scripts/transcript_utterances.py +222 -0
- package/scripts/utterance_intake.sh +424 -0
- package/scripts/validate_yaml.sh +27 -0
- package/scripts/worktree_reclaim.sh +95 -0
- package/templates/.git-hooks/pre-commit +353 -1
- package/templates/.git-hooks/pre-push +91 -0
- package/templates/RED_TEAM_REPORT.md +49 -0
- package/templates/settings.PreToolUse.snippet.json +65 -1
|
@@ -69,7 +69,7 @@ Collect from five sources (bash per source in §Collection-Bash):
|
|
|
69
69
|
|
|
70
70
|
| Source | Method | Cap |
|
|
71
71
|
|---|---|---|
|
|
72
|
-
| HackerNews | Algolia API, score >
|
|
72
|
+
| HackerNews | Algolia API `search_by_date` (date-sorted, never `/search`), score > 30, keyword-relevant | 15 items |
|
|
73
73
|
| arxiv | export API, latest by submittedDate | 6 items |
|
|
74
74
|
| TLDR AI | RSS, title + link | 5 items |
|
|
75
75
|
| The Batch (deeplearning.ai) | HTML scraping, title + issue slug | 5 items |
|
|
@@ -215,7 +215,7 @@ Present Step 4 menu options [1]–[5]. Do not skip to [5] silently — surface t
|
|
|
215
215
|
## Simplification Guards
|
|
216
216
|
|
|
217
217
|
- Video Tier-3 probe fails (any of `yt-dlp` / `curl_cffi` / `ffmpeg` missing, or timedtext returns 429) → fall through to operator summary; never assume `yt-dlp` works
|
|
218
|
-
- If 3+ arxiv queries fail
|
|
218
|
+
- If 3+ arxiv queries fail (HTTP 429): back off once → WebFetch the date-sorted `arxiv.org/list/cs.SE/recent` listing → only then HN-only. Never substitute WebSearch (recall channel, 5/5 REPEATs measured 2026-09-04); report `arxiv FAILED (429)`, never `0 items`. Detail: `SKILL_detail.md §Execution form`
|
|
219
219
|
- On curl timeout, skip that item and continue with the rest
|
|
220
220
|
- If synthesis result exceeds 400 characters, retain top 3 items and truncate the rest
|
|
221
221
|
- Without `--save`, do not create files (conversation output only)
|
|
@@ -21,20 +21,40 @@ load: on-demand
|
|
|
21
21
|
> through WebFetch on the same URLs. If an environment *does* allow the curl form, it is equivalent —
|
|
22
22
|
> the transport is not the contract, the endpoints and criteria are.
|
|
23
23
|
>
|
|
24
|
-
> **arXiv HTTP 429 is a
|
|
25
|
-
>
|
|
26
|
-
> `
|
|
24
|
+
> **arXiv HTTP 429 — fallback is a TRANSPORT change, never a channel change (revised 2026-09-04).**
|
|
25
|
+
> The export API rate-limits this runner (N=11 consecutive by 08-10; 3 consecutive days 09-02·03·04
|
|
26
|
+
> = past the `N≥3` mechanization bar). The 08-10 prescription («fall back to per-item `abs/{id}`
|
|
27
|
+
> reads») was unrunnable in practice — a 429 on the *query* leaves you with no IDs to read — so runs
|
|
28
|
+
> fell back to **WebSearch**, and the 09-04 run measured what that does: 5/5 papers it returned were
|
|
29
|
+
> REPEATs. Search sorts by *canonicity*, the export API by *submission date* — a search fallback is
|
|
30
|
+
> a **recall channel, not a discovery channel**, and swapping it in silently converts «unrun» into
|
|
31
|
+
> «nothing new» (`not found ≠ 0`). Order on 429:
|
|
32
|
+
> 1. back off once (≥30s), retry the same query;
|
|
33
|
+
> 2. still 429 → WebFetch the **date-sorted HTML listing** `https://arxiv.org/list/cs.SE/recent`
|
|
34
|
+
> (and `https://arxiv.org/list/cs.AI/recent` if the first is empty of agent/harness items), keep
|
|
35
|
+
> items whose title/abstract match the three query phrases, cap 6 — same sort axis as the API;
|
|
36
|
+
> 3. only if that also fails → HN-only, and the progress line says `arxiv FAILED (429)`, never `0 items`.
|
|
37
|
+
> 🟥 WebSearch is **not** on this ladder. If a run used it anyway, the arxiv leg is reported as
|
|
38
|
+
> `RECALL-ONLY (WebSearch)` and its items are excluded from the NEW/REPEAT count.
|
|
27
39
|
|
|
28
40
|
### HackerNews (Algolia API)
|
|
29
41
|
|
|
30
42
|
```bash
|
|
31
43
|
for KW in "AI agent" "LLM harness" "Claude" "multi-agent" "context engineering"; do
|
|
32
44
|
curl -s --max-time 8 \
|
|
33
|
-
"https://hn.algolia.com/api/v1/
|
|
45
|
+
"https://hn.algolia.com/api/v1/search_by_date?query=$(echo $KW | tr ' ' '+')&tags=story&hitsPerPage=5&numericFilters=points>30"
|
|
34
46
|
done
|
|
35
47
|
```
|
|
36
48
|
|
|
37
|
-
Collection criteria: score >
|
|
49
|
+
Collection criteria: score > 30, keyword-relevant items only. Max 15 items.
|
|
50
|
+
|
|
51
|
+
> **`/search_by_date`, not `/search` (revised 2026-09-04; prescribed 2026-09-01, unapplied for 3 runs).**
|
|
52
|
+
> `/search` ranks by relevance and returned items dated 2025-11→2026-06 on 2026-09-01 — a *relevant-but-stale*
|
|
53
|
+
> leg, the mirror image of the 2026-07-26 arXiv *fresh-but-off-axis* failure, and the HN leg carried neither
|
|
54
|
+
> refresh trigger. `/search_by_date` sorts by submission time; the points floor rises 10→30 to keep the
|
|
55
|
+
> date-sorted window from filling with noise (measured 2026-09-01: `points>30` gave a clean 08-27→08-31
|
|
56
|
+
> window). Same rule as the arXiv leg: a leg that silently falls back to a relevance/recall channel converts
|
|
57
|
+
> «unrun» into «nothing new».
|
|
38
58
|
|
|
39
59
|
### arxiv
|
|
40
60
|
|
|
@@ -98,6 +98,17 @@ verdict NOT_CONFIGURED, or the
|
|
|
98
98
|
returned, what was found — never the matched lines themselves. Same rule as the marker: the file is
|
|
99
99
|
a local artifact, the capsule is what crosses the boundary.
|
|
100
100
|
|
|
101
|
+
**Noise control — two rules that decide what reaches the comment (absorbed 2026-09-04 from the
|
|
102
|
+
AI-Native SDLC playbook's `REVIEW.md`; FH already tiers findings M/S/R, but had no cap and no
|
|
103
|
+
exclusion list, so a long PR produced a long comment that buried the one finding that mattered):**
|
|
104
|
+
- **Nit cap = 5.** A nit is any finding that would not change behavior, leak data, or breach a
|
|
105
|
+
baseline rule (style · naming · wording · ordering). Report at most five nits, then one line:
|
|
106
|
+
`+N more nits (not listed)`. M-tier and S-tier findings are never capped.
|
|
107
|
+
- **Do-not-report list.** Skip anything a mechanical gate already enforces on this repo (the
|
|
108
|
+
pre-commit 4-axis gate, `validate` CI, `regression_guard`, public-surface scan) and generated or
|
|
109
|
+
vendored files. Reporting what a gate already blocks is double coverage that reads as noise; if
|
|
110
|
+
a gate *should* have caught it and did not, that is an S-tier finding about the gate, not a nit.
|
|
111
|
+
|
|
101
112
|
Then attach the review comment (8-matrix results + self-catch + refinement suggestions + merge
|
|
102
113
|
recommendation) via `gh pr comment`.
|
|
103
114
|
|
|
@@ -28,6 +28,9 @@ gh pr comment "$PR_NUMBER" --body "$(cat <<'EOF'
|
|
|
28
28
|
### Refinement Suggestions (following simplification guard / areas for subsequent rounds outside this PR)
|
|
29
29
|
{Subsequent round areas / omit if 0 items}
|
|
30
30
|
|
|
31
|
+
### Nits (max 5; +N more not listed)
|
|
32
|
+
{At most five style/naming/wording items; omit the section if 0. Never list what a gate already enforces.}
|
|
33
|
+
|
|
31
34
|
### Admin Override Merge Recommendation
|
|
32
35
|
{User decision delegation / beta stage policy adherence}
|
|
33
36
|
EOF
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "fh-qp",
|
|
3
|
+
"version": "3.1.0",
|
|
4
|
+
"engines": {
|
|
5
|
+
"claudeCode": ">=1.0.0"
|
|
6
|
+
},
|
|
7
|
+
"description": "QP (Quality Platform) — 4 skills (qp · qp-plan · qp-run · qp-regress). Plan→Automation→Regression loop for web and desktop apps with zero domain constants. Calls a registered qasp typed capability when one is present (strictest-wins); otherwise drives the target through the session's Playwright MCP (web) or computer-use MCP (desktop). Mobile deferred.",
|
|
8
|
+
"author": {
|
|
9
|
+
"name": "chrono-meta",
|
|
10
|
+
"email": "chrono-meta@users.noreply.github.com"
|
|
11
|
+
},
|
|
12
|
+
"keywords": [
|
|
13
|
+
"qa",
|
|
14
|
+
"quality-platform",
|
|
15
|
+
"test-planning",
|
|
16
|
+
"e2e",
|
|
17
|
+
"regression",
|
|
18
|
+
"surface-reach",
|
|
19
|
+
"mtm",
|
|
20
|
+
"domain-agnostic"
|
|
21
|
+
]
|
|
22
|
+
}
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# fh-qp — QP (Quality Platform)
|
|
2
|
+
|
|
3
|
+
QP(Quality Platform) — qasp 의 PAR 를 도메인 상수 0 으로 옮긴 FH 동봉 플러그인.
|
|
4
|
+
|
|
5
|
+
> Namespace note: in the field harness's own documents «QP» has meant *QA-Prism* (a roadmap label). This plugin's QP is **Quality Platform** — a different name space; nothing here refers to that roadmap.
|
|
6
|
+
|
|
7
|
+
**What it is for**: you are inside Claude Code, you have a web page or a desktop app in front of you, and you want a repeatable **plan → run → regress** loop with a QA lens — not a one-off "click around for me".
|
|
8
|
+
|
|
9
|
+
## Install / prerequisites (read this first — the first run fails without them)
|
|
10
|
+
|
|
11
|
+
| you test | you need connected in this session | how QP detects it |
|
|
12
|
+
|---|---|---|
|
|
13
|
+
| a **web** page | Playwright MCP (`mcp__playwright__*`) — or the Claude-in-Chrome extension | `qp_tools.sh adapter-probe --need web` |
|
|
14
|
+
| a **desktop** app | computer-use MCP (`mcp__computer-use__*`) | `qp_tools.sh adapter-probe --need desktop` |
|
|
15
|
+
| a **mobile** app | — deferred; not supported | — |
|
|
16
|
+
|
|
17
|
+
No adapter → every stage stops with `HARNESS_ERROR` (exit 10, plus a `qp/HARNESS_ERROR.txt`). That is a typed non-pass, never a skip.
|
|
18
|
+
Nothing else to install: the plugin is four skills + one bash script; no Python, no npm packages.
|
|
19
|
+
|
|
20
|
+
## First command
|
|
21
|
+
|
|
22
|
+
```
|
|
23
|
+
/qp https://app.example.com/ # plan → run → regress, stops at the first HARNESS_ERROR
|
|
24
|
+
/qp plan https://app.example.com/ # one stage
|
|
25
|
+
/qp app:"Example App" # desktop (needs a profile — see below)
|
|
26
|
+
```
|
|
27
|
+
You get files under `qp/`: `plan/inventory.md` · `plan/tcs.tsv` · `run/<ts>/verdicts.tsv` + masked evidence · `regress/<ts>/surface_reach.txt` + `delta.tsv` + `report.md`. You succeeded when `report.md` exists and its first 8 lines name the engine, the MTM state and a surface_reach line.
|
|
28
|
+
|
|
29
|
+
## Public vs profile-required targets (zero domain constants)
|
|
30
|
+
|
|
31
|
+
The plugin ships **no** host names, app names, credentials or organisation words. A target that is not public (private IP, `localhost`, `*.internal`/`*.lan`/`*.local`, a bare hostname, or any desktop app) is `PROFILE_REQUIRED` (exit 4) until you supply a profile:
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
cp plugins/fh-qp/qp_profile.example.yaml <a gitignored path>/qp_profile.yaml # fill hosts/apps, reference env vars for secrets
|
|
35
|
+
/qp https://private.example.internal/ --profile <that path>
|
|
36
|
+
```
|
|
37
|
+
The profile is the only place a target-specific constant may live, and it is yours, never the plugin's.
|
|
38
|
+
|
|
39
|
+
## Engine — form C (skills are canonical; machinery is pluggable)
|
|
40
|
+
|
|
41
|
+
1. **Typed capability** — if a QA harness registers a capability whose id ends in `:par`, QP calls its entry and merges constraints strictest-wins (`knowledge/shared/harness-core/capability_composition_contract.md`). **Status 2026-09-05: no such capability is registered anywhere** (FH registry and the field harness both checked — zero cap files). Registration is the field harness's move; until then this bridge is a declared slot. Asking for it explicitly (`--engine capability`) returns `HARNESS_ERROR reason=capability-not-registered`, not a silent fallback.
|
|
42
|
+
2. **MCP fallback** — the path that exists today. Every artifact header says `engine=mcp-fallback adapter=<name> evidence=dom|pixel`, so a reader can tell which edition produced the report.
|
|
43
|
+
3. Desktop through computer-use is the **weaker edition** (pixel evidence, no DOM): `verify` steps close by readable screenshot text only, surface_reach hashes screenshots. The report header says so.
|
|
44
|
+
|
|
45
|
+
## What is mechanical vs what is prose (honest boundary)
|
|
46
|
+
|
|
47
|
+
| check | where | closed by |
|
|
48
|
+
|---|---|---|
|
|
49
|
+
| target public / profile-required | `scripts/qp_tools.sh target-class` | exit enum 0/4/10 |
|
|
50
|
+
| adapter present | `adapter-probe` | exit 0/10 (unknown ≠ present) |
|
|
51
|
+
| evidence masking, residue 0 | `mask` | exit 0/5 |
|
|
52
|
+
| surface_reach (TCs beyond entry / all TCs) | `surface-reach` | REACHED · PARTIAL · NOT_REACHED · UNMEASURED |
|
|
53
|
+
| verdict record shape — status/branch/closure/verb enums, `MACHINE` needs an assertion, branch↔`expected_source` binding | `mtm-check` | exit 0/10 |
|
|
54
|
+
| ≥1 state-changing verb executed | `run-verbs` | exit 0/5 |
|
|
55
|
+
| **whether a label is TRUE** (is this really *code differs*?) | prose in `qp-run` | judgment — paired, not mechanised (a frozen rule would be tomorrow's ceiling) |
|
|
56
|
+
|
|
57
|
+
The last row is the honest residual: QP types *that* a verdict is attributable and non-vacuous, not *that* it is correct.
|
|
58
|
+
|
|
59
|
+
## Skills
|
|
60
|
+
| skill | stage | Done When (summary) |
|
|
61
|
+
|---|---|---|
|
|
62
|
+
| `qp` | router | target classified · adapter probed · every artifact carries `engine=`/`evidence=` |
|
|
63
|
+
| `qp-plan` | Prepare | inventory ≥1 observed surface · `tcs.tsv` 8 fields with `expected_source` · ≥1 click/input TC |
|
|
64
|
+
| `qp-run` | Automation | `verdicts.tsv` passes `mtm-check` + `run-verbs` · evidence masked, residue 0 |
|
|
65
|
+
| `qp-regress` | Regression | `surface_reach.txt` from the tool · one delta row per TC · NOT_REACHED sentence present |
|
|
66
|
+
|
|
67
|
+
## Lane
|
|
68
|
+
`bash scripts/test_fh_qp_lanes.sh` — known pairs for the three negatives (profile-required → refuse · no adapter → HARNESS_ERROR · dirty evidence → masked, residue 0), surface_reach four states, mtm-check good/bad, run-verbs, and a residency scan of the whole plugin (fail-closed if the scanner cannot run).
|
|
69
|
+
|
|
70
|
+
## Prior art this is built on (curated, not reinvented)
|
|
71
|
+
Playwright Test Agents (planner / generator / healer) and Playwright MCP do the exploring and the driving; Cypress UI Coverage and mabl cover page-level coverage; `page.screenshot({mask})` covers web masking. QP adds only the judgment layer on top: the verdict record contract (`mtm-check`), first-step BLOCKED attribution, surface_reach as a ratio over all TCs, and profile-gated targets.
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
step 1 login as qa.tester@example.com
|
|
2
|
+
Authorization: Bearer abcdefghijklmnopqrstuvwxyz0123456789
|
|
3
|
+
token=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJVadQssw5c
|
|
4
|
+
password: hunter2hunter2
|
|
5
|
+
sha=0123456789abcdef0123456789abcdef01234567
|
|
6
|
+
plain text stays
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# qp_profile.example.yaml — the ONLY place a target-specific constant may live.
|
|
2
|
+
# Copy to a gitignored path (e.g. tracks/_meta/qp_profile.yaml) and pass it with --profile.
|
|
3
|
+
# The plugin itself ships ZERO domain constants; without a profile, only public targets run.
|
|
4
|
+
#
|
|
5
|
+
# Every value below is a placeholder. Nothing in this file names a real organisation.
|
|
6
|
+
profile_name: example
|
|
7
|
+
targets:
|
|
8
|
+
web:
|
|
9
|
+
- id: example-web
|
|
10
|
+
base_url: https://app.example.com # non-public hosts (10.x, *.internal, *.lan, bare names) REQUIRE this file
|
|
11
|
+
route_prefix: "" # e.g. "#/" for hash routers
|
|
12
|
+
entry_route: /
|
|
13
|
+
desktop:
|
|
14
|
+
- id: example-desktop
|
|
15
|
+
app_name: Example App # window title / bundle name as the OS shows it
|
|
16
|
+
credentials:
|
|
17
|
+
# Never put secrets here. Reference an env var; qp-run reads it at type-time and masks it in evidence.
|
|
18
|
+
example-web:
|
|
19
|
+
username_env: QP_EXAMPLE_USER
|
|
20
|
+
password_env: QP_EXAMPLE_PASS
|
|
21
|
+
tc_vocabulary:
|
|
22
|
+
# Optional overrides for the step-verb keywords qp-run recognises (defaults are English + Korean).
|
|
23
|
+
click: []
|
|
24
|
+
input: []
|
|
25
|
+
navigate: []
|
|
26
|
+
verify: []
|
|
27
|
+
masking:
|
|
28
|
+
# Extra evidence-masking regexes (appended to the built-in email/token/password set).
|
|
29
|
+
extra_patterns: []
|
|
@@ -0,0 +1,216 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# qp_tools.sh — the MECHANICAL half of fh-qp (QP · Quality Platform).
|
|
3
|
+
#
|
|
4
|
+
# The four qp-* skills are prose; a prose rule is muscle (a strong model carrying it), not skeleton.
|
|
5
|
+
# Everything a floor-tier session must NOT be trusted to "just do" lives here as a typed check with a
|
|
6
|
+
# closed exit enum, so a report can be read without trusting the narrator:
|
|
7
|
+
#
|
|
8
|
+
# target-class <url|app:NAME> [--profile FILE] → PUBLIC | PROFILE_REQUIRED | PROFILE_OK | UNKNOWN
|
|
9
|
+
# adapter-probe --need web|desktop --tools "a,b,c" → ADAPTER=<name> | HARNESS_ERROR
|
|
10
|
+
# mask <in> <out> → MASKED (residue 0) | RESIDUE
|
|
11
|
+
# surface-reach <evidence.tsv> → REACHED | PARTIAL | NOT_REACHED | UNMEASURED
|
|
12
|
+
# mtm-check <verdicts.tsv> → OK | INVALID
|
|
13
|
+
# run-verbs <verdicts.tsv> → OK (≥1 state-changing verb) | VERIFY_ONLY
|
|
14
|
+
# screen-id <snapshot-file> → 12-hex stable id of a DOM/pixel observation
|
|
15
|
+
#
|
|
16
|
+
# Exit enum (shared): 0 = pass/positive · 4 = PROFILE_REQUIRED · 5 = negative-but-typed (RESIDUE ·
|
|
17
|
+
# NOT_REACHED · PARTIAL · VERIFY_ONLY · INVALID) · 10 = HARNESS_ERROR / UNMEASURED (fail-closed:
|
|
18
|
+
# never read 10 as "clean" or "reached").
|
|
19
|
+
#
|
|
20
|
+
# Domain constants: ZERO. Every host/app name comes from the caller or the profile file.
|
|
21
|
+
# bash 3.2 compatible (macOS default). No python required.
|
|
22
|
+
set -uo pipefail
|
|
23
|
+
|
|
24
|
+
usage() { sed -n '2,20p' "$0" >&2; exit 10; }
|
|
25
|
+
[ $# -ge 1 ] || usage
|
|
26
|
+
CMD="$1"; shift
|
|
27
|
+
|
|
28
|
+
# ── helpers ────────────────────────────────────────────────────────────────────
|
|
29
|
+
_host_of() { # strip scheme, path, port, creds
|
|
30
|
+
printf '%s' "$1" | sed -E 's#^[a-zA-Z][a-zA-Z0-9+.-]*://##; s#^[^/@]*@##; s#[/?].*$##; s#:[0-9]+$##' | tr 'A-Z' 'a-z'
|
|
31
|
+
}
|
|
32
|
+
_is_private_host() { # RFC1918 / loopback / link-local / non-public suffix / bare name
|
|
33
|
+
local h="$1"
|
|
34
|
+
case "$h" in
|
|
35
|
+
localhost|127.*|10.*|192.168.*|169.254.*|0.0.0.0|::1) return 0 ;;
|
|
36
|
+
172.1[6-9].*|172.2[0-9].*|172.3[01].*) return 0 ;;
|
|
37
|
+
*.local|*.internal|*.lan|*.intra|*.home|*.test|*.localhost|*.private) return 0 ;;
|
|
38
|
+
esac
|
|
39
|
+
case "$h" in *.*) return 1 ;; *) return 0 ;; esac # bare hostname (no dot) → not public
|
|
40
|
+
}
|
|
41
|
+
_profile_lists() { # $1 = profile file, $2 = needle (host or app name). YAML-lite: grep the value.
|
|
42
|
+
[ -f "$1" ] || return 1
|
|
43
|
+
/usr/bin/grep -qiF -- "$2" "$1"
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
# ── target-class ───────────────────────────────────────────────────────────────
|
|
47
|
+
cmd_target_class() {
|
|
48
|
+
local target="" profile=""
|
|
49
|
+
while [ $# -gt 0 ]; do case "$1" in --profile) profile="${2:-}"; shift 2 ;; *) target="$1"; shift ;; esac; done
|
|
50
|
+
[ -n "$target" ] || { echo "target-class: missing target" >&2; exit 10; }
|
|
51
|
+
case "$target" in
|
|
52
|
+
app:*)
|
|
53
|
+
local app="${target#app:}"
|
|
54
|
+
if [ -n "$profile" ] && _profile_lists "$profile" "$app"; then echo "PROFILE_OK kind=desktop app=$app"; exit 0; fi
|
|
55
|
+
echo "PROFILE_REQUIRED kind=desktop app=$app reason=desktop-targets-always-need-a-profile"; exit 4 ;;
|
|
56
|
+
http://*|https://*)
|
|
57
|
+
local h; h="$(_host_of "$target")"
|
|
58
|
+
[ -n "$h" ] || { echo "UNKNOWN reason=no-host"; exit 10; }
|
|
59
|
+
if _is_private_host "$h"; then
|
|
60
|
+
if [ -n "$profile" ] && _profile_lists "$profile" "$h"; then echo "PROFILE_OK kind=web host=$h"; exit 0; fi
|
|
61
|
+
echo "PROFILE_REQUIRED kind=web host=$h reason=non-public-host"; exit 4
|
|
62
|
+
fi
|
|
63
|
+
echo "PUBLIC kind=web host=$h"; exit 0 ;;
|
|
64
|
+
*) echo "UNKNOWN reason=unrecognised-target-form (want http(s)://… or app:NAME)"; exit 10 ;;
|
|
65
|
+
esac
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
# ── adapter-probe ──────────────────────────────────────────────────────────────
|
|
69
|
+
# bash cannot introspect the session's MCP servers; the SESSION passes the tool names it actually has.
|
|
70
|
+
# Omitting --tools is HARNESS_ERROR: unknown is not present.
|
|
71
|
+
cmd_adapter_probe() {
|
|
72
|
+
local need="" tools=""
|
|
73
|
+
while [ $# -gt 0 ]; do case "$1" in --need) need="${2:-}"; shift 2 ;; --tools) tools="${2:-}"; shift 2 ;; *) shift ;; esac; done
|
|
74
|
+
[ "$need" = web ] || [ "$need" = desktop ] || { echo "HARNESS_ERROR reason=--need must be web|desktop"; exit 10; }
|
|
75
|
+
[ -n "$tools" ] || { echo "HARNESS_ERROR reason=tool-list-not-supplied (unknown ≠ present)"; exit 10; }
|
|
76
|
+
local t; for t in $(printf '%s' "$tools" | tr ',' ' '); do
|
|
77
|
+
case "$need:$t" in
|
|
78
|
+
web:mcp__playwright__browser_navigate) echo "ADAPTER=playwright-mcp evidence=dom"; exit 0 ;;
|
|
79
|
+
web:mcp__claude-in-chrome__navigate) echo "ADAPTER=claude-in-chrome evidence=dom"; exit 0 ;;
|
|
80
|
+
desktop:mcp__computer-use__screenshot) echo "ADAPTER=computer-use-mcp evidence=pixel"; exit 0 ;;
|
|
81
|
+
esac
|
|
82
|
+
done
|
|
83
|
+
echo "HARNESS_ERROR reason=no-$need-adapter-among-supplied-tools"; exit 10
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
# ── mask ───────────────────────────────────────────────────────────────────────
|
|
87
|
+
# Record-layer masking (the typing path still sends the real value). Residue 0 is asserted AFTER writing.
|
|
88
|
+
_MASK_EMAIL='[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}'
|
|
89
|
+
_MASK_JWT='eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}'
|
|
90
|
+
_MASK_BEARER='[Bb]earer[[:space:]]+[A-Za-z0-9._~+/=-]{16,}'
|
|
91
|
+
_MASK_KEYLIKE='(sk|pk|ghp|gho|xox[abp]|AKIA)[-_][A-Za-z0-9_-]{12,}'
|
|
92
|
+
_MASK_HEX='[A-Fa-f0-9]{40,}'
|
|
93
|
+
_MASK_PWKV='([Pp]ass(word|wd)?|[Ss]ecret|[Tt]oken)[[:space:]]*[:=][[:space:]]*"?[^"[:space:],}]{4,}'
|
|
94
|
+
cmd_mask() {
|
|
95
|
+
local in="${1:-}" out="${2:-}"
|
|
96
|
+
[ -f "$in" ] || { echo "HARNESS_ERROR reason=input-missing"; exit 10; }
|
|
97
|
+
[ -n "$out" ] || { echo "HARNESS_ERROR reason=output-missing"; exit 10; }
|
|
98
|
+
sed -E \
|
|
99
|
+
-e "s/$_MASK_JWT/__REDACTED_TOKEN__/g" \
|
|
100
|
+
-e "s/$_MASK_BEARER/Bearer __REDACTED_TOKEN__/g" \
|
|
101
|
+
-e "s/$_MASK_KEYLIKE/__REDACTED_TOKEN__/g" \
|
|
102
|
+
-e "s/$_MASK_HEX/__REDACTED_TOKEN__/g" \
|
|
103
|
+
-e "s/$_MASK_PWKV/\1=__REDACTED_SECRET__/g" \
|
|
104
|
+
-e "s/$_MASK_EMAIL/__REDACTED_EMAIL__/g" \
|
|
105
|
+
"$in" > "$out" || { echo "HARNESS_ERROR reason=sed-failed"; exit 10; }
|
|
106
|
+
local residue
|
|
107
|
+
# Strip our own markers before re-scanning — the Bearer pattern matches "__REDACTED_TOKEN__" itself
|
|
108
|
+
# (18 word-chars). Known-pair caught this on first run: an instrument matching its own output is
|
|
109
|
+
# [[feedback_lane_vocabulary_blind_to_its_own_fix]] in miniature.
|
|
110
|
+
residue=$(sed -E 's/__REDACTED_[A-Z]+__//g' "$out" | /usr/bin/grep -cE "$_MASK_EMAIL|$_MASK_JWT|$_MASK_BEARER|$_MASK_HEX" 2>/dev/null); residue="${residue:-0}"
|
|
111
|
+
if [ "$residue" -eq 0 ]; then echo "MASKED residue=0 out=$out"; exit 0; fi
|
|
112
|
+
echo "RESIDUE residue=$residue out=$out"; exit 5
|
|
113
|
+
}
|
|
114
|
+
|
|
115
|
+
# ── surface-reach ──────────────────────────────────────────────────────────────
|
|
116
|
+
# evidence.tsv: tc_id <TAB> step <TAB> screen_hash (one row per observed screen; hash = any stable id)
|
|
117
|
+
# Entry screen = the modal hash across the batch. A TC is "beyond entry" iff any of its hashes ≠ entry.
|
|
118
|
+
# Denominator = ALL tc_ids in the file. An empty file is UNMEASURED (10), never NOT_REACHED.
|
|
119
|
+
cmd_surface_reach() {
|
|
120
|
+
local f="${1:-}"
|
|
121
|
+
[ -f "$f" ] || { echo "UNMEASURED reason=evidence-missing"; exit 10; }
|
|
122
|
+
[ -s "$f" ] || { echo "UNMEASURED reason=evidence-empty (empty set is not reached)"; exit 10; }
|
|
123
|
+
LC_ALL=C awk -F'\t' '
|
|
124
|
+
NF < 3 || $1 ~ /^#/ { next }
|
|
125
|
+
{ tc[$1]=1; n_rows++; cnt[$3]++; seen[$1 SUBSEP $3]=1 }
|
|
126
|
+
END {
|
|
127
|
+
if (n_rows == 0) { print "UNMEASURED reason=no-rows"; exit 10 }
|
|
128
|
+
best=""; bestc=-1
|
|
129
|
+
for (h in cnt) if (cnt[h] > bestc) { bestc=cnt[h]; best=h }
|
|
130
|
+
total=0; beyond=0
|
|
131
|
+
for (t in tc) { total++; b=0; for (k in seen) { split(k, p, SUBSEP); if (p[1]==t && p[2]!=best) b=1 } beyond+=b }
|
|
132
|
+
st = (beyond==total) ? "REACHED" : (beyond==0 ? "NOT_REACHED" : "PARTIAL")
|
|
133
|
+
printf "%s tcs_beyond_entry=%d tcs_total=%d entry_screen=%s\n", st, beyond, total, best
|
|
134
|
+
exit (st=="REACHED") ? 0 : 5
|
|
135
|
+
}' "$f"
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
# ── mtm-check ──────────────────────────────────────────────────────────────────
|
|
139
|
+
# verdicts.tsv: first non-comment line MUST be #mtm: ACTIVE|UNAVAILABLE|FAILED|DISABLED
|
|
140
|
+
# rows: tc_id <TAB> status <TAB> branch <TAB> closure <TAB> verb <TAB> assertion <TAB> expected_source
|
|
141
|
+
# status ∈ PASS FAIL BLOCKED AMBIGUOUS
|
|
142
|
+
# branch ∈ AS_PLANNED DIFFERS_FROM_PLAN CODE_DIFFERS NONE
|
|
143
|
+
# closure ∈ MACHINE JUDGMENT (MACHINE requires a non-empty assertion — the record that closed it)
|
|
144
|
+
# verb ∈ navigate click input verify
|
|
145
|
+
# expected_source ∈ PLAN_DOC INVENTORY CODE HUMAN NONE — WHERE the expected value came from.
|
|
146
|
+
# The branch→source binding is the mechanical discriminator (challenger #9, governor 2026-09-05):
|
|
147
|
+
# DIFFERS_FROM_PLAN ⇒ expected_source = PLAN_DOC and #mtm: ACTIVE (no plan doc → this branch is unreachable)
|
|
148
|
+
# CODE_DIFFERS ⇒ expected_source ∈ {CODE, PLAN_DOC}
|
|
149
|
+
# AS_PLANNED ⇒ expected_source ≠ NONE
|
|
150
|
+
# NONE ⇒ status ∈ {BLOCKED, AMBIGUOUS} (a verdict with no branch is a non-verdict)
|
|
151
|
+
# Any violation → INVALID (10). The check asserts the RECORD's properties (typed · attributable ·
|
|
152
|
+
# non-vacuous), never whether the label is TRUE — that stays judgment (CLAUDE.md §Mechanization Boundary).
|
|
153
|
+
cmd_mtm_check() {
|
|
154
|
+
local f="${1:-}"
|
|
155
|
+
[ -f "$f" ] || { echo "INVALID reason=file-missing"; exit 10; }
|
|
156
|
+
local mtm; mtm=$(/usr/bin/grep -m1 -E '^#mtm:' "$f" | sed -E 's/^#mtm:[[:space:]]*//' | tr -d '[:space:]')
|
|
157
|
+
case "$mtm" in ACTIVE|UNAVAILABLE|FAILED|DISABLED) ;; *) echo "INVALID reason=mtm-state-missing-or-bad (#mtm: ACTIVE|UNAVAILABLE|FAILED|DISABLED)"; exit 10 ;; esac
|
|
158
|
+
LC_ALL=C awk -F'\t' -v mtm="$mtm" '
|
|
159
|
+
/^#/ || NF==0 { next }
|
|
160
|
+
{ rows++
|
|
161
|
+
if (NF < 7) { bad++; why="row-has-" NF "-fields-need-7"; next }
|
|
162
|
+
if ($2 !~ /^(PASS|FAIL|BLOCKED|AMBIGUOUS)$/) { bad++; why="status:" $2; next }
|
|
163
|
+
if ($3 !~ /^(AS_PLANNED|DIFFERS_FROM_PLAN|CODE_DIFFERS|NONE)$/) { bad++; why="branch:" $3; next }
|
|
164
|
+
if ($3 == "DIFFERS_FROM_PLAN" && mtm != "ACTIVE") { bad++; why="DIFFERS_FROM_PLAN-needs-mtm-ACTIVE"; next }
|
|
165
|
+
if ($4 !~ /^(MACHINE|JUDGMENT)$/) { bad++; why="closure:" $4; next }
|
|
166
|
+
if ($4 == "MACHINE" && $6 == "") { bad++; why="MACHINE-closure-without-assertion"; next }
|
|
167
|
+
if ($5 !~ /^(navigate|click|input|verify)$/) { bad++; why="verb:" $5; next }
|
|
168
|
+
if ($7 !~ /^(PLAN_DOC|INVENTORY|CODE|HUMAN|NONE)$/) { bad++; why="expected_source:" $7; next }
|
|
169
|
+
if ($3 == "DIFFERS_FROM_PLAN" && $7 != "PLAN_DOC") { bad++; why="DIFFERS_FROM_PLAN-needs-expected_source=PLAN_DOC"; next }
|
|
170
|
+
if ($3 == "CODE_DIFFERS" && $7 !~ /^(CODE|PLAN_DOC)$/) { bad++; why="CODE_DIFFERS-needs-expected_source=CODE|PLAN_DOC"; next }
|
|
171
|
+
if ($3 == "AS_PLANNED" && $7 == "NONE") { bad++; why="AS_PLANNED-with-expected_source=NONE"; next }
|
|
172
|
+
if ($3 == "NONE" && $2 !~ /^(BLOCKED|AMBIGUOUS)$/) { bad++; why="branch=NONE-only-for-BLOCKED|AMBIGUOUS"; next }
|
|
173
|
+
if ($4=="MACHINE") m++; else j++
|
|
174
|
+
}
|
|
175
|
+
END {
|
|
176
|
+
if (rows == 0) { print "INVALID reason=no-rows"; exit 10 }
|
|
177
|
+
if (bad > 0) { printf "INVALID rows=%d bad=%d first=%s\n", rows, bad, why; exit 10 }
|
|
178
|
+
printf "OK mtm=%s rows=%d machine_closed=%d judgment_left=%d\n", mtm, rows, m+0, j+0; exit 0
|
|
179
|
+
}' "$f"
|
|
180
|
+
}
|
|
181
|
+
|
|
182
|
+
# ── screen-id ──────────────────────────────────────────────────────────────────
|
|
183
|
+
# First-use finding (2026-09-05, gh-pages target): Playwright MCP accessibility snapshots carry per-navigation
|
|
184
|
+
# ref tokens (`[ref=e12]`, then `[ref=f2e12]`, `[ref=f3e12]` …) — the SAME page hashed to three different ids
|
|
185
|
+
# across three visits, so surface_reach credited two TCs for "leaving the entry screen" before they clicked.
|
|
186
|
+
# The screen id must be a CONTENT hash: strip ref/active tokens and whitespace runs before hashing.
|
|
187
|
+
# Binary files (screenshots) are hashed as-is.
|
|
188
|
+
cmd_screen_id() {
|
|
189
|
+
local f="${1:-}"
|
|
190
|
+
[ -f "$f" ] || { echo "HARNESS_ERROR reason=file-missing"; exit 10; }
|
|
191
|
+
if LC_ALL=C /usr/bin/grep -qI . "$f"; then
|
|
192
|
+
sed -E 's/\[ref=[A-Za-z0-9]+\]//g; s/\[active\]//g; s/[[:space:]]+/ /g' "$f" | shasum -a 256 | cut -c1-12
|
|
193
|
+
else
|
|
194
|
+
shasum -a 256 "$f" | cut -c1-12
|
|
195
|
+
fi
|
|
196
|
+
}
|
|
197
|
+
|
|
198
|
+
# ── run-verbs (challenger #13: verify-only evidence never exercised the runner) ─
|
|
199
|
+
cmd_run_verbs() {
|
|
200
|
+
local f="${1:-}"
|
|
201
|
+
[ -f "$f" ] || { echo "HARNESS_ERROR reason=file-missing"; exit 10; }
|
|
202
|
+
local n; n=$(LC_ALL=C awk -F'\t' '!/^#/ && NF>=5 && ($5=="click" || $5=="input"){c++} END{print c+0}' "$f")
|
|
203
|
+
if [ "$n" -ge 1 ]; then echo "OK state_changing_verbs=$n"; exit 0; fi
|
|
204
|
+
echo "VERIFY_ONLY state_changing_verbs=0 (runner not exercised)"; exit 5
|
|
205
|
+
}
|
|
206
|
+
|
|
207
|
+
case "$CMD" in
|
|
208
|
+
target-class) cmd_target_class "$@" ;;
|
|
209
|
+
adapter-probe) cmd_adapter_probe "$@" ;;
|
|
210
|
+
mask) cmd_mask "$@" ;;
|
|
211
|
+
surface-reach) cmd_surface_reach "$@" ;;
|
|
212
|
+
mtm-check) cmd_mtm_check "$@" ;;
|
|
213
|
+
run-verbs) cmd_run_verbs "$@" ;;
|
|
214
|
+
screen-id) cmd_screen_id "$@" ;;
|
|
215
|
+
*) usage ;;
|
|
216
|
+
esac
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: qp
|
|
3
|
+
description: QP (Quality Platform) router — takes a web URL or a desktop app name and runs the plan → run → regress loop (qp-plan, qp-run, qp-regress) end to end, or routes to one stage. Classifies the target (public vs profile-required), probes which adapter the session has (Playwright MCP for web, computer-use MCP for desktop), and writes every artifact with a typed engine/adapter marker. Triggers on "test this website for me", "run a QA pass on this app", "check the app end to end", "이 사이트 QA 돌려줘", "이 앱 테스트해줘", "회귀 돌려줘".
|
|
4
|
+
user-invocable: true
|
|
5
|
+
allowed-tools: ["Read", "Bash", "Write", "Glob", "Grep"]
|
|
6
|
+
model: sonnet
|
|
7
|
+
origin: chamber run fh-qa-generic (2026-09-05) — generic edition of a field QA harness's P·A·R loop; zero domain constants
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# qp — Quality Platform router
|
|
11
|
+
|
|
12
|
+
One entry point for people who want to **dynamically test a web or desktop app with a QA lens from inside Claude Code**.
|
|
13
|
+
Three stages, each its own skill, each leaving files:
|
|
14
|
+
|
|
15
|
+
| stage | skill | you get |
|
|
16
|
+
|---|---|---|
|
|
17
|
+
| **P**lan | `qp-plan` | `qp/plan/inventory.md` (surfaces the app exposes) + `qp/plan/tcs.tsv` (test cases) |
|
|
18
|
+
| **A**utomation (run) | `qp-run` | `qp/run/<ts>/verdicts.tsv` + masked evidence (DOM snapshots or screenshots) |
|
|
19
|
+
| **R**egression | `qp-regress` | `qp/regress/<ts>/surface_reach.txt` + per-TC delta vs the previous run |
|
|
20
|
+
|
|
21
|
+
Glossary (one line each, because a first-time reader asked): **TC** = test case · **inventory** = the list of screens/routes/menus the app exposes · **MTM** = each finding is labeled *as planned* / *differs from the plan document* / *code differs* · **precondition** = what must already be true before a step runs (logged in, page loaded), checked not assumed — a failure before it is met is `BLOCKED`, not `FAIL` · **surface_reach** = how many TCs actually left the entry screen · **engine** = which machinery drove the run (see §Engine).
|
|
22
|
+
|
|
23
|
+
## Step 0 — classify the target (mechanical, never by eye)
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
bash plugins/fh-qp/scripts/qp_tools.sh target-class <https://… | app:NAME> [--profile <file>]
|
|
27
|
+
```
|
|
28
|
+
| result | rc | what you do |
|
|
29
|
+
|---|---|---|
|
|
30
|
+
| `PUBLIC` | 0 | proceed |
|
|
31
|
+
| `PROFILE_OK` | 0 | proceed with the profile |
|
|
32
|
+
| `PROFILE_REQUIRED` | 4 | **stop.** Say: *"this target is not public; QP needs a profile file — copy `plugins/fh-qp/qp_profile.example.yaml` to a gitignored path, fill the host/app, and re-run with `--profile`."* Do not improvise a run against a private host. |
|
|
33
|
+
| `UNKNOWN` | 10 | stop, report the target form QP accepts |
|
|
34
|
+
|
|
35
|
+
## Step 1 — probe the adapter (mechanical; unknown ≠ present)
|
|
36
|
+
|
|
37
|
+
List the MCP tool names this session actually has (from your tool list — do not guess), then:
|
|
38
|
+
```
|
|
39
|
+
bash plugins/fh-qp/scripts/qp_tools.sh adapter-probe --need web|desktop --tools "<comma-separated tool names>"
|
|
40
|
+
```
|
|
41
|
+
`ADAPTER=… evidence=dom|pixel` → proceed and **write that line into every artifact header**.
|
|
42
|
+
`HARNESS_ERROR` (rc 10) → **stop.** Write `qp/HARNESS_ERROR.txt` with the reason and tell the user which MCP to connect (Playwright MCP for web · computer-use MCP for desktop). This is not a pass and not a skip.
|
|
43
|
+
|
|
44
|
+
## Engine — typed capability vs MCP fallback (form C)
|
|
45
|
+
|
|
46
|
+
- If a QA harness has **registered** a typed capability whose id ends in `:par` (check `.claude/capabilities/**/*.cap` and `bash scripts/cluster_capability_scan.sh` when in an FH hub), call its `entry` and merge constraints **strictest-wins** (`capability_composition_contract.md`). Engine marker: `engine=capability:<id>`.
|
|
47
|
+
- Otherwise drive the target yourself through the adapter from Step 1. Engine marker: `engine=mcp-fallback adapter=<name> evidence=dom|pixel`.
|
|
48
|
+
- **As of 2026-09-05 no such capability is registered anywhere** — the fallback is the only path that exists. If the user explicitly asks for the capability engine (`--engine capability`) and none is registered → `HARNESS_ERROR reason=capability-not-registered`, never a silent fallback.
|
|
49
|
+
- Desktop (`evidence=pixel`) is the **weaker edition**: no DOM, so `verify` closes only by screenshot text you can read, and surface_reach hashes screenshots, not DOM. Say so in the report header.
|
|
50
|
+
|
|
51
|
+
## Step 2 — run the stages
|
|
52
|
+
|
|
53
|
+
`qp <target>` = plan → run → regress in order, stopping at the first `HARNESS_ERROR`. `qp plan|run|regress <target>` = one stage. Each stage skill carries its own Done When.
|
|
54
|
+
|
|
55
|
+
## Done When
|
|
56
|
+
1. Target classified with a `target-class` line recorded in `qp/plan/inventory.md` header · **[mandatory-pass]** — rc is the check.
|
|
57
|
+
2. Adapter probed and `ADAPTER=` or `HARNESS_ERROR` written to a file · **[mandatory-pass]** — file exists and matches the probe output.
|
|
58
|
+
3. Every artifact header carries `engine=` and `evidence=` · **[measured]** — `grep -L 'engine=' qp/**/*.md qp/**/*.tsv` is empty.
|
|
59
|
+
4. Stages chained in P→A→R order with no stage skipped silently · **[judged — pairing: the lane `scripts/test_fh_qp_lanes.sh` and the per-stage Done When files]**.
|
|
60
|
+
|
|
61
|
+
## Independently executable
|
|
62
|
+
Needs only `bash` + the session's MCP adapter. No other FH skill is required. Inside an FH hub the capability scan is optional.
|
|
63
|
+
|
|
64
|
+
## Not this skill
|
|
65
|
+
Mobile (deferred). Verifying FH itself (use FH's own gates). Fixing the app.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: qp-plan
|
|
3
|
+
description: Prepare stage of QP — builds a surface inventory of a web page or desktop app (routes, menus, forms, dialogs), checks it is MECE (no overlap, nothing missing that is visible), and designs test cases as a TSV with preconditions and expected values whose SOURCE is named. Works without a spec document (then expected values come from the inventory, not a plan). Triggers on "what should I test on this page", "make a test plan for this app", "list the screens of this app", "테스트 케이스 뽑아줘", "이 화면 인벤토리 만들어줘".
|
|
4
|
+
user-invocable: true
|
|
5
|
+
allowed-tools: ["Read", "Bash", "Write", "Glob", "Grep"]
|
|
6
|
+
model: sonnet
|
|
7
|
+
origin: chamber run fh-qa-generic (2026-09-05)
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# qp-plan — Prepare (inventory → MECE → TCs)
|
|
11
|
+
|
|
12
|
+
Input: a target that passed `qp_tools.sh target-class` (PUBLIC or PROFILE_OK) and an adapter line from `adapter-probe`. Optional: a plan/spec document path (`--plan <file>`). Optional profile.
|
|
13
|
+
|
|
14
|
+
## 1. Surface inventory — observe, do not guess
|
|
15
|
+
Open the entry route with the adapter (web: navigate + accessibility snapshot; desktop: screenshot). Record every **navigable surface** you can see: links/routes, menu items, buttons that open a dialog, forms, tabs. One row each:
|
|
16
|
+
|
|
17
|
+
```
|
|
18
|
+
| id | kind (route·menu·dialog·form·tab) | label as shown | how reached (from which surface, which action) | observed? (yes/no) |
|
|
19
|
+
```
|
|
20
|
+
Rules: ⓐ **do not invent routes** — a surface is listed only if you saw its trigger; ⓑ parameterised routes (`/item/:id`) are listed as `ROUTE_PARAMETERIZED`, not filled with a made-up id; ⓒ if the entry route did not load → stop with `HARNESS_ERROR reason=entry-unreachable`. Write `qp/plan/inventory.md` with the header lines `target-class: …` (the tool's literal output), `ADAPTER=… evidence=dom|pixel` (the probe's literal output), `engine=mcp-fallback` (or `engine=capability:<id>` only if the `qp` router found a registered capability — never a browser name; a floor-tier sim wrote `engine=chromium` when this line did not say so, 2026-09-05), `plan_doc: <path|NONE>`.
|
|
21
|
+
|
|
22
|
+
## 2. MECE check
|
|
23
|
+
Walk the inventory once more against the snapshot: any visible navigable element not in the list → add it (**missing**); any two rows reaching the same surface → merge (**overlap**). Write the count of each fix under `## MECE` in inventory.md. Zero fixes is a legitimate value; an unstated count is not.
|
|
24
|
+
|
|
25
|
+
## 3. Test-case design — `qp/plan/tcs.tsv`
|
|
26
|
+
Tab-separated, header row:
|
|
27
|
+
```
|
|
28
|
+
tc_id surface_id precondition step_verb step_target expected expected_source priority
|
|
29
|
+
```
|
|
30
|
+
- `step_verb` ∈ `navigate click input verify` (one verb per row; a multi-step TC is several rows sharing `tc_id`).
|
|
31
|
+
- `expected_source` ∈ `PLAN_DOC INVENTORY CODE HUMAN` — **where the expected value came from**. With no plan document every row is `INVENTORY` (or `HUMAN` if the user told you). This field is what lets `qp-run` label a finding *differs from plan* vs *code differs*; a TC without it cannot be run.
|
|
32
|
+
- `precondition` is what must be true before step 1 (e.g. `at:/`, `logged-in`). Write `none` explicitly when there is none.
|
|
33
|
+
- **At least one TC must carry a state-changing verb (`click` or `input`)** — a verify-only plan never exercises the runner (`qp_tools.sh run-verbs` enforces this on the run output).
|
|
34
|
+
- Minimum: inventory ≥ 5 surfaces, TCs ≥ 3 for a first pass; say when the target is too small to reach that (a one-page site can be).
|
|
35
|
+
|
|
36
|
+
## Done When
|
|
37
|
+
1. `qp/plan/inventory.md` exists with the four header lines and ≥1 observed surface · **[mandatory-pass]**.
|
|
38
|
+
2. `qp/plan/tcs.tsv` parses: every row has 8 fields, `step_verb` and `expected_source` in their enums, ≥1 click/input row · **[measured]** — `awk -F'\t' 'NF!=8' tcs.tsv` prints nothing; `run-verbs` on the later verdicts is OK.
|
|
39
|
+
3. No surface in the inventory was invented (each has `observed? yes` or a `ROUTE_PARAMETERIZED` mark) · **[judged — pairing: a second session re-opens the entry route and diffs the inventory; the lane's residency/zero-constant grep on the plan files]**.
|
|
40
|
+
|
|
41
|
+
## Independently executable
|
|
42
|
+
Yes — bash + adapter. Does not need `qp-run`.
|