@topy-ai/maggie 0.7.13 → 0.7.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +28 -6
- package/README.zh-TW.md +8 -3
- package/bin/maggie.js +2 -0
- package/bundled-skills/README.md +1 -0
- package/bundled-skills/catalog.json +4 -0
- package/bundled-skills/maggie-qa-workflow/SKILL.md +102 -0
- package/bundled-tools/clis/maggie_qa_workflow.py +367 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -58,7 +58,23 @@ Then invoke the installed skills from your coding agent, for example:
|
|
|
58
58
|
|
|
59
59
|
Maggie keeps the existing project foundation and asks for decisions before
|
|
60
60
|
shared routes, analytics, or publishing boundaries change. The current
|
|
61
|
-
package ships
|
|
61
|
+
package ships 19 installable skills and a local-first MaggieDash foundation.
|
|
62
|
+
|
|
63
|
+
For repeatable browser QA, create a project-owned scenario manifest and record
|
|
64
|
+
the test, fix, and retest lifecycle:
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
maggie qa start --project . \
|
|
68
|
+
--scenario-file .maggie/scenario-manifest.json \
|
|
69
|
+
--environment local --base-url http://localhost:4321
|
|
70
|
+
maggie qa record --project . --run <run-id> \
|
|
71
|
+
--scenario HOME-001 --phase test --status pass \
|
|
72
|
+
--summary "Homepage smoke passed" --evidence .maggie/qa/home.png
|
|
73
|
+
maggie qa summary --project . --run <run-id>
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
The QA workflow stores secret-free run state under `.maggie/qa-runs/` and
|
|
77
|
+
keeps project-specific scenarios and evidence outside the npm package.
|
|
62
78
|
|
|
63
79
|
For Google integrations, validate a redacted provider matrix before reporting
|
|
64
80
|
access. The command fails closed on unknown scopes, missing Ads prerequisites,
|
|
@@ -114,6 +130,7 @@ maggie clone-to-template ... # URL → validated marketplace template
|
|
|
114
130
|
maggie marketplace ... # catalog and on-demand template workflow
|
|
115
131
|
maggie memory ... # confirmed preferences and lessons
|
|
116
132
|
maggie feedback ... # redact, preview, submit, list
|
|
133
|
+
maggie qa ... # scenario browser QA, fix/retest, release gate
|
|
117
134
|
maggie localization ... # plan, validate, review, publish, stale
|
|
118
135
|
maggie service ... # import, sync, generate, validate
|
|
119
136
|
maggie seo performance ... # sampled PageSpeed/CWV report and baseline
|
|
@@ -220,8 +237,8 @@ artifact schemas.
|
|
|
220
237
|
Recommended upgrade sequence for the current release:
|
|
221
238
|
|
|
222
239
|
```bash
|
|
223
|
-
npx @topy-ai/maggie@0.7.
|
|
224
|
-
npx @topy-ai/maggie@0.7.
|
|
240
|
+
npx @topy-ai/maggie@0.7.14 update --project . --force
|
|
241
|
+
npx @topy-ai/maggie@0.7.14 cleanup --project .
|
|
225
242
|
```
|
|
226
243
|
|
|
227
244
|
Maintainers should pass npm credentials through the repository helper, never
|
|
@@ -231,7 +248,10 @@ as a command-line argument:
|
|
|
231
248
|
node scripts/publish-npm.mjs --maggie-env-file ../.env
|
|
232
249
|
```
|
|
233
250
|
|
|
234
|
-
The 0.7.
|
|
251
|
+
The 0.7.14 workflow adds the general `maggie-qa-workflow` skill and `maggie qa`
|
|
252
|
+
CLI for scenario manifests, secret-free browser evidence metadata, test/fix/
|
|
253
|
+
retest lifecycle, adjacent regression checks, and explicit release gates. The
|
|
254
|
+
0.7.13 workflow adds field-aware section fan-out and locale coverage,
|
|
235
255
|
sibling-copy/media checks, disjoint page inventory, binding validation,
|
|
236
256
|
idempotency and database-target identity gates, full W3C sitemap lastmod
|
|
237
257
|
validation, and the accepted `X-Robots-Tag: noindex` response contract. The
|
|
@@ -484,6 +504,7 @@ python3 tools/clis/maggie_design.py rebrand \
|
|
|
484
504
|
| `maggie-memory` | Persist confirmed preferences, conventions, lessons, and errors |
|
|
485
505
|
| `maggie-content-localization` | Manage locale-aware translation, review, provenance, stale state, and publication gates |
|
|
486
506
|
| `maggie-feedback` | Collect redacted feedback drafts and explicitly submit them to the NoBlox feedback endpoint |
|
|
507
|
+
| `maggie-qa-workflow` | Run scenario-based browser QA with evidence, fix/retest lifecycle, and release gates |
|
|
487
508
|
| `maggie-auth-reference` | Generate and validate traditional email/password auth with secure server-side sessions |
|
|
488
509
|
| `maggie-blog` | Run a provider-neutral blog lifecycle with stable identity, topics, feeds, settings, and rollback |
|
|
489
510
|
|
|
@@ -528,12 +549,13 @@ repairs with `maggie dash sections validate`, `fanout-validate`,
|
|
|
528
549
|
`maggie dash inventory` to classify published pages once into disjoint kinds.
|
|
529
550
|
See the [quality contract examples](https://github.com/TOPY-AI-LTD/ai-cmo-skills/blob/main/contracts/maggiedash/quality-contracts.md).
|
|
530
551
|
|
|
531
|
-
The package includes all
|
|
552
|
+
The package includes all 19 installable skills: `maggie-blog-bootstrap`,
|
|
532
553
|
`maggie-dash`, `maggie-clone`, `maggie-clone-to-template`, `maggie-marketplace`,
|
|
533
554
|
`maggie-template`, `maggie-design`, `maggie-ops`, `maggie-deployment`,
|
|
534
555
|
`maggie-project-context`, `maggie-seo-geo`, `maggie-social-share`,
|
|
535
556
|
`maggie-service-booking`, `maggie-memory`, `maggie-content-localization`,
|
|
536
|
-
`maggie-feedback`, `maggie-
|
|
557
|
+
`maggie-feedback`, `maggie-qa-workflow`, `maggie-auth-reference`, and
|
|
558
|
+
`maggie-blog`. Use the stable
|
|
537
559
|
commands below after installation:
|
|
538
560
|
|
|
539
561
|
```bash
|
package/README.zh-TW.md
CHANGED
|
@@ -2,13 +2,13 @@
|
|
|
2
2
|
|
|
3
3
|
[English README](README.md) · [完整繁體中文 README](https://github.com/TOPY-AI-LTD/ai-cmo-skills/blob/main/README.zh-TW.md)
|
|
4
4
|
|
|
5
|
-
`@topy-ai/maggie` 提供
|
|
5
|
+
`@topy-ai/maggie` 提供 19 個可安裝的 AI website workflow skills,支援
|
|
6
6
|
Codex、Claude Code 與相容的 coding agents。
|
|
7
7
|
|
|
8
8
|
## 安裝
|
|
9
9
|
|
|
10
10
|
```bash
|
|
11
|
-
npx @topy-ai/maggie@0.7.
|
|
11
|
+
npx @topy-ai/maggie@0.7.14 init --agent all
|
|
12
12
|
npx @topy-ai/maggie doctor --project .
|
|
13
13
|
```
|
|
14
14
|
|
|
@@ -50,8 +50,13 @@ maggie dash inventory --pages-file .maggie/published-pages.json
|
|
|
50
50
|
# Migration target identity (never prints or stores a database URL)
|
|
51
51
|
maggie migration identity --identity-file .maggie/db-identity.json \
|
|
52
52
|
--expected-file .maggie/service-db-identity.json
|
|
53
|
+
|
|
54
|
+
# Scenario browser QA
|
|
55
|
+
maggie qa start --project . --scenario-file .maggie/scenario-manifest.json \
|
|
56
|
+
--environment local --base-url http://localhost:4321
|
|
57
|
+
maggie qa summary --project . --run <run-id>
|
|
53
58
|
```
|
|
54
59
|
|
|
55
|
-
完整中文說明、
|
|
60
|
+
完整中文說明、19 個 skills 清單和 roadmap:
|
|
56
61
|
[繁體中文 README](https://github.com/TOPY-AI-LTD/ai-cmo-skills/blob/main/README.zh-TW.md)
|
|
57
62
|
· [Roadmap](https://github.com/TOPY-AI-LTD/ai-cmo-skills/blob/main/ROADMAP.md)
|
package/bin/maggie.js
CHANGED
|
@@ -114,6 +114,7 @@ Usage:
|
|
|
114
114
|
maggie localization <extract|plan|generate|preview|validate|review|publish|stale|glossary> [options]
|
|
115
115
|
maggie seo performance|images|sitemap [options] (sitemap supports strict validate and agent-files)
|
|
116
116
|
maggie feedback <collect|preview|submit|list> [options]
|
|
117
|
+
maggie qa <start|record|summary|export> [options]
|
|
117
118
|
maggie site-audit URL [--crawl] [--access-log FILE] [--require-sitemap-request] [--languages en-GB,es-MX,ja-JP] [--check-hreflang]
|
|
118
119
|
maggie site-audit URL --crawl --save-baseline FILE --reviewer NAME
|
|
119
120
|
maggie site-audit URL --crawl --baseline FILE
|
|
@@ -406,6 +407,7 @@ try {
|
|
|
406
407
|
else if (command === "memory") workflowCli("maggie_memory.py", args);
|
|
407
408
|
else if (command === "localization") workflowCli("maggie_localization.py", args);
|
|
408
409
|
else if (command === "feedback") workflowCli("maggie_feedback.py", args);
|
|
410
|
+
else if (command === "qa") workflowCli("maggie_qa_workflow.py", args);
|
|
409
411
|
else if (command === "site-audit") workflowCli("site_audit.py", args);
|
|
410
412
|
else if (command === "browser-audit") workflowCli("maggie_browser_audit.py", args);
|
|
411
413
|
else if (command === "verification") workflowCli("maggie_verification.py", args);
|
package/bundled-skills/README.md
CHANGED
|
@@ -20,6 +20,7 @@ Skills are the agent-facing workflows. They compose with the tools in
|
|
|
20
20
|
| `maggie-memory` | Persist confirmed preferences, project conventions, lessons, and error history across skill runs | local `.maggie/memory/` state |
|
|
21
21
|
| `maggie-content-localization` | Plan, validate, review, publish, and age localized content with market, locale, provenance, and translation safeguards | localization contract, locale CLI |
|
|
22
22
|
| `maggie-feedback` | Collect, redact, review, and explicitly submit feedback from skill runs | feedback CLI, hosted endpoint, GitHub Issue Forms |
|
|
23
|
+
| `maggie-qa-workflow` | Run scenario-based browser QA with evidence, fix/retest lifecycle, and release gates | browser adapter, QA workflow CLI, scenario manifest |
|
|
23
24
|
|
|
24
25
|
Read only the selected skill and its linked references for a task. Do not load
|
|
25
26
|
all skills as one undifferentiated prompt.
|
|
@@ -58,6 +58,10 @@
|
|
|
58
58
|
"name": "maggie-project-context",
|
|
59
59
|
"description": "Sync a site's safe AI CMO Project, selected brand voice, site settings, and CTA context into a local generated context file. Use when connecting a vibe-coded blog or existing website to AI CMO, refreshing brand context, or diagnosing missing CTA/project data."
|
|
60
60
|
},
|
|
61
|
+
{
|
|
62
|
+
"name": "maggie-qa-workflow",
|
|
63
|
+
"description": "Run scenario-based browser QA with explicit evidence, fix/retest lifecycle, and release-gate decisions for web projects."
|
|
64
|
+
},
|
|
61
65
|
{
|
|
62
66
|
"name": "maggie-seo-geo",
|
|
63
67
|
"description": "Plan, audit, create, rewrite, and measure content for AI CMO's paid SEO and GEO workflow. Use for topic opportunities, AI visibility, technical SEO, extractable article structure, sitemap-based rewrites, GSC readback, or SEO/GEO client reports."
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: maggie-qa-workflow
|
|
3
|
+
description: Run scenario-based browser QA with explicit evidence, fix/retest lifecycle, and release-gate decisions for web projects.
|
|
4
|
+
metadata:
|
|
5
|
+
version: 1.0.0
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Maggie QA Workflow
|
|
9
|
+
|
|
10
|
+
Use this skill when a web project needs repeatable user-scenario testing that
|
|
11
|
+
connects browser evidence to a fix, a same-scenario retest, adjacent regression
|
|
12
|
+
checks, and an explicit Pass/Blocked/Fail release decision. It is framework
|
|
13
|
+
neutral and does not replace the project's browser-control or test tools.
|
|
14
|
+
Follow the shared [memory hook](../../references/memory-hook.md) before and
|
|
15
|
+
after the run; memory and feedback remain bounded by the privacy rules below.
|
|
16
|
+
|
|
17
|
+
## Prepare a scenario manifest
|
|
18
|
+
|
|
19
|
+
Create `.maggie/scenario-manifest.json` or pass another JSON file with a
|
|
20
|
+
non-empty `scenarios` array. Each scenario needs a unique `id`; useful fields
|
|
21
|
+
include `title`, `group`, `priority`, `routes`, `persona`, `auth`, `browser`,
|
|
22
|
+
and `viewport`. Keep project-specific scenarios in the project; do not ship
|
|
23
|
+
them in the Maggie package.
|
|
24
|
+
|
|
25
|
+
## Start and record a run
|
|
26
|
+
|
|
27
|
+
The CLI stores only secret-free run state under `.maggie/qa-runs/`. Screenshots
|
|
28
|
+
and logs stay in the project and are represented by relative paths and hashes.
|
|
29
|
+
Use a real browser adapter for visible interaction, computed layout,
|
|
30
|
+
console/network evidence, final URLs, and response status:
|
|
31
|
+
|
|
32
|
+
```bash
|
|
33
|
+
maggie qa start --project . \
|
|
34
|
+
--scenario-file .maggie/scenario-manifest.json \
|
|
35
|
+
--run-id local-qa-001 --environment local \
|
|
36
|
+
--base-url http://localhost:4321 --browser chrome \
|
|
37
|
+
--commit "$(git rev-parse --short HEAD)"
|
|
38
|
+
|
|
39
|
+
maggie qa record --project . --run local-qa-001 \
|
|
40
|
+
--scenario HOME-001 --phase test --status pass \
|
|
41
|
+
--summary "Homepage navigation and consent passed" \
|
|
42
|
+
--evidence .maggie/qa-runs/local-qa-001/home.png
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
For a failure, record expected behavior, actual behavior, and a stable error
|
|
46
|
+
fingerprint. A failed scenario cannot be silently changed to Pass:
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
maggie qa record --project . --run local-qa-001 \
|
|
50
|
+
--scenario SEARCH-001 --phase test --status fail \
|
|
51
|
+
--summary "Search dialog did not open" \
|
|
52
|
+
--expected "Selecting a result opens its detail page" \
|
|
53
|
+
--actual "The click leaves the dialog open" \
|
|
54
|
+
--error-fingerprint search-dialog-click-stale \
|
|
55
|
+
--evidence .maggie/qa-runs/local-qa-001/search-console.txt
|
|
56
|
+
|
|
57
|
+
maggie qa record --project . --run local-qa-001 \
|
|
58
|
+
--scenario SEARCH-001 --phase fix \
|
|
59
|
+
--resolution "Guarded the dialog transition after the selected result is resolved" \
|
|
60
|
+
--validation "focused browser check and project build passed"
|
|
61
|
+
|
|
62
|
+
maggie qa record --project . --run local-qa-001 \
|
|
63
|
+
--scenario SEARCH-001 --phase retest --status pass \
|
|
64
|
+
--summary "Search result opens the detail page" \
|
|
65
|
+
--evidence .maggie/qa-runs/local-qa-001/search-retest.png
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
`fix` must follow a recorded failure. `retest` must follow a test and, when
|
|
69
|
+
the test failed, a fix. After a fix, retest the original scenario and at least
|
|
70
|
+
one adjacent scenario affected by the same surface.
|
|
71
|
+
|
|
72
|
+
## Gate and release evidence
|
|
73
|
+
|
|
74
|
+
The run gate is `fail` when any scenario fails, `blocked` when there is no
|
|
75
|
+
failure but pending/blocked/inconclusive scenarios remain, and `pass` only
|
|
76
|
+
when every scenario passes. Missing credentials, unsafe fixtures, a real
|
|
77
|
+
mobile viewport, a payment sandbox, or a fault-injection environment are
|
|
78
|
+
explicit blockers; never guess around them.
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
maggie qa summary --project . --run local-qa-001
|
|
82
|
+
maggie qa summary --project . --run local-qa-001 --format markdown
|
|
83
|
+
maggie qa export --project . --run local-qa-001 \
|
|
84
|
+
--output docs/qa-runs/local-qa-001.md
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Before calling a release Pass, also run the relevant build, accessibility,
|
|
88
|
+
SEO, media, API, and deployment-canary checks. A green build, HTTP 200, source
|
|
89
|
+
class, or one guest smoke is not browser QA evidence.
|
|
90
|
+
|
|
91
|
+
## Privacy and feedback
|
|
92
|
+
|
|
93
|
+
Do not put passwords, tokens, cookies, full private URLs, personal data,
|
|
94
|
+
provider response bodies, or secrets in run state, screenshots, or exported
|
|
95
|
+
reports. When the same workflow defect is reusable across projects, create a
|
|
96
|
+
privacy-safe draft through `maggie-feedback`; submitting it remains an explicit
|
|
97
|
+
user-confirmed action. Do not promote a project-specific preference directly
|
|
98
|
+
to active memory.
|
|
99
|
+
|
|
100
|
+
The implementation is `tools/clis/maggie_qa_workflow.py`; it is intentionally
|
|
101
|
+
small enough to run without a browser dependency and delegates browser
|
|
102
|
+
interaction to the host agent/browser capability.
|
|
@@ -0,0 +1,367 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Track project scenario-based browser QA from test through retest.
|
|
3
|
+
|
|
4
|
+
The command stores secret-free run state in ``.maggie/qa-runs``. Browser
|
|
5
|
+
interaction remains with the selected browser skill; this CLI records the
|
|
6
|
+
scenario, evidence references, fix event and final release decision.
|
|
7
|
+
"""
|
|
8
|
+
|
|
9
|
+
from __future__ import annotations
|
|
10
|
+
|
|
11
|
+
import argparse
|
|
12
|
+
import hashlib
|
|
13
|
+
import json
|
|
14
|
+
import os
|
|
15
|
+
import re
|
|
16
|
+
import sys
|
|
17
|
+
from datetime import datetime, timezone
|
|
18
|
+
from pathlib import Path
|
|
19
|
+
from urllib.parse import urlsplit, urlunsplit
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
DEFAULT_SCENARIOS = Path(".maggie") / "scenario-manifest.json"
|
|
23
|
+
VALID_STATUSES = {"pending", "pass", "fail", "blocked", "inconclusive"}
|
|
24
|
+
VALID_PHASES = {"test", "fix", "retest"}
|
|
25
|
+
SECRET_RE = re.compile(
|
|
26
|
+
r"(?i)(bearer\s+|(?:api[_-]?key|token|secret|password|authorization|cookie)\s*[=:]\s*)[^\s,;]+"
|
|
27
|
+
)
|
|
28
|
+
EMAIL_RE = re.compile(r"\b[\w.+-]+@[\w.-]+\.[A-Za-z]{2,}\b")
|
|
29
|
+
QUERY_SECRET_RE = re.compile(r"(?i)([?&](?:token|key|secret|password|code|id_token|access_token)=)[^&\s]+")
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
def now() -> str:
|
|
33
|
+
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
|
34
|
+
|
|
35
|
+
|
|
36
|
+
def safe_text(value: object, limit: int = 1200) -> str:
|
|
37
|
+
text = str(value or "").strip()
|
|
38
|
+
text = SECRET_RE.sub(r"\1[REDACTED]", text)
|
|
39
|
+
text = QUERY_SECRET_RE.sub(r"\1[REDACTED]", text)
|
|
40
|
+
text = EMAIL_RE.sub("[REDACTED_EMAIL]", text)
|
|
41
|
+
return text[:limit]
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
def safe_base_url(value: object) -> str:
|
|
45
|
+
"""Keep the origin only so reports do not retain private paths or queries."""
|
|
46
|
+
text = safe_text(value, 500)
|
|
47
|
+
try:
|
|
48
|
+
parsed = urlsplit(text)
|
|
49
|
+
if parsed.scheme and parsed.hostname:
|
|
50
|
+
netloc = parsed.hostname
|
|
51
|
+
if parsed.port:
|
|
52
|
+
netloc = f"{netloc}:{parsed.port}"
|
|
53
|
+
return urlunsplit((parsed.scheme, netloc, "", "", ""))
|
|
54
|
+
except ValueError:
|
|
55
|
+
pass
|
|
56
|
+
return text
|
|
57
|
+
|
|
58
|
+
|
|
59
|
+
def project_root(value: str | os.PathLike[str]) -> Path:
|
|
60
|
+
return Path(value).expanduser().resolve()
|
|
61
|
+
|
|
62
|
+
|
|
63
|
+
def load_json(path: Path) -> dict:
|
|
64
|
+
try:
|
|
65
|
+
value = json.loads(path.read_text(encoding="utf-8"))
|
|
66
|
+
except (OSError, json.JSONDecodeError) as error:
|
|
67
|
+
raise ValueError(f"could not read JSON {path}: {error}") from error
|
|
68
|
+
if not isinstance(value, dict):
|
|
69
|
+
raise ValueError(f"JSON root must be an object: {path}")
|
|
70
|
+
return value
|
|
71
|
+
|
|
72
|
+
|
|
73
|
+
def scenario_records(path: Path) -> list[dict]:
|
|
74
|
+
manifest = load_json(path)
|
|
75
|
+
scenarios = manifest.get("scenarios")
|
|
76
|
+
if not isinstance(scenarios, list) or not scenarios:
|
|
77
|
+
raise ValueError("scenario manifest must contain a non-empty scenarios array")
|
|
78
|
+
result = []
|
|
79
|
+
seen: set[str] = set()
|
|
80
|
+
for item in scenarios:
|
|
81
|
+
if not isinstance(item, dict) or not isinstance(item.get("id"), str):
|
|
82
|
+
raise ValueError("each scenario must be an object with an id")
|
|
83
|
+
scenario_id = item["id"]
|
|
84
|
+
if scenario_id in seen:
|
|
85
|
+
raise ValueError(f"duplicate scenario id: {scenario_id}")
|
|
86
|
+
seen.add(scenario_id)
|
|
87
|
+
result.append(item)
|
|
88
|
+
return result
|
|
89
|
+
|
|
90
|
+
|
|
91
|
+
def run_directory(project: Path) -> Path:
|
|
92
|
+
directory = project / ".maggie" / "qa-runs"
|
|
93
|
+
directory.mkdir(parents=True, exist_ok=True)
|
|
94
|
+
return directory
|
|
95
|
+
|
|
96
|
+
|
|
97
|
+
def normal_run_id(value: str | None) -> str:
|
|
98
|
+
if value:
|
|
99
|
+
value = re.sub(r"[^A-Za-z0-9._-]+", "-", value).strip("-")
|
|
100
|
+
if not value:
|
|
101
|
+
raise ValueError("run id cannot be empty")
|
|
102
|
+
return value
|
|
103
|
+
return "qa-" + datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
|
|
104
|
+
|
|
105
|
+
|
|
106
|
+
def run_path(project: Path, value: str) -> Path:
|
|
107
|
+
candidate = Path(value).expanduser()
|
|
108
|
+
if candidate.suffix == ".json" or candidate.parent != Path("."):
|
|
109
|
+
path = candidate if candidate.is_absolute() else project / candidate
|
|
110
|
+
else:
|
|
111
|
+
path = run_directory(project) / f"{normal_run_id(value)}.json"
|
|
112
|
+
path = path.resolve()
|
|
113
|
+
if not path.is_file():
|
|
114
|
+
raise ValueError(f"QA run does not exist: {path}")
|
|
115
|
+
return path
|
|
116
|
+
|
|
117
|
+
|
|
118
|
+
def write_json(path: Path, value: dict) -> None:
|
|
119
|
+
path.parent.mkdir(parents=True, exist_ok=True)
|
|
120
|
+
temporary = path.with_suffix(path.suffix + ".tmp")
|
|
121
|
+
temporary.write_text(json.dumps(value, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
|
|
122
|
+
temporary.replace(path)
|
|
123
|
+
|
|
124
|
+
|
|
125
|
+
def calculate(run: dict) -> dict:
|
|
126
|
+
counts = {status: 0 for status in VALID_STATUSES}
|
|
127
|
+
lifecycle_counts = {"ready": 0, "fix_pending": 0}
|
|
128
|
+
for item in run.get("scenarios", {}).values():
|
|
129
|
+
status = item.get("status", "pending")
|
|
130
|
+
counts[status] = counts.get(status, 0) + 1
|
|
131
|
+
lifecycle_counts["fix_pending" if item.get("lifecycle") == "fix_pending" else "ready"] += 1
|
|
132
|
+
if counts["fail"]:
|
|
133
|
+
gate = "fail"
|
|
134
|
+
elif counts["pending"] or counts["blocked"] or counts["inconclusive"]:
|
|
135
|
+
gate = "blocked"
|
|
136
|
+
else:
|
|
137
|
+
gate = "pass"
|
|
138
|
+
return {"gate": gate, "counts": counts, "lifecycle": lifecycle_counts, "total": sum(counts.values())}
|
|
139
|
+
|
|
140
|
+
|
|
141
|
+
def start(args: argparse.Namespace) -> int:
|
|
142
|
+
project = project_root(args.project)
|
|
143
|
+
manifest_path = project_root(args.scenario_file) if Path(args.scenario_file).is_absolute() else project / args.scenario_file
|
|
144
|
+
records = scenario_records(manifest_path)
|
|
145
|
+
run_id = normal_run_id(args.run_id)
|
|
146
|
+
path = run_directory(project) / f"{run_id}.json"
|
|
147
|
+
if path.exists():
|
|
148
|
+
raise ValueError(f"QA run already exists: {path}")
|
|
149
|
+
scenarios = {
|
|
150
|
+
item["id"]: {
|
|
151
|
+
"id": item["id"],
|
|
152
|
+
"group": item.get("group", ""),
|
|
153
|
+
"title": item.get("title", ""),
|
|
154
|
+
"priority": item.get("priority", "P2"),
|
|
155
|
+
"routes": item.get("routes", []),
|
|
156
|
+
"auth": item.get("auth", ""),
|
|
157
|
+
"browser": bool(item.get("browser", True)),
|
|
158
|
+
"status": "pending",
|
|
159
|
+
"lifecycle": "ready",
|
|
160
|
+
"history": [],
|
|
161
|
+
}
|
|
162
|
+
for item in records
|
|
163
|
+
}
|
|
164
|
+
try:
|
|
165
|
+
manifest_label = str(manifest_path.relative_to(project))
|
|
166
|
+
except ValueError:
|
|
167
|
+
# Do not persist an absolute path outside the project. The basename is
|
|
168
|
+
# enough to identify the source without leaking a local filesystem
|
|
169
|
+
# layout into a shareable run artifact.
|
|
170
|
+
manifest_label = manifest_path.name
|
|
171
|
+
run = {
|
|
172
|
+
"schemaVersion": "maggie.qa-run.v1",
|
|
173
|
+
"runId": run_id,
|
|
174
|
+
"createdAt": now(),
|
|
175
|
+
"updatedAt": now(),
|
|
176
|
+
"project": hashlib.sha256(str(project).encode()).hexdigest()[:16],
|
|
177
|
+
"environment": safe_text(args.environment),
|
|
178
|
+
"baseUrl": safe_base_url(args.base_url),
|
|
179
|
+
"browser": safe_text(args.browser),
|
|
180
|
+
"commit": safe_text(args.commit),
|
|
181
|
+
"release": safe_text(args.release),
|
|
182
|
+
"scenarioManifest": manifest_label,
|
|
183
|
+
"scenarios": scenarios,
|
|
184
|
+
}
|
|
185
|
+
run["summary"] = calculate(run)
|
|
186
|
+
write_json(path, run)
|
|
187
|
+
print(json.dumps({"runId": run_id, "path": str(path), "summary": run["summary"]}, ensure_ascii=False, indent=2))
|
|
188
|
+
return 0
|
|
189
|
+
|
|
190
|
+
|
|
191
|
+
def evidence(value: str, project: Path) -> dict:
|
|
192
|
+
clean = safe_text(value, 500)
|
|
193
|
+
path = Path(value).expanduser()
|
|
194
|
+
if path.is_file():
|
|
195
|
+
resolved = path.resolve()
|
|
196
|
+
try:
|
|
197
|
+
display = str(resolved.relative_to(project))
|
|
198
|
+
except ValueError:
|
|
199
|
+
display = resolved.name
|
|
200
|
+
return {"path": display, "sha256": hashlib.sha256(resolved.read_bytes()).hexdigest()}
|
|
201
|
+
return {"reference": clean}
|
|
202
|
+
|
|
203
|
+
|
|
204
|
+
def record(args: argparse.Namespace) -> int:
|
|
205
|
+
project = project_root(args.project)
|
|
206
|
+
path = run_path(project, args.run)
|
|
207
|
+
run = load_json(path)
|
|
208
|
+
scenarios = run.get("scenarios", {})
|
|
209
|
+
item = scenarios.get(args.scenario)
|
|
210
|
+
if not isinstance(item, dict):
|
|
211
|
+
raise ValueError(f"scenario is not in this run: {args.scenario}")
|
|
212
|
+
phase = args.phase
|
|
213
|
+
if phase not in VALID_PHASES:
|
|
214
|
+
raise ValueError(f"phase must be one of: {', '.join(sorted(VALID_PHASES))}")
|
|
215
|
+
if phase in {"test", "retest"} and args.status not in VALID_STATUSES - {"pending"}:
|
|
216
|
+
raise ValueError("test and retest require status pass, fail, blocked or inconclusive")
|
|
217
|
+
if phase == "fix" and not args.resolution:
|
|
218
|
+
raise ValueError("fix requires --resolution")
|
|
219
|
+
if phase == "fix" and item.get("status") != "fail":
|
|
220
|
+
raise ValueError("fix can only follow a recorded fail; use retest after a blocked or inconclusive run")
|
|
221
|
+
if phase == "retest" and not any(event.get("phase") == "test" for event in item.get("history", [])):
|
|
222
|
+
raise ValueError("retest requires an earlier test event")
|
|
223
|
+
if phase == "retest" and item.get("status") == "fail" and not any(event.get("phase") == "fix" for event in item.get("history", [])):
|
|
224
|
+
raise ValueError("a failed scenario must have a recorded fix before retest")
|
|
225
|
+
if phase == "test" and args.status == "fail" and not (args.expected and args.actual and args.error_fingerprint):
|
|
226
|
+
raise ValueError("a failed test requires --expected, --actual and --error-fingerprint")
|
|
227
|
+
event = {
|
|
228
|
+
"at": now(),
|
|
229
|
+
"phase": phase,
|
|
230
|
+
"status": args.status if phase != "fix" else "fix_pending",
|
|
231
|
+
"summary": safe_text(args.summary),
|
|
232
|
+
"expected": safe_text(args.expected),
|
|
233
|
+
"actual": safe_text(args.actual),
|
|
234
|
+
"errorFingerprint": safe_text(args.error_fingerprint, 300),
|
|
235
|
+
"issue": safe_text(args.issue, 500),
|
|
236
|
+
"resolution": safe_text(args.resolution),
|
|
237
|
+
"validation": safe_text(args.validation),
|
|
238
|
+
"evidence": [evidence(value, project) for value in args.evidence],
|
|
239
|
+
}
|
|
240
|
+
item.setdefault("history", []).append(event)
|
|
241
|
+
if phase == "fix":
|
|
242
|
+
item["lifecycle"] = "fix_pending"
|
|
243
|
+
else:
|
|
244
|
+
item["status"] = args.status
|
|
245
|
+
item["lifecycle"] = "ready"
|
|
246
|
+
run["updatedAt"] = now()
|
|
247
|
+
run["summary"] = calculate(run)
|
|
248
|
+
write_json(path, run)
|
|
249
|
+
print(json.dumps({"runId": run.get("runId"), "scenario": args.scenario, "event": event, "summary": run["summary"]}, ensure_ascii=False, indent=2))
|
|
250
|
+
return 0
|
|
251
|
+
|
|
252
|
+
|
|
253
|
+
def markdown(run: dict) -> str:
|
|
254
|
+
summary = run.get("summary") or calculate(run)
|
|
255
|
+
lines = [
|
|
256
|
+
f"# Maggie QA run `{run.get('runId', '')}`",
|
|
257
|
+
"",
|
|
258
|
+
"<!-- Generated by maggie_qa_workflow.py. Evidence is metadata-only. -->",
|
|
259
|
+
"",
|
|
260
|
+
f"- Environment: `{run.get('environment') or 'not provided'}`",
|
|
261
|
+
f"- Base URL: `{run.get('baseUrl') or 'not provided'}`",
|
|
262
|
+
f"- Browser: `{run.get('browser') or 'not provided'}`",
|
|
263
|
+
f"- Commit/release: `{run.get('commit') or 'not provided'}` / `{run.get('release') or 'not provided'}`",
|
|
264
|
+
f"- Gate: **{summary.get('gate', 'blocked').upper()}**",
|
|
265
|
+
"",
|
|
266
|
+
"## Scenario summary",
|
|
267
|
+
"",
|
|
268
|
+
"| ID | Priority | Scenario | Status | Lifecycle | Last result |",
|
|
269
|
+
"|---|---|---|---|---|---|",
|
|
270
|
+
]
|
|
271
|
+
for scenario in run.get("scenarios", {}).values():
|
|
272
|
+
history = scenario.get("history", [])
|
|
273
|
+
last = history[-1] if history else {}
|
|
274
|
+
result = safe_text(last.get("summary") or last.get("actual") or "Not tested", 180).replace("|", "\\|")
|
|
275
|
+
lines.append(f"| {scenario.get('id')} | {scenario.get('priority')} | {scenario.get('title')} | **{scenario.get('status')}** | {scenario.get('lifecycle')} | {result} |")
|
|
276
|
+
lines += ["", "## Fix and retest history", ""]
|
|
277
|
+
for scenario in run.get("scenarios", {}).values():
|
|
278
|
+
for event in scenario.get("history", []):
|
|
279
|
+
lines.append(f"### {scenario.get('id')} — {event.get('phase')} ({event.get('at')})")
|
|
280
|
+
lines.append("")
|
|
281
|
+
if event.get("summary"): lines.append(f"- Summary: {event['summary']}")
|
|
282
|
+
if event.get("expected"): lines.append(f"- Expected: {event['expected']}")
|
|
283
|
+
if event.get("actual"): lines.append(f"- Actual: {event['actual']}")
|
|
284
|
+
if event.get("errorFingerprint"): lines.append(f"- Error fingerprint: `{event['errorFingerprint']}`")
|
|
285
|
+
if event.get("resolution"): lines.append(f"- Resolution: {event['resolution']}")
|
|
286
|
+
if event.get("validation"): lines.append(f"- Validation: {event['validation']}")
|
|
287
|
+
references = event.get("evidence", [])
|
|
288
|
+
if references: lines.append(f"- Evidence: {', '.join(str(value.get('path') or value.get('reference')) for value in references)}")
|
|
289
|
+
lines.append("")
|
|
290
|
+
return "\n".join(lines).rstrip() + "\n"
|
|
291
|
+
|
|
292
|
+
|
|
293
|
+
def summary(args: argparse.Namespace) -> int:
|
|
294
|
+
project = project_root(args.project)
|
|
295
|
+
run = load_json(run_path(project, args.run))
|
|
296
|
+
result = run.get("summary") or calculate(run)
|
|
297
|
+
if args.format == "markdown":
|
|
298
|
+
print(markdown(run))
|
|
299
|
+
else:
|
|
300
|
+
print(json.dumps({"runId": run.get("runId"), **result}, ensure_ascii=False, indent=2))
|
|
301
|
+
return 0 if result.get("gate") == "pass" else 1
|
|
302
|
+
|
|
303
|
+
|
|
304
|
+
def export_run(args: argparse.Namespace) -> int:
|
|
305
|
+
project = project_root(args.project)
|
|
306
|
+
run = load_json(run_path(project, args.run))
|
|
307
|
+
output = Path(args.output).expanduser()
|
|
308
|
+
if not output.is_absolute(): output = project / output
|
|
309
|
+
output.parent.mkdir(parents=True, exist_ok=True)
|
|
310
|
+
output.write_text(markdown(run), encoding="utf-8")
|
|
311
|
+
print(json.dumps({"runId": run.get("runId"), "output": str(output.resolve()), "gate": run.get("summary", {}).get("gate")}, ensure_ascii=False, indent=2))
|
|
312
|
+
return 0
|
|
313
|
+
|
|
314
|
+
|
|
315
|
+
def parser() -> argparse.ArgumentParser:
|
|
316
|
+
root = argparse.ArgumentParser(description=__doc__)
|
|
317
|
+
root.add_argument("--project", default=".")
|
|
318
|
+
commands = root.add_subparsers(dest="command", required=True)
|
|
319
|
+
|
|
320
|
+
start_parser = commands.add_parser("start")
|
|
321
|
+
start_parser.add_argument("--project", default=argparse.SUPPRESS)
|
|
322
|
+
start_parser.add_argument("--scenario-file", default=str(DEFAULT_SCENARIOS))
|
|
323
|
+
start_parser.add_argument("--run-id")
|
|
324
|
+
start_parser.add_argument("--environment", required=True)
|
|
325
|
+
start_parser.add_argument("--base-url", required=True)
|
|
326
|
+
start_parser.add_argument("--browser", default="chrome")
|
|
327
|
+
start_parser.add_argument("--commit", default="")
|
|
328
|
+
start_parser.add_argument("--release", default="")
|
|
329
|
+
|
|
330
|
+
record_parser = commands.add_parser("record")
|
|
331
|
+
record_parser.add_argument("--project", default=argparse.SUPPRESS)
|
|
332
|
+
record_parser.add_argument("--run", required=True)
|
|
333
|
+
record_parser.add_argument("--scenario", required=True)
|
|
334
|
+
record_parser.add_argument("--phase", choices=sorted(VALID_PHASES), required=True)
|
|
335
|
+
record_parser.add_argument("--status", choices=sorted(VALID_STATUSES - {"pending"}))
|
|
336
|
+
record_parser.add_argument("--summary", default="")
|
|
337
|
+
record_parser.add_argument("--expected", default="")
|
|
338
|
+
record_parser.add_argument("--actual", default="")
|
|
339
|
+
record_parser.add_argument("--error-fingerprint", default="")
|
|
340
|
+
record_parser.add_argument("--issue", default="")
|
|
341
|
+
record_parser.add_argument("--resolution", default="")
|
|
342
|
+
record_parser.add_argument("--validation", default="")
|
|
343
|
+
record_parser.add_argument("--evidence", action="append", default=[])
|
|
344
|
+
|
|
345
|
+
for name in ("summary", "export"):
|
|
346
|
+
command = commands.add_parser(name)
|
|
347
|
+
command.add_argument("--project", default=argparse.SUPPRESS)
|
|
348
|
+
command.add_argument("--run", required=True)
|
|
349
|
+
if name == "summary": command.add_argument("--format", choices=("json", "markdown"), default="json")
|
|
350
|
+
else: command.add_argument("--output", required=True)
|
|
351
|
+
return root
|
|
352
|
+
|
|
353
|
+
|
|
354
|
+
def main() -> int:
|
|
355
|
+
args = parser().parse_args()
|
|
356
|
+
try:
|
|
357
|
+
if args.command == "start": return start(args)
|
|
358
|
+
if args.command == "record": return record(args)
|
|
359
|
+
if args.command == "summary": return summary(args)
|
|
360
|
+
return export_run(args)
|
|
361
|
+
except (OSError, ValueError, json.JSONDecodeError) as error:
|
|
362
|
+
print(f"QA workflow error: {safe_text(error)}", file=sys.stderr)
|
|
363
|
+
return 2
|
|
364
|
+
|
|
365
|
+
|
|
366
|
+
if __name__ == "__main__":
|
|
367
|
+
raise SystemExit(main())
|