@jevitate/cli 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +23 -0
- package/dist/ai-cli.d.ts +31 -11
- package/dist/ai-cli.js +131 -88
- package/dist/ai-cli.js.map +1 -1
- package/dist/author-journey-cli.d.ts +4 -0
- package/dist/author-journey-cli.js +111 -0
- package/dist/author-journey-cli.js.map +1 -0
- package/dist/bin.js +60349 -16477
- package/dist/bin.js.map +1 -1
- package/dist/browser-run-options.d.ts +89 -0
- package/dist/browser-run-options.js +175 -0
- package/dist/browser-run-options.js.map +1 -0
- package/dist/build-info.generated.d.ts +2 -0
- package/dist/build-info.generated.js +10 -0
- package/dist/build-info.generated.js.map +1 -0
- package/dist/campaign-api.d.ts +151 -0
- package/dist/campaign-api.js +446 -0
- package/dist/campaign-api.js.map +1 -0
- package/dist/campaign-cli.d.ts +6 -0
- package/dist/campaign-cli.js +66 -0
- package/dist/campaign-cli.js.map +1 -0
- package/dist/check-api.d.ts +5 -0
- package/dist/check-api.js +234 -0
- package/dist/check-api.js.map +1 -0
- package/dist/check-budget.d.ts +26 -0
- package/dist/check-budget.js +90 -0
- package/dist/check-budget.js.map +1 -0
- package/dist/check-cli.d.ts +31 -0
- package/dist/check-cli.js +109 -0
- package/dist/check-cli.js.map +1 -0
- package/dist/check-execute.d.ts +29 -0
- package/dist/check-execute.js +420 -0
- package/dist/check-execute.js.map +1 -0
- package/dist/check-plan.d.ts +118 -0
- package/dist/check-plan.js +516 -0
- package/dist/check-plan.js.map +1 -0
- package/dist/check-suite.d.ts +184 -0
- package/dist/check-suite.js +540 -0
- package/dist/check-suite.js.map +1 -0
- package/dist/check-types.d.ts +181 -0
- package/dist/check-types.js +35 -0
- package/dist/check-types.js.map +1 -0
- package/dist/cli-args.d.ts +26 -0
- package/dist/cli-args.js +50 -0
- package/dist/cli-args.js.map +1 -0
- package/dist/cli-output.d.ts +106 -0
- package/dist/cli-output.js +563 -0
- package/dist/cli-output.js.map +1 -0
- package/dist/cli-reference.d.ts +3 -0
- package/dist/cli-reference.js +81 -0
- package/dist/cli-reference.js.map +1 -0
- package/dist/cli-refusal.d.ts +12 -0
- package/dist/cli-refusal.js +52 -0
- package/dist/cli-refusal.js.map +1 -0
- package/dist/cli-shared.d.ts +340 -0
- package/dist/cli-shared.js +537 -0
- package/dist/cli-shared.js.map +1 -0
- package/dist/cli-surface.d.ts +33 -0
- package/dist/cli-surface.js +48 -0
- package/dist/cli-surface.js.map +1 -0
- package/dist/conversation-options.d.ts +15 -0
- package/dist/conversation-options.js +10 -0
- package/dist/conversation-options.js.map +1 -0
- package/dist/credentials-file.d.ts +15 -0
- package/dist/credentials-file.js +56 -0
- package/dist/credentials-file.js.map +1 -0
- package/dist/data-dir.d.ts +1 -1
- package/dist/data-dir.js +1 -1
- package/dist/defect-evidence.d.ts +141 -0
- package/dist/defect-evidence.js +505 -0
- package/dist/defect-evidence.js.map +1 -0
- package/dist/demo-aspect-api.d.ts +209 -0
- package/dist/demo-aspect-api.js +398 -0
- package/dist/demo-aspect-api.js.map +1 -0
- package/dist/demo-aspect-cli.d.ts +3 -0
- package/dist/demo-aspect-cli.js +243 -0
- package/dist/demo-aspect-cli.js.map +1 -0
- package/dist/demo-capture.d.ts +142 -0
- package/dist/demo-capture.js +537 -0
- package/dist/demo-capture.js.map +1 -0
- package/dist/engine.d.ts +35 -0
- package/dist/engine.js +25 -0
- package/dist/engine.js.map +1 -0
- package/dist/environments.d.ts +155 -0
- package/dist/environments.js +364 -0
- package/dist/environments.js.map +1 -0
- package/dist/exit-codes.d.ts +46 -0
- package/dist/exit-codes.js +91 -0
- package/dist/exit-codes.js.map +1 -0
- package/dist/explore-adversarial.d.ts +144 -0
- package/dist/explore-adversarial.js +227 -0
- package/dist/explore-adversarial.js.map +1 -0
- package/dist/explore-api.d.ts +7 -187
- package/dist/explore-api.js +7 -301
- package/dist/explore-api.js.map +1 -1
- package/dist/explore-author.d.ts +79 -0
- package/dist/explore-author.js +86 -0
- package/dist/explore-author.js.map +1 -0
- package/dist/explore-cli.d.ts +7 -0
- package/dist/explore-cli.js +1199 -0
- package/dist/explore-cli.js.map +1 -0
- package/dist/explore-coverage.d.ts +173 -0
- package/dist/explore-coverage.js +246 -0
- package/dist/explore-coverage.js.map +1 -0
- package/dist/explore-feature.d.ts +125 -0
- package/dist/explore-feature.js +218 -0
- package/dist/explore-feature.js.map +1 -0
- package/dist/explore-goal.d.ts +271 -0
- package/dist/explore-goal.js +417 -0
- package/dist/explore-goal.js.map +1 -0
- package/dist/explore-shared.d.ts +177 -0
- package/dist/explore-shared.js +149 -0
- package/dist/explore-shared.js.map +1 -0
- package/dist/explore-specs.d.ts +53 -0
- package/dist/explore-specs.js +333 -0
- package/dist/explore-specs.js.map +1 -0
- package/dist/findings-filing.d.ts +64 -0
- package/dist/findings-filing.js +108 -0
- package/dist/findings-filing.js.map +1 -0
- package/dist/fixture-auth.d.ts +63 -0
- package/dist/fixture-auth.js +95 -0
- package/dist/fixture-auth.js.map +1 -0
- package/dist/fixture-cli.d.ts +99 -0
- package/dist/fixture-cli.js +292 -0
- package/dist/fixture-cli.js.map +1 -0
- package/dist/github-issue-filer.d.ts +62 -0
- package/dist/github-issue-filer.js +189 -0
- package/dist/github-issue-filer.js.map +1 -0
- package/dist/host-health-run.d.ts +25 -0
- package/dist/host-health-run.js +33 -0
- package/dist/host-health-run.js.map +1 -0
- package/dist/http-5xx-outcome.d.ts +6 -0
- package/dist/http-5xx-outcome.js +39 -0
- package/dist/http-5xx-outcome.js.map +1 -0
- package/dist/inbox-cli.d.ts +7 -0
- package/dist/inbox-cli.js +197 -0
- package/dist/inbox-cli.js.map +1 -0
- package/dist/index.d.ts +1 -0
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/init-cli.d.ts +4 -0
- package/dist/init-cli.js +156 -0
- package/dist/init-cli.js.map +1 -0
- package/dist/init-keys.d.ts +30 -2
- package/dist/init-keys.js +19 -5
- package/dist/init-keys.js.map +1 -1
- package/dist/init-next-steps.d.ts +35 -0
- package/dist/init-next-steps.js +79 -0
- package/dist/init-next-steps.js.map +1 -0
- package/dist/invariants-file.d.ts +34 -0
- package/dist/invariants-file.js +68 -0
- package/dist/invariants-file.js.map +1 -0
- package/dist/invariants-validate.d.ts +30 -0
- package/dist/invariants-validate.js +105 -0
- package/dist/invariants-validate.js.map +1 -0
- package/dist/journey-annotate-api.d.ts +76 -0
- package/dist/journey-annotate-api.js +269 -0
- package/dist/journey-annotate-api.js.map +1 -0
- package/dist/journey-annotate-cli.d.ts +9 -0
- package/dist/journey-annotate-cli.js +181 -0
- package/dist/journey-annotate-cli.js.map +1 -0
- package/dist/journey-api.d.ts +104 -2
- package/dist/journey-api.js +163 -20
- package/dist/journey-api.js.map +1 -1
- package/dist/journey-cli.d.ts +4 -0
- package/dist/journey-cli.js +411 -0
- package/dist/journey-cli.js.map +1 -0
- package/dist/journey-demo-api.d.ts +114 -0
- package/dist/journey-demo-api.js +405 -0
- package/dist/journey-demo-api.js.map +1 -0
- package/dist/journey-demo-cli.d.ts +10 -0
- package/dist/journey-demo-cli.js +158 -0
- package/dist/journey-demo-cli.js.map +1 -0
- package/dist/journey-prefix.d.ts +194 -0
- package/dist/journey-prefix.js +279 -0
- package/dist/journey-prefix.js.map +1 -0
- package/dist/key-report.d.ts +49 -0
- package/dist/key-report.js +101 -0
- package/dist/key-report.js.map +1 -0
- package/dist/kill-signal.d.ts +193 -0
- package/dist/kill-signal.js +283 -0
- package/dist/kill-signal.js.map +1 -0
- package/dist/launch-armed.d.ts +37 -0
- package/dist/launch-armed.js +31 -0
- package/dist/launch-armed.js.map +1 -0
- package/dist/ledger-api.d.ts +106 -0
- package/dist/ledger-api.js +239 -0
- package/dist/ledger-api.js.map +1 -0
- package/dist/ledger-cli.d.ts +18 -0
- package/dist/ledger-cli.js +107 -0
- package/dist/ledger-cli.js.map +1 -0
- package/dist/load-api.d.ts +29 -11
- package/dist/load-api.js +23 -15
- package/dist/load-api.js.map +1 -1
- package/dist/load-cli.d.ts +4 -0
- package/dist/load-cli.js +97 -0
- package/dist/load-cli.js.map +1 -0
- package/dist/log-correlation.d.ts +200 -0
- package/dist/log-correlation.js +394 -0
- package/dist/log-correlation.js.map +1 -0
- package/dist/log-lines.d.ts +131 -0
- package/dist/log-lines.js +389 -0
- package/dist/log-lines.js.map +1 -0
- package/dist/log-sources.d.ts +71 -0
- package/dist/log-sources.js +236 -0
- package/dist/log-sources.js.map +1 -0
- package/dist/log-trace.d.ts +80 -0
- package/dist/log-trace.js +210 -0
- package/dist/log-trace.js.map +1 -0
- package/dist/logs-cli.d.ts +3 -0
- package/dist/logs-cli.js +51 -0
- package/dist/logs-cli.js.map +1 -0
- package/dist/logs-retention.d.ts +28 -0
- package/dist/logs-retention.js +85 -0
- package/dist/logs-retention.js.map +1 -0
- package/dist/masked-input.d.ts +22 -0
- package/dist/masked-input.js +99 -0
- package/dist/masked-input.js.map +1 -0
- package/dist/mcp-api.d.ts +91 -1
- package/dist/mcp-api.js +514 -18
- package/dist/mcp-api.js.map +1 -1
- package/dist/mcp-args.d.ts +50 -0
- package/dist/mcp-args.js +164 -0
- package/dist/mcp-args.js.map +1 -0
- package/dist/mcp-cli-bridge.d.ts +34 -0
- package/dist/mcp-cli-bridge.js +62 -0
- package/dist/mcp-cli-bridge.js.map +1 -0
- package/dist/mcp-cli-runner.d.ts +29 -0
- package/dist/mcp-cli-runner.js +44 -0
- package/dist/mcp-cli-runner.js.map +1 -0
- package/dist/mcp-cli-tools.d.ts +129 -0
- package/dist/mcp-cli-tools.js +737 -0
- package/dist/mcp-cli-tools.js.map +1 -0
- package/dist/mcp-paths.d.ts +28 -0
- package/dist/mcp-paths.js +76 -0
- package/dist/mcp-paths.js.map +1 -0
- package/dist/mission-actors.d.ts +40 -0
- package/dist/mission-actors.js +72 -0
- package/dist/mission-actors.js.map +1 -0
- package/dist/mission-api.d.ts +35 -0
- package/dist/mission-api.js +60 -3
- package/dist/mission-api.js.map +1 -1
- package/dist/mission-cli.d.ts +4 -0
- package/dist/mission-cli.js +248 -0
- package/dist/mission-cli.js.map +1 -0
- package/dist/mission-exit.d.ts +22 -0
- package/dist/mission-exit.js +26 -0
- package/dist/mission-exit.js.map +1 -0
- package/dist/mission-fixtures.d.ts +234 -0
- package/dist/mission-fixtures.js +862 -0
- package/dist/mission-fixtures.js.map +1 -0
- package/dist/mission-journal.d.ts +60 -0
- package/dist/mission-journal.js +115 -0
- package/dist/mission-journal.js.map +1 -0
- package/dist/mission-queue-cli.d.ts +10 -0
- package/dist/mission-queue-cli.js +198 -0
- package/dist/mission-queue-cli.js.map +1 -0
- package/dist/mission-queue-runner.d.ts +126 -0
- package/dist/mission-queue-runner.js +406 -0
- package/dist/mission-queue-runner.js.map +1 -0
- package/dist/multi-run-cli.d.ts +37 -0
- package/dist/multi-run-cli.js +116 -0
- package/dist/multi-run-cli.js.map +1 -0
- package/dist/multi-run.d.ts +339 -0
- package/dist/multi-run.js +619 -0
- package/dist/multi-run.js.map +1 -0
- package/dist/openrouter-call.d.ts +22 -0
- package/dist/openrouter-call.js +64 -0
- package/dist/openrouter-call.js.map +1 -0
- package/dist/profile-cli.d.ts +4 -0
- package/dist/profile-cli.js +57 -0
- package/dist/profile-cli.js.map +1 -0
- package/dist/program.d.ts +3 -75
- package/dist/program.js +123 -1921
- package/dist/program.js.map +1 -1
- package/dist/project-dir.d.ts +96 -0
- package/dist/project-dir.js +221 -0
- package/dist/project-dir.js.map +1 -0
- package/dist/record-api.d.ts +13 -7
- package/dist/record-api.js +44 -30
- package/dist/record-api.js.map +1 -1
- package/dist/record-cli.d.ts +4 -0
- package/dist/record-cli.js +76 -0
- package/dist/record-cli.js.map +1 -0
- package/dist/recording-cli.d.ts +4 -0
- package/dist/recording-cli.js +248 -0
- package/dist/recording-cli.js.map +1 -0
- package/dist/regression-api.d.ts +107 -0
- package/dist/regression-api.js +387 -6
- package/dist/regression-api.js.map +1 -1
- package/dist/regression-cli.d.ts +4 -0
- package/dist/regression-cli.js +284 -0
- package/dist/regression-cli.js.map +1 -0
- package/dist/report-api.d.ts +146 -0
- package/dist/report-api.js +431 -0
- package/dist/report-api.js.map +1 -0
- package/dist/report-cli.d.ts +12 -0
- package/dist/report-cli.js +147 -0
- package/dist/report-cli.js.map +1 -0
- package/dist/resource-preflight.d.ts +87 -0
- package/dist/resource-preflight.js +160 -0
- package/dist/resource-preflight.js.map +1 -0
- package/dist/result-schema.d.ts +22 -0
- package/dist/result-schema.js +28 -0
- package/dist/result-schema.js.map +1 -0
- package/dist/run-index.d.ts +29 -0
- package/dist/run-index.js +70 -0
- package/dist/run-index.js.map +1 -0
- package/dist/run-key-preflight.d.ts +42 -0
- package/dist/run-key-preflight.js +82 -0
- package/dist/run-key-preflight.js.map +1 -0
- package/dist/run-screenshots.d.ts +123 -0
- package/dist/run-screenshots.js +235 -0
- package/dist/run-screenshots.js.map +1 -0
- package/dist/runtime-env.d.ts +8 -0
- package/dist/runtime-env.js +23 -0
- package/dist/runtime-env.js.map +1 -0
- package/dist/secret-args.d.ts +23 -0
- package/dist/secret-args.js +45 -0
- package/dist/secret-args.js.map +1 -0
- package/dist/serve-cli.d.ts +4 -0
- package/dist/serve-cli.js +95 -0
- package/dist/serve-cli.js.map +1 -0
- package/dist/server-log-verify.d.ts +36 -0
- package/dist/server-log-verify.js +120 -0
- package/dist/server-log-verify.js.map +1 -0
- package/dist/session-check.d.ts +14 -0
- package/dist/session-check.js +44 -0
- package/dist/session-check.js.map +1 -0
- package/dist/site-cli.d.ts +4 -0
- package/dist/site-cli.js +127 -0
- package/dist/site-cli.js.map +1 -0
- package/dist/site-gate-cli.d.ts +28 -0
- package/dist/site-gate-cli.js +59 -0
- package/dist/site-gate-cli.js.map +1 -0
- package/dist/source-api.js +20 -4
- package/dist/source-api.js.map +1 -1
- package/dist/source-cli.d.ts +4 -0
- package/dist/source-cli.js +284 -0
- package/dist/source-cli.js.map +1 -0
- package/dist/source-run-api.d.ts +27 -1
- package/dist/source-run-api.js +22 -11
- package/dist/source-run-api.js.map +1 -1
- package/dist/storage-state-snapshot.d.ts +50 -0
- package/dist/storage-state-snapshot.js +87 -0
- package/dist/storage-state-snapshot.js.map +1 -0
- package/dist/suite-explore-options.d.ts +321 -0
- package/dist/suite-explore-options.js +162 -0
- package/dist/suite-explore-options.js.map +1 -0
- package/dist/target-config.d.ts +115 -0
- package/dist/target-config.js +225 -0
- package/dist/target-config.js.map +1 -0
- package/dist/transcript-file.d.ts +12 -0
- package/dist/transcript-file.js +23 -0
- package/dist/transcript-file.js.map +1 -0
- package/dist/type-fixture-file.d.ts +9 -0
- package/dist/type-fixture-file.js +48 -0
- package/dist/type-fixture-file.js.map +1 -0
- package/dist/ui-api.d.ts +3 -0
- package/dist/ui-api.js +11 -4
- package/dist/ui-api.js.map +1 -1
- package/dist/usability-capture.d.ts +43 -0
- package/dist/usability-capture.js +327 -0
- package/dist/usability-capture.js.map +1 -0
- package/dist/usage-config.d.ts +36 -0
- package/dist/usage-config.js +114 -0
- package/dist/usage-config.js.map +1 -0
- package/dist/ux-api.d.ts +3 -75
- package/dist/ux-api.js +3 -220
- package/dist/ux-api.js.map +1 -1
- package/dist/ux-claim-probe.d.ts +45 -0
- package/dist/ux-claim-probe.js +386 -0
- package/dist/ux-claim-probe.js.map +1 -0
- package/dist/ux-cli.d.ts +4 -0
- package/dist/ux-cli.js +109 -0
- package/dist/ux-cli.js.map +1 -0
- package/dist/ux-config.d.ts +29 -0
- package/dist/ux-config.js +103 -0
- package/dist/ux-config.js.map +1 -0
- package/dist/ux-evidence.d.ts +54 -0
- package/dist/ux-evidence.js +197 -0
- package/dist/ux-evidence.js.map +1 -0
- package/dist/ux-product.d.ts +14 -0
- package/dist/ux-product.js +39 -0
- package/dist/ux-product.js.map +1 -0
- package/dist/ux-review.d.ts +116 -0
- package/dist/ux-review.js +171 -0
- package/dist/ux-review.js.map +1 -0
- package/dist/ux-usability.d.ts +263 -0
- package/dist/ux-usability.js +580 -0
- package/dist/ux-usability.js.map +1 -0
- package/dist/verify-fix-api.d.ts +194 -0
- package/dist/verify-fix-api.js +506 -0
- package/dist/verify-fix-api.js.map +1 -0
- package/dist/verify-fix-cli.d.ts +4 -0
- package/dist/verify-fix-cli.js +115 -0
- package/dist/verify-fix-cli.js.map +1 -0
- package/package.json +23 -18
- package/skills/jevitate-ci-check/SKILL.md +77 -0
- package/skills/jevitate-demo/SKILL.md +61 -0
- package/skills/jevitate-explore/SKILL.md +96 -87
- package/skills/jevitate-getting-started/SKILL.md +89 -0
- package/skills/jevitate-load-test/SKILL.md +11 -2
- package/skills/jevitate-mission-scope/SKILL.md +19 -11
- package/skills/jevitate-record/SKILL.md +23 -5
- package/skills/jevitate-run-journey/SKILL.md +61 -17
- package/skills/jevitate-sources/SKILL.md +57 -14
- package/skills/jevitate-test-campaign/SKILL.md +203 -0
- package/skills/jevitate-ux-review/SKILL.md +106 -16
- package/skills/jevitate-verify-fix/SKILL.md +76 -0
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: jevitate-test-campaign
|
|
3
|
+
description: Plan and run an end-to-end release test campaign of a whole web app with jevitate: preflight, a jobs-to-be-done catalog, discovery runs → draft Journeys, journey-anchored adversarial/exploratory missions, full-stack evidence, triage by exit code, verify-fix and regressions, budgets, CI suites and the release gate. Use to test a release or "the whole app" across many user jobs. For one run, use that tool's skill.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
A campaign is many bounded runs, chosen on purpose and read honestly. An untargeted sweep from the
|
|
7
|
+
landing page stays shallow (a handful of states near the entry pages) and never reaches the core
|
|
8
|
+
jobs. Anchor everything on the jobs users actually do. Load `jevitate-explore`,
|
|
9
|
+
`jevitate-run-journey`, `jevitate-verify-fix` and `jevitate-ci-check` for the flags of each step.
|
|
10
|
+
|
|
11
|
+
## 0. Targets and environments
|
|
12
|
+
|
|
13
|
+
- Test a non-production environment the human has authorized: a local dev server, a preview or a
|
|
14
|
+
staging build. Name each in `.jevitate/environments.json` so Journeys run with `--env <name>`.
|
|
15
|
+
- Iterate on a dev or debug build (hot reload makes frontend fixes cheap). Timing findings there
|
|
16
|
+
are advisory. Run the final round on the compiled release build before the gate, and leave load
|
|
17
|
+
testing (`jevitate-load-test`) to a release-like environment.
|
|
18
|
+
- Never point a sweep at a database other test work is using. Seed or reset it (`--fixtures`, or an
|
|
19
|
+
operator `--before`/`--after` hook) so runs don't pollute each other.
|
|
20
|
+
|
|
21
|
+
## 1. Preflight (stop on any failure)
|
|
22
|
+
|
|
23
|
+
- `jevitate ai status --json`: the keys must be accepted live, not just present.
|
|
24
|
+
- One tiny `--real` goal run proves the model path end to end before you queue an hour of runs.
|
|
25
|
+
- One Playwright storageState per persona (kept under `~/.jevitate/`, never committed). Check each
|
|
26
|
+
still logs in: a quick goal run per persona with `--storage-state <file>` and a `--success`
|
|
27
|
+
check on a page only that persona sees. Use `--save-storage-state` on the same file when the app
|
|
28
|
+
rotates refresh tokens.
|
|
29
|
+
- Target health: the start URL answers and the backend is up. A dead target makes every run
|
|
30
|
+
`inconclusive`, which tells you nothing.
|
|
31
|
+
|
|
32
|
+
## 2. Build the job catalog first
|
|
33
|
+
|
|
34
|
+
List every user job from the app's routes, API surface and product docs: persona, job, and the
|
|
35
|
+
signal that proves it worked. Write one goal spec per job (template below): the outcome only,
|
|
36
|
+
never the click path; independent `--success` checks (page or network state); preconditions;
|
|
37
|
+
whether it mutates state; and a run order (jobs that create data run before jobs that use it).
|
|
38
|
+
A job you can't write a checkable `--success` for is too vague. Split it or narrow it.
|
|
39
|
+
|
|
40
|
+
## 3. Discovery: one goal run per job
|
|
41
|
+
|
|
42
|
+
For each job, as the right persona, with no scripted path:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
jevitate explore-author-journey --url <start-url> --goal "<outcome only>" --success "<check>" --id <job-id> --name "<job>" --storage-state <persona-file> --real --json
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
- Reached: an UNPROMOTED draft Journey is written. Promotion is a human decision. List the drafts
|
|
49
|
+
and ask the human to review and run `jevitate journey promote <id>`. Never promote yourself.
|
|
50
|
+
- Stuck, detour, `exhausted` or `inconclusive`: that is a finding about the job itself
|
|
51
|
+
(discoverability, UX, a broken step), not only a crash. Log it in the register.
|
|
52
|
+
- A question rather than a task ("how many seats does the plan allow?") is a find-out run: a
|
|
53
|
+
`--goal "find out …; report the answer"` with no `--success`, read-only by default.
|
|
54
|
+
|
|
55
|
+
## 4. Journey-anchored missions (where most defects are)
|
|
56
|
+
|
|
57
|
+
Defects cluster mid-flow: a verification step, a checkout, a half-finished build. Start missions
|
|
58
|
+
from a promoted Journey's key intermediate states, not the landing page. The Journey's prefix is
|
|
59
|
+
replayed in the mission's OWN browser context (page, form contents and session kept), then the
|
|
60
|
+
mission takes over on the live page:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
jevitate journey anchors <journey-id> --json # the named states worth exploring from, with suggested probes
|
|
64
|
+
jevitate explore --from-journey <journey-id> --at-step <anchor|n> --param <k=v> --env <env> --strategy adversarial --storage-state <persona-file> --max-actions 60 --evidence-video --real --json
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
- `--at-step` takes an anchor name (from `metadata.anchors`) or a 1-based step number. Only the
|
|
68
|
+
prefix's own `--param`s are required. `--strategy` may be goal, coverage, exploratory,
|
|
69
|
+
adversarial or usability (`--feature`, `--repeat`, `--persona` and `--actor` are refused).
|
|
70
|
+
- A prefix that no longer replays ends the run `inconclusive` with `failure.kind: "journey-stale"`
|
|
71
|
+
(exit 2): the Journey drifted. Fix or re-author it; never fall back to a bare `--url`.
|
|
72
|
+
- Results and findings carry `branch: {journeyId, step, anchor}`: the exact point they came from.
|
|
73
|
+
- `--at-step all` (or `anchors`) sweeps every step: each in a fresh session restored by
|
|
74
|
+
`--fixtures`, `--max-actions` split evenly per step, one deduped report naming each defect's steps.
|
|
75
|
+
- `verify-fix` and regressions replay such a finding THROUGH the Journey prefix (re-supply secret
|
|
76
|
+
params with `--param`); a stale prefix is `inconclusive` (`journey-stale`), never `fixed`.
|
|
77
|
+
- `--strategy adversarial` attacks the anchor: replaying a one-time code, swapping a tenant id
|
|
78
|
+
mid-flow, a double submit at checkout, an expired share link, skipping a step-up (the anchor's
|
|
79
|
+
`probes` list what to try). `--strategy exploratory` follows newly revealed controls from there.
|
|
80
|
+
`--strategy usability --goal "<job>" --app-class <class>` critiques the job (`jevitate-ux-review`).
|
|
81
|
+
- `--persona <name=storageState>` (repeatable, without `--from-journey`) runs the same mission per
|
|
82
|
+
role and diffs them; a 403 for one and 200 for another is a candidate authorization finding.
|
|
83
|
+
- `--invariants <file>` turns app rules ("Saved means stored") into code-decided defects. Lint
|
|
84
|
+
them first: `jevitate invariants validate <files...> --url <url>`.
|
|
85
|
+
|
|
86
|
+
Sweep many anchors with a campaign (template below): discovery replays each job's whole Journey (a
|
|
87
|
+
stale one skips its missions), then every anchored mission runs in order with the spec's
|
|
88
|
+
`fixtures` restore around each run, and ONE deduped report comes back (each defect lists the
|
|
89
|
+
Journey steps it branched from):
|
|
90
|
+
|
|
91
|
+
```bash
|
|
92
|
+
jevitate campaign run .jevitate/campaign/campaign.json --real --json
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
An invalid spec is refused with every problem listed (exit 64). Shell `before`/`after` restore
|
|
96
|
+
hooks in the spec run only with `--allow-shell-hooks`, and only when the human asked for them.
|
|
97
|
+
|
|
98
|
+
## 5. A small untargeted pass, plus non-journey attacks
|
|
99
|
+
|
|
100
|
+
Keep one `--strategy coverage --scope app` run per persona to find orphan and dead-end screens.
|
|
101
|
+
Attacks that no Journey reaches (direct API authorization, forged or reordered webhooks,
|
|
102
|
+
cross-tenant requests, session fixation across hosts) are scripted by the operator; record what
|
|
103
|
+
was tried in the register, so the gate can show they were covered.
|
|
104
|
+
|
|
105
|
+
## 6. Full-stack evidence on every run
|
|
106
|
+
|
|
107
|
+
Tail the backend while the browser runs, so each finding carries its server-side cause:
|
|
108
|
+
`--log-source docker:<container>` (or `file:<path>`), `--log-defect error`, and
|
|
109
|
+
`--log-correlation-header <name>` when the app propagates a request id. Matching log lines are
|
|
110
|
+
redacted and attached to the step that caused them. Add `--evidence-video` (a captioned clip per
|
|
111
|
+
defect) or `--screenshots`, and link the files from the issue.
|
|
112
|
+
|
|
113
|
+
## 7. Triage by exit code, then collect before fixing
|
|
114
|
+
|
|
115
|
+
`0` clean · `1` defects found · `2` inconclusive (proved nothing: never a pass; fix the budget,
|
|
116
|
+
session or check and rerun) · `3` hang · `4` intermittent · `64` refused, nothing ran. Report the
|
|
117
|
+
outcome you got. Never round it up.
|
|
118
|
+
|
|
119
|
+
1. Collect: a campaign's own `campaign.md` is already one deduped list; across campaigns and
|
|
120
|
+
ad-hoc runs, `jevitate report --target <origin> --since <campaign-start> --json` dedupes every run
|
|
121
|
+
into one defect list. Keep one register (template below) tagged by layer and severity.
|
|
122
|
+
2. Fix: frontend fixes under the hot-reload dev server, rechecked at once; backend fixes batched
|
|
123
|
+
into one rebuild.
|
|
124
|
+
3. Prove: `jevitate verify-fix --result <run>.result.json --fingerprint <fp> --replays 3 --json` for
|
|
125
|
+
every finding. Only `fixed` counts. Then `jevitate ledger add <run>.result.json <fp> --json` and,
|
|
126
|
+
when the human asks, `jevitate regression capture --from <run>.json --result <run>.result.json --fingerprint <fp> --id <id> --json`.
|
|
127
|
+
4. Re-sweep the jobs the fixes touched, and compare: `jevitate diff <runA> <runB> --json`.
|
|
128
|
+
|
|
129
|
+
## 8. Budgets and machine limits
|
|
130
|
+
|
|
131
|
+
- Every run has a cap: `--max-actions` (and `--max-decisions` for model-driven runs). Start at
|
|
132
|
+
40–60 per mission; raise it only for a run that ended `exhausted` on a real job.
|
|
133
|
+
- Run missions one at a time. Check free memory before each; a browser killed by the OOM killer
|
|
134
|
+
reads as `crashed`, not as an app defect.
|
|
135
|
+
- Mark paid controls with `--paid <pattern>` so retries and hang replays never repeat them. Never
|
|
136
|
+
add `--allow-destructive` or `--allow-writes` unless the human asked.
|
|
137
|
+
|
|
138
|
+
## 9. CI: promoted Journeys + a check suite
|
|
139
|
+
|
|
140
|
+
Put every promoted Journey, the invariants, the key goals and a few anchored missions in a suite,
|
|
141
|
+
and gate with `jevitate check --suite <suite.json> --target-build <sha> --json` (MCP `run_check`;
|
|
142
|
+
see `jevitate-ci-check`). Tag the release candidate's clean run as a baseline:
|
|
143
|
+
`jevitate baseline tag <name> <runs...>`. In CI, exit 1 fails the build, and exit 2 or 64 is a
|
|
144
|
+
broken check, never green.
|
|
145
|
+
|
|
146
|
+
## 10. The gate
|
|
147
|
+
|
|
148
|
+
Release only when: every job in the catalog has a promoted Journey that passes; every finding is
|
|
149
|
+
`fixed` by verify-fix (with a ledger entry or regression); the final round ran on the release
|
|
150
|
+
build; and no `inconclusive` run was counted as a pass. Anything short of that, report it as open.
|
|
151
|
+
|
|
152
|
+
## Templates
|
|
153
|
+
|
|
154
|
+
Job spec (one per job, `.jevitate/campaign/jobs.json`) — the catalog you plan from:
|
|
155
|
+
|
|
156
|
+
```json
|
|
157
|
+
[{
|
|
158
|
+
"id": "invite-teammate",
|
|
159
|
+
"persona": "admin",
|
|
160
|
+
"storageState": "~/.jevitate/sessions/admin.json",
|
|
161
|
+
"goal": "invite a new teammate by email and see them listed as pending",
|
|
162
|
+
"success": ["textIncludes:role=table|Pending", "responseStatus:POST /api/invites=2xx"],
|
|
163
|
+
"preconditions": ["an org with a free seat"],
|
|
164
|
+
"mutates": true,
|
|
165
|
+
"order": 3
|
|
166
|
+
}]
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Anchors live on the promoted Journey itself (`metadata.anchors`, listed by `jevitate journey anchors <id>`):
|
|
170
|
+
|
|
171
|
+
```json
|
|
172
|
+
"anchors": [{ "name": "invite-form", "step": 4, "description": "the filled invite form", "probes": ["double submit", "invite into another org id"] }]
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
Campaign spec (`.jevitate/campaign/campaign.json`, for `jevitate campaign run`):
|
|
176
|
+
|
|
177
|
+
```json
|
|
178
|
+
{
|
|
179
|
+
"version": 1,
|
|
180
|
+
"name": "release-candidate",
|
|
181
|
+
"env": "staging",
|
|
182
|
+
"fixtures": "restore.json",
|
|
183
|
+
"maxRuns": 40,
|
|
184
|
+
"maxActions": 60,
|
|
185
|
+
"jobs": [
|
|
186
|
+
{ "id": "invite-teammate", "journey": "invite-teammate", "storageState": "~/.jevitate/sessions/admin.json",
|
|
187
|
+
"params": { "email": "new@example.test" }, "anchors": ["invite-form"], "strategies": ["adversarial", "exploratory"] },
|
|
188
|
+
{ "id": "checkout", "journey": "checkout", "anchors": ["payment", 6], "strategies": ["adversarial"], "maxActions": 40 }
|
|
189
|
+
]
|
|
190
|
+
}
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
Issue register (one row per deduped finding, `.jevitate/campaign/register.md`):
|
|
194
|
+
|
|
195
|
+
```text
|
|
196
|
+
| fingerprint | job | layer (fe/be) | severity | outcome | evidence | status (open/fixed/verified) | regression |
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
## What you must never do
|
|
200
|
+
|
|
201
|
+
- Never promote a Journey, approve an inbox item or capture a regression on your own. Ask.
|
|
202
|
+
- Never count `inconclusive`, `intermittent` or a single clean replay as passing.
|
|
203
|
+
- Never widen `--allow` beyond what the human authorized, and never test production.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: jevitate-ux-review
|
|
3
|
-
description: Produce ranked, cited, advisory usability findings for a flow — offline over a saved Recording (`jevitate ux <recording> --app-class`) or live
|
|
3
|
+
description: Produce ranked, cited, advisory usability findings (preview) for a flow — offline over a saved Recording (`jevitate ux <recording> --app-class`, MCP `ux_review`) or live on an authorized target (`jevitate explore --strategy usability --goal --app-class`, MCP `run_exploration`). Use when the user wants a UX or usability critique, not a pass/fail test. Findings are advisory and never gate a run.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
You produce a UX review: ranked, cited, evidence-anchored usability findings for
|
|
@@ -9,22 +9,40 @@ fails a run or a build. Every finding is calibrated to an app class (a consumer
|
|
|
9
9
|
signup and an internal admin tool are held to different bars) and cites a rubric
|
|
10
10
|
item; the platform structurally cannot emit an uncited finding.
|
|
11
11
|
|
|
12
|
+
PREVIEW (0.2.0): these findings are a preview feature. The independent quality
|
|
13
|
+
grader is not yet calibrated across apps, and how findings are grouped/deduped
|
|
14
|
+
is still being redesigned (issues #133, #198). `report.preview` is `true` and
|
|
15
|
+
`report.headline` says so; tell the human this is a preview, not a finished
|
|
16
|
+
verdict.
|
|
17
|
+
|
|
12
18
|
## Two modes — pick by what you have
|
|
13
19
|
|
|
14
20
|
- OFFLINE, over a saved Recording — `jevitate ux <recording.json> --app-class
|
|
15
|
-
<class> [--persona <p>] [--job "<text>"] [--out <dir>] --
|
|
21
|
+
<class> [--persona <p>] [--job "<text>"] [--out <dir>] [--min-confidence <n>]
|
|
22
|
+
[--show <labels>] --real --json`. Analyzes a
|
|
16
23
|
Recording you already captured (e.g. from `jevitate-record` or a successful
|
|
17
24
|
`jevitate explore`). It launches no browser. `--app-class` is REQUIRED
|
|
18
|
-
(e.g. `consumer`, `admin`, `internal`); `--job` sharpens relevance.
|
|
25
|
+
(e.g. `consumer`, `admin`, `internal`); `--job` sharpens relevance. It reads
|
|
26
|
+
the artifacts next to `<stem>.recording.json` automatically: the live
|
|
27
|
+
usability run's evidence sidecar `<stem>.evidence.json` (the screens as
|
|
28
|
+
analyzed plus the run signals), `<stem>.result.json` or
|
|
29
|
+
`<stem>.transcript.json`, and `<stem>.screens/`. With the sidecar, offline
|
|
30
|
+
review reproduces the live run's findings. Without it, `report.evidenceCaveats`
|
|
31
|
+
names what could not be checked. `--evidence` and `--result` override the
|
|
32
|
+
discovered files.
|
|
19
33
|
- LIVE, against an authorized target — `jevitate explore --strategy usability
|
|
20
34
|
--url <authorized-url> --goal "<the job>" --app-class <class> [--allow
|
|
21
|
-
<origin>] [--secret <value>] [--max-actions <n>] --
|
|
35
|
+
<origin>] [--secret <value>] [--max-actions <n>] [--min-confidence <n>]
|
|
36
|
+
[--show <labels>] --real --json`. Drives the
|
|
22
37
|
exploration loop toward the job (Jev makes the browsing decisions, not you)
|
|
23
38
|
and reviews each observed screen against the rubric as it goes. Needs a
|
|
24
39
|
gateway: `--real` (live, after `jevitate ai setup`) or `--fake-ai` (a
|
|
25
40
|
deterministic pipeline smoke that will NOT produce a real review). No
|
|
26
41
|
selection fails closed.
|
|
27
42
|
|
|
43
|
+
Over MCP: `ux_review` is the offline mode (`recording`, `appClass`, …); `run_exploration`
|
|
44
|
+
with `strategy: "usability"` is the live mode. Usability is not a queueable mission strategy.
|
|
45
|
+
|
|
28
46
|
Prefer OFFLINE when a Recording already exists — it is cheaper, deterministic,
|
|
29
47
|
and touches no live site. Use LIVE only when there is no Recording and the human
|
|
30
48
|
authorized the target.
|
|
@@ -43,21 +61,93 @@ authorized the target.
|
|
|
43
61
|
authorized. Pass real secrets/PII via `--secret` so they stay out of every
|
|
44
62
|
model call.
|
|
45
63
|
|
|
64
|
+
## Filtering what is shown
|
|
65
|
+
|
|
66
|
+
- `--min-confidence <0..1>` (also `JEVITATE_UX_MIN_CONFIDENCE`, or `ux.minConfidence` in
|
|
67
|
+
`~/.jevitate/config.json`; default 0.3): findings below it are suppressed.
|
|
68
|
+
- `--show <labels>` (also `JEVITATE_UX_SHOW`, or `ux.show` in the config): an
|
|
69
|
+
opt-in filter on the quality grade. The grades are `actionable`,
|
|
70
|
+
`relevant-minor`, `generic` and `wrong`. The default shows ALL grades. The
|
|
71
|
+
grader is not calibrated (no held-out, multi-rater kappa ≥ 0.4 off its tuning
|
|
72
|
+
app), so it labels findings and does not hide them. Each finding carries its
|
|
73
|
+
grade in `quality`, and `report.qualityFiltered` says whether a filter was
|
|
74
|
+
applied. `--show actionable,relevant-minor` restores the old filter.
|
|
75
|
+
- `--max-findings-per-page <n>` (also `JEVITATE_UX_MAX_FINDINGS_PER_PAGE`, or
|
|
76
|
+
`ux.maxFindingsPerPage` in the config; default 5): caps findings per route,
|
|
77
|
+
highest-confidence first (0.2.0, issue #198 interim — a full redesign is
|
|
78
|
+
tracked there). The rest are counted in `report.suppressed` as
|
|
79
|
+
`per-page-cap`, never dropped silently.
|
|
80
|
+
- Nothing is dropped silently: every suppressed candidate is counted in
|
|
81
|
+
`report.suppressed`. Change a filter only when the human asks, and say which
|
|
82
|
+
filter you changed.
|
|
83
|
+
|
|
46
84
|
## Reading the result
|
|
47
85
|
|
|
48
|
-
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
86
|
+
- Lead with `report.headline`. It gives the findings grounded in observed run
|
|
87
|
+
behavior, the heuristic-only count in the appendix, and "N suppressed (by
|
|
88
|
+
rubric item: …)".
|
|
89
|
+
- `report.findings` holds only findings with behavioral evidence:
|
|
90
|
+
- run signals (tier `signal`): a hung request, a stuck job still offered
|
|
91
|
+
again, a duplicate write or create, a failed submit with no or only generic
|
|
92
|
+
error copy, a repeated assistant reply, an inert control, an internal id,
|
|
93
|
+
or a URL whose page shows another route's content. Each cites its steps,
|
|
94
|
+
requests, text and screenshot in `signal`.
|
|
95
|
+
- rubric findings grounded in the friction the run hit on that screen: a
|
|
96
|
+
backtrack, a retry, a dead end, a long wait, an error, an abandoned field,
|
|
97
|
+
or the goal not reached. `journeyEvidence` gives the kind and step range.
|
|
98
|
+
- Their `severity` and `impact` come from the observed impact on the job:
|
|
99
|
+
`blocked` > `slowed` > `confused`. Findings on the same friction point are
|
|
100
|
+
collapsed into one, with the others listed in `contributing`.
|
|
101
|
+
- `report.heuristicAppendix` holds rubric findings with no observed friction.
|
|
102
|
+
They are screen-level heuristics only, capped at `info`. Mention them as an
|
|
103
|
+
appendix, never as the review's findings.
|
|
104
|
+
- Each finding is specific and grounded:
|
|
105
|
+
- `observation`: what is wrong, naming the control, label or text, relative to
|
|
106
|
+
the job.
|
|
107
|
+
- `userImpact`: what it costs the user.
|
|
108
|
+
- `recommendation`: a concrete change to that control or text.
|
|
109
|
+
- `controls` and `quotes`: the implicated controls and verbatim on-screen text.
|
|
110
|
+
Independent code checked that both exist on the screen.
|
|
111
|
+
- `evidenceRefs`: only those controls, not every control on the page.
|
|
112
|
+
- `route`, `screenId`, `occurrences` and `screenIds`: where it was seen. The
|
|
113
|
+
same item on the same route and controls is one finding with a count.
|
|
114
|
+
- `severity` (`info` | `minor` | `major`) and a rubric `citation` (`source` +
|
|
115
|
+
`ref`).
|
|
116
|
+
- `quality`: the independent grader's label and its confidence.
|
|
117
|
+
- `confidence` with its `confidenceBasis`: violation × applicability ×
|
|
118
|
+
grounding × agreement. Quote the observation and recommendation; don't
|
|
119
|
+
paraphrase them into generic heuristic advice.
|
|
120
|
+
- `contributing` (0.2.0, issue #198 interim): DIFFERENT rubric items that
|
|
121
|
+
fired on the SAME control on the SAME route are collapsed into one
|
|
122
|
+
finding — the highest-confidence one leads, the rest are listed here with
|
|
123
|
+
their own `rubricItemId`, `citation`, `observation` and `occurrences`. Read
|
|
124
|
+
it as "these principles all flagged the same thing", not as noise to
|
|
125
|
+
ignore — mention the other principles when it matters.
|
|
126
|
+
- Rank your summary the way the report does: by `impact`, then severity. Lead
|
|
127
|
+
with what blocked the job.
|
|
128
|
+
- `report.suppressed` counts what was not shown: `byReason`, `byRubricItem` and
|
|
129
|
+
`byRubricItemRoute`. The reasons are:
|
|
130
|
+
- `ungrounded`: named no control or text.
|
|
131
|
+
- `rejected-evidence`: cited a control or text that is not on the screen.
|
|
132
|
+
- `not-confirmed`: the specifics step found no concrete violation.
|
|
133
|
+
- `below-min-confidence`
|
|
134
|
+
- `quality-policy`: a grade outside an explicit `--show` filter.
|
|
135
|
+
- `per-page-cap` (0.2.0, issue #198 interim): the route already has
|
|
136
|
+
`--max-findings-per-page` findings shown; this one ranked lower.
|
|
137
|
+
Say how many were suppressed. `clean: true` only happens with zero findings
|
|
138
|
+
AND zero suppressed, so never describe a report with suppressions as "no
|
|
139
|
+
issues".
|
|
140
|
+
- `coverage.notApplicable` lists items whose rubric precondition did not hold,
|
|
141
|
+
for example choice overload on a 2-control screen. Those items are evaluated,
|
|
142
|
+
not skipped.
|
|
143
|
+
- Report findings as ADVISORY recommendations, with their citations. They are
|
|
144
|
+
never defects that block shipping. Never restate a finding without the rubric
|
|
145
|
+
citation the tool attached; the citation is what makes it more than an opinion.
|
|
56
146
|
- An analysis that FAILED (`E_UX_ANALYSIS`) is a real failure to surface
|
|
57
|
-
honestly
|
|
58
|
-
Offline review over a bare Recording is coverage-limited
|
|
59
|
-
the Recording captured
|
|
60
|
-
the Recording is thin, rather than implying the flow is fully vetted.
|
|
147
|
+
honestly. Never fabricate a clean "no issues found" report from a failed run.
|
|
148
|
+
Offline review over a bare Recording is coverage-limited: it sees only what
|
|
149
|
+
the Recording captured, with no live a11y tree and no runtime contrast. When
|
|
150
|
+
the Recording is thin, say so rather than implying the flow is fully vetted.
|
|
61
151
|
|
|
62
152
|
## What you must never do
|
|
63
153
|
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: jevitate-verify-fix
|
|
3
|
+
description: Reproduce a jevitate finding by its fingerprint and prove a fix (`jevitate verify-fix`, MCP `verify_fix`); attach evidence (`--evidence-video` clips, `--record-video` before/after, `--screenshots`); keep findings re-checkable (`jevitate ledger`); lock a bug in as a committed regression (`jevitate regression capture|run`). Use after a run reports a defect, when fixing one, or when a PR needs proof a bug is gone. Not for finding new bugs (jevitate-explore).
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Every finding jevitate reports has a stable `fingerprint` and its own reproduction: the Recording
|
|
7
|
+
and the step to replay up to. Your job is to replay it, read the verdict literally, and attach
|
|
8
|
+
the evidence. No model decides any of these verdicts.
|
|
9
|
+
|
|
10
|
+
## Reproduce or verify a fix
|
|
11
|
+
|
|
12
|
+
- CLI: `jevitate verify-fix --result <run>.result.json --fingerprint <fp> [--replays 3] --json`.
|
|
13
|
+
The result file is `<stem>.result.json` beside the run's Recording (under `.jevitate/logs/<date>/`
|
|
14
|
+
by default). The fingerprint is from the result's `defects[]` (or `hangs[]`).
|
|
15
|
+
- MCP: `verify_fix({ id, fingerprint, replays?, recordVideo?, screenshots?, storageState? })`,
|
|
16
|
+
where `id` is the result stem (e.g. `adversarial-2026-09-23T00-00-00-000Z`) or a finished
|
|
17
|
+
`queue_exploration` missionId.
|
|
18
|
+
- Verdicts and exit codes: `fixed` (0, the signal was absent on every replay) · `still-reproduces`
|
|
19
|
+
(1) · `inconclusive` (2, a replay couldn't reach the step, e.g. a missing session or a changed
|
|
20
|
+
page) · `intermittent` (4, it fired on some replays) · 64 (unusable input). Only `fixed` means
|
|
21
|
+
fixed. A single clean replay isn't evidence, so never lower `--replays` below 3 to make a check
|
|
22
|
+
pass.
|
|
23
|
+
- An authenticated app needs `--storage-state <file>` (a path, never cookie contents), or the
|
|
24
|
+
target's session in `~/.jevitate/targets.json`. A finding recorded at a viewport or device
|
|
25
|
+
replays at that emulation. `--viewport`/`--device` that differ are refused unless
|
|
26
|
+
`--allow-emulation-override` is passed.
|
|
27
|
+
|
|
28
|
+
## Evidence for a PR or an issue
|
|
29
|
+
|
|
30
|
+
- At discovery, add `--evidence-video` to `jevitate explore` (or `mission queue`). Each defect
|
|
31
|
+
(up to 5) then gets a captioned clip of its minimal repro with the failing step marked by the
|
|
32
|
+
real signal, plus before and at screenshots, under `<run>.evidence/<fingerprint>/`. The result
|
|
33
|
+
gets `defects[].evidence`, and the issue drafts link to them.
|
|
34
|
+
- At fix time, `jevitate verify-fix ... --record-video` writes the before/after pair:
|
|
35
|
+
`evidence.before` (the run's own clip) and `evidence.after` (a captioned replay now, ending on the
|
|
36
|
+
verdict).
|
|
37
|
+
- `--screenshots [screens|steps|<dir>]` writes masked screenshots and an `index.md`. It works on
|
|
38
|
+
`explore`, `verify-fix`, `journey run` and `mission queue`.
|
|
39
|
+
- Secrets are masked in pixels. A capture whose mask can't be proven isn't written
|
|
40
|
+
(`screenshotsSkipped`/`captureSkips` say why). Report that. Never work around it.
|
|
41
|
+
- Link to the files. GitHub's API can't upload media, so commit them under `.jevitate/` or
|
|
42
|
+
publish them as CI artifacts.
|
|
43
|
+
|
|
44
|
+
## Keep it re-checkable: the ledger
|
|
45
|
+
|
|
46
|
+
Run output under `.jevitate/logs/` is pruned by retention. To re-check a finding months later:
|
|
47
|
+
|
|
48
|
+
- `jevitate ledger add <run>.result.json <fp> [--ticket JEV-123] --json` stores the repro material
|
|
49
|
+
(no sessions, no page text) in `.jevitate/regressions/ledger/<fp>.json`. Commit it with the fix.
|
|
50
|
+
- After that, `jevitate verify-fix <fp> --json` works without `--result`, and
|
|
51
|
+
`jevitate ledger verify [--ticket JEV-123] --json` re-checks every entry. It exits with the worst
|
|
52
|
+
verdict. An empty match is exit 2 ("nothing verified"), never a pass.
|
|
53
|
+
- Over MCP, the `ledger` tool takes `add`/`list`/`verify` as its action.
|
|
54
|
+
|
|
55
|
+
## Lock it in: a committed regression
|
|
56
|
+
|
|
57
|
+
- `jevitate regression capture --from <run>.json --result <run>.result.json [--fingerprint <fp>]
|
|
58
|
+
--id <id> --json` replays the failure `--attempts` times (default 3) and commits
|
|
59
|
+
`<id>.recording.json` and `<id>.meta.json` to `.jevitate/regressions/`. A failure that doesn't
|
|
60
|
+
reproduce every time is labeled flaky and isn't committed. It captures a failed goal `--success`
|
|
61
|
+
check (minimized), an app-caused failed action, a failed network check, a declared-invariant
|
|
62
|
+
violation, or a Recording whose last `expect` fails.
|
|
63
|
+
- A hard-signal fingerprint (HTTP 5xx, a hang, a server-log defect) is refused (64) with a pointer
|
|
64
|
+
to `ledger add`/`verify-fix`. To make one a regression, declare the rule it breaks as an
|
|
65
|
+
invariant (`--invariants`) and capture the invariant violation.
|
|
66
|
+
- `jevitate regression run <id> [--env <name>] --json` exits 1 (`reproduces`) while the bug is
|
|
67
|
+
there and 0 (`fixed`) once it's gone. An existing `--id` is refused unless `--force` is passed.
|
|
68
|
+
Only overwrite when the human asks.
|
|
69
|
+
- MCP: `regressions` with `action: "capture"` or `action: "run"`.
|
|
70
|
+
|
|
71
|
+
## What you must never do
|
|
72
|
+
|
|
73
|
+
- Never report `inconclusive` or `intermittent` as fixed, and never retry until you happen to get a
|
|
74
|
+
clean replay.
|
|
75
|
+
- Never edit a result file, a ledger entry or a committed regression to change a verdict.
|
|
76
|
+
- Never capture or commit a regression the human didn't ask for. Recommend it and say why.
|