@cassiomc1/forgeloop 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.cursor/rules/project-loop.mdc +18 -0
- package/.forgeloop/.gitignore +2 -0
- package/.github/copilot-instructions.md +16 -0
- package/AGENTS.md +16 -0
- package/AGENT_COMPATIBILITY.md +147 -0
- package/CLAUDE.md +14 -0
- package/CONTRACT_COVERAGE.md +27 -0
- package/DELEGATION_PROTOCOL.md +91 -0
- package/ENG/accessibility-eng.md +155 -0
- package/ENG/clean-code-eng.md +223 -0
- package/ENG/design-code-eng.md +511 -0
- package/ENG/games-code-design-web-eng.md +751 -0
- package/ENG/perf-code-eng.md +441 -0
- package/ENG/premium-sites-studio-eng.md +320 -0
- package/ENG/sec-code-eng.md +706 -0
- package/ENG/test-code-eng.md +257 -0
- package/EXECUTION_STATE.md +107 -0
- package/GUIDE_ROUTER.md +274 -0
- package/LICENSE +21 -0
- package/LICENSE-DOCS.md +13 -0
- package/LOOP_ENGINEERING.md +551 -0
- package/LOOP_SYSTEM_DESIGN.md +394 -0
- package/ORCHESTRATOR_INTEGRATION.md +106 -0
- package/PROJECT_PROFILE.md +124 -0
- package/QUALITY_SCORECARD.md +54 -0
- package/README.md +492 -0
- package/TERMINOLOGY.md +21 -0
- package/THIRD_PARTY_NOTICES.md +129 -0
- package/THREAT_MODEL.md +35 -0
- package/package.json +51 -0
- package/schemas/delegated-result.schema.json +33 -0
- package/schemas/evidence.schema.json +15 -0
- package/schemas/execution-receipt.schema.json +46 -0
- package/schemas/routing-input.schema.json +17 -0
- package/schemas/routing-result.schema.json +17 -0
- package/schemas/task-brief.schema.json +24 -0
- package/schemas/work-state.schema.json +46 -0
- package/src/cli.js +341 -0
- package/src/commands/clear-state.js +11 -0
- package/src/commands/doctor.js +165 -0
- package/src/commands/init.js +42 -0
- package/src/commands/inspect.js +17 -0
- package/src/commands/route.js +32 -0
- package/src/commands/status.js +29 -0
- package/src/commands/update.js +109 -0
- package/src/commands/validate-protocol.js +133 -0
- package/src/commands/validate-receipt.js +19 -0
- package/src/commands/validate-state.js +30 -0
- package/src/core/agent-support.js +89 -0
- package/src/core/conformance.js +133 -0
- package/src/core/delegation.js +283 -0
- package/src/core/evidence.js +56 -0
- package/src/core/filesystem.js +122 -0
- package/src/core/inspect.js +115 -0
- package/src/core/json-safety.js +54 -0
- package/src/core/manifest.js +75 -0
- package/src/core/protocol.js +81 -0
- package/src/core/receipt.js +129 -0
- package/src/core/repository.js +19 -0
- package/src/core/router.js +296 -0
- package/src/core/schema-validation.js +179 -0
- package/src/core/templates.js +56 -0
- package/src/core/work-state.js +471 -0
|
@@ -0,0 +1,257 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: test-code-eng
|
|
3
|
+
language: en
|
|
4
|
+
description: "Risk-based testing strategy and tools for modern languages and platforms."
|
|
5
|
+
version: "2026.09"
|
|
6
|
+
last-reviewed: "2026-08-10"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Risk-Based Testing Guide by Language and Technology
|
|
10
|
+
|
|
11
|
+
> Practical instructions for selecting, running, and approving software tests according to risk, stack, compatibility, and operational evidence.
|
|
12
|
+
|
|
13
|
+
**Related documents**: for general clean code principles (names, small functions, formatting), see [`clean-code-eng.md`](./clean-code-eng.md). For security, including SAST/DAST, see [`sec-code-eng.md`](./sec-code-eng.md). For accessibility, use the repository's specific guide together with [WCAG 2.2](https://www.w3.org/TR/WCAG22/). For HTML video and motion, see [HyperFrames](https://hyperframes.heygen.com). This file is the canonical reference for deciding the testing strategy by language and platform.
|
|
14
|
+
|
|
15
|
+
**Tooling policy**: identify the stack, the stage, and the applicable checks; prefer an already available equivalent that produces compatible evidence. Ask for authorization before installing a tool or changing the environment. If no safe equivalent exists, record the required check as blocked and never claim that it passed. Do not install merely optional resources.
|
|
16
|
+
|
|
17
|
+
## Start with risk, not a tool
|
|
18
|
+
|
|
19
|
+
Before writing or selecting tests, record:
|
|
20
|
+
|
|
21
|
+
- the protected asset, journey, or rule;
|
|
22
|
+
- failure impact on users, revenue, data, security, accessibility, and operations;
|
|
23
|
+
- regression probability, change complexity, and incident history;
|
|
24
|
+
- supported platforms, versions, devices, permissions, and network conditions;
|
|
25
|
+
- observable oracle, environment, data, and owner of the result;
|
|
26
|
+
- required evidence and reassessment date.
|
|
27
|
+
|
|
28
|
+
Classify each item locally. The labels below do not impose a fixed number of tests:
|
|
29
|
+
|
|
30
|
+
| Class | Example | Minimum approval evidence |
|
|
31
|
+
| --- | --- | --- |
|
|
32
|
+
| Critical | authentication, payment, data loss, authorization, destructive migration | unit/property tests for rules, realistic integration, contract, and an E2E journey; manual review when the oracle requires judgment |
|
|
33
|
+
| High | frequent flow, external integration, compatibility, or accessibility | relevant branch tests, integration/contract, and smoke on the affected platform |
|
|
34
|
+
| Standard | content, presentation, or reversible low-impact logic | checks focused on the change and regression proportional to its reach |
|
|
35
|
+
|
|
36
|
+
The testing pyramid is a cost heuristic, not a quota. Keep most confidence in fast, deterministic tests, but use integration, contract, UI, performance, security, and manual evaluation when only those layers can observe the risk.
|
|
37
|
+
|
|
38
|
+
## Coverage and suite strength
|
|
39
|
+
|
|
40
|
+
- Do not adopt universal coverage percentages or require one test per function. Define the repository contract from a measured baseline, criticality, language, and maintenance cost.
|
|
41
|
+
- Observe repository coverage, changed lines, branches/conditions, and critical paths separately. A change should not lower the applicable baseline without a rationale, owner, and accepted risk.
|
|
42
|
+
- For critical rules, list cases, boundaries, invalid states, and properties. Line coverage does not prove that assertions distinguish correct from incorrect behavior.
|
|
43
|
+
- Run **mutation testing** on modules where the cost is justified, using [Stryker](https://stryker-mutator.io/), [PIT](https://pitest.org/), or a stack-equivalent tool. Compare the mutation score with the project's own baseline and investigate relevant surviving mutants; do not turn a generic number into a universal target.
|
|
44
|
+
- Every bug fix should retain a regression test that fails in the defective state and passes with the fix, unless an impossibility is documented.
|
|
45
|
+
|
|
46
|
+
## Tool selection matrix
|
|
47
|
+
|
|
48
|
+
A tool is a candidate, not a ranking winner. Record the decision in the repository and use this order:
|
|
49
|
+
|
|
50
|
+
1. Eliminate options incompatible with the runtime, framework, browser, operating system, architecture, security policy, or CI.
|
|
51
|
+
2. In an existing project, keep the adopted tool when it covers the risk and receives maintenance compatible with the product lifecycle; migration needs a measured benefit and exit plan.
|
|
52
|
+
3. Run a proof with a critical journey. Measure execution and diagnosis time, stability, parallelism, artifacts, runner cost, and maintenance effort.
|
|
53
|
+
4. Pin versions, commands, supported platforms, owner, and reassessment date. Do not base the choice on satisfaction, popularity, or adoption from a single survey.
|
|
54
|
+
|
|
55
|
+
| Context | Candidates with official documentation | Reproducible decision basis |
|
|
56
|
+
| --- | --- | --- |
|
|
57
|
+
| Node.js/TypeScript | [`node:test`](https://nodejs.org/api/test.html), [Vitest](https://vitest.dev/guide/), [Jest](https://jestjs.io/docs/getting-started) | existing framework/build, ESM/CJS, TypeScript, mocks, watch, coverage, and CI runtime |
|
|
58
|
+
| Components and browser | [Testing Library](https://testing-library.com/docs/), [Playwright](https://playwright.dev/docs/intro), [Cypress](https://docs.cypress.io/) | required browsers, isolation, failure artifacts, parallelism, accessibility, and app compatibility |
|
|
59
|
+
| Python | [pytest](https://docs.pytest.org/), [`unittest`](https://docs.python.org/3/library/unittest.html) | dependency policy, fixtures/plugins, Python versions, and existing integration |
|
|
60
|
+
| .NET | [xUnit.net](https://xunit.net/), [NUnit](https://docs.nunit.org/), [MSTest](https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-intro) | .NET version, runner/IDE, fixture model, assertions, and existing projects |
|
|
61
|
+
| Java/JVM | [JUnit](https://docs.junit.org/current/overview.html), [TestNG](https://testng.org/) | Maven/Gradle build, extensions, parallelism, integration, and existing conventions |
|
|
62
|
+
| Go | [`testing`](https://pkg.go.dev/testing), [Testify](https://pkg.go.dev/github.com/stretchr/testify) | sufficient standard library or a proven need for helpers, suites, and mocks |
|
|
63
|
+
| Ruby | [RSpec](https://rspec.info/documentation/), [Minitest](https://ruby-doc.org/3.4/gems/minitest/Minitest.html) | Rails/Ruby conventions, fixtures, DSL, and existing suite |
|
|
64
|
+
| PHP | [PHPUnit](https://docs.phpunit.de/), [Pest](https://pestphp.com/docs/installation) | PHP/framework version, plugins, and migration cost |
|
|
65
|
+
| Real services | [Testcontainers](https://testcontainers.com/), stack-native harness | required fidelity, container availability, time, and CI isolation |
|
|
66
|
+
| HTTP/event contracts | [Pact](https://docs.pact.io/), OpenAPI/JSON Schema/GraphQL schema | consumer/provider ownership, versioning, and compatibility across deployments |
|
|
67
|
+
| Performance | [k6](https://grafana.com/docs/k6/latest/), [Gatling](https://docs.gatling.io/), [JMeter](https://jmeter.apache.org/usermanual/), or [Locust](https://docs.locust.io/) | protocol, load model, volume, distribution, metrics, and team operability |
|
|
68
|
+
|
|
69
|
+
Mocks, stubs, and fakes belong in isolated tests. For integration, prefer disposable dependencies equivalent to production when this reduces relevant differences. Typing, linting, and static analysis complement tests; they do not replace executing behavior.
|
|
70
|
+
|
|
71
|
+
## Operational matrix by domain and framework
|
|
72
|
+
|
|
73
|
+
Run only the rows applicable to the product and recorded risk. A tool name without a hypothesis, oracle, and gate is not a strategy.
|
|
74
|
+
|
|
75
|
+
| Context | Risk requiring evidence | Operational approach and gate | Official tools/sources |
|
|
76
|
+
| --- | --- | --- | --- |
|
|
77
|
+
| APIs and schemas | drift between implementation and contract, incompatible change, undocumented error/authorization | validate requests/responses and negative cases against the schema version; run consumer/provider contracts; block breaking changes without a compatible transition and provider verification | [OpenAPI](https://spec.openapis.org/oas/), [JSON Schema](https://json-schema.org/learn), [GraphQL](https://spec.graphql.org/), and [Pact](https://docs.pact.io/) |
|
|
78
|
+
| ASP.NET Core | routing, middleware, authentication, serialization, and persistence differences | test rules in isolation; use `WebApplicationFactory`/`TestServer` for the pipeline and a disposable real dependency when the database matters; approve only with status, headers, auth, persistence, and critical failures observed | [Microsoft — integration tests](https://learn.microsoft.com/aspnet/core/test/integration-tests) and [Testcontainers for .NET](https://dotnet.testcontainers.org/) |
|
|
79
|
+
| Spring Boot | a slice masks configuration, incorrect profile/security, divergent transaction or integration | use slices for focused feedback and `@SpringBootTest`/random port when risk crosses layers; validate profile, security chain, transactions, migration, and relevant dependencies | [Spring Boot — testing](https://docs.spring.io/spring-boot/reference/testing/spring-boot-applications.html) and [Testcontainers](https://java.testcontainers.org/) |
|
|
80
|
+
| Rails | callbacks, validations, routes, jobs, mailers, and system flow diverge | combine model/request/job/mailbox/system tests according to risk and use the supported database adapter in integration; the gate observes the rule, response, side effects, and at least the affected critical journey | [Rails — Testing Rails Applications](https://guides.rubyonrails.org/testing.html) |
|
|
81
|
+
| Laravel | middleware, validation, policies/auth, queues, and database are not exercised by a pure unit test | separate unit from feature/HTTP/database tests; fake only the boundary outside the oracle and verify persisted effects; approve a critical endpoint with applicable auth, validation, response, database, and job/event evidence | [Laravel — testing](https://laravel.com/docs/12.x/testing), [HTTP tests](https://laravel.com/docs/12.x/http-tests), and [database testing](https://laravel.com/docs/12.x/database-testing) |
|
|
82
|
+
| HTML and HyperFrames | invalid markup, overflow, nondeterministic timing, or an incorrect critical frame | validate HTML; run `hyperframes lint`/`hyperframes check` when exposed by the pinned version; preview/render declared formats and review critical frames/boundaries; any structural, timing, or layout error blocks | [WHATWG — validators](https://whatwg.org/validator/), [Nu HTML Checker](https://validator.w3.org/nu/), and [HyperFrames](https://hyperframes.heygen.com) |
|
|
83
|
+
| CSS and visual regression | cascade, responsiveness, theme, font, or environment changes layout/behavior | run Stylelint and semantic/interaction assertions; compare screenshots only in a controlled environment and risk matrix; diffs require explicit review and the gate does not depend on a generated CSS snapshot | [Stylelint](https://stylelint.io/user-guide/get-started/) and [Playwright visual comparisons](https://playwright.dev/docs/test-snapshots) |
|
|
84
|
+
| Terraform | valid configuration produces unexpected replace/destroy, cost, exposure, or state | run `fmt`/`validate`, test module assertions, and review `plan`; use a dedicated account/environment for `terraform test` that creates resources; block unexpected actions and confirm cleanup of every test resource | [Terraform validate](https://developer.hashicorp.com/terraform/cli/commands/validate), [test](https://developer.hashicorp.com/terraform/cli/commands/test), and [plan](https://developer.hashicorp.com/terraform/cli/commands/plan) |
|
|
85
|
+
| Kubernetes and containers | a schema accepted locally fails in-cluster, or rollout/probe/rollback/image behavior diverges | use strict validation and server-side dry-run; use an ephemeral cluster when rollout/controller risk justifies it; run build checks and image smoke; the gate covers probes, resources, permissions, applicable rollback, and cleanup | [kubectl apply](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_apply/) and [Docker build checks](https://docs.docker.com/reference/build-checks/) |
|
|
86
|
+
| Ansible | a playbook is not idempotent, makes a destructive change, or reaches an incorrect state | run syntax/check when safe and a Molecule scenario with converge, verify, idempotence, and destroy according to risk; approve only if the second convergence causes no unintended change and the environment is cleaned | [Ansible Molecule](https://docs.ansible.com/projects/molecule/) and [test sequence](https://docs.ansible.com/projects/molecule/usage/) |
|
|
87
|
+
|
|
88
|
+
## Fixtures, cassettes, and oracles
|
|
89
|
+
|
|
90
|
+
- Prefer minimal synthetic data. When fixtures are derived from production, anonymize them irreversibly and review reidentification, retention, and licensing risk before committing.
|
|
91
|
+
- Control clock, locale, timezone, IDs, seeds, and order. Each test creates and removes its own state or uses proven isolation.
|
|
92
|
+
- VCR/VCR.py cassettes are versioned code. Before recording, filter tokens, cookies, authorization headers, parameters, request/response bodies, and all PII; run secret scanning before committing and prohibit automatic recording in CI.
|
|
93
|
+
- Define cassette expiration and rerecording so old responses do not hide contract changes. Recorded playback does not replace periodic tests against a real sandbox.
|
|
94
|
+
- A snapshot approves only a reviewed representation. A generated CSS snapshot is not a primary oracle: combine it with semantic assertions, interaction, and visual regression in relevant viewports/themes. Bulk updates require diff review and a reason.
|
|
95
|
+
|
|
96
|
+
## Accessibility: automation plus human evaluation
|
|
97
|
+
|
|
98
|
+
`axe-core` and Lighthouse detect only an automatable subset. A result with no violations means only that those checks found no failures; it does not prove WCAG conformance. W3C describes automated rules as partial checks and requires human judgment to cover all aspects of the criteria.
|
|
99
|
+
|
|
100
|
+
For each critical journey and reusable component:
|
|
101
|
+
|
|
102
|
+
- run automated `axe-core`/Lighthouse checks on visible states, including modals, errors, and dynamically loaded content;
|
|
103
|
+
- perform manual review against WCAG 2.2 at the product's adopted level;
|
|
104
|
+
- traverse everything with a keyboard and validate focus order/visibility, absence of traps, and focus return;
|
|
105
|
+
- test names, roles, states, announcements, and reading with representative assistive technologies, not just the accessibility tree;
|
|
106
|
+
- validate contrast, zoom, reflow, orientation, reduced motion, and magnification without loss of content or function;
|
|
107
|
+
- on native mobile, test a real device with VoiceOver/TalkBack and platform gestures, focus, external keyboard, text size, rotation, and controls.
|
|
108
|
+
|
|
109
|
+
Record browser/OS, assistive technology and version, WCAG criterion, result, evidence, and evaluator. Critical keyboard, focus, name/role/state, or journey-access defects block approval.
|
|
110
|
+
|
|
111
|
+
## Functional approval and critical journeys
|
|
112
|
+
|
|
113
|
+
An approved journey should cover happy, empty, invalid, denied, interrupted, and recovered states according to risk. Include when applicable:
|
|
114
|
+
|
|
115
|
+
- slow network, lost/resumed connectivity, timeout, idempotent retry, and offline operation;
|
|
116
|
+
- first run, upgrade, logout/login, session expiration, and account switching;
|
|
117
|
+
- permissions denied, granted later, and revoked during use;
|
|
118
|
+
- background/foreground, suspension, termination, and state restoration;
|
|
119
|
+
- locale, timezone, full storage, low memory, and unavailable dependencies;
|
|
120
|
+
- OS, browser, and device combinations defined by the support contract and telemetry, with real devices for risks emulators cannot reproduce.
|
|
121
|
+
|
|
122
|
+
The oracle must observe the user outcome and data invariants, not incidental implementation details.
|
|
123
|
+
|
|
124
|
+
## Performance and resilience
|
|
125
|
+
|
|
126
|
+
Before testing, turn expectations into pass/fail criteria tied to product SLOs or budgets:
|
|
127
|
+
|
|
128
|
+
- p95 and p99 latency per journey/endpoint, sustained throughput, and error rate;
|
|
129
|
+
- CPU, memory, energy, connection, and artifact-size consumption when relevant;
|
|
130
|
+
- volume, operation distribution, concurrency, dataset, and dependency limits;
|
|
131
|
+
- a regression budget relative to a comparable baseline, in addition to absolute SLO limits.
|
|
132
|
+
|
|
133
|
+
Document hardware, region, build, configuration, dataset, and interference. Separate **warm-up** from the measured window. When performance is relevant, use a capacity smoke and comparable baseline as minimum evidence. Add a **spike** only when bursts, rate limits, autoscaling, or recovery are risks; a **soak** only when duration, leaks, pools, queues, cache, or long sessions are hypotheses; stress/breakpoint only when limits, headroom, or failure mode must be known, always in an authorized environment. The frequency of each mode derives from the SLO, load profile, changes, and cost: there is no universal type or cadence. Fail the gate when a versioned threshold is missed; an isolated average does not replace p95/p99 or the distribution.
|
|
134
|
+
|
|
135
|
+
## Mobile and desktop
|
|
136
|
+
|
|
137
|
+
### Mobile
|
|
138
|
+
|
|
139
|
+
- Test logic and components with platform harnesses; cover native journeys with [XCTest/XCUITest](https://developer.apple.com/documentation/xctest), [Espresso](https://developer.android.com/training/testing/espresso), [Detox](https://wix.github.io/Detox/), or [`integration_test`](https://docs.flutter.dev/testing/integration-tests) only when compatible with the stack.
|
|
140
|
+
- Build the version, manufacturer, and device matrix from the support contract, telemetry, and hardware risks. Emulators accelerate feedback; real devices approve sensors, performance, accessibility, permissions, and OS behavior.
|
|
141
|
+
- Exercise install/upgrade, deep links, notifications, interruptions, degraded/offline connectivity, and background/foreground transitions.
|
|
142
|
+
|
|
143
|
+
### Windows
|
|
144
|
+
|
|
145
|
+
- Isolate WinUI/WPF/WinForms/MAUI logic through MVVM/MVU and test it with the adopted .NET runner.
|
|
146
|
+
- As of **2026-08-08**, the path maintained by the Appium ecosystem for Windows UI is the [Appium Windows Driver](https://github.com/appium/appium-windows-driver), installed and versioned as an Appium driver. It still proxies the WinAppDriver server, and its own documentation warns that the Microsoft server has not received regular maintenance for years.
|
|
147
|
+
- Treat a harness that starts **WinAppDriver directly** as legacy: preserve it only when an existing suite requires it, pin the version, and maintain a replacement plan. Do not create a new architecture directly coupled to the legacy server.
|
|
148
|
+
- Validate installer, privileges/UAC, DPI, multiple windows, shortcuts, and post-install smoke on compatible Windows runners; driver limitations must appear as risk, never as a silently omitted check.
|
|
149
|
+
|
|
150
|
+
### macOS and Electron
|
|
151
|
+
|
|
152
|
+
- For native Apple apps, use XCTest/XCUITest according to risk and validate signing, notarization, permissions, menus, shortcuts, and post-install smoke on real macOS.
|
|
153
|
+
- For Electron, select [Playwright Electron](https://playwright.dev/docs/api/class-electron) or [WebdriverIO Electron Service](https://webdriver.io/docs/desktop-testing/electron/) by version compatibility, required capabilities, and CI environment.
|
|
154
|
+
|
|
155
|
+
### Tauri
|
|
156
|
+
|
|
157
|
+
- Test the Rust backend with `cargo test` and the isolated frontend with the adopted web tool.
|
|
158
|
+
- For packaged-app E2E, as of **2026-08-08** Tauri documentation points to WebdriverIO with [`@wdio/tauri-service`](https://webdriver.io/docs/wdio-tauri-service/); the service can use an embedded server or `tauri-driver` on supported platforms. Pin versions and follow the published matrix.
|
|
159
|
+
- Do not describe Playwright attached to a Tauri window as a supported path. If a project retains its own experiment, it needs an owner, per-platform evidence, and explicit acceptance of maintenance cost.
|
|
160
|
+
|
|
161
|
+
## Database migrations and cross-version compatibility
|
|
162
|
+
|
|
163
|
+
- Test migrations against an empty database and a sanitized copy with production-like volume and distribution. Verify time, locks, indexes, constraints, integrity, and recovery after failure.
|
|
164
|
+
- Use **expand-migrate-contract** for incompatible changes: expand the schema without removing the old shape; release code that coexists with both shapes and migrate/backfill data; only after telemetry and the compatibility window, contract the schema in a separate deployment.
|
|
165
|
+
- During rollout, prove compatibility between old/new application and old/expanded schema. Test jobs, consumers, replication, and partial deployment.
|
|
166
|
+
- In production, prefer **roll-forward** by correcting the cause. A `down` is an option only when reversal is implemented, tested, and does not lose data; otherwise mark the migration irreversible and use the recovery runbook.
|
|
167
|
+
- A backup alone is insufficient: validate restore in an isolated environment, RPO/RTO, permissions, and operational sequence before a critical change.
|
|
168
|
+
|
|
169
|
+
## Flaky tests, retries, and quarantine
|
|
170
|
+
|
|
171
|
+
- A test that passes on retry remains **flaky**. Preserve the initial failure and artifacts; a limited retry is for diagnosis/known transients, never to turn the gate green or hide failure rate.
|
|
172
|
+
- Quarantine requires an issue, owner, reason, minimal scope, entry date, and removal deadline. The protected journey needs substitute evidence while the test is outside the gate.
|
|
173
|
+
- Expose flakiness count and rate per test/suite. Exceeding the local budget fails the gate or blocks suite expansion according to the recorded policy.
|
|
174
|
+
- Fix synchronization, isolation, state, dependency, and environment; do not use increasing `sleep` as a solution. Remove a redundant test only with proof of equivalent coverage.
|
|
175
|
+
|
|
176
|
+
## CI cadence and evidence
|
|
177
|
+
|
|
178
|
+
| Event | Expected gates |
|
|
179
|
+
| --- | --- |
|
|
180
|
+
| Every PR | applicable lint/typecheck, impacted unit and integration tests, smoke for affected critical journeys, automated accessibility/security checks, coverage/mutation according to risk |
|
|
181
|
+
| Merge/protected branch | contracts, broader integration, and reproducible artifacts; exception policies validated |
|
|
182
|
+
| Scheduled | broad suites selected by risk: platform matrix, E2E, external sandbox, expensive mutation testing, justified load modes, and extensive scans |
|
|
183
|
+
| Release | installer/upgrade/operational rollback, post-deploy smoke, completed manual accessibility evidence, restore/migration, and release SLO criteria; without that evidence, block the release or record a formal exception with an owner and deadline |
|
|
184
|
+
|
|
185
|
+
A broad suite need not run in full on every PR when its cost reduces feedback without lowering risk. Impact selection must be conservative, auditable, and backed by scheduled execution that detects gaps. A documented command must reproduce each gate; any unexecuted check is `blocked` or `not applicable` with a reason, never `approved`.
|
|
186
|
+
|
|
187
|
+
## Instruction template for CLAUDE.md / AGENTS.md
|
|
188
|
+
|
|
189
|
+
```markdown
|
|
190
|
+
## Testing
|
|
191
|
+
|
|
192
|
+
### Risk contract
|
|
193
|
+
|
|
194
|
+
- Critical journeys and owners: [links]
|
|
195
|
+
- Support contract: [OS/browser/device/runtime]
|
|
196
|
+
- SLOs and budgets: [p95/p99 latency, throughput, errors, resources]
|
|
197
|
+
- Baseline: [repository/change/branch coverage, mutation score, flakiness]
|
|
198
|
+
|
|
199
|
+
### Selection and commands
|
|
200
|
+
|
|
201
|
+
- Tools/versions and stack-compatible rationale: [record]
|
|
202
|
+
- PR smoke: `[command]`
|
|
203
|
+
- Scheduled broad suite: `[command]`
|
|
204
|
+
- Performance/accessibility/migration: `[commands or runbooks]`
|
|
205
|
+
- Domain gates (API/IaC/framework/HTML/CSS): `[applicable commands and evidence]`
|
|
206
|
+
|
|
207
|
+
### Gates
|
|
208
|
+
|
|
209
|
+
- Fixes preserve a red/green regression; changes cover branches and risks.
|
|
210
|
+
- Coverage does not regress against the applicable baseline without documented acceptance.
|
|
211
|
+
- Mutation testing applies to critical modules according to the local budget.
|
|
212
|
+
- Retry does not erase the initial failure; quarantine requires an issue, owner, and expiry.
|
|
213
|
+
- Cassettes contain no tokens, cookies, PII, or secrets in requests/responses.
|
|
214
|
+
- Accessibility combines automation with manual WCAG 2.2/AT evaluation.
|
|
215
|
+
- Migrations use expand-migrate-contract, roll-forward, and rehearsed restore.
|
|
216
|
+
- An unexecuted check is recorded as blocked or not applicable.
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
## Approval checklist
|
|
220
|
+
|
|
221
|
+
- [ ] The risk and protected journey are named, with an owner and oracle.
|
|
222
|
+
- [ ] Tool selection considers stack, compatibility, maintenance, runtime, existing project, and CI.
|
|
223
|
+
- [ ] Coverage uses repository/change/branch/critical-path baselines, without a universal percentage.
|
|
224
|
+
- [ ] For changed critical modules, mutation testing was run or rejected with a cost/risk rationale.
|
|
225
|
+
- [ ] PR smoke and scheduled broad suites have separate commands and evidence.
|
|
226
|
+
- [ ] SLOs/budgets include p95/p99, throughput, error rate, warm-up, and soak/spike where applicable.
|
|
227
|
+
- [ ] Every applicable domain/framework has a recorded risk, approach, tool, and operational gate.
|
|
228
|
+
- [ ] Journeys cover degraded/offline network, permissions, and background/foreground where relevant.
|
|
229
|
+
- [ ] The OS/device matrix includes real hardware where an emulator does not prove the risk.
|
|
230
|
+
- [ ] For affected user interfaces, automated accessibility was complemented by keyboard, AT, zoom/reflow, focus, contrast, and native-mobile checks where applicable.
|
|
231
|
+
- [ ] For Windows desktop applications, Appium Windows Driver was used; direct WinAppDriver use is recorded as legacy.
|
|
232
|
+
- [ ] For Tauri packaged-app E2E, use `@wdio/tauri-service`/`tauri-driver`, not Playwright attached to the window.
|
|
233
|
+
- [ ] Cassettes were redacted and scanned; CSS snapshots are not the primary oracle.
|
|
234
|
+
- [ ] Quarantine has an issue, owner, deadline, and substitute evidence; retries do not mask failures.
|
|
235
|
+
- [ ] Migrations follow expand-migrate-contract, roll-forward, compatibility, and restore; `down` is truly reversible.
|
|
236
|
+
- [ ] Exceptions, blocked checks, and reassessment dates are documented.
|
|
237
|
+
|
|
238
|
+
## Auditable official sources
|
|
239
|
+
|
|
240
|
+
Time-sensitive statements in this revision were verified on **2026-08-08**. Revalidate versions, maintenance, and platform support before adopting or updating a tool.
|
|
241
|
+
|
|
242
|
+
- [W3C — Web Content Accessibility Guidelines (WCAG) 2.2](https://www.w3.org/TR/WCAG22/): normative reference and the combination of automation with human evaluation.
|
|
243
|
+
- [W3C — Understanding ACT Rules](https://www.w3.org/WAI/WCAG22/Understanding/understanding-act-rules.html): automated rules are partial checks and a pass does not prove complete conformance.
|
|
244
|
+
- [Deque — axe-core](https://github.com/dequelabs/axe-core): automated scope, `incomplete` results, and the need for manual review.
|
|
245
|
+
- [Chrome for Developers — Lighthouse accessibility audits](https://developer.chrome.com/docs/lighthouse/accessibility/): automated checks and complementary manual items.
|
|
246
|
+
- [Appium — drivers](https://appium.io/docs/en/latest/ecosystem/drivers/) and [Appium Windows Driver](https://github.com/appium/appium-windows-driver): driver installation/support and WinAppDriver dependency/limitations.
|
|
247
|
+
- [Tauri — WebDriver](https://v2.tauri.app/develop/tests/webdriver/) and [WebdriverIO — Tauri Service](https://webdriver.io/docs/wdio-tauri-service/): `@wdio/tauri-service`, embedded server, and `tauri-driver`; Tauri page updated 2026-06-29.
|
|
248
|
+
- [W3C — WebDriver](https://www.w3.org/TR/webdriver2/): standardized protocol used by drivers.
|
|
249
|
+
- [Grafana k6 — thresholds](https://grafana.com/docs/k6/latest/using-k6/thresholds/) and [test types](https://grafana.com/docs/k6/latest/testing-guides/test-types/): pass/fail criteria, percentiles, smoke, spike, and soak.
|
|
250
|
+
- [Playwright — retries](https://playwright.dev/docs/test-retries): distinguishes passed, flaky, and failed-after-retry outcomes.
|
|
251
|
+
- [VCR.py — filtering sensitive data](https://vcrpy.readthedocs.io/en/latest/advanced.html#filter-sensitive-data-from-the-request): filtering headers, query, body, and responses before recording.
|
|
252
|
+
- [Prisma — expand-and-contract migrations](https://docs.prisma.io/docs/guides/database/data-migration): expand, migrate data, and contract sequence for production.
|
|
253
|
+
- [Stryker — mutation testing](https://stryker-mutator.io/docs/): mutants used to evaluate the suite's detection ability.
|
|
254
|
+
- [Spring Boot testing](https://docs.spring.io/spring-boot/reference/testing/spring-boot-applications.html), [ASP.NET Core integration tests](https://learn.microsoft.com/aspnet/core/test/integration-tests), [Rails testing](https://guides.rubyonrails.org/testing.html), and [Laravel testing](https://laravel.com/docs/12.x/testing): official framework harnesses.
|
|
255
|
+
- [OpenAPI](https://spec.openapis.org/oas/), [JSON Schema](https://json-schema.org/learn), [GraphQL](https://spec.graphql.org/), and [Pact](https://docs.pact.io/): API schemas and contracts.
|
|
256
|
+
- [Terraform testing](https://developer.hashicorp.com/terraform/cli/test), [Kubernetes kubectl apply](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_apply/), [Ansible Molecule](https://docs.ansible.com/projects/molecule/), and [Docker build checks](https://docs.docker.com/reference/build-checks/): operational IaC/DevOps evidence.
|
|
257
|
+
- [Stylelint](https://stylelint.io/user-guide/get-started/), [Playwright visual comparisons](https://playwright.dev/docs/test-snapshots), [WHATWG validators](https://whatwg.org/validator/), and [HyperFrames](https://hyperframes.heygen.com): CSS, visual, HTML, and composition checks.
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
# Execution state and resume protocol
|
|
2
|
+
|
|
3
|
+
Compatible agents may persist a handoff checkpoint at:
|
|
4
|
+
|
|
5
|
+
```text
|
|
6
|
+
.forgeloop/work-state.json
|
|
7
|
+
```
|
|
8
|
+
|
|
9
|
+
The file is local, ignored by Git, schema-versioned, and never a replacement
|
|
10
|
+
for the manifest or `PROJECT_PROFILE.md`. It contains no secrets and is
|
|
11
|
+
untrusted on read.
|
|
12
|
+
|
|
13
|
+
## Shape
|
|
14
|
+
|
|
15
|
+
```json
|
|
16
|
+
{
|
|
17
|
+
"schemaVersion": 1,
|
|
18
|
+
"protocolVersion": 1,
|
|
19
|
+
"taskId": "task-123",
|
|
20
|
+
"contractFingerprint": "lowercase-sha256",
|
|
21
|
+
"repositoryFingerprint": { "branch": "main", "head": "commit-sha" },
|
|
22
|
+
"phase": "VERIFYING",
|
|
23
|
+
"selectedGuides": ["clean", "test"],
|
|
24
|
+
"completedSteps": ["discovery", "implementation"],
|
|
25
|
+
"pendingSteps": ["verification", "review"],
|
|
26
|
+
"requiredArtifacts": [
|
|
27
|
+
{ "path": "src/api/auth.js", "sha256": "lowercase-sha256" }
|
|
28
|
+
],
|
|
29
|
+
"checks": [],
|
|
30
|
+
"failures": [],
|
|
31
|
+
"blockers": [],
|
|
32
|
+
"verificationEvidence": [],
|
|
33
|
+
"lastUpdated": "2026-08-11T12:00:00.000Z"
|
|
34
|
+
}
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
The CLI validates the protocol version, phase enum, guide IDs, fingerprints,
|
|
38
|
+
transition, status-specific evidence, bounded strings, and secret-free shape.
|
|
39
|
+
`COMPLETE` requires verification evidence. `BLOCKED` requires a blocker
|
|
40
|
+
category and evidence. `CORRECTING` requires a diagnosed hypothesis.
|
|
41
|
+
|
|
42
|
+
## Atomicity and stale state
|
|
43
|
+
|
|
44
|
+
State writes validate the complete JSON value, write a unique temporary file,
|
|
45
|
+
and rename it into place. The host filesystem's rename guarantee is the
|
|
46
|
+
atomicity boundary; no database or remote state is involved. A truncated,
|
|
47
|
+
malformed, or secret-bearing file is invalid and is never resumed silently.
|
|
48
|
+
|
|
49
|
+
Before resuming, compare:
|
|
50
|
+
|
|
51
|
+
- the task contract fingerprint;
|
|
52
|
+
- the Git branch and HEAD when the target is a Git checkout;
|
|
53
|
+
- the protocol version;
|
|
54
|
+
- required artifacts and assumptions recorded by the task.
|
|
55
|
+
|
|
56
|
+
Any material difference produces `REVALIDATION_REQUIRED`. A non-Git target
|
|
57
|
+
reports that branch/HEAD drift is not verifiable. Cheap checks may be rerun,
|
|
58
|
+
but a completed destructive or publication action is never rerun automatically.
|
|
59
|
+
|
|
60
|
+
The current contract is compared only when its JSON file is supplied:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
forgeloop status --contract-file .forgeloop/current-contract.json --json
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Without that file, contract comparison is `NOT_VERIFIED`; the status does not
|
|
67
|
+
claim full freshness. Required artifact hashes report missing or changed files.
|
|
68
|
+
An optional age threshold may recommend cheap verification with
|
|
69
|
+
`CHECKPOINT_OLD` without changing a fresh result.
|
|
70
|
+
|
|
71
|
+
`inspect`, `status`, and `validate-protocol` use the same derived freshness
|
|
72
|
+
classifier. Protocol validation can be run against the complete artifact set:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
forgeloop validate-protocol \
|
|
76
|
+
--route-file ./routing-result.json \
|
|
77
|
+
--state-file .forgeloop/work-state.json \
|
|
78
|
+
--receipt-file ./execution-receipt.json \
|
|
79
|
+
--contract-file .forgeloop/current-contract.json \
|
|
80
|
+
--json
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
`validate-protocol` reports `STALE` when the current repository fingerprint,
|
|
84
|
+
contract, or required-artifact fingerprints require revalidation. Its JSON
|
|
85
|
+
result exposes the comparison values, reasons, and warnings; the human output
|
|
86
|
+
shows the same evidence. The persisted work-state file remains schema-compatible
|
|
87
|
+
and never stores derived `status`, `stale`, or `fresh` fields.
|
|
88
|
+
|
|
89
|
+
State reports use the shared evidence kinds `OBSERVED`, `INFERRED`,
|
|
90
|
+
`NOT_VERIFIED`, and `BLOCKED`. A parse failure is structured as `INVALID` and
|
|
91
|
+
is never resumed silently.
|
|
92
|
+
|
|
93
|
+
## Commands
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
forgeloop status
|
|
97
|
+
forgeloop status --json
|
|
98
|
+
forgeloop validate-state
|
|
99
|
+
forgeloop validate-state --json
|
|
100
|
+
forgeloop clear-state
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
`status` explains whether state is absent, fresh, or requires revalidation.
|
|
104
|
+
`validate-state` performs schema and semantic checks without mutation.
|
|
105
|
+
`clear-state` affects only `.forgeloop/work-state.json` and prints the exact
|
|
106
|
+
relative path it removed; it never deletes the directory, manifest, or project
|
|
107
|
+
files.
|