bmad-method-test-architecture-enterprise 1.22.4 → 1.22.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/CHANGELOG.md +16 -0
- package/docs/reference/knowledge-base.md +14 -14
- package/package.json +1 -1
- package/src/agents/bmad-tea/resources/knowledge/evidence-integrity.md +41 -22
- package/src/agents/bmad-tea/resources/knowledge/maestro-flows.md +52 -19
- package/src/agents/bmad-tea/resources/knowledge/mobile-ci-device-lab.md +41 -21
- package/src/agents/bmad-tea/resources/knowledge/mobile-test-strategy.md +3 -0
- package/src/agents/bmad-tea/resources/tea-index.csv +4 -4
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/evidence-integrity.md +41 -22
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/maestro-flows.md +52 -19
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/mobile-ci-device-lab.md +41 -21
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/mobile-test-strategy.md +3 -0
- package/src/workflows/testarch/bmad-testarch-atdd/resources/tea-index.csv +4 -4
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/evidence-integrity.md +41 -22
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/maestro-flows.md +52 -19
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/mobile-ci-device-lab.md +41 -21
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/mobile-test-strategy.md +3 -0
- package/src/workflows/testarch/bmad-testarch-automate/resources/tea-index.csv +4 -4
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/evidence-integrity.md +41 -22
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/maestro-flows.md +52 -19
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/mobile-ci-device-lab.md +41 -21
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/mobile-test-strategy.md +3 -0
- package/src/workflows/testarch/bmad-testarch-ci/resources/tea-index.csv +4 -4
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/evidence-integrity.md +41 -22
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/maestro-flows.md +52 -19
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/mobile-ci-device-lab.md +41 -21
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/mobile-test-strategy.md +3 -0
- package/src/workflows/testarch/bmad-testarch-framework/resources/tea-index.csv +4 -4
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/evidence-integrity.md +41 -22
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/maestro-flows.md +52 -19
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/mobile-ci-device-lab.md +41 -21
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/mobile-test-strategy.md +3 -0
- package/src/workflows/testarch/bmad-testarch-nfr/resources/tea-index.csv +4 -4
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/evidence-integrity.md +41 -22
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/maestro-flows.md +52 -19
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/mobile-ci-device-lab.md +41 -21
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/mobile-test-strategy.md +3 -0
- package/src/workflows/testarch/bmad-testarch-test-design/resources/tea-index.csv +4 -4
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/evidence-integrity.md +41 -22
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/maestro-flows.md +52 -19
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/mobile-ci-device-lab.md +41 -21
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/mobile-test-strategy.md +3 -0
- package/src/workflows/testarch/bmad-testarch-test-review/resources/tea-index.csv +4 -4
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/evidence-integrity.md +41 -22
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/maestro-flows.md +52 -19
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/mobile-ci-device-lab.md +41 -21
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/mobile-test-strategy.md +3 -0
- package/src/workflows/testarch/bmad-testarch-trace/resources/tea-index.csv +4 -4
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
"name": "bmad-method-test-architecture-enterprise",
|
|
32
32
|
"source": "./",
|
|
33
33
|
"description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
|
|
34
|
-
"version": "1.22.
|
|
34
|
+
"version": "1.22.5",
|
|
35
35
|
"author": {
|
|
36
36
|
"name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
|
|
37
37
|
},
|
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [1.22.5] - 2026-08-14
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- `maestro-flows.md`: visible means inside the viewport, not present in the view hierarchy. `assertVisible` and `extendedWaitUntil: visible` fail on an element that is rendered, correct, and below the fold, and the failure reads as a broken feature rather than a flow that never scrolled. Carries the cheap diagnostic tell (find the id in the failing step's `screen-hierarchy`: present with out-of-bounds coordinates is a scroll problem, absent is a different bug) and the rule that any assertion inside a scrolling section is a latent screen-height dependency.
|
|
15
|
+
- `mobile-ci-device-lab.md`: local and CI must run the same device profile. Measured, a runner on a roughly 807dp-tall profile against a local 914dp one failed four unrelated flows with "not visible" while every developer machine stayed green, each looking like a product defect. Names the trap that the comparison is density-independent height rather than pixel resolution, since two profiles can share `1080x` and differ by 100dp.
|
|
16
|
+
- `mobile-ci-device-lab.md`: a native module in a development shell does not always fail loudly. Some SDKs detect the shell and degrade to a fallback path (one logs `Expo Go app detected. Using RevenueCat in Browser Mode.`), so nothing errors and the suite proves a code path users never run. A hard `undefined` announces itself; a silent degradation produces a green flow covering the wrong implementation.
|
|
17
|
+
- `mobile-ci-device-lab.md`: check present-but-off-screen before anything else when a step fails on "not visible". One hierarchy lookup separates a flow-level scroll problem from an application defect.
|
|
18
|
+
- `evidence-integrity.md`: screen geometry and accumulated local credentials as environment-asymmetry axes. Both read as trivia and are not: viewport height decides what "visible" means, and a session file or cached certificate on a developer machine makes a whole code path invisible locally.
|
|
19
|
+
- `evidence-integrity.md`: rank hypotheses by the cost of the measurement that would kill them. One investigation produced three plausible mechanisms, all wrong, before two cheap observations eliminated the set; the tell is a session holding several explanations and no new measurements. The corollary is that the discriminating measurement should already be in the captured artifacts, so "the artifacts could not tell us" is a finding about the harness.
|
|
20
|
+
- `mobile-test-strategy.md`: the PR-gate device profile must match the local one, so the matrix can find fragmentation defects instead of manufacturing them at the gate.
|
|
21
|
+
|
|
22
|
+
### Changed
|
|
23
|
+
|
|
24
|
+
- `maestro-flows.md`: withdrew the mechanism claim about lost taps. The measurements rule out a missing element and an occluding overlay; they do not establish where the touch is lost, and the earlier text ("not a scroll, not an overlay, not the keyboard") asserted more than the evidence supports. The gesture-responder explanation is now labelled an untested hypothesis and the retry-with-assertion pattern is labelled a countermeasure rather than a fix, per the fragment's own rule that a stated mechanism is a claim needing a source.
|
|
25
|
+
|
|
10
26
|
## [1.22.4] - 2026-08-14
|
|
11
27
|
|
|
12
28
|
### Added
|
|
@@ -73,15 +73,15 @@ CI/CD patterns, burn-in testing, and selective test execution.
|
|
|
73
73
|
|
|
74
74
|
Test quality standards, test level selection, TDD patterns, and the generation-safety gate.
|
|
75
75
|
|
|
76
|
-
| Fragment | Tier | Description
|
|
77
|
-
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
|
|
78
|
-
| [test-quality](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-quality.md) | core | Execution limits, isolation rules, green criteria
|
|
79
|
-
| [test-levels](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-levels-framework.md) | core | Guidelines for choosing unit, integration, or end-to-end coverage
|
|
80
|
-
| [test-priorities](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-priorities-matrix.md) | core | P0–P3 criteria, coverage targets, execution ordering
|
|
81
|
-
| [test-healing-patterns](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-healing-patterns.md) | core | Common failure patterns and automated fixes
|
|
82
|
-
| [confidence-gate](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/confidence-gate.md) | core | 1-10 confidence score with a stop-and-ask rule below threshold, so the agent declares unknowns instead of fabricating them
|
|
83
|
-
| [evidence-integrity](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/evidence-integrity.md) | core | Checks that cannot fail, three-state diagnostics, probes that issue the client's own request, verifying the outcome rather than the act, verifying framework properties before use, stating environment asymmetry, and recording what a change did rather than what it was for | Falsifiability, hollow green, diagnostics |
|
|
84
|
-
| [component-tdd](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/component-tdd.md) | extended | Red→green→refactor workflow, provider isolation
|
|
76
|
+
| Fragment | Tier | Description | Key Topics |
|
|
77
|
+
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
|
|
78
|
+
| [test-quality](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-quality.md) | core | Execution limits, isolation rules, green criteria | DoD, best practices, anti-patterns |
|
|
79
|
+
| [test-levels](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-levels-framework.md) | core | Guidelines for choosing unit, integration, or end-to-end coverage | Test pyramid, level selection |
|
|
80
|
+
| [test-priorities](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-priorities-matrix.md) | core | P0–P3 criteria, coverage targets, execution ordering | Prioritization, risk-based testing |
|
|
81
|
+
| [test-healing-patterns](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-healing-patterns.md) | core | Common failure patterns and automated fixes | Debugging, healing, fixes |
|
|
82
|
+
| [confidence-gate](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/confidence-gate.md) | core | 1-10 confidence score with a stop-and-ask rule below threshold, so the agent declares unknowns instead of fabricating them | Agent safety, generation, governance |
|
|
83
|
+
| [evidence-integrity](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/evidence-integrity.md) | core | Checks that cannot fail, three-state diagnostics, probes that issue the client's own request, verifying the outcome rather than the act, verifying framework properties before use, stating environment asymmetry including screen geometry and accumulated credentials, ranking hypotheses by the cost of the measurement that kills them, and recording what a change did rather than what it was for | Falsifiability, hollow green, diagnostics |
|
|
84
|
+
| [component-tdd](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/component-tdd.md) | extended | Red→green→refactor workflow, provider isolation | TDD, component testing |
|
|
85
85
|
|
|
86
86
|
**Used in:** `test-design`, `atdd`, `automate`, `test-review`, `trace`
|
|
87
87
|
|
|
@@ -179,11 +179,11 @@ Delivery-side testing for asynchronous, eventually-consistent webhook flows usin
|
|
|
179
179
|
|
|
180
180
|
Maestro device flows and the level discipline that decides what becomes a flow at all. Loaded when `test_stack_type` is `mobile` or when the review set contains a Maestro flow (`.yaml`/`.yml` under `maestro/` or `.maestro/`, or `*.flow.yaml` or `*.flow.yml`).
|
|
181
181
|
|
|
182
|
-
| Fragment | Tier | Description
|
|
183
|
-
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
|
|
184
|
-
| [mobile-test-strategy](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/mobile-test-strategy.md) | specialized | Mobile test level framework, what belongs in a device flow, mobile risk categories, device matrix, CI shape, no live third-party flag evaluation in the run path
|
|
185
|
-
| [maestro-flows](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/maestro-flows.md) | specialized | Flow structure, selector hierarchy, `clearState` isolation, synchronization without sleeps, subflow composition, command semantics that differ by platform, `text:` selectors as whole-element regex, taps that report COMPLETED without being handled
|
|
186
|
-
| [mobile-ci-device-lab](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/mobile-ci-device-lab.md) | specialized | Build artifact selection including why a debug-variant development build does not solve it, dev-server manifest signing in non-interactive CI, emulator snapshot caching, repairing locally created AVDs, per-device identity for sharded runs, runner version pinning, artifact layout and failure diagnosis | Build artifact, emulator, caching, pinning, sharding, artifacts |
|
|
182
|
+
| Fragment | Tier | Description | Key Topics |
|
|
183
|
+
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- |
|
|
184
|
+
| [mobile-test-strategy](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/mobile-test-strategy.md) | specialized | Mobile test level framework, what belongs in a device flow, mobile risk categories, device matrix with a gate profile matching local, CI shape, no live third-party flag evaluation in the run path | Levels, risk, device matrix, permissions, lifecycle |
|
|
185
|
+
| [maestro-flows](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/maestro-flows.md) | specialized | Flow structure, selector hierarchy, `clearState` isolation, synchronization without sleeps, subflow composition, command semantics that differ by platform, `text:` selectors as whole-element regex, taps that report COMPLETED without being handled, visible meaning inside the viewport | Maestro, selectors, isolation, regex selectors, anti-patterns |
|
|
186
|
+
| [mobile-ci-device-lab](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/mobile-ci-device-lab.md) | specialized | Build artifact selection including why a debug-variant development build does not solve it, dev-server manifest signing in non-interactive CI, one device profile across local and CI, native modules that degrade silently in a shell, emulator snapshot caching, repairing locally created AVDs, per-device identity for sharded runs, runner version pinning, artifact layout and failure diagnosis | Build artifact, emulator, caching, pinning, sharding, artifacts |
|
|
187
187
|
|
|
188
188
|
**Used in:** `framework`, `automate`, `atdd`, `test-design`, `test-review`, `ci` (when `test_stack_type` is `mobile` or a Maestro flow is present)
|
|
189
189
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.22.
|
|
4
|
+
"version": "1.22.5",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
|
@@ -151,16 +151,23 @@ The host and the device are different network namespaces. A host that resolves a
|
|
|
151
151
|
|
|
152
152
|
Write the asymmetry down whenever a local result enters a CI argument:
|
|
153
153
|
|
|
154
|
-
| Axis
|
|
155
|
-
|
|
|
156
|
-
| OS / arch
|
|
157
|
-
| Platform ver.
|
|
158
|
-
| Image variant
|
|
159
|
-
| Acceleration
|
|
160
|
-
| Provisioning
|
|
154
|
+
| Axis | Local | CI |
|
|
155
|
+
| --------------- | ------------------------------------------ | ---------------------------------- |
|
|
156
|
+
| OS / arch | macOS, arm64 | Linux, x86_64 |
|
|
157
|
+
| Platform ver. | newest API level | two levels older |
|
|
158
|
+
| Image variant | vendor image with store services signed in | plain AOSP-style image, no account |
|
|
159
|
+
| Acceleration | native hypervisor | KVM, may be unavailable |
|
|
160
|
+
| Provisioning | long-lived machine | fresh runner every job |
|
|
161
|
+
| Screen geometry | 914dp tall (1080x2400 at 420dpi) | 807dp tall (1080x2220 at 440dpi) |
|
|
162
|
+
| Credentials | a developer session already on disk | whatever the secret store supplies |
|
|
161
163
|
|
|
162
164
|
A local pass proves the application path. It proves nothing about acceleration, snapshot restore, `PATH` handling, or an image variant the local machine never runs. Naming the axes converts "it works on my machine" from an argument into a scoped fact.
|
|
163
165
|
|
|
166
|
+
Two of those axes are worth calling out because they read as trivia and are not:
|
|
167
|
+
|
|
168
|
+
- **Screen geometry decides what is on screen, and "visible" usually means on screen.** The row above is a real pair: a 12% shorter viewport pushed content below the fold and failed four unrelated UI assertions on the runner while every developer machine stayed green. Compare the density-independent height, not the pixel resolution; the two profiles in that row share `1080x` and are different screens.
|
|
169
|
+
- **A developer machine accumulates credentials that a fresh runner has never had.** A cached session file or a fetched certificate sitting in a home directory makes a whole code path invisible locally. When a failure exists only in CI and nothing about the code explains it, ask what the local machine has lying around that the runner does not.
|
|
170
|
+
|
|
164
171
|
### Example 6: Resolve Environment-Dependent Values Before Anything Derives From Them
|
|
165
172
|
|
|
166
173
|
**Context**: A harness that probed which host address the device could reach, and ran the probe after that address had already been baked into the built artifact.
|
|
@@ -183,23 +190,34 @@ One investigation removed a live feature-flag service from an end-to-end run, so
|
|
|
183
190
|
|
|
184
191
|
A change described by its intent after its effect is known is a landmine for the next investigation, because the next reader takes the commit message as evidence that the cause was found and stops looking. State the outcome separately from the intent. "Kept for a different reason, and labelled as such" costs one sentence and saves someone a re-derivation.
|
|
185
192
|
|
|
193
|
+
### Example 9: Order Hypotheses by the Cost of the Measurement That Kills Them
|
|
194
|
+
|
|
195
|
+
**Context**: A UI assertion failing on "not visible". Three plausible mechanisms, all wrong.
|
|
196
|
+
|
|
197
|
+
One investigation built three separate explanations for the same failure, each with a real code path behind it: a remote flag service answering false, an unresolved dynamic import leaving state null, and a stuck initialization call. Each was written up with its reasoning. Two measurements ended all three at once: a device log line showing the module had loaded, and a direct API query showing the flag was true. The actual cause was that the element sat below the fold, which one lookup in the captured view hierarchy would have shown at the start.
|
|
198
|
+
|
|
199
|
+
**Rule**: before elaborating a mechanism, list the measurements available and what each would eliminate, then take the cheapest one that kills a whole class of hypothesis. A plausible mechanism is not evidence, and producing more of them feels like progress while the discriminating observation goes untaken. The tell is a session holding several explanations and no new measurements.
|
|
200
|
+
|
|
201
|
+
The corollary for a suite: that cheap discriminating measurement should already be sitting in the artifacts. This is why capturing state at failure earns its storage cost, and why "the artifacts could not tell us" is a finding about the harness rather than an inconvenience.
|
|
202
|
+
|
|
186
203
|
## Anti-Patterns
|
|
187
204
|
|
|
188
|
-
| Anti-pattern
|
|
189
|
-
|
|
|
190
|
-
| Assertion with a soft or optional modifier as default
|
|
191
|
-
| `continue-on-error` on the test step
|
|
192
|
-
| Runner manifest listing a subset of the suite
|
|
193
|
-
| Missing tool reported as a failed condition
|
|
194
|
-
| Probe observing a proxy that correlates with the target
|
|
195
|
-
| Probe sending a different request than the client
|
|
196
|
-
| Written setting read back and reported as effect
|
|
197
|
-
| Change described by its intent once its effect is known
|
|
198
|
-
|
|
|
199
|
-
|
|
|
200
|
-
|
|
|
201
|
-
|
|
|
202
|
-
|
|
|
205
|
+
| Anti-pattern | Why it fails | Fix |
|
|
206
|
+
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
|
207
|
+
| Assertion with a soft or optional modifier as default | Cannot go red; reports coverage that does not exist | Reserve softness for genuinely optional UI, and assert the outcome hard |
|
|
208
|
+
| `continue-on-error` on the test step | The suite cannot fail the build | Put it on artifact collection only; use `if: always()` for uploads |
|
|
209
|
+
| Runner manifest listing a subset of the suite | Files silently never run; the count is the only clue | Include by pattern; assert the executed count against the file count |
|
|
210
|
+
| Missing tool reported as a failed condition | Sends the investigation at the wrong subsystem | Three-state probes; distinct exit code for could-not-measure |
|
|
211
|
+
| Probe observing a proxy that correlates with the target | Passes for a reason unrelated to the claim, and a green is never re-examined | Ask what else could make this pass; observe the thing itself |
|
|
212
|
+
| Probe sending a different request than the client | Takes a different branch through the server: healthy probe, failing app, both correct | Copy the client's method, headers, and body into the probe, and log what was sent |
|
|
213
|
+
| Written setting read back and reported as effect | Proves the write; some configuration latches at boot and never applies live | Assert the observable the claim is about, not the act that was supposed to produce it |
|
|
214
|
+
| Change described by its intent once its effect is known | Reads as a found root cause and stops the next investigation looking | Record what it actually did, and why it was kept |
|
|
215
|
+
| Mechanisms elaborated while the cheap measurement goes untaken | Plausibility feels like progress; several explanations, no new evidence | Rank hypotheses by the cost of the observation that would kill them |
|
|
216
|
+
| Verdict emitted by the side that cannot observe it | Proves the wrong namespace | Move the assertion to the party whose route or state is in question |
|
|
217
|
+
| Comment asserting a mechanism with no source read | Propagates into other files and into other people's reasoning | Cite the doc or source line, or omit the mechanism |
|
|
218
|
+
| Local result used as a CI argument, asymmetry unstated | Hides the axes that actually differ | Tabulate the differing axes with the claim |
|
|
219
|
+
| Environment probe running after its consumers | The answer arrives too late to configure anything | Resolve environment-dependent values first, then derive |
|
|
220
|
+
| Reverting a newly-red check as a regression | Restores the hollow green and loses the finding | Treat the red as the pre-existing defect it exposed |
|
|
203
221
|
|
|
204
222
|
## Evidence Integrity Checklist
|
|
205
223
|
|
|
@@ -219,6 +237,7 @@ A change described by its intent after its effect is known is a landmine for the
|
|
|
219
237
|
- [ ] **Environment asymmetry stated**: local-versus-CI arguments list the differing axes
|
|
220
238
|
- [ ] **Resolution precedes derivation**: environment-dependent values resolved before any consumer is built
|
|
221
239
|
- [ ] **Effects recorded separately from intent**: a change kept for a reason other than the one it was made for says so
|
|
240
|
+
- [ ] **Cheapest discriminating measurement taken first**: no mechanism elaborated while an available observation would eliminate a class of hypothesis
|
|
222
241
|
|
|
223
242
|
## Integration Points
|
|
224
243
|
|
|
@@ -362,7 +362,9 @@ tags:
|
|
|
362
362
|
|
|
363
363
|
**Context**: A checkbox in a virtualized list. Maestro reported the tap COMPLETED and the app never saw it.
|
|
364
364
|
|
|
365
|
-
Measured in isolation: the tap completed in 2.4 seconds, the target was present at `[45,1807][1035,1924]` with `clickable=true`, nothing sat above it in the hierarchy at the point tapped, and the count label still read `0 of 10` ten seconds later.
|
|
365
|
+
Measured in isolation: the tap completed in 2.4 seconds, the target was present at `[45,1807][1035,1924]` with `clickable=true`, nothing sat above it in the hierarchy at the point tapped, and the count label still read `0 of 10` ten seconds later. `tapOn` reports COMPLETED once it has resolved the element and dispatched a touch, so its status describes the driver's action and not the app's response. One passing run recorded five taps for two selections, which makes the loss frequent rather than exotic.
|
|
366
|
+
|
|
367
|
+
**The mechanism is not established.** Those measurements rule out a missing element and an occluding overlay; they do not say where between the driver and the app's handler the touch went. Gesture-responder interaction with a list that has visually stopped moving is a plausible candidate and has not been tested. Treat that as an open question rather than a cause, and treat the pattern below as what it is: a countermeasure that keeps the flow honest while the cause is unknown, per the `evidence-integrity.md` rule that a stated mechanism is a claim needing a source.
|
|
366
368
|
|
|
367
369
|
**Implementation**:
|
|
368
370
|
|
|
@@ -401,26 +403,56 @@ Measured in isolation: the tap completed in 2.4 seconds, the target was present
|
|
|
401
403
|
- The retry does not weaken the check. The assertion inside it has to pass on its own, so what the retry absorbs is a lost touch, which is a property of the driver rather than of the app.
|
|
402
404
|
- If a step routinely needs its retry to land, that is a finding about the driver or the list, not an ordinary step. Record it rather than letting the retry hide it.
|
|
403
405
|
|
|
406
|
+
### Example 9: Visible Means Inside the Viewport, Not Present in the Hierarchy
|
|
407
|
+
|
|
408
|
+
**Context**: An element that is rendered, correct, and below the fold. `assertVisible` fails on it.
|
|
409
|
+
|
|
410
|
+
`assertVisible` and `extendedWaitUntil: visible` require the element to be **on screen**, not merely present in the view tree. An element in a scrolling section that has not been scrolled to is present, correct, and not visible, and the failure reads as "the feature is broken" rather than "the flow never scrolled".
|
|
411
|
+
|
|
412
|
+
**Implementation**:
|
|
413
|
+
|
|
414
|
+
```yaml
|
|
415
|
+
# ❌ Latent screen-height dependency: passes on a tall device, fails on a short one
|
|
416
|
+
- extendedWaitUntil:
|
|
417
|
+
visible:
|
|
418
|
+
id: 'premium_unavailable'
|
|
419
|
+
timeout: 20000
|
|
420
|
+
|
|
421
|
+
# ✅ Scroll to what you assert on
|
|
422
|
+
- scrollUntilVisible:
|
|
423
|
+
element:
|
|
424
|
+
id: 'premium_unavailable'
|
|
425
|
+
- assertVisible:
|
|
426
|
+
id: 'premium_unavailable'
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
**Key points**:
|
|
430
|
+
|
|
431
|
+
- **The diagnostic tell is cheap and decisive.** Dump the failing step's `screen-hierarchy` entry and look for the id. Present with bounds outside the screen is a scroll problem; absent is a different bug entirely. Checking this first is worth more than any hypothesis about the feature, and one investigation built three separate plausible mechanisms for such a failure (a remote flag answer, an unresolved dynamic import, a stuck initialization call) before two measurements killed all three and left the viewport.
|
|
432
|
+
- **Any `assertVisible` on an element inside a scrolling section is a screen-height dependency** until the flow scrolls to it. It will pass wherever it was written and fail on the first shorter device.
|
|
433
|
+
- This is the failure mode most likely to reproduce only in CI, because the runner's device profile is rarely the one the flow was written against. `mobile-ci-device-lab.md` carries the profile-parity rule.
|
|
434
|
+
|
|
404
435
|
## Anti-Patterns
|
|
405
436
|
|
|
406
|
-
| Anti-pattern
|
|
407
|
-
|
|
|
408
|
-
| Flow with no `assertVisible`/`assertTrue`
|
|
409
|
-
| `sleep` used as synchronization
|
|
410
|
-
| `tapOn: index:` on a list
|
|
411
|
-
| `tapOn: point:` coordinates
|
|
412
|
-
| No `clearState`
|
|
413
|
-
| Hardcoded credential or PII
|
|
414
|
-
| One flow covering six user journeys
|
|
415
|
-
| Required assertion inside `when:`
|
|
416
|
-
| `back` used as a cross-platform step
|
|
417
|
-
| `hideKeyboard` with a modal open
|
|
418
|
-
| `optional: true` on the assertion that carries the outcome
|
|
419
|
-
| `waitForAnimationToEnd` used as a wait-for-content
|
|
420
|
-
| Unescaped regex characters in a `text:` selector
|
|
421
|
-
| Tap status treated as proof the app handled the tap
|
|
422
|
-
| One assertion at the end of a tap sequence
|
|
423
|
-
| `scrollUntilVisible` reused after an earlier search moved the list
|
|
437
|
+
| Anti-pattern | Why it fails | Fix |
|
|
438
|
+
| ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
|
|
439
|
+
| Flow with no `assertVisible`/`assertTrue` | Passes as long as taps land; proves nothing about behavior | Assert the destination state of every flow |
|
|
440
|
+
| `sleep` used as synchronization | Flaky under load, slow when not | `extendedWaitUntil` on the real condition |
|
|
441
|
+
| `tapOn: index:` on a list | Breaks when the list reorders or the backend returns a different order | Scope by `text` with `below`/`containsChild` |
|
|
442
|
+
| `tapOn: point:` coordinates | Breaks on a different screen size or density | Address the element by `id` |
|
|
443
|
+
| No `clearState` | Flow depends on whatever ran before it; unreproducible in isolation | `clearState` before `launchApp` |
|
|
444
|
+
| Hardcoded credential or PII | Leaks in the repo and in CI logs | `${ENV_VAR}` sourced from the CI secret store |
|
|
445
|
+
| One flow covering six user journeys | A failure names the flow, not the behavior; slow to diagnose | One journey per flow, composed from subflows |
|
|
446
|
+
| Required assertion inside `when:` | Turns a real failure into a silent pass | Guard only genuinely optional UI (permission dialogs, upsells) |
|
|
447
|
+
| `back` used as a cross-platform step | Documented for Android and Web only; on iOS it does nothing and reports COMPLETED | Split with `runFlow: when: platform:`; tap the app's own control on iOS |
|
|
448
|
+
| `hideKeyboard` with a modal open | The Android implementation is the system back key, which dismisses a React Native modal | Tap a non-interactive element to drop the keyboard |
|
|
449
|
+
| `optional: true` on the assertion that carries the outcome | The step cannot fail, so the flow reports coverage it does not have | Assert hard; reserve `optional` for genuinely optional UI |
|
|
450
|
+
| `waitForAnimationToEnd` used as a wait-for-content | It succeeds when its cap is reached, so it cannot fail | `extendedWaitUntil` on the content that must appear |
|
|
451
|
+
| Unescaped regex characters in a `text:` selector | The value is a regex matched against the element's entire text, so it can never pass | Escape literal `(`, `)`, `[`, `]`, `.`; pad partial matches with `.*` |
|
|
452
|
+
| Tap status treated as proof the app handled the tap | `tapOn` reports COMPLETED once the touch is dispatched, and touches do get lost | Pair each state-changing tap with its own assertion inside a `retry` |
|
|
453
|
+
| One assertion at the end of a tap sequence | Cannot name which tap was lost, so the first hypothesis is a guess | Assert after every action that changes state |
|
|
454
|
+
| `scrollUntilVisible` reused after an earlier search moved the list | It travels only in the direction given, so the second search scrolls away from the target | Name the opposite `direction`, or return to a known position first |
|
|
455
|
+
| `assertVisible` on an element inside a scrolling section, with no scroll | Visible means inside the viewport; the element is present, correct, and below the fold | `scrollUntilVisible` first, then assert |
|
|
424
456
|
|
|
425
457
|
## Maestro Flow Checklist
|
|
426
458
|
|
|
@@ -440,6 +472,7 @@ Before merging a flow:
|
|
|
440
472
|
- [ ] **`text:` selectors read as regex**: literal `(`, `)`, `[`, `]`, `.` escaped, and whole-element matching accounted for
|
|
441
473
|
- [ ] **Every state-changing tap has its own assertion**, rather than one assertion covering a sequence
|
|
442
474
|
- [ ] **`retry` scoped to a single step**, with `maxRetries` inside the documented 0-3 range
|
|
475
|
+
- [ ] **Nothing asserted below the fold**: every assertion on an element inside a scrolling section is preceded by a scroll to it
|
|
443
476
|
|
|
444
477
|
## Integration Points
|
|
445
478
|
|
|
@@ -27,6 +27,8 @@ Before a single flow is written, decide **which artifact the flows run against**
|
|
|
27
27
|
|
|
28
28
|
**Rule**: the prebuilt shell is the wrong artifact for E2E. It cannot load your native modules, so any flow touching notifications, OAuth, deep or universal links, maps, in-app purchases, or any feature that hands an API key to native code can only assert the feature is **absent**. Deep linking is how most device flows enter a screen, which makes the hole load-bearing rather than marginal.
|
|
29
29
|
|
|
30
|
+
**"Absent" is not always the shape it takes, and the other shape is worse.** Some SDKs detect the shell and degrade instead of failing. One in-app-purchase SDK logs `Expo Go app detected. Using RevenueCat in Browser Mode.` and keeps working through a different code path. Nothing errors, the flow proceeds, and what the suite proves is that the fallback path works in a container users never run. A hard `undefined` at least announces itself; a silent degradation gives you a green flow covering the wrong implementation. When a flow touches a native module in a shell, check the device log for what the module decided to do rather than assuming it did nothing.
|
|
31
|
+
|
|
30
32
|
Expo's own CI tutorial builds a dedicated EAS profile for this (`e2e-test`, with `withoutCredentials: true`, Android `buildType: "apk"`, and iOS `simulator: true`) and runs Maestro against those builds. It never runs the flows through Expo Go. See <https://docs.expo.dev/tutorial/cicd/e2e-tests/> and <https://expo.dev/blog/expo-go-vs-development-builds>.
|
|
31
33
|
|
|
32
34
|
The cost of getting this wrong is measurable in flow source. In one audit, about 120 of 195 lines in a single launch subflow existed solely to fight the development shell (dev-server readiness, manifest retries, a third-party app's own UI), and most of the defects fixed that week would not have existed against a compiled build. Workarounds for a wrong artifact do not stay in the harness; they migrate into the flows and become the suite.
|
|
@@ -55,6 +57,18 @@ Sometimes the compiled build is not ready yet and the dev-server path has to wor
|
|
|
55
57
|
- **Expect the app config to be evaluated per request.** The manifest handler re-reads the project config on every manifest request, so config plugins run per request. Anything environment-sensitive in that config is a live macOS-versus-Linux divergence axis.
|
|
56
58
|
- **Do not build on undocumented packager host variables.** They carry a "drop the undocumented env variables" note upstream, and setting one can break a working `adb reverse` plus loopback setup by advertising a different host back to the client.
|
|
57
59
|
|
|
60
|
+
## Local and CI Run the Same Device Profile
|
|
61
|
+
|
|
62
|
+
Pin one device profile and use it on both sides. A different profile is a different layout, and a different layout is a class of failure that reproduces nowhere but CI.
|
|
63
|
+
|
|
64
|
+
Measured: CI booted a `pixel_3a` (1080x2220 at 440dpi, roughly 807dp tall) while every local run used a `medium_phone` (1080x2400 at 420dpi, roughly 914dp). A screen about 12% shorter pushes more of each scrolling section below the fold, and four unrelated flows failed on the runner with "not visible" while staying green on every developer machine. Each one read like a product defect. None was.
|
|
65
|
+
|
|
66
|
+
The height that matters is **density-independent pixels**, not the pixel resolution, because the layout is laid out in dp. Two profiles with the same `1080x` resolution and different densities are different screens, and comparing the resolutions alone will tell you they match.
|
|
67
|
+
|
|
68
|
+
- Name the profile in the harness, not in a person's local setup, so both sides read the same value.
|
|
69
|
+
- When the matrix genuinely needs more than one profile, keep the PR-gate profile identical to the local one and let the extra profiles run nightly, where a difference is information rather than noise.
|
|
70
|
+
- A suite running two profiles cannot distinguish a real regression from a screen-height artifact, and the artifact is far more common. That ambiguity costs more than the coverage the second profile adds at the gate.
|
|
71
|
+
|
|
58
72
|
## Local Emulators Need Repair After Creation
|
|
59
73
|
|
|
60
74
|
`avdmanager create avd` does not hand back a device a UI driver can use. Three defects, all measured on an Apple Silicon host, all needing a post-creation edit:
|
|
@@ -123,6 +137,8 @@ Read, in this order:
|
|
|
123
137
|
3. **Device logs** for the app's own errors.
|
|
124
138
|
4. **The screenshot, last and with suspicion.** It is captured after teardown, so it frequently shows the launcher rather than the failing screen. Diagnosing from it produces confident wrong answers.
|
|
125
139
|
|
|
140
|
+
**Check present-but-off-screen before anything else.** When a step fails on "not visible", find the id in that step's `screen-hierarchy` entry. Present with bounds outside the screen is a scroll problem in the flow; absent is a defect in the app. This one lookup separates the two most common causes of a red run and costs seconds. Skipping it is how an investigation spends hours building plausible mechanisms for a feature that was working the whole time.
|
|
141
|
+
|
|
126
142
|
**Count root causes, not red flows.** One serial run failed with `Maestro Android driver did not start up in time`, and three further flows then failed in one to two seconds each with no artifacts written. Four red flows, one defect. A flow that failed in seconds and wrote nothing did not run; treat it as could-not-measure and diagnose the first failure, because a defect count inflated by a cascade sends the investigation at four subsystems instead of one.
|
|
127
143
|
|
|
128
144
|
## Parallelism
|
|
@@ -134,27 +150,30 @@ Read, in this order:
|
|
|
134
150
|
|
|
135
151
|
## Anti-Patterns
|
|
136
152
|
|
|
137
|
-
| Anti-pattern | Why it fails
|
|
138
|
-
| ------------------------------------------------------------------ |
|
|
139
|
-
| E2E against a prebuilt development shell | Native modules absent; adds a CI-only launch path; flows fill with workarounds
|
|
140
|
-
| Development build adopted as the fix for the dev-server dependency | `developmentClient: true` builds a debug variant, which does not embed the JS bundle, so a packager is still required
|
|
141
|
-
| E2E affordance gated behind `__DEV__` | `__DEV__` is false in a release build, so the affordance is absent from the build under test
|
|
142
|
-
| Signed-manifest path left intact in CI | The certificate fetch needs an account session; non-interactive CI cannot answer the prompt
|
|
143
|
-
|
|
|
144
|
-
|
|
|
145
|
-
|
|
|
146
|
-
|
|
|
147
|
-
|
|
|
148
|
-
|
|
|
149
|
-
|
|
|
150
|
-
|
|
|
151
|
-
|
|
|
152
|
-
|
|
|
153
|
-
|
|
|
154
|
-
|
|
|
155
|
-
|
|
|
156
|
-
|
|
|
157
|
-
|
|
|
153
|
+
| Anti-pattern | Why it fails | Fix |
|
|
154
|
+
| ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
|
155
|
+
| E2E against a prebuilt development shell | Native modules absent; adds a CI-only launch path; flows fill with workarounds | Build a release-shaped artifact and install it |
|
|
156
|
+
| Development build adopted as the fix for the dev-server dependency | `developmentClient: true` builds a debug variant, which does not embed the JS bundle, so a packager is still required | Build a release variant; only it embeds the bundle |
|
|
157
|
+
| E2E affordance gated behind `__DEV__` | `__DEV__` is false in a release build, so the affordance is absent from the build under test | Gate on an `EXPO_PUBLIC_` variable and keep what it gates safe to ship in plain text |
|
|
158
|
+
| Signed-manifest path left intact in CI | The certificate fetch needs an account session; non-interactive CI cannot answer the prompt | Supply `EXPO_TOKEN`, or serve the manifest `--offline`, on every path |
|
|
159
|
+
| Local and CI on different device profiles | A shorter screen pushes content below the fold, producing "not visible" failures that reproduce only in CI and read as product defects | Pin one profile for both; extra profiles run nightly, not at the gate |
|
|
160
|
+
| Native module assumed absent in a development shell | Some SDKs detect the shell and degrade silently, so the flow covers a fallback path users never run | Read the device log for what the module decided; prefer a real build |
|
|
161
|
+
| "Not visible" diagnosed before the hierarchy is read | Present-but-off-screen and genuinely-absent are different bugs behind the same message | Look for the id and its bounds in that step's `screen-hierarchy` first |
|
|
162
|
+
| AVD used as the creation tool produced it | `target=android-0`, GPU off, keyboard off: acceleration silently drops or boot never completes | Repair the `.ini` and `config.ini`, then read the values back |
|
|
163
|
+
| Android device identity read rather than written | Every emulator reports the product model, so parallel shards share one map key and one account | Write `device_name` / `bluetooth_name` per device and hard-fail duplicates |
|
|
164
|
+
| Shard count chosen from per-flow duration | Oversubscription stretches each flow while still shortening the run | Choose on measured wall clock, which is what gates the PR |
|
|
165
|
+
| Every red flow counted as its own defect | A driver timeout cascades into fast, artifact-less failures | Diagnose the first failure; artifact-less seconds-long failures are could-not-measure |
|
|
166
|
+
| Multi-line `script:` in the emulator action | Each line is a separate `sh -c`; `set -e` and every variable are lost | One line invoking a real script file |
|
|
167
|
+
| Hardware inputs on both the create and the test step | `config.ini` is re-appended every run; the snapshot is rejected at boot | Pass them on the creation step only, or not at all |
|
|
168
|
+
| Cache key without an image version component | Runner-image bump silently invalidates the snapshot; permanent cold boots | Key on API level, target, arch, and image version |
|
|
169
|
+
| Combined cache step for the AVD | Saves only on success, so the run that built the snapshot never stores it | Split `cache/restore` and `cache/save` |
|
|
170
|
+
| ATD image under a UI driver | SystemUI, launcher, and IME are stripped; hardware rendering is off | Use a standard system image |
|
|
171
|
+
| Floating runner install | Local and CI drift apart silently; behavior differs with no version in the logs | Pin the version and assert the resolved version |
|
|
172
|
+
| Flat artifact glob | Breaks on the run-directory layout change | Resolve the newest run directory and walk it |
|
|
173
|
+
| Diagnosing from the failure screenshot | Taken after teardown; usually shows the launcher | Read the per-step status and the hierarchy dump |
|
|
174
|
+
| Host-side reachability check standing in for the device | Different network namespace; proves nothing about the guest | Prove it from the device, or forward the port over the debug bridge |
|
|
175
|
+
| Device-side connect used to verify an `adb reverse` forward | The mapping itself answers, so the check passes with nothing behind it | Accept a socket in the host process and assert the accept happened |
|
|
176
|
+
| Retries added over a configuration defect | Converts a reproducible failure into an intermittent one | Fix the configuration; keep retries for genuinely nondeterministic steps |
|
|
158
177
|
|
|
159
178
|
## Device Lab Checklist
|
|
160
179
|
|
|
@@ -168,6 +187,7 @@ Read, in this order:
|
|
|
168
187
|
- [ ] **Cache key carries an image version component**
|
|
169
188
|
- [ ] **Hardware acceleration verified on**, not left to chance
|
|
170
189
|
- [ ] **Standard system image**, not an ATD variant
|
|
190
|
+
- [ ] **One device profile across local and CI**, compared on density-independent height rather than pixel resolution
|
|
171
191
|
- [ ] **Locally created AVDs asserted after creation**: `target`, GPU, and keyboard read back rather than assumed written
|
|
172
192
|
- [ ] **Per-device identity written and proven unique** before any sharded run, with a duplicate failing the job
|
|
173
193
|
- [ ] **Artifacts uploaded**: per-step statuses, hierarchy dumps, screenshots, and device logs, resolved by run directory or written to a named output directory
|
|
@@ -75,6 +75,8 @@ Pick the matrix from risk, not from availability. A defensible minimum:
|
|
|
75
75
|
|
|
76
76
|
Run the full matrix nightly and on release candidates. Run the primary target only on PRs, because a PR gate that boots six emulators stops being a gate people wait for.
|
|
77
77
|
|
|
78
|
+
**The PR-gate profile and the local profile must be the same one.** A different screen height moves content below the fold, and `assertVisible` means inside the viewport, so a shorter runner device fails UI assertions that pass on every developer machine and reads as a product defect. Compare on density-independent height rather than pixel resolution: two profiles can share `1080x` and differ by 100dp. The matrix exists to find fragmentation defects, and it can only do that from a gate that is not producing them by accident. See `mobile-ci-device-lab.md`.
|
|
79
|
+
|
|
78
80
|
## CI Shape
|
|
79
81
|
|
|
80
82
|
- **Build artifact first**: decide what the flows run against before writing any of them. A release-shaped build (unsigned APK, simulator IPA) is the default; a prebuilt development shell such as Expo Go is not a CI artifact, because the native modules the flows need are absent and the launch path exists only in CI. This decision sets the failure surface of the whole suite; see `mobile-ci-device-lab.md`.
|
|
@@ -106,6 +108,7 @@ Run the full matrix nightly and on release candidates. Run the primary target on
|
|
|
106
108
|
- [ ] **Duplicate coverage checked**: no device flow proving something a cheaper level already proves
|
|
107
109
|
- [ ] **Mobile risk categories scored**: permissions, lifecycle, connectivity, fragmentation, upgrade
|
|
108
110
|
- [ ] **Device matrix justified**: primary, floor, and form factor chosen from usage data
|
|
111
|
+
- [ ] **PR-gate profile matches the local profile**, compared on density-independent height
|
|
109
112
|
- [ ] **PR gate bounded**: P0 flows on the primary target only
|
|
110
113
|
- [ ] **NFR evidence separated**: cold start, frame rate, memory, binary size instrumented rather than asserted in flows
|
|
111
114
|
- [ ] **No live third-party flag or experiment service in the run path**: remote evaluation disabled, values seeded as test data
|
|
@@ -54,7 +54,7 @@ webhook-timeout-error,WebhookTimeoutError Debugging,"templateName, timeoutMs, to
|
|
|
54
54
|
webhook-providers,Webhook Provider Patterns,"WireMock (deleteById supported), MockServer (deleteById no-op), Mockoon (deleteById no-op, 100-entry limit), custom WebhookProvider interface","webhook,providers,playwright-utils,wiremock,mockserver,mockoon",extended,knowledge/webhook-providers.md
|
|
55
55
|
webhook-risk,Webhook Testing Risk Guidance,"When webhook tests are required, P2×I3 default risk score, complete test checklist, failure patterns and mitigations, TA assessment checklist","webhook,risk,assessment,event-driven,async,playwright-utils,governance",core,knowledge/webhook-risk-guidance.md
|
|
56
56
|
confidence-gate,Confidence Gate,"1-10 confidence scoring with stop-and-ask rule below threshold for selectors, endpoints, risk classification, fixtures, schemas, and data factories — prevents agent fabrication","reliability,agent-safety,generation,quality,governance",core,knowledge/confidence-gate.md
|
|
57
|
-
maestro-flows,Maestro Flow Patterns,"Declarative mobile flow structure, selector hierarchy (id > text > scoped text), clearState isolation, synchronization without sleeps, subflow composition, text selectors as whole-element regex, taps that report COMPLETED without being handled, anti-patterns","mobile,maestro,ios,android,flows,selectors,ui",specialized,knowledge/maestro-flows.md
|
|
58
|
-
mobile-test-strategy,Mobile Test Strategy,"Mobile test level framework (unit/component/contract/device flow), what belongs in a device flow, mobile risk categories (permissions, lifecycle, connectivity, fragmentation, upgrade), device matrix, CI shape, no live third-party flag evaluation in the run path","mobile,maestro,strategy,risk,levels,ci,ios,android",specialized,knowledge/mobile-test-strategy.md
|
|
59
|
-
evidence-integrity,Evidence Integrity,"Falsifiability of checks (hollow green, optional assertions, continue-on-error, partial manifests), three-state diagnostics (pass/fail/could-not-measure), probes that issue the client's own request, verifying the outcome rather than the act, verifying framework properties before use, proving claims from the side that can observe them, environment asymmetry, recording what a change did rather than what it was for","quality,evidence,diagnostics,review,ci,gates,falsifiability",core,knowledge/evidence-integrity.md
|
|
60
|
-
mobile-ci-device-lab,Mobile CI Device Lab,"Build artifact selection (release build vs development shell vs debug-variant development build), development-server manifest signing in non-interactive CI, Android emulator snapshot caching and config.ini pitfalls, repairing locally created AVDs, per-device identity for sharded runs, runner version pinning, artifact layout and failure diagnosis, sharding measured on wall clock","mobile,maestro,ci,android,ios,expo,emulator,artifacts",specialized,knowledge/mobile-ci-device-lab.md
|
|
57
|
+
maestro-flows,Maestro Flow Patterns,"Declarative mobile flow structure, selector hierarchy (id > text > scoped text), clearState isolation, synchronization without sleeps, subflow composition, text selectors as whole-element regex, taps that report COMPLETED without being handled, visible meaning inside the viewport rather than present in the hierarchy, anti-patterns","mobile,maestro,ios,android,flows,selectors,ui",specialized,knowledge/maestro-flows.md
|
|
58
|
+
mobile-test-strategy,Mobile Test Strategy,"Mobile test level framework (unit/component/contract/device flow), what belongs in a device flow, mobile risk categories (permissions, lifecycle, connectivity, fragmentation, upgrade), device matrix with a gate profile matching local, CI shape, no live third-party flag evaluation in the run path","mobile,maestro,strategy,risk,levels,ci,ios,android",specialized,knowledge/mobile-test-strategy.md
|
|
59
|
+
evidence-integrity,Evidence Integrity,"Falsifiability of checks (hollow green, optional assertions, continue-on-error, partial manifests), three-state diagnostics (pass/fail/could-not-measure), probes that issue the client's own request, verifying the outcome rather than the act, verifying framework properties before use, proving claims from the side that can observe them, environment asymmetry including screen geometry and accumulated credentials, ranking hypotheses by the cost of the measurement that kills them, recording what a change did rather than what it was for","quality,evidence,diagnostics,review,ci,gates,falsifiability",core,knowledge/evidence-integrity.md
|
|
60
|
+
mobile-ci-device-lab,Mobile CI Device Lab,"Build artifact selection (release build vs development shell vs debug-variant development build), native modules that degrade silently in a shell, development-server manifest signing in non-interactive CI, one device profile across local and CI, Android emulator snapshot caching and config.ini pitfalls, repairing locally created AVDs, per-device identity for sharded runs, runner version pinning, artifact layout and failure diagnosis, sharding measured on wall clock","mobile,maestro,ci,android,ios,expo,emulator,artifacts",specialized,knowledge/mobile-ci-device-lab.md
|