bmad-method-test-architecture-enterprise 1.22.5 → 1.22.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/CHANGELOG.md +13 -0
- package/docs/reference/knowledge-base.md +14 -14
- package/package.json +1 -1
- package/src/agents/bmad-tea/resources/knowledge/evidence-integrity.md +35 -18
- package/src/agents/bmad-tea/resources/knowledge/maestro-flows.md +76 -19
- package/src/agents/bmad-tea/resources/knowledge/mobile-ci-device-lab.md +6 -2
- package/src/agents/bmad-tea/resources/knowledge/mobile-test-strategy.md +13 -11
- package/src/agents/bmad-tea/resources/tea-index.csv +3 -3
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/evidence-integrity.md +35 -18
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/maestro-flows.md +76 -19
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/mobile-ci-device-lab.md +6 -2
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/mobile-test-strategy.md +13 -11
- package/src/workflows/testarch/bmad-testarch-atdd/resources/tea-index.csv +3 -3
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/evidence-integrity.md +35 -18
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/maestro-flows.md +76 -19
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/mobile-ci-device-lab.md +6 -2
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/mobile-test-strategy.md +13 -11
- package/src/workflows/testarch/bmad-testarch-automate/resources/tea-index.csv +3 -3
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/evidence-integrity.md +35 -18
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/maestro-flows.md +76 -19
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/mobile-ci-device-lab.md +6 -2
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/mobile-test-strategy.md +13 -11
- package/src/workflows/testarch/bmad-testarch-ci/resources/tea-index.csv +3 -3
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/evidence-integrity.md +35 -18
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/maestro-flows.md +76 -19
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/mobile-ci-device-lab.md +6 -2
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/mobile-test-strategy.md +13 -11
- package/src/workflows/testarch/bmad-testarch-framework/resources/tea-index.csv +3 -3
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/evidence-integrity.md +35 -18
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/maestro-flows.md +76 -19
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/mobile-ci-device-lab.md +6 -2
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/mobile-test-strategy.md +13 -11
- package/src/workflows/testarch/bmad-testarch-nfr/resources/tea-index.csv +3 -3
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/evidence-integrity.md +35 -18
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/maestro-flows.md +76 -19
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/mobile-ci-device-lab.md +6 -2
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/mobile-test-strategy.md +13 -11
- package/src/workflows/testarch/bmad-testarch-test-design/resources/tea-index.csv +3 -3
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/evidence-integrity.md +35 -18
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/maestro-flows.md +76 -19
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/mobile-ci-device-lab.md +6 -2
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/mobile-test-strategy.md +13 -11
- package/src/workflows/testarch/bmad-testarch-test-review/resources/tea-index.csv +3 -3
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/evidence-integrity.md +35 -18
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/maestro-flows.md +76 -19
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/mobile-ci-device-lab.md +6 -2
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/mobile-test-strategy.md +13 -11
- package/src/workflows/testarch/bmad-testarch-trace/resources/tea-index.csv +3 -3
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
"name": "bmad-method-test-architecture-enterprise",
|
|
32
32
|
"source": "./",
|
|
33
33
|
"description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
|
|
34
|
-
"version": "1.22.
|
|
34
|
+
"version": "1.22.6",
|
|
35
35
|
"author": {
|
|
36
36
|
"name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
|
|
37
37
|
},
|
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [1.22.6] - 2026-08-14
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
|
|
14
|
+
- `mobile-ci-device-lab.md` and `mobile-test-strategy.md`: corrected the claim, shipped in 1.22.4, that deep links cannot be exercised in a prebuilt development shell. A shell does not register the app's custom scheme, so a `myapp://` URL fails there, but it routes its own URL form (`exp://<host>/--/<path>?<query>`) into the app with path and query intact. Link parsing, routing, and the resulting state changes are therefore all testable in a shell; only the OS-level handoff needs the registered scheme, which means a cold start from a real widget or notification tap. The originating suite had recorded a deep-link flow as impossible for this reason, and it passed on the first attempt once the URL was built the shell's way. The related bullet no longer lists deep links among the surfaces a shell can only assert absent.
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
|
|
18
|
+
- `evidence-integrity.md`: a fifth shape of check that cannot fail, and the hardest of the five to catch in review: the assertion is already true before the action runs. A flow opened a deep link and asserted a container belonging to the screen it was already on, so it passed whether or not the link did anything, and the suite reported every flow green with it included. Carries two tells, both cheap: the test's name promises an effect no assertion mentions, and the result tracks an environment difference the assertions never mention (the identical vacuous flow was green in CI and red locally, because the unresolvable link errored on one API level and resolved on another).
|
|
19
|
+
- `evidence-integrity.md` and `maestro-flows.md`: assert the transition rather than the state. Asserting that an action produced a state passes whenever that state is the application's default, which is the same hollow shape one level up. Where only a single state is available, choose an input whose expected value differs from the default. `maestro-flows.md` carries the worked example with state selectors, and the related preference for a deterministic input over stubbing a clock, since the deterministic input exercises the identical code path with no test-only seam.
|
|
20
|
+
- `maestro-flows.md`: `selected`, `checked`, `enabled`, and `focused` are documented state selectors that compose with `id` and `text` on `tapOn`, `assertVisible`, and `assertNotVisible`, and `assertNotVisible` with a state selector is how "no longer selected" is expressed. Also adds `maestro check-syntax` as the device-free way to confirm a selector or field exists on the pinned version before a run.
|
|
21
|
+
- `mobile-test-strategy.md` and `evidence-integrity.md`: a surface declared untestable is a claim needing evidence like any other. Dropping coverage on an unchecked assumption costs real coverage, and the fix is to test the claim and then name precisely which part is out of reach.
|
|
22
|
+
|
|
10
23
|
## [1.22.5] - 2026-08-14
|
|
11
24
|
|
|
12
25
|
### Added
|
|
@@ -73,15 +73,15 @@ CI/CD patterns, burn-in testing, and selective test execution.
|
|
|
73
73
|
|
|
74
74
|
Test quality standards, test level selection, TDD patterns, and the generation-safety gate.
|
|
75
75
|
|
|
76
|
-
| Fragment | Tier | Description
|
|
77
|
-
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
|
|
78
|
-
| [test-quality](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-quality.md) | core | Execution limits, isolation rules, green criteria
|
|
79
|
-
| [test-levels](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-levels-framework.md) | core | Guidelines for choosing unit, integration, or end-to-end coverage
|
|
80
|
-
| [test-priorities](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-priorities-matrix.md) | core | P0–P3 criteria, coverage targets, execution ordering
|
|
81
|
-
| [test-healing-patterns](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-healing-patterns.md) | core | Common failure patterns and automated fixes
|
|
82
|
-
| [confidence-gate](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/confidence-gate.md) | core | 1-10 confidence score with a stop-and-ask rule below threshold, so the agent declares unknowns instead of fabricating them
|
|
83
|
-
| [evidence-integrity](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/evidence-integrity.md) | core | Checks that cannot fail, three-state diagnostics, probes that issue the client's own request, verifying the outcome rather than the act, verifying framework properties before use, stating environment asymmetry including screen geometry and accumulated credentials, ranking hypotheses by the cost of the measurement that kills them, and recording what a change did rather than what it was for | Falsifiability, hollow green, diagnostics |
|
|
84
|
-
| [component-tdd](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/component-tdd.md) | extended | Red→green→refactor workflow, provider isolation
|
|
76
|
+
| Fragment | Tier | Description | Key Topics |
|
|
77
|
+
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
|
|
78
|
+
| [test-quality](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-quality.md) | core | Execution limits, isolation rules, green criteria | DoD, best practices, anti-patterns |
|
|
79
|
+
| [test-levels](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-levels-framework.md) | core | Guidelines for choosing unit, integration, or end-to-end coverage | Test pyramid, level selection |
|
|
80
|
+
| [test-priorities](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-priorities-matrix.md) | core | P0–P3 criteria, coverage targets, execution ordering | Prioritization, risk-based testing |
|
|
81
|
+
| [test-healing-patterns](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/test-healing-patterns.md) | core | Common failure patterns and automated fixes | Debugging, healing, fixes |
|
|
82
|
+
| [confidence-gate](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/confidence-gate.md) | core | 1-10 confidence score with a stop-and-ask rule below threshold, so the agent declares unknowns instead of fabricating them | Agent safety, generation, governance |
|
|
83
|
+
| [evidence-integrity](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/evidence-integrity.md) | core | Checks that cannot fail including assertions already true before the action, three-state diagnostics, probes that issue the client's own request, verifying the outcome rather than the act, verifying framework properties before use, stating environment asymmetry including screen geometry and accumulated credentials, ranking hypotheses by the cost of the measurement that kills them, and recording what a change did rather than what it was for | Falsifiability, hollow green, diagnostics |
|
|
84
|
+
| [component-tdd](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/component-tdd.md) | extended | Red→green→refactor workflow, provider isolation | TDD, component testing |
|
|
85
85
|
|
|
86
86
|
**Used in:** `test-design`, `atdd`, `automate`, `test-review`, `trace`
|
|
87
87
|
|
|
@@ -179,11 +179,11 @@ Delivery-side testing for asynchronous, eventually-consistent webhook flows usin
|
|
|
179
179
|
|
|
180
180
|
Maestro device flows and the level discipline that decides what becomes a flow at all. Loaded when `test_stack_type` is `mobile` or when the review set contains a Maestro flow (`.yaml`/`.yml` under `maestro/` or `.maestro/`, or `*.flow.yaml` or `*.flow.yml`).
|
|
181
181
|
|
|
182
|
-
| Fragment | Tier | Description
|
|
183
|
-
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
|
|
184
|
-
| [mobile-test-strategy](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/mobile-test-strategy.md) | specialized | Mobile test level framework, what belongs in a device flow, mobile risk categories, device matrix with a gate profile matching local, CI shape, no live third-party flag evaluation in the run path
|
|
185
|
-
| [maestro-flows](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/maestro-flows.md) | specialized | Flow structure, selector hierarchy, `clearState` isolation, synchronization without sleeps, subflow composition, command semantics that differ by platform, `text:` selectors as whole-element regex, taps that report COMPLETED without being handled, visible meaning inside the viewport
|
|
186
|
-
| [mobile-ci-device-lab](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/mobile-ci-device-lab.md) | specialized | Build artifact selection including why a debug-variant development build does not solve it, dev-server manifest signing in non-interactive CI, one device profile across local and CI, native modules that degrade silently in a shell, emulator snapshot caching, repairing locally created AVDs, per-device identity for sharded runs, runner version pinning, artifact layout and failure diagnosis | Build artifact, emulator, caching, pinning, sharding, artifacts |
|
|
182
|
+
| Fragment | Tier | Description | Key Topics |
|
|
183
|
+
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
|
|
184
|
+
| [mobile-test-strategy](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/mobile-test-strategy.md) | specialized | Mobile test level framework, what belongs in a device flow, mobile risk categories, device matrix with a gate profile matching local, CI shape, no live third-party flag evaluation in the run path | Levels, risk, device matrix, permissions, lifecycle |
|
|
185
|
+
| [maestro-flows](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/maestro-flows.md) | specialized | Flow structure, selector hierarchy, `clearState` isolation, synchronization without sleeps, subflow composition, command semantics that differ by platform, `text:` selectors as whole-element regex, taps that report COMPLETED without being handled, visible meaning inside the viewport, asserting the transition rather than a state that may already hold | Maestro, selectors, isolation, regex selectors, anti-patterns |
|
|
186
|
+
| [mobile-ci-device-lab](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/blob/main/src/agents/bmad-tea/resources/knowledge/mobile-ci-device-lab.md) | specialized | Build artifact selection including why a debug-variant development build does not solve it, dev-server manifest signing in non-interactive CI, one device profile across local and CI, native modules that degrade silently in a shell, deep links reachable through the shell's routed URL form, emulator snapshot caching, repairing locally created AVDs, per-device identity for sharded runs, runner version pinning, artifact layout and failure diagnosis | Build artifact, emulator, caching, pinning, sharding, artifacts |
|
|
187
187
|
|
|
188
188
|
**Used in:** `framework`, `automate`, `atdd`, `test-design`, `test-review`, `ci` (when `test_stack_type` is `mobile` or a Maestro flow is present)
|
|
189
189
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.22.
|
|
4
|
+
"version": "1.22.6",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
|
@@ -21,7 +21,7 @@ A suite lies in two ways. A test that **cannot fail** reports coverage it does n
|
|
|
21
21
|
|
|
22
22
|
### Example 1: The Check That Cannot Fail
|
|
23
23
|
|
|
24
|
-
**Context**:
|
|
24
|
+
**Context**: Five shapes found live in one suite that reported success on every run.
|
|
25
25
|
|
|
26
26
|
**Implementation**:
|
|
27
27
|
|
|
@@ -81,10 +81,20 @@ flows:
|
|
|
81
81
|
|
|
82
82
|
**Shape 4** has no snippet, because the step looks correct: an assertion passes on iOS because the element is still in the hierarchy behind a presented modal, and fails on Android where the modal replaces the hierarchy. Same assertion, different meaning per platform. Any assertion whose truth depends on how a platform composes its view tree needs its own per-platform expectation, not one shared line.
|
|
83
83
|
|
|
84
|
+
**Shape 5: the assertion is about something that was already true before the action.** A flow opened a deep link and then asserted that a container belonging to the screen it was already on was visible. The container predated the link, so the check held whether or not the link did anything, and on one platform it did nothing. The suite reported all flows green with this one included, and the green was stable rather than intermittent.
|
|
85
|
+
|
|
86
|
+
Two tells for it, both cheap:
|
|
87
|
+
|
|
88
|
+
- **The name promises an effect the assertions never mention.** "Widget Deep Link Hydration" asserted the presence of a container, not that anything had hydrated. Read the flow's name as a claim and check that some assertion carries it. A name is the only place many suites record what a test was for, which makes disagreement between name and assertion a reliable smell.
|
|
89
|
+
- **The result differs across environments for reasons unrelated to what it asserts.** The identical vacuous flow was green in CI and red locally, because on one API level the unresolvable link errored and on another it resolved somewhere and the open step completed. When a flow's outcome tracks an environment difference that its assertions never mention, suspect that the assertions are not what is deciding the result.
|
|
90
|
+
|
|
91
|
+
The fix generalizes past this shape: **assert the transition rather than the state.** Where only a single state is available, choose an input whose expected value differs from the application's default, so agreement with the default cannot carry the pass. Asserting "the morning option is selected" after an action that selects morning proves nothing in an app that starts on morning; asserting that a second action **moved** the selection, and that the first option is no longer selected, cannot pass without the action working.
|
|
92
|
+
|
|
84
93
|
**Key points**:
|
|
85
94
|
|
|
86
95
|
- Name the input that would turn each check red. If none exists, the check is decoration.
|
|
87
|
-
- `optional: true`, `continue-on-error`, a partial manifest, and a soft assertion are
|
|
96
|
+
- `optional: true`, `continue-on-error`, a partial manifest, and a soft assertion are four common ways a result stops being falsifiable.
|
|
97
|
+
- A fifth, and the hardest to see in review: the assertion is true before the action runs. Nothing about the step looks wrong, and the green is stable.
|
|
88
98
|
- **When you make a hollow check falsifiable and it goes red, the red is the finding.** It is a defect that was always there and is now visible. Reporting it as a regression you introduced is the wrong read and usually gets the fix reverted.
|
|
89
99
|
|
|
90
100
|
### Example 2: Diagnostics Need a Could-Not-Measure State
|
|
@@ -202,27 +212,34 @@ The corollary for a suite: that cheap discriminating measurement should already
|
|
|
202
212
|
|
|
203
213
|
## Anti-Patterns
|
|
204
214
|
|
|
205
|
-
| Anti-pattern
|
|
206
|
-
|
|
|
207
|
-
| Assertion with a soft or optional modifier as default
|
|
208
|
-
|
|
|
209
|
-
|
|
|
210
|
-
|
|
|
211
|
-
|
|
|
212
|
-
|
|
|
213
|
-
|
|
|
214
|
-
|
|
|
215
|
-
|
|
|
216
|
-
|
|
|
217
|
-
|
|
|
218
|
-
|
|
|
219
|
-
|
|
|
220
|
-
|
|
|
215
|
+
| Anti-pattern | Why it fails | Fix |
|
|
216
|
+
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
|
217
|
+
| Assertion with a soft or optional modifier as default | Cannot go red; reports coverage that does not exist | Reserve softness for genuinely optional UI, and assert the outcome hard |
|
|
218
|
+
| Assertion that was already true before the action | Cannot distinguish the action working from the action doing nothing | Assert the transition, or an input whose expected state differs from the default |
|
|
219
|
+
| Flow name promising an effect no assertion mentions | The name records the intent and nothing checks it | Read the name as a claim; make some assertion carry it |
|
|
220
|
+
| Outcome tracking an environment difference the assertions never mention | Something other than the assertions is deciding the result | Suspect a vacuous check before investigating the environment |
|
|
221
|
+
| A surface declared untestable without a test of the claim | Coverage is dropped on an assumption, and the assumption is often wrong | Try it; then name precisely which part is out of reach and why |
|
|
222
|
+
| `continue-on-error` on the test step | The suite cannot fail the build | Put it on artifact collection only; use `if: always()` for uploads |
|
|
223
|
+
| Runner manifest listing a subset of the suite | Files silently never run; the count is the only clue | Include by pattern; assert the executed count against the file count |
|
|
224
|
+
| Missing tool reported as a failed condition | Sends the investigation at the wrong subsystem | Three-state probes; distinct exit code for could-not-measure |
|
|
225
|
+
| Probe observing a proxy that correlates with the target | Passes for a reason unrelated to the claim, and a green is never re-examined | Ask what else could make this pass; observe the thing itself |
|
|
226
|
+
| Probe sending a different request than the client | Takes a different branch through the server: healthy probe, failing app, both correct | Copy the client's method, headers, and body into the probe, and log what was sent |
|
|
227
|
+
| Written setting read back and reported as effect | Proves the write; some configuration latches at boot and never applies live | Assert the observable the claim is about, not the act that was supposed to produce it |
|
|
228
|
+
| Change described by its intent once its effect is known | Reads as a found root cause and stops the next investigation looking | Record what it actually did, and why it was kept |
|
|
229
|
+
| Mechanisms elaborated while the cheap measurement goes untaken | Plausibility feels like progress; several explanations, no new evidence | Rank hypotheses by the cost of the observation that would kill them |
|
|
230
|
+
| Verdict emitted by the side that cannot observe it | Proves the wrong namespace | Move the assertion to the party whose route or state is in question |
|
|
231
|
+
| Comment asserting a mechanism with no source read | Propagates into other files and into other people's reasoning | Cite the doc or source line, or omit the mechanism |
|
|
232
|
+
| Local result used as a CI argument, asymmetry unstated | Hides the axes that actually differ | Tabulate the differing axes with the claim |
|
|
233
|
+
| Environment probe running after its consumers | The answer arrives too late to configure anything | Resolve environment-dependent values first, then derive |
|
|
234
|
+
| Reverting a newly-red check as a regression | Restores the hollow green and loses the finding | Treat the red as the pre-existing defect it exposed |
|
|
221
235
|
|
|
222
236
|
## Evidence Integrity Checklist
|
|
223
237
|
|
|
224
238
|
- [ ] **Every check is falsifiable**: for each assertion, the input that turns it red is nameable
|
|
225
239
|
- [ ] **No soft assertion by default**: optional modifiers only on genuinely optional UI, with a comment
|
|
240
|
+
- [ ] **The assertion carrying the outcome post-dates its action**: a precondition assertion is allowed when it is labelled as one, and nothing already true before the step is presented as proof of it
|
|
241
|
+
- [ ] **Names reconciled with assertions**: what a test is called is carried by something that can fail
|
|
242
|
+
- [ ] **Untestable claims tested**: a surface is dropped from coverage only after the claim itself has been checked
|
|
226
243
|
- [ ] **No `continue-on-error` on a test step**: only on artifact collection
|
|
227
244
|
- [ ] **Executed count reconciled**: the number of tests that ran matches the number of test files discovered
|
|
228
245
|
- [ ] **Platform-divergent assertions split**: no single assertion whose meaning depends on how a platform composes its view tree
|
|
@@ -432,27 +432,81 @@ Measured in isolation: the tap completed in 2.4 seconds, the target was present
|
|
|
432
432
|
- **Any `assertVisible` on an element inside a scrolling section is a screen-height dependency** until the flow scrolls to it. It will pass wherever it was written and fail on the first shorter device.
|
|
433
433
|
- This is the failure mode most likely to reproduce only in CI, because the runner's device profile is rarely the one the flow was written against. `mobile-ci-device-lab.md` carries the profile-parity rule.
|
|
434
434
|
|
|
435
|
+
### Example 10: Assert the Change, Not a State That May Already Hold
|
|
436
|
+
|
|
437
|
+
**Context**: A deep-link flow named "Widget Deep Link Hydration" that opened a link and asserted the home screen's own container was visible. The container was there before the link, so the assertion held whether or not the link did anything. On one platform it did nothing, and the suite reported 18 of 18.
|
|
438
|
+
|
|
439
|
+
Two shapes are stacked here, and the second one catches people who have already fixed the first.
|
|
440
|
+
|
|
441
|
+
**Shape one: the assertion targets something that predates the action.** A container, a screen root, or a nav bar that was on screen before the step cannot be evidence that the step worked. The tell is that the flow's name describes an effect the assertions never mention.
|
|
442
|
+
|
|
443
|
+
**Shape two: the assertion targets the right thing, in a state the app might already be in.** Asserting "morning is selected" after a link that selects morning passes whenever morning is the default. The check is about the action but still cannot fail.
|
|
444
|
+
|
|
445
|
+
**Implementation**:
|
|
446
|
+
|
|
447
|
+
```yaml
|
|
448
|
+
# ❌ Both shapes. The container predates the link, and even a corrected
|
|
449
|
+
# assertion on the default scenario would pass without the link doing anything.
|
|
450
|
+
- openLink: ${WIDGET_URL_AM}
|
|
451
|
+
- assertVisible:
|
|
452
|
+
id: 'scenario-toggles' # already on screen before the link
|
|
453
|
+
|
|
454
|
+
# ✅ Assert the transition. State the precondition, act, then assert both that
|
|
455
|
+
# the new state holds and that the old one no longer does. Every link in the
|
|
456
|
+
# flow has to earn its own pass this way, including the first.
|
|
457
|
+
- assertVisible:
|
|
458
|
+
id: 'scenario-toggle-morning'
|
|
459
|
+
selected: true # PRECONDITION, not the outcome: names the state moved FROM
|
|
460
|
+
|
|
461
|
+
- openLink: ${WIDGET_URL_EVENING}
|
|
462
|
+
- assertVisible:
|
|
463
|
+
id: 'scenario-toggle-evening'
|
|
464
|
+
selected: true
|
|
465
|
+
- assertNotVisible:
|
|
466
|
+
id: 'scenario-toggle-morning'
|
|
467
|
+
selected: true # the move is what proves this link was applied
|
|
468
|
+
|
|
469
|
+
- openLink: ${WIDGET_URL_AM}
|
|
470
|
+
- assertVisible:
|
|
471
|
+
id: 'scenario-toggle-morning'
|
|
472
|
+
selected: true
|
|
473
|
+
- assertNotVisible:
|
|
474
|
+
id: 'scenario-toggle-evening'
|
|
475
|
+
selected: true # and the second link proves itself the same way
|
|
476
|
+
```
|
|
477
|
+
|
|
478
|
+
**Key points**:
|
|
479
|
+
|
|
480
|
+
- **Prefer asserting a change over asserting a state.** Where a single state is all you have, pick an input whose expected result differs from the app's default, so agreement with the default cannot carry the pass.
|
|
481
|
+
- **A precondition assertion is legitimate, and it is not the outcome.** Naming the state being moved from is what makes the following change meaningful, so mark it as a precondition and never let it stand in for proof that the action worked. Ordering the links so the first one also moves the selection is what stops the first step passing on the default, which is the trap the corrected block above is arranged to avoid.
|
|
482
|
+
- **Choose a deterministic input.** The same flow had link variants resolving from the current time and the day's forecast, which cannot be asserted without freezing the clock. Two other variants mapped fixed values straight through the same parse-route-apply path. Reach for the deterministic input rather than reaching for a clock stub: it exercises the identical code path and needs no test-only seam.
|
|
483
|
+
- `selected`, `checked`, `enabled`, and `focused` are documented state selectors and compose with `id` and `text` on `tapOn`, `assertVisible`, and `assertNotVisible`. `assertNotVisible` with a state selector is how "no longer selected" is expressed.
|
|
484
|
+
- **Check the syntax before the run.** `maestro check-syntax` validates flow files without a device, which is the cheap way to confirm a selector or field exists on the version you pin rather than discovering it in a red run.
|
|
485
|
+
|
|
435
486
|
## Anti-Patterns
|
|
436
487
|
|
|
437
|
-
| Anti-pattern | Why it fails | Fix
|
|
438
|
-
| ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------- |
|
|
439
|
-
| Flow with no `assertVisible`/`assertTrue` | Passes as long as taps land; proves nothing about behavior | Assert the destination state of every flow
|
|
440
|
-
| `sleep` used as synchronization | Flaky under load, slow when not | `extendedWaitUntil` on the real condition
|
|
441
|
-
| `tapOn: index:` on a list | Breaks when the list reorders or the backend returns a different order | Scope by `text` with `below`/`containsChild`
|
|
442
|
-
| `tapOn: point:` coordinates | Breaks on a different screen size or density | Address the element by `id`
|
|
443
|
-
| No `clearState` | Flow depends on whatever ran before it; unreproducible in isolation | `clearState` before `launchApp`
|
|
444
|
-
| Hardcoded credential or PII | Leaks in the repo and in CI logs | `${ENV_VAR}` sourced from the CI secret store
|
|
445
|
-
| One flow covering six user journeys | A failure names the flow, not the behavior; slow to diagnose | One journey per flow, composed from subflows
|
|
446
|
-
| Required assertion inside `when:` | Turns a real failure into a silent pass | Guard only genuinely optional UI (permission dialogs, upsells)
|
|
447
|
-
| `back` used as a cross-platform step | Documented for Android and Web only; on iOS it does nothing and reports COMPLETED | Split with `runFlow: when: platform:`; tap the app's own control on iOS
|
|
448
|
-
| `hideKeyboard` with a modal open | The Android implementation is the system back key, which dismisses a React Native modal | Tap a non-interactive element to drop the keyboard
|
|
449
|
-
| `optional: true` on the assertion that carries the outcome | The step cannot fail, so the flow reports coverage it does not have | Assert hard; reserve `optional` for genuinely optional UI
|
|
450
|
-
| `waitForAnimationToEnd` used as a wait-for-content | It succeeds when its cap is reached, so it cannot fail | `extendedWaitUntil` on the content that must appear
|
|
451
|
-
| Unescaped regex characters in a `text:` selector | The value is a regex matched against the element's entire text, so it can never pass | Escape literal `(`, `)`, `[`, `]`, `.`; pad partial matches with `.*`
|
|
452
|
-
| Tap status treated as proof the app handled the tap | `tapOn` reports COMPLETED once the touch is dispatched, and touches do get lost | Pair each state-changing tap with its own assertion inside a `retry`
|
|
453
|
-
| One assertion at the end of a tap sequence | Cannot name which tap was lost, so the first hypothesis is a guess | Assert after every action that changes state
|
|
454
|
-
| `scrollUntilVisible` reused after an earlier search moved the list | It travels only in the direction given, so the second search scrolls away from the target | Name the opposite `direction`, or return to a known position first
|
|
455
|
-
| `assertVisible` on an element inside a scrolling section, with no scroll | Visible means inside the viewport; the element is present, correct, and below the fold | `scrollUntilVisible` first, then assert
|
|
488
|
+
| Anti-pattern | Why it fails | Fix |
|
|
489
|
+
| ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
|
490
|
+
| Flow with no `assertVisible`/`assertTrue` | Passes as long as taps land; proves nothing about behavior | Assert the destination state of every flow |
|
|
491
|
+
| `sleep` used as synchronization | Flaky under load, slow when not | `extendedWaitUntil` on the real condition |
|
|
492
|
+
| `tapOn: index:` on a list | Breaks when the list reorders or the backend returns a different order | Scope by `text` with `below`/`containsChild` |
|
|
493
|
+
| `tapOn: point:` coordinates | Breaks on a different screen size or density | Address the element by `id` |
|
|
494
|
+
| No `clearState` | Flow depends on whatever ran before it; unreproducible in isolation | `clearState` before `launchApp` |
|
|
495
|
+
| Hardcoded credential or PII | Leaks in the repo and in CI logs | `${ENV_VAR}` sourced from the CI secret store |
|
|
496
|
+
| One flow covering six user journeys | A failure names the flow, not the behavior; slow to diagnose | One journey per flow, composed from subflows |
|
|
497
|
+
| Required assertion inside `when:` | Turns a real failure into a silent pass | Guard only genuinely optional UI (permission dialogs, upsells) |
|
|
498
|
+
| `back` used as a cross-platform step | Documented for Android and Web only; on iOS it does nothing and reports COMPLETED | Split with `runFlow: when: platform:`; tap the app's own control on iOS |
|
|
499
|
+
| `hideKeyboard` with a modal open | The Android implementation is the system back key, which dismisses a React Native modal | Tap a non-interactive element to drop the keyboard |
|
|
500
|
+
| `optional: true` on the assertion that carries the outcome | The step cannot fail, so the flow reports coverage it does not have | Assert hard; reserve `optional` for genuinely optional UI |
|
|
501
|
+
| `waitForAnimationToEnd` used as a wait-for-content | It succeeds when its cap is reached, so it cannot fail | `extendedWaitUntil` on the content that must appear |
|
|
502
|
+
| Unescaped regex characters in a `text:` selector | The value is a regex matched against the element's entire text, so it can never pass | Escape literal `(`, `)`, `[`, `]`, `.`; pad partial matches with `.*` |
|
|
503
|
+
| Tap status treated as proof the app handled the tap | `tapOn` reports COMPLETED once the touch is dispatched, and touches do get lost | Pair each state-changing tap with its own assertion inside a `retry` |
|
|
504
|
+
| One assertion at the end of a tap sequence | Cannot name which tap was lost, so the first hypothesis is a guess | Assert after every action that changes state |
|
|
505
|
+
| `scrollUntilVisible` reused after an earlier search moved the list | It travels only in the direction given, so the second search scrolls away from the target | Name the opposite `direction`, or return to a known position first |
|
|
506
|
+
| `assertVisible` on an element inside a scrolling section, with no scroll | Visible means inside the viewport; the element is present, correct, and below the fold | `scrollUntilVisible` first, then assert |
|
|
507
|
+
| Assertion targets a container that predates the action | It was on screen before the step, so it cannot be evidence the step worked | Assert the action's own effect; the flow's name should name what it asserts |
|
|
508
|
+
| Assertion on a state the app may already be in | Passes whenever the expected value is the default, so the action is not under test | Assert the transition, or pick an input whose expected state differs from the default |
|
|
509
|
+
| Time- or forecast-dependent input in a flow assertion | The expected value cannot be stated without freezing the clock | Pick a deterministic input through the same code path |
|
|
456
510
|
|
|
457
511
|
## Maestro Flow Checklist
|
|
458
512
|
|
|
@@ -473,6 +527,9 @@ Before merging a flow:
|
|
|
473
527
|
- [ ] **Every state-changing tap has its own assertion**, rather than one assertion covering a sequence
|
|
474
528
|
- [ ] **`retry` scoped to a single step**, with `maxRetries` inside the documented 0-3 range
|
|
475
529
|
- [ ] **Nothing asserted below the fold**: every assertion on an element inside a scrolling section is preceded by a scroll to it
|
|
530
|
+
- [ ] **The assertion carrying the outcome post-dates its action**: a precondition assertion is allowed when it is labelled as one, and nothing already true before the step is presented as proof of it
|
|
531
|
+
- [ ] **Transitions asserted where possible**: a single-state assertion is justified only when the expected value differs from the default
|
|
532
|
+
- [ ] **Syntax checked**: `maestro check-syntax` run against the pinned version before the flow reaches a device
|
|
476
533
|
|
|
477
534
|
## Integration Points
|
|
478
535
|
|
|
@@ -25,7 +25,9 @@ Before a single flow is written, decide **which artifact the flows run against**
|
|
|
25
25
|
| **Development build / dev client** | The app with dev tooling, all native modules present | A build step, plus a dev server when the JS bundle is served rather than embedded | Local iteration, debug flows |
|
|
26
26
|
| **Prebuilt development shell (for example Expo Go)** | That the JS runs inside someone else's container | A live dev server, a manifest exchange, and a launch through a third-party app | Manual smoke work only |
|
|
27
27
|
|
|
28
|
-
**Rule**: the prebuilt shell is the wrong artifact for E2E. It cannot load your native modules, so any flow touching notifications, OAuth,
|
|
28
|
+
**Rule**: the prebuilt shell is the wrong artifact for E2E. It cannot load your native modules, so for any flow touching notifications, OAuth, maps, in-app purchases, or any feature that hands an API key to native code, **a pass in the shell does not prove the native implementation ran**. It proves that whatever the app did in the shell's absence of that module did not throw. Where the module has a fallback path, the flow is exercising the fallback, and the native behavior stays unverified no matter how green the run is. Require evidence that the native path itself was exercised before counting such a flow as coverage, which in practice means running it against a real build.
|
|
29
|
+
|
|
30
|
+
**Be precise about which part is unreachable, because the coarse version of this rule is wrong and gets flows abandoned that would pass.** Deep links are the case worth stating carefully. A shell cannot register the app's custom scheme, so a `myapp://` URL fails there. It can still route its own URL form into the app with the path and query intact, so the app's link parsing, routing, and resulting state changes are all testable. Only the OS-level handoff is out of reach: a cold start from a real widget or notification tap. One suite recorded a deep-link flow as impossible under the shell for exactly this reason, and the flow passed on the first attempt once the URL was built the shell's way. "Untestable here" is a claim that needs the same evidence as any other.
|
|
29
31
|
|
|
30
32
|
**"Absent" is not always the shape it takes, and the other shape is worse.** Some SDKs detect the shell and degrade instead of failing. One in-app-purchase SDK logs `Expo Go app detected. Using RevenueCat in Browser Mode.` and keeps working through a different code path. Nothing errors, the flow proceeds, and what the suite proves is that the fallback path works in a container users never run. A hard `undefined` at least announces itself; a silent degradation gives you a green flow covering the wrong implementation. When a flow touches a native module in a shell, check the device log for what the module decided to do rather than assuming it did nothing.
|
|
31
33
|
|
|
@@ -42,7 +44,9 @@ Getting to a release-variant APK without an account or a build service:
|
|
|
42
44
|
- **`eas build --local` composes badly with CI caching.** Expo documents "Caching is not supported" for local builds, and they still require `eas login` or an `EXPO_TOKEN`. `npx expo prebuild` followed by `./gradlew :app:assembleRelease` is the path that caches. See <https://docs.expo.dev/build-reference/local-builds/>.
|
|
43
45
|
- **A locally prebuilt release APK is debug-signed.** The generated `android/app/build.gradle` sets `release { signingConfig signingConfigs.debug }`, so the artifact installs on an emulator with no credentials, which is exactly what a device lab needs and not something to "fix".
|
|
44
46
|
- **`__DEV__` is `false` in a release build.** Any E2E affordance gated behind it silently disappears in the one build the suite is meant to run against. Move the switch to an `EXPO_PUBLIC_`-prefixed variable, which is inlined into the bundle at build time. Expo documents these as "visible in plain-text in your compiled application", so whatever the switch gates must be safe to ship: a throwaway credential against a disposable environment, never a real one.
|
|
45
|
-
- **
|
|
47
|
+
- **Custom-scheme deep links need a different URL in the shell, and that is a constraint rather than a hole.** `scheme` is documented as "a build-time configuration, it has no effect in Expo Go", so the shell never registers the app's own scheme and a `myapp://` link fails there with `Activity not started, unable to resolve Intent`. What works is the shell's routed form, `exp://<host>/--/<path>?<query>`, which delivers the path and query string into the app. The parsing, routing, and application logic behind a deep link is therefore fully reachable in a shell. What is not reachable is anything that needs the production scheme registered with the OS: a cold start from a real widget or notification tap, or another app handing the link over. Build the URL from one helper used by every flow, so no single platform branch quietly hardcodes the unregistered scheme.
|
|
48
|
+
|
|
49
|
+
Universal Links and Android App Links are a different mechanism and get no such reprieve. They are HTTPS links resolved through a domain association (`apple-app-site-association`, `assetlinks.json`) that the OS fetches and verifies, so the shell's routed form exercises the in-app routing and says nothing about whether the handoff would have happened. Verify the association against a real build, and treat routing coverage and handoff coverage as two separate claims.
|
|
46
50
|
|
|
47
51
|
## If the Suite Must Run Against a Dev Server
|
|
48
52
|
|
|
@@ -35,7 +35,8 @@ Promote to a Maestro flow when the risk is in the **integration**, not the logic
|
|
|
35
35
|
|
|
36
36
|
- P0 revenue or access journeys end to end (sign in, purchase, submit claim)
|
|
37
37
|
- OS permission grants and denials, including the denied path
|
|
38
|
-
- Deep link and
|
|
38
|
+
- Deep link entry into a specific screen. The app's own parsing and routing is testable even in a development shell, through the shell's routed URL form; the OS-level handoff needs the app's custom scheme registered, which a shell does not do.
|
|
39
|
+
- Universal Links and Android App Links are a **separate** surface from custom-scheme deep links, and the same shell trick does not cover them. A verified HTTPS link depends on the platform's domain association (`apple-app-site-association`, `assetlinks.json`) being served and accepted, so a shell-routed URL exercises the in-app routing while proving nothing about whether the OS would have handed the link over at all. Score and cover the association separately from the routing.
|
|
39
40
|
- Background, foreground, and process-death restoration
|
|
40
41
|
- Offline and reconnect behavior
|
|
41
42
|
- Push notification tap-through
|
|
@@ -90,16 +91,17 @@ Run the full matrix nightly and on release candidates. Run the primary target on
|
|
|
90
91
|
|
|
91
92
|
## Anti-Patterns
|
|
92
93
|
|
|
93
|
-
| Anti-pattern | Why it fails
|
|
94
|
-
| ------------------------------------------------------ |
|
|
95
|
-
| Every acceptance criterion becomes a device flow | Suite runtime explodes; failures are slow to diagnose
|
|
96
|
-
| Full device matrix on every PR | Gate becomes too slow to block on, so people bypass it
|
|
97
|
-
| Performance asserted inside a Maestro flow | Measures the harness, not the app
|
|
98
|
-
| Flows depend on a shared logged-in account | Parallel runs collide; failures are not reproducible
|
|
99
|
-
| No offline or permission-denied coverage | The paths users actually hit in the wild are the untested ones
|
|
100
|
-
| Testing against a production backend | Non-deterministic data, and a test order can mutate real state
|
|
101
|
-
| Device flows run through a prebuilt development shell | Native modules are absent, so
|
|
102
|
-
|
|
|
94
|
+
| Anti-pattern | Why it fails | Fix |
|
|
95
|
+
| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
|
96
|
+
| Every acceptance criterion becomes a device flow | Suite runtime explodes; failures are slow to diagnose | Apply the level framework and the duplicate-coverage guard |
|
|
97
|
+
| Full device matrix on every PR | Gate becomes too slow to block on, so people bypass it | Primary target on PRs, matrix nightly |
|
|
98
|
+
| Performance asserted inside a Maestro flow | Measures the harness, not the app | Platform instrumentation as NFR evidence |
|
|
99
|
+
| Flows depend on a shared logged-in account | Parallel runs collide; failures are not reproducible | Per-run accounts/data or explicit backend reset when server state changes |
|
|
100
|
+
| No offline or permission-denied coverage | The paths users actually hit in the wild are the untested ones | Score them as risks; they are usually P0 or P1 |
|
|
101
|
+
| Testing against a production backend | Non-deterministic data, and a test order can mutate real state | Dedicated environment or a stubbed backend |
|
|
102
|
+
| Device flows run through a prebuilt development shell | Native modules are absent, so notifications and payments can only be asserted missing; the launch path is CI-only | Build a release-shaped artifact and install it (`mobile-ci-device-lab.md`) |
|
|
103
|
+
| A surface written off as untestable without a check | The coarse claim is often wrong: a shell routes its own deep-link URL form into the app, so link handling is reachable | Test the claim before dropping coverage; name the part that is genuinely out of reach |
|
|
104
|
+
| Flow asserts behavior behind a remotely evaluated flag | A third-party service decides the outcome, for a user created seconds earlier | Unconfigure the remote provider so the seeded local value wins, and seed it as test data |
|
|
103
105
|
|
|
104
106
|
## Mobile Strategy Checklist
|
|
105
107
|
|
|
@@ -54,7 +54,7 @@ webhook-timeout-error,WebhookTimeoutError Debugging,"templateName, timeoutMs, to
|
|
|
54
54
|
webhook-providers,Webhook Provider Patterns,"WireMock (deleteById supported), MockServer (deleteById no-op), Mockoon (deleteById no-op, 100-entry limit), custom WebhookProvider interface","webhook,providers,playwright-utils,wiremock,mockserver,mockoon",extended,knowledge/webhook-providers.md
|
|
55
55
|
webhook-risk,Webhook Testing Risk Guidance,"When webhook tests are required, P2×I3 default risk score, complete test checklist, failure patterns and mitigations, TA assessment checklist","webhook,risk,assessment,event-driven,async,playwright-utils,governance",core,knowledge/webhook-risk-guidance.md
|
|
56
56
|
confidence-gate,Confidence Gate,"1-10 confidence scoring with stop-and-ask rule below threshold for selectors, endpoints, risk classification, fixtures, schemas, and data factories — prevents agent fabrication","reliability,agent-safety,generation,quality,governance",core,knowledge/confidence-gate.md
|
|
57
|
-
maestro-flows,Maestro Flow Patterns,"Declarative mobile flow structure, selector hierarchy (id > text > scoped text), clearState isolation, synchronization without sleeps, subflow composition, text selectors as whole-element regex, taps that report COMPLETED without being handled, visible meaning inside the viewport rather than
|
|
57
|
+
maestro-flows,Maestro Flow Patterns,"Declarative mobile flow structure, selector hierarchy (id > text > scoped text), clearState isolation, synchronization without sleeps, subflow composition, text selectors as whole-element regex, taps that report COMPLETED without being handled, visible meaning inside the viewport, asserting the transition rather than a state that may already hold, anti-patterns","mobile,maestro,ios,android,flows,selectors,ui",specialized,knowledge/maestro-flows.md
|
|
58
58
|
mobile-test-strategy,Mobile Test Strategy,"Mobile test level framework (unit/component/contract/device flow), what belongs in a device flow, mobile risk categories (permissions, lifecycle, connectivity, fragmentation, upgrade), device matrix with a gate profile matching local, CI shape, no live third-party flag evaluation in the run path","mobile,maestro,strategy,risk,levels,ci,ios,android",specialized,knowledge/mobile-test-strategy.md
|
|
59
|
-
evidence-integrity,Evidence Integrity,"Falsifiability of checks (hollow green, optional assertions, continue-on-error, partial manifests), three-state diagnostics (pass/fail/could-not-measure), probes that issue the client's own request, verifying the outcome rather than the act, verifying framework properties before use, proving claims from the side that can observe them, environment asymmetry including screen geometry and accumulated credentials, ranking hypotheses by the cost of the measurement that kills them, recording what a change did rather than what it was for","quality,evidence,diagnostics,review,ci,gates,falsifiability",core,knowledge/evidence-integrity.md
|
|
60
|
-
mobile-ci-device-lab,Mobile CI Device Lab,"Build artifact selection (release build vs development shell vs debug-variant development build), native modules that degrade silently in a shell, development-server manifest signing in non-interactive CI, one device profile across local and CI, Android emulator snapshot caching and config.ini pitfalls, repairing locally created AVDs, per-device identity for sharded runs, runner version pinning, artifact layout and failure diagnosis, sharding measured on wall clock","mobile,maestro,ci,android,ios,expo,emulator,artifacts",specialized,knowledge/mobile-ci-device-lab.md
|
|
59
|
+
evidence-integrity,Evidence Integrity,"Falsifiability of checks (hollow green, optional assertions, continue-on-error, partial manifests, assertions already true before the action), three-state diagnostics (pass/fail/could-not-measure), probes that issue the client's own request, verifying the outcome rather than the act, verifying framework properties before use, proving claims from the side that can observe them, environment asymmetry including screen geometry and accumulated credentials, ranking hypotheses by the cost of the measurement that kills them, recording what a change did rather than what it was for","quality,evidence,diagnostics,review,ci,gates,falsifiability",core,knowledge/evidence-integrity.md
|
|
60
|
+
mobile-ci-device-lab,Mobile CI Device Lab,"Build artifact selection (release build vs development shell vs debug-variant development build), native modules that degrade silently in a shell, deep links reachable through the shell routed URL form, development-server manifest signing in non-interactive CI, one device profile across local and CI, Android emulator snapshot caching and config.ini pitfalls, repairing locally created AVDs, per-device identity for sharded runs, runner version pinning, artifact layout and failure diagnosis, sharding measured on wall clock","mobile,maestro,ci,android,ios,expo,emulator,artifacts",specialized,knowledge/mobile-ci-device-lab.md
|
|
@@ -21,7 +21,7 @@ A suite lies in two ways. A test that **cannot fail** reports coverage it does n
|
|
|
21
21
|
|
|
22
22
|
### Example 1: The Check That Cannot Fail
|
|
23
23
|
|
|
24
|
-
**Context**:
|
|
24
|
+
**Context**: Five shapes found live in one suite that reported success on every run.
|
|
25
25
|
|
|
26
26
|
**Implementation**:
|
|
27
27
|
|
|
@@ -81,10 +81,20 @@ flows:
|
|
|
81
81
|
|
|
82
82
|
**Shape 4** has no snippet, because the step looks correct: an assertion passes on iOS because the element is still in the hierarchy behind a presented modal, and fails on Android where the modal replaces the hierarchy. Same assertion, different meaning per platform. Any assertion whose truth depends on how a platform composes its view tree needs its own per-platform expectation, not one shared line.
|
|
83
83
|
|
|
84
|
+
**Shape 5: the assertion is about something that was already true before the action.** A flow opened a deep link and then asserted that a container belonging to the screen it was already on was visible. The container predated the link, so the check held whether or not the link did anything, and on one platform it did nothing. The suite reported all flows green with this one included, and the green was stable rather than intermittent.
|
|
85
|
+
|
|
86
|
+
Two tells for it, both cheap:
|
|
87
|
+
|
|
88
|
+
- **The name promises an effect the assertions never mention.** "Widget Deep Link Hydration" asserted the presence of a container, not that anything had hydrated. Read the flow's name as a claim and check that some assertion carries it. A name is the only place many suites record what a test was for, which makes disagreement between name and assertion a reliable smell.
|
|
89
|
+
- **The result differs across environments for reasons unrelated to what it asserts.** The identical vacuous flow was green in CI and red locally, because on one API level the unresolvable link errored and on another it resolved somewhere and the open step completed. When a flow's outcome tracks an environment difference that its assertions never mention, suspect that the assertions are not what is deciding the result.
|
|
90
|
+
|
|
91
|
+
The fix generalizes past this shape: **assert the transition rather than the state.** Where only a single state is available, choose an input whose expected value differs from the application's default, so agreement with the default cannot carry the pass. Asserting "the morning option is selected" after an action that selects morning proves nothing in an app that starts on morning; asserting that a second action **moved** the selection, and that the first option is no longer selected, cannot pass without the action working.
|
|
92
|
+
|
|
84
93
|
**Key points**:
|
|
85
94
|
|
|
86
95
|
- Name the input that would turn each check red. If none exists, the check is decoration.
|
|
87
|
-
- `optional: true`, `continue-on-error`, a partial manifest, and a soft assertion are
|
|
96
|
+
- `optional: true`, `continue-on-error`, a partial manifest, and a soft assertion are four common ways a result stops being falsifiable.
|
|
97
|
+
- A fifth, and the hardest to see in review: the assertion is true before the action runs. Nothing about the step looks wrong, and the green is stable.
|
|
88
98
|
- **When you make a hollow check falsifiable and it goes red, the red is the finding.** It is a defect that was always there and is now visible. Reporting it as a regression you introduced is the wrong read and usually gets the fix reverted.
|
|
89
99
|
|
|
90
100
|
### Example 2: Diagnostics Need a Could-Not-Measure State
|
|
@@ -202,27 +212,34 @@ The corollary for a suite: that cheap discriminating measurement should already
|
|
|
202
212
|
|
|
203
213
|
## Anti-Patterns
|
|
204
214
|
|
|
205
|
-
| Anti-pattern
|
|
206
|
-
|
|
|
207
|
-
| Assertion with a soft or optional modifier as default
|
|
208
|
-
|
|
|
209
|
-
|
|
|
210
|
-
|
|
|
211
|
-
|
|
|
212
|
-
|
|
|
213
|
-
|
|
|
214
|
-
|
|
|
215
|
-
|
|
|
216
|
-
|
|
|
217
|
-
|
|
|
218
|
-
|
|
|
219
|
-
|
|
|
220
|
-
|
|
|
215
|
+
| Anti-pattern | Why it fails | Fix |
|
|
216
|
+
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
|
217
|
+
| Assertion with a soft or optional modifier as default | Cannot go red; reports coverage that does not exist | Reserve softness for genuinely optional UI, and assert the outcome hard |
|
|
218
|
+
| Assertion that was already true before the action | Cannot distinguish the action working from the action doing nothing | Assert the transition, or an input whose expected state differs from the default |
|
|
219
|
+
| Flow name promising an effect no assertion mentions | The name records the intent and nothing checks it | Read the name as a claim; make some assertion carry it |
|
|
220
|
+
| Outcome tracking an environment difference the assertions never mention | Something other than the assertions is deciding the result | Suspect a vacuous check before investigating the environment |
|
|
221
|
+
| A surface declared untestable without a test of the claim | Coverage is dropped on an assumption, and the assumption is often wrong | Try it; then name precisely which part is out of reach and why |
|
|
222
|
+
| `continue-on-error` on the test step | The suite cannot fail the build | Put it on artifact collection only; use `if: always()` for uploads |
|
|
223
|
+
| Runner manifest listing a subset of the suite | Files silently never run; the count is the only clue | Include by pattern; assert the executed count against the file count |
|
|
224
|
+
| Missing tool reported as a failed condition | Sends the investigation at the wrong subsystem | Three-state probes; distinct exit code for could-not-measure |
|
|
225
|
+
| Probe observing a proxy that correlates with the target | Passes for a reason unrelated to the claim, and a green is never re-examined | Ask what else could make this pass; observe the thing itself |
|
|
226
|
+
| Probe sending a different request than the client | Takes a different branch through the server: healthy probe, failing app, both correct | Copy the client's method, headers, and body into the probe, and log what was sent |
|
|
227
|
+
| Written setting read back and reported as effect | Proves the write; some configuration latches at boot and never applies live | Assert the observable the claim is about, not the act that was supposed to produce it |
|
|
228
|
+
| Change described by its intent once its effect is known | Reads as a found root cause and stops the next investigation looking | Record what it actually did, and why it was kept |
|
|
229
|
+
| Mechanisms elaborated while the cheap measurement goes untaken | Plausibility feels like progress; several explanations, no new evidence | Rank hypotheses by the cost of the observation that would kill them |
|
|
230
|
+
| Verdict emitted by the side that cannot observe it | Proves the wrong namespace | Move the assertion to the party whose route or state is in question |
|
|
231
|
+
| Comment asserting a mechanism with no source read | Propagates into other files and into other people's reasoning | Cite the doc or source line, or omit the mechanism |
|
|
232
|
+
| Local result used as a CI argument, asymmetry unstated | Hides the axes that actually differ | Tabulate the differing axes with the claim |
|
|
233
|
+
| Environment probe running after its consumers | The answer arrives too late to configure anything | Resolve environment-dependent values first, then derive |
|
|
234
|
+
| Reverting a newly-red check as a regression | Restores the hollow green and loses the finding | Treat the red as the pre-existing defect it exposed |
|
|
221
235
|
|
|
222
236
|
## Evidence Integrity Checklist
|
|
223
237
|
|
|
224
238
|
- [ ] **Every check is falsifiable**: for each assertion, the input that turns it red is nameable
|
|
225
239
|
- [ ] **No soft assertion by default**: optional modifiers only on genuinely optional UI, with a comment
|
|
240
|
+
- [ ] **The assertion carrying the outcome post-dates its action**: a precondition assertion is allowed when it is labelled as one, and nothing already true before the step is presented as proof of it
|
|
241
|
+
- [ ] **Names reconciled with assertions**: what a test is called is carried by something that can fail
|
|
242
|
+
- [ ] **Untestable claims tested**: a surface is dropped from coverage only after the claim itself has been checked
|
|
226
243
|
- [ ] **No `continue-on-error` on a test step**: only on artifact collection
|
|
227
244
|
- [ ] **Executed count reconciled**: the number of tests that ran matches the number of test files discovered
|
|
228
245
|
- [ ] **Platform-divergent assertions split**: no single assertion whose meaning depends on how a platform composes its view tree
|