@pzy560117/opentest 0.1.10 → 0.1.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/assets/manifest.json +15 -1
- package/assets/skills/opentest/references/android-app-testing.md +78 -0
- package/assets/skills/opentest/references/api-testing.md +77 -0
- package/assets/skills/opentest/references/codex-harness-coverage-heuristics.md +18 -1
- package/assets/skills/opentest/references/complete-testing-workflow.md +23 -0
- package/assets/skills/opentest/references/desktop-gui-testing.md +52 -0
- package/assets/skills/opentest/references/matrix-format.md +8 -2
- package/assets/skills/opentest/references/test-asset-layout.md +64 -0
- package/assets/skills/opentest/references/test-surfaces.md +101 -0
- package/assets/skills/opentest/references/web-browser-testing.md +40 -0
- package/assets/skills/opentest/templates/acceptance-template.md +3 -1
- package/assets/skills/opentest/templates/android-app-acceptance-template.md +46 -0
- package/assets/skills/opentest/templates/api-acceptance-template.md +44 -0
- package/assets/skills/opentest/templates/desktop-gui-acceptance-template.md +43 -0
- package/assets/skills/opentest/templates/matrix-template.md +12 -11
- package/assets/skills/opentest/templates/plan-template.md +2 -2
- package/assets/skills/opentest/templates/web-acceptance-template.md +27 -0
- package/assets/skills/opentest-accept/SKILL.md +18 -7
- package/assets/skills/opentest-android-app/SKILL.md +25 -0
- package/assets/skills/opentest-api/SKILL.md +25 -0
- package/assets/skills/opentest-author/SKILL.md +2 -1
- package/assets/skills/opentest-desktop-gui/SKILL.md +24 -0
- package/assets/skills/opentest-plan/SKILL.md +16 -9
- package/assets/skills/opentest-run/SKILL.md +15 -8
- package/assets/skills/opentest-web-browser/SKILL.md +26 -0
- package/assets/skills-zh/opentest/references/android-app-testing.md +78 -0
- package/assets/skills-zh/opentest/references/api-testing.md +77 -0
- package/assets/skills-zh/opentest/references/codex-harness-coverage-heuristics.md +18 -1
- package/assets/skills-zh/opentest/references/complete-testing-workflow.md +23 -0
- package/assets/skills-zh/opentest/references/desktop-gui-testing.md +52 -0
- package/assets/skills-zh/opentest/references/matrix-format.md +8 -2
- package/assets/skills-zh/opentest/references/test-asset-layout.md +64 -0
- package/assets/skills-zh/opentest/references/test-surfaces.md +101 -0
- package/assets/skills-zh/opentest/references/web-browser-testing.md +40 -0
- package/assets/skills-zh/opentest/templates/acceptance-template.md +3 -1
- package/assets/skills-zh/opentest/templates/android-app-acceptance-template.md +46 -0
- package/assets/skills-zh/opentest/templates/api-acceptance-template.md +44 -0
- package/assets/skills-zh/opentest/templates/desktop-gui-acceptance-template.md +43 -0
- package/assets/skills-zh/opentest/templates/matrix-template.md +12 -11
- package/assets/skills-zh/opentest/templates/plan-template.md +2 -2
- package/assets/skills-zh/opentest/templates/web-acceptance-template.md +27 -0
- package/assets/skills-zh/opentest-accept/SKILL.md +16 -5
- package/assets/skills-zh/opentest-android-app/SKILL.md +25 -0
- package/assets/skills-zh/opentest-api/SKILL.md +25 -0
- package/assets/skills-zh/opentest-author/SKILL.md +2 -1
- package/assets/skills-zh/opentest-desktop-gui/SKILL.md +24 -0
- package/assets/skills-zh/opentest-plan/SKILL.md +14 -7
- package/assets/skills-zh/opentest-run/SKILL.md +14 -7
- package/assets/skills-zh/opentest-web-browser/SKILL.md +26 -0
- package/package.json +1 -1
- package/scripts/smoke-test.js +321 -0
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# API Acceptance
|
|
2
|
+
|
|
3
|
+
## ACC-API-001
|
|
4
|
+
|
|
5
|
+
- execution surface: api
|
|
6
|
+
- acceptance mode: n/a
|
|
7
|
+
- tool route: project API command | pytest + httpx/requests | curl/httpie | Postman/Newman | contract tool
|
|
8
|
+
- evidence layer: contract | integration | smoke | security-review
|
|
9
|
+
- base URL:
|
|
10
|
+
- auth/role:
|
|
11
|
+
- fixture/seed:
|
|
12
|
+
- teardown:
|
|
13
|
+
- status: pending
|
|
14
|
+
|
|
15
|
+
### Request
|
|
16
|
+
|
|
17
|
+
- method:
|
|
18
|
+
- path:
|
|
19
|
+
- headers:
|
|
20
|
+
- query:
|
|
21
|
+
- body:
|
|
22
|
+
|
|
23
|
+
### Expected Response
|
|
24
|
+
|
|
25
|
+
- status code:
|
|
26
|
+
- headers:
|
|
27
|
+
- schema/source:
|
|
28
|
+
- payload assertions:
|
|
29
|
+
- error contract:
|
|
30
|
+
|
|
31
|
+
### Read-Back Contract
|
|
32
|
+
|
|
33
|
+
- API read endpoint:
|
|
34
|
+
- DB/storage/log/event assertion:
|
|
35
|
+
- idempotency/retry assertion:
|
|
36
|
+
- cleanup assertion:
|
|
37
|
+
|
|
38
|
+
### Evidence
|
|
39
|
+
|
|
40
|
+
- status:
|
|
41
|
+
- request/response record:
|
|
42
|
+
- report path:
|
|
43
|
+
- artifacts:
|
|
44
|
+
- blockers:
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Desktop GUI Acceptance
|
|
2
|
+
|
|
3
|
+
## ACC-Desktop-001
|
|
4
|
+
|
|
5
|
+
- execution surface: desktop-gui
|
|
6
|
+
- acceptance mode: n/a
|
|
7
|
+
- tool route: project GUI automation | @midscene/computer | accessibility/window metadata | scripted manual GUI acceptance
|
|
8
|
+
- evidence layer: gui-acceptance | visual-acceptance | integration | smoke
|
|
9
|
+
- target app/window:
|
|
10
|
+
- launch command:
|
|
11
|
+
- fixture/reset:
|
|
12
|
+
- status: pending
|
|
13
|
+
|
|
14
|
+
### Environment
|
|
15
|
+
|
|
16
|
+
- OS/display/RDP:
|
|
17
|
+
- model env status:
|
|
18
|
+
- app version/build:
|
|
19
|
+
- target process/window metadata:
|
|
20
|
+
|
|
21
|
+
### Steps
|
|
22
|
+
|
|
23
|
+
1.
|
|
24
|
+
|
|
25
|
+
### Expected Outcome
|
|
26
|
+
|
|
27
|
+
-
|
|
28
|
+
|
|
29
|
+
### Read-Back Contract
|
|
30
|
+
|
|
31
|
+
- persisted result surface:
|
|
32
|
+
- reopen/restart check:
|
|
33
|
+
- accessibility/window metadata assertion:
|
|
34
|
+
- file/config/app-state assertion:
|
|
35
|
+
|
|
36
|
+
### Evidence
|
|
37
|
+
|
|
38
|
+
- status:
|
|
39
|
+
- screenshots/recording:
|
|
40
|
+
- GUI action log:
|
|
41
|
+
- window/app metadata:
|
|
42
|
+
- Midscene/computer report or log:
|
|
43
|
+
- blockers:
|
|
@@ -1,13 +1,14 @@
|
|
|
1
1
|
# Acceptance-to-Test Matrix
|
|
2
2
|
|
|
3
|
-
| ID | Requirement source | Intent | Coverage dimension | Trigger/Input | Expected behavior | Risk | Evidence layer | Framework/command | Required evidence | Gap/blocker | Status |
|
|
4
|
-
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
|
5
|
-
| ACC-001 | REQ-001 / user story | create succeeds | create | valid fixture entity | entity is created and visible in list/detail | high |
|
|
6
|
-
| ACC-002 | REQ-001 /
|
|
7
|
-
| ACC-003 | REQ-
|
|
8
|
-
| ACC-004 | REQ-
|
|
9
|
-
| ACC-005 |
|
|
10
|
-
| ACC-006 |
|
|
11
|
-
| ACC-007 |
|
|
12
|
-
| ACC-
|
|
13
|
-
| ACC-
|
|
3
|
+
| ID | Requirement source | Intent | Execution surface | Acceptance mode | Coverage dimension | Trigger/Input | Expected behavior | Risk | Evidence layer | Framework/command | Required evidence | Gap/blocker | Status |
|
|
4
|
+
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
|
5
|
+
| ACC-001 | REQ-001 / user story | create succeeds in web UI | web-browser | instant-acceptance | create | valid fixture entity | entity is created and visible in list/detail | high | browser-acceptance + integration | Playwright MCP / Playwright CLI + project command | create evidence, UI/API/DB assertion, fixture used | none | pending |
|
|
6
|
+
| ACC-002 | REQ-WEB-REG-001 / regression need | web CRUD has durable regression | web-browser | durable-regression | read/list/detail + end-to-end CRUD | committed E2E test | create -> list -> detail -> update -> read back -> delete is repeatable | high | e2e | `npx playwright test` or project E2E command | test file, trace/report, stable locator assertions | none | pending |
|
|
7
|
+
| ACC-003 | REQ-WEB-VIS-001 / visual risk | visual web state is verified | web-browser | visual-ai-assist | visual behavior | weak selector or canvas UI | Midscene helps locate UI and deterministic read-back proves result | medium | visual-acceptance | Midscene + Playwright route | Midscene report, screenshot, read/assert result | none | pending |
|
|
8
|
+
| ACC-004 | REQ-ANDROID-001 / mobile flow | create succeeds in Android app | android-app | n/a | create | APK installed on emulator/device | task is created and visible after app refresh | high | visual-acceptance + e2e | `opentest-android-app` / `android-midscene-pytest` / `python -m pytest tests_py -v` | pytest report, ADB smoke, Midscene HTML report, screenshot, logcat, device/app metadata, `midscene_run` logs | missing ADB/APK/device/package/model env/result surface if unavailable | pending |
|
|
9
|
+
| ACC-005 | REQ-DESKTOP-001 / settings workflow | desktop setting saves | desktop-gui | n/a | update | desktop app window is open | setting persists after save and reopen | high | gui-acceptance + integration | `opentest-desktop-gui` / project GUI automation / `@midscene/computer` | screenshot or recording, GUI action log, app/window metadata, deterministic read-back, Midscene/computer report when used | missing display/RDP/app launch/window identity/model env/result surface if unavailable | pending |
|
|
10
|
+
| ACC-006 | REQ-API-001 / contract | API create succeeds | api | n/a | create | valid request payload | response status and payload confirm created entity | high | contract + integration | `opentest-api` / project API command / `python -m pytest tests/api -v` | request/response record, status code, payload/schema assertion, read-after-write, data consistency, cleanup/teardown | missing base URL/auth/fixture/seed/teardown/dependency/schema/result surface if unavailable | pending |
|
|
11
|
+
| ACC-007 | Risk: invalid and unauthorized input | failure and boundary paths are handled | api | n/a | failure/boundary | invalid, empty, duplicate, unauthorized, expired, forbidden, or stale fixture data | clear error contract without corrupting data | high | contract + security-review | `opentest-api` / project command / `python -m pytest tests/api -v` | validation, auth/permission, duplicate/idempotency, stale-state, error schema, sensitive-field evidence | none | pending |
|
|
12
|
+
| ACC-007A | Risk: list contract drift | API list/search behavior is stable | api | n/a | list/filter/sort/pagination | seeded collection and query params | pagination, filtering, sorting, and empty result follow contract | medium | contract + integration | `opentest-api` / project API command | request/response record, schema assertion, pagination metadata, fixture isolation | not applicable if no list endpoint changed | pending |
|
|
13
|
+
| ACC-008 | Quality gate requirement | smoke gate passes | web-browser | instant-acceptance | smoke | app starts and core entry points open | core route/API/CRUD happy path does not crash | high | smoke | project smoke command or targeted workflow | smoke_report path | none | pending |
|
|
14
|
+
| ACC-009 | Delivery gate requirement | pre-push gate passes | api | n/a | pre-push | staged change before push | format/lint/type/unit/integration/smoke/diff checks pass or block push | high | pre-push | project command sequence | pre_push_report path | none | pending |
|
|
@@ -24,5 +24,5 @@
|
|
|
24
24
|
|
|
25
25
|
## Evidence Plan
|
|
26
26
|
|
|
27
|
-
| Evidence layer | Applicable scenario | Command
|
|
28
|
-
| --- | --- | --- | --- | --- |
|
|
27
|
+
| Execution surface | Acceptance mode | Evidence layer | Applicable scenario | Command/tool | Artifact path | Status |
|
|
28
|
+
| --- | --- | --- | --- | --- | --- | --- |
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Web Browser Acceptance
|
|
2
|
+
|
|
3
|
+
- ACC ID:
|
|
4
|
+
- execution surface: web-browser
|
|
5
|
+
- acceptance mode: instant-acceptance | durable-regression | visual-ai-assist
|
|
6
|
+
- tool route: Playwright MCP | Playwright CLI | @playwright/test | Midscene
|
|
7
|
+
- page/route:
|
|
8
|
+
- actor/role:
|
|
9
|
+
- fixture:
|
|
10
|
+
|
|
11
|
+
## Steps
|
|
12
|
+
|
|
13
|
+
1. open:
|
|
14
|
+
2. snapshot:
|
|
15
|
+
3. fill/input:
|
|
16
|
+
4. submit/confirm:
|
|
17
|
+
5. snapshot after submit:
|
|
18
|
+
6. read/assert changed result:
|
|
19
|
+
7. screenshot/report:
|
|
20
|
+
|
|
21
|
+
## Evidence
|
|
22
|
+
|
|
23
|
+
- status:
|
|
24
|
+
- changed result asserted:
|
|
25
|
+
- artifact paths:
|
|
26
|
+
- console/network notes:
|
|
27
|
+
- blocked reason:
|
|
@@ -1,23 +1,34 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: opentest-accept
|
|
3
|
-
description: "OpenTest phase 4: execute
|
|
3
|
+
description: "OpenTest phase 4: execute acceptance and write back evidence."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# OpenTest Accept
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
Write PASS, FAIL, or blocked evidence to cases and matrix.
|
|
9
9
|
|
|
10
10
|
## Required references
|
|
11
11
|
|
|
12
12
|
- `opentest/references/acceptance-evidence.md`
|
|
13
13
|
- `opentest/references/complete-testing-workflow.md`
|
|
14
|
+
- `opentest/references/test-surfaces.md`
|
|
15
|
+
- `opentest/references/android-app-testing.md`
|
|
16
|
+
- `opentest/references/web-browser-testing.md`
|
|
17
|
+
- `opentest/references/desktop-gui-testing.md`
|
|
18
|
+
- `opentest/references/api-testing.md`
|
|
14
19
|
- `opentest/templates/acceptance-template.md`
|
|
20
|
+
- `opentest/templates/android-app-acceptance-template.md`
|
|
21
|
+
- `opentest/templates/desktop-gui-acceptance-template.md`
|
|
22
|
+
- `opentest/templates/api-acceptance-template.md`
|
|
15
23
|
|
|
16
24
|
## Steps
|
|
17
25
|
|
|
18
26
|
1. Read the matrix, fixtures, and `docs/opentest/acceptance/`.
|
|
19
|
-
2.
|
|
20
|
-
3. For
|
|
21
|
-
4. For
|
|
22
|
-
5.
|
|
23
|
-
6.
|
|
27
|
+
2. Select the acceptance tool from the matrix execution surface.
|
|
28
|
+
3. For `web-browser`, use `opentest-web-browser`: Playwright MCP/CLI, `@playwright/test` for durable regression, Midscene only for visual assist.
|
|
29
|
+
4. For `android-app`, use `opentest-android-app` and `android-midscene-pytest`: `python -m pytest tests_py -v`, ADB smoke, Midscene HTML, logcat, and `midscene_run`; block missing prerequisites.
|
|
30
|
+
5. For `desktop-gui`, use `opentest-desktop-gui`: project GUI automation or `@midscene/computer`, screenshot/recording, metadata, and read-back.
|
|
31
|
+
6. For `api`, use `opentest-api`: project API command or `pytest` with `httpx`/`requests`, schema checks, fixtures, read-after-write, and cleanup/teardown.
|
|
32
|
+
7. For CRUD/data changes, execute the full chain from the workflow reference.
|
|
33
|
+
8. Record feedback location/shape, artifacts, blocked evidence, and matching ACC IDs.
|
|
34
|
+
9. Update acceptance records and run `bash "$OPENTEST_GUARD" accept --apply`.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: opentest-android-app
|
|
3
|
+
description: "OpenTest Android app execution surface adapter. Use when planning, authoring, running, or accepting Android APK/app GUI workflows through android-midscene-pytest, ADB, emulator/device setup, screenshots, logcat, and Midscene reports."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# OpenTest Android App
|
|
7
|
+
|
|
8
|
+
Use this adapter for `android-app` matrix rows.
|
|
9
|
+
|
|
10
|
+
## Required references
|
|
11
|
+
|
|
12
|
+
- `opentest/references/android-app-testing.md`
|
|
13
|
+
- `opentest/templates/android-app-acceptance-template.md`
|
|
14
|
+
|
|
15
|
+
## Route
|
|
16
|
+
|
|
17
|
+
1. Use `android-midscene-pytest` as the detailed Android execution skill when available.
|
|
18
|
+
2. Keep durable Android assets in the fixed layout from `test-asset-layout.md`: `tests/android/tests_py/`, `tests/android/midscene/`, and `scripts/opentest-run-android.ps1` or an existing Android harness command.
|
|
19
|
+
3. User-facing execution should enter through pytest, normally `python -m pytest tests_py -v` in an existing harness or the repository's Android test entry.
|
|
20
|
+
4. Run `npm run test:android` only when model environment variables are ready or when debugging the Midscene layer.
|
|
21
|
+
5. Mark blocked when ADB, emulator/device, APK path, package name, model credentials, fixture data, or stable result surface is missing.
|
|
22
|
+
|
|
23
|
+
## Evidence Contract
|
|
24
|
+
|
|
25
|
+
PASS requires pytest result, ADB smoke, Midscene HTML report when visual automation runs, screenshots, logcat, device/app metadata, and any available `midscene_run` logs. Static screenshots alone are not Android app acceptance.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: opentest-api
|
|
3
|
+
description: "OpenTest API execution surface adapter. Use when planning, authoring, running, or accepting HTTP API, RPC, backend workflow, contract, auth, data consistency, idempotency, and integration evidence."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# OpenTest API
|
|
7
|
+
|
|
8
|
+
Use this adapter for `api` matrix rows.
|
|
9
|
+
|
|
10
|
+
## Required references
|
|
11
|
+
|
|
12
|
+
- `opentest/references/api-testing.md`
|
|
13
|
+
- `opentest/templates/api-acceptance-template.md`
|
|
14
|
+
|
|
15
|
+
## Route
|
|
16
|
+
|
|
17
|
+
1. Prefer the repository's existing API/integration test command when it is explicit and repeatable.
|
|
18
|
+
2. If no project command exists, default to `pytest` with `httpx` or `requests`, schema checks via `jsonschema` or existing Pydantic/DTO models, and fixtures for seed/teardown.
|
|
19
|
+
3. Use OpenAPI, protobuf, or existing contract docs as the contract source when present; otherwise write the expected status, headers, response schema, and business payload in the acceptance case.
|
|
20
|
+
4. Mock or stub third-party APIs unless the requirement explicitly needs a live external dependency.
|
|
21
|
+
5. Mark blocked when base URL, auth token, fixture data, seed/teardown, dependency service, or stable read-back surface is missing.
|
|
22
|
+
|
|
23
|
+
## Evidence Contract
|
|
24
|
+
|
|
25
|
+
PASS requires request/response records, status code, response payload or schema assertion, auth/permission result when applicable, data consistency/read-after-write evidence, and cleanup or teardown proof. API smoke only proves liveness; it does not replace contract, boundary, permission, or data consistency evidence for high-risk changes.
|
|
@@ -11,6 +11,7 @@ Turn the matrix into executable tests, fixtures, seed/teardown notes, and accept
|
|
|
11
11
|
|
|
12
12
|
- `opentest/references/opentest-driven-development.md`
|
|
13
13
|
- `opentest/references/complete-testing-workflow.md`
|
|
14
|
+
- `opentest/references/test-asset-layout.md`
|
|
14
15
|
- `opentest/templates/fixtures-template.md`
|
|
15
16
|
- `opentest/templates/acceptance-template.md`
|
|
16
17
|
|
|
@@ -18,7 +19,7 @@ Turn the matrix into executable tests, fixtures, seed/teardown notes, and accept
|
|
|
18
19
|
|
|
19
20
|
1. Read `matrix` and `fixtures` from `.opentest.yaml`.
|
|
20
21
|
2. Preserve each row's requirement source and expected behavior. Do not rewrite acceptance cases around current implementation names, component internals, or existing test files.
|
|
21
|
-
3.
|
|
22
|
+
3. Place assets in the fixed layout from `test-asset-layout.md`; default to pytest under `tests/` when no project framework exists. Missing implementation means evidence stays pending.
|
|
22
23
|
4. Create/update fixtures, seed, teardown, users, roles, entities, files/images, and assertion surfaces.
|
|
23
24
|
5. For CRUD/data changes, author the full acceptance flow: create -> list -> detail -> update -> read back -> delete -> confirm absence -> teardown.
|
|
24
25
|
6. Record any gap/blocker with reason and risk.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: opentest-desktop-gui
|
|
3
|
+
description: "OpenTest desktop GUI execution surface adapter. Use when planning, authoring, running, or accepting native desktop, Electron, Tauri, Windows, macOS, Linux, or RDP workflows and deciding between project GUI automation, @midscene/computer, accessibility metadata, screenshots, and scripted manual acceptance."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# OpenTest Desktop GUI
|
|
7
|
+
|
|
8
|
+
Use this adapter for `desktop-gui` matrix rows.
|
|
9
|
+
|
|
10
|
+
## Required references
|
|
11
|
+
|
|
12
|
+
- `opentest/references/desktop-gui-testing.md`
|
|
13
|
+
- `opentest/templates/desktop-gui-acceptance-template.md`
|
|
14
|
+
|
|
15
|
+
## Route
|
|
16
|
+
|
|
17
|
+
1. Prefer the repository's existing GUI automation command when it is explicit and repeatable.
|
|
18
|
+
2. Use `@midscene/computer` for visual desktop automation, weak selectors, native controls, cross-window flows, or remote Windows RDP when deterministic project automation is unavailable.
|
|
19
|
+
3. For Electron or Tauri apps that expose a browser context, prefer `web-browser` or Playwright routes for DOM-verifiable flows, and use `desktop-gui` only for native shell, tray, file picker, menu, OS dialog, or multi-window behavior.
|
|
20
|
+
4. If model credentials, desktop access, display/RDP availability, app launch command, or stable result surface is missing, record `blocked` instead of PASS.
|
|
21
|
+
|
|
22
|
+
## Evidence Contract
|
|
23
|
+
|
|
24
|
+
PASS requires the executed steps, screenshot or screen recording, window/app metadata, GUI action log, and a deterministic read-back result after writes. `@midscene/computer` assertions are visual assistance; they must not replace persisted-result checks such as reopening a settings screen, reading a file/config value, querying an app state surface, or confirming the expected window state after restart.
|
|
@@ -1,27 +1,34 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: opentest-plan
|
|
3
|
-
description: "OpenTest phase 1:
|
|
3
|
+
description: "OpenTest phase 1: create test strategy and acceptance matrix."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# OpenTest Plan
|
|
7
7
|
|
|
8
|
-
Create
|
|
8
|
+
Create plan, matrix, and fixtures before implementation.
|
|
9
9
|
|
|
10
10
|
## Required references
|
|
11
11
|
|
|
12
12
|
- `opentest/references/codex-harness-coverage-heuristics.md`
|
|
13
13
|
- `opentest/references/matrix-format.md`
|
|
14
14
|
- `opentest/references/complete-testing-workflow.md`
|
|
15
|
+
- `opentest/references/test-asset-layout.md`
|
|
16
|
+
- `opentest/references/test-surfaces.md`
|
|
17
|
+
- `opentest/references/android-app-testing.md`
|
|
18
|
+
- `opentest/references/web-browser-testing.md`
|
|
19
|
+
- `opentest/references/desktop-gui-testing.md`
|
|
20
|
+
- `opentest/references/api-testing.md`
|
|
15
21
|
|
|
16
22
|
## Steps
|
|
17
23
|
|
|
18
|
-
1. Read
|
|
19
|
-
2. Treat requirements
|
|
20
|
-
3. Apply
|
|
21
|
-
4.
|
|
22
|
-
5.
|
|
23
|
-
6.
|
|
24
|
+
1. Read rules, requirements/diff, commands, and detection output.
|
|
25
|
+
2. Treat requirements and risks as sources; inspect current code only for project facts.
|
|
26
|
+
3. Apply CRUD baseline and test data rules for data-writing/API/form/file/stateful changes.
|
|
27
|
+
4. Classify execution surface and evidence layer separately: `web-browser`, `android-app`, `desktop-gui`, or `api`; use `opentest-android-app`, web modes, `opentest-desktop-gui`, and `opentest-api` where applicable.
|
|
28
|
+
5. Produce a requirement-first matrix with source, behavior, surface, mode, evidence, command, gap/blocker, status, and fixed layout.
|
|
29
|
+
6. Write `.opentest.yaml` fields: `plan`, `matrix`, `fixtures`.
|
|
30
|
+
7. Update handoff if present, then run `bash "$OPENTEST_GUARD" plan --apply`.
|
|
24
31
|
|
|
25
32
|
## Gate
|
|
26
33
|
|
|
27
|
-
Every
|
|
34
|
+
Every behavior, failure path, boundary, and risk needs evidence or gap/blocker. Every row cites a source and includes surface plus evidence layer; web rows include acceptance mode. Do not use unit/component/integration/contract/smoke as the execution surface. Do not drop or narrow acceptance because current code has no matching file.
|
|
@@ -5,12 +5,18 @@ description: "OpenTest phase 3: run project verification commands in targeted, f
|
|
|
5
5
|
|
|
6
6
|
# OpenTest Run
|
|
7
7
|
|
|
8
|
-
Run matrix-driven commands and write
|
|
8
|
+
Run matrix-driven commands and write reports under `docs/opentest/runs/`.
|
|
9
9
|
|
|
10
10
|
## Required references
|
|
11
11
|
|
|
12
12
|
- `opentest/references/command-routing.md`
|
|
13
13
|
- `opentest/references/complete-testing-workflow.md`
|
|
14
|
+
- `opentest/references/test-asset-layout.md`
|
|
15
|
+
- `opentest/references/test-surfaces.md`
|
|
16
|
+
- `opentest/references/android-app-testing.md`
|
|
17
|
+
- `opentest/references/web-browser-testing.md`
|
|
18
|
+
- `opentest/references/desktop-gui-testing.md`
|
|
19
|
+
- `opentest/references/api-testing.md`
|
|
14
20
|
|
|
15
21
|
## Modes
|
|
16
22
|
|
|
@@ -22,10 +28,11 @@ Run matrix-driven commands and write evidence reports under `docs/opentest/runs/
|
|
|
22
28
|
|
|
23
29
|
## Steps
|
|
24
30
|
|
|
25
|
-
1. Read `run_mode`, matrix, fixtures, and
|
|
26
|
-
2.
|
|
27
|
-
3.
|
|
28
|
-
4.
|
|
29
|
-
5.
|
|
30
|
-
6.
|
|
31
|
-
7.
|
|
31
|
+
1. Read `run_mode`, matrix, fixtures, required evidence, and fixed asset layout.
|
|
32
|
+
2. Choose by matrix execution surface: MCP/CLI or `npx playwright test` for `web-browser`; `opentest-android-app` or `python -m pytest tests_py -v` for `android-app`, with `npm run test:android` only when model env is ready or debugging Midscene; `opentest-desktop-gui` for `desktop-gui`; `opentest-api` or `python -m pytest tests/api -v` for `api`.
|
|
33
|
+
3. Prefer explicit project commands; otherwise use `python -m pytest` for code-level tests.
|
|
34
|
+
4. For coverage, prefer `python -m pytest --cov=. --cov-report=term-missing`.
|
|
35
|
+
5. Smoke evidence is required unless the matrix says not applicable.
|
|
36
|
+
6. For `pre-push`, run or record format/check, lint, type, unit, targeted integration, smoke, and `git diff --check`.
|
|
37
|
+
7. Write `run_report`, `coverage_report`, `smoke_report`, or `pre_push_report` when required.
|
|
38
|
+
8. Run `bash "$OPENTEST_GUARD" run --apply`.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: opentest-web-browser
|
|
3
|
+
description: "OpenTest web-browser execution surface adapter. Use when planning, authoring, running, or accepting browser-rendered web workflows and deciding between Playwright MCP, Playwright CLI, @playwright/test, and Midscene."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# OpenTest Web Browser
|
|
7
|
+
|
|
8
|
+
Use this adapter for `web-browser` matrix rows.
|
|
9
|
+
|
|
10
|
+
## Required references
|
|
11
|
+
|
|
12
|
+
- `opentest/references/web-browser-testing.md`
|
|
13
|
+
- `opentest/references/complete-testing-workflow.md`
|
|
14
|
+
- `opentest/templates/web-acceptance-template.md`
|
|
15
|
+
|
|
16
|
+
## Steps
|
|
17
|
+
|
|
18
|
+
1. Decide `acceptance_mode`: `instant-acceptance`, `durable-regression`, or `visual-ai-assist`.
|
|
19
|
+
2. For `instant-acceptance`, use Playwright MCP first; if browser MCP is unavailable or unstable, use Playwright CLI.
|
|
20
|
+
3. For `durable-regression`, write or run `@playwright/test` or the repository's existing E2E framework.
|
|
21
|
+
4. Use Midscene only for visual, weak-selector, canvas, cross-frame, or other UI surfaces selectors cannot prove well.
|
|
22
|
+
5. For writes, run the full chain: open -> snapshot -> fill/input -> submit -> snapshot -> read/assert changed result -> screenshot -> PASS/FAIL.
|
|
23
|
+
|
|
24
|
+
## Gate
|
|
25
|
+
|
|
26
|
+
MCP or Playwright CLI evidence proves this acceptance run only. Do not count it as durable regression unless the repository has a committed repeatable test, command, and report path. Midscene must not replace read-after-write assertions.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# Android App 测试
|
|
2
|
+
|
|
3
|
+
用于 `android-app` 执行面的矩阵行。
|
|
4
|
+
|
|
5
|
+
## 适配边界
|
|
6
|
+
|
|
7
|
+
`opentest-android-app` 是 OpenTest 的执行面适配器。它让 Android App 测试在安装后的 OpenTest skills 中可见,并在可用时把详细执行路由到 `android-midscene-pytest`。
|
|
8
|
+
|
|
9
|
+
这里不复制完整 Android Midscene SOP。本文只定义 OpenTest 矩阵、目录、证据和阻塞规则。
|
|
10
|
+
|
|
11
|
+
## 默认目录
|
|
12
|
+
|
|
13
|
+
稳定 Android 资产遵循 `opentest/references/test-asset-layout.md`:
|
|
14
|
+
|
|
15
|
+
```text
|
|
16
|
+
tests/android/
|
|
17
|
+
tests_py/
|
|
18
|
+
midscene/
|
|
19
|
+
|
|
20
|
+
scripts/
|
|
21
|
+
opentest-run-android.ps1
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
如果仓库已有 Android harness,优先使用最接近的现有路径,但逻辑槽位必须保持一致:pytest wrapper、Midscene 脚本、设备准备和运行入口。
|
|
25
|
+
|
|
26
|
+
标准 Midscene 槽位是 `tests/android/midscene/`。
|
|
27
|
+
|
|
28
|
+
## 默认路线
|
|
29
|
+
|
|
30
|
+
```text
|
|
31
|
+
opentest-android-app
|
|
32
|
+
-> android-midscene-pytest
|
|
33
|
+
-> pytest wrapper
|
|
34
|
+
-> npm/Vitest wrapper
|
|
35
|
+
-> @midscene/android
|
|
36
|
+
-> ADB + Android emulator/device
|
|
37
|
+
-> screenshots + logcat + Midscene HTML report
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
面向用户的入口是 pytest。优先使用现有 Android harness 中的 `python -m pytest tests_py -v`,或仓库采用固定目录时的 `scripts/opentest-run-android.ps1`。
|
|
41
|
+
|
|
42
|
+
只有 Midscene 模型环境变量齐全,或需要直接排查 Midscene 层时,才运行 `npm run test:android`。
|
|
43
|
+
|
|
44
|
+
## 必需证据
|
|
45
|
+
|
|
46
|
+
`android-app` 行需要:
|
|
47
|
+
|
|
48
|
+
- pytest 报告或命令输出
|
|
49
|
+
- ADB 冒烟结果
|
|
50
|
+
- APK 路径、package name、模拟器/真机标识
|
|
51
|
+
- 关键步骤后的截图
|
|
52
|
+
- logcat 或定向 Android 日志
|
|
53
|
+
- 执行视觉自动化时的 Midscene HTML 报告
|
|
54
|
+
- 可用的 `midscene_run/log/ai-call.log`、`agent.log`、`android-device.log` 和 `midscene_run/report/*.html`
|
|
55
|
+
- App 写数据时的写后读或持久化状态证明
|
|
56
|
+
|
|
57
|
+
## 阻塞规则
|
|
58
|
+
|
|
59
|
+
缺少任一必需前置条件时记录 `blocked`:
|
|
60
|
+
|
|
61
|
+
- ADB
|
|
62
|
+
- 模拟器或真机
|
|
63
|
+
- APK 路径
|
|
64
|
+
- package name 或 launch activity
|
|
65
|
+
- 需要视觉自动化时的 Midscene 模型凭据
|
|
66
|
+
- fixture 数据、reset 路径或稳定结果面
|
|
67
|
+
|
|
68
|
+
不得只凭静态截图把 Android App 验收标为 PASS。
|
|
69
|
+
|
|
70
|
+
## 矩阵要求
|
|
71
|
+
|
|
72
|
+
`android-app` 行必须包含:
|
|
73
|
+
|
|
74
|
+
- `执行面`:`android-app`
|
|
75
|
+
- `验收模式`:`n/a`
|
|
76
|
+
- `证据层级`:`visual-acceptance`、`e2e`、`integration` 或 `smoke`
|
|
77
|
+
- `框架/命令`:`opentest-android-app`、`android-midscene-pytest`、`python -m pytest tests_py -v` 或 `scripts/opentest-run-android.ps1`
|
|
78
|
+
- `必需证据`:pytest、ADB 冒烟、使用时的 Midscene HTML 报告、截图、logcat、设备/App 元数据、`midscene_run` 日志,以及不可用时的 blocked 前置条件
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# API 测试
|
|
2
|
+
|
|
3
|
+
用于 `api` 执行面的矩阵行。
|
|
4
|
+
|
|
5
|
+
## 默认架构
|
|
6
|
+
|
|
7
|
+
优先使用仓库已有 API 测试框架。项目没有明确 API 测试命令时,默认采用:
|
|
8
|
+
|
|
9
|
+
```text
|
|
10
|
+
pytest
|
|
11
|
+
-> httpx 或 requests client
|
|
12
|
+
-> pytest fixtures 管 seed/teardown
|
|
13
|
+
-> jsonschema 或项目已有 Pydantic/DTO 模型做契约断言
|
|
14
|
+
-> 可选 DB/存储/日志回读
|
|
15
|
+
-> pytest report 或 JUnit XML
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
依赖需要隔离时,用 Docker Compose、testcontainers 或仓库已有本地服务启动方式。第三方 API 默认 mock/stub;打 live 外部服务必须有明确需求和记录的风险说明。
|
|
19
|
+
|
|
20
|
+
稳定 API 资产放入 `opentest/references/test-asset-layout.md` 约定的 `tests/api/`:client 放 `tests/api/clients/`,fixtures 放 `tests/api/fixtures/`,schemas 放 `tests/api/schemas/`,可重复入口使用 `scripts/opentest-run-api.ps1` 或仓库等价命令。
|
|
21
|
+
|
|
22
|
+
## 证据层级
|
|
23
|
+
|
|
24
|
+
| 层级 | 证明内容 | 常见命令/工具 |
|
|
25
|
+
| --- | --- | --- |
|
|
26
|
+
| contract | 状态码、headers、响应字段、schema、错误形态 | 项目 contract tests、`pytest`、OpenAPI 检查 |
|
|
27
|
+
| integration | API handler、service、数据库/存储、队列/日志副作用 | 项目 integration 命令、`pytest`、本地服务 |
|
|
28
|
+
| smoke | base URL 和关键 endpoint 存活 | 项目 smoke 命令、小型 `pytest`/curl 脚本 |
|
|
29
|
+
| security-review | 鉴权、授权、敏感字段泄漏、注入风险 | 定向测试 + review 记录 |
|
|
30
|
+
|
|
31
|
+
## 必需 API 用例
|
|
32
|
+
|
|
33
|
+
API 变更应按适用性写入矩阵:
|
|
34
|
+
|
|
35
|
+
- 主路径:期望状态码、payload、headers 和业务状态
|
|
36
|
+
- 校验失败:非法、空、边界、格式错误、不支持字段
|
|
37
|
+
- 鉴权和权限:未登录、token 过期、角色错误、对象级授权
|
|
38
|
+
- 未找到和过期状态:资源不存在、已删除资源、stale version
|
|
39
|
+
- 冲突和幂等:重复创建、重复提交、带 idempotency key 的 retry、并发修改
|
|
40
|
+
- 适用时的限流或节流
|
|
41
|
+
- 列表 endpoint 变更时的分页、过滤、排序和空结果
|
|
42
|
+
- 数据一致性:响应、DB/存储、事件、队列消息、文件或日志
|
|
43
|
+
- teardown/cleanup:创建资源已删除,或隔离 fixture namespace 已重置
|
|
44
|
+
|
|
45
|
+
## 契约来源
|
|
46
|
+
|
|
47
|
+
契约来源优先级:
|
|
48
|
+
|
|
49
|
+
1. 仓库提交的 OpenAPI/protobuf/schema 文件。
|
|
50
|
+
2. 既有 request/response DTO、Pydantic model、serializer 或 typed client。
|
|
51
|
+
3. 明确写出字段和错误的需求/设计文档。
|
|
52
|
+
4. 验收用例中手写 schema。
|
|
53
|
+
|
|
54
|
+
当当前实现行为与需求冲突时,不得只按实现反推契约。
|
|
55
|
+
|
|
56
|
+
## 阻塞规则
|
|
57
|
+
|
|
58
|
+
缺少任一必需前置条件时记录 `blocked`:
|
|
59
|
+
|
|
60
|
+
- base URL 或服务启动命令
|
|
61
|
+
- auth token、角色或测试用户
|
|
62
|
+
- fixture seed/teardown 路径
|
|
63
|
+
- 依赖服务、数据库、队列或 mock server
|
|
64
|
+
- 稳定契约来源或期望 schema
|
|
65
|
+
- 确定性写后读结果面
|
|
66
|
+
|
|
67
|
+
不得只凭 2xx 响应把 API 验收标为 PASS。写操作必须包含可信结果面的写后读证据,例如 API 查询 endpoint、DB/存储记录、队列/事件/日志,或另一个项目自有状态面。
|
|
68
|
+
|
|
69
|
+
## 矩阵要求
|
|
70
|
+
|
|
71
|
+
`api` 行必须包含:
|
|
72
|
+
|
|
73
|
+
- `执行面`:`api`
|
|
74
|
+
- `验收模式`:`n/a`
|
|
75
|
+
- `证据层级`:`contract`、`integration`、`smoke` 或 `security-review`
|
|
76
|
+
- `框架/命令`:项目 API 命令、`python -m pytest tests/api -v`、curl/httpie 脚本、项目已使用时的 Postman/Newman,或契约工具
|
|
77
|
+
- `必需证据`:请求/响应记录、状态码、payload/schema 断言、适用时的鉴权/权限断言、写后读/数据一致性,以及 cleanup/teardown 证明
|
|
@@ -13,6 +13,10 @@ OpenTest 应根据变更类型、风险和项目事实选择适用覆盖面。
|
|
|
13
13
|
- `tdd-workflow`
|
|
14
14
|
- `e2e-runner`
|
|
15
15
|
- `browser-e2e-testing`
|
|
16
|
+
- `opentest-android-app`
|
|
17
|
+
- `android-midscene-pytest`
|
|
18
|
+
- `opentest-desktop-gui`
|
|
19
|
+
- `opentest-api`
|
|
16
20
|
- `verification-loop`
|
|
17
21
|
- `code-reviewer`
|
|
18
22
|
- `speckit-checklist`
|
|
@@ -28,6 +32,19 @@ OpenTest 应根据变更类型、风险和项目事实选择适用覆盖面。
|
|
|
28
32
|
| 权限、支付、安全、数据写入、跨页面闭环 | high-risk 验收或 E2E 证据 |
|
|
29
33
|
| 文案、配置、小范围无行为变更 | targeted review 或轻量证据 |
|
|
30
34
|
|
|
35
|
+
## 执行面选择
|
|
36
|
+
|
|
37
|
+
执行面和证据层级必须分开选择:
|
|
38
|
+
|
|
39
|
+
| 执行面 | 默认路由 |
|
|
40
|
+
| --- | --- |
|
|
41
|
+
| `web-browser` | Chrome DevTools MCP、Playwright CLI 或浏览器验收 |
|
|
42
|
+
| `android-app` | `opentest-android-app`;已安装时路由到 `android-midscene-pytest`,由 `python -m pytest tests_py -v` 通过 ADB 驱动 Midscene Android |
|
|
43
|
+
| `desktop-gui` | `opentest-desktop-gui`;优先项目 GUI 自动化,视觉/原生/RDP GUI 流程用 `@midscene/computer`,无自动化时才用脚本化人工 GUI 验收 |
|
|
44
|
+
| `api` | `opentest-api`;优先项目 API/integration 命令,否则用 `pytest` + `httpx`/`requests`、schema 校验、fixtures、写后读和 cleanup/teardown |
|
|
45
|
+
|
|
46
|
+
不得把 unit、component、integration、contract、smoke 或 security review 这类代码检查归类成执行面;它们只能作为证据层级。
|
|
47
|
+
|
|
31
48
|
## 前端验收维度
|
|
32
49
|
|
|
33
50
|
前端或真实链路验收可从以下维度中选择适用项,不要求每次全部覆盖:
|
|
@@ -74,7 +91,7 @@ OpenTest plan 阶段默认检查以下问题;适用则进入矩阵,不适用
|
|
|
74
91
|
| ACC-001 | 用户保存后看到成功反馈 | medium | UI 验收 | pending |
|
|
75
92
|
```
|
|
76
93
|
|
|
77
|
-
|
|
94
|
+
当矩阵需要驱动验收执行或风险要求时,再增加执行面、证据层级、命令/工具、证据路径和阻塞原因列。
|
|
78
95
|
|
|
79
96
|
## 质量门启发
|
|
80
97
|
|