opencode-codeops 2.5.0 → 2.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +13 -0
- package/README.md +39 -0
- package/_shared/spec-first-ordering.md +6 -2
- package/_shared/test-selection.md +130 -0
- package/bin/codeops-test.mjs +855 -0
- package/bin/lib/test-branch-policy.d.mts +51 -0
- package/bin/lib/test-branch-policy.mjs +185 -0
- package/bin/lib/test-map-validate.d.mts +42 -0
- package/bin/lib/test-map-validate.mjs +618 -0
- package/bin/lib/test-map.d.mts +114 -0
- package/bin/lib/test-map.mjs +384 -0
- package/bin/lib/test-provenance.d.mts +73 -0
- package/bin/lib/test-provenance.mjs +188 -0
- package/bin/lib/test-runner.d.mts +72 -0
- package/bin/lib/test-runner.mjs +350 -0
- package/bin/lib/test-selection-changes.d.mts +76 -0
- package/bin/lib/test-selection-changes.mjs +272 -0
- package/bin/lib/test-selection-explain.mjs +75 -0
- package/bin/lib/test-selection-glob.d.mts +32 -0
- package/bin/lib/test-selection-glob.mjs +206 -0
- package/bin/lib/test-selection.d.mts +80 -0
- package/bin/lib/test-selection.mjs +418 -0
- package/package.json +4 -2
- package/schemas/test-map.schema.json +164 -0
- package/scripts/__pycache__/codeops_plan.cpython-314.pyc +0 -0
- package/scripts/__pycache__/install_agents.cpython-314.pyc +0 -0
- package/skills/analyze-project/SKILL.md +8 -1
- package/skills/exec-plan/SKILL.md +16 -8
- package/skills/exec-plan/execution-protocol.md +15 -2
- package/skills/git-commit/SKILL.md +1 -1
- package/skills/make-plan/SKILL.md +10 -3
- package/skills/make-plan/quality-checklist.md +4 -0
- package/skills/make-plan/templates.md +19 -1
- package/skills/organize-tests/SKILL.md +169 -0
- package/skills/organize-tests/references/agents-testing-block.md +80 -0
- package/skills/organize-tests/references/analysis-protocol.md +94 -0
- package/skills/organize-tests/references/ci-github-actions.md +178 -0
- package/skills/organize-tests/references/language-adapters.md +54 -0
- package/skills/organize-tests/references/map-health.md +47 -0
- package/skills/organize-tests/references/project-wrapper.md +72 -0
- package/skills/organize-tests/references/shadow-mode.md +60 -0
- package/standards/coding-standards-full.md +28 -0
- package/standards/coding-standards.md +29 -0
- package/scripts/__pycache__/codeops_plan.cpython-312.pyc +0 -0
- package/scripts/__pycache__/install_agents.cpython-312.pyc +0 -0
|
@@ -50,6 +50,11 @@ Suggestion only — the user may proceed without upgrading.
|
|
|
50
50
|
|
|
51
51
|
### Phase start
|
|
52
52
|
|
|
53
|
+
Record the phase's expected modification set from its task target paths and deliverables. The set
|
|
54
|
+
drives the fallback rule: a task-time change outside it, or a shared/configuration/test-infrastructure
|
|
55
|
+
change, runs the full gate. For an old plan, the rule is unchanged: a single `**Verify**` line is the gate
|
|
56
|
+
command; with a map, derive the task command, otherwise use the same line for tasks.
|
|
57
|
+
|
|
53
58
|
Before a phase's first task (and before a T-NN mini-plan's first task), snapshot the complete
|
|
54
59
|
non-ignored worktree—including staged, unstaged, and untracked files—into a temporary Git tree,
|
|
55
60
|
without changing the real index:
|
|
@@ -102,8 +107,16 @@ complete. For each task, in order:
|
|
|
102
107
|
and update the Progress header. Do this before running verification or anything else — if the
|
|
103
108
|
session crashes now, the implementation progress is preserved and the resume session knows the
|
|
104
109
|
task still needs verification.
|
|
105
|
-
4. Run verification
|
|
106
|
-
|
|
110
|
+
4. Run verification, with output captured per the **Verify-output capture rule** below. Use the
|
|
111
|
+
task command for a normal task: the plan's validated `**Task verify**` line, or the map's
|
|
112
|
+
`task` gate with `--baseline <phase-baseline-tree>` and `--include <declared-spec-file>` for
|
|
113
|
+
every plan-declared specification test file. If neither exists, use the plan's
|
|
114
|
+
`**Verify (gate)**` command, or the project's full verify command (from AGENTS.md or detected
|
|
115
|
+
conventions). Run the full gate at phase exit, plan
|
|
116
|
+
completion, before push/release, and whenever a changed path falls outside the phase's
|
|
117
|
+
expected modification set or touches shared/configuration/test infrastructure. A single
|
|
118
|
+
`**Verify**` line is the gate command; a map may derive the task command from it, otherwise
|
|
119
|
+
that line serves both.
|
|
107
120
|
- **PASS** → before promoting, run the **documentation-standard self-check** below on the files
|
|
108
121
|
this task changed. Missing documentation and leaked plan references are invisible to
|
|
109
122
|
build+test, so a green verify is NOT sufficient — the self-check is a hard part of the
|
|
@@ -14,7 +14,7 @@ A request to commit authorizes a local commit only. Push only when the user expl
|
|
|
14
14
|
1. Resolve the repository root and inspect `git status --short`, staged/unstaged diffs, and recent commit style.
|
|
15
15
|
2. If clean, report `nothing to commit` and stop.
|
|
16
16
|
3. Inspect every untracked file. Stop and ask before staging likely secrets, credentials, build output, large binaries, or unrelated scratch files.
|
|
17
|
-
4. Resolve the authoritative verification command from `AGENTS.md`, CodeOps plan Verify lines, or project manifests. Run
|
|
17
|
+
4. Resolve the authoritative verification command from `AGENTS.md`, CodeOps plan Verify lines, or project manifests. For a normal commit, a validated scoped `**Task verify**` command or the map's `task` gate may replace the full command; record the command and scope in the verification note. Run the full gate for a release, merge, or push commit, and whenever the staged diff touches shared, configuration, manifest, or test infrastructure. Never bypass hooks. Run with output captured to a log under `$CODEOPS_TMPDIR` (fallback: the OS temp directory), never in the repository. On failure, report the failure tail and do not stage or commit. The gate rules are in [_shared/test-selection.md](../../_shared/test-selection.md).
|
|
18
18
|
5. Stage deliberately by explicit paths or coherent groups. Avoid `git add .` when the working directory may be a monorepo subtree or unrelated changes exist.
|
|
19
19
|
6. Review `git diff --cached --check`, the staged stat, and the staged diff. Ensure the commit contains one coherent purpose and preserves unrelated user changes.
|
|
20
20
|
7. Write a Conventional Commit message to a temporary file under `$CODEOPS_TMPDIR` (fallback: the OS temp directory) and commit with `git commit -F`. Never use an inline multiline message.
|
|
@@ -233,9 +233,16 @@ decisions remain user-owned.
|
|
|
233
233
|
3b. Every addition accepted through scope exploration must also carry its owning `SE-*`
|
|
234
234
|
back-reference. `Defer`, `Discard`, and `Superseded` entries never produce plan tasks.
|
|
235
235
|
4. `07-testing-strategy.md` must contain concrete **Specification Test Cases (ST-*)** with input→expected-output pairs, each traced to a requirement / spec doc / AR entry. Expectations come from the SPEC, never from imagined implementation behavior.
|
|
236
|
-
4b. **Confirm the verify
|
|
237
|
-
project's AGENTS.md or manifests — state what you
|
|
238
|
-
entry like any decision). If nothing is
|
|
236
|
+
4b. **Confirm the verify commands once.** The gate command that fills every `**Verify**` /
|
|
237
|
+
`**Verify (gate)**` line comes from the project's AGENTS.md or manifests — state what you
|
|
238
|
+
detected and have the user confirm it (an AR entry like any decision). If nothing is
|
|
239
|
+
detectable, ask — **never invent a command**. Optionally record a `**Task verify**` scoped
|
|
240
|
+
command for task/commit verification. Run the task command once during authoring, record the
|
|
241
|
+
observed test count as `**Task verify evidence**: <N> tests observed, <date>`, and reject a
|
|
242
|
+
command that runs zero tests. Record the plan's declared spec file(s) so exec-plan can
|
|
243
|
+
pass them with `--include`. With no map and no scoped command, generate exactly today's plan
|
|
244
|
+
(one `**Verify**` line). The canonical gate ladder is
|
|
245
|
+
[../../_shared/test-selection.md](../../_shared/test-selection.md).
|
|
239
246
|
5. `99-execution-plan.md` must structure every feature phase with the mandatory three-session ordering (Spec Tests → Implementation → Impl Tests & Hardening), carrying each phase's tasks as a **single checkbox list** — the plan's single source of truth for progress; a task line appears exactly once in the document. The full ordering rules are in [templates.md](templates.md).
|
|
240
247
|
6. If you discover a NEW ambiguity while writing, STOP immediately and add it to the register.
|
|
241
248
|
In normal mode get the user's decision; with active auto-design resolve an eligible technical
|
|
@@ -44,6 +44,10 @@ Run this before finalizing the plan documents. The **Specification-First Testing
|
|
|
44
44
|
- [ ] Spec test tasks reference ST-cases from `07-testing-strategy.md`
|
|
45
45
|
- [ ] Spec test and impl test files use separate naming conventions (`*.spec.test.*` and `*.impl.test.*`)
|
|
46
46
|
- [ ] A red-phase verification task exists in the execution plan (verify spec tests fail before implementation)
|
|
47
|
+
- [ ] `07-testing-strategy.md` carries the optional `Test slices` table, or records `N/A — full verify only`
|
|
48
|
+
- [ ] Every ST-case maps to a test slice or a validated scoped command
|
|
49
|
+
- [ ] Every slice has a task-tier test and a release/acceptance group or recorded acceptance reason
|
|
50
|
+
- [ ] The full gate command is recorded; a `**Task verify**` command, when present, proved a nonzero test count
|
|
47
51
|
|
|
48
52
|
## ✅ No Dead Code
|
|
49
53
|
- [ ] No unused parameters (except interface contracts, overrides, and framework-required signatures)
|
|
@@ -312,6 +312,21 @@ With no RD upstream, `01-requirements.md` IS the owning requirements doc:
|
|
|
312
312
|
- Adjust targets per project in `01-requirements.md` (an AR-referenced decision) — never
|
|
313
313
|
silently.
|
|
314
314
|
|
|
315
|
+
## Test Slices & Groups (optional)
|
|
316
|
+
|
|
317
|
+
> Fill this table when the project has a test map (`codeops/test-map.json` or
|
|
318
|
+
> `codeops/test-map/`) or when the plan records a validated `**Task verify**` scoped command.
|
|
319
|
+
> For a map-less project with no scoped command, write `N/A — full verify only` and keep the
|
|
320
|
+
> existing plan shape.
|
|
321
|
+
|
|
322
|
+
| Slice | Tier | Test Files | Sources | Depends On | Gate |
|
|
323
|
+
| ----- | ---- | ---------- | ------- | ---------- | ---- |
|
|
324
|
+
| [slice] | task | [paths] | [paths] | [slice/group] | task |
|
|
325
|
+
| [slice] | release | [paths] | [paths] | [task group] | release |
|
|
326
|
+
|
|
327
|
+
Every slice needs a task-tier test. Every slice needs a release or acceptance group, or a
|
|
328
|
+
recorded manual/external/N/A acceptance reason.
|
|
329
|
+
|
|
315
330
|
## 🚨 Specification Test Cases (MANDATORY — NON-NEGOTIABLE)
|
|
316
331
|
|
|
317
332
|
> These test cases are derived EXCLUSIVELY from requirements (`01-requirements.md`),
|
|
@@ -478,7 +493,10 @@ task-size criteria in [quality-checklist.md](quality-checklist.md))
|
|
|
478
493
|
- [ ] Deliverable 1
|
|
479
494
|
- [ ] All verification passing
|
|
480
495
|
|
|
481
|
-
**
|
|
496
|
+
**Test slices**: [optional — slice/tier summary, or `N/A — full verify only`]
|
|
497
|
+
**Task verify**: [optional validated scoped command and observed test count, or omit]
|
|
498
|
+
**Verify (gate)**: [full project verify command from AGENTS.md / detected conventions]
|
|
499
|
+
**Verify**: [legacy alias — same as the gate command when only one line is used]
|
|
482
500
|
|
|
483
501
|
---
|
|
484
502
|
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: organize-tests
|
|
3
|
+
description: Build or refresh a test-impact map for an existing project so local development runs only affected feature-slice test groups while CI keeps a full-suite release gate. Use for organize tests, group tests, test impact map, speed up tests, affected-only testing, test selection, or test-suite health. Detects runners and language adapters, inventories tests, proposes feature slices and tiered groups, runs shadow-mode adoption, updates AGENTS.md and CI additively, and audits drift. Never edits tests or weakens the full gate.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Organize Tests
|
|
7
|
+
|
|
8
|
+
`organize-tests` adopts opt-in test-impact selection in an existing project. It
|
|
9
|
+
reads the project, proposes feature-slice groups and language adapters, previews
|
|
10
|
+
every write, proves the selection in shadow mode, and only then applies the map.
|
|
11
|
+
The full suite remains the release, acceptance, and CI gate.
|
|
12
|
+
|
|
13
|
+
The skill is additive. It never edits, moves, deletes, or weakens a test. It
|
|
14
|
+
never removes a CI job or bypasses the full gate. A project can stop after any
|
|
15
|
+
preview and keep its current verification flow.
|
|
16
|
+
|
|
17
|
+
## Modes
|
|
18
|
+
|
|
19
|
+
| Mode | Behavior |
|
|
20
|
+
| --- | --- |
|
|
21
|
+
| default | Analyze, propose, preview, ask, then apply |
|
|
22
|
+
| `--dry-run` | Analysis and proposal only; write nothing |
|
|
23
|
+
| `--check` | Read-only validation and drift report |
|
|
24
|
+
| `--refresh` | Update globs, edges, and timings; preserve manual group edits |
|
|
25
|
+
| `--shadow` | Run the selected and full suites, write evidence, never enable automatically |
|
|
26
|
+
| `--audit` | Report orphans, overlaps, slow groups, never-selected groups, and flaky candidates |
|
|
27
|
+
| `--no-ci` | Skip the CI patch; map and AGENTS block only |
|
|
28
|
+
| `--custom` | Force the custom-command adapter for an unsupported language |
|
|
29
|
+
|
|
30
|
+
## Protocol
|
|
31
|
+
|
|
32
|
+
### 1. Preflight
|
|
33
|
+
|
|
34
|
+
Resolve the Git root, layout, existing map, manifests, runner configuration, CI
|
|
35
|
+
files, and `AGENTS.md` managed blocks. Refuse unsafe writes: a symlinked or
|
|
36
|
+
non-regular map, a malformed existing map, or a project with no full command.
|
|
37
|
+
Never overwrite a map the skill does not own.
|
|
38
|
+
|
|
39
|
+
### 2. Inventory tests
|
|
40
|
+
|
|
41
|
+
Enumerate test units with the project's native discovery. Record the owning
|
|
42
|
+
source path, the test path or target, and the observed test count. The recipes
|
|
43
|
+
are in [references/analysis-protocol.md](references/analysis-protocol.md) and
|
|
44
|
+
[references/language-adapters.md](references/language-adapters.md).
|
|
45
|
+
|
|
46
|
+
### 3. Propose slices
|
|
47
|
+
|
|
48
|
+
Use feature slices for business applications, pipeline stages for compilers,
|
|
49
|
+
packages for libraries, and workspace packages for monorepos. One test unit owns
|
|
50
|
+
exactly one group. Target 5–15 groups; split a slice that is slow or spans
|
|
51
|
+
unrelated behavior.
|
|
52
|
+
|
|
53
|
+
### 4. Derive dependencies
|
|
54
|
+
|
|
55
|
+
Use the native graph (Cargo metadata, Go list, Bazel/Nx, Maven/Gradle modules)
|
|
56
|
+
or production imports. `dependsOn` points upstream: a group lists what it needs.
|
|
57
|
+
A change selects its own group and downstream dependents, never upstream
|
|
58
|
+
providers. Never let shared test utilities pull unrelated groups together; when
|
|
59
|
+
uncertain, add an edge or make the group `alwaysRun`.
|
|
60
|
+
|
|
61
|
+
### 5. Assign tiers
|
|
62
|
+
|
|
63
|
+
`task` is the fast unit, component, or contract tier. `phase` covers API, DB,
|
|
64
|
+
and integration behavior. `release` covers end-to-end and acceptance behavior.
|
|
65
|
+
Every slice needs at least one task-tier group and either a release/acceptance
|
|
66
|
+
group or a per-slice acceptance record:
|
|
67
|
+
|
|
68
|
+
```json
|
|
69
|
+
{ "acceptance": { "mode": "manual", "reason": "approved by the owning team" } }
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Valid modes are `manual`, `external`, and `na`. Every record needs a reason.
|
|
73
|
+
|
|
74
|
+
### 6. Synthesize commands
|
|
75
|
+
|
|
76
|
+
Build argv commands per adapter. Run each candidate once and record the observed
|
|
77
|
+
test count; a command that runs zero tests is invalid. Resolve `{tests}` to the
|
|
78
|
+
group's full test set. For target-based runners (Rust, Go, C/C++), omit `tests`
|
|
79
|
+
when the command performs its own discovery; ownership then comes from `sources`
|
|
80
|
+
plus the command smoke test. Add `policy.prSmoke` groups for the PR gate and a
|
|
81
|
+
`requirements` probe for every `requires` name.
|
|
82
|
+
|
|
83
|
+
### 7. Preview
|
|
84
|
+
|
|
85
|
+
Show a summary table (slice, tier, tests, sources, dependsOn, duration, command),
|
|
86
|
+
file coverage, the map layout, the AGENTS block diff, and the CI diff. On a
|
|
87
|
+
non-integration branch, ask before writing.
|
|
88
|
+
|
|
89
|
+
### 8. Shadow
|
|
90
|
+
|
|
91
|
+
Run the selected groups, then the full suite, regardless of the first result.
|
|
92
|
+
Adoption requires zero missed failures. An escaped failure refuses adoption and
|
|
93
|
+
is reported with the group and failure tail. Never enable the map from shadow
|
|
94
|
+
mode. See [references/shadow-mode.md](references/shadow-mode.md).
|
|
95
|
+
|
|
96
|
+
### 9. Apply with staging
|
|
97
|
+
|
|
98
|
+
Render the map, policy, fragments, optional wrapper, `CODEOPS-TESTING` block,
|
|
99
|
+
CI diff, and package scripts in a staging area under `$CODEOPS_TMPDIR`. Validate
|
|
100
|
+
the map and every diff. Back up existing files, write in a fixed order, and
|
|
101
|
+
restore every prior file on failure. Never leave a partial map or a half-updated
|
|
102
|
+
block. Validate the staged result with `check --strict`.
|
|
103
|
+
|
|
104
|
+
### 10. Verify
|
|
105
|
+
|
|
106
|
+
Run `check`, one real selection, the full suite once, and `npm pack` or the
|
|
107
|
+
project build as applicable. Record the evidence. When `--metrics` is enabled,
|
|
108
|
+
emit only content-free fields: gate, selected/total, budget status, fallback
|
|
109
|
+
reason, and map hash. Never include test names, paths, output, or user text.
|
|
110
|
+
|
|
111
|
+
### 11. Refresh/audit
|
|
112
|
+
|
|
113
|
+
`--refresh` updates globs, edges, and timings without discarding manual groups or
|
|
114
|
+
owner fields. `--check` reports drift. `--audit` reports orphans, overlaps,
|
|
115
|
+
slices without a task tier, slow groups, never-selected groups, and flaky
|
|
116
|
+
candidates. The health rules are in
|
|
117
|
+
[references/map-health.md](references/map-health.md).
|
|
118
|
+
|
|
119
|
+
## AGENTS.md testing block
|
|
120
|
+
|
|
121
|
+
Only `organize-tests` writes the `<!-- CODEOPS-TESTING:START -->` /
|
|
122
|
+
`<!-- CODEOPS-TESTING:END -->` block. `analyze-project` detects it, preserves it
|
|
123
|
+
byte-for-byte, and warns when it is missing or stale; it never renders the block.
|
|
124
|
+
The contract, the optional `Test slices` table (slice, tier, tests, sources,
|
|
125
|
+
gate), and the wrapper guidance are in
|
|
126
|
+
[references/agents-testing-block.md](references/agents-testing-block.md).
|
|
127
|
+
|
|
128
|
+
## CI patch rules
|
|
129
|
+
|
|
130
|
+
- Add a required full gate for release, acceptance, and package branches.
|
|
131
|
+
- Add an optional affected PR job behind a stable `verify` aggregator.
|
|
132
|
+
- Add nightly only when the project wants it; use the change marker.
|
|
133
|
+
- Add provenance only when the project builds a deployable artifact.
|
|
134
|
+
- Suggest a `.github/CODEOWNERS` entry for `codeops/test-map/**`; on approval,
|
|
135
|
+
add it without touching existing ownership rules.
|
|
136
|
+
- Preserve existing jobs, caching, toolchain setup, and branch filters. Preview
|
|
137
|
+
every diff and never remove a job.
|
|
138
|
+
- Never use `pull_request_target` with fork code or expose secrets to fork PRs.
|
|
139
|
+
|
|
140
|
+
The advisory templates are in
|
|
141
|
+
[references/ci-github-actions.md](references/ci-github-actions.md) and
|
|
142
|
+
[references/project-wrapper.md](references/project-wrapper.md). They are
|
|
143
|
+
previewed like every other change and are never applied automatically.
|
|
144
|
+
|
|
145
|
+
## Safety rules
|
|
146
|
+
|
|
147
|
+
- Never edit, move, delete, or weaken a test.
|
|
148
|
+
- Never remove a CI job or weaken the full gate.
|
|
149
|
+
- No adoption on shadow misses.
|
|
150
|
+
- Refuse malformed or symlinked maps and paths outside the root.
|
|
151
|
+
- Stage and validate every write; restore prior files on failure.
|
|
152
|
+
- Idempotent refresh; preserve manual group edits and owner fields.
|
|
153
|
+
- No map write without preview and approval on non-integration branches.
|
|
154
|
+
|
|
155
|
+
## Output contract
|
|
156
|
+
|
|
157
|
+
The preview names the map layout, each slice and tier, the command per group, the
|
|
158
|
+
file coverage, and every file that will change. The shadow report records the
|
|
159
|
+
verdict, selected groups, selected result, full result, escaped groups, and
|
|
160
|
+
`enabled: false`. The apply report lists the written files and the validation
|
|
161
|
+
result. The audit report names every finding with a path or group ID.
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
|
|
165
|
+
> **Workspace hygiene (non-negotiable):** put every temporary artifact this run creates in
|
|
166
|
+
> `$CODEOPS_TMPDIR` (fallback: the OS temp directory), never in the repository, and delete them
|
|
167
|
+
> all before reporting completion. Never delete user files, versioned artifacts, worktrees, or
|
|
168
|
+
> another session's temporary files. Full rules:
|
|
169
|
+
> [_shared/workspace-hygiene.md](../../_shared/workspace-hygiene.md).
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# AGENTS.md Testing Block
|
|
2
|
+
|
|
3
|
+
> **CodeOps Artifact Schema**: 1
|
|
4
|
+
|
|
5
|
+
The testing block tells an agent how to verify this project without re-reading
|
|
6
|
+
the map. It lives inside the existing `CODEOPS-PROJECT` managed block and is
|
|
7
|
+
owned by one writer.
|
|
8
|
+
|
|
9
|
+
## Ownership
|
|
10
|
+
|
|
11
|
+
`organize-tests` is the **sole writer** of the block. `analyze-project` detects
|
|
12
|
+
the block and preserves it byte-for-byte on every refresh; it never renders or
|
|
13
|
+
overwrites it, and it warns when the block is missing or stale. A project that
|
|
14
|
+
has no map is not given a block by `analyze-project`.
|
|
15
|
+
|
|
16
|
+
The block markers are:
|
|
17
|
+
|
|
18
|
+
```markdown
|
|
19
|
+
<!-- CODEOPS-TESTING:START -->
|
|
20
|
+
<!-- CODEOPS-TESTING:END -->
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
## With a map
|
|
24
|
+
|
|
25
|
+
```markdown
|
|
26
|
+
<!-- CODEOPS-TESTING:START -->
|
|
27
|
+
- Test map: `codeops/test-map/` (8 groups, 3 slices)
|
|
28
|
+
- Task verify: `npm run verify:task` (affected task-tier groups + dependents)
|
|
29
|
+
- Phase verify: `npm run verify:phase` (task + phase tiers)
|
|
30
|
+
- Gate verify: `npm run verify` (full suite; release/acceptance/CI)
|
|
31
|
+
- Map check: `npm run test:map:check`
|
|
32
|
+
- Last map check: 2026-11-11 23:36
|
|
33
|
+
<!-- CODEOPS-TESTING:END -->
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
The commands are the project's real commands. Never invent a script name; if the
|
|
37
|
+
project does not want a wrapper, point at the CLI command instead.
|
|
38
|
+
|
|
39
|
+
## Optional Test slices table
|
|
40
|
+
|
|
41
|
+
A project that wants a map-less reference may add a human-readable table after
|
|
42
|
+
the commands. It is informational; the map remains the source of truth.
|
|
43
|
+
|
|
44
|
+
| Slice | Tier | Tests | Sources | Gate |
|
|
45
|
+
| --- | --- | --- | --- | --- |
|
|
46
|
+
| `billing.invoice-dialog` | task | `web/billing/**/*.test.tsx` | `web/billing/**` | task |
|
|
47
|
+
| `billing.invoice-dialog` | release | `tests/e2e/billing/**` | `web/billing/**` | release |
|
|
48
|
+
| `shared.auth` | task | `shared/auth/**/*.test.ts` | `shared/auth/**` | always-run |
|
|
49
|
+
|
|
50
|
+
The table columns are slice, tier, tests, sources, and gate. Keep it short; a
|
|
51
|
+
large table belongs in the map, not in `AGENTS.md`.
|
|
52
|
+
|
|
53
|
+
## Without a map
|
|
54
|
+
|
|
55
|
+
```markdown
|
|
56
|
+
<!-- CODEOPS-TESTING:START -->
|
|
57
|
+
- Testing: full suite only (no test map)
|
|
58
|
+
- Gate verify: `npm run verify`
|
|
59
|
+
<!-- CODEOPS-TESTING:END -->
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
The no-map text preserves today's behavior. `analyze-project` leaves an existing
|
|
63
|
+
no-map block untouched.
|
|
64
|
+
|
|
65
|
+
## Refresh rules
|
|
66
|
+
|
|
67
|
+
- Replace only the block contents; preserve surrounding managed and
|
|
68
|
+
hand-authored text byte-for-byte.
|
|
69
|
+
- Keep exactly one block; stop on duplicate or reversed markers and name the
|
|
70
|
+
manual fix.
|
|
71
|
+
- Update `Last map check` only after a successful `check`.
|
|
72
|
+
- Write with the file's dominant line ending; keep exactly one block; fail without writing on duplicate or reversed markers.
|
|
73
|
+
- Never put secrets, absolute machine paths, or test output in the block.
|
|
74
|
+
|
|
75
|
+
## Wrapper guidance
|
|
76
|
+
|
|
77
|
+
A thin project wrapper is optional. If the project wants npm scripts, generate a
|
|
78
|
+
small file that calls the plugin CLI with the project's map path and forwards
|
|
79
|
+
arguments. The wrapper contains no selection logic; it must not grow into a
|
|
80
|
+
second implementation. Record the wrapper path in the block.
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# Analysis and Inventory Protocol
|
|
2
|
+
|
|
3
|
+
> **CodeOps Artifact Schema**: 1
|
|
4
|
+
|
|
5
|
+
This reference describes how `organize-tests` turns a repository into a proposed
|
|
6
|
+
test map. It is evidence-first: every group traces to a real source path, a real
|
|
7
|
+
test unit, and a command that was observed to run at least one test.
|
|
8
|
+
|
|
9
|
+
## 1. Project reconnaissance
|
|
10
|
+
|
|
11
|
+
Read, without writing:
|
|
12
|
+
|
|
13
|
+
- the Git root and current branch;
|
|
14
|
+
- `codeops/test-map.json` and `codeops/test-map/` when present;
|
|
15
|
+
- manifests and lockfiles (`package.json`, `Cargo.toml`, `go.mod`,
|
|
16
|
+
`pyproject.toml`, `pom.xml`, `build.gradle`, `*.sln`, `CMakeLists.txt`);
|
|
17
|
+
- runner configuration (`vitest.config.*`, `jest.config.*`, `pytest.ini`,
|
|
18
|
+
`nextest.toml`, CTest files);
|
|
19
|
+
- CI workflows, especially the full verify job;
|
|
20
|
+
- the `CODEOPS-PROJECT` and `CODEOPS-TESTING` blocks in `AGENTS.md`.
|
|
21
|
+
|
|
22
|
+
Record the project's full verify command and the exact file or script that
|
|
23
|
+
defines it. A missing full command stops adoption.
|
|
24
|
+
|
|
25
|
+
## 2. Test inventory
|
|
26
|
+
|
|
27
|
+
For each test unit, record:
|
|
28
|
+
|
|
29
|
+
| Field | Meaning |
|
|
30
|
+
| --- | --- |
|
|
31
|
+
| `test` | Repo-relative file path, target name, or test ID |
|
|
32
|
+
| `source` | The production path or target that owns it |
|
|
33
|
+
| `runner` | Detected adapter |
|
|
34
|
+
| `count` | Observed number of tests |
|
|
35
|
+
| `durationSec` | Observed or estimated duration |
|
|
36
|
+
|
|
37
|
+
Use native discovery where possible. Do not guess a test count; run the
|
|
38
|
+
candidate command and read the reported number.
|
|
39
|
+
|
|
40
|
+
## 3. Slice proposal
|
|
41
|
+
|
|
42
|
+
Group units by the feature or pipeline stage a developer changes, not by folder
|
|
43
|
+
alone:
|
|
44
|
+
|
|
45
|
+
- business app: dialog, invoice, inventory, auth;
|
|
46
|
+
- compiler: lexer, parser, codegen, assembler, linker, e2e;
|
|
47
|
+
- library: public API, adapters, compatibility;
|
|
48
|
+
- monorepo: workspace package boundaries.
|
|
49
|
+
|
|
50
|
+
One unit belongs to exactly one group. A slice carries one or more tiers of the
|
|
51
|
+
same feature. Split a slice when its task tier exceeds the budget or when its
|
|
52
|
+
tests cover unrelated behavior.
|
|
53
|
+
|
|
54
|
+
## 4. Dependency graph
|
|
55
|
+
|
|
56
|
+
Collect edges from the native graph first. Fall back to production imports, not
|
|
57
|
+
test imports. `dependsOn` lists upstream groups: a parser group is upstream of a
|
|
58
|
+
codegen group. Selection walks reverse edges, so changing the parser selects
|
|
59
|
+
codegen and its dependents, while changing codegen does not select the parser.
|
|
60
|
+
|
|
61
|
+
Shared test utilities are not a dependency shortcut. If a helper is used by
|
|
62
|
+
unrelated slices, keep it in a narrow shared group or mark the consumers
|
|
63
|
+
`alwaysRun`; never merge unrelated slices to satisfy the helper.
|
|
64
|
+
|
|
65
|
+
## 5. Command synthesis
|
|
66
|
+
|
|
67
|
+
Build an argv command per group. Verify it:
|
|
68
|
+
|
|
69
|
+
1. run it once on the current tree;
|
|
70
|
+
2. confirm it exits zero;
|
|
71
|
+
3. confirm the reported test count is greater than zero;
|
|
72
|
+
4. confirm it runs a subset, not the whole suite.
|
|
73
|
+
|
|
74
|
+
Record the count in the preview. A zero-test command is rejected. A command that
|
|
75
|
+
runs the whole suite defeats the purpose and is moved back to the full gate.
|
|
76
|
+
|
|
77
|
+
## 6. Preview contents
|
|
78
|
+
|
|
79
|
+
The preview must show:
|
|
80
|
+
|
|
81
|
+
- the chosen layout (single file or policy plus fragments);
|
|
82
|
+
- every slice, tier, source glob, test glob, dependency, and duration;
|
|
83
|
+
- the exact command per group and its observed count;
|
|
84
|
+
- files that will be created or changed;
|
|
85
|
+
- the AGENTS block diff and the CI diff;
|
|
86
|
+
- validation findings from the staged map.
|
|
87
|
+
|
|
88
|
+
Nothing is written before the preview is accepted.
|
|
89
|
+
|
|
90
|
+
## 7. Evidence to keep
|
|
91
|
+
|
|
92
|
+
Record the observed counts, the shadow verdict, the `check --strict` result, and
|
|
93
|
+
the date of the last map check. Evidence lives in the plan or the adoption
|
|
94
|
+
report, not in shipped source comments.
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
# GitHub Actions Test-Selection Templates
|
|
2
|
+
|
|
3
|
+
> **CodeOps Artifact Schema**: 1
|
|
4
|
+
|
|
5
|
+
These templates are advisory. `organize-tests` previews every diff and never
|
|
6
|
+
removes an existing job. A project can keep its current full gate and adopt only
|
|
7
|
+
the map, or adopt the affected PR job later.
|
|
8
|
+
|
|
9
|
+
## Shape
|
|
10
|
+
|
|
11
|
+
```text
|
|
12
|
+
PR:
|
|
13
|
+
plan(affected) -> test shards -> verify aggregator (required)
|
|
14
|
+
|
|
15
|
+
release/** / hotfix/** / v* tag:
|
|
16
|
+
full gate -> build artifact -> package -> provenance -> deploy
|
|
17
|
+
|
|
18
|
+
acceptance/**:
|
|
19
|
+
full gate + acceptance groups -> provenance
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## Affected plan and shards
|
|
23
|
+
|
|
24
|
+
```yaml
|
|
25
|
+
jobs:
|
|
26
|
+
plan:
|
|
27
|
+
runs-on: ubuntu-latest
|
|
28
|
+
outputs:
|
|
29
|
+
matrix: ${{ steps.plan.outputs.matrix }}
|
|
30
|
+
mode: ${{ steps.plan.outputs.mode }}
|
|
31
|
+
steps:
|
|
32
|
+
- uses: actions/checkout@v7
|
|
33
|
+
with:
|
|
34
|
+
fetch-depth: 0
|
|
35
|
+
- run: npm ci --ignore-scripts
|
|
36
|
+
- id: plan
|
|
37
|
+
run: node bin/codeops-test.mjs plan --gate pr --since "origin/${{ github.base_ref }}" --json > plan.json
|
|
38
|
+
- run: echo "matrix=$(jq -c .matrix plan.json)" >> "$GITHUB_OUTPUT"
|
|
39
|
+
- run: echo "mode=$(jq -r .mode plan.json)" >> "$GITHUB_OUTPUT"
|
|
40
|
+
- uses: actions/upload-artifact@v4
|
|
41
|
+
with:
|
|
42
|
+
name: test-plan
|
|
43
|
+
path: plan.json
|
|
44
|
+
|
|
45
|
+
test:
|
|
46
|
+
needs: plan
|
|
47
|
+
if: needs.plan.outputs.mode == 'groups' && needs.plan.outputs.matrix != '{"include":[]}'
|
|
48
|
+
strategy:
|
|
49
|
+
fail-fast: false
|
|
50
|
+
matrix: ${{ fromJSON(needs.plan.outputs.matrix) }}
|
|
51
|
+
runs-on: ubuntu-latest
|
|
52
|
+
steps:
|
|
53
|
+
- uses: actions/checkout@v7
|
|
54
|
+
- run: npm ci --ignore-scripts
|
|
55
|
+
- uses: actions/download-artifact@v4
|
|
56
|
+
with:
|
|
57
|
+
name: test-plan
|
|
58
|
+
- run: node bin/codeops-test.mjs run --selection plan.json --group "${{ matrix.group }}"
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
The plan has four modes: `groups`, `full`, `none`, and `skip`. Every mode
|
|
62
|
+
returns a matrix object. `none` and `skip` use an empty matrix. A full run is a
|
|
63
|
+
separate job, never a fake group entry, so `run --group full` remains a usage
|
|
64
|
+
error.
|
|
65
|
+
|
|
66
|
+
```yaml
|
|
67
|
+
full:
|
|
68
|
+
needs: plan
|
|
69
|
+
if: needs.plan.outputs.mode == 'full'
|
|
70
|
+
uses: ./.github/workflows/verify-full.yml
|
|
71
|
+
|
|
72
|
+
check:
|
|
73
|
+
needs: plan
|
|
74
|
+
runs-on: ubuntu-latest
|
|
75
|
+
steps:
|
|
76
|
+
- uses: actions/checkout@v7
|
|
77
|
+
- run: npm ci --ignore-scripts
|
|
78
|
+
- run: node bin/codeops-test.mjs check --strict
|
|
79
|
+
|
|
80
|
+
verify:
|
|
81
|
+
needs: [plan, test, full, check]
|
|
82
|
+
if: always()
|
|
83
|
+
runs-on: ubuntu-latest
|
|
84
|
+
steps:
|
|
85
|
+
- run: |
|
|
86
|
+
test "${{ needs.plan.result }}" = "success"
|
|
87
|
+
test "${{ needs.check.result }}" = "success"
|
|
88
|
+
if [ "${{ needs.plan.outputs.mode }}" = "groups" ]; then
|
|
89
|
+
test "${{ needs.test.result }}" = "success"
|
|
90
|
+
fi
|
|
91
|
+
if [ "${{ needs.plan.outputs.mode }}" = "full" ]; then
|
|
92
|
+
test "${{ needs.full.result }}" = "success"
|
|
93
|
+
fi
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
The stable `verify` job is the required branch-protection check. Dynamic matrix
|
|
97
|
+
names are never required checks. `check --strict` runs beside it so a malformed
|
|
98
|
+
map stays visible even when selection falls back to full.
|
|
99
|
+
|
|
100
|
+
## Release and acceptance
|
|
101
|
+
|
|
102
|
+
Release and acceptance always run the full gate. Build the artifact only after
|
|
103
|
+
the gate, or build first and let the gate consume the artifact.
|
|
104
|
+
|
|
105
|
+
```yaml
|
|
106
|
+
verify:
|
|
107
|
+
uses: ./.github/workflows/verify-full.yml
|
|
108
|
+
|
|
109
|
+
package:
|
|
110
|
+
needs: verify
|
|
111
|
+
runs-on: ubuntu-latest
|
|
112
|
+
steps:
|
|
113
|
+
- uses: actions/checkout@v7
|
|
114
|
+
- run: npm ci --ignore-scripts
|
|
115
|
+
- run: npm run build
|
|
116
|
+
- run: node bin/codeops-test.mjs provenance write --gate release --artifact dist/app.tar.gz --source-commit "$GITHUB_SHA" --record release.json
|
|
117
|
+
- uses: actions/upload-artifact@v4
|
|
118
|
+
with:
|
|
119
|
+
name: release-evidence
|
|
120
|
+
path: release.json
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
`--source-commit` records the commit the full gate tested. When the gate ran on
|
|
124
|
+
source, the artifact must be reproducible; `provenance verify` fails on a
|
|
125
|
+
different digest or source commit. Acceptance uses `--gate acceptance` and
|
|
126
|
+
records the manual or external result.
|
|
127
|
+
|
|
128
|
+
## Nightly
|
|
129
|
+
|
|
130
|
+
```yaml
|
|
131
|
+
name: Nightly Full
|
|
132
|
+
on:
|
|
133
|
+
schedule: [{ cron: "0 3 * * *" }]
|
|
134
|
+
workflow_dispatch:
|
|
135
|
+
concurrency:
|
|
136
|
+
group: nightly-full
|
|
137
|
+
cancel-in-progress: false
|
|
138
|
+
jobs:
|
|
139
|
+
plan:
|
|
140
|
+
runs-on: ubuntu-latest
|
|
141
|
+
outputs:
|
|
142
|
+
run_full: ${{ steps.plan.outputs.run_full }}
|
|
143
|
+
steps:
|
|
144
|
+
- uses: actions/checkout@v7
|
|
145
|
+
with: { fetch-depth: 0 }
|
|
146
|
+
- run: npm ci --ignore-scripts
|
|
147
|
+
- id: plan
|
|
148
|
+
env:
|
|
149
|
+
CODEOPS_LAST_FULL_SHA: ${{ vars.CODEOPS_LAST_FULL_SHA }}
|
|
150
|
+
run: node bin/codeops-test.mjs plan --gate nightly --last-full-sha "$CODEOPS_LAST_FULL_SHA" --json > plan.json
|
|
151
|
+
- run: echo "run_full=$(jq -r '.mode != "skip"' plan.json)" >> "$GITHUB_OUTPUT"
|
|
152
|
+
full:
|
|
153
|
+
needs: plan
|
|
154
|
+
if: needs.plan.outputs.run_full == 'true'
|
|
155
|
+
uses: ./.github/workflows/verify-full.yml
|
|
156
|
+
update-marker:
|
|
157
|
+
needs: full
|
|
158
|
+
if: needs.full.result == 'success'
|
|
159
|
+
runs-on: ubuntu-latest
|
|
160
|
+
permissions:
|
|
161
|
+
actions: write
|
|
162
|
+
steps:
|
|
163
|
+
- run: gh variable set CODEOPS_LAST_FULL_SHA --body "$GITHUB_SHA"
|
|
164
|
+
env:
|
|
165
|
+
GH_TOKEN: ${{ github.token }}
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
The marker updates only after a successful full run. Never update it from a
|
|
169
|
+
failed or skipped job.
|
|
170
|
+
|
|
171
|
+
## Governance
|
|
172
|
+
|
|
173
|
+
- Protect `codeops/test-map/**` with a `CODEOWNERS` entry; map changes are
|
|
174
|
+
reviewed like CI changes.
|
|
175
|
+
- Use `pull_request`, never `pull_request_target` with fork code. The map
|
|
176
|
+
contains commands and must be treated as code.
|
|
177
|
+
- Preserve existing caching, toolchain setup, permissions, and branch filters.
|
|
178
|
+
- Do not add a path filter that lets release or acceptance skip the full gate.
|