@cleocode/core 2026.4.5 → 2026.4.6

package/src/validation/protocols/protocols-markdown/testing.md

@@ -0,0 +1,287 @@
+---
+id: TEST
+title: Testing Protocol (Project-Agnostic IVT Loop)
+version: 2.0.0
+status: active
+type: base
+audience: [llm-agent, orchestrator]
+tags: [testing, ivt-loop, autonomous, framework-agnostic, compliance]
+skillRef: ct-ivt-looper
+lastUpdated: 2026-04-07
+enforcement: strict
+---
+
+# Testing Protocol — Project-Agnostic IVT Loop
+
+**Provenance**: @task T260 (replaces T3155 BATS-locked v1)
+**Version**: 2.0.0
+**Type**: Base Protocol
+**Stage**: IVTR — T (Testing, the closure of the IVT loop)
+**Skill**: `ct-ivt-looper`
+
+This protocol defines the **autonomous compliance layer** that runs before any release or pull request. It is intentionally **project-agnostic** — it works in any git worktree regardless of language, framework, or directory layout. The previous v1 of this protocol was hardcoded to BATS (Bash testing) and is fully superseded.
+
+---
+
+## Trigger Conditions
+
+This protocol activates when the task involves:
+
+| Trigger | Keywords | Context |
+|---------|----------|---------|
+| Loop Closure | "ivt loop", "implement and verify", "ship this task" | Autonomous compliance run |
+| Test Execution | "run tests", "verify", "test this" | Test running across any framework |
+| Coverage | "coverage", "test coverage", "missing tests" | Coverage gap analysis |
+| Spec Compliance | "satisfy spec", "verify against spec", "match the requirements" | Spec-driven verification |
+| Release Gate | "before release", "pre-PR validation", "ship verified" | Last gate before release stage |
+
+**Explicit Override**: `--protocol testing` flag on task creation.
+
+**Lifecycle Position**: After Validation (V), before Release (R). The Testing stage is the **closure of the Implement → Validate → Test loop**, not a one-shot operation.
+
+---
+
+## Core Principle
+
+> **The loop converges on the spec, not on "tests pass". Passing tests that don't cover the spec are a failure.**
+
+The Testing stage is autonomous: it iterates Implement → Validate → Test until the implementation satisfies every MUST requirement of the specification. It is **not** a single "run the test suite" operation. Convergence is measured against the spec, not against test count.
+
+---
+
+## Project-Agnostic Mandate
+
+This protocol MUST work in any project. It MUST NOT assume:
+
+- A specific language (TypeScript, Python, Rust, Go, Ruby, PHP, Bash, …)
+- A specific test framework (vitest, jest, pytest, cargo test, …)
+- A specific directory layout (`tests/`, `__tests__/`, `spec/`, `_test.go`, …)
+- A specific test command (`pnpm test`, `pytest`, `cargo test`, `go test ./...`, …)
+
+The implementation skill (`ct-ivt-looper`) MUST detect the framework dynamically from project signals before invoking any test command.
+
+---
+
+## Requirements (RFC 2119)
+
+### MUST
+
+| Requirement | Description |
+|-------------|-------------|
+| TEST-001 | MUST identify the project's test framework via dynamic detection (config files, lockfile, devDependencies, language toolchain). The framework MUST NOT be hardcoded. |
+| TEST-002 | MUST run the IVT loop with an explicit iteration counter and a hard `MAX_ITERATIONS` cap (default 5). |
+| TEST-003 | MUST trace each MUST requirement from the upstream specification to at least one passing test. |
+| TEST-004 | MUST achieve 100% pass rate before exiting with `ivtLoopConverged: true`. Any failing test blocks convergence. |
+| TEST-005 | MUST exit the loop only when the spec is satisfied (every MUST has a passing test) or `MAX_ITERATIONS` is reached. |
+| TEST-006 | MUST write a test summary to the manifest entry's `key_findings` array — at minimum framework, total run, passed, failed, coverage, iterations. |
+| TEST-007 | MUST set `agent_type: "testing"` in the manifest entry. |
+| TEST-008 | MUST run on a feature branch — never on `main`/`master`. The skill MUST stop and request a feature branch if the worktree is on the trunk. |
+| TEST-009 | MUST escalate non-convergence to HITL via exit code 65 with a manifest entry containing `ivtLoopConverged: false` and `ivtLoopIterations: <n>`. |
+
+### SHOULD
+
+| Requirement | Description |
+|-------------|-------------|
+| TEST-010 | SHOULD test edge cases and error paths, not just happy paths. |
+| TEST-011 | SHOULD include setup/teardown fixtures appropriate to the detected framework. |
+| TEST-012 | SHOULD use descriptive test names tied to spec requirement codes (e.g., "ADR-003: rejects accepted status without HITL review"). |
+| TEST-013 | SHOULD report a coverage percentage when the framework supports it; warn (non-blocking) if below `coverageThreshold`. |
+| TEST-014 | SHOULD prefer `.cleo/project-context.json#testing.command` when present — that file is CLEO's per-project canonical test command. |
+
+### MAY
+
+| Requirement | Description |
+|-------------|-------------|
+| TEST-020 | MAY add golden tests for output verification. |
+| TEST-021 | MAY add performance benchmarks. |
+| TEST-022 | MAY add stress / concurrency tests. |
+| TEST-023 | MAY parallelize the IVT loop across independent specs when the agent has worktree isolation. |
+
+---
+
+## The IVT Loop
+
+```
+┌──────────────────────────────────────────────────────────────────┐
+│  Implement → Validate → Test                                     │
+│                                                                  │
+│  1. Load spec (MUST requirements)                                │
+│  2. Apply implementation changes                                 │
+│  3. Run validation (lint + typecheck + spec-match)               │
+│  4. Detect framework, run tests                                  │
+│  5. Convergence check:                                           │
+│     - All spec MUSTs have passing tests?                         │
+│     - Validation clean? Tests 100% pass?                         │
+│     - YES → record convergence, exit loop                        │
+│     - NO  → analyse failures, increment counter                  │
+│  6. If counter >= MAX_ITERATIONS → escalate HITL, exit code 65   │
+│  7. Else → generate fix, GOTO 2                                  │
+└──────────────────────────────────────────────────────────────────┘
+```
+
+The loop is **autonomous within MAX_ITERATIONS**. It never spins forever and never exits silently with unmet requirements.
+
+---
+
+## Framework Detection (Project-Agnostic)
+
+The skill MUST resolve the test command in this priority order:
+
+1. **`.cleo/project-context.json#testing.command`** — CLEO's per-project canonical (highest priority)
+2. **Auto-detection signals** in the worktree:
+
+| Signal | Framework | Canonical Command |
+|--------|-----------|-------------------|
+| `vitest.config.*` or `vitest` in package.json devDeps | vitest | `pnpm vitest run` (or `npx vitest run`) |
+| `jest.config.*` or `jest` in package.json devDeps | jest | `npx jest --ci` |
+| `mocha` in package.json devDeps and `.mocharc*` | mocha | `npx mocha` |
+| `pytest.ini` / `pyproject.toml [tool.pytest]` / `conftest.py` | pytest | `pytest -q` |
+| `tests/__init__.py` or `unittest` in stdlib usage | unittest | `python -m unittest discover -q` |
+| `Cargo.toml` | cargo-test | `cargo test --quiet` |
+| `go.mod` | go-test | `go test ./...` |
+| `Gemfile` with `rspec` | rspec | `bundle exec rspec` |
+| `phpunit.xml*` | phpunit | `vendor/bin/phpunit` |
+| `*.bats` files in `tests/` | bats | `bats tests/` |
+| none of the above | other | fall back to project-context.json or HITL ask |
+
+3. **Fallback** — if no signal matches, the skill MUST stop and ask the operator to declare the test command in `.cleo/project-context.json`.
+
+---
+
+## Convergence Criteria
+
+The loop is converged when **ALL** of these are true:
+
+- Every MUST requirement from the upstream specification maps to at least one passing test (TEST-003).
+- The test suite reports `failed: 0`.
+- Lint / format / typecheck reports zero errors.
+- Spec-match validation (validation stage protocol VALID-001) confirms implementation matches the spec.
+- No runtime errors in CI-equivalent local commands.
+
+If any of the above is false, the loop continues until `MAX_ITERATIONS` is hit.
+
+---
+
+## Branch Discipline
+
+- Detect the current branch via `git branch --show-current`.
+- If the branch is `main`, `master`, `trunk`, `develop`, or matches `release/*` → STOP.
+- Refuse to run the IVT loop on a trunk branch. Request a feature branch from the operator before continuing.
+- The IVT loop modifies code; trunk branches MUST be protected.
+
+---
+
+## Manifest Entry
+
+Use `cleo research add` to record the IVT loop result:
+
+```bash
+cleo research add \
+  --title "Testing IVT loop: <feature>" \
+  --file "T####-ivt-report.md" \
+  --topics "testing,ivt-loop,<framework>" \
+  --findings "framework: <name>,run: N,passed: N,failed: 0,iterations: <n>,converged: true" \
+  --status complete \
+  --task T#### \
+  --agent-type testing
+```
+
+The orchestrator MUST then call:
+
+```bash
+cleo check protocol --protocolType testing \
+  --taskId T#### \
+  --framework <name> \
+  --testsRun N --testsPassed N --testsFailed 0 \
+  --ivtLoopConverged true --ivtLoopIterations <n>
+```
+
+This invokes the canonical pure validator (`validateTestingProtocol` in `packages/core/src/orchestration/protocol-validators.ts`) and is the runtime authority for whether the testing stage is complete.
+
+---
+
+## Integration Points
+
+### With Implementation Protocol (the "I" of IVT)
+
+The IVT loop wraps `implementation` work — every fix iteration is governed by IMPL-001..007. The loop MUST NOT skip the implementation protocol's own gates.
+
+### With Validation Protocol (the "V" of IVT)
+
+The validation stage runs **inside** each IVT iteration (step 3), not just before testing. Validation reports feed back into the convergence check.
+
+### With Release Protocol
+
+```
+Testing (T) ──► Release (R)
+       │              │
+       │ Converged?   │ Requires:
+       │ Yes          │ - ivtLoopConverged: true
+       │              │ - testsFailed: 0
+       │              │ - All MUSTs covered
+```
+
+The release stage MUST refuse to advance if the testing stage's manifest entry has `ivtLoopConverged: false`.
+
+---
+
+## Exit Codes
+
+| Code | Constant | Meaning |
+|------|----------|---------|
+| 0 | SUCCESS | Loop converged, manifest recorded |
+| 65 | HANDOFF_REQUIRED | Non-convergence after MAX_ITERATIONS — HITL escalation |
+| 67 | CONCURRENT_SESSION | Generic testing protocol violation (used by orchestration validator) |
+| 80 | LIFECYCLE_GATE_FAILED | A required gate (validation, spec-match, branch discipline) failed |
+
+---
+
+## HITL Escalation
+
+When the loop fails to converge:
+
+1. Write a manifest entry with:
+   - `agent_type: "testing"`
+   - `key_findings: ["framework: <name>", "iterations: <MAX>", "converged: false", "remaining failures: <list>"]`
+   - `actionable: true`
+   - `needs_followup: ["HITL review required"]`
+2. Exit with code 65 (`HANDOFF_REQUIRED`).
+3. Surface the unmet spec requirements to the operator with concrete next steps.
+
+The operator picks up the loop manually, fixes the blocking failure, and re-launches the IVT loop. The skill MUST NOT silently exit, MUST NOT loosen the spec, and MUST NOT mark the task complete.
+
+---
+
+## Anti-Patterns
+
+| Pattern | Why Avoid |
+|---------|-----------|
+| Hardcoding vitest/jest/pytest | Violates the project-agnostic mandate (TEST-001) |
+| Running tests once and reporting result | Testing is a LOOP, not a one-shot (TEST-005) |
+| Skipping validation between iterations | Validation gates are part of IVT, not a pre-step (TEST-009 wraps both) |
+| Bailing silently on non-convergence | Must escalate to HITL with exit 65 (TEST-009) |
+| Marking task complete with `failed > 0` | TEST-004: 100% pass required for convergence |
+| Modifying main/master branch | TEST-008: branch discipline mandatory |
+| Ignoring spec-requirement traceability | TEST-003: every MUST needs a test |
+| Treating coverage as the convergence metric | Coverage is advisory; spec satisfaction is the gate |
+| Editing test expectations to match broken code | The fix goes in the implementation, not the test |
+
+---
+
+## Cross-Cutting: Contribution Protocol
+
+Multi-agent IVT loops MUST record contributor identity per iteration in the contribution protocol. See `protocols-markdown/contribution.md`.
+
+---
+
+## References
+
+- **Skill**: `packages/skills/skills/ct-ivt-looper/SKILL.md` (project-agnostic loop logic)
+- **Pure validator**: `packages/core/src/orchestration/protocol-validators.ts#validateTestingProtocol`
+- **Wrapper**: `packages/core/src/validation/protocols/testing.ts`
+- **Engine op**: `packages/cleo/src/dispatch/engines/validate-engine.ts#validateProtocolTesting`
+- **Dispatch case**: `packages/cleo/src/dispatch/domains/check.ts` (case `'testing'`)
+- **Per-project test command override**: `.cleo/project-context.json#testing.command`
+- **Implementation protocol** (the "I"): `protocols-markdown/implementation.md`
+- **Validation protocol** (the "V"): `protocols-markdown/validation.md`
+- **Release protocol** (downstream gate): `protocols-markdown/release.md`

package/src/validation/protocols/protocols-markdown/validation.md

@@ -0,0 +1,242 @@
+---
+id: VALID
+title: Validation Protocol
+version: 1.0.0
+status: active
+type: base
+audience: [llm-agent, orchestrator]
+tags: [validation, quality, compliance]
+skillRef: ct-validator
+lastUpdated: 2026-02-24
+enforcement: strict
+---
+
+# Validation Protocol
+
+**Provenance**: @task T3155, @epic T3147
+**Version**: 1.0.1
+**Type**: Conditional Protocol
+**Stage**: IVTR - V (Validation)
+**Max Active**: 3 protocols (including base)
+
+---
+
+## Trigger Conditions
+
+This protocol activates when the task involves:
+
+| Trigger | Keywords | Context |
+|---------|----------|---------|
+| Verification | "validate", "verify", "check", "audit" | Correctness checking |
+| Quality | "quality", "qa", "review" | Quality assurance |
+| Compliance | "compliance", "conform", "standard" | Standards adherence |
+| Smoke Test | "smoke", "sanity", "basic test" | Initial verification |
+
+**Explicit Override**: `--protocol validation` flag on task creation.
+
+**Lifecycle Position**: After Implementation (I), before Testing (T)
+
+---
+
+## Requirements (RFC 2119)
+
+### MUST
+
+| Requirement | Description |
+|-------------|-------------|
+| VALID-001 | MUST verify implementation matches specification |
+| VALID-002 | MUST run existing test suite and report results |
+| VALID-003 | MUST check protocol compliance via `lib/protocol-validation.sh` |
+| VALID-004 | MUST document pass/fail status for each validation check |
+| VALID-005 | MUST write validation summary to manifest |
+| VALID-006 | MUST set `agent_type: "validation"` in manifest |
+| VALID-007 | MUST block progression if critical validations fail |
+
+### SHOULD
+
+| Requirement | Description |
+|-------------|-------------|
+| VALID-010 | SHOULD verify edge cases identified in specification |
+| VALID-011 | SHOULD check for regressions in related functionality |
+| VALID-012 | SHOULD validate error handling paths |
+| VALID-013 | SHOULD measure against acceptance criteria |
+
+### MAY
+
+| Requirement | Description |
+|-------------|-------------|
+| VALID-020 | MAY perform performance validation |
+| VALID-021 | MAY verify security constraints |
+| VALID-022 | MAY suggest additional test cases |
+
+---
+
+## Validation Checklist
+
+### Code Validation
+
+```bash
+# 1. Syntax check
+bash -n scripts/*.sh lib/*.sh
+
+# 2. Protocol compliance
+source lib/protocol-validation.sh
+validate_implementation_protocol "$TASK_ID"
+
+# 3. Run existing tests
+bats tests/unit/*.bats
+bats tests/integration/*.bats
+```
+
+### Specification Compliance
+
+| Check | Command | Pass Criteria |
+|-------|---------|---------------|
+| Spec exists | `ls docs/specs/*.md` | File present |
+| RFC 2119 keywords | `grep -E "MUST|SHOULD|MAY"` | Keywords present |
+| Implementation matches | Manual review | All MUST satisfied |
+
+### Exit Code Validation
+
+| Exit Code | Meaning | Action |
+|-----------|---------|--------|
+| 0 | All validations pass | Proceed to Testing |
+| 62 | Specification mismatch | Fix implementation |
+| 64 | Implementation protocol violation | Fix provenance |
+| 67 | Generic protocol violation | Review and fix |
+
+---
+
+## Output Format
+
+### Validation Report
+
+```markdown
+# Validation Report: T####
+
+**Date**: YYYY-MM-DD
+**Validator**: agent-id
+**Task**: T####
+**Epic**: T####
+
+## Summary
+
+- **Status**: PASS | FAIL | PARTIAL
+- **Checks Passed**: X/Y
+- **Critical Issues**: N
+
+## Detailed Results
+
+| Check | Result | Notes |
+|-------|--------|-------|
+| Syntax | PASS | No errors |
+| Tests | PASS | 48/48 pass |
+| Protocol | PASS | Compliance 95% |
+
+## Issues Found
+
+1. [Issue description]
+2. [Issue description]
+
+## Recommendations
+
+1. [Action item]
+```
+
+### Manifest Entry
+
+@skills/_shared/manifest-operations.md
+
+Use `cleo research add` to create the manifest entry:
+
+```bash
+cleo research add \
+  --title "Validation: [Feature Name]" \
+  --file "T####-validation-report.md" \
+  --topics "validation,qa" \
+  --findings "RESULT: X/Y checks passed,ISSUES: N found,STATUS: PASS|FAIL" \
+  --status complete \
+  --task T#### \
+  --actionable \
+  --agent-type validation
+```
+
+---
+
+## Integration Points
+
+### With Implementation Protocol
+
+```
+Implementation (I) ──► Validation (V)
+                           │
+                           ├── Run tests
+                           ├── Check compliance
+                           └── Verify spec match
+```
+
+### With Testing Protocol
+
+```
+Validation (V) ──► Testing (T)
+       │                │
+       │                ├── Write new tests
+       │                └── Full coverage
+       │
+       └── Identifies gaps for Testing to fill
+```
+
+### With lib/protocol-validation.sh
+
+The validation protocol uses these functions:
+
+| Function | Purpose | Exit Code |
+|----------|---------|-----------|
+| `validate_implementation_protocol()` | Check IMPL-* compliance | 64 |
+| `validate_specification_protocol()` | Check SPEC-* compliance | 62 |
+| `validate_protocol()` | Generic protocol check | 67 |
+
+---
+
+## Enforcement
+
+### Via lib/protocol-validation.sh
+
+```bash
+source lib/protocol-validation.sh
+
+# Validate a specific protocol
+validate_implementation_protocol "$TASK_ID" "$MANIFEST_ENTRY" "true"
+
+# Generic validation
+validate_protocol "implementation" "$TASK_ID" "$DATA"
+```
+
+### Exit Codes
+
+| Code | Constant | Description |
+|------|----------|-------------|
+| 62 | EXIT_PROTOCOL_SPECIFICATION | Spec validation failed |
+| 64 | EXIT_PROTOCOL_IMPLEMENTATION | Implementation validation failed |
+| 67 | EXIT_PROTOCOL_GENERIC | Generic validation failed |
+| 68 | EXIT_VALIDATION_INCOMPLETE | Validation not finished |
+
+---
+
+## Cross-Cutting: Contribution Protocol
+
+When validation involves multi-agent work:
+
+1. **Record validator identity** in manifest
+2. **Attribute findings** to validating agent
+3. **Track validation consensus** if multiple validators
+
+See: `src/protocols/contribution.md` for attribution requirements.
+
+---
+
+## References
+
+- **Specification**: `docs/specs/PROTOCOL-ENFORCEMENT-SPEC.md` *(ARCHIVED — modernization tracked in T5492)*
+- **Implementation**: `lib/protocol-validation.sh`
+- **Tests**: `tests/unit/protocol-validation.bats`

package/src/validation/protocols/provenance.ts

@@ -0,0 +1,50 @@
+/**
+ * Provenance protocol — thin wrapper delegating to the canonical pure validator.
+ *
+ * Provenance is a cross-cutting protocol that composes with release via
+ * artifact-publish: artifact-publish delegates signing, SBOM generation, and
+ * in-toto attestation chain assembly to provenance (see provenance.md Pipeline
+ * Integration and release-engine.ts releaseShip()). The CI pipeline already
+ * uses `npm publish --provenance` for SLSA L3 keyless attestation via OIDC.
+ *
+ * @task T260 — create the missing provenance protocol wrapper
+ */
+
+import {
+  type ProtocolValidationResult,
+  validateProvenanceProtocol,
+} from '../../orchestration/protocol-validators.js';
+import {
+  loadManifestEntryByTaskId,
+  loadManifestEntryFromFile,
+  throwIfStrictFailed,
+} from './_shared.js';
+
+interface ProvenanceOpts {
+  strict?: boolean;
+  hasAttestation?: boolean;
+  hasSbom?: boolean;
+}
+
+/** Validate provenance protocol for a task. */
+export async function validateProvenanceTask(
+  taskId: string,
+  opts: ProvenanceOpts,
+): Promise<ProtocolValidationResult> {
+  const entry = loadManifestEntryByTaskId(taskId);
+  const result = validateProvenanceProtocol(entry, opts);
+  throwIfStrictFailed(result, opts, 'provenance', taskId);
+  return result;
+}
+
+/** Validate provenance protocol from a manifest file. */
+export async function checkProvenanceManifest(
+  manifestFile: string,
+  opts: ProvenanceOpts,
+): Promise<ProtocolValidationResult> {
+  const entry = loadManifestEntryFromFile(manifestFile);
+  const taskId = entry.linked_tasks?.[0] ?? 'UNKNOWN';
+  const result = validateProvenanceProtocol(entry, opts);
+  throwIfStrictFailed(result, opts, 'provenance', taskId);
+  return result;
+}

package/src/validation/protocols/release.ts

@@ -0,0 +1,44 @@
+/**
+ * Release protocol — thin wrapper delegating to the canonical pure validator.
+ *
+ * Renamed from release-protocol.ts in T260. The release protocol orchestrates
+ * version bumping, changelog generation, tagging, and pushing. It composes
+ * with artifact-publish and provenance at the release stage (see release.md
+ * Pipeline Integration + release-engine.ts releaseShip()).
+ *
+ * @task T4804
+ * @task T260 — drop -protocol suffix, delegate to orchestration validator
+ */
+
+import {
+  type ProtocolValidationResult,
+  validateReleaseProtocol,
+} from '../../orchestration/protocol-validators.js';
+import {
+  loadManifestEntryByTaskId,
+  loadManifestEntryFromFile,
+  throwIfStrictFailed,
+} from './_shared.js';
+
+/** Validate release protocol for a task. */
+export async function validateReleaseTask(
+  taskId: string,
+  opts: { strict?: boolean; version?: string; hasChangelog?: boolean },
+): Promise<ProtocolValidationResult> {
+  const entry = loadManifestEntryByTaskId(taskId);
+  const result = validateReleaseProtocol(entry, opts);
+  throwIfStrictFailed(result, opts, 'release', taskId);
+  return result;
+}
+
+/** Validate release protocol from a manifest file. */
+export async function checkReleaseManifest(
+  manifestFile: string,
+  opts: { strict?: boolean; version?: string; hasChangelog?: boolean },
+): Promise<ProtocolValidationResult> {
+  const entry = loadManifestEntryFromFile(manifestFile);
+  const taskId = entry.linked_tasks?.[0] ?? 'UNKNOWN';
+  const result = validateReleaseProtocol(entry, opts);
+  throwIfStrictFailed(result, opts, 'release', taskId);
+  return result;
+}