yam-flow 2.8.4 → 2.8.5

package/references/test-check.md

@@ -0,0 +1,192 @@
+# Test check completion gate
+
+Test check validates changed work before a completion claim, even without a commit
+or push request. It is separate from report formatting and Security check. It does
+not certify every possible input or prove a live deployment from a fixture.
+
+## Contract
+
+1. At work start, identify the exact baseline commit and expected behavior. Review
+   the project-owned policy before executing its commands. Commands run as the
+   caller without a shell; they are authorized project code, not a security sandbox.
+2. Configure `testing/policy.json` (shared) or `.yam/testing/policy.json` (private).
+   `yam template test-check` prints a starter that cannot pass without an adapter.
+3. Each changed source file must match a surface's exact path or `directory/**`.
+   Declare risk, targets and required scenarios. Built-in risk floors cannot be
+   reduced by deleting checks. API/DB/auth/UI require integration or E2E evidence.
+4. At implementation end, `plan` checks impact and required scenarios; `run` executes
+   adapters and records exit codes/output hashes. Empty changes, zero tests, zero
+   assertions, skips, failing results and missing observations block. One identical
+   command executes once per run, even when it reports several scenarios.
+5. `check` revalidates the policy, receipt, note, entire source snapshot, environment
+   identity and required scenario inventory. A relevant edit invalidates the result.
+6. Completion requires PASS. An unavailable environment is BLOCKED, with a truthful
+   incomplete handoff permitted; it must not be described as completed verification.
+
+Risk floors:
+
+| Risk | Required scenarios |
+| --- | --- |
+| docs | inspection |
+| logic | success, invalid, boundary |
+| api | success, invalid, contract |
+| db | success, denied, invalid, readback, rollback, migration_upgrade, migration_fresh |
+| auth | success, denied, cross_scope, revoked |
+| ui | success, error |
+| release | artifact, integrity |
+
+Path heuristics are conservative supplementary detection, not a complete dependency
+or semantic graph. SQL functions/procedures/policies/triggers found in the old and
+new SQL file must have explicit targets, e.g. `function:public.receiving`. Quoted
+identifiers and dynamic SQL currently block for manual review; overloaded signatures,
+indirect effects and application-specific dependencies require explicit project tests.
+Do not label a function passing merely because a broad file-level test passed.
+
+## Commands and activation
+
+```bash
+mkdir -p testing
+yam template test-check > testing/policy.json
+# Edit policy and implement the referenced adapter; capture BASE before editing.
+yam test-check plan --base "$BASE" --json
+yam test-check run --base "$BASE" --json
+yam test-check check --json
+```
+
+`BASE` must be an exact existing commit ID. A base equal to the candidate with no
+changes cannot authorize completion. Generated/private `.yam`, dist, build, coverage,
+node_modules, host agent configs and tarball/log files are excluded from source
+snapshots; runtime dependency/configuration identities belong in environment evidence.
+Source symlinks/submodules are unsupported and block, rather than silently omitted.
+
+For a configured project, the existing opt-in `study-note` prompt/Stop hook also
+runs Test check validation. Enable through `yam hook enable study-note --project .`
+(or use an existing global study-note hook). Prompt captures changed-turn scope;
+Stop checks the receipt only, never launches long tests. Unlike the report-format
+one-correction rule, Test check remains blocked on repeated Stop calls. A response
+containing the marker `Test check: BLOCKED` on its own status line may finish as an incomplete-status
+handoff. This marker grants no machine gate approval. No-change/read-only turns
+skip the completion check. Without a configured policy/hook the route instruction
+is guidance, not host enforcement; missing host discovery must be reported.
+
+Before commit use `check --stage commit`: the index must match the tested working
+snapshot. Before push use `check --stage push --source "$SOURCE"` for every outgoing
+commit; SOURCE must be an exact commit ID and match the tested snapshot. `check --base <expected starting commit>` rejects a receipt from a different baseline;
+Security check supplies the actual CI base automatically. Stage ci
+or deploy additionally requires an administrator-reviewed `--policy-digest`.
+Missing CI/deploy pins are rejected before any project test command executes.
+`run` supports the same stage/source requirements. Deploy additionally requires
+`--artifact <repository-local-file> --configuration <repository-local-file>`
+and a deployment-stage receipt. Both files are measured before/after the tests
+and remeasured on check; completion-only evidence cannot approve deployment.
+Adapters receive YAM_TEST_ARTIFACT / YAM_TEST_CONFIGURATION and must test those
+exact inputs rather than create a substitute package. Run stage checks before operations,
+not concurrently with index/source changes. Existing Security check invokes this
+validation automatically when a default project policy exists; CI/deploy callers
+must supply `yam security check --test-policy-digest "$PIN"`. Existing custom project
+runners must generate the Test check receipt first. Do not overwrite other hooks.
+
+Pins bind reviewed bytes; they do not independently approve policy authorship.
+Policy/scenario removal and changed test adapters require renewed engineering review
+and CI-protected policy approval. CI must independently run the reviewed tests;
+local hooks/Stop hooks can be bypassed or disabled. This feature does not configure
+GitHub protection or change installed global skills/hooks by package installation.
+
+## Adapter protocol
+
+A command emits one JSON document on stdout (logs can go to stderr). Each selected
+check must appear exactly once. Reports are data and are never executed.
+
+```json
+{
+  "schema": "yam.test-observation.v1",
+  "tests": [{
+    "id": "save-success",
+    "target": "function:public.receiving",
+    "scenario": "success",
+    "evidence": "integration",
+    "status": "passed",
+    "tests": 1,
+    "assertions": 4,
+    "failed": 0,
+    "skipped": 0,
+    "mocked": false,
+    "environment": {}
+  }]
+}
+```
+
+The empty environment above is illustrative and cannot pass DB validation. DB
+checks require `environment` to point to a repository-local expected manifest,
+preferably ignored `.yam/testing/environment.json`, containing:
+
+- schema `yam.test-environment.v1`, engine and exact engine_version;
+- source_digest from `yam test-check plan`;
+- configuration_digest, roles_digest and fixtures_digest (`sha256:...`);
+- migrations: the complete sorted candidate `migrations/` inventory, each with
+  file and SHA256. Existing baseline migrations must also be included.
+
+The adapter queries the *actual* isolated environment and returns the same structured
+fields only after comparing applied migration order/content, runtime/roles/settings
+and synthetic fixture identity. Returning the expected file without measuring the
+DB is not valid evidence. yam compares reports and candidate manifests; adapter
+assertions and environment truth are still trusted project code, not independently
+attested by the generic runner. Misreported `mocked:false` cannot be detected by a
+JSON validator. Review and protect the adapters as rigorously as the application.
+
+Use actual DB operations with synthetic records and representative roles. Success
+means saved headers/items and separate readback match, not just HTTP 200. Verify
+failure rollback, duplicate/retry behavior, revoked access and relevant triggers.
+Do not use owner/service credentials to stand in for ordinary-user authorization.
+Test in disposable environments. Transaction rollback does not undo email, external
+files, webhooks or all sequence effects; verify external isolation and cleanup.
+
+## Records, safety and limits
+
+Each run appends an explained `-- Test check --` block to private
+`.yam/testing/notes.md`, writes an immutable JSON under `runs/` and updates
+`latest.json`. Note/receipt write failure blocks; concurrent persistence uses an
+exclusive lock, rejects links/hardlinks and never removes someone else's lock.
+Raw stdout/stderr, command arguments and environment values are not persisted;
+only allowlisted facts and hashes are recorded. Do not put secrets in explanations
+or command arguments. Use the project's existing least-privilege test environment.
+
+A checksum is not a signature against a hostile same-user process. Timeout marks a
+command failed; descendant-process cleanup is not attested by spawnSync. Native
+JUnit/Playwright/Postgres result parsers, property/stateful/combination testing and
+mutation testing are project adapters or future extensions, not built-in scanners.
+The regression suite uses real Git/process/filesystem fixtures and simulated DB
+observations; it does not claim to have tested any production RO database.
+
+Implementation boundaries: existing verification-closure remains an immutable
+operator-declared command receipt. Test check adds actual process execution and
+scenario checks without promoting those older receipts into runtime proof.
+
+## Hosted rollout
+
+The yam repository workflow runs Test check before Security check and uploads its
+sanitized notes/runs. Activation after review requires `YAM_TEST_POLICY_SHA256`
+alongside the existing reviewed source/security-policy pins. This change does not
+set repository variables, update global hooks, commit, push or publish. An absent
+or outdated pin intentionally blocks hosted approval rather than silently using
+candidate-computed trust. Local project source-review manifests must be refreshed
+only after the executable changes have been reviewed.
+
+## Optional real PostgreSQL simulation
+
+From a source checkout, `npm run test-check-postgres-smoke` requires Docker access and an existing local
+`postgres:17-alpine` image (override with YAM_TEST_PG_IMAGE). It never pulls an
+image, publishes a port, mounts host data, or connects to a production database.
+The disposable database reproduces SQLSTATE 42702 after successful function DDL,
+then verifies fixed normal save/readback, access denial/revocation, invalid input,
+rollback, retry, fresh/upgrade migrations, out-of-band function drift, and a
+16-case role/brand/quantity matrix. The
+adapter queries actual server/function/ACL/migration state rather than echoing an
+expected manifest. Cleanup must confirm the owned container is absent.
+This is real PostgreSQL with synthetic application data, not verification of any
+production application's SQL or authorization rules. It stays opt-in, outside
+ordinary verification so a missing Docker daemon never silently becomes a mock.
+
+Invalid stage/base values are rejected before persistence. Malformed run options
+leave a sanitized failed note. Git discovery failure does not disable a previously
+configured Stop gate; quoted examples of the handoff marker cannot approve it.

package/skills/deep/SKILL.md

@@ -74,6 +74,10 @@ Stop condition:
 - Stop earlier when the same blocker repeats, a required tool/auth/runtime is unavailable, or a user decision is needed.
 - Do not keep retrying the same failure; classify the cause, give a recovery hint, and hand off.
 
+## Test check before completion
+
+Before claiming changed work complete, use `references/test-check.md`. With a configured project policy run `yam test-check run --base <exact starting commit> --json` and `yam test-check check --json`. Require mapped scenarios, actual project-runner evidence, matching source/environment and a persisted Test check note. Required missing/failed/skipped checks block approval; give an honest `Test check: BLOCKED` handoff when unable to verify. Recheck exact staged/outgoing/deployment targets before their operations. Do not substitute mock DB results, report validation or old operator-declared receipts. If project adapters/policy are missing, report the activation gap instead of claiming gated completion. Read-only advisory work needs no execution gate; tiny changes use fitting inspection checks.
+
 ## Commit, push and deployment security gate
 
 Before performing a requested commit, push or deployment, follow `references/security-check.md`: validate stage-bound evidence with `yam security check`, require a successful persisted Security check note, and stop the operation when the gate blocks. If the project is not configured, report that blocker rather than claim security approval. Do not overwrite existing hooks or treat supplied evidence as an automatic vulnerability scan.

package/skills/mission/SKILL.md

@@ -121,6 +121,10 @@ Good mission prompts include:
 
 If the user invokes `$mission` without an approved plan, ask for the plan or propose a compact plan first.
 
+## Test check before completion
+
+Before claiming changed work complete, use `references/test-check.md`. With a configured project policy run `yam test-check run --base <exact starting commit> --json` and `yam test-check check --json`. Require mapped scenarios, actual project-runner evidence, matching source/environment and a persisted Test check note. Required missing/failed/skipped checks block approval; give an honest `Test check: BLOCKED` handoff when unable to verify. Recheck exact staged/outgoing/deployment targets before their operations. Do not substitute mock DB results, report validation or old operator-declared receipts. If project adapters/policy are missing, report the activation gap instead of claiming gated completion. Read-only advisory work needs no execution gate; tiny changes use fitting inspection checks.
+
 ## Commit, push and deployment security gate
 
 Before performing a requested commit, push or deployment, follow `references/security-check.md`: validate stage-bound evidence with `yam security check`, require a successful persisted Security check note, and stop the operation when the gate blocks. If the project is not configured, report that blocker rather than claim security approval. Do not overwrite existing hooks or treat supplied evidence as an automatic vulnerability scan.

package/skills/quick/SKILL.md

@@ -75,6 +75,10 @@ Use `references/study-note.md` when any code, config, docs, or project artifact
 Use `references/final-report.md` to close with remaining tasks and fix-first items when useful.
 Use `references/token-budget-reporter.md` when a run needs measured budget feedback.
 
+## Test check before completion
+
+Before claiming changed work complete, use `references/test-check.md`. With a configured project policy run `yam test-check run --base <exact starting commit> --json` and `yam test-check check --json`. Require mapped scenarios, actual project-runner evidence, matching source/environment and a persisted Test check note. Required missing/failed/skipped checks block approval; give an honest `Test check: BLOCKED` handoff when unable to verify. Recheck exact staged/outgoing/deployment targets before their operations. Do not substitute mock DB results, report validation or old operator-declared receipts. If project adapters/policy are missing, report the activation gap instead of claiming gated completion. Read-only advisory work needs no execution gate; tiny changes use fitting inspection checks.
+
 ## Commit, push and deployment security gate
 
 Before performing a requested commit, push or deployment, follow `references/security-check.md`: validate stage-bound evidence with `yam security check`, require a successful persisted Security check note, and stop the operation when the gate blocks. If the project is not configured, report that blocker rather than claim security approval. Do not overwrite existing hooks or treat supplied evidence as an automatic vulnerability scan.

package/skills/ueye/SKILL.md

@@ -183,6 +183,10 @@ Always consider:
 - Interaction affordance.
 - Consistency with the project's visual language.
 
+## Test check before completion
+
+Before claiming changed work complete, use `references/test-check.md`. With a configured project policy run `yam test-check run --base <exact starting commit> --json` and `yam test-check check --json`. Require mapped scenarios, actual project-runner evidence, matching source/environment and a persisted Test check note. Required missing/failed/skipped checks block approval; give an honest `Test check: BLOCKED` handoff when unable to verify. Recheck exact staged/outgoing/deployment targets before their operations. Do not substitute mock DB results, report validation or old operator-declared receipts. If project adapters/policy are missing, report the activation gap instead of claiming gated completion. Read-only advisory work needs no execution gate; tiny changes use fitting inspection checks.
+
 ## Commit, push and deployment security gate
 
 Before performing a requested commit, push or deployment, follow `references/security-check.md`: validate stage-bound evidence with `yam security check`, require a successful persisted Security check note, and stop the operation when the gate blocks. If the project is not configured, report that blocker rather than claim security approval. Do not overwrite existing hooks or treat supplied evidence as an automatic vulnerability scan.

package/src/bin/yam.ts

@@ -1,4 +1,5 @@
 #!/usr/bin/env node
+import { runTestCheckCommand, testCheckStop, testCheckConfigured } from '../lib/test-check.js';
 import { spawnSync } from 'node:child_process';
 import { studyNoteTurnScope } from '../lib/study-note-scope.js';
 import { selectHookNode, probeHookDiscovery, inspectHookObservation, recordHookObservation } from '../lib/hook-readiness.js';
@@ -228,6 +229,7 @@ Usage:
   yam proof [dir|--from file] [--route route] [--truth status] [--command text] [--evidence text]
   yam proof write [dir] [--format json|md] [--out file] [--route route] [--truth status] [--command text]
   yam study-note check [dir] [--report file|--text text] [--json]
+  yam test-check <plan|run|check|status> [--base commit] [--json]
   yam security <init|check|hooks|status> [--stage commit|push|ci|deploy] [--json]
   yam next-step <report|verify> [--spec file|--receipt file] [--json]
   yam loop report [--route route] [--intent text] [--stage id:status:note] [--evidence text] [--json]
@@ -254,7 +256,7 @@ Usage:
   yam safety [text...]
   yam memory <init|add|list|summary|resolve> [dir] [options]
   yam hook <status|enable|disable|run> [lite|study-note] [--global|--project dir]
-  yam template <project|ueye|mission|proof|tuning|security|security-ci>
+  yam template <project|ueye|mission|proof|tuning|security|security-ci|test-check>
   yam tune-log [dir]
   yam install [--dry-run] [--json] [--replace-user-skill name]
   yam uninstall
@@ -576,6 +578,7 @@ async function verify({ quiet = false } = {}) {
     'markdown-management.md',
     'study-note.md',
     'security-check.md',
+    'test-check.md',
     'final-report.md',
     'token-budget-reporter.md',
     'memory.md'
@@ -591,7 +594,7 @@ async function verify({ quiet = false } = {}) {
       if (!hasHeading(projectTemplate, section)) issues.push(`project template missing section: ${section}`);
     }
   }
-  for (const template of ['ueye-review.md', 'ueye-comparison.md', 'mission-plan.md', 'runtime-proof.md', 'tuning-log.md', 'security-check-note.md', 'security-gate.yml']) {
+  for (const template of ['ueye-review.md', 'ueye-comparison.md', 'mission-plan.md', 'runtime-proof.md', 'tuning-log.md', 'security-check-note.md', 'security-gate.yml', 'test-check-policy.json']) {
     if (!await exists(path.join(ROOT, 'templates', template))) issues.push(`missing template: ${template}`);
   }
   for (const module of ['trust-kernel.js', 'ueye-artifacts.js']) {
@@ -2524,6 +2527,8 @@ async function buildYamLiteContext({ cwd, prompt }) {
 async function buildStudyNoteStopOutput(input: AnyRecord = {}, cwd = process.cwd()) {
   const lastAssistantMessage = String(input?.last_assistant_message || input?.lastAssistantMessage || '');
   const scope = studyNoteTurnScope(input, cwd);
+  const testGate = testCheckStop(cwd, !scope.available || scope.files.length > 0, lastAssistantMessage);
+  if (testGate) return testGate;
   const result = await buildStudyNoteGuardResult(path.resolve(cwd), lastAssistantMessage, 'stop_hook_last_assistant_message', scope);
   if (result.truth_status === 'partial' && !result.changed_file_detection.available) {
     return {
@@ -2554,8 +2559,9 @@ async function buildStudyNoteStopOutput(input: AnyRecord = {}, cwd = process.cwd
 
 async function buildStudyNoteTurnContext(input: AnyRecord, cwd: string) {
   const scope = studyNoteTurnScope(input, cwd, true);
+  const testReminder = testCheckConfigured(cwd) ? ' Test check active: before completion run yam test-check run --base <exact starting commit> --json and yam test-check check --json. Missing required tests/evidence block completion; report Test check: BLOCKED if unable to verify. This is separate from report formatting and Security check.' : '';
   if (!scope.available) return 'yam Study Note: turn scope is unavailable; do not treat this as a clean project. Inspect changed artifacts manually; no strict block will be applied without a baseline.';
-  return `yam Study Note turn baseline saved (hash-only). Mode: ${scope.mode}. Only changes since this prompt count; known generated logs, mission/security receipts and screenshots are excluded unless explicitly included. Include a concise Study Note followed by Next step when product artifacts change. ${scope.mode === 'strict' ? 'Risk work uses the detailed report contract and one correction pass.' : 'Ordinary work is advisory: a short explanation, verification/limits and remaining work (or none) suffice.'}`;
+  return `yam Study Note turn baseline saved (hash-only). Mode: ${scope.mode}. Only changes since this prompt count; known generated logs, mission/security receipts and screenshots are excluded unless explicitly included. Include a concise Study Note followed by Next step when product artifacts change. ${scope.mode === 'strict' ? 'Risk work uses the detailed report contract and one correction pass.' : 'Ordinary work is advisory: a short explanation, verification/limits and remaining work (or none) suffice.'}${testReminder}`;
 }
 
 function yamLiteRouteHint(prompt = '') {
@@ -6149,11 +6155,12 @@ async function printTemplate(name = '') {
     runtime: 'runtime-proof.md',
     tuning: 'tuning-log.md',
     security: 'security-check-note.md',
-    'security-ci': 'security-gate.yml'
+    'security-ci': 'security-gate.yml',
+    'test-check': 'test-check-policy.json'
   };
   const file = map[key];
   if (!file) {
-    console.error('usage: yam template <project|ueye|ueye-comparison|mission|proof|tuning|security|security-ci>');
+    console.error('usage: yam template <project|ueye|ueye-comparison|mission|proof|tuning|security|security-ci|test-check>');
     process.exitCode = 1;
     return;
   }
@@ -6456,6 +6463,13 @@ function showRequestedHelp(args: string[]) {
 async function main() {
   const args = process.argv.slice(2);
   const command = args[0] || 'help';
+  if (command === 'test-check') {
+    if (!args[1] || ['help', '--help', '-h'].includes(args[1])) {
+      console.log('yam test-check run --base <exact commit> [--policy file --policy-digest sha256] [--json]\nyam test-check check [--stage complete|commit|push|ci|deploy --source <exact commit> --policy-digest sha256 --receipt file] [--json]\nyam test-check status\nyam test-check plan --base <exact commit> --json\nSee references/test-check.md for required scenarios, adapters and activation.'); return;
+    }
+    const result = runTestCheckCommand(args.slice(1));
+    console.log(JSON.stringify(result, null, 2)); if (!result.ok) process.exitCode = 1; return;
+  }
   if (showRequestedHelp(args)) return;
   if (command === 'help') return usage();
   if (command === 'security') {

package/src/lib/security-integration.ts

@@ -1,3 +1,4 @@
+import { checkTestCheck, testCheckConfigured } from './test-check.js';
 import { createHash, randomUUID } from 'node:crypto';
 import { execFileSync } from 'node:child_process';
 import fs from 'node:fs/promises';
@@ -27,7 +28,7 @@ function parseArgs(args: string[]): Map<string, string> {
   const out = new Map<string, string>();
   for (let i = 0; i < args.length; i++) {
     if (args[i] === '--json') continue;
-    if (!['--policy', '--policy-digest', '--evidence', '--note', '--stage', '--remote', '--destination', '--source', '--base', '--artifact', '--artifact-digest', '--configuration', '--configuration-digest', '--environment', '--cli'].includes(args[i]) || !args[i + 1] || args[i + 1].startsWith('--') || out.has(args[i])) throw new Error('invalid or duplicate security option');
+    if (!['--policy', '--policy-digest', '--evidence', '--note', '--stage', '--remote', '--destination', '--source', '--base', '--artifact', '--artifact-digest', '--configuration', '--configuration-digest', '--environment', '--cli', '--test-policy-digest'].includes(args[i]) || !args[i + 1] || args[i + 1].startsWith('--') || out.has(args[i])) throw new Error('invalid or duplicate security option');
     out.set(args[i], args[++i]);
   }
   return out;
@@ -262,6 +263,13 @@ async function check(cwd: string, options: Map<string, string>, settings: Securi
     result = validateSecurityCheck(input, binding as any);
     gateErrors.push('security_input_or_binding_unavailable');
   }
+  if (testCheckConfigured(cwd) || config.test_policy_digest) {
+    const pin = options.get('--test-policy-digest') || config.test_policy_digest;
+    const sources = stage === 'push' ? (binding.push_refs as any[] || []).map(ref => ref.local_oid)
+      : [binding.source_revision as string | undefined];
+    const targets = sources.length ? sources : [undefined];
+    if (targets.some(source => !checkTestCheck(cwd, { stage, source, pin, base: stage === 'ci' ? binding.base_revision as string : undefined, artifact: options.get('--artifact'), configuration: options.get('--configuration') }).ok)) gateErrors.push('test_check_blocked');
+  }
   if (gateErrors.length) {
     result.result = 'BLOCKED';
     result.errors = [...new Set([...result.errors, ...gateErrors])];