forgedock 1.0.17 → 1.0.19

package/commands/quality-gate.md

@@ -47,6 +47,7 @@ This quality gate uses **domain detection** to adapt to your project's actual te
 | IMPORT_RESOLUTION | Python files with new `from app.*` imports added outside `try:` blocks |
 | INFRA | Dockerfiles or entrypoints with runtime UID changes |
 | ROUTER_BUG | Router Python files where gate conditions are narrowed |
+| FORGE_GRAPH | `commands/*.md` specs or `scripts/*` files change (ForgeDock self-consistency) |
 
 **Stack-agnostic coverage**: The example domains above reflect common patterns caught in production; they are not requirements. A Go or Ruby project benefits from SECURITY checks. A Node.js project benefits from FRONTEND, PROXY, and DEPLOY checks. Any project benefits from DATABASE and WORKFLOW checks when those file types are present. The stack-specific examples (Python routers, SOPS secrets chain, FastAPI layouts, appleboy SSH deploys) are illustrative — the domain detection system applies the applicable subset to any codebase.
 
@@ -98,6 +99,7 @@ Before running checks, classify the changed files into domains. This avoids runn
 | `*.py` with new `from app.` import statements added outside `try:` blocks | IMPORT_RESOLUTION |
 | `Dockerfile*` or `entrypoint*.sh` with added/changed `USER`, `su-exec`, `gosu`, or `setuid` | INFRA |
 | `*.py` in `routers/` where the diff removes a line containing an or-gate condition (lines starting with `-` containing `if.*or`) | ROUTER_BUG |
+| `commands/*.md` or `scripts/*` files (ForgeDock repo only — dogfoods its own spec graph) | FORGE_GRAPH |
 
 **Apply the classification:**
 
@@ -125,6 +127,9 @@ for f in {CHANGED_FILES}; do
             echo "$f" | grep -qE '\.py$' && DOMAINS="$DOMAINS STRING_SETS"
             ;;
     esac
+    case "$f" in
+        commands/*.md|scripts/*) DOMAINS="$DOMAINS FORGE_GRAPH" ;;
+    esac
     case "$f" in
         .github/workflows/*.yml) DOMAINS="$DOMAINS WORKFLOW" ;;
     esac
@@ -412,6 +417,58 @@ done
 
 Report any hit as **HIGH** — the DB connection will fail in any environment where PostgreSQL is not on `127.0.0.1:5432`.
 
+### 2G.6: Spec graph self-consistency (FORGE_GRAPH domain)
+
+**Triggered when**: changed files include `commands/*.md` specs or `scripts/*` files (ForgeDock repo only — it dogfoods its own Spec Knowledge Graph). Skips silently in any project without `scripts/validate-spec-graph.sh`.
+
+**Why this matters**: ForgeDock's pipeline behavior is encoded in a ~1.2 MB prose spec corpus where drift is invisible — a `FORGE:*` annotation written but never read, a `Skill(skill="X")` referencing a sub-phase that no longer exists, or a `workflow:*` label set by no command. The Spec Knowledge Graph (`build-spec-graph.mjs`) makes this queryable; `validate-spec-graph.sh` consumes it and partitions findings into HARD (dangling cross-refs, broken label transitions) and SOFT (orphan annotations).
+
+```bash
+# Run only when commands/*.md or scripts/* changed and the validator ships in this repo.
+VALIDATOR="$FORGEDOCK_SCRIPTS/validate-spec-graph.sh"
+[ -f "$VALIDATOR" ] || VALIDATOR="{WORKTREE_PATH}/scripts/validate-spec-graph.sh"
+if [ -f "$VALIDATOR" ]; then
+    # The graph is auto-built from the worktree (gitignored JSON not required).
+    # --soft keeps the gate non-blocking on the documented baseline orphans;
+    # HARD findings are still reported below and surfaced as gate findings.
+    GRAPH_REPORT=$(bash "$VALIDATOR" --root "{WORKTREE_PATH}" --soft 2>&1 || true)
+    echo "$GRAPH_REPORT"
+    # Map any HARD finding to a quality-gate finding (dangling ref / broken transition).
+    echo "$GRAPH_REPORT" | grep -E '^\s*\[HARD\]' | while IFS= read -r hit; do
+        echo "FORGE_GRAPH | HIGH | spec-graph | ${hit#*] }"
+    done
+fi
+```
+
+Report each `[HARD]` line as **HIGH** — a dangling `Skill()` target or a label set by no command is a real pipeline break (an agent will invoke a phase that does not exist, or a state will never be entered/exited). Orphan-annotation `[SOFT]` lines are advisory: surface them in the report for periodic triage but do NOT block the commit. Scaffolding `[INFO]` lines are expected and need no action. To gate on HARD findings (fail the build), drop `--soft`; the default keeps existing baseline orphans from blocking until they are triaged. <!-- Added: forge#869 -->
+
+### 2G.7: Native command conflict check (FORGE_GRAPH domain)
+
+**Triggered when**: changed files include `commands/*.md` files (top-level, not sub-phase subdirectories). Skips silently in any project without `scripts/check-native-conflicts.sh`.
+
+**Why this matters**: ForgeDock commands are installed into `~/.claude/commands/`, the same namespace as native Claude Code built-in slash commands. A top-level command file whose basename matches a native command name (e.g. `resume.md` → `/resume`, `status.md` → `/status`) completely shadows the native command — users lose access to core Claude Code functionality with no error message. This check prevents the bug class from ever reaching `staging` or `npm publish`. <!-- Added: forge#1074 -->
+
+```bash
+# Run only when top-level commands/*.md files changed and the script ships in this repo.
+# Sub-phase files in subdirectories (commands/work-on/*.md) are installed with their
+# relative path preserved and resolve to namespaced commands — they cannot shadow
+# root-level native commands and are not scanned.
+CONFLICT_CHECKER="$FORGEDOCK_SCRIPTS/check-native-conflicts.sh"
+[ -f "$CONFLICT_CHECKER" ] || CONFLICT_CHECKER="{WORKTREE_PATH}/scripts/check-native-conflicts.sh"
+if [ -f "$CONFLICT_CHECKER" ]; then
+    CONFLICT_REPORT=$(bash "$CONFLICT_CHECKER" "{WORKTREE_PATH}/commands" 2>&1) || CONFLICT_EXIT=$?
+    echo "$CONFLICT_REPORT"
+    if [ "${CONFLICT_EXIT:-0}" -ne 0 ]; then
+        # Extract each conflicting file from the report and emit a gate finding.
+        echo "$CONFLICT_REPORT" | grep -E '^\s+commands/' | while IFS= read -r hit; do
+            echo "FORGE_GRAPH | HIGH | native-conflict | ${hit#*  }"
+        done
+    fi
+fi
+```
+
+Report each conflict as **HIGH** — a shadowed native command is a silent user-facing regression (the native command becomes unreachable without warning). The builder must rename the conflicting file before the commit. Blocklist updates are in `scripts/check-native-conflicts.sh` (NATIVE_COMMANDS array).
+
 ### 2H: Asyncio cancellation safety (Python async code)
 
 **Triggered when**: changed Python files contain `asyncio.shield`, `asyncio.wait_for`, or `Task.cancel`.

package/commands/review-pr-staging.md

@@ -26,9 +26,13 @@ GH_REPO=$(yq '.project.owner + "/" + .project.repo' "$CONFIG_FILE")
 GH_FLAG="-R $GH_REPO"
 DEFAULT_BRANCH=$(yq '.branches.default' "$CONFIG_FILE")
 STAGING_BRANCH=$(yq '.branches.staging' "$CONFIG_FILE")
+
+# Test-gate config (Phase 6.5) — read here so vars are available before first use
+GATE_POSTURE=$(yq '.verification.test_gate.posture // "blocking"' "$CONFIG_FILE" 2>/dev/null || echo "blocking")
+OVERRIDE_PHRASE=$(yq '.verification.test_gate.override_phrase // "OVERRIDE: shipping with test failures —"' "$CONFIG_FILE" 2>/dev/null || echo "OVERRIDE: shipping with test failures —")
 ```
 
-All `$DEFAULT_BRANCH` and `$STAGING_BRANCH` references below are populated from `forge.yaml`.
+All `$DEFAULT_BRANCH`, `$STAGING_BRANCH`, `$GATE_POSTURE`, and `$OVERRIDE_PHRASE` references below are populated from `forge.yaml`.
 
 ---
 
@@ -342,6 +346,132 @@ Agent maps dependencies, assesses integration points (service boundaries, env va
 
 ---
 
+## Phase 6.5: Runtime Test Gate (BLOCKING — runs after static analysis, before finding triage)
+
+**Purpose**: Verify the integrated bundle's acceptance criteria against running code before the deploy verdict. Catches runtime defects (cross-PR interactions, container/startup failures, regression in tested behaviour) that static review cannot surface.
+
+**Why here**: Phase 6 completes all static analysis (regression risk, security, quality). Phase 6.5 adds the runtime dimension before Phase 7 triages findings — so any test-gate failures can be filed as `test-failure` issues by `/test-gate` itself, then surface in Phase 7's triage pass.
+
+**Posture**: Controlled by `verification.test_gate.posture` in `forge.yaml` (resolved in Config Resolution as `$GATE_POSTURE`). Default: `blocking`. Set to `advisory` to surface failures without preventing deploy. <!-- Added: forge#906 -->
+
+```bash
+echo "=== Phase 6.5: Runtime Test Gate ==="
+echo "Bundle PRs: $(echo $ALL_PR_NUMBERS | tr '\n' ' ')"
+echo "Posture: ${GATE_POSTURE}"
+
+# Initialize test-gate verdict (default SKIP — safe if Phase 6.5 is bypassed)
+TEST_GATE_VERDICT="SKIP"
+TEST_GATE_REASON="Phase 6.5 not yet run"
+
+# Invoke /test-gate with the bundle PRs already computed in Phase 0A
+# ALL_PR_NUMBERS is the de-duplicated union of both scan methods (commit log + merge subjects)
+GATE_OUTPUT=$(Skill("test-gate", "--prs \"$(echo $ALL_PR_NUMBERS | tr '\n' ' ' | xargs)\" --base $DEFAULT_BRANCH"))
+
+# Extract machine-readable verdict from Skill output
+TEST_GATE_VERDICT=$(echo "$GATE_OUTPUT" | grep -oP '(?<=FORGE:TEST_GATE:RESULT=)(BLOCK|PASS|SKIP)' | tail -1 || echo "SKIP")
+
+echo "Test-gate verdict: ${TEST_GATE_VERDICT}"
+```
+
+**Verdict handling**:
+
+```bash
+case "$TEST_GATE_VERDICT" in
+
+  SKIP)
+    echo "ℹ️  Test gate: SKIP — no executable changes or tests not configured."
+    echo "   This gap will appear in the Phase 8 summary. No deploy impact."
+    TEST_GATE_REASON="SKIP — no runtime tests ran (docs-only bundle, no integration tests configured, or manual-only criteria)"
+    ;;
+
+  PASS)
+    echo "✅ Test gate: PASS — all test clusters passed. Deploy may proceed."
+    TEST_GATE_REASON="PASS — all automated test clusters passed"
+    ;;
+
+  BLOCK)
+    # Check for override comment on the staging→main PR (mirrors Phase 0A pattern)
+    if [ -n "$PR_NUMBER" ]; then
+      TG_OVERRIDE=$(gh pr view "$PR_NUMBER" ${GH_FLAG} \
+        --json comments \
+        --jq "[.comments[].body | select(startswith(\"${OVERRIDE_PHRASE}\"))] | length" 2>/dev/null || echo 0)
+    else
+      TG_OVERRIDE=0
+    fi
+
+    if [ "${TG_OVERRIDE:-0}" -gt 0 ]; then
+      OVERRIDE_REASON=$(gh pr view "$PR_NUMBER" ${GH_FLAG} \
+        --json comments \
+        --jq "[.comments[].body | select(startswith(\"${OVERRIDE_PHRASE}\"))] | last" 2>/dev/null || echo "(reason not captured)")
+      echo "⚠️  Test gate: BLOCK — but override comment detected on PR #${PR_NUMBER}."
+      echo "   Override: ${OVERRIDE_REASON}"
+      echo "   Proceeding with deploy. Override is logged in Phase 8 summary."
+      TEST_GATE_VERDICT="PASS"
+      TEST_GATE_REASON="BLOCK downgraded to PASS by override: ${OVERRIDE_REASON}"
+
+    elif [ "$GATE_POSTURE" = "advisory" ]; then
+      echo "⚠️  Test gate: BLOCK (advisory posture) — runtime failures detected but deploy is NOT prevented."
+      echo "   Switch verification.test_gate.posture to 'blocking' in forge.yaml to enforce this gate."
+      TEST_GATE_REASON="BLOCK (advisory) — runtime failures detected; deploy allowed by advisory posture"
+
+    else
+      # blocking posture (default) — STOP
+      echo "⛔ DEPLOY BLOCKED — /test-gate returned BLOCK verdict."
+      echo ""
+      echo "Runtime failures were detected in the staging→${DEFAULT_BRANCH} bundle."
+      echo "The failures were batch-introduced (not pre-existing on ${DEFAULT_BRANCH} baseline)."
+      echo ""
+      echo "Options:"
+      echo "  1. Fix the failing tests and merge fixes to staging before retrying the deploy."
+      echo "  2. Post a comment on this PR starting with \"${OVERRIDE_PHRASE} <reason>\" to bypass this gate."
+      echo "  3. Set verification.test_gate.posture: advisory in forge.yaml to downgrade to a warning."
+      echo ""
+      echo "RESULT: BLOCK DEPLOY"
+
+      # Post structured FORGE:GATE_FAILURE comment for pipeline-health tracking
+      GATE_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
+      if [ -n "$PR_NUMBER" ]; then
+        gh pr comment "$PR_NUMBER" ${GH_FLAG} --body "<!-- FORGE:GATE_FAILURE -->
+## Deploy Gate: BLOCKED
+
+**Gate**: test-gate
+**Timestamp**: ${GATE_TIMESTAMP}
+**Bundle PRs**: $(echo $ALL_PR_NUMBERS | tr '\n' ' ')
+**Posture**: ${GATE_POSTURE}
+
+### What Happened
+
+\`/test-gate\` detected runtime failures in the staging→\`${DEFAULT_BRANCH}\` bundle that were NOT present on the \`${DEFAULT_BRANCH}\` baseline. These are batch-introduced regressions.
+
+\`/test-gate\` has filed \`test-failure\` issues for each failing cluster — check the issue tracker for details.
+
+### Resolution
+
+1. Fix the failing tests and merge fixes to staging.
+2. Retry the staging→\`${DEFAULT_BRANCH}\` deploy.
+
+To override (ship known failures with documented reason), post a comment containing:
+\`${OVERRIDE_PHRASE} <reason>\`
+
+<!-- FORGE:GATE_FAILURE:TYPE=test-gate|BUNDLE=$(echo $ALL_PR_NUMBERS | tr '\n' ' ' | xargs) -->" 2>/dev/null || true
+      fi
+      exit 1
+    fi
+    ;;
+
+  *)
+    echo "⚠️  Test gate: unrecognised verdict '${TEST_GATE_VERDICT}' — treating as SKIP."
+    TEST_GATE_VERDICT="SKIP"
+    TEST_GATE_REASON="SKIP — unrecognised verdict from /test-gate (treated as SKIP)"
+    ;;
+
+esac
+```
+
+If the gate exits with `RESULT: BLOCK DEPLOY` → **STOP**. A `<!-- FORGE:GATE_FAILURE -->` structured comment is automatically posted on the staging→main PR (if `$PR_NUMBER` is set) for pipeline-health tracking. `/test-gate` will have filed `test-failure` issues for each failing cluster before returning BLOCK.
+
+---
+
 ## Phase 7: Finding Triage & Issue Creation
 
 ### 7A: Extract Findings
@@ -424,10 +554,36 @@ Labels: `review-finding` + `needs-validation` + `staging-review` + priority (`pr
 Post summary with verdict:
 1. CI failed + autofix failed → BLOCK DEPLOY
 2. CI failed + autofix succeeded → continue
+2.5. Test gate returned BLOCK (blocking posture, no override) → BLOCK DEPLOY *(handled in Phase 6.5 — if Phase 8 is reached, BLOCK was either overridden or posture is advisory)*
 3. CONFIRMED CRITICAL (non-CI) → BLOCK DEPLOY
 4. CONFIRMED HIGH blocking (crashes, data loss) → NEEDS FIXES FIRST
 5. All else → APPROVE FOR DEPLOY
 
-Include: Material Changes Summary, Risk Matrix (CI, Build, Bugs, Security, Billing, Quality, Regression), Finding Triage Results, Blocking Issues, Deployment Checklist (pre-deploy, deploy, post-deploy verification, rollback triggers), Stats.
+Include: Material Changes Summary, Risk Matrix (CI, Build, Bugs, Security, Billing, Quality, Regression, **Test Gate**), Finding Triage Results, Blocking Issues, Deployment Checklist (pre-deploy, deploy, post-deploy verification, rollback triggers), Stats.
+
+**Risk Matrix must include a Test Gate row** (use `$TEST_GATE_VERDICT` and `$TEST_GATE_REASON` set in Phase 6.5):
+
+| Domain | Result | Notes |
+|--------|--------|-------|
+| CI | ... | ... |
+| Build | ... | ... |
+| Bugs | ... | ... |
+| Security | ... | ... |
+| Billing | ... | ... |
+| Quality | ... | ... |
+| Regression | ... | ... |
+| **Test Gate** | `${TEST_GATE_VERDICT:-SKIP}` | `${TEST_GATE_REASON:-Phase 6.5 not run}` |
+
+If `TEST_GATE_VERDICT` is `SKIP`, surface the gap explicitly in the summary:
+
+> **Test Gate: SKIP** — No runtime tests ran for this bundle. This means acceptance criteria were NOT verified against running code before deploy. Cause: `${TEST_GATE_REASON}`. To enable runtime testing, configure `verification.integration_tests` in `forge.yaml`.
+
+If `TEST_GATE_VERDICT` is `PASS`, note it as a positive signal:
+
+> **Test Gate: PASS** — Acceptance criteria verified against running code. No batch-introduced runtime failures detected.
+
+If `TEST_GATE_VERDICT` is `BLOCK` and Phase 8 was reached (advisory posture or override active), note it as a risk:
+
+> **Test Gate: BLOCK (override/advisory)** — Runtime failures were detected but deploy is proceeding. Reason: `${TEST_GATE_REASON}`. Filed `test-failure` issues track the failures.
 
 **CRITICAL**: This review NEVER merges staging → main. User makes deploy decision via GitHub web UI.