@sdd-method/sdd-cli 0.110.2 → 0.111.1

package/scripts/README.md

@@ -1,10 +1,17 @@
 # Bundled upstream scripts
 
-Frozen copies of bash scripts from
-[`sdd-method`](https://github.com/tomfosteruk/sdd-method) that
-`sdd-cli` either shells out to (`validate-*`) or retains as a regression
-oracle (`sync-method-baseline.sh`, which is superseded by native TypeScript
-sync in v0.2.0 but kept so `tests/parity/` can keep catching drift).
+Copies of bash scripts from
+[`sdd-method`](https://github.com/tomfosteruk/sdd-method) that `sdd-cli` shells
+out to at runtime (`validate-*`). These SHIP in the npm package, because the
+`sdd-cli validate` dispatcher spawns them.
+
+⚠ `sync-method-baseline.sh` is no longer here. It was never shelled out to — it is
+a parity oracle — and it now lives at `tests/parity/fixtures/`, which is where its
+only consumer is and, crucially, is not published. While it sat in this directory it
+shipped to adopters, and it had drifted 386 lines behind upstream: it wrote a lock
+with a single bundle entry, which would have dropped every other bundle's entry on
+the 97-of-105 adopter locks that are multi-bundle. Adopters receive the
+authoritative script in `platform-baseline`; nothing needed a copy here.
 
 Files in this directory are **byte-identical** to their upstream source. Do
 not edit here — edit upstream, then run the refresh procedure below to copy
@@ -13,9 +20,13 @@ signal.
 
 ## Provenance
 
+⚠ **These SHAs are historical.** Byte-identity is now asserted by
+`tests/parity/vendored-script-drift.test.ts` against the sibling sdd-method checkout,
+so the pin is the test rather than this table. The table is retained only as a record
+of where each file came from.
+
 | File | Upstream path | Pinned to |
 |---|---|---|
-| `sync-method-baseline.sh` | `orchestration/scripts/sync-method-baseline.sh` | `4d80826` |
 | `validate-app-docs.sh` | `orchestration/scripts/validate-app-docs.sh` | `db4cb25` |
 | `validate-app-dependencies.sh` | `orchestration/scripts/validate-app-dependencies.sh` | `db4cb25` |
 | `validate-integration-docs.sh` | `orchestration/scripts/validate-integration-docs.sh` | `db4cb25` |
@@ -34,14 +45,29 @@ cp ../sdd-method/orchestration/scripts/<script>.sh scripts/
 # Then update the SHA in this README and commit both together.
 ```
 
-Automation (pre-build sync step) is a later milestone. Refresh is currently
-manual and visible.
+⚠ **Manual-and-visible turned out to be manual-and-invisible**: this README mandated byte-identity ("do not edit here …
+byte-identity keeps `diff` usable as a drift-detection signal") while its Lifecycle
+section simultaneously described the sync oracle as "frozen at 4a8cd05" — and the
+provenance table pinned a *different* commit. Either reading was defensible, so a
+386-line drift looked intentional to anyone who checked. The parity oracle now has
+the automation this section called "a later milestone":
+`scripts/sync-vendored-script.sh` plus `tests/parity/vendored-script-drift.test.ts`,
+which fails with a re-sync instruction when any copy differs.
+
+**All four files are now covered**, which mattered: when the guard was generalised it
+immediately found that two of the three validators had ALSO drifted —
+`validate-app-dependencies.sh` by 35 lines and `validate-integration-docs.sh` by 182,
+the latter missing an entire check family. Those files are executed by
+`sdd-cli validate`, so adopters had been running weaker validation than the bundle
+provides. Re-synced.
 
 ## Lifecycle
 
-The `sync-method-baseline.sh` oracle is frozen at 4a8cd05 and stays for
-regression testing — the native TS sync is tested against it in
-`tests/parity/`.
+The `sync-method-baseline.sh` oracle stays for regression testing — the native TS
+sync is compared against it in `tests/parity/` — but it is tracked at upstream HEAD,
+not frozen. Frozen was the wrong model: parity against a frozen oracle stops testing
+the CLI against what adopters run and starts testing it against history, which is how
+a real functional gap (the CLI not applying ADR transitions) stayed invisible.
 
 The `validate-*.sh` scripts are what `sdd-cli validate` spawns. Native TS
 reimplementation of individual validators is v0.3.0+, smallest-first

package/scripts/sync-vendored-script.sh

@@ -1,43 +1,83 @@
 #!/usr/bin/env bash
-# Sync the authoritative `sync-method-baseline.sh` from sdd-method into sdd-cli's
-# vendored copy. Mirrors scripts/sync-canonical-schema.sh: direction of truth is
-# one-way, sdd-method -> vendored, and the vendored copy is never hand-edited.
+# Sync every vendored bash script from sdd-method into this repo. Mirrors
+# scripts/sync-canonical-schema.sh: direction of truth is one-way,
+# sdd-method -> vendored, and a vendored copy is never hand-edited.
 #
-# The vendored copy matters twice over: the parity suite compares sdd-cli against
-# it, and it ships in the npm package (`files` includes `scripts`), so an adopter
-# can invoke it from node_modules. It once fell 386 lines behind and, among other
-# losses, wrote a lock with a single bundle entry — dropping every other bundle's
-# entry, on an estate where 97 of 105 locks are multi-bundle. Because parity
-# compared against the stale copy, nothing caught it.
+# FOUR FILES, TWO DESTINATIONS, and the distinction matters:
+#
+#   scripts/validate-*.sh          SHIP in the npm package, and the CLI shells out to
+#                                  them at runtime via the `sdd-cli validate`
+#                                  dispatcher. A stale copy here means adopters run
+#                                  weaker validation than the bundle provides — and
+#                                  two of the three were found 35 and 182 lines behind,
+#                                  the latter missing an entire check family.
+#
+#   tests/parity/fixtures/         the sync ORACLE, kept only so tests/parity can
+#     sync-method-baseline.sh      compare the TS implementation against the reference.
+#                                  Deliberately NOT in scripts/, because that directory
+#                                  ships: while it lived there it was 386 lines stale
+#                                  and would have written a single-bundle lock,
+#                                  dropping entries from the 97-of-105 adopter locks
+#                                  that are multi-bundle. Adopters receive the
+#                                  authoritative script in platform-baseline.
+#
+# `scripts/README.md` pinned each file to an upstream SHA by hand and called refresh
+# "manual and visible". It was manual and INVISIBLE: three of four copies had drifted.
+# This script plus tests/parity/vendored-script-drift.test.ts replace that.
 #
 # Usage:
 #   scripts/sync-vendored-script.sh                     # from $SDD_METHOD_PATH or ~/Projects/method/sdd-method
 #   scripts/sync-vendored-script.sh /path/to/sdd-method # explicit path
 #
-# After syncing, run `npx vitest run tests/parity` to confirm the CLI still matches
-# the reference — a diff there is a real finding, not a fixture to adjust.
+# After syncing run `npx vitest run tests/parity` — a diff there is a real finding
+# about the TS implementation, not a fixture to adjust.
 
 set -euo pipefail
 
 SDD_METHOD_PATH="${1:-${SDD_METHOD_PATH:-$HOME/Projects/method/sdd-method}}"
-SOURCE="$SDD_METHOD_PATH/orchestration/scripts/sync-method-baseline.sh"
-DEST="$(dirname "$0")/sync-method-baseline.sh"
+SRC_DIR="$SDD_METHOD_PATH/orchestration/scripts"
+HERE="$(cd "$(dirname "$0")" && pwd)"
 
-if [[ ! -f "$SOURCE" ]]; then
-  echo "ERROR: authoritative script not found at $SOURCE" >&2
+# name:destination-relative-to-repo-root
+VENDORED=(
+  "sync-method-baseline.sh:tests/parity/fixtures"
+  "validate-app-docs.sh:scripts"
+  "validate-app-dependencies.sh:scripts"
+  "validate-integration-docs.sh:scripts"
+)
+
+if [[ ! -d "$SRC_DIR" ]]; then
+  echo "ERROR: sdd-method scripts not found at $SRC_DIR" >&2
   echo "       Pass the sdd-method repo path as the first argument." >&2
   exit 1
 fi
 
-if cmp -s "$SOURCE" "$DEST"; then
-  echo "Vendored sync-method-baseline.sh already matches sdd-method ($(wc -l < "$DEST" | tr -d ' ') lines)."
-  exit 0
-fi
+changed=0
+for entry in "${VENDORED[@]}"; do
+  name="${entry%%:*}"
+  dest_dir="${entry#*:}"
+  src="$SRC_DIR/$name"
+  dest="$HERE/../$dest_dir/$name"
 
-OLD_LINES=0
-[[ -f "$DEST" ]] && OLD_LINES=$(wc -l < "$DEST" | tr -d ' ')
-cp "$SOURCE" "$DEST"
-NEW_LINES=$(wc -l < "$DEST" | tr -d ' ')
+  if [[ ! -f "$src" ]]; then
+    echo "ERROR: authoritative script missing: $src" >&2
+    exit 1
+  fi
+  if cmp -s "$src" "$dest"; then
+    printf '  ok      %-32s (%s lines)\n' "$name" "$(wc -l < "$dest" | tr -d ' ')"
+    continue
+  fi
+  old=0
+  [[ -f "$dest" ]] && old=$(wc -l < "$dest" | tr -d ' ')
+  mkdir -p "$(dirname "$dest")"
+  cp "$src" "$dest"
+  printf '  SYNCED  %-32s %s -> %s lines\n' "$name" "$old" "$(wc -l < "$dest" | tr -d ' ')"
+  changed=$((changed + 1))
+done
 
-echo "Synced sync-method-baseline.sh: ${OLD_LINES} -> ${NEW_LINES} lines"
-echo "Run: npx vitest run tests/parity"
+echo
+if [[ "$changed" -gt 0 ]]; then
+  echo "$changed file(s) re-synced. Run: npx vitest run tests/parity"
+else
+  echo "All vendored scripts already match sdd-method."
+fi

package/scripts/validate-app-dependencies.sh

@@ -7,7 +7,9 @@
 #
 # Checks (for each entry in app-platform-dependencies.yaml):
 #   1. required_stability is a valid value (provisional | locked)
-#   2. service_id exists in platform service-architecture.md
+#   2. service_id resolves against the platform source of truth:
+#      catalogues/services.csv (exact id match) when the catalogue exists,
+#      else docs/platform/service-architecture.md (word-anchored match)
 #   3. contract_path exists (as a file) when required_stability is "locked"
 #
 # Outputs a dependency status table.
@@ -43,7 +45,8 @@ Usage: $(basename "$0") --app-root <path> --platform-root <path>
 
 Options:
   --app-root       Path to the app repository root (must contain app-platform-dependencies.yaml)
-  --platform-root  Path to the platform repository root (must contain docs/service-architecture.md)
+  --platform-root  Path to the platform repository root (service_id checks use
+                   catalogues/services.csv when present, else docs/platform/service-architecture.md)
 USAGE
 }
 
@@ -93,6 +96,7 @@ fi
 
 APP_DEPS_FILE="$APP_ROOT/app-platform-dependencies.yaml"
 PLATFORM_SERVICE_ARCH="$PLATFORM_ROOT/docs/platform/service-architecture.md"
+PLATFORM_SERVICES_CSV="$PLATFORM_ROOT/catalogues/services.csv"
 
 VALID_STABILITY_VALUES=("provisional" "locked")
 
@@ -117,13 +121,33 @@ check_prerequisites() {
     fi
     log_pass "yq available"
 
+    if [[ -f "$PLATFORM_SERVICES_CSV" ]]; then
+        log_pass "Platform services catalogue found (canonical service_id source)"
+    else
+        PLATFORM_SERVICES_CSV=""
+    fi
+
     if [[ ! -f "$PLATFORM_SERVICE_ARCH" ]]; then
         log_warn "Platform service-architecture.md not found: $PLATFORM_SERVICE_ARCH"
-        log_warn "service_id validation will be skipped"
         PLATFORM_SERVICE_ARCH=""
     else
         log_pass "Platform service-architecture.md found"
     fi
+
+    if [[ -z "$PLATFORM_SERVICES_CSV" && -z "$PLATFORM_SERVICE_ARCH" ]]; then
+        log_warn "Neither services catalogue nor service-architecture.md found — service_id validation will be skipped"
+    fi
+}
+
+# Word-anchored match: the id must not be a substring of a longer slug token.
+service_id_in_doc() {
+    local service_id="$1"
+    grep -qE "(^|[^a-zA-Z0-9-])${service_id}([^a-zA-Z0-9-]|$)" "$PLATFORM_SERVICE_ARCH" 2>/dev/null
+}
+
+service_id_in_catalogue() {
+    local service_id="$1"
+    awk -F',' -v id="$service_id" 'NR>1 && $1==id {found=1} END {exit !found}' "$PLATFORM_SERVICES_CSV" 2>/dev/null
 }
 
 # ──────────────────────────────────────────────────────────────────────────────
@@ -158,12 +182,23 @@ validate_dependency() {
         ((dep_errors++))
     fi
 
-    # Check 2: service_id exists in platform service-architecture.md
-    if [[ -n "$PLATFORM_SERVICE_ARCH" ]]; then
-        if [[ -z "$service_id" || "$service_id" == "null" ]]; then
-            log_fail "$dep_label: service_id is empty or missing"
+    # Check 2: service_id resolves against the platform source of truth.
+    # The services catalogue (when present) is canonical — it is the same source
+    # the aggregator FK-validates dependency edges against. The prose
+    # service-architecture.md is the fallback for pre-catalogue platforms.
+    if [[ -z "$service_id" || "$service_id" == "null" ]]; then
+        log_fail "$dep_label: service_id is empty or missing"
+        ((dep_errors++))
+    elif [[ -n "$PLATFORM_SERVICES_CSV" ]]; then
+        if ! service_id_in_catalogue "$service_id"; then
+            log_fail "$dep_label: service_id '$service_id' not found in platform services catalogue (catalogues/services.csv)"
             ((dep_errors++))
-        elif ! grep -q "$service_id" "$PLATFORM_SERVICE_ARCH" 2>/dev/null; then
+        elif [[ -n "$PLATFORM_SERVICE_ARCH" ]] && ! service_id_in_doc "$service_id"; then
+            log_warn "$dep_label: service_id '$service_id' is in the services catalogue but not mentioned in service-architecture.md (doc drift)"
+            ((WARNINGS++))
+        fi
+    elif [[ -n "$PLATFORM_SERVICE_ARCH" ]]; then
+        if ! service_id_in_doc "$service_id"; then
             log_fail "$dep_label: service_id '$service_id' not found in platform service-architecture.md"
             ((dep_errors++))
         fi

package/scripts/validate-integration-docs.sh

@@ -159,6 +159,181 @@ check_manifest() {
         *)
             ;;
     esac
+
+    # Interaction patterns (ADR 0171) — optional fields, validated only when present.
+    # Existing manifests predate the field; absence is not a failure. Directionality
+    # mismatch (pattern present but the matching capability flag is false) is a warning.
+    local ingestion invocation
+    ingestion="$(yq e '.ingestionPattern // ""' "$manifest")"
+    invocation="$(yq e '.invocationPattern // ""' "$manifest")"
+
+    if [[ -n "$ingestion" && "$ingestion" != "null" ]]; then
+        case "$ingestion" in
+            event-stream-consume|webhook|socket-listen|poll)
+                log_pass "integration-manifest.yaml: ingestionPattern '$ingestion' valid"
+                # if/fi for the same reason as the invocationPattern warn below:
+                # safe only because more code follows, which is not a property
+                # worth relying on.
+                if [[ "$inbound" != "true" ]]; then
+                    log_warn "integration-manifest.yaml: ingestionPattern set but capabilityFlags.inbound is not true"
+                fi
+                ;;
+            *)
+                log_fail "integration-manifest.yaml: ingestionPattern '$ingestion' not in {event-stream-consume, webhook, socket-listen, poll}"
+                FAILURES=$((FAILURES + 1))
+                ;;
+        esac
+    fi
+
+    if [[ -n "$invocation" && "$invocation" != "null" ]]; then
+        case "$invocation" in
+            request-response|async-request|publish)
+                log_pass "integration-manifest.yaml: invocationPattern '$invocation' valid"
+                # Explicit if/fi, NOT `[[ ... ]] && log_warn`. As the last
+                # statement in this function, a false test made the function
+                # return 1, and under `set -e` that aborted the whole run before
+                # Group 4 — so the check fired on the CORRECT case (a manifest
+                # that declares invocationPattern and properly sets
+                # outbound: true), and the better-formed the manifest, the more
+                # certainly its grounding evidence went unchecked.
+                if [[ "$outbound" != "true" ]]; then
+                    log_warn "integration-manifest.yaml: invocationPattern set but capabilityFlags.outbound is not true"
+                fi
+                ;;
+            *)
+                log_fail "integration-manifest.yaml: invocationPattern '$invocation' not in {request-response, async-request, publish}"
+                FAILURES=$((FAILURES + 1))
+                ;;
+        esac
+    fi
+}
+
+# ---------------------------------------------------------------------------
+# Unedited-scaffold-residue lint (WARN-only).
+#
+# WHAT THIS MEASURES, and what it deliberately does not: residue measures
+# **authoring completeness** — how much of the scaffold is still untouched. It is
+# NOT a signal for whether an SDD models a real provider. Those are independent
+# properties, measured to be so: on one estate 5 SDDs are demonstrations and 11
+# carry residue, because six REAL integrations are simply not authored yet. A
+# residue list is therefore an authoring WORK QUEUE, never a census of
+# demonstrations. Do not re-couple it to the `exemplar` trait.
+#
+# THE TEMPLATE IS THE ORACLE. Residue is defined as template text that survived
+# into the authored document, so the shipped template is read at runtime and the
+# comparison is exact. This matters for two reasons:
+#   1. It is self-maintaining. An earlier attempt at this check hardcoded a list
+#      of placeholder strings; the list silently went stale, so tokens from other
+#      templates read as real content.
+#   2. It separates residue from a brace that is legitimately authored. A JSON
+#      example containing `{` is not residue; the exact string `{provider field}`
+#      is, because the template is where it comes from.
+#
+# WHAT IS LEGITIMATELY RETAINED — state this half or the rule flags correct
+# authoring. Template text kept as STRUCTURE is right and must never be reported:
+#   - section headings (`^#`) — a cohort of SDDs sharing one standard deliberately
+#     keeps the method's section shape while filling it with real content, and that
+#     is exactly correct authoring;
+#   - table header and separator rows (`^|`);
+#   - short shared lines — a bare path, a field label — dropped by a length floor.
+# Residue is template text left IN THE SLOTS, never template text AS STRUCTURE.
+#
+# THREE SHAPES OF RESIDUE, and only two are detectable:
+#   1. Unfilled placeholder tokens      — detected (exact token match).
+#   2. Leftover instruction prose       — detected (exact line match).
+#   3. Unmodified example table rows    — NOT ATTEMPTED, on evidence rather than
+#      difficulty: every placeholder-free table row in the shipped templates is a
+#      table HEADER row, which an authored document is supposed to retain. Rows
+#      that do carry placeholders are already caught by shape 1. So verbatim-row
+#      matching would flag precisely the lines that should survive and add no
+#      signal that shape 1 misses.
+# ---------------------------------------------------------------------------
+
+# Templates that may reach an integration SDD (forward via integration-baseline,
+# reverse via reverse-extension-baseline). Whichever are present are used.
+GROUNDING_TEMPLATES=(
+    "docs/method/methods/templates/feature-grounding-integration-template.md"
+    "docs/method/methods/reverse/templates/integration-product/docs/design/integration/feature-grounding.md"
+)
+
+# Cap on reported items per class — an untouched scaffold would otherwise bury
+# the rest of the run. The count is always reported in full, per no-silent-caps.
+RESIDUE_REPORT_CAP=5
+
+check_grounding_residue() {
+    local grounding="$1"
+    local rel="docs/design/integration/feature-grounding.md"
+
+    local -a templates=()
+    local t
+    for t in "${GROUNDING_TEMPLATES[@]}"; do
+        [[ -f "$REPO_ROOT/$t" ]] && templates+=("$REPO_ROOT/$t")
+    done
+
+    # Never pass silently for want of an oracle: "no residue found" and "could
+    # not look" must not be the same output.
+    if [[ ${#templates[@]} -eq 0 ]]; then
+        log_warn "$rel: cannot check scaffold residue — no grounding template present in this repo (expected one of: ${GROUNDING_TEMPLATES[*]})"
+        return
+    fi
+
+    # --- Shape 1: unfilled placeholder tokens -------------------------------
+    # `|| true` on every match pipeline below: grep exits 1 on no-match, and
+    # under `set -euo pipefail` that aborts the run at the assignment. "Found
+    # nothing" is the expected result for a well-authored document, so it must not
+    # be fatal — this is the same trap that was silently killing Group 4.
+    local tokens
+    tokens="$(grep -ohE '\{[^}]{1,80}\}' "${templates[@]}" 2>/dev/null | sort -u || true)"
+
+    local -a hits=()
+    local token
+    while IFS= read -r token; do
+        [[ -z "$token" ]] && continue
+        local lines
+        lines="$(grep -Fn -- "$token" "$grounding" 2>/dev/null | cut -d: -f1 | paste -sd, - || true)"
+        [[ -n "$lines" ]] && hits+=("line ${lines}: ${token}")
+    done <<< "$tokens"
+
+    if [[ ${#hits[@]} -gt 0 ]]; then
+        log_warn "$rel: ${#hits[@]} unfilled placeholder token(s) from the shipped template — authoring incomplete (not a demonstration marker)"
+        local i
+        for ((i = 0; i < ${#hits[@]} && i < RESIDUE_REPORT_CAP; i++)); do
+            echo "    - ${hits[$i]}"
+        done
+        if [[ ${#hits[@]} -gt $RESIDUE_REPORT_CAP ]]; then
+            echo "    - … and $(( ${#hits[@]} - RESIDUE_REPORT_CAP )) more"
+        fi
+    fi
+
+    # --- Shape 2: leftover template instruction prose -----------------------
+    # Prose only: headings and table rows are structure an authored document
+    # legitimately keeps. The length floor drops short shared lines (a bare path
+    # or a field label) that carry no instruction and would false-positive.
+    local -a prose_hits=()
+    local line
+    while IFS= read -r line; do
+        [[ ${#line} -lt 60 ]] && continue
+        if grep -Fxq -- "$line" "$grounding" 2>/dev/null; then
+            prose_hits+=("$line")
+        fi
+    done < <(grep -hvE '^#|^\||^$|^[[:space:]]*$' "${templates[@]}" 2>/dev/null | sort -u || true)
+
+    if [[ ${#prose_hits[@]} -gt 0 ]]; then
+        log_warn "$rel: ${#prose_hits[@]} line(s) of template instruction prose retained verbatim"
+        local j
+        for ((j = 0; j < ${#prose_hits[@]} && j < RESIDUE_REPORT_CAP; j++)); do
+            local shown="${prose_hits[$j]}"
+            [[ ${#shown} -gt 90 ]] && shown="${shown:0:90}…"
+            echo "    - ${shown}"
+        done
+        if [[ ${#prose_hits[@]} -gt $RESIDUE_REPORT_CAP ]]; then
+            echo "    - … and $(( ${#prose_hits[@]} - RESIDUE_REPORT_CAP )) more"
+        fi
+    fi
+
+    if [[ ${#hits[@]} -eq 0 && ${#prose_hits[@]} -eq 0 ]]; then
+        log_pass "$rel: no unedited scaffold residue detected"
+    fi
 }
 
 check_grounding_doc() {
@@ -197,6 +372,13 @@ check_grounding_doc() {
             FAILURES=$((FAILURES + 1))
         fi
     fi
+
+    # Substance, not just presence. The checks above establish that the required
+    # sections EXIST; this one asks whether they were filled in. WARN-only and
+    # never incrementing FAILURES: an estate must not turn red for being honest
+    # about its authoring state, and an exploration-stage integration is a
+    # legitimate thing to have.
+    check_grounding_residue "$grounding"
 }
 
 main() {