qualia-framework 5.1.0 → 5.4.0

package/tests/skills.test.sh

@@ -0,0 +1,143 @@
+#!/bin/bash
+# Qualia Framework — skill smoke tests
+# Verifies every skills/*/SKILL.md is well-formed:
+#   - YAML frontmatter present and parseable
+#   - name field matches folder name
+#   - description present and substantive
+#   - description has trigger phrases (or skill is disable-model-invocation)
+#   - body has at least one h1 heading and 2+ sections
+#
+# Run: bash tests/skills.test.sh
+
+PASS=0
+FAIL=0
+SKILLS_DIR="$(cd "$(dirname "$0")/../skills" && pwd)"
+
+# Skills allowed to ship without trigger phrases — disable-model-invocation
+# skills only fire on explicit slash command, so triggers are optional.
+SKIP_TRIGGER_CHECK=("qualia-road" "qualia-handoff")
+
+pass() {
+  echo "  ✓ $1"
+  PASS=$((PASS + 1))
+}
+
+fail_case() {
+  echo "  ✗ $1"
+  echo "    $2"
+  FAIL=$((FAIL + 1))
+}
+
+is_in_skip_list() {
+  local needle="$1"
+  for x in "${SKIP_TRIGGER_CHECK[@]}"; do
+    [ "$x" = "$needle" ] && return 0
+  done
+  return 1
+}
+
+echo "skills.test.sh — smoke tests for every skills/*/SKILL.md"
+echo ""
+
+for skill_dir in "$SKILLS_DIR"/*/; do
+  name=$(basename "$skill_dir")
+  skill_md="$skill_dir/SKILL.md"
+
+  # Existence
+  if [ ! -f "$skill_md" ]; then
+    fail_case "$name" "SKILL.md not found at $skill_md"
+    continue
+  fi
+
+  # Frontmatter present
+  if ! head -1 "$skill_md" | grep -q "^---$"; then
+    fail_case "$name: frontmatter" "first line is not '---'"
+    continue
+  fi
+  if ! sed -n '2,30p' "$skill_md" | grep -q "^---$"; then
+    fail_case "$name: frontmatter" "no closing --- within first 30 lines"
+    continue
+  fi
+
+  # name field matches folder
+  fm_name=$(grep "^name:" "$skill_md" | head -1 | sed 's/^name:[[:space:]]*//' | tr -d '"')
+  if [ "$fm_name" != "$name" ]; then
+    fail_case "$name: name field" "frontmatter says name=\"$fm_name\", folder is \"$name\""
+    continue
+  fi
+  pass "$name: frontmatter name matches folder"
+
+  # description field present + substantive
+  fm_desc=$(grep "^description:" "$skill_md" | head -1 | sed 's/^description:[[:space:]]*//')
+  desc_len=${#fm_desc}
+  if [ "$desc_len" -lt 50 ]; then
+    fail_case "$name: description" "description is $desc_len chars, expected >= 50"
+    continue
+  fi
+  pass "$name: description present (${desc_len} chars)"
+
+  # Trigger phrases (unless disable-model-invocation or in transitional skip list)
+  if ! is_in_skip_list "$name"; then
+    has_disable=$(grep -c "disable-model-invocation:[[:space:]]*true" "$skill_md")
+    if [ "$has_disable" = "0" ]; then
+      if echo "$fm_desc" | grep -qiE "trigger|when user|use when|invoke|says|use this|fire on|user types"; then
+        pass "$name: description has trigger guidance"
+      else
+        fail_case "$name: triggers" "description lacks trigger phrases (Trigger:/Use when:/'says'/etc.) and skill is not disable-model-invocation"
+      fi
+    fi
+  fi
+
+  # Body has an h1 heading
+  h1_count=$(grep -cE "^# " "$skill_md")
+  if [ "$h1_count" -ge 1 ]; then
+    pass "$name: body has h1 heading"
+  else
+    fail_case "$name: body" "no h1 heading (^# ) in body"
+  fi
+
+  # Body has at least one section (any ## heading)
+  h2_count=$(grep -cE "^## " "$skill_md")
+  if [ "$h2_count" -ge 1 ]; then
+    pass "$name: body has section heading (${h2_count} found)"
+  else
+    fail_case "$name: body" "no '## ' section heading; every skill needs at least one"
+  fi
+
+  # Cache-aware spawn audit (per rules/grounding.md):
+  # Every spawn to a CUSTOM (qualia-*) agent must anchor the prompt with
+  # `@~/.claude/agents/{name}.md` (either `Role: @...` or `Read your role:
+  # @...` — both forms accepted). The role file is session-stable; placing
+  # it first lets Anthropic's prompt cache reuse the prefix across spawns
+  # (documented 81-90% cost reduction). If task-specific content lands
+  # before the role anchor, the entire prefix recomputes on every spawn.
+  #
+  # Built-in subagent types (Explore, general-purpose, Plan, etc.) have
+  # stable system-prompt baselines on Anthropic's side; no Role anchor
+  # required. We count only `subagent_type="qualia-*"` spawns.
+  #
+  # Some skills follow progressive-disclosure discipline (e.g.
+  # qualia-polish-loop) and put the literal spawn template in REFERENCE.md
+  # while SKILL.md mentions the spawn in prose. We scan both.
+  custom_spawn_count=$(grep -c 'subagent_type="qualia-' "$skill_md")
+  ref_md="$skill_dir/REFERENCE.md"
+  if [ -f "$ref_md" ]; then
+    custom_spawn_count=$((custom_spawn_count + $(grep -c 'subagent_type="qualia-' "$ref_md")))
+  fi
+  if [ "${custom_spawn_count:-0}" -gt 0 ]; then
+    role_count=$(grep -cE '@~/\.claude/agents/' "$skill_md")
+    if [ -f "$ref_md" ]; then
+      role_count=$((role_count + $(grep -cE '@~/\.claude/agents/' "$ref_md")))
+    fi
+    if [ "${role_count:-0}" -ge "$custom_spawn_count" ]; then
+      pass "$name: spawn audit ($custom_spawn_count custom spawn(s), all role-anchored for cache)"
+    else
+      fail_case "$name: spawn audit" "$custom_spawn_count custom spawn(s) but only ${role_count:-0} '@~/.claude/agents/' anchors — prompt cache will miss"
+    fi
+  fi
+done
+
+echo ""
+echo "=== Results: $PASS passed, $FAIL failed ==="
+
+[ "$FAIL" = "0" ]

package/tests/slop-detect.test.sh

@@ -0,0 +1,160 @@
+#!/bin/bash
+# Qualia Framework — bin/slop-detect.mjs behavior tests
+# Verifies the AI-tells gatekeeper actually catches what it claims to catch.
+#
+# Run: bash tests/slop-detect.test.sh
+
+PASS=0
+FAIL=0
+SLOP_DETECT="$(cd "$(dirname "$0")/../bin" && pwd)/slop-detect.mjs"
+NODE="${NODE:-node}"
+
+TMP_DIRS=()
+cleanup() {
+  for d in "${TMP_DIRS[@]}"; do
+    [ -d "$d" ] && rm -rf "$d"
+  done
+}
+trap cleanup EXIT
+
+mktmp() {
+  local TMP
+  TMP=$(mktemp -d)
+  TMP_DIRS+=("$TMP")
+  echo "$TMP"
+}
+
+pass() {
+  echo "  ✓ $1"
+  PASS=$((PASS + 1))
+}
+
+fail_case() {
+  echo "  ✗ $1"
+  echo "    $2"
+  FAIL=$((FAIL + 1))
+}
+
+echo "slop-detect.test.sh — bin/slop-detect.mjs behavioral tests"
+echo ""
+
+# ── Sanity: file exists and parses ────────────────────────────────────
+if [ ! -f "$SLOP_DETECT" ]; then
+  fail_case "slop-detect exists" "$SLOP_DETECT not found"
+  echo "=== Results: $PASS passed, $FAIL failed ==="
+  exit 1
+fi
+pass "slop-detect.mjs exists at expected path"
+
+if ! $NODE --check "$SLOP_DETECT" 2>&1 | head -1; then
+  pass "slop-detect.mjs parses as valid JS"
+fi
+# Above is a heredoc test — node --check succeeds silently on valid JS.
+# Re-test explicitly so a parse error fails the suite.
+if $NODE --check "$SLOP_DETECT" 2>/dev/null; then
+  pass "slop-detect.mjs syntax is valid"
+else
+  fail_case "syntax check" "node --check failed on $SLOP_DETECT"
+fi
+
+# ── Clean file: should exit 0 ─────────────────────────────────────────
+TMP=$(mktmp)
+cat > "$TMP/clean.tsx" <<'EOF'
+import { Button } from '@/components/ui/button';
+
+export default function Page() {
+  return (
+    <div className="bg-surface text-foreground">
+      <h1 className="text-display">Welcome</h1>
+      <Button variant="primary">Continue setup</Button>
+    </div>
+  );
+}
+EOF
+if $NODE "$SLOP_DETECT" "$TMP/clean.tsx" >/dev/null 2>&1; then
+  pass "exits 0 on a clean .tsx file"
+else
+  fail_case "clean file" "exit non-zero on a deliberately clean file"
+fi
+
+# ── Em-dash detection (HIGH severity — reported, doesn't block) ──────
+# Em-dash is HIGH not CRITICAL, so default exit is 0; we verify the
+# FINDING is reported to stdout/stderr, not the exit code.
+TMP2=$(mktmp)
+cat > "$TMP2/emdash.tsx" <<'EOF'
+export default function Page() {
+  return <p>Welcome — to our amazing platform</p>;
+}
+EOF
+OUT=$($NODE "$SLOP_DETECT" "$TMP2/emdash.tsx" 2>&1 || true)
+if echo "$OUT" | grep -qiE "em.?dash|—"; then
+  pass "reports em-dash finding (HIGH severity, non-blocking)"
+else
+  fail_case "em-dash detection" "no em-dash mention in output: $(echo "$OUT" | head -c 120)"
+fi
+
+# ── Banned-font detection ─────────────────────────────────────────────
+TMP3=$(mktmp)
+cat > "$TMP3/font.css" <<'EOF'
+body { font-family: "Inter", sans-serif; }
+EOF
+EXIT_CODE=0
+$NODE "$SLOP_DETECT" "$TMP3/font.css" >/dev/null 2>&1 || EXIT_CODE=$?
+if [ "$EXIT_CODE" = "1" ]; then
+  pass "exits 1 on banned font (Inter) in CSS"
+else
+  fail_case "banned-font detection" "expected exit 1, got $EXIT_CODE"
+fi
+
+# ── Purple-blue gradient detection ────────────────────────────────────
+TMP4=$(mktmp)
+cat > "$TMP4/gradient.tsx" <<'EOF'
+export default function Hero() {
+  return <div className="bg-gradient-to-r from-blue-500 to-purple-600">Hi</div>;
+}
+EOF
+EXIT_CODE=0
+$NODE "$SLOP_DETECT" "$TMP4/gradient.tsx" >/dev/null 2>&1 || EXIT_CODE=$?
+if [ "$EXIT_CODE" = "1" ]; then
+  pass "exits 1 on purple-blue gradient (the #1 AI-design tell)"
+else
+  fail_case "gradient detection" "expected exit 1, got $EXIT_CODE"
+fi
+
+# ── Existing fixture: skills/qualia-polish-loop/fixtures/broken.html ──
+FIXTURE="$(cd "$(dirname "$0")/.." && pwd)/skills/qualia-polish-loop/fixtures/broken.html"
+if [ -f "$FIXTURE" ]; then
+  EXIT_CODE=0
+  $NODE "$SLOP_DETECT" "$FIXTURE" >/dev/null 2>&1 || EXIT_CODE=$?
+  if [ "$EXIT_CODE" = "1" ]; then
+    pass "exits 1 on the broken.html fixture (designed to hit critical bans)"
+  else
+    fail_case "fixture detection" "broken.html fixture exited $EXIT_CODE; expected 1"
+  fi
+else
+  echo "  - broken.html fixture not present, skipping"
+fi
+
+# ── --json flag produces JSON output ─────────────────────────────────
+TMP5=$(mktmp)
+cp "$TMP3/font.css" "$TMP5/font.css"
+JSON_OUT=$($NODE "$SLOP_DETECT" --json "$TMP5/font.css" 2>/dev/null || true)
+if echo "$JSON_OUT" | head -1 | grep -qE "^[\{\[]"; then
+  pass "--json flag produces JSON-shaped output"
+else
+  fail_case "--json output" "first line is not JSON-shaped: '$(echo "$JSON_OUT" | head -c 80)'"
+fi
+
+# ── Invocation error: no path provided AND no default repo ───────────
+EXIT_CODE=0
+$NODE "$SLOP_DETECT" /nonexistent/path/that/cannot/exist >/dev/null 2>&1 || EXIT_CODE=$?
+if [ "$EXIT_CODE" = "2" ] || [ "$EXIT_CODE" = "0" ]; then
+  pass "handles missing path gracefully (exit=$EXIT_CODE — 0=skip, 2=invocation error)"
+else
+  fail_case "missing path" "unexpected exit $EXIT_CODE on /nonexistent path"
+fi
+
+echo ""
+echo "=== Results: $PASS passed, $FAIL failed ==="
+
+[ "$FAIL" = "0" ]

package/docs/playwright-loop-review-2026-05-03.md

@@ -1,65 +0,0 @@
-# Playwright Visual-Polish Loop — Adversarial Review 2026-05-03
-
-## TL;DR
-
-**Recommendation:** **NO-SHIP**
-
-**Headline finding:** The feature does not exist in the repository. No `skills/qualia-polish-loop/` folder, no pilot results doc, no design notes, no v5.1.0 CHANGELOG entry, no version bump, no commits past the prompt-only commit `8e7d33d`. There is nothing to evaluate against the builder spec. The v5.1 "autonomous visual-polish loop" remains in the state declared at `CHANGELOG.md:280-285` — Deferred.
-
-## Gate-by-gate verdict
-
-| Gate | Status | Evidence |
-|---|---|---|
-| 1 — Builder claim integrity | **FAIL** | No claims to verify. `docs/playwright-loop-pilot-results.md` does not exist (`ls docs/playwright-loop-*` returns only `builder-prompt.md` + `tester-prompt.md`). `docs/playwright-loop-design-notes.md` does not exist. `git log 8e7d33d..HEAD` returns empty. `package.json:3` still reads `"version": "5.0.0"`. `CHANGELOG.md` has no `[5.1.0]` entry. |
-| 2 — Framework regression | **PASS (degenerate)** | `npm test` reports 14 + 59 + 66 + 101 + 15 = **255 passing, 0 failed** across 5 suites. Pass solely because no code changed. Note: the spec claims "260+ tests"; actual baseline is 255. Spec figure is loose. |
-| 3 — Skill structural validity | **FAIL** | `ls skills/qualia-polish-loop/` returns `No such file or directory`. SKILL.md, REFERENCE.md, `scripts/playwright-capture.mjs`, `scripts/score.mjs` — none exist. Gate cannot proceed. |
-| 4 — Pilot results audit | **FAIL** | `docs/playwright-loop-pilot-results.md` does not exist. Scenario 1 / 2 / 3 unverifiable. No `qpl-N:` commit prefixes anywhere in `git log`. |
-| 5 — Adversarial probes | **N/A — 0 PASS, 0 FAIL** | No artifact to probe. Each of 5a–5h marked INSUFFICIENT EVIDENCE: no skill installed → nothing to invoke → no behavior to observe. Re-run when builder ships. |
-| 6 — Token cost reality | **FAIL** | No iterations executed. Token-budget claim ("≤100K per loop") in `playwright-loop-builder-prompt.md:108` and `:32` is unverifiable. |
-| 7 — Security review | **FAIL** | Spec attack surface (Playwright MCP + Bash + Edit + Write + user-provided URL → shell) has no implementation to audit. Must be re-run post-build. |
-| 8 — Doc accuracy | **FAIL** | No CHANGELOG v5.1.0 entry to verify (`grep -n "v5.1\|polish-loop" CHANGELOG.md` returns only the existing "Deferred to v5.1" section at lines 272/280/288/291). No design-notes doc to verify. |
-
-## Critical findings (CRITICAL severity, must-fix before ship)
-
-### C1 — Builder produced zero artifacts
-
-- **Severity:** CRITICAL — matches Severity Rubric line "feature broken for >50% of users; ... wiring missing (component exists but unreachable)" trivially: nothing exists to wire or reach.
-- **Evidence:**
-  - `git log --oneline -30` — most recent commit is `8e7d33d docs(v5.1-prep): Playwright visual-polish loop prompts (builder + reviewer)`. That commit added only the two prompt markdowns.
-  - `git status` — clean, branch `feat/env-empty-guard`, no uncommitted work.
-  - `ls skills/` — 32 skills, none named `qualia-polish-loop`.
-  - `ls docs/` — pilot-results.md and design-notes.md absent.
-  - `package.json:3` — `"version": "5.0.0"`.
-- **Impact:** v5.1 cannot ship. Users invoking `/qualia-polish-loop` will hit a routing miss. The spec's success criteria (`playwright-loop-builder-prompt.md:164-173`) — items 1, 2, 3, 4, 5, 6 — all fail.
-- **Action:** Re-run the builder prompt in a fresh session. Verify the agent actually executes (does not silently hallucinate completion).
-
-## High findings (HIGH severity, fix in v5.1.1 patch)
-
-None applicable — the feature must exist before HIGH-severity behavioral findings can be raised.
-
-## Medium findings (MEDIUM severity, v5.2 backlog)
-
-### M1 — Builder spec contains a verifiable factual error
-
-- **Severity:** MEDIUM — "feature works but missing states; ... contract drift between docs and behavior" applied to the spec itself.
-- **Evidence:** `playwright-loop-builder-prompt.md:9` claims the framework has "260+ tests." `npm test` totals on the current main of `feat/env-empty-guard` give **255**. Off-by-five against the stated baseline. Either tests were lost since the figure was written, or the figure was rounded up.
-- **Impact:** Tester Gate 2 step 1 ("all suites pass with the same count or higher than v5.0.0 baseline (260 tests)") is impossible to satisfy as written. A future builder reading this spec literally will treat 255 as a regression.
-- **Action:** Patch the spec line to `255` or run `git log --all --grep test` to find where the lost 5+ tests went and restore them before v5.1 begins.
-
-## What works well (give credit honestly)
-
-- **The prompt pair is well-engineered.** `playwright-loop-builder-prompt.md` is concrete, cites `file:line` for every integration point, lists 7 hard constraints with named failure modes, mandates 3 self-test scenarios with quantitative expected outcomes, and explicitly forbids silent workarounds (`docs/playwright-loop-builder-prompt.md:175-181`). This is the rare AI-build prompt that survives adversarial reading.
-- **Tester prompt enforces grounding discipline.** Cites `rules/grounding.md`, mandates `file:line` evidence, prohibits hedging, caps tool budget at 50 calls (`playwright-loop-tester-prompt.md:204-205`). The reviewer-side rigor is in place; the builder-side execution is not.
-- **CHANGELOG is honest about deferred work.** `CHANGELOG.md:280-291` already lists the visual-polish loop as v5.1 deferred, with accurate reasoning. The framework owner's documented intent and the current repo state are consistent — the spec was set up correctly; the build run did not happen.
-- **Framework regression test still green.** 255/255 passing on the baseline (`npm test`). The reviewer harness is healthy and ready when the build lands.
-
-## Recommended next steps
-
-1. **Re-spawn the builder agent in a fresh session and verify it actually writes files.** The most likely failure mode is: builder session died, was killed, or hallucinated DONE without committing. Watch the session log; require `git log` output proving commits exist before declaring DONE.
-2. **Patch the test-baseline figure in `playwright-loop-builder-prompt.md:9`** — change "260+ tests" to "255 tests" or audit the git history for missing tests. This unblocks Gate 2.
-3. **Defer this review.** When the builder produces real artifacts, re-run `playwright-loop-tester-prompt.md` against them. The current review is a no-op except for documenting that the build run did not occur.
-4. **Consider a builder pre-flight check.** Add a heartbeat to the builder agent: write `docs/.qpl-builder-started` when the session begins and `docs/.qpl-builder-progress` after every major file. The reviewer can then distinguish "builder didn't run" from "builder ran and failed silently" — a real failure mode given the spec's complexity (Playwright MCP install + Vercel preview deploy + 3 self-tests + commit discipline + slop-detect gate, all in one session).
-
----
-
-**Reviewer note (honesty over signoff, per `playwright-loop-tester-prompt.md:213`):** This review took ~10 tool calls because the absence of artifacts halted Gates 3–8 immediately. The remaining 40 calls of budget are reserved for the next review pass when real artifacts exist. No CRITICAL findings beyond C1 are surfaceable at this stage; once the loop ships, expect Gate 5 (adversarial probes) and Gate 7 (security) to do most of the work.