comp-gate 0.1.0

package/src/templates/SKILL.md

@@ -0,0 +1,253 @@
+---
+name: testme
+description: Generate and verify codebase comprehension tests before committing or pushing. Use when the user runs /testme, asks to commit or push, or when a hook blocks git commit/push.
+disable-model-invocation: true
+---
+
+# testme
+
+> **Gate off?** Run `npx comp-gate testing status`. If the gate is disabled, do not run this workflow — tell the user to run `/testing` to re-enable.
+
+Gate `git commit` (when `gateCommits` is enabled) and `git push` to protected branches behind a comprehension check. The agent generates reference answers, quizzes you in chat, then grades your replies by **conceptual alignment** — not keyword matching.
+
+## When to use
+
+- User runs `/testme`
+- User asks to commit or push to a protected branch
+- Hook blocks a commit or push and asks for verification
+
+## Workflow
+
+**Use conversational Q&A in chat.** Do not ask the user to manually edit `.testme/` JSON files.
+
+```
+Task Progress:
+- [ ] Step 1: Update PROMPTS.md with session changes
+- [ ] Step 2: Run npx comp-gate generate
+- [ ] Step 3: Read referenced files; write .testme/references.json
+- [ ] Step 4: Ask each question in chat; collect answers from user replies
+- [ ] Step 5: Grade semantically; write .testme/judgments.json; run npx comp-gate verify
+- [ ] Step 6: Commit and push only after PASS
+```
+
+### Step 1: Update PROMPTS.md
+
+Before generating questions, ensure `PROMPTS.md` documents what changed:
+
+```markdown
+# Session Changes
+
+## src/auth.ts
+- Added validateToken middleware; checks JWT expiry before route handlers
+```
+
+Write bullets as you work. If empty, update from `git diff --stat` first.
+
+### Step 2: Generate questions
+
+```bash
+npx comp-gate generate
+```
+
+Reads `SUMMARY.md`, `PROMPTS.md`, and git diff. Writes `.testme/session.json`.
+
+Announce how many questions were generated and list their ids/categories.
+
+### Step 3: Generate reference answers
+
+Read `.testme/session.json`. For each question, read **only** files in `files[]`.
+
+Write `.testme/references.json`:
+
+```json
+{
+  "diffHash": "<same as session.json>",
+  "generatedAt": "<ISO timestamp>",
+  "references": {
+    "q1": "A complete reference answer explaining what changed and why, based on the code and PROMPTS.md.",
+    "q2": "A reference answer describing how the change fits the project architecture."
+  }
+}
+```
+
+Reference answers are the **gold standard** you will compare user replies against. Write them from your understanding of the code — not by copying PROMPTS.md bullets verbatim.
+
+### Step 4: Conversational Q&A (in chat)
+
+Ask questions **one at a time**. Wait for the user's reply before asking the next.
+
+**Format each question:**
+
+```
+**Question 1 of N** · `q1` · changeRationale
+
+What changed in src/auth.ts and why?
+
+_Files: src/auth.ts_
+```
+
+Rules:
+- Show progress, question id, category, prompt verbatim, and referenced files
+- Do **not** show reference answers or rubric terms to the user
+- Nudge once if an answer is too vague: "Can you explain what changed and why?"
+- Store each answer keyed by question id (`q1`, `q2`, …)
+
+### Step 5: Semantic grading and verify
+
+For each question, compare the user's answer to the reference in `.testme/references.json`:
+
+1. **Summarize** the user's answer in 1–2 sentences (`userSummary`)
+2. **Score accuracy** from 0–100 (`accuracy`) — how well they understand what changed vs the reference
+3. **Judge alignment**: `high` (core concepts correct), `medium` (mostly right, minor gaps), `low` (wrong or too vague)
+4. **Pass** a question if `passed: true`, `accuracy >= passThreshold` (default 70 from config), and alignment is `high` or `medium`
+5. Do **not** fail answers for missing specific keywords — judge meaning, not wording
+
+Write `.testme/answers.json` and `.testme/judgments.json`:
+
+```json
+{
+  "diffHash": "<same as session.json>",
+  "gradedAt": "<ISO timestamp>",
+  "judgments": {
+    "q1": {
+      "passed": true,
+      "accuracy": 88,
+      "userSummary": "User explained the README now documents setup and the chat-based workflow.",
+      "alignment": "high",
+      "feedback": "Strong understanding of the documentation changes; minor detail on troubleshooting missing."
+    }
+  }
+}
+```
+
+Share grading feedback in chat for each question:
+- accuracy score (0–100)
+- alignment (`low` / `medium` / `high`)
+- brief summary of what they got right or wrong
+
+Then run:
+
+```bash
+npx comp-gate verify
+```
+
+**On PASS:** commit and push to protected branches are allowed.
+
+**On FAIL:** re-ask only failed questions in chat, update answers + judgments, run `verify` again. Do not re-run `generate`.
+
+### Step 6: Commit and push
+
+Only after verify passes:
+
+```bash
+git commit -m "your message"
+git push origin <protected-branch>
+```
+
+## Gate configuration
+
+In `testme.config.json`:
+
+```json
+{
+  "protectedBranches": ["main"],
+  "gateCommits": true,
+  "autoProtectCurrentBranch": true
+}
+```
+
+- `gateCommits: true` blocks `git commit` until `/testme` passes
+- `autoProtectCurrentBranch: true` (default) also protects your current branch — no config edits when switching branches
+- Pushes to protected branches are blocked with: `You must run the /testme skill before pushing to '<branch>'.`
+- `npx comp-gate init` also installs **git** `pre-commit` / `pre-push` hooks (blocks the Cursor terminal and any other terminal)
+
+## Token efficiency rules
+
+- Never scan the full repo for this workflow
+- Read only files referenced in session questions
+- Re-run verify after any last-minute edits (pass invalidates on diff change)
+
+## Grading mode
+
+Default is **semantic** (`grading: "semantic"` in `testme.config.json`). Set `passThreshold` (default `70`) for minimum accuracy % per question. Legacy keyword mode: `"grading": "keywords"`.
+
+## SUMMARY.md
+
+Update only when architecture, stack, or major conventions change — not every commit.
+
+Add a `## Domain` section to steer auto-detection:
+
+```markdown
+## Domain
+
+- Primary: machine learning
+- Focus areas: model training, evaluation metrics, inference API
+```
+
+## Configure categories
+
+Question count and knowledge areas are configured in `testme.config.json` (team defaults) with optional local overrides in `.testme/config.json`.
+
+### Checkbox-style categories
+
+Set booleans in `categories` — `true` enables, `false` disables:
+
+```json
+{
+  "questions": { "min": 2, "max": 5 },
+  "categories": {
+    "changeRationale": true,
+    "symbols": true,
+    "architecture": true,
+    "runtime": false,
+    "dataStructures": false,
+    "testing": true,
+    "machineLearning": false,
+    "cybersecurity": false
+  },
+  "autoDetect": true
+}
+```
+
+### Core categories
+
+`changeRationale`, `symbols`, `architecture`, `runtime`, `dataStructures`, `dependencies`, `testing`, `errorHandling`, `apiContracts`, `security`, `performance`, `database`
+
+### Domain categories (auto-suggested)
+
+`machineLearning`, `cybersecurity`, `frontend`, `backend`, `devops`, `mobile`, `dataEngineering`
+
+Run detection before editing config:
+
+```bash
+npx comp-gate detect
+npx comp-gate config init
+npx comp-gate config show
+```
+
+### One-off overrides
+
+```bash
+npx comp-gate generate --max-questions 3 --category changeRationale,runtime,testing
+```
+
+### Examples
+
+**Machine learning project** — enable `machineLearning`, `dataStructures`; disable `cybersecurity`.
+
+**Security tooling project** — enable `cybersecurity`, `security`; disable `machineLearning`.
+
+Ask the user which categories matter if unclear, then update `testme.config.json`.
+
+## Troubleshooting
+
+| Issue | Fix |
+|-------|-----|
+| Session stale | Re-run `npx comp-gate generate` |
+| Missing terms | Add clearer bullets to `PROMPTS.md` |
+| Commit blocked | `gateCommits` is on — run `/testme` before committing |
+| Push blocked | Run `/testme` or `npx comp-gate generate` → answer → `verify` |
+| Pass invalid | `npx comp-gate status` — verify then commit/push |
+| Reset session | `npx comp-gate reset` |
+| Verify failed | Re-ask failed questions; re-grade and update judgments.json |
+| Missing judgments | Agent must grade semantically before running verify |

package/src/templates/SUMMARY.md

@@ -0,0 +1,29 @@
+# Project Summary
+
+Keep this file updated when architecture, stack, or major conventions change — not on every commit.
+
+## Stack
+
+- Language:
+- Framework:
+- Runtime:
+
+## Architecture
+
+- `src/` — application source
+- `tests/` — test suites
+
+## Conventions
+
+- Follow existing naming and import patterns in each directory.
+- Prefer small, focused modules over large catch-all files.
+
+## Runtime Notes
+
+- How to run locally:
+- How to run tests:
+
+## Domain
+
+- Primary:
+- Focus areas:

package/src/templates/agents/agents-md/section.md

@@ -0,0 +1,11 @@
+## testme comprehension gate
+
+Before committing or pushing to a protected branch, run the testme workflow:
+
+1. Update `PROMPTS.md` with session change bullets per file.
+2. Run `npx comp-gate generate` to create questions in `.testme/session.json`.
+3. Answer each question in chat; the agent writes `.testme/answers.json` and `.testme/judgments.json`.
+4. Run `npx comp-gate verify` — on PASS, `.testme/pass.json` is written and git operations are allowed.
+5. After a successful push, `PROMPTS.md` resets automatically via hook.
+
+Do not commit or push until `npx comp-gate verify` passes.

package/src/templates/agents/claude/SKILL.md

@@ -0,0 +1,8 @@
+---
+name: testme
+description: Generate and verify codebase comprehension tests before committing or pushing. Use when the user asks to commit or push, or when a hook blocks git commit/push.
+---
+
+# testme
+
+Use the shared workflow: update PROMPTS.md, run `npx comp-gate generate`, quiz the user in chat, grade semantically, then `npx comp-gate verify` before commit/push.

package/src/templates/agents/claude/settings.fragment.json

@@ -0,0 +1,16 @@
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "Bash",
+        "hooks": [
+          {
+            "type": "command",
+            "command": ".testme/hooks/claude-pre-tool.sh",
+            "timeout": 15
+          }
+        ]
+      }
+    ]
+  }
+}

package/src/templates/agents/claude/testing-SKILL.md

@@ -0,0 +1,9 @@
+---
+name: testing
+description: Toggle the testme comprehension gate on or off. Use when the user runs /testing or asks to enable/disable testme gating.
+disable-model-invocation: true
+---
+
+# testing
+
+Run `npx comp-gate testing` (toggle), `on`, `off`, or `status`. Show the CLI banner output verbatim in chat.

package/src/templates/agents/cursor/SKILL.md

@@ -0,0 +1,9 @@
+---
+name: testme
+description: Generate and verify codebase comprehension tests before committing or pushing. Use when the user runs /testme, asks to commit or push, or when a hook blocks git commit/push.
+disable-model-invocation: true
+---
+
+# testme
+
+Use the shared workflow in the project `SKILL.md` template. Run `npx comp-gate generate`, quiz the user in chat, grade semantically, then `npx comp-gate verify` before commit/push.

package/src/templates/agents/cursor/hooks.json

@@ -0,0 +1,23 @@
+{
+  "version": 1,
+  "hooks": {
+    "beforeShellExecution": [
+      {
+        "command": ".testme/hooks/block-commit.sh",
+        "matcher": "git\\s+commit",
+        "failClosed": true
+      },
+      {
+        "command": ".testme/hooks/block-push.sh",
+        "matcher": "git\\s+push",
+        "failClosed": true
+      }
+    ],
+    "afterShellExecution": [
+      {
+        "command": ".testme/hooks/after-push.sh",
+        "matcher": "git\\s+push"
+      }
+    ]
+  }
+}

package/src/templates/agents/cursor/testing-SKILL.md

@@ -0,0 +1,9 @@
+---
+name: testing
+description: Toggle the testme comprehension gate on or off. Use when the user runs /testing or asks to enable/disable testme gating.
+disable-model-invocation: true
+---
+
+# testing
+
+Run `npx comp-gate testing` (toggle), `on`, `off`, or `status`. Show the CLI banner output verbatim in chat.

package/src/templates/agents/windsurf/SKILL.md

@@ -0,0 +1,8 @@
+---
+name: testme
+description: Generate and verify codebase comprehension tests before committing or pushing. Use when the user asks to commit or push, or when a hook blocks git commit/push.
+---
+
+# testme
+
+Use the shared workflow: update PROMPTS.md, run `npx comp-gate generate`, quiz the user in chat, grade semantically, then `npx comp-gate verify` before commit/push.

package/src/templates/agents/windsurf/hooks.json

@@ -0,0 +1,10 @@
+{
+  "hooks": {
+    "pre_run_command": [
+      {
+        "command": ".testme/hooks/windsurf-pre-run.sh",
+        "show_output": true
+      }
+    ]
+  }
+}

package/src/templates/agents/windsurf/testing-SKILL.md

@@ -0,0 +1,9 @@
+---
+name: testing
+description: Toggle the testme comprehension gate on or off. Use when the user runs /testing or asks to enable/disable testme gating.
+disable-model-invocation: true
+---
+
+# testing
+
+Run `npx comp-gate testing` (toggle), `on`, `off`, or `status`. Show the CLI banner output verbatim in chat.

package/src/templates/hooks/after-push.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+input=$(cat)
+command=$(node -e 'const i=JSON.parse(process.argv[1]); process.stdout.write(i.command||"")' "$input")
+exit_code=$(node -e 'const i=JSON.parse(process.argv[1]); process.stdout.write(String(i.exit_code ?? i.exitCode ?? 1))' "$input")
+
+if [ "$exit_code" != "0" ]; then
+  exit 0
+fi
+
+if ! echo "$command" | grep -qE '^git[[:space:]]+push'; then
+  exit 0
+fi
+
+npx --yes comp-gate hook after-push --command "$command" 2>/dev/null || true
+exit 0

package/src/templates/hooks/block-commit.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+input=$(cat)
+command=$(node -e 'const i=JSON.parse(process.argv[1]); process.stdout.write(i.command||"")' "$input")
+
+if ! echo "$command" | grep -qE '^git[[:space:]]+commit'; then
+  echo '{"permission":"allow"}'
+  exit 0
+fi
+
+response=$(npx --yes comp-gate hook before-commit --command "$command" --json 2>/dev/null || true)
+
+if [ -n "$response" ]; then
+  echo "$response"
+  if echo "$response" | grep -q '"permission"[[:space:]]*:[[:space:]]*"deny"'; then
+    exit 2
+  fi
+  exit 0
+fi
+
+cat <<'EOF'
+{
+  "permission": "deny",
+  "user_message": "You must run the /testme skill before committing.",
+  "agent_message": "You must run the /testme skill before committing."
+}
+EOF
+exit 2

package/src/templates/hooks/block-push.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+input=$(cat)
+command=$(node -e 'const i=JSON.parse(process.argv[1]); process.stdout.write(i.command||"")' "$input")
+
+if ! echo "$command" | grep -qE '^git[[:space:]]+push'; then
+  echo '{"permission":"allow"}'
+  exit 0
+fi
+
+response=$(npx --yes comp-gate hook before-push --command "$command" --json 2>/dev/null || true)
+
+if [ -n "$response" ]; then
+  echo "$response"
+  if echo "$response" | grep -q '"permission"[[:space:]]*:[[:space:]]*"deny"'; then
+    exit 2
+  fi
+  exit 0
+fi
+
+cat <<'EOF'
+{
+  "permission": "deny",
+  "user_message": "You must run the /testme skill before pushing.",
+  "agent_message": "You must run the /testme skill before pushing."
+}
+EOF
+exit 2

package/src/templates/hooks/claude-pre-tool.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+input=$(cat)
+command=$(node -e '
+const data = JSON.parse(process.argv[1]);
+const tool = data.tool_name || "";
+const toolInput = data.tool_input || {};
+const cmd = toolInput.command || toolInput.cmd || "";
+process.stdout.write(cmd);
+' "$input")
+
+if [ -z "$command" ]; then
+  exit 0
+fi
+
+hook_kind=""
+if echo "$command" | grep -qE '^git[[:space:]]+commit'; then
+  hook_kind="before-commit"
+elif echo "$command" | grep -qE '^git[[:space:]]+push'; then
+  hook_kind="before-push"
+else
+  exit 0
+fi
+
+response=$(npx --yes comp-gate hook "$hook_kind" --command "$command" --json 2>/dev/null || true)
+
+if [ -n "$response" ] && echo "$response" | grep -q '"permission"[[:space:]]*:[[:space:]]*"deny"'; then
+  message=$(echo "$response" | node -e '
+const data = JSON.parse(require("fs").readFileSync(0, "utf8"));
+process.stdout.write(data.agent_message || data.user_message || "Blocked by testme.");
+')
+  node -e "
+const reason = process.argv[1];
+process.stdout.write(JSON.stringify({
+  hookSpecificOutput: {
+    hookEventName: 'PreToolUse',
+    permissionDecision: 'deny',
+    permissionDecisionReason: reason
+  }
+}));
+" "$message"
+  exit 0
+fi
+
+if [ -n "$response" ]; then
+  exit 0
+fi
+
+node -e '
+process.stdout.write(JSON.stringify({
+  hookSpecificOutput: {
+    hookEventName: "PreToolUse",
+    permissionDecision: "deny",
+    permissionDecisionReason: "You must run the testme workflow before committing or pushing."
+  }
+}));
+'
+exit 0

package/src/templates/hooks/git-pre-commit.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+root="$(git rev-parse --show-toplevel)"
+cd "$root"
+
+run_comp_gate() {
+  if [ -x "$root/node_modules/.bin/comp-gate" ]; then
+    "$root/node_modules/.bin/comp-gate" "$@"
+  elif command -v comp-gate >/dev/null 2>&1; then
+    comp-gate "$@"
+  elif [ -x "$root/node_modules/.bin/testme" ]; then
+    "$root/node_modules/.bin/testme" "$@"
+  elif command -v testme >/dev/null 2>&1; then
+    testme "$@"
+  else
+    npx --yes comp-gate "$@"
+  fi
+}
+
+result=$(run_comp_gate hook before-commit --command "git commit" --json 2>/dev/null || true)
+
+if [ -n "$result" ]; then
+  if echo "$result" | grep -q '"permission"[[:space:]]*:[[:space:]]*"deny"'; then
+    echo "$result" | node -e '
+      const input = JSON.parse(require("fs").readFileSync(0, "utf8"));
+      console.error(input.user_message ?? input.agent_message ?? "Commit blocked by testme.");
+    '
+    exit 1
+  fi
+  exit 0
+fi
+
+echo "You must run the /testme skill before committing." >&2
+exit 1

package/src/templates/hooks/git-pre-push.sh

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+root="$(git rev-parse --show-toplevel)"
+cd "$root"
+
+run_comp_gate() {
+  if [ -x "$root/node_modules/.bin/comp-gate" ]; then
+    "$root/node_modules/.bin/comp-gate" "$@"
+  elif command -v comp-gate >/dev/null 2>&1; then
+    comp-gate "$@"
+  elif [ -x "$root/node_modules/.bin/testme" ]; then
+    "$root/node_modules/.bin/testme" "$@"
+  elif command -v testme >/dev/null 2>&1; then
+    testme "$@"
+  else
+    npx --yes comp-gate "$@"
+  fi
+}
+
+zero="0000000000000000000000000000000000000000"
+
+while read -r local_ref local_sha remote_ref remote_sha; do
+  if [ -z "${local_ref:-}" ]; then
+    continue
+  fi
+
+  if [ "$local_sha" = "$zero" ]; then
+    continue
+  fi
+
+  case "$remote_ref" in
+    refs/heads/*) ;;
+    *) continue ;;
+  esac
+
+  result=$(run_comp_gate hook before-push-ref --remote-ref "$remote_ref" --json 2>/dev/null || true)
+
+  if [ -z "$result" ]; then
+    echo "You must run the /testme skill before pushing." >&2
+    exit 1
+  fi
+
+  if echo "$result" | grep -q '"permission"[[:space:]]*:[[:space:]]*"deny"'; then
+    echo "$result" | node -e '
+      const input = JSON.parse(require("fs").readFileSync(0, "utf8"));
+      console.error(input.user_message ?? input.agent_message ?? "Push blocked by testme.");
+    '
+    exit 1
+  fi
+done
+
+exit 0

package/src/templates/hooks/statusline.sh

@@ -0,0 +1,3 @@
+#!/bin/sh
+cd "$(git rev-parse --show-toplevel 2>/dev/null)" || exit 0
+npx comp-gate statusline 2>/dev/null || true

package/src/templates/hooks/windsurf-pre-run.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+input=$(cat)
+command=$(node -e '
+const data = JSON.parse(process.argv[1]);
+const toolInfo = data.tool_info || {};
+process.stdout.write(toolInfo.command_line || "");
+' "$input")
+
+if [ -z "$command" ]; then
+  exit 0
+fi
+
+hook_kind=""
+if echo "$command" | grep -qE '^git[[:space:]]+commit'; then
+  hook_kind="before-commit"
+elif echo "$command" | grep -qE '^git[[:space:]]+push'; then
+  hook_kind="before-push"
+else
+  exit 0
+fi
+
+response=$(npx --yes comp-gate hook "$hook_kind" --command "$command" --json 2>/dev/null || true)
+
+if [ -n "$response" ] && echo "$response" | grep -q '"permission"[[:space:]]*:[[:space:]]*"deny"'; then
+  echo "$response" | node -e '
+const data = JSON.parse(require("fs").readFileSync(0, "utf8"));
+console.error(data.user_message || data.agent_message || "Blocked by testme.");
+' >&2
+  exit 2
+fi
+
+if [ -n "$response" ]; then
+  exit 0
+fi
+
+echo "You must run the testme workflow before committing or pushing." >&2
+exit 2