memento-mori-jester 0.1.51 → 0.1.52

package/CHANGELOG.md

@@ -4,6 +4,12 @@ All notable changes to Memento Mori Jester are tracked here.
 
 ## Unreleased
 
+## 0.1.52
+
+- Added `npm run fixtures:check`, a local fixture authoring validator for duplicate IDs, weak metadata, unsafe-looking content, duplicate content, and explicit expected/absent rule intent.
+- Wired fixture authoring validation into `npm test` so fixture quality is checked alongside the review-engine expectations.
+- Updated maintainer, fixture, release, and production-readiness docs to make fixture validation part of the support-to-coverage workflow.
+
 ## 0.1.51
 
 - Added `docs/MAINTAINER_TRIAGE.md` with a repeatable flow for triaging bugs, false positives, security-sensitive reports, and fixture candidates.

package/README.md

@@ -500,6 +500,7 @@ When filing a bug, include redacted `jester doctor --json` output. The GitHub is
 Use the false-positive template for noisy cautions or blocks. Include `jester summary` and `jester tune <rule-id> --json` output when possible so rule changes can be backed by evidence.
 
 Maintainers can use [docs/MAINTAINER_TRIAGE.md](docs/MAINTAINER_TRIAGE.md) to turn useful false-positive reports into redacted fixtures.
+Run `npm run fixtures:check` before merging fixture changes; it catches duplicate IDs, missing rule metadata, weak descriptions, unsafe-looking content, and duplicate content.
 
 For vulnerabilities, private code exposure, or credential-handling concerns, follow [SECURITY.md](SECURITY.md) instead of opening a public issue with sensitive details.
 

package/ROADMAP.md

@@ -6,6 +6,7 @@ Memento Mori Jester is usable today as a CLI, MCP server, GitHub Action, and git
 
 ## Recently Shipped
 
+- Fixture authoring validator in v0.1.52 for duplicate IDs, missing expected/absent rule intent, weak metadata, unsafe-looking content, and duplicate content.
 - Maintainer triage guide in v0.1.51 for turning useful false-positive reports into redacted fixture coverage.
 - Security policy and GitHub issue templates in v0.1.50 for bug reports, false positives, feature requests, and vulnerability intake.
 - Support-focused `doctor --json` diagnostics in v0.1.49 for package, config, hook, MCP, and GitHub Action state.
@@ -41,7 +42,7 @@ Memento Mori Jester is usable today as a CLI, MCP server, GitHub Action, and git
 ## Product Ideas
 
 - Add more framework-specific false-positive examples from real reports so tuning guidance keeps getting sharper.
-- Add a fixture authoring validator for duplicate IDs, missing expected matches, and weak metadata.
+- Add a fixture report generator that shows which rules, presets, and review kinds still need better pass-case coverage.
 
 ## Quality And Safety
 

package/docs/MAINTAINER_TRIAGE.md

@@ -53,8 +53,8 @@ Good fixture candidates:
 - use the smallest command, plan, diff, or final answer that reproduces the behavior,
 - identify the preset or default config needed to reproduce it,
 - include the expected verdict,
-- include the expected rule id when the fixture should cover a specific rule,
-- include `expectedMatches` for intentional multi-rule overlaps,
+- include `expectedRuleIds` when the fixture should cover specific rules,
+- include `absentRuleIds` when a quiet fixture protects against noisy matches,
 - include `weight` when the case should strongly influence tuning confidence,
 - set `edgeCase: true` when the example is useful but unusual.
 
@@ -74,13 +74,15 @@ Avoid fixtures that:
 
 ```powershell
 npm.cmd test
+npm.cmd run fixtures:check
 node .\dist\cli.js tune <rule-id>
 node .\dist\cli.js tune <rule-id> --json
 node .\dist\cli.js tune coverage
 ```
 
-5. Check whether support/confidence changed in the expected direction.
-6. If the fixture changes verdict behavior, mention the exact rule impact in `CHANGELOG.md`.
+5. Fix any duplicate IDs, missing expected rule metadata, weak descriptions, unsafe content, or duplicate content reported by `fixtures:check`.
+6. Check whether support/confidence changed in the expected direction.
+7. If the fixture changes verdict behavior, mention the exact rule impact in `CHANGELOG.md`.
 
 ## When To Change A Rule
 

package/docs/PRODUCTION_READINESS.md

@@ -52,6 +52,7 @@ This checklist defines what "production grade" means for Memento Mori Jester rig
 - GitHub issue templates collect bug reports, false-positive reports, and feature requests with the diagnostic context maintainers need.
 - `SECURITY.md` routes vulnerability reports away from public issues and asks for redacted diagnostics.
 - `docs/MAINTAINER_TRIAGE.md` explains how to turn useful false-positive reports into fixture coverage before changing rule logic.
+- `npm run fixtures:check` validates fixture IDs, metadata, unsafe-looking content, duplicate content, and explicit expected/absent rule intent.
 - npm publish has a manual workflow fallback, but the normal release path is tag-driven trusted publishing.
 
 ## Static Guard
@@ -65,6 +66,7 @@ This checklist defines what "production grade" means for Memento Mori Jester rig
 - production readiness documentation covers package, GitHub Action, MCP, git hooks, docs, and support,
 - `SECURITY.md` and GitHub issue templates exist and ask for the right diagnostics.
 - maintainer triage docs exist and link noisy-rule reports back to fixture coverage.
+- fixture authoring checks are wired into `npm test`.
 
 `npm test` runs this check after the TypeScript build and unit tests.
 

package/docs/RELEASE.md

@@ -8,6 +8,7 @@ This project publishes GitHub Releases and npm packages from `v*` tags.
 npm.cmd version 0.1.x --no-git-tag-version
 npm.cmd test
 npm.cmd run production:check
+npm.cmd run fixtures:check
 npm.cmd run pack:dry
 git diff --check
 ```

package/docs/RELEASE_NOTES_v0.1.52.md

@@ -0,0 +1,35 @@
+# v0.1.52 Release Notes
+
+This release adds a fixture authoring validator so noisy-rule reports become clean, deterministic fixture coverage instead of fragile JSON edits.
+
+## What Changed
+
+- Added `scripts/check-fixtures.mjs`.
+- Added `npm run fixtures:check`.
+- Wired `fixtures:check` into `npm test`.
+- The validator checks:
+  - duplicate fixture IDs,
+  - valid preset, kind, verdict, and rule-id metadata,
+  - missing `expectedRuleIds` / `absentRuleIds` intent,
+  - invalid `weight` / `edgeCase` values,
+  - duplicate content for the same preset and kind,
+  - unsafe-looking fixture content such as private keys, provider tokens, and absolute home-directory paths.
+- Updated maintainer triage docs, fixture docs, release docs, roadmap, changelog, and production readiness checks.
+
+## Behavior Notes
+
+- No CLI, MCP, config, rule, playground, GitHub Action runtime, or release automation behavior changed.
+- Existing review fixture expectations remain unchanged.
+
+## Release Validation
+
+```powershell
+npm.cmd test
+npm.cmd run fixtures:check
+npm.cmd run production:check
+npm.cmd run demo:svg:check
+npm.cmd run pack:dry
+git diff --check
+node .\dist\cli.js doctor --json
+git diff | node .\dist\cli.js diff --fail-on block --subject "v0.1.52 fixture authoring validator"
+```

package/examples/fixtures/README.md

@@ -19,6 +19,7 @@ Maintainer triage guidance lives in [docs/MAINTAINER_TRIAGE.md](../../docs/MAINT
 
 ```powershell
 npm.cmd test
+npm.cmd run fixtures:check
 ```
 
 For one-off manual review, paste a fixture `content` value into:
@@ -34,8 +35,10 @@ Use the smallest redacted example that still reproduces the behavior. A good fix
 - the review `kind`,
 - the preset or config needed to reproduce it,
 - the expected verdict,
-- the rule ids that should match,
-- any intentional overlaps in `expectedMatches`,
+- the rule ids that should match in `expectedRuleIds`,
+- noisy rules that must stay absent in `absentRuleIds`,
 - and whether the case is an unusual `edgeCase`.
 
 Do not add secrets, private code, customer data, complete logs, or machine-specific paths. If a false-positive report is safe but broad, add a passing fixture before loosening a rule.
+
+`npm run fixtures:check` validates duplicate IDs, missing expected rule metadata, weak descriptions, unsafe-looking fixture content, and duplicate content before the fixture suite becomes tuning evidence.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "memento-mori-jester",
-  "version": "0.1.51",
+  "version": "0.1.52",
   "description": "A local court-jester sidecar for AI coding agents: review plans, commands, diffs, and final claims before they get too pleased with themselves.",
   "type": "module",
   "repository": {
@@ -40,10 +40,11 @@
     "build": "tsc -p tsconfig.json",
     "start": "node dist/server.js",
     "start:mcp": "node dist/server.js",
-    "test": "npm run build && node scripts/run-tests.mjs && npm run production:check",
+    "test": "npm run build && node scripts/run-tests.mjs && npm run fixtures:check && npm run production:check",
     "doctor": "node dist/cli.js doctor",
     "demo:svg": "node scripts/render-demo-svg.mjs",
     "demo:svg:check": "node scripts/render-demo-svg.mjs --check",
+    "fixtures:check": "node scripts/check-fixtures.mjs",
     "production:check": "node scripts/check-production-readiness.mjs",
     "pack:dry": "npm pack --dry-run",
     "prepare": "npm run build",

package/scripts/check-fixtures.mjs

@@ -0,0 +1,190 @@
+#!/usr/bin/env node
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+
+const root = process.cwd();
+const fixturePath = "examples/fixtures/preset-review-cases.json";
+const failures = [];
+
+function read(path) {
+  return readFileSync(join(root, path), "utf8");
+}
+
+function readConstStringArray(path, constName) {
+  const source = read(path);
+  const match = source.match(new RegExp(`export const ${constName} = \\[([^\\]]+)\\] as const`));
+
+  if (!match) {
+    failures.push(`Could not read ${constName} from ${path}.`);
+    return [];
+  }
+
+  return [...match[1].matchAll(/"([^"]+)"/g)].map((entry) => entry[1]);
+}
+
+const allowedPresets = new Set(readConstStringArray("src/config.ts", "configPresetNames"));
+const allowedKinds = new Set(readConstStringArray("src/types.ts", "reviewKinds"));
+const allowedVerdicts = new Set(["pass", "caution", "block"]);
+const unsafeContentPatterns = [
+  { name: "private key block", pattern: /-----BEGIN [A-Z ]*PRIVATE KEY-----/ },
+  { name: "OpenAI-looking secret key", pattern: /\bsk-(?:proj-)?[A-Za-z0-9_-]{20,}\b/ },
+  { name: "Anthropic-looking secret key", pattern: /\bsk-ant-[A-Za-z0-9_-]{20,}\b/ },
+  { name: "GitHub-looking token", pattern: /\bgh[pousr]_[A-Za-z0-9_]{20,}\b/ },
+  { name: "AWS access key id", pattern: /\bAKIA[0-9A-Z]{16}\b/ },
+  { name: "Slack-looking token", pattern: /\bxox[baprs]-[A-Za-z0-9-]{20,}\b/ },
+  { name: "absolute Unix home path", pattern: /(?:^|[\s"'`])\/(?:Users|home)\/[A-Za-z0-9._-]+/ },
+  { name: "absolute Windows user path", pattern: /[A-Za-z]:\\Users\\[A-Za-z0-9._-]+\\/ }
+];
+
+let fixtures;
+try {
+  fixtures = JSON.parse(read(fixturePath));
+} catch (error) {
+  console.error(`Could not parse ${fixturePath}: ${error instanceof Error ? error.message : String(error)}`);
+  process.exit(1);
+}
+
+if (!Array.isArray(fixtures)) {
+  console.error(`${fixturePath} should contain a JSON array.`);
+  process.exit(1);
+}
+
+const ids = new Set();
+const contentKeys = new Map();
+
+for (const [index, fixture] of fixtures.entries()) {
+  const label = typeof fixture?.id === "string" && fixture.id.length > 0
+    ? fixture.id
+    : `fixture[${index}]`;
+
+  if (!fixture || typeof fixture !== "object" || Array.isArray(fixture)) {
+    failures.push(`${label} should be an object.`);
+    continue;
+  }
+
+  checkString(fixture, "id", label);
+  checkString(fixture, "description", label, { minLength: 20 });
+  checkString(fixture, "content", label, { minLength: 3 });
+
+  if (typeof fixture.id === "string") {
+    if (!/^[a-z0-9]+(?:-[a-z0-9]+)*$/.test(fixture.id)) {
+      failures.push(`${label}.id should use stable kebab-case.`);
+    }
+
+    if (ids.has(fixture.id)) {
+      failures.push(`${label}.id is duplicated.`);
+    }
+    ids.add(fixture.id);
+  }
+
+  if (!allowedPresets.has(fixture.preset)) {
+    failures.push(`${label}.preset should be one of: ${[...allowedPresets].join(", ")}.`);
+  }
+
+  if (!allowedKinds.has(fixture.kind)) {
+    failures.push(`${label}.kind should be one of: ${[...allowedKinds].join(", ")}.`);
+  }
+
+  if (!allowedVerdicts.has(fixture.expectedVerdict)) {
+    failures.push(`${label}.expectedVerdict should be pass, caution, or block.`);
+  }
+
+  checkRuleIdArray(fixture, "expectedRuleIds", label, { required: true });
+  checkRuleIdArray(fixture, "absentRuleIds", label, { required: false });
+
+  const expectedRuleIds = Array.isArray(fixture.expectedRuleIds) ? fixture.expectedRuleIds : [];
+  const absentRuleIds = Array.isArray(fixture.absentRuleIds) ? fixture.absentRuleIds : [];
+  const expectedSet = new Set(expectedRuleIds);
+  for (const ruleId of absentRuleIds) {
+    if (expectedSet.has(ruleId)) {
+      failures.push(`${label} lists ${ruleId} in both expectedRuleIds and absentRuleIds.`);
+    }
+  }
+
+  if (fixture.expectedVerdict !== "pass" && expectedRuleIds.length === 0) {
+    failures.push(`${label} should include at least one expectedRuleIds entry for ${fixture.expectedVerdict} verdicts.`);
+  }
+
+  if (expectedRuleIds.length === 0 && absentRuleIds.length === 0) {
+    failures.push(`${label} should include expectedRuleIds or absentRuleIds so fixture intent is explicit.`);
+  }
+
+  if (fixture.weight !== undefined && (!Number.isInteger(fixture.weight) || fixture.weight < 1 || fixture.weight > 3)) {
+    failures.push(`${label}.weight should be an integer from 1 to 3.`);
+  }
+
+  if (fixture.edgeCase !== undefined && typeof fixture.edgeCase !== "boolean") {
+    failures.push(`${label}.edgeCase should be boolean when present.`);
+  }
+
+  if (typeof fixture.content === "string") {
+    for (const unsafe of unsafeContentPatterns) {
+      if (unsafe.pattern.test(fixture.content)) {
+        failures.push(`${label}.content appears to contain ${unsafe.name}; fixtures should use redacted placeholders.`);
+      }
+    }
+
+    const contentKey = `${fixture.preset}:${fixture.kind}:${fixture.content}`;
+    const previous = contentKeys.get(contentKey);
+    if (previous) {
+      failures.push(`${label}.content duplicates ${previous} for the same preset and kind.`);
+    } else {
+      contentKeys.set(contentKey, label);
+    }
+  }
+}
+
+if (failures.length > 0) {
+  console.error("Fixture authoring check failed:");
+  for (const failure of failures) {
+    console.error(`- ${failure}`);
+  }
+  process.exit(1);
+}
+
+process.stdout.write(`Fixture authoring check passed for ${fixtures.length} fixtures.\n`);
+
+function checkString(fixture, field, label, options = {}) {
+  const value = fixture[field];
+  if (typeof value !== "string" || value.trim().length === 0) {
+    failures.push(`${label}.${field} should be a non-empty string.`);
+    return;
+  }
+
+  if (options.minLength && value.trim().length < options.minLength) {
+    failures.push(`${label}.${field} should be at least ${options.minLength} characters.`);
+  }
+}
+
+function checkRuleIdArray(fixture, field, label, options) {
+  const value = fixture[field];
+
+  if (value === undefined) {
+    if (options.required) {
+      failures.push(`${label}.${field} should be an array.`);
+    }
+    return;
+  }
+
+  if (!Array.isArray(value)) {
+    failures.push(`${label}.${field} should be an array.`);
+    return;
+  }
+
+  const seen = new Set();
+  for (const ruleId of value) {
+    if (typeof ruleId !== "string" || ruleId.length === 0) {
+      failures.push(`${label}.${field} should only contain non-empty strings.`);
+      continue;
+    }
+
+    if (!/^[a-z0-9]+(?:-[a-z0-9]+)*$/.test(ruleId)) {
+      failures.push(`${label}.${field} contains ${ruleId}; rule ids should use kebab-case.`);
+    }
+
+    if (seen.has(ruleId)) {
+      failures.push(`${label}.${field} repeats ${ruleId}.`);
+    }
+    seen.add(ruleId);
+  }
+}

package/scripts/check-production-readiness.mjs

@@ -57,6 +57,7 @@ for (const path of [
   "docs/MAINTAINER_TRIAGE.md",
   `docs/RELEASE_NOTES_${tag}.md`,
   "action.yml",
+  "scripts/check-fixtures.mjs",
   ".github/ISSUE_TEMPLATE/bug_report.yml",
   ".github/ISSUE_TEMPLATE/false_positive.yml",
   ".github/ISSUE_TEMPLATE/feature_request.yml",
@@ -83,6 +84,7 @@ requireText("README.md", /github-action --write/, "GitHub Action onboarding");
 requireText("README.md", /SECURITY\.md/, "security policy link");
 requireText("README.md", /false-positive/i, "false-positive support guidance");
 requireText("README.md", /MAINTAINER_TRIAGE\.md/, "maintainer triage guide link");
+requireText("README.md", /fixtures:check/, "fixture authoring check guidance");
 requireText("README.md", /License: PolyForm Noncommercial/, "the noncommercial license badge");
 requireText("docs/PRODUCTION_READINESS.md", /npm package/i, "npm package readiness");
 requireText("docs/PRODUCTION_READINESS.md", /GitHub Action/i, "GitHub Action readiness");
@@ -93,13 +95,20 @@ requireText("docs/PRODUCTION_READINESS.md", /doctor --json/, "doctor JSON suppor
 requireText("docs/PRODUCTION_READINESS.md", /SECURITY\.md/, "security policy readiness");
 requireText("docs/PRODUCTION_READINESS.md", /issue templates/i, "issue template readiness");
 requireText("docs/PRODUCTION_READINESS.md", /MAINTAINER_TRIAGE\.md/, "maintainer triage readiness");
+requireText("docs/PRODUCTION_READINESS.md", /fixtures:check/, "fixture authoring check readiness");
 requireText("docs/CLI.md", /jester doctor --json/, "doctor JSON CLI docs");
 requireText("docs/MAINTAINER_TRIAGE.md", /doctor --json/, "doctor JSON triage prompt");
 requireText("docs/MAINTAINER_TRIAGE.md", /tune <rule-id> --json/, "tune JSON triage prompt");
 requireText("docs/MAINTAINER_TRIAGE.md", /preset-review-cases\.json/, "fixture suite link");
-requireText("docs/MAINTAINER_TRIAGE.md", /expectedMatches/, "fixture overlap guidance");
+requireText("docs/MAINTAINER_TRIAGE.md", /expectedRuleIds/, "fixture expected rule guidance");
+requireText("docs/MAINTAINER_TRIAGE.md", /absentRuleIds/, "fixture absent rule guidance");
 requireText("examples/fixtures/README.md", /MAINTAINER_TRIAGE\.md/, "maintainer triage link");
 requireText("examples/fixtures/README.md", /Adding A Fixture From A Report/, "fixture report conversion guidance");
+requireText("examples/fixtures/README.md", /fixtures:check/, "fixture authoring check guidance");
+requireText("scripts/check-fixtures.mjs", /duplicated/, "duplicate fixture id check");
+requireText("scripts/check-fixtures.mjs", /unsafeContentPatterns/, "unsafe fixture content checks");
+requireText("package.json", /"fixtures:check": "node scripts\/check-fixtures\.mjs"/, "fixture authoring check script");
+requireText("package.json", /npm run fixtures:check/, "fixture authoring check in npm test");
 requireText("SECURITY.md", /doctor --json/, "doctor JSON redaction guidance");
 requireText("SECURITY.md", /security\/advisories\/new/, "private vulnerability report link");
 requireText(".github/ISSUE_TEMPLATE/bug_report.yml", /doctor --json/, "doctor JSON support prompt");