ai-spec-dev 0.30.1 → 0.31.0

package/README.md

@@ -93,7 +93,7 @@ ai-spec create "给用户模块增加登录功能"
         [cycle 2/2] Running tests: npm test
         ✔ Tests passed.
         ✔ All checks passed after 2 cycle(s).
-[9/9]   Automated code review (3-pass: architecture + implementation + impact/complexity)...
+[9/10]  Automated code review (3-pass: architecture + implementation + impact/complexity)...
         Pass 1/3: Architecture review...
         Pass 2/3: Implementation review...
         Pass 3/3: Impact & complexity assessment...
@@ -1259,6 +1259,32 @@ Attempting auto-fix (3 error(s))...
 
 ---
 
+### Step 10 — Harness Self-Eval
+
+代码审查完成后自动执行，**零 AI 调用**，纯确定性评分：
+
+| 维度 | 评分逻辑 |
+|------|---------|
+| DSL Coverage (0-10) | 生成文件是否覆盖 DSL 声明的 endpoint 层和 model 层 |
+| Compile Score (0-10) | error feedback 全通过 → 10；未通过 / 跳过 → 5 |
+| Review Score (0-10) | 从 3-pass review 文本提取 `Score: X/10` |
+
+**Harness Score** = 加权平均（DSL 40% + Compile 30% + Review 30%）
+
+```
+─── Harness Self-Eval ───────────────────────────
+  Score  : [████████░░] 7.8/10
+  DSL    : 8/10  Compile: pass  Review: 7.2/10
+  Prompt : a3f2c1d8
+─────────────────────────────────────────────────
+```
+
+- `harnessScore` 和 `promptHash` 写入 RunLog（`.ai-spec-logs/<runId>.json`）
+- 每次改动 prompt 文件后，`promptHash` 自动变化，结合 `harnessScore` 可量化 prompt 改动的效果
+- 后续可通过脚本聚合多个 RunLog，绘制 harnessScore × promptHash 的趋势图
+
+---
+
 ## 多 Repo 工作区模式
 
 当父目录中存在 `.ai-spec-workspace.json` 时，`ai-spec create` 自动切换为**多 Repo 联动模式**，一句需求驱动前后端全链路实现。
@@ -1525,6 +1551,8 @@ ai-spec-dev-poc/
 │   ├── reviewer.ts                 # AI 代码审查（git diff / 文件内容双模式）
 │   ├── test-generator.ts           # 测试骨架生成器（DSL → Jest/Vitest 骨架）
 │   ├── error-feedback.ts           # 错误反馈自动修复（测试+lint检测 · 依赖图排序修复 · AI修复循环）
+│   ├── prompt-hasher.ts            # [v0.31.0] Prompt Hash：6 个核心 prompt 的 SHA-256 短 hash
+│   ├── self-evaluator.ts           # [v0.31.0] Harness Self-Eval：零 AI 调用，DSL覆盖+编译+review加权评分
 │   ├── knowledge-memory.ts         # 经验积累：审查 issue → 宪法§9
 │   ├── workspace-loader.ts         # [Phase 4] 工作区配置加载 + repo 类型自动检测
 │   ├── requirement-decomposer.ts   # [Phase 4] 需求跨 repo 拆分 + UX 决策生成

package/RELEASE_LOG.md

@@ -2,6 +2,39 @@
 
 ---
 
+## [0.31.0] 2026-03-29 — Harness Engineer：Prompt Hash + Create 内联 Self-Eval
+
+### 新增内容
+
+**Feature #1 — Prompt Hash 关联（`core/prompt-hasher.ts`、`core/run-logger.ts`）**
+
+- 新增 `computePromptHash()` — 对 6 个核心 prompt 字符串（codegen、DSL extractor、spec generator、review 三 pass）计算 SHA-256 并取前 8 位，返回形如 `a3f2c1d8` 的短 hex 字符串
+- `RunLog` 新增 `promptHash?: string` 字段；`RunLogger` 新增 `setPromptHash()` + `setHarnessScore()` 方法
+- `ai-spec create` 运行开始时立即调用 `computePromptHash()` 写入 RunLog，任何 prompt 文件改动都会产生不同的 hash
+- **目的**：跨多次运行对比 `harnessScore` 时，可以精确知道「这两次用的 prompt 版本是否相同」，将 prompt 改动的效果从模型随机性中解耦
+
+**Feature #2 — Create 内联 Harness Self-Eval（`core/self-evaluator.ts`、`cli/index.ts`）**
+
+- 新增 `core/self-evaluator.ts` — 零 AI 调用的确定性评分模块：
+  - **DSL Coverage Score (0-10)**：检查 `generatedFiles` 中是否存在 endpoint 层文件（`src/api*`、`src/routes*`、`src/controller*`…）和 model 层文件（`src/model*`、`prisma/`、`src/db*`…），与 DSL 中声明的 endpoint / model 数量对照
+  - **Compile Score (0-10)**：`runErrorFeedback()` 返回 `true` → 10，未通过 / 跳过 → 5
+  - **Review Score (0-10)**：从 3-pass review 文本中提取 `Score: X/10`（与 `reviewer.ts` 同规则），review 跳过时为 null
+  - **Harness Score**：加权平均（有 review：DSL×40% + Compile×30% + Review×30%；无 review：DSL×55% + Compile×45%）
+- `runErrorFeedback()` 的返回值（`boolean`）现在被接住赋给 `compilePassed`，传入 self-eval
+- `ai-spec create` Step 9（code review）之后新增 **Step 10: Harness Self-Eval**，完成后打印：
+
+  ```
+  ─── Harness Self-Eval ───────────────────────────
+    Score  : [████████░░] 7.8/10
+    DSL    : 8/10  Compile: pass  Review: 7.2/10
+    Prompt : a3f2c1d8
+  ─────────────────────────────────────────────────
+  ```
+
+- `harnessScore` 和所有维度分数写入 RunLog 的 `self_eval:done` 事件 + 根级 `harnessScore` 字段，便于后续脚本聚合分析
+
+---
+
 ## [0.30.0] 2026-03-29 — 错误修复依赖图排序 + 前端 Import 多行感知解析
 
 ### 改进内容

package/cli/index.ts

@@ -73,6 +73,8 @@ import { SpecUpdater } from "../core/spec-updater";
 import { exportOpenApi } from "../core/openapi-exporter";
 import { generateRunId, RunLogger, setActiveLogger } from "../core/run-logger";
 import { RunSnapshot, setActiveSnapshot } from "../core/run-snapshot";
+import { computePromptHash } from "../core/prompt-hasher";
+import { runSelfEval, printSelfEval } from "../core/self-evaluator";
 
 // ─── Config File ──────────────────────────────────────────────────────────────
 
@@ -305,6 +307,11 @@ program
     });
     setActiveLogger(runLogger);
 
+    // Record prompt hash immediately — links this RunLog to the prompt version
+    // in use, enabling cross-run harnessScore comparisons (Harness Engineering).
+    const promptHash = computePromptHash();
+    runLogger.setPromptHash(promptHash);
+
     // ── Step 1: Context ───────────────────────────────────────────────────────
     console.log(chalk.blue("[1/6] Loading project context..."));
     runLogger.stageStart("context_load");
@@ -625,14 +632,16 @@ program
     // ── Step 8: Error Feedback Loop ───────────────────────────────────────────
     // In TDD mode, the error feedback loop is the primary driver:
     // it runs tests, collects failures, and fixes implementation until tests pass.
+    let compilePassed = false;
     if (opts.skipErrorFeedback) {
       console.log(chalk.gray("[8/9] Skipping error feedback (--skip-error-feedback)."));
+      compilePassed = true; // treat skip as neutral pass for self-eval
     } else {
       if (opts.tdd) {
         console.log(chalk.cyan("[8/9] TDD mode — error feedback loop driving implementation to pass tests..."));
       }
       runLogger.stageStart("error_feedback");
-      await runErrorFeedback(codegenProvider, workingDir, extractedDsl, {
+      compilePassed = await runErrorFeedback(codegenProvider, workingDir, extractedDsl, {
         maxCycles: opts.tdd ? 3 : 2,  // TDD gets one extra cycle
       });
       runLogger.stageEnd("error_feedback");
@@ -665,6 +674,20 @@ program
       await accumulateReviewKnowledge(specProvider, currentDir, reviewResult);
     }
 
+    // ── Step 10: Harness Self-Evaluation ──────────────────────────────────────
+    // Zero AI calls — deterministic scoring from file-system state + review text.
+    // Records harnessScore + promptHash in RunLog for cross-run trend analysis.
+    runLogger.stageStart("self_eval");
+    const selfEvalResult = runSelfEval({
+      dsl: extractedDsl,
+      generatedFiles,
+      compilePassed,
+      reviewText: reviewResult,
+      promptHash,
+      logger: runLogger,
+    });
+    printSelfEval(selfEvalResult);
+
     // ── Done ──────────────────────────────────────────────────────────────────
     runLogger.finish();
     console.log(chalk.bold.green("\n✔ All done!"));

package/core/prompt-hasher.ts

@@ -0,0 +1,42 @@
+import { createHash } from "crypto";
+
+import { codeGenSystemPrompt } from "../prompts/codegen.prompt";
+import {
+  reviewArchitectureSystemPrompt,
+  reviewImplementationSystemPrompt,
+  reviewImpactComplexitySystemPrompt,
+} from "../prompts/codegen.prompt";
+import { dslSystemPrompt } from "../prompts/dsl.prompt";
+import { specPrompt } from "../prompts/spec.prompt";
+
+/**
+ * Compute a short deterministic hash of the key prompt strings used in a run.
+ *
+ * Why this matters (Harness Engineering):
+ *   When you change a prompt and re-run `ai-spec create`, the resulting RunLog
+ *   will have a different promptHash. Cross-referencing RunLogs by promptHash
+ *   lets you quantify whether a prompt change improved or degraded harnessScore
+ *   without keeping a separate changelog.
+ *
+ * Coverage: codegen system prompt (TS), DSL extractor, spec generator, and all
+ *   three review-pass prompts — these drive the vast majority of token spend and
+ *   output variance.
+ *
+ * Returns: 8-char lowercase hex (e.g. "a3f2c1d8"). Collision probability for
+ *   practical prompt-tweak scenarios is negligible.
+ */
+export function computePromptHash(): string {
+  const segments = [
+    codeGenSystemPrompt,
+    dslSystemPrompt,
+    specPrompt,
+    reviewArchitectureSystemPrompt,
+    reviewImplementationSystemPrompt,
+    reviewImpactComplexitySystemPrompt,
+  ];
+
+  return createHash("sha256")
+    .update(segments.join("\x00"))   // \x00 separator prevents segment-boundary collisions
+    .digest("hex")
+    .slice(0, 8);
+}

package/core/run-logger.ts

@@ -20,6 +20,15 @@ export interface RunLog {
   provider?: string;
   model?: string;
   specPath?: string;
+  /**
+   * 8-char hex hash of the key prompt strings used in this run.
+   * Changes whenever any of: codegen, DSL, spec, or review prompts are edited.
+   * Use this to correlate RunLogs across runs and measure whether a prompt
+   * change improved or degraded harnessScore (Harness Engineering observability).
+   */
+  promptHash?: string;
+  /** Harness self-evaluation score recorded at end of `create` (0-10). */
+  harnessScore?: number;
   entries: LogEntry[];
   filesWritten: string[];
   errors: string[];
@@ -73,6 +82,18 @@ export class RunLogger {
     this.flush();
   }
 
+  /** Record the prompt hash for this run (call once at run start). */
+  setPromptHash(hash: string): void {
+    this.log.promptHash = hash;
+    this.flush();
+  }
+
+  /** Record the harness self-eval score (call once at run end). */
+  setHarnessScore(score: number): void {
+    this.log.harnessScore = score;
+    this.flush();
+  }
+
   fileWritten(filePath: string): void {
     if (!this.log.filesWritten.includes(filePath)) {
       this.log.filesWritten.push(filePath);

package/core/self-evaluator.ts

@@ -0,0 +1,172 @@
+import chalk from "chalk";
+import { SpecDSL } from "./dsl-types";
+import { RunLogger } from "./run-logger";
+
+// ─── Types ────────────────────────────────────────────────────────────────────
+
+export interface SelfEvalResult {
+  /** 0-10: did generated files cover the endpoint + model layers declared in DSL? */
+  dslCoverageScore: number;
+  /** 0-10: 10 = error feedback passed cleanly, 5 = partial / skipped */
+  compileScore: number;
+  /** 0-10 extracted from 3-pass review text, or null when review was skipped */
+  reviewScore: number | null;
+  /** 0-10 weighted overall — the "Harness Score" recorded in RunLog */
+  harnessScore: number;
+  /** Prompt hash at the time this run executed */
+  promptHash: string;
+  detail: {
+    endpointsTotal: number;
+    endpointLayerCovered: boolean;
+    modelsTotal: number;
+    modelLayerCovered: boolean;
+    filesWritten: number;
+  };
+}
+
+// ─── Helpers ──────────────────────────────────────────────────────────────────
+
+/** File-path patterns that indicate an API / controller / route layer file. */
+const ENDPOINT_LAYER_PATTERNS = [
+  /src\/api/,
+  /src\/routes?/,
+  /src\/controller/,
+  /src\/handler/,
+  /src\/endpoints?/,
+];
+
+/** File-path patterns that indicate a data / model / schema layer file. */
+const MODEL_LAYER_PATTERNS = [
+  /src\/model/,
+  /src\/schema/,
+  /src\/entit/,
+  /src\/db/,
+  /prisma/,
+  /src\/data/,
+  /src\/domain/,
+];
+
+/**
+ * Extract a numeric score from review text.
+ * Matches the same "Score: X/10" pattern as `reviewer.ts → extractScore()`.
+ */
+function extractReviewScore(reviewText: string): number | null {
+  const match = reviewText.match(/Score:\s*(\d+(?:\.\d+)?)\s*\/\s*10/i);
+  return match ? parseFloat(match[1]) : null;
+}
+
+// ─── Main ─────────────────────────────────────────────────────────────────────
+
+/**
+ * Run a lightweight self-evaluation at the end of `ai-spec create`.
+ *
+ * Design goals (Harness Engineering):
+ *  - Zero AI calls: all scoring is deterministic file-system + text checks
+ *  - Produces a single `harnessScore` (0-10) stored in RunLog alongside `promptHash`
+ *  - Lets you compare runs across prompt versions: did harnessScore go up or down?
+ *
+ * Scoring weights:
+ *  | Dimension       | Weight (with review) | Weight (review skipped) |
+ *  |-----------------|----------------------|-------------------------|
+ *  | DSL Coverage    | 40 %                 | 55 %                    |
+ *  | Compile/Error   | 30 %                 | 45 %                    |
+ *  | Review Score    | 30 %                 | —                       |
+ */
+export function runSelfEval(opts: {
+  dsl: SpecDSL | null;
+  generatedFiles: string[];
+  /** true = error-feedback loop ended with all checks passing */
+  compilePassed: boolean;
+  /** Full text of the 3-pass review output; empty string if review was skipped */
+  reviewText: string;
+  promptHash: string;
+  logger: RunLogger;
+}): SelfEvalResult {
+  const { dsl, generatedFiles, compilePassed, reviewText, promptHash, logger } = opts;
+
+  // ── DSL Coverage Score ────────────────────────────────────────────────────
+  const endpointsTotal = dsl?.endpoints?.length ?? 0;
+  const modelsTotal    = dsl?.models?.length    ?? 0;
+
+  const endpointLayerCovered = generatedFiles.some((f) =>
+    ENDPOINT_LAYER_PATTERNS.some((p) => p.test(f))
+  );
+  const modelLayerCovered = generatedFiles.some((f) =>
+    MODEL_LAYER_PATTERNS.some((p) => p.test(f))
+  );
+
+  let dslCoverageScore = 10;
+  if (generatedFiles.length === 0) {
+    dslCoverageScore = 0;
+  } else {
+    if (endpointsTotal > 0 && !endpointLayerCovered) dslCoverageScore -= 4;
+    if (modelsTotal    > 0 && !modelLayerCovered)    dslCoverageScore -= 3;
+  }
+
+  // ── Compile Score ─────────────────────────────────────────────────────────
+  // 10 = clean pass, 5 = error feedback ran but didn't fully clear / was skipped
+  const compileScore = compilePassed ? 10 : 5;
+
+  // ── Review Score ──────────────────────────────────────────────────────────
+  const reviewScore = reviewText ? extractReviewScore(reviewText) : null;
+
+  // ── Harness Score (weighted average) ──────────────────────────────────────
+  const harnessScore = reviewScore !== null
+    ? Math.round((dslCoverageScore * 0.4 + compileScore * 0.3 + reviewScore * 0.3) * 10) / 10
+    : Math.round((dslCoverageScore * 0.55 + compileScore * 0.45) * 10) / 10;
+
+  const result: SelfEvalResult = {
+    dslCoverageScore,
+    compileScore,
+    reviewScore,
+    harnessScore,
+    promptHash,
+    detail: {
+      endpointsTotal,
+      endpointLayerCovered,
+      modelsTotal,
+      modelLayerCovered,
+      filesWritten: generatedFiles.length,
+    },
+  };
+
+  // Persist to RunLog
+  logger.setHarnessScore(harnessScore);
+  logger.stageEnd("self_eval", {
+    harnessScore,
+    dslCoverageScore,
+    compileScore,
+    reviewScore: reviewScore ?? undefined,
+    promptHash,
+  });
+
+  return result;
+}
+
+// ─── Display ──────────────────────────────────────────────────────────────────
+
+export function printSelfEval(result: SelfEvalResult): void {
+  const scoreColor =
+    result.harnessScore >= 8 ? chalk.green :
+    result.harnessScore >= 6 ? chalk.yellow :
+    chalk.red;
+
+  const filled = Math.round(result.harnessScore);
+  const bar    = "█".repeat(filled) + "░".repeat(10 - filled);
+
+  const compileTag = result.compileScore === 10
+    ? chalk.green("pass")
+    : chalk.yellow("partial");
+  const reviewTag  = result.reviewScore !== null
+    ? `Review: ${result.reviewScore}/10`
+    : chalk.gray("Review: skipped");
+
+  console.log(chalk.cyan("\n─── Harness Self-Eval ───────────────────────────"));
+  console.log(`  Score  : ${scoreColor(`[${bar}] ${result.harnessScore}/10`)}`);
+  console.log(
+    `  DSL    : ${scoreColor(result.dslCoverageScore + "/10")}  ` +
+    `Compile: ${compileTag}  ${reviewTag}`
+  );
+  console.log(chalk.gray(`  Prompt : ${result.promptHash}`));
+  console.log(chalk.gray("─".repeat(49)));
+}