pcl-mcp 0.2.4 → 0.3.0

package/README.md

@@ -1,17 +1,24 @@
+<div align="center">
+
 # PCL — Product Context Layer
 
 **Give AI coding agents persistent, structured knowledge of your product.**
 
-Instead of re-explaining your personas, journeys, and architecture decisions every session, PCL serves them via MCP on demand. Any agent (Claude Code, Cursor, Windsurf) queries exactly what it needs, when it needs it.
-
-## Quick Start
+[![npm version](https://img.shields.io/npm/v/pcl-mcp?color=brightgreen)](https://www.npmjs.com/package/pcl-mcp)
+[![npm downloads](https://img.shields.io/npm/dm/pcl-mcp)](https://www.npmjs.com/package/pcl-mcp)
+[![Node >=22](https://img.shields.io/badge/node-%3E%3D22-blue)](https://nodejs.org)
+[![MIT License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
 
 ```bash
-npm install pcl-mcp
-npx pcl init
-# add MCP config (see Agent Configuration below), then start a new agent session
+npx pcl-mcp init
 ```
 
+</div>
+
+Instead of re-explaining your personas, journeys, and architecture decisions every session, PCL serves them via MCP on demand. Any agent (Claude Code, Cursor, Windsurf) queries exactly what it needs, when it needs it.
+
+---
+
 ## Why PCL?
 
 **Without PCL**, every coding session starts from scratch:
@@ -34,21 +41,40 @@ You ask your agent: *"Build the checkout flow"*
 
 **With PCL:** The agent auto-loads critical billing rules at session start (~200 tokens). When it starts the checkout feature, it pulls the relevant persona, fetches the journey steps, and checks the spec's acceptance criteria — all on-demand, only what's needed. Every session, automatically.
 
+---
+
+## Quick Start
+
+```bash
+npm install pcl-mcp
+npx pcl init            # prompts before adding example files, sets up CLAUDE.md
+# add MCP config (see Agent Configuration below), then start a new agent session
+```
+
+---
+
 ## Stack
 
 | Layer | Technology | Why |
 |---|---|---|
 | Protocol | MCP (stdio) | Universal — works with every major agent |
 | Storage | SQLite + FTS5 | Zero infra, git-friendly, offline |
-| Keyword search | BM25 via FTS5 | Best-in-class for exact term matching |
-| Semantic search | `all-MiniLM-L6-v2` (local) | 23MB, zero API cost, ~3ms/doc |
-| Hybrid fusion | Reciprocal Rank Fusion (k=60) | Better than either alone, no tuning |
-| Validation | Zod schemas | Agents rely on predictable frontmatter |
+| Keyword search | BM25 via FTS5 (title-weighted 10×) | Best-in-class for exact terms, IDs, proper nouns |
+| Semantic search | `all-mpnet-base-v2` (local, 768d) | Higher quality than MiniLM, zero API cost, ~3ms/doc |
+| Embedding strategy | Split body + title embeddings | Separate semantic channels for body and title matching |
+| Hybrid fusion | Adaptive RRF (corpus-size-aware k) | Better recall on both small and large corpora |
+| Score filtering | 15% gap threshold | Prevents low-quality tail results from surfacing |
+| Cross-references | Auto frontmatter link resolution | Pulls related files into results automatically |
+| Validation | Zod schemas | Agents get predictable, parseable frontmatter |
 | File watching | Chokidar v4 | Live reindex on save |
 
+---
+
 ## Prerequisites
 
-- **Node.js >= 22** (required — PCL uses modern Node APIs)
+**Node.js >= 22** (required — PCL uses modern Node APIs)
+
+---
 
 ## Install
 
@@ -59,6 +85,8 @@ npx pcl init            # creates ./product with templates
 
 Also available on GitHub Packages as `@michaelgorski/pcl-mcp`.
 
+---
+
 ## Import existing docs
 
 If you already have markdown documentation in your repo, PCL can scan, classify, and import it automatically:
@@ -77,8 +105,12 @@ The scanner:
 
 Supported classifications: **persona**, **journey**, **spec**, **decision**, **domain**, **product**
 
+---
+
 ## Agent configuration
 
+Works with any MCP-compatible agent. Configuration examples below.
+
 ### Claude Code — `.claude/mcp.json`
 ```json
 {
@@ -113,6 +145,8 @@ Supported classifications: **persona**, **journey**, **spec**, **decision**, **d
 }
 ```
 
+---
+
 ## File structure
 
 ```
@@ -131,19 +165,23 @@ Supported classifications: **persona**, **journey**, **spec**, **decision**, **d
   .pcl.db                 ← SQLite index (auto-generated, gitignore this)
 ```
 
+---
+
 ## Tools available to agents
 
 | Tool | Params | Description |
 |---|---|---|
 | `pcl_product_summary` | — | Load the product north-star document. Call at session start. |
-| `pcl_get_persona(id)` | `id`: persona ID | Get a user persona by ID. Call before any user-facing feature. |
-| `pcl_get_journey(id)` | `id`: journey ID | Get a user journey by ID including step-by-step detail. |
-| `pcl_get_spec(id)` | `id`: spec ID | Get a feature spec by ID including acceptance criteria. |
-| `pcl_get_decision(id)` | `id`: decision ID | Get an architecture decision record (ADR) by ID. |
-| `pcl_get_domain(id)` | `id`: domain ID or `"*critical"` | Get domain rules by ID. Pass `"*critical"` to load all critical rules. |
-| `pcl_list({ type })` | `type`: `"personas"` \| `"journeys"` \| `"specs"` \| `"decisions"` \| `"domain"` | List all files of a given type with IDs, titles, and summaries. |
-| `pcl_search({ query })` | `query`, `mode?` (`"hybrid"` \| `"semantic"` \| `"keyword"`), `types?`, `top_k?` | Hybrid semantic + keyword search across all product files. |
-| `pcl_related(id)` | `id`: source file ID, `top_k?` | Find files semantically related to a given file ID. |
+| `pcl_get_persona` | `id` | Get a user persona by ID. Call before any user-facing feature. |
+| `pcl_get_journey` | `id` | Get a user journey by ID including step-by-step detail. |
+| `pcl_get_spec` | `id` | Get a feature spec by ID including acceptance criteria. |
+| `pcl_get_decision` | `id` | Get an architecture decision record (ADR) by ID. |
+| `pcl_get_domain` | `id` or `"*critical"` | Get domain rules by ID. Pass `"*critical"` to load all critical rules. |
+| `pcl_list` | `type`: `"personas"` \| `"journeys"` \| `"specs"` \| `"decisions"` \| `"domain"` | List all files of a given type with IDs, titles, and summaries. |
+| `pcl_search` | `query`, `mode?` (`"hybrid"` \| `"semantic"` \| `"keyword"`), `types?`, `top_k?` | Hybrid semantic + keyword search across all product files. |
+| `pcl_related` | `id`, `top_k?` | Find files semantically related to a given file ID. |
+
+---
 
 ## Prompts & Resources
 
@@ -153,25 +191,92 @@ In addition to tools, PCL exposes MCP prompts and resources:
 
 **Resources: `pcl://files/{type}/{id}`** — Each indexed file is available as an MCP resource. Agents can browse and read individual files directly via the resource URI (e.g., `pcl://files/persona/example-user`).
 
+---
+
 ## How hybrid search works
 
+PCL runs three parallel retrieval signals and fuses them with Reciprocal Rank Fusion:
+
 ```
 query: "what does Max find frustrating about onboarding"
 
-BM25 (FTS5):      [persona-max, journey-onboarding, spec-magic-link, ...]
-                   ↓ ranked by term frequency + IDF
+BM25 (FTS5, title-weighted 10×):
+  → persona-max, journey-onboarding, spec-magic-link
+  ↓ ranked by bm25(title=10×, body=1×) — exact terms, IDs, proper nouns
+
+Semantic — body embedding (all-mpnet-base-v2, 768d):
+  → journey-onboarding, persona-max, domain-core-rules
+  ↓ cosine similarity on full-text embedding
 
-Cosine similarity: [journey-onboarding, persona-max, domain-core-rules, ...]
-                   ↓ ranked by embedding dot product (MiniLM-L6-v2)
+Semantic — title embedding (all-mpnet-base-v2, 768d):
+  → persona-max, journey-onboarding, spec-onboarding-ux
+  ↓ cosine similarity on title + summary embedding
 
-RRF fusion:        score(d) = Σ 1 / (60 + rank(d))
-                   ↓ combines both rankings without weight tuning
+Adaptive RRF (k = corpus_size / 10):
+  score(d) = Σ 1 / (k + rank(d))   fused across all three lists
 
-Result:            1. journey-onboarding (0.94)
-                   2. persona-max (0.87)
-                   3. spec-onboarding-ux (0.71)
+Score gap filter (15% threshold):
+  Drops results below 0.15 × top_score — removes noise
+
+Cross-reference resolution:
+  journey-onboarding.frontmatter.persona = "max"
+  → auto-includes persona-max even if it ranked outside top-k
+
+Result:  1. journey-onboarding  (0.94)
+         2. persona-max         (0.87)
+         3. spec-onboarding-ux  (0.71)
 ```
 
+**Why split embeddings?** Body and title carry different semantic signals. A query like *"checkout persona"* should match a persona file by title even if its body content is mostly demographic data. Indexing them separately gives the fusion step two distinct semantic channels rather than one diluted one.
+
+**Why adaptive RRF k?** Fixed k=60 over-smooths rankings on small corpora (10–20 files). Corpus-aware k scales down on small collections to let strong matches separate from weak ones.
+
+---
+
+## Testing & Benchmarks
+
+PCL ships with a full test suite and a multi-dimensional benchmark framework.
+
+### Tests
+
+```bash
+npm test            # run all tests (vitest)
+npm run test:watch  # watch mode
+```
+
+Six test suites cover the full stack:
+
+| Suite | Coverage |
+|---|---|
+| `db.test.ts` | SQLite operations, FTS5 queries, embedding storage |
+| `embeddings.test.ts` | Embedding generation, cache hits, dimension checks |
+| `indexer.test.ts` | File discovery, schema extraction, change detection |
+| `schemas.test.ts` | Zod frontmatter validation for all file types |
+| `search.test.ts` | Hybrid search, RRF, multi-hop decomposition, cross-refs |
+| `tools.test.ts` | MCP tool handlers, response formatting, error paths |
+
+### Benchmarks
+
+```bash
+npm run bench           # all benchmarks
+npm run bench:perf      # latency benchmarks (search + embedding speed)
+npm run bench:quality   # search quality: Precision@k, Recall@k, NDCG, MRR
+npm run bench:tokens    # token efficiency across search modes
+npm run bench:ablation  # hybrid vs keyword-only vs semantic-only comparison
+npm run bench:ai        # Claude-judged result quality (requires ANTHROPIC_API_KEY)
+npm run bench:report    # generate markdown report from results
+```
+
+| Suite | Measures |
+|---|---|
+| Performance | Search + embedding latency (p50/p95) |
+| Search quality | Precision@k, Recall@k, NDCG, MRR on labeled corpus |
+| Token efficiency | Tokens consumed per query across search modes |
+| Ablation | Quality delta: hybrid vs keyword-only vs semantic-only |
+| AI quality | Claude-judged relevance score for top-k results |
+
+---
+
 ## Human workflow
 
 The system is only as good as what you put in. Discipline:
@@ -184,12 +289,16 @@ The system is only as good as what you put in. Discipline:
 
 The agent does the rest.
 
+---
+
 ## Gitignore
 
 ```gitignore
 product/.pcl.db      # SQLite index — auto-regenerated
 ```
 
+---
+
 ## License
 
 MIT

package/dist/benchmarks/evaluators/context-retrieval-quality.d.ts

@@ -0,0 +1,30 @@
+/**
+ * Context Retrieval Quality — measures if PCL retrieves the RIGHT documents.
+ * Disentangles "did PCL find the right docs" from "did the LLM use them well."
+ */
+export interface ContextMetrics {
+    recall: number;
+    precision: number;
+    f1: number;
+    retrieved: string[];
+    required: string[];
+    hits: string[];
+    misses: string[];
+    noise: string[];
+}
+/**
+ * Compute context retrieval quality metrics.
+ *
+ * @param retrievedIds - Document IDs returned by PCL search
+ * @param requiredIds - Document IDs that the task actually needs
+ */
+export declare function measureContextRetrieval(retrievedIds: string[], requiredIds: string[]): ContextMetrics;
+/**
+ * Aggregate context metrics across multiple tasks.
+ */
+export declare function averageContextMetrics(metrics: ContextMetrics[]): {
+    recall: number;
+    precision: number;
+    f1: number;
+};
+//# sourceMappingURL=context-retrieval-quality.d.ts.map

package/dist/benchmarks/evaluators/context-retrieval-quality.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"context-retrieval-quality.d.ts","sourceRoot":"","sources":["../../../benchmarks/evaluators/context-retrieval-quality.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,MAAM,WAAW,cAAc;IAC7B,MAAM,EAAE,MAAM,CAAC;IACf,SAAS,EAAE,MAAM,CAAC;IAClB,EAAE,EAAE,MAAM,CAAC;IACX,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,IAAI,EAAE,MAAM,EAAE,CAAC;IACf,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,KAAK,EAAE,MAAM,EAAE,CAAC;CACjB;AAED;;;;;GAKG;AACH,wBAAgB,uBAAuB,CACrC,YAAY,EAAE,MAAM,EAAE,EACtB,WAAW,EAAE,MAAM,EAAE,GACpB,cAAc,CA0BhB;AAED;;GAEG;AACH,wBAAgB,qBAAqB,CACnC,OAAO,EAAE,cAAc,EAAE,GACxB;IAAE,MAAM,EAAE,MAAM,CAAC;IAAC,SAAS,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,MAAM,CAAA;CAAE,CAenD"}

package/dist/benchmarks/evaluators/context-retrieval-quality.js

@@ -0,0 +1,50 @@
+/**
+ * Context Retrieval Quality — measures if PCL retrieves the RIGHT documents.
+ * Disentangles "did PCL find the right docs" from "did the LLM use them well."
+ */
+/**
+ * Compute context retrieval quality metrics.
+ *
+ * @param retrievedIds - Document IDs returned by PCL search
+ * @param requiredIds - Document IDs that the task actually needs
+ */
+export function measureContextRetrieval(retrievedIds, requiredIds) {
+    const retrievedSet = new Set(retrievedIds);
+    const requiredSet = new Set(requiredIds);
+    const hits = requiredIds.filter((id) => retrievedSet.has(id));
+    const misses = requiredIds.filter((id) => !retrievedSet.has(id));
+    const noise = retrievedIds.filter((id) => !requiredSet.has(id));
+    const recall = requiredIds.length > 0 ? hits.length / requiredIds.length : 1;
+    const precision = retrievedIds.length > 0 ? hits.length / retrievedIds.length : 1;
+    const f1 = recall + precision > 0
+        ? (2 * recall * precision) / (recall + precision)
+        : 0;
+    return {
+        recall,
+        precision,
+        f1,
+        retrieved: retrievedIds,
+        required: requiredIds,
+        hits,
+        misses,
+        noise,
+    };
+}
+/**
+ * Aggregate context metrics across multiple tasks.
+ */
+export function averageContextMetrics(metrics) {
+    if (metrics.length === 0)
+        return { recall: 0, precision: 0, f1: 0 };
+    const sum = metrics.reduce((acc, m) => ({
+        recall: acc.recall + m.recall,
+        precision: acc.precision + m.precision,
+        f1: acc.f1 + m.f1,
+    }), { recall: 0, precision: 0, f1: 0 });
+    return {
+        recall: sum.recall / metrics.length,
+        precision: sum.precision / metrics.length,
+        f1: sum.f1 / metrics.length,
+    };
+}
+//# sourceMappingURL=context-retrieval-quality.js.map

package/dist/benchmarks/evaluators/context-retrieval-quality.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"context-retrieval-quality.js","sourceRoot":"","sources":["../../../benchmarks/evaluators/context-retrieval-quality.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAaH;;;;;GAKG;AACH,MAAM,UAAU,uBAAuB,CACrC,YAAsB,EACtB,WAAqB;IAErB,MAAM,YAAY,GAAG,IAAI,GAAG,CAAC,YAAY,CAAC,CAAC;IAC3C,MAAM,WAAW,GAAG,IAAI,GAAG,CAAC,WAAW,CAAC,CAAC;IAEzC,MAAM,IAAI,GAAG,WAAW,CAAC,MAAM,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,YAAY,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC;IAC9D,MAAM,MAAM,GAAG,WAAW,CAAC,MAAM,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,CAAC,YAAY,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC;IACjE,MAAM,KAAK,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,CAAC,WAAW,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC;IAEhE,MAAM,MAAM,GAAG,WAAW,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,GAAG,WAAW,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC;IAC7E,MAAM,SAAS,GACb,YAAY,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC;IAClE,MAAM,EAAE,GACN,MAAM,GAAG,SAAS,GAAG,CAAC;QACpB,CAAC,CAAC,CAAC,CAAC,GAAG,MAAM,GAAG,SAAS,CAAC,GAAG,CAAC,MAAM,GAAG,SAAS,CAAC;QACjD,CAAC,CAAC,CAAC,CAAC;IAER,OAAO;QACL,MAAM;QACN,SAAS;QACT,EAAE;QACF,SAAS,EAAE,YAAY;QACvB,QAAQ,EAAE,WAAW;QACrB,IAAI;QACJ,MAAM;QACN,KAAK;KACN,CAAC;AACJ,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,qBAAqB,CACnC,OAAyB;IAEzB,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,MAAM,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC;IACpE,MAAM,GAAG,GAAG,OAAO,CAAC,MAAM,CACxB,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC;QACX,MAAM,EAAE,GAAG,CAAC,MAAM,GAAG,CAAC,CAAC,MAAM;QAC7B,SAAS,EAAE,GAAG,CAAC,SAAS,GAAG,CAAC,CAAC,SAAS;QACtC,EAAE,EAAE,GAAG,CAAC,EAAE,GAAG,CAAC,CAAC,EAAE;KAClB,CAAC,EACF,EAAE,MAAM,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE,EAAE,EAAE,CAAC,EAAE,CACnC,CAAC;IACF,OAAO;QACL,MAAM,EAAE,GAAG,CAAC,MAAM,GAAG,OAAO,CAAC,MAAM;QACnC,SAAS,EAAE,GAAG,CAAC,SAAS,GAAG,OAAO,CAAC,MAAM;QACzC,EAAE,EAAE,GAAG,CAAC,EAAE,GAAG,OAAO,CAAC,MAAM;KAC5B,CAAC;AACJ,CAAC"}

package/dist/benchmarks/evaluators/ir-metrics.d.ts

@@ -0,0 +1,32 @@
+/**
+ * Precision@K: fraction of top-K results that are relevant.
+ */
+export declare function precisionAtK(retrieved: string[], relevant: Set<string>, k: number): number;
+/**
+ * Recall@K: fraction of relevant docs found in top-K.
+ */
+export declare function recallAtK(retrieved: string[], relevant: Set<string>, k: number): number;
+/**
+ * MRR (Mean Reciprocal Rank): 1 / rank of first relevant result.
+ */
+export declare function reciprocalRank(retrieved: string[], relevant: Set<string>): number;
+/**
+ * NDCG@K: normalized DCG using ideal ranking.
+ */
+export declare function ndcgAtK(retrieved: string[], relevanceScores: Map<string, number>, k: number): number;
+/**
+ * Compute all metrics for a single query.
+ */
+export declare function computeMetrics(retrieved: string[], relevanceScores: Map<string, number>, relevantThreshold?: number): {
+    precisionAt1: number;
+    precisionAt3: number;
+    precisionAt5: number;
+    recallAt5: number;
+    mrr: number;
+    ndcgAt5: number;
+};
+/**
+ * Average metrics across multiple queries.
+ */
+export declare function averageMetrics(results: Array<ReturnType<typeof computeMetrics>>): ReturnType<typeof computeMetrics>;
+//# sourceMappingURL=ir-metrics.d.ts.map

package/dist/benchmarks/evaluators/ir-metrics.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"ir-metrics.d.ts","sourceRoot":"","sources":["../../../benchmarks/evaluators/ir-metrics.ts"],"names":[],"mappings":"AAAA;;GAEG;AACH,wBAAgB,YAAY,CAC1B,SAAS,EAAE,MAAM,EAAE,EACnB,QAAQ,EAAE,GAAG,CAAC,MAAM,CAAC,EACrB,CAAC,EAAE,MAAM,GACR,MAAM,CAKR;AAED;;GAEG;AACH,wBAAgB,SAAS,CACvB,SAAS,EAAE,MAAM,EAAE,EACnB,QAAQ,EAAE,GAAG,CAAC,MAAM,CAAC,EACrB,CAAC,EAAE,MAAM,GACR,MAAM,CAKR;AAED;;GAEG;AACH,wBAAgB,cAAc,CAC5B,SAAS,EAAE,MAAM,EAAE,EACnB,QAAQ,EAAE,GAAG,CAAC,MAAM,CAAC,GACpB,MAAM,CAKR;AAmBD;;GAEG;AACH,wBAAgB,OAAO,CACrB,SAAS,EAAE,MAAM,EAAE,EACnB,eAAe,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,EACpC,CAAC,EAAE,MAAM,GACR,MAAM,CASR;AAED;;GAEG;AACH,wBAAgB,cAAc,CAC5B,SAAS,EAAE,MAAM,EAAE,EACnB,eAAe,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,EACpC,iBAAiB,SAAI,GACpB;IACD,YAAY,EAAE,MAAM,CAAC;IACrB,YAAY,EAAE,MAAM,CAAC;IACrB,YAAY,EAAE,MAAM,CAAC;IACrB,SAAS,EAAE,MAAM,CAAC;IAClB,GAAG,EAAE,MAAM,CAAC;IACZ,OAAO,EAAE,MAAM,CAAC;CACjB,CAeA;AAED;;GAEG;AACH,wBAAgB,cAAc,CAC5B,OAAO,EAAE,KAAK,CAAC,UAAU,CAAC,OAAO,cAAc,CAAC,CAAC,GAChD,UAAU,CAAC,OAAO,cAAc,CAAC,CA0BnC"}

package/dist/benchmarks/evaluators/ir-metrics.js

@@ -0,0 +1,98 @@
+/**
+ * Precision@K: fraction of top-K results that are relevant.
+ */
+export function precisionAtK(retrieved, relevant, k) {
+    const topK = retrieved.slice(0, k);
+    if (topK.length === 0)
+        return 0;
+    const hits = topK.filter(id => relevant.has(id)).length;
+    return hits / topK.length;
+}
+/**
+ * Recall@K: fraction of relevant docs found in top-K.
+ */
+export function recallAtK(retrieved, relevant, k) {
+    if (relevant.size === 0)
+        return 1; // no relevant docs = perfect recall vacuously
+    const topK = retrieved.slice(0, k);
+    const hits = topK.filter(id => relevant.has(id)).length;
+    return hits / relevant.size;
+}
+/**
+ * MRR (Mean Reciprocal Rank): 1 / rank of first relevant result.
+ */
+export function reciprocalRank(retrieved, relevant) {
+    for (let i = 0; i < retrieved.length; i++) {
+        if (relevant.has(retrieved[i]))
+            return 1 / (i + 1);
+    }
+    return 0;
+}
+/**
+ * DCG@K with graded relevance (relevance scores 0-3).
+ */
+function dcgAtK(retrieved, relevanceScores, k) {
+    let dcg = 0;
+    const topK = retrieved.slice(0, k);
+    for (let i = 0; i < topK.length; i++) {
+        const rel = relevanceScores.get(topK[i]) ?? 0;
+        dcg += (Math.pow(2, rel) - 1) / Math.log2(i + 2); // i+2 because log2(1)=0
+    }
+    return dcg;
+}
+/**
+ * NDCG@K: normalized DCG using ideal ranking.
+ */
+export function ndcgAtK(retrieved, relevanceScores, k) {
+    const dcg = dcgAtK(retrieved, relevanceScores, k);
+    // Ideal ranking: sort all docs by relevance descending
+    const idealOrder = [...relevanceScores.entries()]
+        .sort(([, a], [, b]) => b - a)
+        .map(([id]) => id);
+    const idcg = dcgAtK(idealOrder, relevanceScores, k);
+    if (idcg === 0)
+        return 0;
+    return dcg / idcg;
+}
+/**
+ * Compute all metrics for a single query.
+ */
+export function computeMetrics(retrieved, relevanceScores, relevantThreshold = 1) {
+    const relevant = new Set([...relevanceScores.entries()]
+        .filter(([, score]) => score >= relevantThreshold)
+        .map(([id]) => id));
+    return {
+        precisionAt1: precisionAtK(retrieved, relevant, 1),
+        precisionAt3: precisionAtK(retrieved, relevant, 3),
+        precisionAt5: precisionAtK(retrieved, relevant, 5),
+        recallAt5: recallAtK(retrieved, relevant, 5),
+        mrr: reciprocalRank(retrieved, relevant),
+        ndcgAt5: ndcgAtK(retrieved, relevanceScores, 5),
+    };
+}
+/**
+ * Average metrics across multiple queries.
+ */
+export function averageMetrics(results) {
+    const n = results.length;
+    if (n === 0) {
+        return { precisionAt1: 0, precisionAt3: 0, precisionAt5: 0, recallAt5: 0, mrr: 0, ndcgAt5: 0 };
+    }
+    const sum = results.reduce((acc, r) => ({
+        precisionAt1: acc.precisionAt1 + r.precisionAt1,
+        precisionAt3: acc.precisionAt3 + r.precisionAt3,
+        precisionAt5: acc.precisionAt5 + r.precisionAt5,
+        recallAt5: acc.recallAt5 + r.recallAt5,
+        mrr: acc.mrr + r.mrr,
+        ndcgAt5: acc.ndcgAt5 + r.ndcgAt5,
+    }), { precisionAt1: 0, precisionAt3: 0, precisionAt5: 0, recallAt5: 0, mrr: 0, ndcgAt5: 0 });
+    return {
+        precisionAt1: sum.precisionAt1 / n,
+        precisionAt3: sum.precisionAt3 / n,
+        precisionAt5: sum.precisionAt5 / n,
+        recallAt5: sum.recallAt5 / n,
+        mrr: sum.mrr / n,
+        ndcgAt5: sum.ndcgAt5 / n,
+    };
+}
+//# sourceMappingURL=ir-metrics.js.map

package/dist/benchmarks/evaluators/ir-metrics.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"ir-metrics.js","sourceRoot":"","sources":["../../../benchmarks/evaluators/ir-metrics.ts"],"names":[],"mappings":"AAAA;;GAEG;AACH,MAAM,UAAU,YAAY,CAC1B,SAAmB,EACnB,QAAqB,EACrB,CAAS;IAET,MAAM,IAAI,GAAG,SAAS,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IACnC,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,CAAC,CAAC;IAChC,MAAM,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,EAAE,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC;IACxD,OAAO,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC;AAC5B,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,SAAS,CACvB,SAAmB,EACnB,QAAqB,EACrB,CAAS;IAET,IAAI,QAAQ,CAAC,IAAI,KAAK,CAAC;QAAE,OAAO,CAAC,CAAC,CAAC,8CAA8C;IACjF,MAAM,IAAI,GAAG,SAAS,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IACnC,MAAM,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,EAAE,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC;IACxD,OAAO,IAAI,GAAG,QAAQ,CAAC,IAAI,CAAC;AAC9B,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,cAAc,CAC5B,SAAmB,EACnB,QAAqB;IAErB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,SAAS,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QAC1C,IAAI,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAE,CAAC;YAAE,OAAO,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IACtD,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAED;;GAEG;AACH,SAAS,MAAM,CACb,SAAmB,EACnB,eAAoC,EACpC,CAAS;IAET,IAAI,GAAG,GAAG,CAAC,CAAC;IACZ,MAAM,IAAI,GAAG,SAAS,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IACnC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,MAAM,GAAG,GAAG,eAAe,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,CAAE,CAAC,IAAI,CAAC,CAAC;QAC/C,GAAG,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,wBAAwB;IAC5E,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,OAAO,CACrB,SAAmB,EACnB,eAAoC,EACpC,CAAS;IAET,MAAM,GAAG,GAAG,MAAM,CAAC,SAAS,EAAE,eAAe,EAAE,CAAC,CAAC,CAAC;IAClD,uDAAuD;IACvD,MAAM,UAAU,GAAG,CAAC,GAAG,eAAe,CAAC,OAAO,EAAE,CAAC;SAC9C,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC;SAC7B,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;IACrB,MAAM,IAAI,GAAG,MAAM,CAAC,UAAU,EAAE,eAAe,EAAE,CAAC,CAAC,CAAC;IACpD,IAAI,IAAI,KAAK,CAAC;QAAE,OAAO,CAAC,CAAC;IACzB,OAAO,GAAG,GAAG,IAAI,CAAC;AACpB,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,cAAc,CAC5B,SAAmB,EACnB,eAAoC,EACpC,iBAAiB,GAAG,CAAC;IASrB,MAAM,QAAQ,GAAG,IAAI,GAAG,CACtB,CAAC,GAAG,eAAe,CAAC,OAAO,EAAE,CAAC;SAC3B,MAAM,CAAC,CAAC,CAAC,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,KAAK,IAAI,iBAAiB,CAAC;SACjD,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,EAAE,CAAC,CACrB,CAAC;IAEF,OAAO;QACL,YAAY,EAAE,YAAY,CAAC,SAAS,EAAE,QAAQ,EAAE,CAAC,CAAC;QAClD,YAAY,EAAE,YAAY,CAAC,SAAS,EAAE,QAAQ,EAAE,CAAC,CAAC;QAClD,YAAY,EAAE,YAAY,CAAC,SAAS,EAAE,QAAQ,EAAE,CAAC,CAAC;QAClD,SAAS,EAAE,SAAS,CAAC,SAAS,EAAE,QAAQ,EAAE,CAAC,CAAC;QAC5C,GAAG,EAAE,cAAc,CAAC,SAAS,EAAE,QAAQ,CAAC;QACxC,OAAO,EAAE,OAAO,CAAC,SAAS,EAAE,eAAe,EAAE,CAAC,CAAC;KAChD,CAAC;AACJ,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,cAAc,CAC5B,OAAiD;IAEjD,MAAM,CAAC,GAAG,OAAO,CAAC,MAAM,CAAC;IACzB,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;QACZ,OAAO,EAAE,YAAY,EAAE,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,CAAC;IACjG,CAAC;IAED,MAAM,GAAG,GAAG,OAAO,CAAC,MAAM,CACxB,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC;QACX,YAAY,EAAE,GAAG,CAAC,YAAY,GAAG,CAAC,CAAC,YAAY;QAC/C,YAAY,EAAE,GAAG,CAAC,YAAY,GAAG,CAAC,CAAC,YAAY;QAC/C,YAAY,EAAE,GAAG,CAAC,YAAY,GAAG,CAAC,CAAC,YAAY;QAC/C,SAAS,EAAE,GAAG,CAAC,SAAS,GAAG,CAAC,CAAC,SAAS;QACtC,GAAG,EAAE,GAAG,CAAC,GAAG,GAAG,CAAC,CAAC,GAAG;QACpB,OAAO,EAAE,GAAG,CAAC,OAAO,GAAG,CAAC,CAAC,OAAO;KACjC,CAAC,EACF,EAAE,YAAY,EAAE,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,CACxF,CAAC;IAEF,OAAO;QACL,YAAY,EAAE,GAAG,CAAC,YAAY,GAAG,CAAC;QAClC,YAAY,EAAE,GAAG,CAAC,YAAY,GAAG,CAAC;QAClC,YAAY,EAAE,GAAG,CAAC,YAAY,GAAG,CAAC;QAClC,SAAS,EAAE,GAAG,CAAC,SAAS,GAAG,CAAC;QAC5B,GAAG,EAAE,GAAG,CAAC,GAAG,GAAG,CAAC;QAChB,OAAO,EAAE,GAAG,CAAC,OAAO,GAAG,CAAC;KACzB,CAAC;AACJ,CAAC"}

package/dist/benchmarks/evaluators/structured-judge.d.ts

@@ -0,0 +1,34 @@
+/**
+ * Structured Judge — evaluates AI output with specific yes/no questions per criterion.
+ *
+ * Instead of "rate 0-10", asks: "Does the code implement X? YES/NO"
+ * Then scores = (yes_count / total_questions) * 10.
+ */
+import Anthropic from "@anthropic-ai/sdk";
+import type { CodingTask } from "../lib/types.js";
+export interface JudgmentResult {
+    taskId: string;
+    totalQuestions: number;
+    yesCount: number;
+    score: number;
+    details: Array<{
+        criterion: string;
+        answer: "YES" | "NO" | "PARTIAL";
+        reasoning: string;
+    }>;
+}
+/**
+ * Evaluate an AI-generated output against structured criteria.
+ */
+export declare function structuredJudge(client: Anthropic, task: CodingTask, output: string, contextDocs: string): Promise<JudgmentResult>;
+/**
+ * Check if generated code contains valid TypeScript/JSX syntax.
+ * Uses the TypeScript compiler API in syntax-only mode for accurate parsing
+ * of template literals, JSX, and other complex syntax.
+ */
+export declare function checkTypeScriptSyntax(output: string): {
+    valid: boolean;
+    errorCount: number;
+    codeBlockCount: number;
+};
+//# sourceMappingURL=structured-judge.d.ts.map

package/dist/benchmarks/evaluators/structured-judge.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"structured-judge.d.ts","sourceRoot":"","sources":["../../../benchmarks/evaluators/structured-judge.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AACH,OAAO,SAAS,MAAM,mBAAmB,CAAC;AAC1C,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AAIlD,MAAM,WAAW,cAAc;IAC7B,MAAM,EAAE,MAAM,CAAC;IACf,cAAc,EAAE,MAAM,CAAC;IACvB,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,KAAK,CAAC;QACb,SAAS,EAAE,MAAM,CAAC;QAClB,MAAM,EAAE,KAAK,GAAG,IAAI,GAAG,SAAS,CAAC;QACjC,SAAS,EAAE,MAAM,CAAC;KACnB,CAAC,CAAC;CACJ;AAiCD;;GAEG;AACH,wBAAsB,eAAe,CACnC,MAAM,EAAE,SAAS,EACjB,IAAI,EAAE,UAAU,EAChB,MAAM,EAAE,MAAM,EACd,WAAW,EAAE,MAAM,GAClB,OAAO,CAAC,cAAc,CAAC,CAsGzB;AAED;;;;GAIG;AACH,wBAAgB,qBAAqB,CACnC,MAAM,EAAE,MAAM,GACb;IAAE,KAAK,EAAE,OAAO,CAAC;IAAC,UAAU,EAAE,MAAM,CAAC;IAAC,cAAc,EAAE,MAAM,CAAA;CAAE,CAgDhE"}