@modular-prompt/experiment 0.3.0 → 0.3.2

package/README.md

@@ -1,412 +1,79 @@
 # @modular-prompt/experiment
 
-Experiment framework for comparing and evaluating modular prompt modules.
+プロンプトモジュールの比較・評価フレームワーク。
 
-## Overview
-
-This framework provides tools to compare and evaluate different prompt module variations under identical conditions. It integrates with the `@modular-prompt/core` system to test multiple prompt variations and evaluate their output quality.
-
-### Use Cases
-
-- **Prompt Engineering**: Validate the effectiveness of new prompt structures
-- **Module Separation**: Verify that modularized prompts produce equivalent outputs
-- **Quality Evaluation**: Assess output stability and consistency through repeated executions
-- **Multi-Model Testing**: Test across different LLM providers (MLX, VertexAI, GoogleGenAI, etc.)
-
-## Features
-
-- ✅ **Dynamic Module Loading**: Load prompt modules from external files or inline definitions
-- ✅ **Flexible Evaluators**: Support both code-based and AI-based evaluation
-- ✅ **Statistical Analysis**: Analyze success rates, execution times, and output consistency
-- ✅ **Prompt Diff Detection**: Automatically detect differences between module outputs
-- ✅ **Driver Caching**: Reuse drivers for improved memory efficiency
-- ✅ **Detailed Logging**: Comprehensive logging of all executions
-
-## Installation
+## インストール
 
 ```bash
-pnpm add @modular-prompt/experiment
+npm install @modular-prompt/experiment
 ```
 
-## Quick Start
+## 概要
 
-### 1. Create Configuration File
+複数のプロンプトモジュールを同一条件下で比較・評価する。YAML設定で実験を定義し、CLIで実行。
 
-You can use either YAML or TypeScript format.
+- **プロンプト比較**: 異なるプロンプト構造の効果を定量的に比較
+- **マルチモデルテスト**: 異なるLLMプロバイダーでの動作比較
+- **品質評価**: 繰り返し実行による安定性・一貫性の評価
+- **柔軟な評価器**: コードベース・AIベースの評価をサポート
 
-#### Option A: YAML Configuration (Recommended for static configurations)
+## クイックスタート
 
-Create `examples/experiment.yaml`:
+### 1. 設定ファイルを作成
 
 ```yaml
+# experiment.yaml
 models:
-  gemini-fast:
-    provider: vertexai
-    model: gemini-2.0-flash-exp
-    capabilities: ["tools", "fast"]
-    enabled: true
+  gpt4o:
+    provider: openai
+    model: gpt-4o
 
 drivers:
-  vertexai:
-    projectId: your-project-id
-    location: us-central1
-    # Paths are resolved relative to this config file
-    # Can use ~/ for home directory or absolute paths
-    credentialsPath: ./credentials.json
+  openai:
+    apiKey: ${OPENAI_API_KEY}
 
 modules:
   - name: my-module
     path: ./my-module.ts
-    description: My custom prompt module
 
 testCases:
-  - name: Basic Test
-    description: Test basic functionality
-    input:  # Structured context object (passed to module.compile)
-      query: user question
-      context: additional information
-    models:  # Optional: specify which models to test (uses all enabled if not specified)
-      - gemini-fast
-
-evaluators:
-  # Built-in evaluators (name only)
-  - name: structured-output-presence
-  - name: llm-requirement-fulfillment
-  # Or external evaluator (with path)
-  - name: custom-validator
-    path: ./evaluators/custom-validator.ts
-  # Or inline prompt evaluator
-  - name: quality-check
-    prompt:
-      objective:
-        - Evaluate output quality
-      instructions:
-        - Check clarity and accuracy
-
-evaluation:
-  enabled: true
-  model: gemini-fast  # Reference by model name
-```
-
-#### Option B: TypeScript Configuration (For dynamic configurations)
-
-Create `examples/experiment.ts`:
-
-```typescript
-export default {
-  models: {
-    'gemini-fast': {
-      provider: 'vertexai',
-      model: 'gemini-2.0-flash-exp',
-      capabilities: ['tools', 'fast'],
-      enabled: true,
-    },
-  },
-  drivers: {
-    vertexai: {
-      projectId: 'your-project-id',
-      location: 'us-central1',
-      credentialsPath: './credentials.json',
-    },
-  },
-  modules: [
-    {
-      name: 'my-module',
-      path: './my-module.ts',
-      description: 'My custom prompt module',
-    },
-  ],
-  testCases: [
-    {
-      name: 'Basic Test',
-      description: 'Test basic functionality',
-      input: {  // Structured context object
-        query: 'user question',
-        options: { temperature: 0.7 },
-      },
-      models: ['gemini-fast'],  // Optional
-    },
-  ],
-  evaluators: [
-    // Built-in evaluators (name only)
-    { name: 'structured-output-presence' },
-    { name: 'llm-requirement-fulfillment' },
-    // Or external evaluator (with path)
-    {
-      name: 'custom-validator',
-      path: './evaluators/custom-validator.ts',
-    },
-  ],
-  evaluation: {
-    enabled: true,
-    model: 'gemini-fast',  // Reference by model name
-  },
-};
-```
-
-**TypeScript Support**: TypeScript configuration files are automatically transpiled using [jiti](https://github.com/unjs/jiti). You can use TypeScript syntax directly without pre-compilation. Type annotations are stripped automatically, and the file is executed as JavaScript.
-
-**Important**: All file paths in the configuration (modules, evaluators, credentials) are resolved relative to the config file location.
-
-### 2. Run Experiment
-
-```bash
-# Validate configuration and display execution plan (recommended first step)
-npx modular-experiment examples/experiment.yaml --dry-run
-
-# Run with YAML config
-npx modular-experiment examples/experiment.yaml
-
-# Run with TypeScript config
-npx modular-experiment examples/experiment.ts
-
-# Run specific module
-npx modular-experiment examples/experiment.yaml --modules my-module
-
-# Run with evaluation
-npx modular-experiment examples/experiment.yaml --evaluate
-
-# Run multiple times for statistics
-npx modular-experiment examples/experiment.yaml --repeat 10
-
-# Run with detailed logging to JSONL file
-npx modular-experiment examples/experiment.yaml --log-file experiment.jsonl
-
-# Run with verbose output (show internal operations)
-npx modular-experiment examples/experiment.yaml --verbose
-
-# Combine options
-npx modular-experiment examples/experiment.yaml --evaluate --log-file experiment.jsonl --verbose
-```
-
-## Configuration
-
-### Module Definition
-
-Modules can be defined inline or loaded from external files:
-
-```typescript
-// External file
-export const modules: ModuleReference[] = [
-  {
-    name: 'my-module',
-    path: './modules/my-module.ts',
-    description: 'Description',
-  },
-];
-```
-
-A module file should export a default object with:
-
-```typescript
-import { compile } from '@modular-prompt/core';
-import { myPromptModule } from './prompts.js';
+  - name: 基本テスト
+    input:
+      query: "TypeScriptについて説明して"
 
-export default {
-  name: 'My Module',
-  description: 'Module description',
-  compile: (context: any) => compile(myPromptModule, context),
-};
+evaluators: []
 ```
 
-### Evaluator Definition
-
-Two types of evaluators are supported:
-
-#### 1. Code Evaluator
-
-Programmatic validation (e.g., JSON structure validation):
-
-```typescript
-import type { CodeEvaluator, EvaluationContext, EvaluationResult } from '@modular-prompt/experiment';
-
-export default {
-  name: 'JSON Validator',
-  description: 'Validates JSON structure in output',
-
-  async evaluate(context: EvaluationContext): Promise<EvaluationResult> {
-    // Validation logic
-    return {
-      evaluator: 'json-validator',
-      moduleName: context.moduleName,
-      score: 10,
-      reasoning: 'Valid JSON structure',
-    };
-  },
-} satisfies CodeEvaluator;
-```
-
-#### 2. Prompt Evaluator
-
-AI-based evaluation using LLM:
+### 2. モジュールファイルを作成
 
 ```typescript
-import type { PromptEvaluator, EvaluationContext } from '@modular-prompt/experiment';
+// my-module.ts
 import type { PromptModule } from '@modular-prompt/core';
 
-const evaluationModule: PromptModule<EvaluationContext> = {
-  createContext: (): EvaluationContext => ({
-    moduleName: '',
-    prompt: '',
-    runs: [],
-  }),
-
-  objective: [
-    '- Assess output quality',
-  ],
-
+const module: PromptModule<{ query: string }> = {
+  objective: ['ユーザーの質問に回答する'],
   instructions: [
-    '- Evaluate clarity and accuracy',
+    (ctx) => `質問: ${ctx.query}`,
   ],
 };
 
-export default {
-  name: 'Quality Evaluator',
-  description: 'Evaluates output quality',
-  module: evaluationModule,
-} satisfies PromptEvaluator;
+export default module;
 ```
 
-All prompt evaluators are automatically merged with the base evaluation module.
-
-## Built-in Evaluators
-
-The framework includes built-in evaluators that can be referenced by name only (no path required):
-
-### structured-output-presence
-
-- **Type**: Code Evaluator
-- **What it measures**: Checks if `structuredOutput` exists and is a valid object
-- **Evaluation logic**:
-  - Verifies presence of `structuredOutput` in query result
-  - Confirms it's a non-null object type
-- **Score**: `(validCount / totalRuns) * 10`
-- **Use case**: Verify that the model returns structured JSON output (essential for structured output workflows)
-- **Usage**:
-  ```yaml
-  evaluators:
-    - name: "structured-output-presence"
-  ```
-
-### llm-requirement-fulfillment
-
-- **Type**: Prompt Evaluator (uses LLM for evaluation)
-- **What it measures**: Uses LLM to comprehensively evaluate whether output meets functional requirements
-- **Evaluation criteria**:
-  1. **Requirement Fulfillment**: Does it satisfy the intent described in the prompt?
-  2. **Parameter Correctness**: Are all required parameters present and correct?
-  3. **Parameter Completeness**: Are optional parameters appropriately used or omitted?
-  4. **Logical Consistency**: Is the output logically consistent with the facts?
-- **Score**: 0-10 overall score with detailed sub-scores for each criterion
-- **Use case**: Comprehensive quality assessment of output (requires evaluation model to be configured)
-- **Usage**:
-  ```yaml
-  evaluators:
-    - name: "llm-requirement-fulfillment"
-
-  evaluation:
-    enabled: true
-    model: "gemini-fast"  # Model used for evaluation
-  ```
-
-**Note**: `llm-requirement-fulfillment` requires an evaluation model to be configured in the `evaluation` section.
-
-## Architecture
-
-```
-┌─────────────────────────────────────────┐
-│ run-comparison.ts (CLI Entry Point)    │
-└─────────────────────────────────────────┘
-              │
-    ┌─────────┼─────────┐
-    ▼         ▼         ▼
-┌────────┐ ┌────────┐ ┌────────┐
-│ Config │ │ Runner │ │Reporter│
-│ Loader │ │        │ │        │
-└────────┘ └────────┘ └────────┘
-    │         │
-    ▼         ▼
-┌────────┐ ┌────────┐
-│Dynamic │ │Driver  │
-│Loader  │ │Manager │
-└────────┘ └────────┘
-```
-
-### Components
-
-| Component | Responsibility |
-|-----------|----------------|
-| `config/loader.ts` | Load YAML configuration |
-| `config/dynamic-loader.ts` | Dynamic module/evaluator loading |
-| `runner/experiment.ts` | Orchestrate experiment execution |
-| `runner/evaluator.ts` | Execute evaluations |
-| `runner/driver-manager.ts` | Cache and manage AI drivers |
-| `reporter/statistics.ts` | Generate statistical reports |
-| `base-evaluation-module.ts` | Base evaluation prompt module |
-| `evaluators/index.ts` | Built-in evaluator registry |
-
-## Examples
-
-See `examples/experiment.yaml` for a complete configuration template with:
-- Model definitions (MLX, Vertex AI, Google GenAI)
-- Driver configurations with credential paths
-- Evaluation settings
-- Empty sections for modules, test cases, and evaluators (ready for your content)
-
-## API
-
-### Programmatic Usage
-
-```typescript
-import {
-  loadExperimentConfig,
-  loadModules,
-  loadEvaluators,
-  ExperimentRunner,
-  DriverManager,
-} from '@modular-prompt/experiment';
-
-const { serverConfig, aiService } = loadExperimentConfig('config.yaml');
-const modules = await loadModules(moduleRefs, basePath);
-const evaluators = await loadEvaluators(evaluatorRefs, basePath);
-
-const driverManager = new DriverManager();
-const runner = new ExperimentRunner(
-  aiService,
-  driverManager,
-  modules,
-  testCases,
-  models,
-  repeatCount,
-  evaluators,
-  evaluatorModel
-);
-
-const results = await runner.run();
-await driverManager.cleanup();
-```
-
-## CLI Options
+### 3. 実行
 
+```bash
+npx modular-experiment experiment.yaml --dry-run    # 確認
+npx modular-experiment experiment.yaml              # 実行
+npx modular-experiment experiment.yaml --evaluate   # 評価付き
+npx modular-experiment experiment.yaml --repeat 10  # 複数回実行
 ```
-Usage: modular-experiment <config> [options]
 
-Arguments:
-  <config>                Config file path (YAML or TypeScript)
+設定ファイルの詳細、評価器の書き方、プログラマティックAPIについては `skills/experiment/SKILL.md` を参照。
 
-Options:
-  --test-case <name>      Test case name filter
-  --model <provider>      Model provider filter
-  --modules <names>       Comma-separated module names (default: all)
-  --repeat <count>        Number of repetitions (default: 1)
-  --evaluate              Enable evaluation phase
-  --evaluators <names>    Comma-separated evaluator names (default: all)
-  --dry-run               Display execution plan without running the experiment
-  --log-file <path>       Log file path for JSONL output (detailed logs)
-  --verbose               Enable verbose output (show detailed internal operations)
-```
+## Skills (for Claude Code)
 
-**Note**: All paths specified in the config file are resolved relative to the config file's directory.
+This package includes `skills/experiment/SKILL.md`. It can be used as a Claude Code skill to guide experiment framework usage.
 
 ## License
 

package/dist/config/dynamic-loader.js

@@ -105,8 +105,8 @@ export async function loadModules(refs, basePath) {
         }
         modules.push({
             name: ref.name,
-            description: ref.description || module.description || '',
-            compile: module.compile,
+            description: ref.description || '',
+            module: module,
         });
     }
     return modules;

package/dist/config/dynamic-loader.js.map

@@ -1 +1 @@
-{"version":3,"file":"dynamic-loader.js","sourceRoot":"","sources":["../../src/config/dynamic-loader.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,EAAE,KAAK,EAAE,MAAM,sBAAsB,CAAC;AAC7C,OAAO,EAAE,aAAa,EAAE,MAAM,KAAK,CAAC;AACpC,OAAO,EAAE,OAAO,EAAE,MAAM,MAAM,CAAC;AAO/B,OAAO,EAAE,oBAAoB,EAAE,MAAM,8BAA8B,CAAC;AACpE,OAAO,EAAE,mBAAmB,EAAE,MAAM,wBAAwB,CAAC;AAC7D,OAAO,EAAE,MAAM,IAAI,UAAU,EAAE,MAAM,cAAc,CAAC;AAEpD,MAAM,MAAM,GAAG,UAAU,CAAC,OAAO,CAAC,gBAAgB,CAAC,CAAC;AAapD;;;;;;GAMG;AACH,MAAM,CAAC,KAAK,UAAU,cAAc,CAClC,IAA0B,EAC1B,QAAgB;IAEhB,MAAM,UAAU,GAAsB,EAAE,CAAC;IAEzC,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACvB,IAAI,SAAsD,CAAC;QAE3D,IAAI,MAAM,IAAI,GAAG,EAAE,CAAC;YAClB,gBAAgB;YAChB,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC,IAAI,CAAC,CAAC;YAC7C,MAAM,OAAO,GAAG,aAAa,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC;YAC7C,MAAM,QAAQ,GAAG,MAAM,MAAM,CAAC,OAAO,CAAC,CAAC;YACvC,SAAS,GAAG,QAAQ,CAAC,OAAO,CAAC;YAE7B,IAAI,CAAC,SAAS,EAAE,CAAC;gBACf,MAAM,CAAC,IAAI,CAAC,wBAAwB,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC;gBAChD,SAAS;YACX,CAAC;QACH,CAAC;aAAM,IAAI,QAAQ,IAAI,GAAG,EAAE,CAAC;YAC3B,oDAAoD;YACpD,MAAM,YAAY,GAAG,KAAK,CAAC,oBAAoB,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;YAC7D,UAAU,CAAC,IAAI,CAAC;gBACd,IAAI,EAAE,GAAG,CAAC,IAAI;gBACd,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,EAAE;gBAClC,IAAI,EAAE,QAAQ;gBACd,eAAe,EAAE;oBACf,IAAI,EAAE,GAAG,CAAC,IAAI;oBACd,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,EAAE;oBAClC,MAAM,EAAE,YAAY;iBACrB;aACF,CAAC,CAAC;YACH,SAAS;QACX,CAAC;aAAM,CAAC;YACN,gCAAgC;YAChC,SAAS,GAAG,mBAAmB,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;YAE1C,IAAI,CAAC,SAAS,EAAE,CAAC;gBACf,MAAM,CAAC,IAAI,CAAC,gCAAgC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC;gBACxD,SAAS;YACX,CAAC;QACH,CAAC;QAED,qCAAqC;QACrC,IAAI,UAAU,IAAI,SAAS,IAAI,OAAO,SAAS,CAAC,QAAQ,KAAK,UAAU,EAAE,CAAC;YACxE,iBAAiB;YACjB,UAAU,CAAC,IAAI,CAAC;gBACd,IAAI,EAAE,GAAG,CAAC,IAAI;gBACd,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,SAAS,CAAC,WAAW,IAAI,EAAE;gBAC3D,IAAI,EAAE,MAAM;gBACZ,aAAa,EAAE,SAA0B;aAC1C,CAAC,CAAC;QACL,CAAC;aAAM,IAAI,QAAQ,IAAI,SAAS,EAAE,CAAC;YACjC,4CAA4C;YAC5C,MAAM,YAAY,GAAG,KAAK,CAAC,oBAAoB,EAAE,SAAS,CAAC,MAAM,CAAC,CAAC;YACnE,UAAU,CAAC,IAAI,CAAC;gBACd,IAAI,EAAE,GAAG,CAAC,IAAI;gBACd,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,SAAS,CAAC,WAAW,IAAI,EAAE;gBAC3D,IAAI,EAAE,QAAQ;gBACd,eAAe,EAAE;oBACf,IAAI,EAAE,SAAS,CAAC,IAAI;oBACpB,WAAW,EAAE,SAAS,CAAC,WAAW;oBAClC,MAAM,EAAE,YAAY;iBACrB;aACF,CAAC,CAAC;QACL,CAAC;aAAM,CAAC;YACN,MAAM,CAAC,IAAI,CAAC,2BAA2B,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC;QACrD,CAAC;IACH,CAAC;IAED,OAAO,UAAU,CAAC;AACpB,CAAC;AAQD;;;;;;GAMG;AACH,MAAM,CAAC,KAAK,UAAU,WAAW,CAC/B,IAAuB,EACvB,QAAgB;IAEhB,MAAM,OAAO,GAAuB,EAAE,CAAC;IAEvC,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACvB,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC,IAAI,CAAC,CAAC;QAC7C,MAAM,OAAO,GAAG,aAAa,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC;QAC7C,MAAM,QAAQ,GAAG,MAAM,MAAM,CAAC,OAAO,CAAC,CAAC;QACvC,MAAM,MAAM,GAAG,QAAQ,CAAC,OAAO,CAAC;QAEhC,IAAI,CAAC,MAAM,EAAE,CAAC;YACZ,MAAM,CAAC,IAAI,CAAC,wBAAwB,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC;YAChD,SAAS;QACX,CAAC;QAED,OAAO,CAAC,IAAI,CAAC;YACX,IAAI,EAAE,GAAG,CAAC,IAAI;YACd,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,MAAM,CAAC,WAAW,IAAI,EAAE;YACxD,OAAO,EAAE,MAAM,CAAC,OAAO;SACxB,CAAC,CAAC;IACL,CAAC;IAED,OAAO,OAAO,CAAC;AACjB,CAAC"}
+{"version":3,"file":"dynamic-loader.js","sourceRoot":"","sources":["../../src/config/dynamic-loader.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,EAAE,KAAK,EAAE,MAAM,sBAAsB,CAAC;AAC7C,OAAO,EAAE,aAAa,EAAE,MAAM,KAAK,CAAC;AACpC,OAAO,EAAE,OAAO,EAAE,MAAM,MAAM,CAAC;AAO/B,OAAO,EAAE,oBAAoB,EAAE,MAAM,8BAA8B,CAAC;AACpE,OAAO,EAAE,mBAAmB,EAAE,MAAM,wBAAwB,CAAC;AAC7D,OAAO,EAAE,MAAM,IAAI,UAAU,EAAE,MAAM,cAAc,CAAC;AAEpD,MAAM,MAAM,GAAG,UAAU,CAAC,OAAO,CAAC,gBAAgB,CAAC,CAAC;AAapD;;;;;;GAMG;AACH,MAAM,CAAC,KAAK,UAAU,cAAc,CAClC,IAA0B,EAC1B,QAAgB;IAEhB,MAAM,UAAU,GAAsB,EAAE,CAAC;IAEzC,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACvB,IAAI,SAAsD,CAAC;QAE3D,IAAI,MAAM,IAAI,GAAG,EAAE,CAAC;YAClB,gBAAgB;YAChB,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC,IAAI,CAAC,CAAC;YAC7C,MAAM,OAAO,GAAG,aAAa,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC;YAC7C,MAAM,QAAQ,GAAG,MAAM,MAAM,CAAC,OAAO,CAAC,CAAC;YACvC,SAAS,GAAG,QAAQ,CAAC,OAAO,CAAC;YAE7B,IAAI,CAAC,SAAS,EAAE,CAAC;gBACf,MAAM,CAAC,IAAI,CAAC,wBAAwB,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC;gBAChD,SAAS;YACX,CAAC;QACH,CAAC;aAAM,IAAI,QAAQ,IAAI,GAAG,EAAE,CAAC;YAC3B,oDAAoD;YACpD,MAAM,YAAY,GAAG,KAAK,CAAC,oBAAoB,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;YAC7D,UAAU,CAAC,IAAI,CAAC;gBACd,IAAI,EAAE,GAAG,CAAC,IAAI;gBACd,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,EAAE;gBAClC,IAAI,EAAE,QAAQ;gBACd,eAAe,EAAE;oBACf,IAAI,EAAE,GAAG,CAAC,IAAI;oBACd,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,EAAE;oBAClC,MAAM,EAAE,YAAY;iBACrB;aACF,CAAC,CAAC;YACH,SAAS;QACX,CAAC;aAAM,CAAC;YACN,gCAAgC;YAChC,SAAS,GAAG,mBAAmB,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;YAE1C,IAAI,CAAC,SAAS,EAAE,CAAC;gBACf,MAAM,CAAC,IAAI,CAAC,gCAAgC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC;gBACxD,SAAS;YACX,CAAC;QACH,CAAC;QAED,qCAAqC;QACrC,IAAI,UAAU,IAAI,SAAS,IAAI,OAAO,SAAS,CAAC,QAAQ,KAAK,UAAU,EAAE,CAAC;YACxE,iBAAiB;YACjB,UAAU,CAAC,IAAI,CAAC;gBACd,IAAI,EAAE,GAAG,CAAC,IAAI;gBACd,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,SAAS,CAAC,WAAW,IAAI,EAAE;gBAC3D,IAAI,EAAE,MAAM;gBACZ,aAAa,EAAE,SAA0B;aAC1C,CAAC,CAAC;QACL,CAAC;aAAM,IAAI,QAAQ,IAAI,SAAS,EAAE,CAAC;YACjC,4CAA4C;YAC5C,MAAM,YAAY,GAAG,KAAK,CAAC,oBAAoB,EAAE,SAAS,CAAC,MAAM,CAAC,CAAC;YACnE,UAAU,CAAC,IAAI,CAAC;gBACd,IAAI,EAAE,GAAG,CAAC,IAAI;gBACd,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,SAAS,CAAC,WAAW,IAAI,EAAE;gBAC3D,IAAI,EAAE,QAAQ;gBACd,eAAe,EAAE;oBACf,IAAI,EAAE,SAAS,CAAC,IAAI;oBACpB,WAAW,EAAE,SAAS,CAAC,WAAW;oBAClC,MAAM,EAAE,YAAY;iBACrB;aACF,CAAC,CAAC;QACL,CAAC;aAAM,CAAC;YACN,MAAM,CAAC,IAAI,CAAC,2BAA2B,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC;QACrD,CAAC;IACH,CAAC;IAED,OAAO,UAAU,CAAC;AACpB,CAAC;AAQD;;;;;;GAMG;AACH,MAAM,CAAC,KAAK,UAAU,WAAW,CAC/B,IAAuB,EACvB,QAAgB;IAEhB,MAAM,OAAO,GAAuB,EAAE,CAAC;IAEvC,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACvB,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC,IAAI,CAAC,CAAC;QAC7C,MAAM,OAAO,GAAG,aAAa,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC;QAC7C,MAAM,QAAQ,GAAG,MAAM,MAAM,CAAC,OAAO,CAAC,CAAC;QACvC,MAAM,MAAM,GAAG,QAAQ,CAAC,OAAO,CAAC;QAEhC,IAAI,CAAC,MAAM,EAAE,CAAC;YACZ,MAAM,CAAC,IAAI,CAAC,wBAAwB,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC;YAChD,SAAS;QACX,CAAC;QAED,OAAO,CAAC,IAAI,CAAC;YACX,IAAI,EAAE,GAAG,CAAC,IAAI;YACd,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,EAAE;YAClC,MAAM,EAAE,MAAM;SACf,CAAC,CAAC;IACL,CAAC;IAED,OAAO,OAAO,CAAC;AACjB,CAAC"}

package/dist/runner/experiment.d.ts

@@ -24,6 +24,14 @@ export declare class ExperimentRunner {
      * @returns Array of TestResult
      */
     run(): Promise<TestResult[]>;
+    /**
+     * Build test plan: expand all testCase × model × module combinations
+     */
+    private buildTestPlan;
+    /**
+     * Execute test plan grouped by model
+     */
+    private executePlan;
     /**
      * Run module test with multiple repetitions
      */
@@ -32,9 +40,5 @@ export declare class ExperimentRunner {
      * Run evaluation phase
      */
     private runEvaluationPhase;
-    /**
-     * Compare prompts across modules
-     */
-    private comparePrompts;
 }
 //# sourceMappingURL=experiment.d.ts.map

package/dist/runner/experiment.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"experiment.d.ts","sourceRoot":"","sources":["../../src/runner/experiment.ts"],"names":[],"mappings":"AAAA;;GAEG;AAGH,OAAO,KAAK,EAAE,SAAS,EAAe,SAAS,EAAE,MAAM,wBAAwB,CAAC;AAChF,OAAO,KAAK,EAAE,gBAAgB,EAAE,UAAU,EAAE,QAAQ,EAAuC,MAAM,aAAa,CAAC;AAC/G,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,qBAAqB,CAAC;AACzD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAC;AAMnE,qBAAa,gBAAgB;IAEzB,OAAO,CAAC,SAAS;IACjB,OAAO,CAAC,aAAa;IACrB,OAAO,CAAC,OAAO;IACf,OAAO,CAAC,SAAS;IACjB,OAAO,CAAC,MAAM;IACd,OAAO,CAAC,WAAW;IACnB,OAAO,CAAC,UAAU,CAAC;IACnB,OAAO,CAAC,cAAc,CAAC;gBAPf,SAAS,EAAE,SAAS,EACpB,aAAa,EAAE,aAAa,EAC5B,OAAO,EAAE,gBAAgB,EAAE,EAC3B,SAAS,EAAE,QAAQ,EAAE,EACrB,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,SAAS,CAAC,EACjC,WAAW,EAAE,MAAM,EACnB,UAAU,CAAC,EAAE,eAAe,EAAE,YAAA,EAC9B,cAAc,CAAC,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,SAAS,CAAA;KAAE,YAAA;IAG5D;;;;OAIG;IACG,GAAG,IAAI,OAAO,CAAC,UAAU,EAAE,CAAC;IAkHlC;;OAEG;YACW,aAAa;IA8D3B;;OAEG;YACW,kBAAkB;IA2BhC;;OAEG;IACH,OAAO,CAAC,cAAc;CAiCvB"}
+{"version":3,"file":"experiment.d.ts","sourceRoot":"","sources":["../../src/runner/experiment.ts"],"names":[],"mappings":"AAAA;;GAEG;AAKH,OAAO,KAAK,EAAE,SAAS,EAAe,SAAS,EAAY,MAAM,wBAAwB,CAAC;AAE1F,OAAO,KAAK,EAAE,gBAAgB,EAAE,UAAU,EAAE,QAAQ,EAAuC,MAAM,aAAa,CAAC;AAC/G,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,qBAAqB,CAAC;AACzD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAC;AAenE,qBAAa,gBAAgB;IAEzB,OAAO,CAAC,SAAS;IACjB,OAAO,CAAC,aAAa;IACrB,OAAO,CAAC,OAAO;IACf,OAAO,CAAC,SAAS;IACjB,OAAO,CAAC,MAAM;IACd,OAAO,CAAC,WAAW;IACnB,OAAO,CAAC,UAAU,CAAC;IACnB,OAAO,CAAC,cAAc,CAAC;gBAPf,SAAS,EAAE,SAAS,EACpB,aAAa,EAAE,aAAa,EAC5B,OAAO,EAAE,gBAAgB,EAAE,EAC3B,SAAS,EAAE,QAAQ,EAAE,EACrB,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,SAAS,CAAC,EACjC,WAAW,EAAE,MAAM,EACnB,UAAU,CAAC,EAAE,eAAe,EAAE,YAAA,EAC9B,cAAc,CAAC,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,SAAS,CAAA;KAAE,YAAA;IAG5D;;;;OAIG;IACG,GAAG,IAAI,OAAO,CAAC,UAAU,EAAE,CAAC;IAmBlC;;OAEG;IACH,OAAO,CAAC,aAAa;IAyCrB;;OAEG;YACW,WAAW;IAmFzB;;OAEG;YACW,aAAa;IAuE3B;;OAEG;YACW,kBAAkB;CA2BjC"}