@modular-prompt/experiment 0.3.0 → 0.3.1

package/README.md

@@ -1,412 +1,75 @@
 # @modular-prompt/experiment
 
-Experiment framework for comparing and evaluating modular prompt modules.
+プロンプトモジュールの比較・評価フレームワーク。
 
-## Overview
-
-This framework provides tools to compare and evaluate different prompt module variations under identical conditions. It integrates with the `@modular-prompt/core` system to test multiple prompt variations and evaluate their output quality.
-
-### Use Cases
-
-- **Prompt Engineering**: Validate the effectiveness of new prompt structures
-- **Module Separation**: Verify that modularized prompts produce equivalent outputs
-- **Quality Evaluation**: Assess output stability and consistency through repeated executions
-- **Multi-Model Testing**: Test across different LLM providers (MLX, VertexAI, GoogleGenAI, etc.)
-
-## Features
-
-- ✅ **Dynamic Module Loading**: Load prompt modules from external files or inline definitions
-- ✅ **Flexible Evaluators**: Support both code-based and AI-based evaluation
-- ✅ **Statistical Analysis**: Analyze success rates, execution times, and output consistency
-- ✅ **Prompt Diff Detection**: Automatically detect differences between module outputs
-- ✅ **Driver Caching**: Reuse drivers for improved memory efficiency
-- ✅ **Detailed Logging**: Comprehensive logging of all executions
-
-## Installation
+## インストール
 
 ```bash
-pnpm add @modular-prompt/experiment
+npm install @modular-prompt/experiment
 ```
 
-## Quick Start
+## 概要
 
-### 1. Create Configuration File
+複数のプロンプトモジュールを同一条件下で比較・評価する。YAML設定で実験を定義し、CLIで実行。
 
-You can use either YAML or TypeScript format.
+- **プロンプト比較**: 異なるプロンプト構造の効果を定量的に比較
+- **マルチモデルテスト**: 異なるLLMプロバイダーでの動作比較
+- **品質評価**: 繰り返し実行による安定性・一貫性の評価
+- **柔軟な評価器**: コードベース・AIベースの評価をサポート
 
-#### Option A: YAML Configuration (Recommended for static configurations)
+## クイックスタート
 
-Create `examples/experiment.yaml`:
+### 1. 設定ファイルを作成
 
 ```yaml
+# experiment.yaml
 models:
-  gemini-fast:
-    provider: vertexai
-    model: gemini-2.0-flash-exp
-    capabilities: ["tools", "fast"]
-    enabled: true
+  gpt4o:
+    provider: openai
+    model: gpt-4o
 
 drivers:
-  vertexai:
-    projectId: your-project-id
-    location: us-central1
-    # Paths are resolved relative to this config file
-    # Can use ~/ for home directory or absolute paths
-    credentialsPath: ./credentials.json
+  openai:
+    apiKey: ${OPENAI_API_KEY}
 
 modules:
   - name: my-module
     path: ./my-module.ts
-    description: My custom prompt module
 
 testCases:
-  - name: Basic Test
-    description: Test basic functionality
-    input:  # Structured context object (passed to module.compile)
-      query: user question
-      context: additional information
-    models:  # Optional: specify which models to test (uses all enabled if not specified)
-      - gemini-fast
+  - name: 基本テスト
+    input:
+      query: "TypeScriptについて説明して"
 
-evaluators:
-  # Built-in evaluators (name only)
-  - name: structured-output-presence
-  - name: llm-requirement-fulfillment
-  # Or external evaluator (with path)
-  - name: custom-validator
-    path: ./evaluators/custom-validator.ts
-  # Or inline prompt evaluator
-  - name: quality-check
-    prompt:
-      objective:
-        - Evaluate output quality
-      instructions:
-        - Check clarity and accuracy
-
-evaluation:
-  enabled: true
-  model: gemini-fast  # Reference by model name
+evaluators: []
 ```
 
-#### Option B: TypeScript Configuration (For dynamic configurations)
-
-Create `examples/experiment.ts`:
-
-```typescript
-export default {
-  models: {
-    'gemini-fast': {
-      provider: 'vertexai',
-      model: 'gemini-2.0-flash-exp',
-      capabilities: ['tools', 'fast'],
-      enabled: true,
-    },
-  },
-  drivers: {
-    vertexai: {
-      projectId: 'your-project-id',
-      location: 'us-central1',
-      credentialsPath: './credentials.json',
-    },
-  },
-  modules: [
-    {
-      name: 'my-module',
-      path: './my-module.ts',
-      description: 'My custom prompt module',
-    },
-  ],
-  testCases: [
-    {
-      name: 'Basic Test',
-      description: 'Test basic functionality',
-      input: {  // Structured context object
-        query: 'user question',
-        options: { temperature: 0.7 },
-      },
-      models: ['gemini-fast'],  // Optional
-    },
-  ],
-  evaluators: [
-    // Built-in evaluators (name only)
-    { name: 'structured-output-presence' },
-    { name: 'llm-requirement-fulfillment' },
-    // Or external evaluator (with path)
-    {
-      name: 'custom-validator',
-      path: './evaluators/custom-validator.ts',
-    },
-  ],
-  evaluation: {
-    enabled: true,
-    model: 'gemini-fast',  // Reference by model name
-  },
-};
-```
-
-**TypeScript Support**: TypeScript configuration files are automatically transpiled using [jiti](https://github.com/unjs/jiti). You can use TypeScript syntax directly without pre-compilation. Type annotations are stripped automatically, and the file is executed as JavaScript.
-
-**Important**: All file paths in the configuration (modules, evaluators, credentials) are resolved relative to the config file location.
-
-### 2. Run Experiment
-
-```bash
-# Validate configuration and display execution plan (recommended first step)
-npx modular-experiment examples/experiment.yaml --dry-run
-
-# Run with YAML config
-npx modular-experiment examples/experiment.yaml
-
-# Run with TypeScript config
-npx modular-experiment examples/experiment.ts
-
-# Run specific module
-npx modular-experiment examples/experiment.yaml --modules my-module
-
-# Run with evaluation
-npx modular-experiment examples/experiment.yaml --evaluate
-
-# Run multiple times for statistics
-npx modular-experiment examples/experiment.yaml --repeat 10
-
-# Run with detailed logging to JSONL file
-npx modular-experiment examples/experiment.yaml --log-file experiment.jsonl
-
-# Run with verbose output (show internal operations)
-npx modular-experiment examples/experiment.yaml --verbose
-
-# Combine options
-npx modular-experiment examples/experiment.yaml --evaluate --log-file experiment.jsonl --verbose
-```
-
-## Configuration
-
-### Module Definition
-
-Modules can be defined inline or loaded from external files:
-
-```typescript
-// External file
-export const modules: ModuleReference[] = [
-  {
-    name: 'my-module',
-    path: './modules/my-module.ts',
-    description: 'Description',
-  },
-];
-```
-
-A module file should export a default object with:
+### 2. モジュールファイルを作成
 
 ```typescript
+// my-module.ts
 import { compile } from '@modular-prompt/core';
-import { myPromptModule } from './prompts.js';
 
 export default {
   name: 'My Module',
-  description: 'Module description',
   compile: (context: any) => compile(myPromptModule, context),
 };
 ```
 
-### Evaluator Definition
-
-Two types of evaluators are supported:
-
-#### 1. Code Evaluator
-
-Programmatic validation (e.g., JSON structure validation):
-
-```typescript
-import type { CodeEvaluator, EvaluationContext, EvaluationResult } from '@modular-prompt/experiment';
-
-export default {
-  name: 'JSON Validator',
-  description: 'Validates JSON structure in output',
-
-  async evaluate(context: EvaluationContext): Promise<EvaluationResult> {
-    // Validation logic
-    return {
-      evaluator: 'json-validator',
-      moduleName: context.moduleName,
-      score: 10,
-      reasoning: 'Valid JSON structure',
-    };
-  },
-} satisfies CodeEvaluator;
-```
-
-#### 2. Prompt Evaluator
-
-AI-based evaluation using LLM:
-
-```typescript
-import type { PromptEvaluator, EvaluationContext } from '@modular-prompt/experiment';
-import type { PromptModule } from '@modular-prompt/core';
-
-const evaluationModule: PromptModule<EvaluationContext> = {
-  createContext: (): EvaluationContext => ({
-    moduleName: '',
-    prompt: '',
-    runs: [],
-  }),
-
-  objective: [
-    '- Assess output quality',
-  ],
-
-  instructions: [
-    '- Evaluate clarity and accuracy',
-  ],
-};
-
-export default {
-  name: 'Quality Evaluator',
-  description: 'Evaluates output quality',
-  module: evaluationModule,
-} satisfies PromptEvaluator;
-```
-
-All prompt evaluators are automatically merged with the base evaluation module.
-
-## Built-in Evaluators
-
-The framework includes built-in evaluators that can be referenced by name only (no path required):
-
-### structured-output-presence
-
-- **Type**: Code Evaluator
-- **What it measures**: Checks if `structuredOutput` exists and is a valid object
-- **Evaluation logic**:
-  - Verifies presence of `structuredOutput` in query result
-  - Confirms it's a non-null object type
-- **Score**: `(validCount / totalRuns) * 10`
-- **Use case**: Verify that the model returns structured JSON output (essential for structured output workflows)
-- **Usage**:
-  ```yaml
-  evaluators:
-    - name: "structured-output-presence"
-  ```
-
-### llm-requirement-fulfillment
-
-- **Type**: Prompt Evaluator (uses LLM for evaluation)
-- **What it measures**: Uses LLM to comprehensively evaluate whether output meets functional requirements
-- **Evaluation criteria**:
-  1. **Requirement Fulfillment**: Does it satisfy the intent described in the prompt?
-  2. **Parameter Correctness**: Are all required parameters present and correct?
-  3. **Parameter Completeness**: Are optional parameters appropriately used or omitted?
-  4. **Logical Consistency**: Is the output logically consistent with the facts?
-- **Score**: 0-10 overall score with detailed sub-scores for each criterion
-- **Use case**: Comprehensive quality assessment of output (requires evaluation model to be configured)
-- **Usage**:
-  ```yaml
-  evaluators:
-    - name: "llm-requirement-fulfillment"
-
-  evaluation:
-    enabled: true
-    model: "gemini-fast"  # Model used for evaluation
-  ```
-
-**Note**: `llm-requirement-fulfillment` requires an evaluation model to be configured in the `evaluation` section.
-
-## Architecture
-
-```
-┌─────────────────────────────────────────┐
-│ run-comparison.ts (CLI Entry Point)    │
-└─────────────────────────────────────────┘
-              │
-    ┌─────────┼─────────┐
-    ▼         ▼         ▼
-┌────────┐ ┌────────┐ ┌────────┐
-│ Config │ │ Runner │ │Reporter│
-│ Loader │ │        │ │        │
-└────────┘ └────────┘ └────────┘
-    │         │
-    ▼         ▼
-┌────────┐ ┌────────┐
-│Dynamic │ │Driver  │
-│Loader  │ │Manager │
-└────────┘ └────────┘
-```
-
-### Components
-
-| Component | Responsibility |
-|-----------|----------------|
-| `config/loader.ts` | Load YAML configuration |
-| `config/dynamic-loader.ts` | Dynamic module/evaluator loading |
-| `runner/experiment.ts` | Orchestrate experiment execution |
-| `runner/evaluator.ts` | Execute evaluations |
-| `runner/driver-manager.ts` | Cache and manage AI drivers |
-| `reporter/statistics.ts` | Generate statistical reports |
-| `base-evaluation-module.ts` | Base evaluation prompt module |
-| `evaluators/index.ts` | Built-in evaluator registry |
-
-## Examples
-
-See `examples/experiment.yaml` for a complete configuration template with:
-- Model definitions (MLX, Vertex AI, Google GenAI)
-- Driver configurations with credential paths
-- Evaluation settings
-- Empty sections for modules, test cases, and evaluators (ready for your content)
-
-## API
-
-### Programmatic Usage
-
-```typescript
-import {
-  loadExperimentConfig,
-  loadModules,
-  loadEvaluators,
-  ExperimentRunner,
-  DriverManager,
-} from '@modular-prompt/experiment';
-
-const { serverConfig, aiService } = loadExperimentConfig('config.yaml');
-const modules = await loadModules(moduleRefs, basePath);
-const evaluators = await loadEvaluators(evaluatorRefs, basePath);
-
-const driverManager = new DriverManager();
-const runner = new ExperimentRunner(
-  aiService,
-  driverManager,
-  modules,
-  testCases,
-  models,
-  repeatCount,
-  evaluators,
-  evaluatorModel
-);
-
-const results = await runner.run();
-await driverManager.cleanup();
-```
-
-## CLI Options
+### 3. 実行
 
+```bash
+npx modular-experiment experiment.yaml --dry-run    # 確認
+npx modular-experiment experiment.yaml              # 実行
+npx modular-experiment experiment.yaml --evaluate   # 評価付き
+npx modular-experiment experiment.yaml --repeat 10  # 複数回実行
 ```
-Usage: modular-experiment <config> [options]
 
-Arguments:
-  <config>                Config file path (YAML or TypeScript)
+設定ファイルの詳細、評価器の書き方、プログラマティックAPIについては `skills/experiment/SKILL.md` を参照。
 
-Options:
-  --test-case <name>      Test case name filter
-  --model <provider>      Model provider filter
-  --modules <names>       Comma-separated module names (default: all)
-  --repeat <count>        Number of repetitions (default: 1)
-  --evaluate              Enable evaluation phase
-  --evaluators <names>    Comma-separated evaluator names (default: all)
-  --dry-run               Display execution plan without running the experiment
-  --log-file <path>       Log file path for JSONL output (detailed logs)
-  --verbose               Enable verbose output (show detailed internal operations)
-```
+## Skills (for Claude Code)
 
-**Note**: All paths specified in the config file are resolved relative to the config file's directory.
+This package includes `skills/experiment/SKILL.md`. It can be used as a Claude Code skill to guide experiment framework usage.
 
 ## License
 

package/examples/tools-experiment.yaml

@@ -12,6 +12,23 @@ models:
     provider: "mlx"
     capabilities: ["local", "tools"]
     priority: 20
+  lfm2.5-jp:
+    model: LiquidAI/LFM2.5-1.2B-JP-MLX-8bit
+    provider: "mlx"
+    capabilities: ["local", "fast", "japanese"]
+    priority: 20
+  lfm2.5-instruct:
+    model: LiquidAI/LFM2.5-1.2B-Instruct-MLX-8bit
+    provider: "mlx"
+    capabilities: ["local", "fast", "japanese"]
+    priority: 20
+    disabled: true
+  lfm2.5-thinking:
+    model: LiquidAI/LFM2.5-1.2B-Thinking-MLX-8bit
+    provider: "mlx"
+    capabilities: ["local", "fast", "japanese"]
+    priority: 20
+    disabled: true
 
 drivers:
   mlx: {}
@@ -55,17 +72,32 @@ testCases:
       tools: *tools_def
       toolChoice: auto
 
-  # --- qwen3-4b ---
-  - name: "[qwen3] 天気ツール呼び出し"
+  # # --- qwen3-4b ---
+  # - name: "[qwen3] 天気ツール呼び出し"
+  #   description: "get_weatherツールを呼び出すことを期待"
+  #   models: ["qwen3-4b"]
+  #   input:
+  #     question: "東京の天気を教えてください。"
+  #   queryOptions: *tool_weather
+
+  # - name: "[qwen3] ツール不要の質問"
+  #   description: "ツールを呼び出さずにテキストで回答することを期待"
+  #   models: ["qwen3-4b"]
+  #   input:
+  #     question: "1 + 1 は何ですか？"
+  #   queryOptions: *tool_math
+
+  # --- lfm2.5 jp ---
+  - name: "[lfm2.5] 天気ツール呼び出し"
     description: "get_weatherツールを呼び出すことを期待"
-    models: ["qwen3-4b"]
+    models: ["lfm2.5-instruct"]
     input:
       question: "東京の天気を教えてください。"
     queryOptions: *tool_weather
 
-  - name: "[qwen3] ツール不要の質問"
+  - name: "[lfm2.5] ツール不要の質問"
     description: "ツールを呼び出さずにテキストで回答することを期待"
-    models: ["qwen3-4b"]
+    models: ["lfm2.5-instruct"]
     input:
       question: "1 + 1 は何ですか？"
     queryOptions: *tool_math

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@modular-prompt/experiment",
-  "version": "0.3.0",
+  "version": "0.3.1",
   "description": "Experiment framework for comparing and evaluating prompt modules",
   "type": "module",
   "main": "./dist/index.js",
@@ -17,6 +17,7 @@
   "files": [
     "dist",
     "examples",
+    "skills",
     "README.md"
   ],
   "dependencies": {
@@ -24,9 +25,9 @@
     "jiti": "^2.4.2",
     "yaml": "^2.3.4",
     "zod": "^3.22.4",
-    "@modular-prompt/driver": "0.8.0",
-    "@modular-prompt/core": "0.1.12",
-    "@modular-prompt/utils": "0.2.3"
+    "@modular-prompt/core": "0.1.13",
+    "@modular-prompt/driver": "0.8.1",
+    "@modular-prompt/utils": "0.2.4"
   },
   "devDependencies": {
     "@eslint/js": "^9.34.0",
@@ -64,7 +65,8 @@
     "test": "vitest",
     "test:ui": "vitest --ui",
     "test:run": "vitest run",
-    "clean": "rm -rf dist",
+    "copy-skills": "mkdir -p skills/experiment && cp ../../skills/experiment/SKILL.md skills/experiment/SKILL.md",
+    "clean": "rm -rf dist skills",
     "lint": "eslint src",
     "typecheck": "tsc --noEmit"
   }

package/skills/experiment/SKILL.md

@@ -0,0 +1,292 @@
+---
+name: experiment
+description: modular-promptの実験フレームワーク（@modular-prompt/experiment）の使い方ガイド。プロンプトモジュールの比較・評価実験の設定、実行、評価器の定義を参照する。
+---
+
+# 実験フレームワーク使い方ガイド
+
+## 実験フレームワークとは
+
+`@modular-prompt/experiment` は、複数のプロンプトモジュールを同一条件下で比較・評価するためのフレームワーク。YAML設定で実験を定義し、CLIまたはプログラマティックに実行できる。
+
+### ユースケース
+
+- **プロンプト比較**: 異なるプロンプト構造の効果を比較検証
+- **モジュール分離検証**: モジュール化したプロンプトが同等の出力を生成するか確認
+- **品質評価**: 繰り返し実行による出力の安定性・一貫性の評価
+- **マルチモデルテスト**: 異なるLLMプロバイダーでの動作比較
+
+## CLI
+
+```bash
+# 設定検証・実行計画表示（まずこれで確認）
+npx modular-experiment config.yaml --dry-run
+
+# 実験実行
+npx modular-experiment config.yaml
+
+# 評価付き実行
+npx modular-experiment config.yaml --evaluate
+
+# 複数回実行（統計用）
+npx modular-experiment config.yaml --repeat 10
+
+# 特定モジュール・テストケースのみ
+npx modular-experiment config.yaml --modules my-module --test-case "Basic Test"
+
+# 詳細ログ出力
+npx modular-experiment config.yaml --log-file experiment.jsonl --verbose
+```
+
+### CLIオプション
+
+| オプション | 説明 |
+|-----------|------|
+| `<config>` | 設定ファイルパス（YAML or TypeScript） |
+| `--dry-run` | 実行計画のみ表示 |
+| `--evaluate` | 評価フェーズを有効化 |
+| `--repeat <count>` | 繰り返し回数（デフォルト: 1） |
+| `--modules <names>` | カンマ区切りモジュール名フィルター |
+| `--test-case <name>` | テストケース名フィルター |
+| `--model <provider>` | モデルプロバイダーフィルター |
+| `--evaluators <names>` | カンマ区切り評価器名フィルター |
+| `--log-file <path>` | JSONLログファイルパス |
+| `--verbose` | 詳細な内部操作を表示 |
+
+## 設定ファイル（YAML）
+
+```yaml
+# モデル定義
+models:
+  gpt4o:
+    provider: openai
+    model: gpt-4o
+    capabilities: ["streaming", "tools", "structured"]
+    enabled: true
+  gemini:
+    provider: vertexai
+    model: gemini-2.0-flash-001
+    capabilities: ["tools", "fast"]
+    enabled: true
+
+# ドライバー認証設定
+drivers:
+  openai:
+    apiKey: ${OPENAI_API_KEY}      # 環境変数
+  vertexai:
+    projectId: my-gcp-project
+    location: us-central1
+
+# デフォルトオプション
+defaultOptions:
+  temperature: 0.7
+  maxTokens: 2048
+
+# テスト対象モジュール
+modules:
+  - name: baseline
+    path: ./modules/baseline.ts     # 設定ファイルからの相対パス
+    description: ベースラインプロンプト
+  - name: optimized
+    path: ./modules/optimized.ts
+    description: 最適化版プロンプト
+
+# テストケース
+testCases:
+  - name: 基本テスト
+    description: 基本的な動作確認
+    input:                          # module.compile に渡すコンテキスト
+      query: "TypeScriptの型推論について説明して"
+    models: [gpt4o]                 # オプション: 未指定時は全有効モデル
+    queryOptions:                   # オプション
+      temperature: 0.5
+
+  - name: ツール呼び出しテスト
+    input:
+      query: "東京の天気を調べて"
+    queryOptions:
+      tools:
+        - name: get_weather
+          description: 天気を取得
+          parameters:
+            type: object
+            properties:
+              city: { type: string }
+            required: [city]
+
+# 評価器
+evaluators:
+  - name: structured-output-presence  # ビルトイン
+  - name: llm-requirement-fulfillment # ビルトイン
+  - name: custom-eval                 # 外部ファイル
+    path: ./evaluators/custom-eval.ts
+
+# 評価設定
+evaluation:
+  enabled: true
+  model: gpt4o                        # 評価に使うモデル
+```
+
+### パス解決
+
+設定ファイル内のパス（modules, evaluators等）は設定ファイルのディレクトリからの相対パスで解決される。`~/` でホームディレクトリ、絶対パスも使用可能。
+
+## モジュール定義
+
+テスト対象のモジュールファイル:
+
+```typescript
+import { compile } from '@modular-prompt/core';
+import { myPromptModule } from './prompts.js';
+
+export default {
+  name: 'My Module',
+  description: 'テスト対象のプロンプトモジュール',
+  compile: (context: any) => compile(myPromptModule, context),
+};
+```
+
+`compile` 関数はテストケースの `input` をコンテキストとして受け取り、CompiledPrompt を返す。
+
+## 評価器
+
+### ビルトイン評価器
+
+**structured-output-presence** - コード評価器
+- `structuredOutput` の存在と有効性を検証
+- スコア: `(validCount / totalRuns) * 10`
+
+**llm-requirement-fulfillment** - プロンプト評価器
+- LLMが要件充足度を包括的に評価
+- 評価基準: 要件充足度、パラメータ正確性、パラメータ完全性、論理的一貫性
+- 評価用モデルの設定が必要（`evaluation.model`）
+
+### カスタム評価器（コード）
+
+```typescript
+import type { CodeEvaluator, EvaluationContext, EvaluationResult } from '@modular-prompt/experiment';
+
+export default {
+  name: 'json-validator',
+  description: 'JSON構造を検証',
+
+  async evaluate(context: EvaluationContext): Promise<EvaluationResult> {
+    const allValid = context.runs.every(run =>
+      run.queryResult.structuredOutput != null
+    );
+    return {
+      evaluator: 'json-validator',
+      moduleName: context.moduleName,
+      score: allValid ? 10 : 0,
+      reasoning: allValid ? '全実行でJSON出力あり' : 'JSON出力なし',
+    };
+  },
+} satisfies CodeEvaluator;
+```
+
+### カスタム評価器（プロンプト）
+
+LLMに評価させる場合:
+
+```typescript
+import type { PromptEvaluator, EvaluationContext } from '@modular-prompt/experiment';
+import type { PromptModule } from '@modular-prompt/core';
+
+const evaluationModule: PromptModule<EvaluationContext> = {
+  createContext: () => ({ moduleName: '', prompt: '', runs: [] }),
+  objective: ['出力の品質を0-10で評価する'],
+  instructions: [
+    '- 明確さ、正確さ、完全性を基準にする',
+    (ctx) => `対象モジュール: ${ctx.moduleName}`,
+    (ctx) => ctx.runs.map((run, i) =>
+      `実行${i + 1}: ${run.queryResult.content.slice(0, 500)}`
+    ),
+  ],
+};
+
+export default {
+  name: 'quality-evaluator',
+  description: '出力品質を評価',
+  module: evaluationModule,   // baseEvaluationModuleと自動マージされる
+} satisfies PromptEvaluator;
+```
+
+## 主要な型
+
+### TestCase
+
+```typescript
+interface TestCase {
+  name: string;
+  description?: string;
+  input: any;                          // module.compileに渡すコンテキスト
+  models?: string[];                   // 未指定時は全有効モデル
+  queryOptions?: Partial<QueryOptions>;
+}
+```
+
+### EvaluationContext
+
+```typescript
+interface EvaluationContext {
+  moduleName: string;
+  prompt: string;         // コンパイル済みプロンプト（文字列化）
+  runs: Array<{
+    queryResult: QueryResult;
+  }>;
+}
+```
+
+### EvaluationResult
+
+```typescript
+interface EvaluationResult {
+  evaluator: string;
+  moduleName: string;
+  score?: number;         // 0-10
+  reasoning?: string;
+  details?: Record<string, any>;
+  error?: string;
+}
+```
+
+## プログラマティック使用
+
+```typescript
+import {
+  loadExperimentConfig,
+  loadModules,
+  loadEvaluators,
+  ExperimentRunner,
+  DriverManager,
+} from '@modular-prompt/experiment';
+
+const { serverConfig, aiService, configDir } = loadExperimentConfig('config.yaml');
+const modules = await loadModules(serverConfig.modules, configDir);
+const evaluators = await loadEvaluators(serverConfig.evaluators, configDir);
+
+const driverManager = new DriverManager();
+const runner = new ExperimentRunner(
+  aiService,
+  driverManager,
+  modules,
+  serverConfig.testCases,
+  serverConfig.models,
+  5,           // repeat count
+  evaluators,
+  evaluatorModel
+);
+
+const results = await runner.run();
+await driverManager.cleanup();
+```
+
+## 実験フロー
+
+```
+設定ロード → モジュール・評価器ロード → テストケースごとに:
+  全モジュールをコンパイル → プロンプト比較 → 各モデルで実行（繰り返し対応）
+→ 評価フェーズ（オプション） → 統計レポート生成 → クリーンアップ
+```
+
+DriverManagerがモデル名をキーにドライバーをキャッシュし、同じモデルであればドライバーを再利用する。異なるモデルに切り替わると前のドライバーをcloseできる。これはローカルLLM（MLX等）のメモリ消費を抑えるための設計。