npm - @modular-prompt/experiment - Versions diffs - 0.1.10 → 0.3.1 - Mend

@modular-prompt/experiment 0.1.10 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (13) hide show

package/README.md +32 -369
package/dist/run-comparison.js +6 -0
package/dist/run-comparison.js.map +1 -1
package/dist/runner/experiment.d.ts.map +1 -1
package/dist/runner/experiment.js +37 -12
package/dist/runner/experiment.js.map +1 -1
package/dist/types.d.ts +5 -1
package/dist/types.d.ts.map +1 -1
package/examples/tools-experiment.yaml +105 -0
package/examples/tools-test-module.mjs +29 -0
package/examples/tools-test-module.ts +29 -0
package/package.json +7 -5
package/skills/experiment/SKILL.md +292 -0

package/README.md CHANGED Viewed

@@ -1,412 +1,75 @@
 # @modular-prompt/experiment
-Experiment framework for comparing and evaluating modular prompt modules.
+プロンプトモジュールの比較・評価フレームワーク。
-## Overview
-This framework provides tools to compare and evaluate different prompt module variations under identical conditions. It integrates with the `@modular-prompt/core` system to test multiple prompt variations and evaluate their output quality.
-### Use Cases
-- **Prompt Engineering**: Validate the effectiveness of new prompt structures
-- **Module Separation**: Verify that modularized prompts produce equivalent outputs
-- **Quality Evaluation**: Assess output stability and consistency through repeated executions
-- **Multi-Model Testing**: Test across different LLM providers (MLX, VertexAI, GoogleGenAI, etc.)
-## Features
-- ✅ **Dynamic Module Loading**: Load prompt modules from external files or inline definitions
-- ✅ **Flexible Evaluators**: Support both code-based and AI-based evaluation
-- ✅ **Statistical Analysis**: Analyze success rates, execution times, and output consistency
-- ✅ **Prompt Diff Detection**: Automatically detect differences between module outputs
-- ✅ **Driver Caching**: Reuse drivers for improved memory efficiency
-- ✅ **Detailed Logging**: Comprehensive logging of all executions
-## Installation
+## インストール
 ```bash
-pnpm add @modular-prompt/experiment
+npm install @modular-prompt/experiment
 ```
-## Quick Start
+## 概要
-### 1. Create Configuration File
+複数のプロンプトモジュールを同一条件下で比較・評価する。YAML設定で実験を定義し、CLIで実行。
-You can use either YAML or TypeScript format.
+- **プロンプト比較**: 異なるプロンプト構造の効果を定量的に比較
+- **マルチモデルテスト**: 異なるLLMプロバイダーでの動作比較
+- **品質評価**: 繰り返し実行による安定性・一貫性の評価
+- **柔軟な評価器**: コードベース・AIベースの評価をサポート
-#### Option A: YAML Configuration (Recommended for static configurations)
+## クイックスタート
-Create `examples/experiment.yaml`:
+### 1. 設定ファイルを作成
 ```yaml
+# experiment.yaml
 models:
-  gemini-fast:
-    provider: vertexai
-    model: gemini-2.0-flash-exp
-    capabilities: ["tools", "fast"]
-    enabled: true
+  gpt4o:
+    provider: openai
+    model: gpt-4o
 drivers:
-  vertexai:
-    projectId: your-project-id
-    location: us-central1
-    # Paths are resolved relative to this config file
-    # Can use ~/ for home directory or absolute paths
-    credentialsPath: ./credentials.json
+  openai:
+    apiKey: ${OPENAI_API_KEY}
 modules:
   - name: my-module
     path: ./my-module.ts
-    description: My custom prompt module
 testCases:
-  - name: Basic Test
-    description: Test basic functionality
-    input:  # Structured context object (passed to module.compile)
-      query: user question
-      context: additional information
-    models:  # Optional: specify which models to test (uses all enabled if not specified)
-      - gemini-fast
+  - name: 基本テスト
+    input:
+      query: "TypeScriptについて説明して"
-evaluators:
-  # Built-in evaluators (name only)
-  - name: structured-output-presence
-  - name: llm-requirement-fulfillment
-  # Or external evaluator (with path)
-  - name: custom-validator
-    path: ./evaluators/custom-validator.ts
-  # Or inline prompt evaluator
-  - name: quality-check
-    prompt:
-      objective:
-        - Evaluate output quality
-      instructions:
-        - Check clarity and accuracy
-evaluation:
-  enabled: true
-  model: gemini-fast  # Reference by model name
+evaluators: []
 ```
-#### Option B: TypeScript Configuration (For dynamic configurations)
-Create `examples/experiment.ts`:
-```typescript
-export default {
-  models: {
-    'gemini-fast': {
-      provider: 'vertexai',
-      model: 'gemini-2.0-flash-exp',
-      capabilities: ['tools', 'fast'],
-      enabled: true,
-    },
-  },
-  drivers: {
-    vertexai: {
-      projectId: 'your-project-id',
-      location: 'us-central1',
-      credentialsPath: './credentials.json',
-    },
-  },
-  modules: [
-    {
-      name: 'my-module',
-      path: './my-module.ts',
-      description: 'My custom prompt module',
-    },
-  ],
-  testCases: [
-    {
-      name: 'Basic Test',
-      description: 'Test basic functionality',
-      input: {  // Structured context object
-        query: 'user question',
-        options: { temperature: 0.7 },
-      },
-      models: ['gemini-fast'],  // Optional
-    },
-  ],
-  evaluators: [
-    // Built-in evaluators (name only)
-    { name: 'structured-output-presence' },
-    { name: 'llm-requirement-fulfillment' },
-    // Or external evaluator (with path)
-    {
-      name: 'custom-validator',
-      path: './evaluators/custom-validator.ts',
-    },
-  ],
-  evaluation: {
-    enabled: true,
-    model: 'gemini-fast',  // Reference by model name
-  },
-};
-```
-**TypeScript Support**: TypeScript configuration files are automatically transpiled using [jiti](https://github.com/unjs/jiti). You can use TypeScript syntax directly without pre-compilation. Type annotations are stripped automatically, and the file is executed as JavaScript.
-**Important**: All file paths in the configuration (modules, evaluators, credentials) are resolved relative to the config file location.
-### 2. Run Experiment
-```bash
-# Validate configuration and display execution plan (recommended first step)
-npx modular-experiment examples/experiment.yaml --dry-run
-# Run with YAML config
-npx modular-experiment examples/experiment.yaml
-# Run with TypeScript config
-npx modular-experiment examples/experiment.ts
-# Run specific module
-npx modular-experiment examples/experiment.yaml --modules my-module
-# Run with evaluation
-npx modular-experiment examples/experiment.yaml --evaluate
-# Run multiple times for statistics
-npx modular-experiment examples/experiment.yaml --repeat 10
-# Run with detailed logging to JSONL file
-npx modular-experiment examples/experiment.yaml --log-file experiment.jsonl
-# Run with verbose output (show internal operations)
-npx modular-experiment examples/experiment.yaml --verbose
-# Combine options
-npx modular-experiment examples/experiment.yaml --evaluate --log-file experiment.jsonl --verbose
-```
-## Configuration
-### Module Definition
-Modules can be defined inline or loaded from external files:
-```typescript
-// External file
-export const modules: ModuleReference[] = [
-  {
-    name: 'my-module',
-    path: './modules/my-module.ts',
-    description: 'Description',
-  },
-];
-```
-A module file should export a default object with:
+### 2. モジュールファイルを作成
 ```typescript
+// my-module.ts
 import { compile } from '@modular-prompt/core';
-import { myPromptModule } from './prompts.js';
 export default {
   name: 'My Module',
-  description: 'Module description',
   compile: (context: any) => compile(myPromptModule, context),
 };
 ```
-### Evaluator Definition
-Two types of evaluators are supported:
-#### 1. Code Evaluator
-Programmatic validation (e.g., JSON structure validation):
-```typescript
-import type { CodeEvaluator, EvaluationContext, EvaluationResult } from '@modular-prompt/experiment';
-export default {
-  name: 'JSON Validator',
-  description: 'Validates JSON structure in output',
-  async evaluate(context: EvaluationContext): Promise<EvaluationResult> {
-    // Validation logic
-    return {
-      evaluator: 'json-validator',
-      moduleName: context.moduleName,
-      score: 10,
-      reasoning: 'Valid JSON structure',
-    };
-  },
-} satisfies CodeEvaluator;
-```
-#### 2. Prompt Evaluator
-AI-based evaluation using LLM:
-```typescript
-import type { PromptEvaluator, EvaluationContext } from '@modular-prompt/experiment';
-import type { PromptModule } from '@modular-prompt/core';
-const evaluationModule: PromptModule<EvaluationContext> = {
-  createContext: (): EvaluationContext => ({
-    moduleName: '',
-    prompt: '',
-    runs: [],
-  }),
-  objective: [
-    '- Assess output quality',
-  ],
-  instructions: [
-    '- Evaluate clarity and accuracy',
-  ],
-};
-export default {
-  name: 'Quality Evaluator',
-  description: 'Evaluates output quality',
-  module: evaluationModule,
-} satisfies PromptEvaluator;
-```
-All prompt evaluators are automatically merged with the base evaluation module.
-## Built-in Evaluators
-The framework includes built-in evaluators that can be referenced by name only (no path required):
-### structured-output-presence
-- **Type**: Code Evaluator
-- **What it measures**: Checks if `structuredOutput` exists and is a valid object
-- **Evaluation logic**:
-  - Verifies presence of `structuredOutput` in query result
-  - Confirms it's a non-null object type
-- **Score**: `(validCount / totalRuns) * 10`
-- **Use case**: Verify that the model returns structured JSON output (essential for structured output workflows)
-- **Usage**:
-  ```yaml
-  evaluators:
-    - name: "structured-output-presence"
-  ```
-### llm-requirement-fulfillment
-- **Type**: Prompt Evaluator (uses LLM for evaluation)
-- **What it measures**: Uses LLM to comprehensively evaluate whether output meets functional requirements
-- **Evaluation criteria**:
-  1. **Requirement Fulfillment**: Does it satisfy the intent described in the prompt?
-  2. **Parameter Correctness**: Are all required parameters present and correct?
-  3. **Parameter Completeness**: Are optional parameters appropriately used or omitted?
-  4. **Logical Consistency**: Is the output logically consistent with the facts?
-- **Score**: 0-10 overall score with detailed sub-scores for each criterion
-- **Use case**: Comprehensive quality assessment of output (requires evaluation model to be configured)
-- **Usage**:
-  ```yaml
-  evaluators:
-    - name: "llm-requirement-fulfillment"
-  evaluation:
-    enabled: true
-    model: "gemini-fast"  # Model used for evaluation
-  ```
-**Note**: `llm-requirement-fulfillment` requires an evaluation model to be configured in the `evaluation` section.
-## Architecture
-```
-┌─────────────────────────────────────────┐
-│ run-comparison.ts (CLI Entry Point)    │
-└─────────────────────────────────────────┘
-              │
-    ┌─────────┼─────────┐
-    ▼         ▼         ▼
-┌────────┐ ┌────────┐ ┌────────┐
-│ Config │ │ Runner │ │Reporter│
-│ Loader │ │        │ │        │
-└────────┘ └────────┘ └────────┘
-    │         │
-    ▼         ▼
-┌────────┐ ┌────────┐
-│Dynamic │ │Driver  │
-│Loader  │ │Manager │
-└────────┘ └────────┘
-```
-### Components
-| Component | Responsibility |
-|-----------|----------------|
-| `config/loader.ts` | Load YAML configuration |
-| `config/dynamic-loader.ts` | Dynamic module/evaluator loading |
-| `runner/experiment.ts` | Orchestrate experiment execution |
-| `runner/evaluator.ts` | Execute evaluations |
-| `runner/driver-manager.ts` | Cache and manage AI drivers |
-| `reporter/statistics.ts` | Generate statistical reports |
-| `base-evaluation-module.ts` | Base evaluation prompt module |
-| `evaluators/index.ts` | Built-in evaluator registry |
-## Examples
-See `examples/experiment.yaml` for a complete configuration template with:
-- Model definitions (MLX, Vertex AI, Google GenAI)
-- Driver configurations with credential paths
-- Evaluation settings
-- Empty sections for modules, test cases, and evaluators (ready for your content)
-## API
-### Programmatic Usage
-```typescript
-import {
-  loadExperimentConfig,
-  loadModules,
-  loadEvaluators,
-  ExperimentRunner,
-  DriverManager,
-} from '@modular-prompt/experiment';
-const { serverConfig, aiService } = loadExperimentConfig('config.yaml');
-const modules = await loadModules(moduleRefs, basePath);
-const evaluators = await loadEvaluators(evaluatorRefs, basePath);
-const driverManager = new DriverManager();
-const runner = new ExperimentRunner(
-  aiService,
-  driverManager,
-  modules,
-  testCases,
-  models,
-  repeatCount,
-  evaluators,
-  evaluatorModel
-);
-const results = await runner.run();
-await driverManager.cleanup();
-```
-## CLI Options
+### 3. 実行
+```bash
+npx modular-experiment experiment.yaml --dry-run    # 確認
+npx modular-experiment experiment.yaml              # 実行
+npx modular-experiment experiment.yaml --evaluate   # 評価付き
+npx modular-experiment experiment.yaml --repeat 10  # 複数回実行
 ```
-Usage: modular-experiment <config> [options]
-Arguments:
-  <config>                Config file path (YAML or TypeScript)
+設定ファイルの詳細、評価器の書き方、プログラマティックAPIについては `skills/experiment/SKILL.md` を参照。
-Options:
-  --test-case <name>      Test case name filter
-  --model <provider>      Model provider filter
-  --modules <names>       Comma-separated module names (default: all)
-  --repeat <count>        Number of repetitions (default: 1)
-  --evaluate              Enable evaluation phase
-  --evaluators <names>    Comma-separated evaluator names (default: all)
-  --dry-run               Display execution plan without running the experiment
-  --log-file <path>       Log file path for JSONL output (detailed logs)
-  --verbose               Enable verbose output (show detailed internal operations)
-```
+## Skills (for Claude Code)
-**Note**: All paths specified in the config file are resolved relative to the config file's directory.
+This package includes `skills/experiment/SKILL.md`. It can be used as a Claude Code skill to guide experiment framework usage.
 ## License

package/dist/run-comparison.js CHANGED Viewed

@@ -159,6 +159,12 @@ const results = await runner.run();
 console.log('='.repeat(80));
 console.log('✨ Experiment completed');
 console.log('='.repeat(80));
+// Flush log file if configured
+if (options.logFile) {
+    const { logger } = await import('./logger.js');
+    await logger.flush();
+    console.log(`📄 Log file written: ${options.logFile}`);
+}
 // Cleanup drivers
 await driverManager.cleanup();
 // Display statistics if repeated

package/dist/run-comparison.js.map CHANGED Viewed

	@@ -1 +1 @@
1	- {"version":3,"file":"run-comparison.js","sourceRoot":"","sources":["../src/run-comparison.ts"],"names":[],"mappings":";AACA;;;;;;;;;;;;;;;;;;;GAmBG;AAEH,OAAO,EAAE,SAAS,EAAE,MAAM,eAAe,CAAC;AAC1C,OAAO,EAAE,oBAAoB,EAAE,MAAM,oBAAoB,CAAC;AAC1D,OAAO,EAAE,WAAW,EAAE,cAAc,EAAE,MAAM,4BAA4B,CAAC;AACzE,OAAO,EAAE,aAAa,EAAE,MAAM,4BAA4B,CAAC;AAC3D,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC;AAC1D,OAAO,EAAE,kBAAkB,EAAE,MAAM,0BAA0B,CAAC;AAC9D,OAAO,EAAE,MAAM,EAAE,MAAM,uBAAuB,CAAC;AAE/C,sBAAsB;AACtB,MAAM,OAAO,GAAG,SAAS,EAAE,CAAC;AAE5B,mBAAmB;AACnB,MAAM,CAAC,SAAS,CAAC;IACf,KAAK,EAAE,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM;IACzC,eAAe,EAAE,OAAO;IACxB,SAAS,EAAE,KAAK;IAChB,UAAU,EAAE,CAAC,CAAC,OAAO,CAAC,OAAO;IAC7B,UAAU,EAAE,KAAK;IACjB,OAAO,EAAE,OAAO,CAAC,OAAO;CACzB,CAAC,CAAC;AAEH,iBAAiB;AACjB,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;AAC5B,OAAO,CAAC,GAAG,CAAC,8BAA8B,CAAC,CAAC;AAC5C,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;AAC5B,OAAO,CAAC,GAAG,CAAC,WAAW,OAAO,CAAC,UAAU,EAAE,CAAC,CAAC;AAC7C,OAAO,CAAC,GAAG,CAAC,qBAAqB,OAAO,CAAC,cAAc,IAAI,KAAK,EAAE,CAAC,CAAC;AACpE,OAAO,CAAC,GAAG,CAAC,iBAAiB,OAAO,CAAC,WAAW,IAAI,oBAAoB,EAAE,CAAC,CAAC;AAC5E,OAAO,CAAC,GAAG,CAAC,YAAY,OAAO,CAAC,YAAY,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,KAAK,EAAE,CAAC,CAAC;AACrE,OAAO,CAAC,GAAG,CAAC,WAAW,OAAO,CAAC,WAAW,UAAU,CAAC,CAAC;AACtD,OAAO,CAAC,GAAG,CAAC,eAAe,OAAO,CAAC,gBAAgB,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU,EAAE,CAAC,CAAC;AAChF,IAAI,OAAO,CAAC,gBAAgB,EAAE,CAAC;IAC7B,OAAO,CAAC,GAAG,CAAC,eAAe,OAAO,CAAC,eAAe,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,KAAK,EAAE,CAAC,CAAC;AAC7E,CAAC;AACD,OAAO,CAAC,GAAG,CAAC,YAAY,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,qBAAqB,CAAC,CAAC,CAAC,UAAU,EAAE,CAAC,CAAC;AAC/E,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;AAC5B,OAAO,CAAC,GAAG,EAAE,CAAC;AAEd,qBAAqB;AACrB,MAAM,EACJ,YAAY,EACZ,OAAO,EAAE,aAAa,EACtB,SAAS,EAAE,eAAe,EAC1B,UAAU,EAAE,gBAAgB,EAC5B,SAAS,EACT,SAAS,EACV,GAAG,MAAM,oBAAoB,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;AAEnD,8CAA8C;AAC9C,MAAM,MAAM,GAAG,YAAY,CAAC,MAAM,CAAC;AAEnC,uCAAuC;AACvC,MAAM,YAAY,GAAG,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,CAAgB,EAAE,EAAE,CAC9E,CAAC,IAAI,CAAC,QAAQ,IAAI,CAAC,CAAC,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,IAAI,KAAK,MAAM,CAAC,CACvD,CAAC;AAEF,IAAI,OAAO,CAAC,WAAW,EAAE,CAAC;IACxB,MAAM,eAAe,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,CAAgB,EAAE,EAAE,CACvE,IAAI,CAAC,QAAQ,KAAK,OAAO,CAAC,WAAW,CACtC,CAAC;IACF,IAAI,eAAe,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACjC,OAAO,CAAC,KAAK,CAAC,gDAAgD,OAAO,CAAC,WAAW,EAAE,CAAC,CAAC;QACrF,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IACD,OAAO,CAAC,GAAG,CAAC,mBAAmB,eAAe,CAAC,MAAM,0BAA0B,OAAO,CAAC,WAAW,IAAI,CAAC,CAAC;IACxG,eAAe,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,IAAI,CAAgB,EAAE,EAAE,CACtD,OAAO,CAAC,GAAG,CAAC,OAAO,IAAI,KAAK,IAAI,CAAC,KAAK,KAAK,IAAI,CAAC,QAAQ,GAAG,CAAC,CAC7D,CAAC;AACJ,CAAC;KAAM,CAAC;IACN,OAAO,CAAC,GAAG,CAAC,mBAAmB,YAAY,CAAC,MAAM,YAAY,CAAC,CAAC;IAChE,YAAY,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,IAAI,CAAgB,EAAE,EAAE,CACnD,OAAO,CAAC,GAAG,CAAC,OAAO,IAAI,KAAK,IAAI,CAAC,KAAK,KAAK,IAAI,CAAC,QAAQ,GAAG,CAAC,CAC7D,CAAC;AACJ,CAAC;AAED,gCAAgC;AAChC,MAAM,MAAM,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,CAAgB,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,KAAK,KAAK,CAAC,CAAC;AACxF,IAAI,MAAM,EAAE,CAAC;IACX,OAAO,CAAC,GAAG,EAAE,CAAC;IACd,OAAO,CAAC,GAAG,CAAC,4GAA4G,CAAC,CAAC;AAC5H,CAAC;AACD,OAAO,CAAC,GAAG,EAAE,CAAC;AAEd,kBAAkB;AAClB,MAAM,YAAY,GAAG,eAAe,CAAC;AACrC,MAAM,SAAS,GAAG,OAAO,CAAC,cAAc;IACtC,CAAC,CAAC,YAAY,CAAC,MAAM,CAAC,CAAC,EAAO,EAAE,EAAE,CAAC,EAAE,CAAC,IAAI,KAAK,OAAO,CAAC,cAAc,CAAC;IACtE,CAAC,CAAC,YAAY,CAAC;AAEjB,IAAI,SAAS,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;IAC3B,OAAO,CAAC,KAAK,CAAC,wBAAwB,OAAO,CAAC,cAAc,CAAC,CAAC,CAAC,cAAc,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IAC9G,OAAO,CAAC,KAAK,CAAC,yCAAyC,CAAC,CAAC;IACzD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC;AAED,OAAO,CAAC,GAAG,CAAC,cAAc,SAAS,CAAC,MAAM,eAAe,CAAC,CAAC;AAC3D,OAAO,CAAC,GAAG,EAAE,CAAC;AAEd,wCAAwC;AACxC,MAAM,UAAU,GAAG,MAAM,WAAW,CAAC,aAAa,EAAE,SAAS,CAAC,CAAC;AAC/D,MAAM,OAAO,GAAG,OAAO,CAAC,YAAY;IAClC,CAAC,CAAC,UAAU,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,OAAO,CAAC,YAAa,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;IAChE,CAAC,CAAC,UAAU,CAAC;AAEf,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;IACzB,OAAO,CAAC,KAAK,CAAC,sBAAsB,CAAC,CAAC;IACtC,OAAO,CAAC,KAAK,CAAC,sCAAsC,CAAC,CAAC;IACtD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC;AAED,OAAO,CAAC,GAAG,CAAC,cAAc,OAAO,CAAC,MAAM,aAAa,CAAC,CAAC;AACvD,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,IAAI,KAAK,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;AACrE,OAAO,CAAC,GAAG,EAAE,CAAC;AAEd,8DAA8D;AAC9D,IAAI,UAAU,CAAC;AACf,IAAI,cAAc,CAAC;AACnB,IAAI,OAAO,CAAC,gBAAgB,EAAE,CAAC;IAC7B,8CAA8C;IAC9C,MAAM,aAAa,GAAG,MAAM,cAAc,CAAC,gBAAgB,EAAE,SAAS,CAAC,CAAC;IACxE,UAAU,GAAG,OAAO,CAAC,eAAe;QAClC,CAAC,CAAC,aAAa,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,OAAO,CAAC,eAAgB,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;QACtE,CAAC,CAAC,aAAa,CAAC;IAElB,IAAI,UAAU,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC5B,OAAO,CAAC,KAAK,CAAC,uBAAuB,CAAC,CAAC;QACvC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAED,8CAA8C;IAC9C,IAAI,CAAC,YAAY,CAAC,UAAU,IAAI,CAAC,YAAY,CAAC,UAAU,CAAC,OAAO,EAAE,CAAC;QACjE,OAAO,CAAC,KAAK,CAAC,+CAA+C,CAAC,CAAC;QAC/D,OAAO,CAAC,KAAK,CAAC,uDAAuD,CAAC,CAAC;QACvE,OAAO,CAAC,KAAK,CAAC,gBAAgB,CAAC,CAAC;QAChC,OAAO,CAAC,KAAK,CAAC,oBAAoB,CAAC,CAAC;QACpC,OAAO,CAAC,KAAK,CAAC,0BAA0B,CAAC,CAAC;QAC1C,OAAO,CAAC,KAAK,CAAC,gCAAgC,CAAC,CAAC;QAChD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAED,MAAM,gBAAgB,GAAG,YAAY,CAAC,UAAU,CAAC;IAEjD,mCAAmC;IACnC,MAAM,SAAS,GAAG,gBAAgB,CAAC,KAAK,CAAC;IACzC,MAAM,SAAS,GAAG,YAAY,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;IAEjD,IAAI,CAAC,SAAS,IAAI,SAAS,CAAC,QAAQ,EAAE,CAAC;QACrC,OAAO,CAAC,KAAK,CAAC,4CAA4C,SAAS,EAAE,CAAC,CAAC;QACvE,OAAO,CAAC,KAAK,CAAC,yEAAyE,CAAC,CAAC;QACzF,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAED,cAAc,GAAG,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,EAAE,SAAS,EAAE,CAAC;IAEtD,OAAO,CAAC,GAAG,CAAC,8BAA8B,UAAU,CAAC,MAAM,gBAAgB,CAAC,CAAC;IAC7E,UAAU,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,OAAO,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,IAAI,KAAK,CAAC,CAAC,IAAI,KAAK,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;IACpF,OAAO,CAAC,GAAG,CAAC,uBAAuB,SAAS,KAAK,SAAS,CAAC,QAAQ,IAAI,SAAS,CAAC,KAAK,GAAG,CAAC,CAAC;IAC3F,OAAO,CAAC,GAAG,EAAE,CAAC;AAChB,CAAC;AAED,kBAAkB;AAClB,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;IACnB,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IAC5B,OAAO,CAAC,GAAG,CAAC,wCAAwC,CAAC,CAAC;IACtD,OAAO,CAAC,GAAG,CAAC,mCAAmC,CAAC,CAAC;IACjD,OAAO,CAAC,GAAG,CAAC,+CAA+C,CAAC,CAAC;IAC7D,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IAC5B,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC;AAED,iBAAiB;AACjB,MAAM,aAAa,GAAG,IAAI,aAAa,EAAE,CAAC;AAC1C,MAAM,MAAM,GAAG,IAAI,gBAAgB,CACjC,SAAS,EACT,aAAa,EACb,OAAO,EACP,SAAS,EACT,MAAM,EACN,OAAO,CAAC,WAAW,EACnB,UAAU,EACV,cAAc,CACf,CAAC;AAEF,MAAM,OAAO,GAAG,MAAM,MAAM,CAAC,GAAG,EAAE,CAAC;AAEnC,qBAAqB;AACrB,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;AAC5B,OAAO,CAAC,GAAG,CAAC,wBAAwB,CAAC,CAAC;AACtC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;AAE5B,kBAAkB;AAClB,MAAM,aAAa,CAAC,OAAO,EAAE,CAAC;AAE9B,iCAAiC;AACjC,IAAI,OAAO,CAAC,WAAW,GAAG,CAAC,EAAE,CAAC;IAC5B,MAAM,QAAQ,GAAG,IAAI,kBAAkB,CAAC,OAAO,CAAC,CAAC;IACjD,QAAQ,CAAC,MAAM,EAAE,CAAC;AACpB,CAAC"}
1	+ {"version":3,"file":"run-comparison.js","sourceRoot":"","sources":["../src/run-comparison.ts"],"names":[],"mappings":";AACA;;;;;;;;;;;;;;;;;;;GAmBG;AAEH,OAAO,EAAE,SAAS,EAAE,MAAM,eAAe,CAAC;AAC1C,OAAO,EAAE,oBAAoB,EAAE,MAAM,oBAAoB,CAAC;AAC1D,OAAO,EAAE,WAAW,EAAE,cAAc,EAAE,MAAM,4BAA4B,CAAC;AACzE,OAAO,EAAE,aAAa,EAAE,MAAM,4BAA4B,CAAC;AAC3D,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC;AAC1D,OAAO,EAAE,kBAAkB,EAAE,MAAM,0BAA0B,CAAC;AAC9D,OAAO,EAAE,MAAM,EAAE,MAAM,uBAAuB,CAAC;AAE/C,sBAAsB;AACtB,MAAM,OAAO,GAAG,SAAS,EAAE,CAAC;AAE5B,mBAAmB;AACnB,MAAM,CAAC,SAAS,CAAC;IACf,KAAK,EAAE,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM;IACzC,eAAe,EAAE,OAAO;IACxB,SAAS,EAAE,KAAK;IAChB,UAAU,EAAE,CAAC,CAAC,OAAO,CAAC,OAAO;IAC7B,UAAU,EAAE,KAAK;IACjB,OAAO,EAAE,OAAO,CAAC,OAAO;CACzB,CAAC,CAAC;AAEH,iBAAiB;AACjB,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;AAC5B,OAAO,CAAC,GAAG,CAAC,8BAA8B,CAAC,CAAC;AAC5C,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;AAC5B,OAAO,CAAC,GAAG,CAAC,WAAW,OAAO,CAAC,UAAU,EAAE,CAAC,CAAC;AAC7C,OAAO,CAAC,GAAG,CAAC,qBAAqB,OAAO,CAAC,cAAc,IAAI,KAAK,EAAE,CAAC,CAAC;AACpE,OAAO,CAAC,GAAG,CAAC,iBAAiB,OAAO,CAAC,WAAW,IAAI,oBAAoB,EAAE,CAAC,CAAC;AAC5E,OAAO,CAAC,GAAG,CAAC,YAAY,OAAO,CAAC,YAAY,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,KAAK,EAAE,CAAC,CAAC;AACrE,OAAO,CAAC,GAAG,CAAC,WAAW,OAAO,CAAC,WAAW,UAAU,CAAC,CAAC;AACtD,OAAO,CAAC,GAAG,CAAC,eAAe,OAAO,CAAC,gBAAgB,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU,EAAE,CAAC,CAAC;AAChF,IAAI,OAAO,CAAC,gBAAgB,EAAE,CAAC;IAC7B,OAAO,CAAC,GAAG,CAAC,eAAe,OAAO,CAAC,eAAe,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,KAAK,EAAE,CAAC,CAAC;AAC7E,CAAC;AACD,OAAO,CAAC,GAAG,CAAC,YAAY,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,qBAAqB,CAAC,CAAC,CAAC,UAAU,EAAE,CAAC,CAAC;AAC/E,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;AAC5B,OAAO,CAAC,GAAG,EAAE,CAAC;AAEd,qBAAqB;AACrB,MAAM,EACJ,YAAY,EACZ,OAAO,EAAE,aAAa,EACtB,SAAS,EAAE,eAAe,EAC1B,UAAU,EAAE,gBAAgB,EAC5B,SAAS,EACT,SAAS,EACV,GAAG,MAAM,oBAAoB,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;AAEnD,8CAA8C;AAC9C,MAAM,MAAM,GAAG,YAAY,CAAC,MAAM,CAAC;AAEnC,uCAAuC;AACvC,MAAM,YAAY,GAAG,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,CAAgB,EAAE,EAAE,CAC9E,CAAC,IAAI,CAAC,QAAQ,IAAI,CAAC,CAAC,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,IAAI,KAAK,MAAM,CAAC,CACvD,CAAC;AAEF,IAAI,OAAO,CAAC,WAAW,EAAE,CAAC;IACxB,MAAM,eAAe,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,CAAgB,EAAE,EAAE,CACvE,IAAI,CAAC,QAAQ,KAAK,OAAO,CAAC,WAAW,CACtC,CAAC;IACF,IAAI,eAAe,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACjC,OAAO,CAAC,KAAK,CAAC,gDAAgD,OAAO,CAAC,WAAW,EAAE,CAAC,CAAC;QACrF,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IACD,OAAO,CAAC,GAAG,CAAC,mBAAmB,eAAe,CAAC,MAAM,0BAA0B,OAAO,CAAC,WAAW,IAAI,CAAC,CAAC;IACxG,eAAe,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,IAAI,CAAgB,EAAE,EAAE,CACtD,OAAO,CAAC,GAAG,CAAC,OAAO,IAAI,KAAK,IAAI,CAAC,KAAK,KAAK,IAAI,CAAC,QAAQ,GAAG,CAAC,CAC7D,CAAC;AACJ,CAAC;KAAM,CAAC;IACN,OAAO,CAAC,GAAG,CAAC,mBAAmB,YAAY,CAAC,MAAM,YAAY,CAAC,CAAC;IAChE,YAAY,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,IAAI,CAAgB,EAAE,EAAE,CACnD,OAAO,CAAC,GAAG,CAAC,OAAO,IAAI,KAAK,IAAI,CAAC,KAAK,KAAK,IAAI,CAAC,QAAQ,GAAG,CAAC,CAC7D,CAAC;AACJ,CAAC;AAED,gCAAgC;AAChC,MAAM,MAAM,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,CAAgB,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,KAAK,KAAK,CAAC,CAAC;AACxF,IAAI,MAAM,EAAE,CAAC;IACX,OAAO,CAAC,GAAG,EAAE,CAAC;IACd,OAAO,CAAC,GAAG,CAAC,4GAA4G,CAAC,CAAC;AAC5H,CAAC;AACD,OAAO,CAAC,GAAG,EAAE,CAAC;AAEd,kBAAkB;AAClB,MAAM,YAAY,GAAG,eAAe,CAAC;AACrC,MAAM,SAAS,GAAG,OAAO,CAAC,cAAc;IACtC,CAAC,CAAC,YAAY,CAAC,MAAM,CAAC,CAAC,EAAO,EAAE,EAAE,CAAC,EAAE,CAAC,IAAI,KAAK,OAAO,CAAC,cAAc,CAAC;IACtE,CAAC,CAAC,YAAY,CAAC;AAEjB,IAAI,SAAS,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;IAC3B,OAAO,CAAC,KAAK,CAAC,wBAAwB,OAAO,CAAC,cAAc,CAAC,CAAC,CAAC,cAAc,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IAC9G,OAAO,CAAC,KAAK,CAAC,yCAAyC,CAAC,CAAC;IACzD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC;AAED,OAAO,CAAC,GAAG,CAAC,cAAc,SAAS,CAAC,MAAM,eAAe,CAAC,CAAC;AAC3D,OAAO,CAAC,GAAG,EAAE,CAAC;AAEd,wCAAwC;AACxC,MAAM,UAAU,GAAG,MAAM,WAAW,CAAC,aAAa,EAAE,SAAS,CAAC,CAAC;AAC/D,MAAM,OAAO,GAAG,OAAO,CAAC,YAAY;IAClC,CAAC,CAAC,UAAU,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,OAAO,CAAC,YAAa,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;IAChE,CAAC,CAAC,UAAU,CAAC;AAEf,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;IACzB,OAAO,CAAC,KAAK,CAAC,sBAAsB,CAAC,CAAC;IACtC,OAAO,CAAC,KAAK,CAAC,sCAAsC,CAAC,CAAC;IACtD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC;AAED,OAAO,CAAC,GAAG,CAAC,cAAc,OAAO,CAAC,MAAM,aAAa,CAAC,CAAC;AACvD,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,IAAI,KAAK,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;AACrE,OAAO,CAAC,GAAG,EAAE,CAAC;AAEd,8DAA8D;AAC9D,IAAI,UAAU,CAAC;AACf,IAAI,cAAc,CAAC;AACnB,IAAI,OAAO,CAAC,gBAAgB,EAAE,CAAC;IAC7B,8CAA8C;IAC9C,MAAM,aAAa,GAAG,MAAM,cAAc,CAAC,gBAAgB,EAAE,SAAS,CAAC,CAAC;IACxE,UAAU,GAAG,OAAO,CAAC,eAAe;QAClC,CAAC,CAAC,aAAa,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,OAAO,CAAC,eAAgB,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;QACtE,CAAC,CAAC,aAAa,CAAC;IAElB,IAAI,UAAU,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC5B,OAAO,CAAC,KAAK,CAAC,uBAAuB,CAAC,CAAC;QACvC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAED,8CAA8C;IAC9C,IAAI,CAAC,YAAY,CAAC,UAAU,IAAI,CAAC,YAAY,CAAC,UAAU,CAAC,OAAO,EAAE,CAAC;QACjE,OAAO,CAAC,KAAK,CAAC,+CAA+C,CAAC,CAAC;QAC/D,OAAO,CAAC,KAAK,CAAC,uDAAuD,CAAC,CAAC;QACvE,OAAO,CAAC,KAAK,CAAC,gBAAgB,CAAC,CAAC;QAChC,OAAO,CAAC,KAAK,CAAC,oBAAoB,CAAC,CAAC;QACpC,OAAO,CAAC,KAAK,CAAC,0BAA0B,CAAC,CAAC;QAC1C,OAAO,CAAC,KAAK,CAAC,gCAAgC,CAAC,CAAC;QAChD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAED,MAAM,gBAAgB,GAAG,YAAY,CAAC,UAAU,CAAC;IAEjD,mCAAmC;IACnC,MAAM,SAAS,GAAG,gBAAgB,CAAC,KAAK,CAAC;IACzC,MAAM,SAAS,GAAG,YAAY,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;IAEjD,IAAI,CAAC,SAAS,IAAI,SAAS,CAAC,QAAQ,EAAE,CAAC;QACrC,OAAO,CAAC,KAAK,CAAC,4CAA4C,SAAS,EAAE,CAAC,CAAC;QACvE,OAAO,CAAC,KAAK,CAAC,yEAAyE,CAAC,CAAC;QACzF,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IAED,cAAc,GAAG,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,EAAE,SAAS,EAAE,CAAC;IAEtD,OAAO,CAAC,GAAG,CAAC,8BAA8B,UAAU,CAAC,MAAM,gBAAgB,CAAC,CAAC;IAC7E,UAAU,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,OAAO,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,IAAI,KAAK,CAAC,CAAC,IAAI,KAAK,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;IACpF,OAAO,CAAC,GAAG,CAAC,uBAAuB,SAAS,KAAK,SAAS,CAAC,QAAQ,IAAI,SAAS,CAAC,KAAK,GAAG,CAAC,CAAC;IAC3F,OAAO,CAAC,GAAG,EAAE,CAAC;AAChB,CAAC;AAED,kBAAkB;AAClB,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;IACnB,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IAC5B,OAAO,CAAC,GAAG,CAAC,wCAAwC,CAAC,CAAC;IACtD,OAAO,CAAC,GAAG,CAAC,mCAAmC,CAAC,CAAC;IACjD,OAAO,CAAC,GAAG,CAAC,+CAA+C,CAAC,CAAC;IAC7D,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IAC5B,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC;AAED,iBAAiB;AACjB,MAAM,aAAa,GAAG,IAAI,aAAa,EAAE,CAAC;AAC1C,MAAM,MAAM,GAAG,IAAI,gBAAgB,CACjC,SAAS,EACT,aAAa,EACb,OAAO,EACP,SAAS,EACT,MAAM,EACN,OAAO,CAAC,WAAW,EACnB,UAAU,EACV,cAAc,CACf,CAAC;AAEF,MAAM,OAAO,GAAG,MAAM,MAAM,CAAC,GAAG,EAAE,CAAC;AAEnC,qBAAqB;AACrB,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;AAC5B,OAAO,CAAC,GAAG,CAAC,wBAAwB,CAAC,CAAC;AACtC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;AAE5B,+BAA+B;AAC/B,IAAI,OAAO,CAAC,OAAO,EAAE,CAAC;IACpB,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,MAAM,CAAC,aAAa,CAAC,CAAC;IAC/C,MAAM,MAAM,CAAC,KAAK,EAAE,CAAC;IACrB,OAAO,CAAC,GAAG,CAAC,wBAAwB,OAAO,CAAC,OAAO,EAAE,CAAC,CAAC;AACzD,CAAC;AAED,kBAAkB;AAClB,MAAM,aAAa,CAAC,OAAO,EAAE,CAAC;AAE9B,iCAAiC;AACjC,IAAI,OAAO,CAAC,WAAW,GAAG,CAAC,EAAE,CAAC;IAC5B,MAAM,QAAQ,GAAG,IAAI,kBAAkB,CAAC,OAAO,CAAC,CAAC;IACjD,QAAQ,CAAC,MAAM,EAAE,CAAC;AACpB,CAAC"}

package/dist/runner/experiment.d.ts.map CHANGED Viewed

	@@ -1 +1 @@
1	- {"version":3,"file":"experiment.d.ts","sourceRoot":"","sources":["../../src/runner/experiment.ts"],"names":[],"mappings":"AAAA;;GAEG;AAGH,OAAO,KAAK,EAAE,SAAS,EAAe,SAAS,EAAE,MAAM,wBAAwB,CAAC;AAChF,OAAO,KAAK,EAAE,gBAAgB,EAAE,UAAU,EAAE,QAAQ,EAAuC,MAAM,aAAa,CAAC;AAC/G,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,qBAAqB,CAAC;AACzD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAC;AAMnE,qBAAa,gBAAgB;IAEzB,OAAO,CAAC,SAAS;IACjB,OAAO,CAAC,aAAa;IACrB,OAAO,CAAC,OAAO;IACf,OAAO,CAAC,SAAS;IACjB,OAAO,CAAC,MAAM;IACd,OAAO,CAAC,WAAW;IACnB,OAAO,CAAC,UAAU,CAAC;IACnB,OAAO,CAAC,cAAc,CAAC;gBAPf,SAAS,EAAE,SAAS,EACpB,aAAa,EAAE,aAAa,EAC5B,OAAO,EAAE,gBAAgB,EAAE,EAC3B,SAAS,EAAE,QAAQ,EAAE,EACrB,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,SAAS,CAAC,EACjC,WAAW,EAAE,MAAM,EACnB,UAAU,CAAC,EAAE,eAAe,EAAE,YAAA,EAC9B,cAAc,CAAC,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,SAAS,CAAA;KAAE,YAAA;IAG5D;;;;OAIG;IACG,GAAG,IAAI,OAAO,CAAC,UAAU,EAAE,CAAC;~~IA4GlC~~;;OAEG;YACW,aAAa;~~IAyC3B~~;;OAEG;YACW,kBAAkB;IA2BhC;;OAEG;IACH,OAAO,CAAC,cAAc;CAiCvB"}
1	+ {"version":3,"file":"experiment.d.ts","sourceRoot":"","sources":["../../src/runner/experiment.ts"],"names":[],"mappings":"AAAA;;GAEG;AAGH,OAAO,KAAK,EAAE,SAAS,EAAe,SAAS,EAAE,MAAM,wBAAwB,CAAC;AAChF,OAAO,KAAK,EAAE,gBAAgB,EAAE,UAAU,EAAE,QAAQ,EAAuC,MAAM,aAAa,CAAC;AAC/G,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,qBAAqB,CAAC;AACzD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAC;AAMnE,qBAAa,gBAAgB;IAEzB,OAAO,CAAC,SAAS;IACjB,OAAO,CAAC,aAAa;IACrB,OAAO,CAAC,OAAO;IACf,OAAO,CAAC,SAAS;IACjB,OAAO,CAAC,MAAM;IACd,OAAO,CAAC,WAAW;IACnB,OAAO,CAAC,UAAU,CAAC;IACnB,OAAO,CAAC,cAAc,CAAC;gBAPf,SAAS,EAAE,SAAS,EACpB,aAAa,EAAE,aAAa,EAC5B,OAAO,EAAE,gBAAgB,EAAE,EAC3B,SAAS,EAAE,QAAQ,EAAE,EACrB,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,SAAS,CAAC,EACjC,WAAW,EAAE,MAAM,EACnB,UAAU,CAAC,EAAE,eAAe,EAAE,YAAA,EAC9B,cAAc,CAAC,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,SAAS,CAAA;KAAE,YAAA;IAG5D;;;;OAIG;IACG,GAAG,IAAI,OAAO,CAAC,UAAU,EAAE,CAAC;IAkHlC;;OAEG;YACW,aAAa;IA8D3B;;OAEG;YACW,kBAAkB;IA2BhC;;OAEG;IACH,OAAO,CAAC,cAAc;CAiCvB"}

package/dist/runner/experiment.js CHANGED Viewed

@@ -32,6 +32,8 @@ export class ExperimentRunner {
     async run() {
         const allResults = [];
         const evaluationContexts = [];
+        // ドライバー切り替えをテストケースをまたいでトラッキング
+        let activeModelName = null;
         for (const testCase of this.testCases) {
             console.log('─'.repeat(80));
             console.log(`Test Case: ${testCase.name}`);
@@ -75,23 +77,26 @@ export class ExperimentRunner {
                 continue;
             }
             // Test with each model
-            let previousDriver = null;
-            let previousModelName = null;
             for (const { name: modelName, spec: modelSpec } of modelsToTest) {
-                console.log(`🤖 Testing with ${modelName} (${modelSpec.provider}:${modelSpec.model})`);
-                // Close previous driver if switching models
-                if (previousDriver && previousModelName && previousModelName !== modelName) {
-                    logger.verbose(`Switching from ${previousModelName} to ${modelName}, closing previous driver...`);
-                    await this.driverManager.close(previousModelName);
-                    previousDriver = null;
+                // モデル切り替え: 前のドライバーをクローズしてから新しいモデルを起動
+                if (activeModelName && activeModelName !== modelName) {
+                    logger.info(`Closing driver: ${activeModelName} (switching to ${modelName})`);
+                    await this.driverManager.close(activeModelName);
+                    activeModelName = null;
+                }
+                if (activeModelName === modelName) {
+                    logger.verbose(`Reusing driver for ${modelName}`);
+                }
+                else {
+                    logger.info(`Creating new driver for ${modelName} (${modelSpec.provider}:${modelSpec.model})`);
                 }
+                console.log(`🤖 Testing with ${modelName} (${modelSpec.provider}:${modelSpec.model})`);
                 // Get or create driver for this model
                 const driver = await this.driverManager.getOrCreate(this.aiService, modelName, modelSpec);
-                previousDriver = driver;
-                previousModelName = modelName;
+                activeModelName = modelName;
                 // Test each module
                 for (const { name, compiled, prompt } of compiledModules) {
-                    const runs = await this.runModuleTest(name, compiled, driver);
+                    const runs = await this.runModuleTest(name, compiled, driver, testCase);
                     allResults.push({
                         testCase: testCase.name,
                         model: modelName,
@@ -100,6 +105,8 @@ export class ExperimentRunner {
                             success: r.success,
                             elapsed: r.elapsed,
                             content: r.queryResult?.content || '',
+                            toolCalls: r.queryResult?.toolCalls,
+                            finishReason: r.queryResult?.finishReason,
                             error: r.error,
                         })),
                     });
@@ -124,7 +131,7 @@ export class ExperimentRunner {
     /**
      * Run module test with multiple repetitions
      */
-    async runModuleTest(moduleName, compiled, driver) {
+    async runModuleTest(moduleName, compiled, driver, testCase) {
         logger.verbose(`Running ${this.repeatCount} time(s) for module: ${moduleName}`);
         const runs = [];
         for (let i = 0; i < this.repeatCount; i++) {
@@ -134,9 +141,27 @@ export class ExperimentRunner {
                 const result = await driver.query(compiled, {
                     temperature: 0.7,
                     maxTokens: 2048,
+                    ...testCase.queryOptions,
                 });
                 const elapsed = Date.now() - startTime;
                 logger.verbose(`Module ${moduleName} run ${i + 1}: Success (${elapsed}ms)`);
+                // Display result summary (思考ブロックはプレビューから除外)
+                // パターン: <think>...</think> または 先頭から</think>まで（テンプレートが<think>を付与する場合）
+                const displayContent = result.content
+                    .replace(/<think>[\s\S]*?<\/think>\s*/g, '')
+                    .replace(/^[\s\S]*?<\/think>\s*/g, '');
+                const contentPreview = displayContent.length > 200
+                    ? displayContent.substring(0, 200) + '...'
+                    : displayContent;
+                console.log(`   ✅ [${moduleName}] run ${i + 1} (${elapsed}ms) finishReason=${result.finishReason || 'unknown'}`);
+                if (result.toolCalls && result.toolCalls.length > 0) {
+                    for (const tc of result.toolCalls) {
+                        console.log(`      🔧 toolCall: ${tc.name}(${JSON.stringify(tc.arguments)})`);
+                    }
+                }
+                if (contentPreview.trim()) {
+                    console.log(`      📝 ${contentPreview}`);
+                }
                 runs.push({
                     success: true,
                     elapsed,

package/dist/runner/experiment.js.map CHANGED Viewed

	@@ -1 +1 @@
1	- {"version":3,"file":"experiment.js","sourceRoot":"","sources":["../../src/runner/experiment.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,sBAAsB,EAAE,MAAM,wBAAwB,CAAC;AAKhE,OAAO,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AACjD,OAAO,EAAE,MAAM,IAAI,UAAU,EAAE,MAAM,cAAc,CAAC;AAEpD,MAAM,MAAM,GAAG,UAAU,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;AAE5C,MAAM,OAAO,gBAAgB;IAEjB;IACA;IACA;IACA;IACA;IACA;IACA;IACA;IARV,YACU,SAAoB,EACpB,aAA4B,EAC5B,OAA2B,EAC3B,SAAqB,EACrB,MAAiC,EACjC,WAAmB,EACnB,UAA8B,EAC9B,cAAkD;QAPlD,cAAS,GAAT,SAAS,CAAW;QACpB,kBAAa,GAAb,aAAa,CAAe;QAC5B,YAAO,GAAP,OAAO,CAAoB;QAC3B,cAAS,GAAT,SAAS,CAAY;QACrB,WAAM,GAAN,MAAM,CAA2B;QACjC,gBAAW,GAAX,WAAW,CAAQ;QACnB,eAAU,GAAV,UAAU,CAAoB;QAC9B,mBAAc,GAAd,cAAc,CAAoC;IACzD,CAAC;IAEJ;;;;OAIG;IACH,KAAK,CAAC,GAAG;QACP,MAAM,UAAU,GAAiB,EAAE,CAAC;QACpC,MAAM,kBAAkB,GAAwB,EAAE,CAAC;QAEnD,KAAK,MAAM,QAAQ,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;YACtC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;YAC5B,OAAO,CAAC,GAAG,CAAC,cAAc,QAAQ,CAAC,IAAI,EAAE,CAAC,CAAC;YAC3C,IAAI,QAAQ,CAAC,WAAW,EAAE,CAAC;gBACzB,OAAO,CAAC,GAAG,CAAC,gBAAgB,QAAQ,CAAC,WAAW,EAAE,CAAC,CAAC;YACtD,CAAC;YACD,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;YAC5B,OAAO,CAAC,GAAG,EAAE,CAAC;YAEd,qDAAqD;YACrD,MAAM,eAAe,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE;gBAChD,MAAM,CAAC,OAAO,CAAC,gCAAgC,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;gBAC9D,MAAM,QAAQ,GAAG,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC;gBAChD,MAAM,MAAM,GAAG,sBAAsB,CAAC,QAAQ,CAAC,CAAC;gBAChD,MAAM,CAAC,OAAO,CAAC,qBAAqB,MAAM,CAAC,IAAI,KAAK,MAAM,CAAC,MAAM,QAAQ,CAAC,CAAC;gBAE3E,OAAO;oBACL,IAAI,EAAE,MAAM,CAAC,IAAI;oBACjB,QAAQ;oBACR,MAAM;iBACP,CAAC;YACJ,CAAC,CAAC,CAAC;YAEH,sCAAsC;YACtC,IAAI,eAAe,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAC/B,IAAI,CAAC,cAAc,CAAC,eAAe,CAAC,CAAC;YACvC,CAAC;YAED,oDAAoD;YACpD,MAAM,YAAY,GAA6C,QAAQ,CAAC,MAAM;gBAC5E,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE;oBACzB,MAAM,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;oBAC/B,IAAI,CAAC,IAAI,EAAE,CAAC;wBACV,OAAO,CAAC,IAAI,CAAC,cAAc,IAAI,wCAAwC,CAAC,CAAC;wBACzE,OAAO,IAAI,CAAC;oBACd,CAAC;oBACD,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;gBACxB,CAAC,CAAC,CAAC,MAAM,CAAC,OAAO,CAA6C;gBAChE,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC;qBACxB,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,CAAC,EAAE,EAAE,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC;qBACrC,GAAG,CAAC,CAAC,CAAC,IAAI,EAAE,IAAI,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;YAE/C,IAAI,YAAY,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBAC9B,OAAO,CAAC,GAAG,CAAC,oDAAoD,CAAC,CAAC;gBAClE,OAAO,CAAC,GAAG,EAAE,CAAC;gBACd,SAAS;YACX,CAAC;YAED,uBAAuB;YACvB,~~IAAI,cAAc,GAAQ,IAAI,CAAC;YAC/B,IAAI,iBAAiB,GAAkB,IAAI,CAAC;YAE5C,~~KAAK,MAAM,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,YAAY,EAAE,CAAC;gBAChE,~~OAAO~~,CAAC,~~GAAG~~,CAAC,mBAAmB,~~SAAS~~,~~KAAK~~,SAAS,CAAC,~~QAAQ~~,IAAI,~~SAAS~~,CAAC,KAAK,~~GAAG~~,CAAC,CAAC;~~gBAEvF~~,~~4CAA4C;gBAC5C~~,IAAI,~~cAAc~~,~~IAAI~~,~~iBAAiB,~~IAAI,~~iBAAiB~~,KAAK,SAAS,EAAE,CAAC;~~oBAC3E~~,MAAM,CAAC,OAAO,CAAC,~~kBAAkB~~,~~iBAAiB~~,~~OAAO~~,~~SAAS~~,~~8BAA8B~~,CAAC,CAAC;~~oBAClG~~,MAAM,IAAI,CAAC,~~aAAa~~,~~CAAC~~,KAAK,CAAC,~~iBAAiB~~,CAAC,CAAC;~~oBAClD~~,~~cAAc~~,GAAG,IAAI,CAAC~~;gBACxB~~,CAAC;~~gBAED~~,sCAAsC;gBACtC,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,SAAS,EAAE,SAAS,EAAE,SAAS,CAAC,CAAC;gBAC1F,~~cAAc~~,GAAG,~~MAAM,CAAC;gBACxB,iBAAiB,GAAG,~~SAAS,CAAC;~~gBAE9B~~,mBAAmB;gBACnB,KAAK,MAAM,EAAE,IAAI,EAAE,QAAQ,EAAE,MAAM,EAAE,IAAI,eAAe,EAAE,CAAC;oBACzD,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,aAAa,CAAC,IAAI,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC;~~oBAE9D~~,UAAU,CAAC,IAAI,CAAC;wBACd,QAAQ,EAAE,QAAQ,CAAC,IAAI;wBACvB,KAAK,EAAE,SAAS;wBAChB,MAAM,EAAE,IAAI;wBACZ,IAAI,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;4BACnB,OAAO,EAAE,CAAC,CAAC,OAAO;4BAClB,OAAO,EAAE,CAAC,CAAC,OAAO;4BAClB,OAAO,EAAE,CAAC,CAAC,WAAW,EAAE,OAAO,IAAI,EAAE;4BACrC,KAAK,EAAE,CAAC,CAAC,KAAK;yBACf,CAAC,CAAC;qBACJ,CAAC,CAAC;oBAEH,iDAAiD;oBACjD,MAAM,cAAc,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;oBACnD,IAAI,cAAc,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;wBAC9B,kBAAkB,CAAC,IAAI,CAAC;4BACtB,UAAU,EAAE,IAAI;4BAChB,MAAM;4BACN,IAAI,EAAE,cAAc,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,WAAW,EAAE,CAAC,CAAC,WAAY,EAAE,CAAC,CAAC;yBACjE,CAAC,CAAC;oBACL,CAAC;gBACH,CAAC;YACH,CAAC;QACH,CAAC;QAED,kDAAkD;QAClD,IAAI,IAAI,CAAC,UAAU,IAAI,IAAI,CAAC,UAAU,CAAC,MAAM,GAAG,CAAC,IAAI,IAAI,CAAC,cAAc,EAAE,CAAC;YACzE,MAAM,IAAI,CAAC,kBAAkB,CAAC,kBAAkB,CAAC,CAAC;QACpD,CAAC;QAED,OAAO,UAAU,CAAC;IACpB,CAAC;IAED;;OAEG;IACK,KAAK,CAAC,aAAa,CACzB,UAAkB,EAClB,QAAa,EACb,MAAW;~~QAEX~~,MAAM,CAAC,OAAO,CAAC,WAAW,IAAI,CAAC,WAAW,wBAAwB,UAAU,EAAE,CAAC,CAAC;QAEhF,MAAM,IAAI,GAA4F,EAAE,CAAC;QAEzG,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,WAAW,EAAE,CAAC,EAAE,EAAE,CAAC;YAC1C,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,IAAI,IAAI,CAAC,WAAW,gBAAgB,UAAU,EAAE,CAAC,CAAC;YAE7E,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAC7B,IAAI,CAAC;gBACH,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,KAAK,CAAC,QAAQ,EAAE;oBAC1C,WAAW,EAAE,GAAG;oBAChB,SAAS,EAAE,IAAI;~~iBAChB~~,CAAC,CAAC;gBACH,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC;gBAEvC,MAAM,CAAC,OAAO,CAAC,UAAU,UAAU,QAAQ,CAAC,GAAG,CAAC,cAAc,OAAO,KAAK,CAAC,CAAC;~~gBAC5E~~,IAAI,CAAC,IAAI,CAAC;oBACR,OAAO,EAAE,IAAI;oBACb,OAAO;oBACP,WAAW,EAAE,MAAM;iBACpB,CAAC,CAAC;YACL,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC;gBACvC,MAAM,YAAY,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;gBAC5E,MAAM,CAAC,KAAK,CAAC,UAAU,UAAU,QAAQ,CAAC,GAAG,CAAC,YAAY,OAAO,QAAQ,YAAY,EAAE,CAAC,CAAC;gBACzF,IAAI,CAAC,IAAI,CAAC;oBACR,OAAO,EAAE,KAAK;oBACd,OAAO;oBACP,KAAK,EAAE,YAAY;iBACpB,CAAC,CAAC;YACL,CAAC;QACH,CAAC;QAED,OAAO,IAAI,CAAC;IACd,CAAC;IAED;;OAEG;IACK,KAAK,CAAC,kBAAkB,CAC9B,kBAAuC;QAEvC,OAAO,CAAC,GAAG,EAAE,CAAC;QACd,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;QAC5B,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,CAAC;QACnC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;QAC5B,OAAO,CAAC,GAAG,EAAE,CAAC;QAEd,MAAM,eAAe,GAAG,IAAI,eAAe,CAAC,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,cAAe,CAAC,IAAI,CAAC,CAAC;QACvF,MAAM,cAAc,GAAuB,EAAE,CAAC;QAE9C,2CAA2C;QAC3C,KAAK,MAAM,OAAO,IAAI,kBAAkB,EAAE,CAAC;YACzC,OAAO,CAAC,GAAG,CAAC,kBAAkB,OAAO,CAAC,UAAU,EAAE,CAAC,CAAC;YACpD,OAAO,CAAC,GAAG,EAAE,CAAC;YAEd,KAAK,MAAM,SAAS,IAAI,IAAI,CAAC,UAAW,EAAE,CAAC;gBACzC,MAAM,MAAM,GAAG,MAAM,eAAe,CAAC,QAAQ,CAAC,SAAS,EAAE,OAAO,CAAC,CAAC;gBAClE,cAAc,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;YAC9B,CAAC;QACH,CAAC;QAED,iCAAiC;QACjC,eAAe,CAAC,cAAc,CAAC,cAAc,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;IAClE,CAAC;IAED;;OAEG;IACK,cAAc,CAAC,eAAwD;QAC7E,OAAO,CAAC,GAAG,CAAC,uBAAuB,CAAC,CAAC;QAErC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,eAAe,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YAChD,MAAM,OAAO,GAAG,eAAe,CAAC,CAAC,CAAC,CAAC;YAEnC,KAAK,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,eAAe,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;gBACpD,MAAM,OAAO,GAAG,eAAe,CAAC,CAAC,CAAC,CAAC;gBAEnC,IAAI,OAAO,CAAC,MAAM,KAAK,OAAO,CAAC,MAAM,EAAE,CAAC;oBACtC,OAAO,CAAC,GAAG,CAAC,SAAS,OAAO,CAAC,IAAI,UAAU,OAAO,CAAC,IAAI,iBAAiB,CAAC,CAAC;gBAC5E,CAAC;qBAAM,CAAC;oBACN,OAAO,CAAC,GAAG,CAAC,WAAW,OAAO,CAAC,IAAI,UAAU,OAAO,CAAC,IAAI,UAAU,CAAC,CAAC;oBACrE,MAAM,CAAC,OAAO,CAAC,4BAA4B,CAAC,CAAC;oBAC7C,MAAM,CAAC,OAAO,CAAC,KAAK,OAAO,CAAC,IAAI,KAAK,OAAO,CAAC,MAAM,CAAC,MAAM,QAAQ,CAAC,CAAC;oBACpE,MAAM,CAAC,OAAO,CAAC,KAAK,OAAO,CAAC,IAAI,KAAK,OAAO,CAAC,MAAM,CAAC,MAAM,QAAQ,CAAC,CAAC;oBACpE,MAAM,CAAC,OAAO,CAAC,WAAW,OAAO,CAAC,MAAM,CAAC,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC,MAAM,QAAQ,CAAC,CAAC;oBAEjF,uCAAuC;oBACvC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;wBAChF,IAAI,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,KAAK,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC;4BAC5C,MAAM,CAAC,OAAO,CAAC,4BAA4B,CAAC,GAAG,CAAC,CAAC;4BACjD,MAAM,CAAC,OAAO,CAAC,OAAO,OAAO,CAAC,IAAI,KAAK,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,EAAE,CAAC,GAAG,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;4BAC9F,MAAM,CAAC,OAAO,CAAC,OAAO,OAAO,CAAC,IAAI,KAAK,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,EAAE,CAAC,GAAG,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;4BAC9F,MAAM;wBACR,CAAC;oBACH,CAAC;gBACH,CAAC;YACH,CAAC;QACH,CAAC;QAED,OAAO,CAAC,GAAG,EAAE,CAAC;IAChB,CAAC;CACF"}
1	+ {"version":3,"file":"experiment.js","sourceRoot":"","sources":["../../src/runner/experiment.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,sBAAsB,EAAE,MAAM,wBAAwB,CAAC;AAKhE,OAAO,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AACjD,OAAO,EAAE,MAAM,IAAI,UAAU,EAAE,MAAM,cAAc,CAAC;AAEpD,MAAM,MAAM,GAAG,UAAU,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;AAE5C,MAAM,OAAO,gBAAgB;IAEjB;IACA;IACA;IACA;IACA;IACA;IACA;IACA;IARV,YACU,SAAoB,EACpB,aAA4B,EAC5B,OAA2B,EAC3B,SAAqB,EACrB,MAAiC,EACjC,WAAmB,EACnB,UAA8B,EAC9B,cAAkD;QAPlD,cAAS,GAAT,SAAS,CAAW;QACpB,kBAAa,GAAb,aAAa,CAAe;QAC5B,YAAO,GAAP,OAAO,CAAoB;QAC3B,cAAS,GAAT,SAAS,CAAY;QACrB,WAAM,GAAN,MAAM,CAA2B;QACjC,gBAAW,GAAX,WAAW,CAAQ;QACnB,eAAU,GAAV,UAAU,CAAoB;QAC9B,mBAAc,GAAd,cAAc,CAAoC;IACzD,CAAC;IAEJ;;;;OAIG;IACH,KAAK,CAAC,GAAG;QACP,MAAM,UAAU,GAAiB,EAAE,CAAC;QACpC,MAAM,kBAAkB,GAAwB,EAAE,CAAC;QAEnD,8BAA8B;QAC9B,IAAI,eAAe,GAAkB,IAAI,CAAC;QAE1C,KAAK,MAAM,QAAQ,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;YACtC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;YAC5B,OAAO,CAAC,GAAG,CAAC,cAAc,QAAQ,CAAC,IAAI,EAAE,CAAC,CAAC;YAC3C,IAAI,QAAQ,CAAC,WAAW,EAAE,CAAC;gBACzB,OAAO,CAAC,GAAG,CAAC,gBAAgB,QAAQ,CAAC,WAAW,EAAE,CAAC,CAAC;YACtD,CAAC;YACD,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;YAC5B,OAAO,CAAC,GAAG,EAAE,CAAC;YAEd,qDAAqD;YACrD,MAAM,eAAe,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE;gBAChD,MAAM,CAAC,OAAO,CAAC,gCAAgC,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;gBAC9D,MAAM,QAAQ,GAAG,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC;gBAChD,MAAM,MAAM,GAAG,sBAAsB,CAAC,QAAQ,CAAC,CAAC;gBAChD,MAAM,CAAC,OAAO,CAAC,qBAAqB,MAAM,CAAC,IAAI,KAAK,MAAM,CAAC,MAAM,QAAQ,CAAC,CAAC;gBAE3E,OAAO;oBACL,IAAI,EAAE,MAAM,CAAC,IAAI;oBACjB,QAAQ;oBACR,MAAM;iBACP,CAAC;YACJ,CAAC,CAAC,CAAC;YAEH,sCAAsC;YACtC,IAAI,eAAe,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAC/B,IAAI,CAAC,cAAc,CAAC,eAAe,CAAC,CAAC;YACvC,CAAC;YAED,oDAAoD;YACpD,MAAM,YAAY,GAA6C,QAAQ,CAAC,MAAM;gBAC5E,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE;oBACzB,MAAM,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;oBAC/B,IAAI,CAAC,IAAI,EAAE,CAAC;wBACV,OAAO,CAAC,IAAI,CAAC,cAAc,IAAI,wCAAwC,CAAC,CAAC;wBACzE,OAAO,IAAI,CAAC;oBACd,CAAC;oBACD,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;gBACxB,CAAC,CAAC,CAAC,MAAM,CAAC,OAAO,CAA6C;gBAChE,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC;qBACxB,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,CAAC,EAAE,EAAE,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC;qBACrC,GAAG,CAAC,CAAC,CAAC,IAAI,EAAE,IAAI,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;YAE/C,IAAI,YAAY,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBAC9B,OAAO,CAAC,GAAG,CAAC,oDAAoD,CAAC,CAAC;gBAClE,OAAO,CAAC,GAAG,EAAE,CAAC;gBACd,SAAS;YACX,CAAC;YAED,uBAAuB;YACvB,KAAK,MAAM,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,YAAY,EAAE,CAAC;gBAChE,qCAAqC;gBACrC,IAAI,eAAe,IAAI,eAAe,KAAK,SAAS,EAAE,CAAC;oBACrD,MAAM,CAAC,IAAI,CAAC,mBAAmB,eAAe,kBAAkB,SAAS,GAAG,CAAC,CAAC;oBAC9E,MAAM,IAAI,CAAC,aAAa,CAAC,KAAK,CAAC,eAAe,CAAC,CAAC;oBAChD,eAAe,GAAG,IAAI,CAAC;gBACzB,CAAC;gBAED,IAAI,eAAe,KAAK,SAAS,EAAE,CAAC;oBAClC,MAAM,CAAC,OAAO,CAAC,sBAAsB,SAAS,EAAE,CAAC,CAAC;gBACpD,CAAC;qBAAM,CAAC;oBACN,MAAM,CAAC,IAAI,CAAC,2BAA2B,SAAS,KAAK,SAAS,CAAC,QAAQ,IAAI,SAAS,CAAC,KAAK,GAAG,CAAC,CAAC;gBACjG,CAAC;gBACD,OAAO,CAAC,GAAG,CAAC,mBAAmB,SAAS,KAAK,SAAS,CAAC,QAAQ,IAAI,SAAS,CAAC,KAAK,GAAG,CAAC,CAAC;gBAEvF,sCAAsC;gBACtC,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,SAAS,EAAE,SAAS,EAAE,SAAS,CAAC,CAAC;gBAC1F,eAAe,GAAG,SAAS,CAAC;gBAE5B,mBAAmB;gBACnB,KAAK,MAAM,EAAE,IAAI,EAAE,QAAQ,EAAE,MAAM,EAAE,IAAI,eAAe,EAAE,CAAC;oBACzD,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,aAAa,CAAC,IAAI,EAAE,QAAQ,EAAE,MAAM,EAAE,QAAQ,CAAC,CAAC;oBAExE,UAAU,CAAC,IAAI,CAAC;wBACd,QAAQ,EAAE,QAAQ,CAAC,IAAI;wBACvB,KAAK,EAAE,SAAS;wBAChB,MAAM,EAAE,IAAI;wBACZ,IAAI,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;4BACnB,OAAO,EAAE,CAAC,CAAC,OAAO;4BAClB,OAAO,EAAE,CAAC,CAAC,OAAO;4BAClB,OAAO,EAAE,CAAC,CAAC,WAAW,EAAE,OAAO,IAAI,EAAE;4BACrC,SAAS,EAAE,CAAC,CAAC,WAAW,EAAE,SAAS;4BACnC,YAAY,EAAE,CAAC,CAAC,WAAW,EAAE,YAAY;4BACzC,KAAK,EAAE,CAAC,CAAC,KAAK;yBACf,CAAC,CAAC;qBACJ,CAAC,CAAC;oBAEH,iDAAiD;oBACjD,MAAM,cAAc,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;oBACnD,IAAI,cAAc,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;wBAC9B,kBAAkB,CAAC,IAAI,CAAC;4BACtB,UAAU,EAAE,IAAI;4BAChB,MAAM;4BACN,IAAI,EAAE,cAAc,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,WAAW,EAAE,CAAC,CAAC,WAAY,EAAE,CAAC,CAAC;yBACjE,CAAC,CAAC;oBACL,CAAC;gBACH,CAAC;YACH,CAAC;QACH,CAAC;QAED,kDAAkD;QAClD,IAAI,IAAI,CAAC,UAAU,IAAI,IAAI,CAAC,UAAU,CAAC,MAAM,GAAG,CAAC,IAAI,IAAI,CAAC,cAAc,EAAE,CAAC;YACzE,MAAM,IAAI,CAAC,kBAAkB,CAAC,kBAAkB,CAAC,CAAC;QACpD,CAAC;QAED,OAAO,UAAU,CAAC;IACpB,CAAC;IAED;;OAEG;IACK,KAAK,CAAC,aAAa,CACzB,UAAkB,EAClB,QAAa,EACb,MAAW,EACX,QAAkB;QAElB,MAAM,CAAC,OAAO,CAAC,WAAW,IAAI,CAAC,WAAW,wBAAwB,UAAU,EAAE,CAAC,CAAC;QAEhF,MAAM,IAAI,GAA4F,EAAE,CAAC;QAEzG,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,WAAW,EAAE,CAAC,EAAE,EAAE,CAAC;YAC1C,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,IAAI,IAAI,CAAC,WAAW,gBAAgB,UAAU,EAAE,CAAC,CAAC;YAE7E,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAC7B,IAAI,CAAC;gBACH,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,KAAK,CAAC,QAAQ,EAAE;oBAC1C,WAAW,EAAE,GAAG;oBAChB,SAAS,EAAE,IAAI;oBACf,GAAG,QAAQ,CAAC,YAAY;iBACzB,CAAC,CAAC;gBACH,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC;gBAEvC,MAAM,CAAC,OAAO,CAAC,UAAU,UAAU,QAAQ,CAAC,GAAG,CAAC,cAAc,OAAO,KAAK,CAAC,CAAC;gBAE5E,4CAA4C;gBAC5C,qEAAqE;gBACrE,MAAM,cAAc,GAAG,MAAM,CAAC,OAAO;qBAClC,OAAO,CAAC,8BAA8B,EAAE,EAAE,CAAC;qBAC3C,OAAO,CAAC,wBAAwB,EAAE,EAAE,CAAC,CAAC;gBACzC,MAAM,cAAc,GAAG,cAAc,CAAC,MAAM,GAAG,GAAG;oBAChD,CAAC,CAAC,cAAc,CAAC,SAAS,CAAC,CAAC,EAAE,GAAG,CAAC,GAAG,KAAK;oBAC1C,CAAC,CAAC,cAAc,CAAC;gBACnB,OAAO,CAAC,GAAG,CAAC,SAAS,UAAU,SAAS,CAAC,GAAG,CAAC,KAAK,OAAO,oBAAoB,MAAM,CAAC,YAAY,IAAI,SAAS,EAAE,CAAC,CAAC;gBACjH,IAAI,MAAM,CAAC,SAAS,IAAI,MAAM,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;oBACpD,KAAK,MAAM,EAAE,IAAI,MAAM,CAAC,SAAS,EAAE,CAAC;wBAClC,OAAO,CAAC,GAAG,CAAC,sBAAsB,EAAE,CAAC,IAAI,IAAI,IAAI,CAAC,SAAS,CAAC,EAAE,CAAC,SAAS,CAAC,GAAG,CAAC,CAAC;oBAChF,CAAC;gBACH,CAAC;gBACD,IAAI,cAAc,CAAC,IAAI,EAAE,EAAE,CAAC;oBAC1B,OAAO,CAAC,GAAG,CAAC,YAAY,cAAc,EAAE,CAAC,CAAC;gBAC5C,CAAC;gBAED,IAAI,CAAC,IAAI,CAAC;oBACR,OAAO,EAAE,IAAI;oBACb,OAAO;oBACP,WAAW,EAAE,MAAM;iBACpB,CAAC,CAAC;YACL,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC;gBACvC,MAAM,YAAY,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;gBAC5E,MAAM,CAAC,KAAK,CAAC,UAAU,UAAU,QAAQ,CAAC,GAAG,CAAC,YAAY,OAAO,QAAQ,YAAY,EAAE,CAAC,CAAC;gBACzF,IAAI,CAAC,IAAI,CAAC;oBACR,OAAO,EAAE,KAAK;oBACd,OAAO;oBACP,KAAK,EAAE,YAAY;iBACpB,CAAC,CAAC;YACL,CAAC;QACH,CAAC;QAED,OAAO,IAAI,CAAC;IACd,CAAC;IAED;;OAEG;IACK,KAAK,CAAC,kBAAkB,CAC9B,kBAAuC;QAEvC,OAAO,CAAC,GAAG,EAAE,CAAC;QACd,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;QAC5B,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,CAAC;QACnC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;QAC5B,OAAO,CAAC,GAAG,EAAE,CAAC;QAEd,MAAM,eAAe,GAAG,IAAI,eAAe,CAAC,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,cAAe,CAAC,IAAI,CAAC,CAAC;QACvF,MAAM,cAAc,GAAuB,EAAE,CAAC;QAE9C,2CAA2C;QAC3C,KAAK,MAAM,OAAO,IAAI,kBAAkB,EAAE,CAAC;YACzC,OAAO,CAAC,GAAG,CAAC,kBAAkB,OAAO,CAAC,UAAU,EAAE,CAAC,CAAC;YACpD,OAAO,CAAC,GAAG,EAAE,CAAC;YAEd,KAAK,MAAM,SAAS,IAAI,IAAI,CAAC,UAAW,EAAE,CAAC;gBACzC,MAAM,MAAM,GAAG,MAAM,eAAe,CAAC,QAAQ,CAAC,SAAS,EAAE,OAAO,CAAC,CAAC;gBAClE,cAAc,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;YAC9B,CAAC;QACH,CAAC;QAED,iCAAiC;QACjC,eAAe,CAAC,cAAc,CAAC,cAAc,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;IAClE,CAAC;IAED;;OAEG;IACK,cAAc,CAAC,eAAwD;QAC7E,OAAO,CAAC,GAAG,CAAC,uBAAuB,CAAC,CAAC;QAErC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,eAAe,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YAChD,MAAM,OAAO,GAAG,eAAe,CAAC,CAAC,CAAC,CAAC;YAEnC,KAAK,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,eAAe,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;gBACpD,MAAM,OAAO,GAAG,eAAe,CAAC,CAAC,CAAC,CAAC;gBAEnC,IAAI,OAAO,CAAC,MAAM,KAAK,OAAO,CAAC,MAAM,EAAE,CAAC;oBACtC,OAAO,CAAC,GAAG,CAAC,SAAS,OAAO,CAAC,IAAI,UAAU,OAAO,CAAC,IAAI,iBAAiB,CAAC,CAAC;gBAC5E,CAAC;qBAAM,CAAC;oBACN,OAAO,CAAC,GAAG,CAAC,WAAW,OAAO,CAAC,IAAI,UAAU,OAAO,CAAC,IAAI,UAAU,CAAC,CAAC;oBACrE,MAAM,CAAC,OAAO,CAAC,4BAA4B,CAAC,CAAC;oBAC7C,MAAM,CAAC,OAAO,CAAC,KAAK,OAAO,CAAC,IAAI,KAAK,OAAO,CAAC,MAAM,CAAC,MAAM,QAAQ,CAAC,CAAC;oBACpE,MAAM,CAAC,OAAO,CAAC,KAAK,OAAO,CAAC,IAAI,KAAK,OAAO,CAAC,MAAM,CAAC,MAAM,QAAQ,CAAC,CAAC;oBACpE,MAAM,CAAC,OAAO,CAAC,WAAW,OAAO,CAAC,MAAM,CAAC,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC,MAAM,QAAQ,CAAC,CAAC;oBAEjF,uCAAuC;oBACvC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;wBAChF,IAAI,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,KAAK,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC;4BAC5C,MAAM,CAAC,OAAO,CAAC,4BAA4B,CAAC,GAAG,CAAC,CAAC;4BACjD,MAAM,CAAC,OAAO,CAAC,OAAO,OAAO,CAAC,IAAI,KAAK,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,EAAE,CAAC,GAAG,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;4BAC9F,MAAM,CAAC,OAAO,CAAC,OAAO,OAAO,CAAC,IAAI,KAAK,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,EAAE,CAAC,GAAG,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;4BAC9F,MAAM;wBACR,CAAC;oBACH,CAAC;gBACH,CAAC;YACH,CAAC;QACH,CAAC;QAED,OAAO,CAAC,GAAG,EAAE,CAAC;IAChB,CAAC;CACF"}

package/dist/types.d.ts CHANGED Viewed

@@ -2,7 +2,7 @@
  * Common type definitions for module comparison experiments
  */
 import type { PromptModule } from '@modular-prompt/core';
-import type { QueryResult } from '@modular-prompt/driver';
+import type { QueryResult, QueryOptions, ToolCall } from '@modular-prompt/driver';
 /**
  * Test case definition
  */
@@ -15,6 +15,8 @@ export interface TestCase {
     input: any;
     /** Model names to use for this test case (optional, uses all enabled models if not specified) */
     models?: string[];
+    /** Query options for this test case (tools, temperature, etc.) */
+    queryOptions?: Partial<QueryOptions>;
 }
 /**
  * Result of a single run
@@ -23,6 +25,8 @@ export interface RunResult {
     success: boolean;
     elapsed: number;
     content: string;
+    toolCalls?: ToolCall[];
+    finishReason?: string;
     error?: string;
 }
 /**

package/dist/types.d.ts.map CHANGED Viewed

	@@ -1 +1 @@
1	- {"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,sBAAsB,CAAC;AACzD,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,wBAAwB,CAAC;~~AAE1D~~;;GAEG;AACH,MAAM,WAAW,QAAQ;IACvB,qBAAqB;IACrB,IAAI,EAAE,MAAM,CAAC;IACb,4BAA4B;IAC5B,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,+CAA+C;IAC/C,KAAK,EAAE,GAAG,CAAC;IACX,iGAAiG;IACjG,MAAM,CAAC,EAAE,MAAM,EAAE,CAAC;~~CACnB~~;AAED;;GAEG;AACH,MAAM,WAAW,SAAS;IACxB,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED;;GAEG;AACH,MAAM,WAAW,UAAU;IACzB,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,IAAI,EAAE,SAAS,EAAE,CAAC;CACnB;AAED;;GAEG;AACH,MAAM,WAAW,iBAAiB;IAChC,UAAU,EAAE,MAAM,CAAC;IACnB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,YAAY,CAAC,EAAE,MAAM,EAAE,CAAC;IACxB,WAAW,EAAE,MAAM,CAAC;CACrB;AAED;;GAEG;AACH,MAAM,WAAW,gBAAgB;IAC/B,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,EAAE,MAAM,CAAC;IACpB,OAAO,EAAE,CAAC,OAAO,EAAE,GAAG,KAAK,GAAG,CAAC;CAChC;AAED;;GAEG;AACH,MAAM,WAAW,iBAAiB;IAChC,UAAU,EAAE,MAAM,CAAC;IACnB,MAAM,EAAE,MAAM,CAAC;IACf,IAAI,EAAE,KAAK,CAAC;QACV,WAAW,EAAE,WAAW,CAAC;KAC1B,CAAC,CAAC;CACJ;AAED;;GAEG;AACH,MAAM,WAAW,gBAAgB;IAC/B,SAAS,EAAE,MAAM,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;IAC9B,GAAG,CAAC,EAAE,GAAG,CAAC;IACV,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED;;GAEG;AACH,MAAM,WAAW,aAAa;IAC5B,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;;;;OASG;IACH,WAAW,EAAE,MAAM,CAAC;IACpB,QAAQ,EAAE,CAAC,OAAO,EAAE,iBAAiB,KAAK,OAAO,CAAC,gBAAgB,CAAC,CAAC;CACrE;AAED;;GAEG;AACH,MAAM,WAAW,eAAe;IAC9B,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;;;;OASG;IACH,WAAW,EAAE,MAAM,CAAC;IACpB,MAAM,EAAE,YAAY,CAAC,iBAAiB,CAAC,CAAC;CACzC;AAED;;GAEG;AACH,MAAM,MAAM,kBAAkB,GAC1B;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,IAAI,EAAE,MAAM,CAAC;IAAC,WAAW,CAAC,EAAE,MAAM,CAAA;CAAE,GACpD;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,YAAY,CAAC,iBAAiB,CAAC,CAAC;IAAC,WAAW,CAAC,EAAE,MAAM,CAAA;CAAE,GAC/E;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,WAAW,CAAC,EAAE,MAAM,CAAA;CAAE,CAAC;AAE3C;;GAEG;AACH,MAAM,WAAW,yBAA0B,SAAQ,iBAAiB;IAClE,gBAAgB,CAAC,EAAE,OAAO,CAAC;IAC3B,eAAe,CAAC,EAAE,MAAM,EAAE,CAAC;IAC3B,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB"}
1	+ {"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,sBAAsB,CAAC;AACzD,OAAO,KAAK,EAAE,WAAW,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,wBAAwB,CAAC;AAElF;;GAEG;AACH,MAAM,WAAW,QAAQ;IACvB,qBAAqB;IACrB,IAAI,EAAE,MAAM,CAAC;IACb,4BAA4B;IAC5B,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,+CAA+C;IAC/C,KAAK,EAAE,GAAG,CAAC;IACX,iGAAiG;IACjG,MAAM,CAAC,EAAE,MAAM,EAAE,CAAC;IAClB,kEAAkE;IAClE,YAAY,CAAC,EAAE,OAAO,CAAC,YAAY,CAAC,CAAC;CACtC;AAED;;GAEG;AACH,MAAM,WAAW,SAAS;IACxB,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,CAAC,EAAE,QAAQ,EAAE,CAAC;IACvB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED;;GAEG;AACH,MAAM,WAAW,UAAU;IACzB,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,IAAI,EAAE,SAAS,EAAE,CAAC;CACnB;AAED;;GAEG;AACH,MAAM,WAAW,iBAAiB;IAChC,UAAU,EAAE,MAAM,CAAC;IACnB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,YAAY,CAAC,EAAE,MAAM,EAAE,CAAC;IACxB,WAAW,EAAE,MAAM,CAAC;CACrB;AAED;;GAEG;AACH,MAAM,WAAW,gBAAgB;IAC/B,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,EAAE,MAAM,CAAC;IACpB,OAAO,EAAE,CAAC,OAAO,EAAE,GAAG,KAAK,GAAG,CAAC;CAChC;AAED;;GAEG;AACH,MAAM,WAAW,iBAAiB;IAChC,UAAU,EAAE,MAAM,CAAC;IACnB,MAAM,EAAE,MAAM,CAAC;IACf,IAAI,EAAE,KAAK,CAAC;QACV,WAAW,EAAE,WAAW,CAAC;KAC1B,CAAC,CAAC;CACJ;AAED;;GAEG;AACH,MAAM,WAAW,gBAAgB;IAC/B,SAAS,EAAE,MAAM,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;IAC9B,GAAG,CAAC,EAAE,GAAG,CAAC;IACV,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED;;GAEG;AACH,MAAM,WAAW,aAAa;IAC5B,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;;;;OASG;IACH,WAAW,EAAE,MAAM,CAAC;IACpB,QAAQ,EAAE,CAAC,OAAO,EAAE,iBAAiB,KAAK,OAAO,CAAC,gBAAgB,CAAC,CAAC;CACrE;AAED;;GAEG;AACH,MAAM,WAAW,eAAe;IAC9B,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;;;;OASG;IACH,WAAW,EAAE,MAAM,CAAC;IACpB,MAAM,EAAE,YAAY,CAAC,iBAAiB,CAAC,CAAC;CACzC;AAED;;GAEG;AACH,MAAM,MAAM,kBAAkB,GAC1B;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,IAAI,EAAE,MAAM,CAAC;IAAC,WAAW,CAAC,EAAE,MAAM,CAAA;CAAE,GACpD;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,YAAY,CAAC,iBAAiB,CAAC,CAAC;IAAC,WAAW,CAAC,EAAE,MAAM,CAAA;CAAE,GAC/E;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,WAAW,CAAC,EAAE,MAAM,CAAA;CAAE,CAAC;AAE3C;;GAEG;AACH,MAAM,WAAW,yBAA0B,SAAQ,iBAAiB;IAClE,gBAAgB,CAAC,EAAE,OAAO,CAAC;IAC3B,eAAe,CAAC,EAAE,MAAM,EAAE,CAAC;IAC3B,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB"}

package/examples/tools-experiment.yaml ADDED Viewed

@@ -0,0 +1,105 @@
+# Tools実験設定
+# MLXドライバーでのtool calling動作確認
+models:
+  qwen3-4b:
+    model: "mlx-community/Qwen3-4B-Thinking-2507-heretic-8bit"
+    provider: "mlx"
+    capabilities: ["local", "tools"]
+    priority: 10
+  gemma3-12b:
+    model: "mlx-community/gemma-3-12b-it-qat-4bit"
+    provider: "mlx"
+    capabilities: ["local", "tools"]
+    priority: 20
+  lfm2.5-jp:
+    model: LiquidAI/LFM2.5-1.2B-JP-MLX-8bit
+    provider: "mlx"
+    capabilities: ["local", "fast", "japanese"]
+    priority: 20
+  lfm2.5-instruct:
+    model: LiquidAI/LFM2.5-1.2B-Instruct-MLX-8bit
+    provider: "mlx"
+    capabilities: ["local", "fast", "japanese"]
+    priority: 20
+    disabled: true
+  lfm2.5-thinking:
+    model: LiquidAI/LFM2.5-1.2B-Thinking-MLX-8bit
+    provider: "mlx"
+    capabilities: ["local", "fast", "japanese"]
+    priority: 20
+    disabled: true
+drivers:
+  mlx: {}
+modules:
+  - name: tools-test
+    path: ./tools-test-module.mjs
+    description: "ツール呼び出し実験用"
+testCases:
+  # --- gemma3-12b ---
+  - name: "[gemma3] 天気ツール呼び出し"
+    description: "get_weatherツールを呼び出すことを期待"
+    models: ["gemma3-12b"]
+    input:
+      question: "東京の天気を教えてください。"
+    queryOptions: &tool_weather
+      temperature: 0.3
+      maxTokens: 512
+      tools: &tools_def
+        - name: get_weather
+          description: "指定された場所の現在の天気を取得する"
+          parameters:
+            type: object
+            properties:
+              location:
+                type: string
+                description: "天気を取得する場所（都市名）"
+            required:
+              - location
+      toolChoice: auto
+  - name: "[gemma3] ツール不要の質問"
+    description: "ツールを呼び出さずにテキストで回答することを期待"
+    models: ["gemma3-12b"]
+    input:
+      question: "1 + 1 は何ですか？"
+    queryOptions: &tool_math
+      temperature: 0.3
+      maxTokens: 1024
+      tools: *tools_def
+      toolChoice: auto
+  # # --- qwen3-4b ---
+  # - name: "[qwen3] 天気ツール呼び出し"
+  #   description: "get_weatherツールを呼び出すことを期待"
+  #   models: ["qwen3-4b"]
+  #   input:
+  #     question: "東京の天気を教えてください。"
+  #   queryOptions: *tool_weather
+  # - name: "[qwen3] ツール不要の質問"
+  #   description: "ツールを呼び出さずにテキストで回答することを期待"
+  #   models: ["qwen3-4b"]
+  #   input:
+  #     question: "1 + 1 は何ですか？"
+  #   queryOptions: *tool_math
+  # --- lfm2.5 jp ---
+  - name: "[lfm2.5] 天気ツール呼び出し"
+    description: "get_weatherツールを呼び出すことを期待"
+    models: ["lfm2.5-instruct"]
+    input:
+      question: "東京の天気を教えてください。"
+    queryOptions: *tool_weather
+  - name: "[lfm2.5] ツール不要の質問"
+    description: "ツールを呼び出さずにテキストで回答することを期待"
+    models: ["lfm2.5-instruct"]
+    input:
+      question: "1 + 1 は何ですか？"
+    queryOptions: *tool_math
+evaluators: []

package/examples/tools-test-module.mjs ADDED Viewed

@@ -0,0 +1,29 @@
+/**
+ * Tools実験用モジュール
+ */
+import { compile } from '@modular-prompt/core';
+export default {
+  name: 'tools-test',
+  description: 'ツール呼び出し実験用モジュール',
+  compile: (context) => {
+    return compile({
+      objective: [
+        '- あなたは利用者からの質問に答えるアシスタントです。',
+      ],
+      instructions: [
+        '- 質問の内容に応じて、適切なツールを使ってください。',
+        '  - ツールの結果が返ってくるまで、推測で答えないでください。',
+        '- 必要がない場合は通常の応答を返します。',
+      ],
+      messages: [
+        {
+          type: 'message',
+          role: 'user',
+          content: context.question || 'Hello',
+        },
+      ],
+    });
+  },
+};

package/examples/tools-test-module.ts ADDED Viewed

@@ -0,0 +1,29 @@
+/**
+ * Tools実験用モジュール
+ */
+import { compile } from '@modular-prompt/core';
+export default {
+  name: 'tools-test',
+  description: 'ツール呼び出し実験用モジュール',
+  compile: (context: any) => {
+    return compile({
+      objective: [
+        'あなたはツールを使って質問に答えるアシスタントです。',
+        '必要に応じてツールを呼び出してください。',
+      ],
+      instructions: [
+        '質問の内容に応じて、適切なツールを使ってください。',
+        'ツールの結果が返ってくるまで、推測で答えないでください。',
+      ],
+      messages: [
+        {
+          type: 'message' as const,
+          role: 'user' as const,
+          content: context.question || 'Hello',
+        },
+      ],
+    });
+  },
+};

package/package.json CHANGED Viewed

@@ -1,6 +1,6 @@
 {
   "name": "@modular-prompt/experiment",
-  "version": "0.1.10",
+  "version": "0.3.1",
   "description": "Experiment framework for comparing and evaluating prompt modules",
   "type": "module",
   "main": "./dist/index.js",
@@ -17,6 +17,7 @@
   "files": [
     "dist",
     "examples",
+    "skills",
     "README.md"
   ],
   "dependencies": {
@@ -24,9 +25,9 @@
     "jiti": "^2.4.2",
     "yaml": "^2.3.4",
     "zod": "^3.22.4",
-    "@modular-prompt/core": "0.1.12",
-    "@modular-prompt/driver": "0.6.3",
-    "@modular-prompt/utils": "0.2.3"
+    "@modular-prompt/core": "0.1.13",
+    "@modular-prompt/driver": "0.8.1",
+    "@modular-prompt/utils": "0.2.4"
   },
   "devDependencies": {
     "@eslint/js": "^9.34.0",
@@ -64,7 +65,8 @@
     "test": "vitest",
     "test:ui": "vitest --ui",
     "test:run": "vitest run",
-    "clean": "rm -rf dist",
+    "copy-skills": "mkdir -p skills/experiment && cp ../../skills/experiment/SKILL.md skills/experiment/SKILL.md",
+    "clean": "rm -rf dist skills",
     "lint": "eslint src",
     "typecheck": "tsc --noEmit"
   }

package/skills/experiment/SKILL.md ADDED Viewed

@@ -0,0 +1,292 @@
+---
+name: experiment
+description: modular-promptの実験フレームワーク（@modular-prompt/experiment）の使い方ガイド。プロンプトモジュールの比較・評価実験の設定、実行、評価器の定義を参照する。
+---
+# 実験フレームワーク使い方ガイド
+## 実験フレームワークとは
+`@modular-prompt/experiment` は、複数のプロンプトモジュールを同一条件下で比較・評価するためのフレームワーク。YAML設定で実験を定義し、CLIまたはプログラマティックに実行できる。
+### ユースケース
+- **プロンプト比較**: 異なるプロンプト構造の効果を比較検証
+- **モジュール分離検証**: モジュール化したプロンプトが同等の出力を生成するか確認
+- **品質評価**: 繰り返し実行による出力の安定性・一貫性の評価
+- **マルチモデルテスト**: 異なるLLMプロバイダーでの動作比較
+## CLI
+```bash
+# 設定検証・実行計画表示（まずこれで確認）
+npx modular-experiment config.yaml --dry-run
+# 実験実行
+npx modular-experiment config.yaml
+# 評価付き実行
+npx modular-experiment config.yaml --evaluate
+# 複数回実行（統計用）
+npx modular-experiment config.yaml --repeat 10
+# 特定モジュール・テストケースのみ
+npx modular-experiment config.yaml --modules my-module --test-case "Basic Test"
+# 詳細ログ出力
+npx modular-experiment config.yaml --log-file experiment.jsonl --verbose
+```
+### CLIオプション
+| オプション | 説明 |
+|-----------|------|
+| `<config>` | 設定ファイルパス（YAML or TypeScript） |
+| `--dry-run` | 実行計画のみ表示 |
+| `--evaluate` | 評価フェーズを有効化 |
+| `--repeat <count>` | 繰り返し回数（デフォルト: 1） |
+| `--modules <names>` | カンマ区切りモジュール名フィルター |
+| `--test-case <name>` | テストケース名フィルター |
+| `--model <provider>` | モデルプロバイダーフィルター |
+| `--evaluators <names>` | カンマ区切り評価器名フィルター |
+| `--log-file <path>` | JSONLログファイルパス |
+| `--verbose` | 詳細な内部操作を表示 |
+## 設定ファイル（YAML）
+```yaml
+# モデル定義
+models:
+  gpt4o:
+    provider: openai
+    model: gpt-4o
+    capabilities: ["streaming", "tools", "structured"]
+    enabled: true
+  gemini:
+    provider: vertexai
+    model: gemini-2.0-flash-001
+    capabilities: ["tools", "fast"]
+    enabled: true
+# ドライバー認証設定
+drivers:
+  openai:
+    apiKey: ${OPENAI_API_KEY}      # 環境変数
+  vertexai:
+    projectId: my-gcp-project
+    location: us-central1
+# デフォルトオプション
+defaultOptions:
+  temperature: 0.7
+  maxTokens: 2048
+# テスト対象モジュール
+modules:
+  - name: baseline
+    path: ./modules/baseline.ts     # 設定ファイルからの相対パス
+    description: ベースラインプロンプト
+  - name: optimized
+    path: ./modules/optimized.ts
+    description: 最適化版プロンプト
+# テストケース
+testCases:
+  - name: 基本テスト
+    description: 基本的な動作確認
+    input:                          # module.compile に渡すコンテキスト
+      query: "TypeScriptの型推論について説明して"
+    models: [gpt4o]                 # オプション: 未指定時は全有効モデル
+    queryOptions:                   # オプション
+      temperature: 0.5
+  - name: ツール呼び出しテスト
+    input:
+      query: "東京の天気を調べて"
+    queryOptions:
+      tools:
+        - name: get_weather
+          description: 天気を取得
+          parameters:
+            type: object
+            properties:
+              city: { type: string }
+            required: [city]
+# 評価器
+evaluators:
+  - name: structured-output-presence  # ビルトイン
+  - name: llm-requirement-fulfillment # ビルトイン
+  - name: custom-eval                 # 外部ファイル
+    path: ./evaluators/custom-eval.ts
+# 評価設定
+evaluation:
+  enabled: true
+  model: gpt4o                        # 評価に使うモデル
+```
+### パス解決
+設定ファイル内のパス（modules, evaluators等）は設定ファイルのディレクトリからの相対パスで解決される。`~/` でホームディレクトリ、絶対パスも使用可能。
+## モジュール定義
+テスト対象のモジュールファイル:
+```typescript
+import { compile } from '@modular-prompt/core';
+import { myPromptModule } from './prompts.js';
+export default {
+  name: 'My Module',
+  description: 'テスト対象のプロンプトモジュール',
+  compile: (context: any) => compile(myPromptModule, context),
+};
+```
+`compile` 関数はテストケースの `input` をコンテキストとして受け取り、CompiledPrompt を返す。
+## 評価器
+### ビルトイン評価器
+**structured-output-presence** - コード評価器
+- `structuredOutput` の存在と有効性を検証
+- スコア: `(validCount / totalRuns) * 10`
+**llm-requirement-fulfillment** - プロンプト評価器
+- LLMが要件充足度を包括的に評価
+- 評価基準: 要件充足度、パラメータ正確性、パラメータ完全性、論理的一貫性
+- 評価用モデルの設定が必要（`evaluation.model`）
+### カスタム評価器（コード）
+```typescript
+import type { CodeEvaluator, EvaluationContext, EvaluationResult } from '@modular-prompt/experiment';
+export default {
+  name: 'json-validator',
+  description: 'JSON構造を検証',
+  async evaluate(context: EvaluationContext): Promise<EvaluationResult> {
+    const allValid = context.runs.every(run =>
+      run.queryResult.structuredOutput != null
+    );
+    return {
+      evaluator: 'json-validator',
+      moduleName: context.moduleName,
+      score: allValid ? 10 : 0,
+      reasoning: allValid ? '全実行でJSON出力あり' : 'JSON出力なし',
+    };
+  },
+} satisfies CodeEvaluator;
+```
+### カスタム評価器（プロンプト）
+LLMに評価させる場合:
+```typescript
+import type { PromptEvaluator, EvaluationContext } from '@modular-prompt/experiment';
+import type { PromptModule } from '@modular-prompt/core';
+const evaluationModule: PromptModule<EvaluationContext> = {
+  createContext: () => ({ moduleName: '', prompt: '', runs: [] }),
+  objective: ['出力の品質を0-10で評価する'],
+  instructions: [
+    '- 明確さ、正確さ、完全性を基準にする',
+    (ctx) => `対象モジュール: ${ctx.moduleName}`,
+    (ctx) => ctx.runs.map((run, i) =>
+      `実行${i + 1}: ${run.queryResult.content.slice(0, 500)}`
+    ),
+  ],
+};
+export default {
+  name: 'quality-evaluator',
+  description: '出力品質を評価',
+  module: evaluationModule,   // baseEvaluationModuleと自動マージされる
+} satisfies PromptEvaluator;
+```
+## 主要な型
+### TestCase
+```typescript
+interface TestCase {
+  name: string;
+  description?: string;
+  input: any;                          // module.compileに渡すコンテキスト
+  models?: string[];                   // 未指定時は全有効モデル
+  queryOptions?: Partial<QueryOptions>;
+}
+```
+### EvaluationContext
+```typescript
+interface EvaluationContext {
+  moduleName: string;
+  prompt: string;         // コンパイル済みプロンプト（文字列化）
+  runs: Array<{
+    queryResult: QueryResult;
+  }>;
+}
+```
+### EvaluationResult
+```typescript
+interface EvaluationResult {
+  evaluator: string;
+  moduleName: string;
+  score?: number;         // 0-10
+  reasoning?: string;
+  details?: Record<string, any>;
+  error?: string;
+}
+```
+## プログラマティック使用
+```typescript
+import {
+  loadExperimentConfig,
+  loadModules,
+  loadEvaluators,
+  ExperimentRunner,
+  DriverManager,
+} from '@modular-prompt/experiment';
+const { serverConfig, aiService, configDir } = loadExperimentConfig('config.yaml');
+const modules = await loadModules(serverConfig.modules, configDir);
+const evaluators = await loadEvaluators(serverConfig.evaluators, configDir);
+const driverManager = new DriverManager();
+const runner = new ExperimentRunner(
+  aiService,
+  driverManager,
+  modules,
+  serverConfig.testCases,
+  serverConfig.models,
+  5,           // repeat count
+  evaluators,
+  evaluatorModel
+);
+const results = await runner.run();
+await driverManager.cleanup();
+```
+## 実験フロー
+```
+設定ロード → モジュール・評価器ロード → テストケースごとに:
+  全モジュールをコンパイル → プロンプト比較 → 各モデルで実行（繰り返し対応）
+→ 評価フェーズ（オプション） → 統計レポート生成 → クリーンアップ
+```
+DriverManagerがモデル名をキーにドライバーをキャッシュし、同じモデルであればドライバーを再利用する。異なるモデルに切り替わると前のドライバーをcloseできる。これはローカルLLM（MLX等）のメモリ消費を抑えるための設計。