@learning-commons/evaluators 0.4.0 → 0.5.0

package/CHANGELOG.md

@@ -2,6 +2,14 @@
 
 All notable changes to the `@learning-commons/evaluators` TypeScript SDK will be documented in this file.
 
+## [0.5.0](https://github.com/learning-commons-org/evaluators/compare/sdks-typescript-v0.4.0...sdks-typescript-v0.5.0) (2026-05-07)
+
+
+### Features
+
+* **ts-sdk:** add modelOverride option to all evaluators ([#34](https://github.com/learning-commons-org/evaluators/issues/34)) ([c57c4fc](https://github.com/learning-commons-org/evaluators/commit/c57c4fc86bc56846afe92e6d451705642e399309))
+* **ts-sdk:** Add Purpose evaluator ([#57](https://github.com/learning-commons-org/evaluators/issues/57)) ([8b6d715](https://github.com/learning-commons-org/evaluators/commit/8b6d715b49ba1911de35ccc1b6aeaef888289a1d))
+
 ## [0.4.0] — 2026-03-23
 
 ### Added

package/README.md

@@ -1,5 +1,7 @@
 # @learning-commons/evaluators
 
+[![npm version](https://img.shields.io/npm/v/@learning-commons/evaluators)](https://www.npmjs.com/package/@learning-commons/evaluators)
+
 TypeScript SDK for Learning Commons educational text complexity evaluators.
 
 ## Installation
@@ -27,7 +29,7 @@ const evaluator = new VocabularyEvaluator({
 });
 
 const result = await evaluator.evaluate("Your text here", "5");
-console.log(result.score); // "moderately complex"
+console.log(result.score); // "Moderately complex"
 ```
 
 ---
@@ -36,7 +38,7 @@ console.log(result.score); // "moderately complex"
 
 ### 1. Vocabulary Evaluator
 
-Evaluates vocabulary complexity using the Qual Text Complexity rubric (SAP).
+Evaluates vocabulary complexity using the Qualitative Text Complexity rubric (SAP).
 
 **Supported Grades:** 3-12
 
@@ -45,12 +47,13 @@ Evaluates vocabulary complexity using the Qual Text Complexity rubric (SAP).
 **Constructor:**
 ```typescript
 const evaluator = new VocabularyEvaluator({
-  googleApiKey?: string;  // Google API key (required by this evaluator)
-  openaiApiKey?: string;  // OpenAI API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - SILENT | ERROR | WARN | INFO | DEBUG (default: WARN)
+  googleApiKey: string;                   // Google API key
+  openaiApiKey: string;                   // OpenAI API key
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -62,13 +65,13 @@ await evaluator.evaluate(text: string, grade: string)
 **Returns:**
 ```typescript
 {
-  score: 'slightly complex' | 'moderately complex' | 'very complex' | 'exceedingly complex';
+  score: 'Slightly complex' | 'Moderately complex' | 'Very complex' | 'Exceedingly complex';
   reasoning: string;
   metadata: {
     model: string;
     processingTimeMs: number;
   };
-  _internal: VocabularyComplexity; // Detailed analysis
+  _internal: VocabularyInternal; // Detailed analysis
 }
 ```
 
@@ -85,11 +88,12 @@ Evaluates sentence structure complexity based on grammatical features.
 **Constructor:**
 ```typescript
 const evaluator = new SentenceStructureEvaluator({
-  openaiApiKey?: string;  // OpenAI API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
+  openaiApiKey: string;                   // OpenAI API key
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -101,7 +105,7 @@ await evaluator.evaluate(text: string, grade: string)
 **Returns:**
 ```typescript
 {
-  score: 'Slightly Complex' | 'Moderately Complex' | 'Very Complex' | 'Exceedingly Complex';
+  score: 'Slightly complex' | 'Moderately complex' | 'Very complex' | 'Exceedingly complex';
   reasoning: string;
   metadata: {
     model: string;
@@ -128,11 +132,12 @@ Evaluates the background knowledge demands of educational texts relative to grad
 **Constructor:**
 ```typescript
 const evaluator = new SmkEvaluator({
-  googleApiKey?: string;  // Google API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
+  googleApiKey: string;                   // Google API key
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -191,11 +196,12 @@ Evaluates how explicit, literal, and straightforward a text's meaning is versus
 **Constructor:**
 ```typescript
 const evaluator = new ConventionalityEvaluator({
-  googleApiKey?: string;  // Google API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
+  googleApiKey: string;                   // Google API key
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -248,17 +254,18 @@ Composite evaluator that analyzes vocabulary, sentence structure, subject matter
 
 **Supported Grades:** 3-12
 
-**Uses:** Google Gemini 2.5 Pro + Google Gemini 3 Flash Preview + OpenAI GPT-4o (composite)
+**Uses:** Google Gemini 2.5 Pro + Google Gemini 3 Flash Preview + OpenAI GPT-4o + OpenAI GPT-4.1 (composite)
 
 **Constructor:**
 ```typescript
 const evaluator = new TextComplexityEvaluator({
-  googleApiKey?: string;  // Google API key (required by this evaluator)
-  openaiApiKey?: string;  // OpenAI API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
+  googleApiKey: string;                   // Google API key
+  openaiApiKey: string;                   // OpenAI API key
+  modelOverride?: ModelOverride;          // Override the default provider and model for all sub-evaluators
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -317,11 +324,12 @@ Determines appropriate grade level for text.
 **Constructor:**
 ```typescript
 const evaluator = new GradeLevelAppropriatenessEvaluator({
-  googleApiKey?: string;  // Google API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
+  googleApiKey: string;                   // Google API key
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -350,6 +358,83 @@ await evaluator.evaluate(text: string)
 
 ---
 
+### 7. Purpose Evaluator
+
+Evaluates the Purpose dimension of qualitative text complexity — how explicitly the text's purpose is stated versus implied, and how that affects comprehension demands for the target grade level.
+
+**Supported Grades:** 3-12
+
+**Uses:** Google Gemini 3 Flash Preview
+
+**Constructor:**
+```typescript
+const evaluator = new PurposeEvaluator({
+  googleApiKey: string;                   // Google API key (required by this evaluator)
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
+});
+```
+
+**API:**
+```typescript
+await evaluator.evaluate(text: string, grade: string)
+```
+
+**Returns:**
+```typescript
+{
+  score: 'Slightly complex' | 'Moderately complex' | 'Very complex' | 'Exceedingly complex' | 'More context needed';
+  reasoning: string;
+  metadata: {
+    model: string;
+    processingTimeMs: number;
+  };
+  _internal: {
+    complexity_score: 'slightly_complex' | 'moderately_complex' | 'very_complex' | 'exceedingly_complex' | 'more_context_needed';
+    reasoning: string;
+    details: {
+      detailed_summary: Array<{
+        factor: string;
+        description: string;
+        effect_on_complexity_dimension: string;
+      }>;
+      adjustment_and_scaffolding: Array<{
+        scaffolding_need: string;
+        suggestion: string;
+      }>;
+      recommended_use_cases: Array<{
+        opportunity: string;
+        suggestion: string;
+      }>;
+    };
+  };
+}
+```
+
+> **Note:** The `'More context needed'` score is used for cases where the text alone is insufficient to determine complexity.
+
+**Example:**
+```typescript
+import { PurposeEvaluator } from '@learning-commons/evaluators';
+
+const evaluator = new PurposeEvaluator({
+  googleApiKey: process.env.GOOGLE_API_KEY,
+});
+
+const result = await evaluator.evaluate(
+  "The author argues that renewable energy is the only viable solution to climate change.",
+  "9"
+);
+console.log(result.score);          // "Moderately complex"
+console.log(result.reasoning);
+console.log(result._internal.details.adjustment_and_scaffolding);
+```
+
+---
+
 ## Batch CSV Evaluation
 
 For evaluating many texts at once, the SDK ships a CLI tool that reads a CSV file, runs all evaluators in a group, and produces CSV and HTML reports.
@@ -399,6 +484,9 @@ try {
   } else if (error instanceof NetworkError) {
     // Network connectivity issues
     console.error('Network error:', error.message);
+  } else if (error instanceof TimeoutError) {
+    // Request timed out
+    console.error('Timeout:', error.message);
   } else if (error instanceof APIError) {
     // Other API errors
     console.error('API error:', error.message, 'Status:', error.statusCode);
@@ -443,6 +531,44 @@ const evaluator = new VocabularyEvaluator({
 
 ---
 
+## Model Override
+
+By default each evaluator uses a recommended provider and model tuned for that task. You can override this with any supported provider — OpenAI, Google, or Anthropic — using the `modelOverride` option.
+
+When `modelOverride` is set:
+- All LLM calls within the evaluator use the specified provider and model
+- Only the API key for the override provider is required (e.g. `anthropicApiKey` when using `Provider.Anthropic`); default provider keys are not validated
+- A warning is logged to indicate results may differ from the defaults
+- Telemetry records `model_override: true` so override usage is tracked separately
+
+**Validation:** The SDK validates `modelOverride` at construction time and throws `ConfigurationError` if:
+- `provider` is not one of the supported `Provider` values (`openai`, `google`, `anthropic`)
+- `model` is empty or blank — no default is assumed; you must always specify the model ID explicitly
+- The API key for the chosen provider is missing
+
+If the model ID is valid at construction but doesn't exist on the provider's API, `ConfigurationError` is thrown when `evaluate()` is called.
+
+```typescript
+import { VocabularyEvaluator, Provider } from '@learning-commons/evaluators';
+
+const evaluator = new VocabularyEvaluator({
+  anthropicApiKey: process.env.ANTHROPIC_API_KEY,
+  modelOverride: {
+    provider: Provider.Anthropic,
+    model: 'claude-sonnet-4-6',
+  },
+});
+
+const result = await evaluator.evaluate("Your text here", "5");
+console.log(result.metadata.model); // "anthropic:claude-sonnet-4-6"
+```
+
+See the [Installation](#installation) section for provider adapter setup if you haven't already.
+
+> **Note:** Evaluators are validated and quality-tested against their default models. Results with other models may vary. Check `result.metadata.model` to confirm which model was used.
+
+---
+
 ## Telemetry & Privacy
 
 See [docs/telemetry.md](./docs/telemetry.md) for telemetry configuration and privacy information.
@@ -455,13 +581,15 @@ All evaluators use the same `BaseEvaluatorConfig` interface:
 
 ```typescript
 interface BaseEvaluatorConfig {
-  googleApiKey?: string;  // Google API key (required by some evaluators)
-  openaiApiKey?: string;  // OpenAI API key (required by some evaluators)
-  maxRetries?: number;    // Max API retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Telemetry config (default: true)
-  logger?: Logger;        // Custom logger (optional)
-  logLevel?: LogLevel;    // Console log level (default: WARN)
-  partnerKey?: string;    // Learning Commons partner key for authenticated telemetry (optional)
+  googleApiKey?: string;                  // Google API key (required by some evaluators)
+  openaiApiKey?: string;                  // OpenAI API key (required by some evaluators)
+  anthropicApiKey?: string;               // Anthropic API key (required if an evaluator defaults to Claude or when `modelOverride` uses `Provider.Anthropic`)
+  modelOverride?: ModelOverride;          // Override the provider and model (see Model Override section)
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
+  partnerKey?: string;                    // Learning Commons partner key for authenticated telemetry
 }
 ```
 
@@ -472,6 +600,9 @@ interface BaseEvaluatorConfig {
 - **Conventionality**: Requires `googleApiKey` only
 - **Text Complexity**: Requires both `googleApiKey` and `openaiApiKey`
 - **Grade Level Appropriateness**: Requires `googleApiKey` only
+- **Purpose**: Requires `googleApiKey` only
+
+When `modelOverride` is set, the default key requirements are bypassed — only the key for the override provider is required (e.g. `anthropicApiKey` when using `Provider.Anthropic`).
 
 ---
 

package/dist/{base-Ced9oKKa.d.cts → base-DKcAYXfb.d.cts}

@@ -1,3 +1,5 @@
+import { z } from 'zod';
+
 /**
  * Logging interface for the Evaluators SDK
  *
@@ -77,6 +79,82 @@ interface Logger {
     error(message: string, context?: LogContext): void;
 }
 
+/**
+ * Message format for LLM conversations
+ */
+interface Message {
+    role: 'system' | 'user' | 'assistant';
+    content: string;
+}
+/**
+ * Request configuration for structured LLM generation
+ */
+interface LLMRequest<T> {
+    messages: Message[];
+    schema: z.ZodSchema<T>;
+    temperature?: number;
+    maxTokens?: number;
+}
+/**
+ * Response from LLM with usage metadata
+ */
+interface LLMResponse<T> {
+    data: T;
+    model: string;
+    usage: {
+        inputTokens: number;
+        outputTokens: number;
+    };
+    latencyMs: number;
+}
+/**
+ * Response from plain text generation
+ */
+interface TextGenerationResponse {
+    text: string;
+    usage: {
+        inputTokens: number;
+        outputTokens: number;
+    };
+    latencyMs: number;
+}
+/**
+ * Base interface for LLM provider implementations
+ */
+interface LLMProvider {
+    /** Canonical label for the provider and model in use (e.g. "openai:gpt-4o") */
+    readonly label: string;
+    /**
+     * Generate structured output from LLM using Zod schema
+     */
+    generateStructured<T>(request: LLMRequest<T>): Promise<LLMResponse<T>>;
+    /**
+     * Generate plain text from LLM
+     */
+    generateText(messages: Message[], temperature?: number): Promise<TextGenerationResponse>;
+}
+/**
+ * Named constants for LLM provider types — use instead of raw string literals.
+ */
+declare const Providers: {
+    readonly google: "google";
+    readonly openai: "openai";
+    readonly anthropic: "anthropic";
+    readonly custom: "custom";
+};
+/**
+ * Configuration for LLM provider
+ */
+interface ProviderConfig {
+    type: 'openai' | 'anthropic' | 'google' | 'custom';
+    apiKey?: string;
+    model?: string;
+    temperature?: number;
+    baseURL?: string;
+    customProvider?: LLMProvider;
+    maxRetries?: number;
+}
+
 /**
  * Evaluation status
  */
@@ -130,6 +208,7 @@ interface TelemetryEvent {
     provider: string;
     token_usage?: TokenUsage;
     metadata?: TelemetryMetadata;
+    model_override?: boolean;
     input_text?: string;
 }
 /**
@@ -166,6 +245,14 @@ declare class TelemetryClient {
     send(event: TelemetryEvent): Promise<void>;
 }
 
+/**
+ * Supported LLM providers
+ */
+declare enum Provider {
+    OpenAI = "openai",
+    Google = "google",
+    Anthropic = "anthropic"
+}
 /**
  * Granular telemetry configuration options
  */
@@ -175,6 +262,24 @@ interface TelemetryOptions {
     /** Record input text in telemetry (default: false) */
     recordInputs?: boolean;
 }
+/**
+ * Override the provider and model used by an evaluator.
+ *
+ * When set, all LLM calls use this provider and model instead of the defaults.
+ * The evaluator's normal key requirements are bypassed — provide the key for
+ * the chosen provider via the matching top-level config field
+ * (e.g. `anthropicApiKey` for `Provider.Anthropic`).
+ *
+ * Both `provider` and `model` are required. An empty or missing `model` throws
+ * `ConfigurationError` at construction time. An unrecognised model ID throws
+ * `ConfigurationError` at evaluation time when the provider rejects it.
+ *
+ * Results may vary; evaluators are validated against their recommended models.
+ */
+interface ModelOverride {
+    provider: Provider;
+    model: string;
+}
 /**
  * Base configuration for all evaluators
  */
@@ -183,8 +288,16 @@ interface BaseEvaluatorConfig {
     googleApiKey?: string;
     /** OpenAI API key (for evaluators using GPT) */
     openaiApiKey?: string;
+    /** Anthropic API key (for evaluators using Claude) */
+    anthropicApiKey?: string;
     /** Learning Commons partner key for authenticated telemetry (optional) */
     partnerKey?: string;
+    /**
+     * Override the provider and model used by this evaluator.
+     * When set, all LLM calls use this provider and model instead of the defaults.
+     * See {@link ModelOverride} for details.
+     */
+    modelOverride?: ModelOverride;
     /**
      * Maximum number of retries for failed API calls (default: 2)
      * Set to 0 to disable retries.
@@ -232,10 +345,8 @@ interface EvaluatorMetadata {
     readonly description: string;
     /** Supported grade levels (e.g., ['3', '4', '5', ...]) */
     readonly supportedGrades: readonly string[];
-    /** Whether this evaluator requires a Google API key */
-    readonly requiresGoogleKey: boolean;
-    /** Whether this evaluator requires an OpenAI API key */
-    readonly requiresOpenAIKey: boolean;
+    /** Providers required by this evaluator's default configuration */
+    readonly defaultProviders: readonly Provider[];
 }
 /**
  * Abstract base class for all evaluators
@@ -254,6 +365,10 @@ declare abstract class BaseEvaluator {
     protected logger: Logger;
     protected config: Required<Pick<BaseEvaluatorConfig, 'maxRetries'>> & {
         telemetry: Required<TelemetryOptions>;
+        modelOverride?: ModelOverride;
+        googleApiKey?: string;
+        openaiApiKey?: string;
+        anthropicApiKey?: string;
     };
     /**
      * Static metadata for the evaluator
@@ -268,13 +383,17 @@ declare abstract class BaseEvaluator {
      *     name: 'My Evaluator',
      *     description: 'Does something useful',
      *     supportedGrades: ['3', '4', '5'],
-     *     requiresGoogleKey: true,
-     *     requiresOpenAIKey: false,
+     *     defaultProviders: [Provider.Google],
      *   };
      * }
      * ```
      */
     static readonly metadata: EvaluatorMetadata;
+    /**
+     * @throws {ConfigurationError} If the subclass has not defined static metadata
+     * @throws {ConfigurationError} If modelOverride has an invalid provider or empty model
+     * @throws {ConfigurationError} If a required API key is missing
+     */
     constructor(config: BaseEvaluatorConfig);
     /**
      * Get metadata for this evaluator instance
@@ -282,8 +401,16 @@ declare abstract class BaseEvaluator {
      */
     protected get metadata(): EvaluatorMetadata;
     /**
-     * Validate that required API keys are provided based on metadata
-     * @throws {ConfigurationError} If required API keys are missing
+     * Validate modelOverride shape: provider must be a known Provider value and
+     * model must be a non-empty string.
+     * @throws {ConfigurationError} If the override is malformed
+     */
+    private validateModelOverride;
+    /**
+     * Validate that the required API key is present.
+     * When modelOverride is set, checks the override provider's key.
+     * Otherwise checks the keys required by the evaluator's default providers.
+     * @throws {ConfigurationError} If a required key is missing
      */
     private validateApiKeys;
     /**
@@ -311,6 +438,12 @@ declare abstract class BaseEvaluator {
      * @throws {ValidationError} If grade is invalid
      */
     protected validateGrade(grade: string, validGrades: Set<string>): void;
+    /**
+     * Create an LLM provider, honouring modelOverride if set.
+     * When override is active, the key for the override provider is resolved
+     * from the matching top-level config field (e.g. anthropicApiKey for Anthropic).
+     */
+    protected createConfiguredProvider(defaultType: Provider, defaultModel: string, defaultApiKey: string | undefined): LLMProvider;
     /**
      * Send telemetry event to analytics service
      * Common helper for all evaluators
@@ -328,4 +461,4 @@ declare abstract class BaseEvaluator {
     }): Promise<void>;
 }
 
-export { BaseEvaluator as B, type EvaluatorMetadata as E, type Logger as L, type TelemetryOptions as T, type BaseEvaluatorConfig as a, type LogContext as b, LogLevel as c };
+export { BaseEvaluator as B, type EvaluatorMetadata as E, type LLMProvider as L, type Message as M, Provider as P, type TelemetryOptions as T, type BaseEvaluatorConfig as a, type LLMRequest as b, type LLMResponse as c, type LogContext as d, LogLevel as e, type Logger as f, type ModelOverride as g, type ProviderConfig as h, Providers as i, type TextGenerationResponse as j };