@learning-commons/evaluators 0.3.0 → 0.5.0

package/CHANGELOG.md

@@ -0,0 +1,52 @@
+# Changelog
+
+All notable changes to the `@learning-commons/evaluators` TypeScript SDK will be documented in this file.
+
+## [0.5.0](https://github.com/learning-commons-org/evaluators/compare/sdks-typescript-v0.4.0...sdks-typescript-v0.5.0) (2026-05-07)
+
+
+### Features
+
+* **ts-sdk:** add modelOverride option to all evaluators ([#34](https://github.com/learning-commons-org/evaluators/issues/34)) ([c57c4fc](https://github.com/learning-commons-org/evaluators/commit/c57c4fc86bc56846afe92e6d451705642e399309))
+* **ts-sdk:** Add Purpose evaluator ([#57](https://github.com/learning-commons-org/evaluators/issues/57)) ([8b6d715](https://github.com/learning-commons-org/evaluators/commit/8b6d715b49ba1911de35ccc1b6aeaef888289a1d))
+
+## [0.4.0] — 2026-03-23
+
+### Added
+
+- **Batch CSV Evaluator** — CLI tool and programmatic API for evaluating multiple texts from a CSV file in parallel. Runs the `text-complexity` group (GLA, SMK, Vocabulary, Sentence Structure, and Conventionality) across up to 50 rows and produces CSV and HTML reports.
+
+---
+
+## [0.3.0] — 2026-03-20
+
+### Added
+
+- **Conventionality Evaluator** — evaluates how explicit, literal, and straightforward a text's meaning is versus how abstract, ironic, figurative, or archaic it is, relative to grades 3–12.
+- **Conventionality added to TextComplexityEvaluator** — composite evaluator now runs vocabulary, sentence structure, SMK, and conventionality in parallel; result includes `conventionality` key.
+
+---
+
+## [0.2.0] — 2026-03-18
+
+### Added
+
+- **Subject Matter Knowledge (SMK) Evaluator** — evaluates background knowledge demands of educational texts relative to grades 3–12.
+- **SMK added to TextComplexityEvaluator** — composite evaluator now runs vocabulary, sentence structure, and SMK in parallel; result includes `subjectMatterKnowledge` key.
+- **Prompt versioning** — prompts updated to v1.3.0 (`evals/prompts/subject-matter-knowledge/`).
+
+---
+
+## [0.1.0] — Early Release
+
+Initial early release of the TypeScript SDK for Learning Commons educational evaluators.
+
+### Added
+
+- **Vocabulary Evaluator** — grades 3–12 vocabulary difficulty assessment.
+- **Sentence Structure Evaluator** — syntactic complexity analysis by grade level.
+- **Grade Level Appropriateness (GLA) Evaluator** — overall grade-level suitability scoring.
+- **Text Complexity Evaluator** — composite evaluation combining Vocabulary, Sentence Structure, and GLA.
+- **Provider abstraction** — model-agnostic via Vercel AI SDK; OpenAI, Google, and Anthropic supported.
+- **Telemetry** — opt-in, with `partnerKey` and `recordInputs` (defaults to `false`).
+- **Prompt versioning** — prompts versioned in `evals/prompts/` (v1.2.0), shared with Python notebooks.

package/README.md

@@ -1,5 +1,7 @@
 # @learning-commons/evaluators
 
+[![npm version](https://img.shields.io/npm/v/@learning-commons/evaluators)](https://www.npmjs.com/package/@learning-commons/evaluators)
+
 TypeScript SDK for Learning Commons educational text complexity evaluators.
 
 ## Installation
@@ -27,7 +29,7 @@ const evaluator = new VocabularyEvaluator({
 });
 
 const result = await evaluator.evaluate("Your text here", "5");
-console.log(result.score); // "moderately complex"
+console.log(result.score); // "Moderately complex"
 ```
 
 ---
@@ -36,7 +38,7 @@ console.log(result.score); // "moderately complex"
 
 ### 1. Vocabulary Evaluator
 
-Evaluates vocabulary complexity using the Qual Text Complexity rubric (SAP).
+Evaluates vocabulary complexity using the Qualitative Text Complexity rubric (SAP).
 
 **Supported Grades:** 3-12
 
@@ -45,12 +47,13 @@ Evaluates vocabulary complexity using the Qual Text Complexity rubric (SAP).
 **Constructor:**
 ```typescript
 const evaluator = new VocabularyEvaluator({
-  googleApiKey?: string;  // Google API key (required by this evaluator)
-  openaiApiKey?: string;  // OpenAI API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - SILENT | ERROR | WARN | INFO | DEBUG (default: WARN)
+  googleApiKey: string;                   // Google API key
+  openaiApiKey: string;                   // OpenAI API key
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -62,13 +65,13 @@ await evaluator.evaluate(text: string, grade: string)
 **Returns:**
 ```typescript
 {
-  score: 'slightly complex' | 'moderately complex' | 'very complex' | 'exceedingly complex';
+  score: 'Slightly complex' | 'Moderately complex' | 'Very complex' | 'Exceedingly complex';
   reasoning: string;
   metadata: {
     model: string;
     processingTimeMs: number;
   };
-  _internal: VocabularyComplexity; // Detailed analysis
+  _internal: VocabularyInternal; // Detailed analysis
 }
 ```
 
@@ -85,11 +88,12 @@ Evaluates sentence structure complexity based on grammatical features.
 **Constructor:**
 ```typescript
 const evaluator = new SentenceStructureEvaluator({
-  openaiApiKey?: string;  // OpenAI API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
+  openaiApiKey: string;                   // OpenAI API key
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -101,7 +105,7 @@ await evaluator.evaluate(text: string, grade: string)
 **Returns:**
 ```typescript
 {
-  score: 'Slightly Complex' | 'Moderately Complex' | 'Very Complex' | 'Exceedingly Complex';
+  score: 'Slightly complex' | 'Moderately complex' | 'Very complex' | 'Exceedingly complex';
   reasoning: string;
   metadata: {
     model: string;
@@ -128,11 +132,12 @@ Evaluates the background knowledge demands of educational texts relative to grad
 **Constructor:**
 ```typescript
 const evaluator = new SmkEvaluator({
-  googleApiKey?: string;  // Google API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
+  googleApiKey: string;                   // Google API key
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -191,11 +196,12 @@ Evaluates how explicit, literal, and straightforward a text's meaning is versus
 **Constructor:**
 ```typescript
 const evaluator = new ConventionalityEvaluator({
-  googleApiKey?: string;  // Google API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
+  googleApiKey: string;                   // Google API key
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -248,17 +254,18 @@ Composite evaluator that analyzes vocabulary, sentence structure, subject matter
 
 **Supported Grades:** 3-12
 
-**Uses:** Google Gemini 2.5 Pro + Google Gemini 3 Flash Preview + OpenAI GPT-4o (composite)
+**Uses:** Google Gemini 2.5 Pro + Google Gemini 3 Flash Preview + OpenAI GPT-4o + OpenAI GPT-4.1 (composite)
 
 **Constructor:**
 ```typescript
 const evaluator = new TextComplexityEvaluator({
-  googleApiKey?: string;  // Google API key (required by this evaluator)
-  openaiApiKey?: string;  // OpenAI API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
+  googleApiKey: string;                   // Google API key
+  openaiApiKey: string;                   // OpenAI API key
+  modelOverride?: ModelOverride;          // Override the default provider and model for all sub-evaluators
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -317,11 +324,12 @@ Determines appropriate grade level for text.
 **Constructor:**
 ```typescript
 const evaluator = new GradeLevelAppropriatenessEvaluator({
-  googleApiKey?: string;  // Google API key (required by this evaluator)
-  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
-  logger?: Logger;        // Optional - Custom logger
-  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
+  googleApiKey: string;                   // Google API key
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
 });
 ```
 
@@ -350,6 +358,98 @@ await evaluator.evaluate(text: string)
 
 ---
 
+### 7. Purpose Evaluator
+
+Evaluates the Purpose dimension of qualitative text complexity — how explicitly the text's purpose is stated versus implied, and how that affects comprehension demands for the target grade level.
+
+**Supported Grades:** 3-12
+
+**Uses:** Google Gemini 3 Flash Preview
+
+**Constructor:**
+```typescript
+const evaluator = new PurposeEvaluator({
+  googleApiKey: string;                   // Google API key (required by this evaluator)
+  modelOverride?: ModelOverride;          // Override the default provider and model
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
+});
+```
+
+**API:**
+```typescript
+await evaluator.evaluate(text: string, grade: string)
+```
+
+**Returns:**
+```typescript
+{
+  score: 'Slightly complex' | 'Moderately complex' | 'Very complex' | 'Exceedingly complex' | 'More context needed';
+  reasoning: string;
+  metadata: {
+    model: string;
+    processingTimeMs: number;
+  };
+  _internal: {
+    complexity_score: 'slightly_complex' | 'moderately_complex' | 'very_complex' | 'exceedingly_complex' | 'more_context_needed';
+    reasoning: string;
+    details: {
+      detailed_summary: Array<{
+        factor: string;
+        description: string;
+        effect_on_complexity_dimension: string;
+      }>;
+      adjustment_and_scaffolding: Array<{
+        scaffolding_need: string;
+        suggestion: string;
+      }>;
+      recommended_use_cases: Array<{
+        opportunity: string;
+        suggestion: string;
+      }>;
+    };
+  };
+}
+```
+
+> **Note:** The `'More context needed'` score is used for cases where the text alone is insufficient to determine complexity.
+
+**Example:**
+```typescript
+import { PurposeEvaluator } from '@learning-commons/evaluators';
+
+const evaluator = new PurposeEvaluator({
+  googleApiKey: process.env.GOOGLE_API_KEY,
+});
+
+const result = await evaluator.evaluate(
+  "The author argues that renewable energy is the only viable solution to climate change.",
+  "9"
+);
+console.log(result.score);          // "Moderately complex"
+console.log(result.reasoning);
+console.log(result._internal.details.adjustment_and_scaffolding);
+```
+
+---
+
+## Batch CSV Evaluation
+
+For evaluating many texts at once, the SDK ships a CLI tool that reads a CSV file, runs all evaluators in a group, and produces CSV and HTML reports.
+
+```bash
+# Run from the directory containing your CSV
+npx evaluators-batch
+```
+
+The CLI will prompt for your CSV path, API keys, and output directory, then process all rows in parallel with real-time progress.
+
+See [`src/batch/README.md`](./src/batch/README.md) for full documentation.
+
+---
+
 ## Error Handling
 
 The SDK provides specific error types to help you handle different scenarios:
@@ -384,6 +484,9 @@ try {
   } else if (error instanceof NetworkError) {
     // Network connectivity issues
     console.error('Network error:', error.message);
+  } else if (error instanceof TimeoutError) {
+    // Request timed out
+    console.error('Timeout:', error.message);
   } else if (error instanceof APIError) {
     // Other API errors
     console.error('API error:', error.message, 'Status:', error.statusCode);
@@ -428,6 +531,44 @@ const evaluator = new VocabularyEvaluator({
 
 ---
 
+## Model Override
+
+By default each evaluator uses a recommended provider and model tuned for that task. You can override this with any supported provider — OpenAI, Google, or Anthropic — using the `modelOverride` option.
+
+When `modelOverride` is set:
+- All LLM calls within the evaluator use the specified provider and model
+- Only the API key for the override provider is required (e.g. `anthropicApiKey` when using `Provider.Anthropic`); default provider keys are not validated
+- A warning is logged to indicate results may differ from the defaults
+- Telemetry records `model_override: true` so override usage is tracked separately
+
+**Validation:** The SDK validates `modelOverride` at construction time and throws `ConfigurationError` if:
+- `provider` is not one of the supported `Provider` values (`openai`, `google`, `anthropic`)
+- `model` is empty or blank — no default is assumed; you must always specify the model ID explicitly
+- The API key for the chosen provider is missing
+
+If the model ID is valid at construction but doesn't exist on the provider's API, `ConfigurationError` is thrown when `evaluate()` is called.
+
+```typescript
+import { VocabularyEvaluator, Provider } from '@learning-commons/evaluators';
+
+const evaluator = new VocabularyEvaluator({
+  anthropicApiKey: process.env.ANTHROPIC_API_KEY,
+  modelOverride: {
+    provider: Provider.Anthropic,
+    model: 'claude-sonnet-4-6',
+  },
+});
+
+const result = await evaluator.evaluate("Your text here", "5");
+console.log(result.metadata.model); // "anthropic:claude-sonnet-4-6"
+```
+
+See the [Installation](#installation) section for provider adapter setup if you haven't already.
+
+> **Note:** Evaluators are validated and quality-tested against their default models. Results with other models may vary. Check `result.metadata.model` to confirm which model was used.
+
+---
+
 ## Telemetry & Privacy
 
 See [docs/telemetry.md](./docs/telemetry.md) for telemetry configuration and privacy information.
@@ -440,13 +581,15 @@ All evaluators use the same `BaseEvaluatorConfig` interface:
 
 ```typescript
 interface BaseEvaluatorConfig {
-  googleApiKey?: string;  // Google API key (required by some evaluators)
-  openaiApiKey?: string;  // OpenAI API key (required by some evaluators)
-  maxRetries?: number;    // Max API retry attempts (default: 2)
-  telemetry?: boolean | TelemetryOptions; // Telemetry config (default: true)
-  logger?: Logger;        // Custom logger (optional)
-  logLevel?: LogLevel;    // Console log level (default: WARN)
-  partnerKey?: string;    // Learning Commons partner key for authenticated telemetry (optional)
+  googleApiKey?: string;                  // Google API key (required by some evaluators)
+  openaiApiKey?: string;                  // OpenAI API key (required by some evaluators)
+  anthropicApiKey?: string;               // Anthropic API key (required if an evaluator defaults to Claude or when `modelOverride` uses `Provider.Anthropic`)
+  modelOverride?: ModelOverride;          // Override the provider and model (see Model Override section)
+  maxRetries?: number;                    // Max retry attempts (default: 2)
+  telemetry?: boolean | TelemetryOptions; // Telemetry settings (default: enabled)
+  logger?: Logger;                        // Custom logger
+  logLevel?: LogLevel;                    // Log verbosity (default: WARN)
+  partnerKey?: string;                    // Learning Commons partner key for authenticated telemetry
 }
 ```
 
@@ -457,6 +600,9 @@ interface BaseEvaluatorConfig {
 - **Conventionality**: Requires `googleApiKey` only
 - **Text Complexity**: Requires both `googleApiKey` and `openaiApiKey`
 - **Grade Level Appropriateness**: Requires `googleApiKey` only
+- **Purpose**: Requires `googleApiKey` only
+
+When `modelOverride` is set, the default key requirements are bypassed — only the key for the override provider is required (e.g. `anthropicApiKey` when using `Provider.Anthropic`).
 
 ---