@dotsetlabs/bellwether 1.0.3 → 2.0.1

package/CHANGELOG.md

@@ -7,6 +7,124 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.0.1] - 2026-02-07
+
+### Added
+
+- **MCP protocol version gating**: New `src/protocol/` module with version-to-feature-flag mapping
+  - Supports MCP protocol versions: `2024-11-05`, `2025-03-26`, `2025-06-18`, `2025-11-25`
+  - `MCPFeatureFlags` interface with 9 feature flags (`toolAnnotations`, `entityTitles`, `completions`, `resourceAnnotations`, `structuredOutput`, `serverInstructions`, `httpVersionHeader`, `tasks`, `icons`)
+  - `getSharedFeatureFlags(v1, v2)` computes AND-intersection for cross-version baseline comparison
+  - All version-specific fields in baselines are now gated by protocol version during conversion and comparison
+- **Version-gated drift detection**: Comparator now detects changes in version-specific fields
+  - Tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint)
+  - Entity titles (tool, prompt, resource, and resource template titles)
+  - Output schema and structured output changes
+  - Execution/task support changes
+  - Server instructions changes
+- **MCPClient protocol version tracking**: Client stores negotiated protocol version after `initialize()`, exposes via getters
+- **Mock server protocol version support**: Mock MCP server now supports `MOCK_PROTOCOL_VERSION` env var for testing
+
+### Fixed
+
+- **20 production-blocking bugs across all layers** (`4717ca1`):
+  - Transport: HTTP transport URL construction, SSE error event handling, MCP client error propagation
+  - Discovery: ResourceTemplate type handling, discovery error handling
+  - Baseline: Converter version-gated field handling, saver hash calculation, comparator severity logic
+  - CLI: Check command exit code handling, explore command cleanup, baseline command error paths
+  - Config: Environment variable expansion edge cases
+  - Docs: Contract and agents generator error handling
+- **Protocol version gating gaps causing false negatives and data loss** (`dce73ed`):
+  - Fixed tool title comparison using wrong feature flag (`toolAnnotations` instead of `entityTitles`)
+  - Fixed tool title comparison condition (AND → OR) to detect added/removed titles
+  - Added missing `execution` and `baselineP99Ms` fields to `ToolFingerprint` type
+  - Added missing fields (`title`, `outputSchema`, `outputSchemaHash`, `annotations`, `execution`, `baselineP99Ms`) to `toToolCapability()` accessor — prevents data loss during incremental check merges
+  - Added `execution` and `baselineP99Ms` mapping to `getToolFingerprints()` accessor
+  - Added prompt title comparison gated by `entityTitles` flag
+  - Added resource title comparison gated by `entityTitles` flag
+  - Added resource template title comparison gated by `entityTitles` flag
+  - Added execution/task support comparison gated by `tasks` flag
+  - Added server instructions comparison gated by `serverInstructions` flag
+  - Gated resource template `title` in converter by `entityTitles` flag
+- **Clean JSON output from baseline commands** (`7aab450`):
+  - `baseline compare --format json` no longer appends summary text after JSON object
+  - `baseline diff --format json` no longer prepends header or appends summary text around JSON object
+  - JSON output is now machine-parseable without text contamination
+
+## [2.0.0] - 2026-02-04
+
+### Breaking Changes
+
+- **Removed cloud-related baseline modules**: The following exports have been removed from the public API:
+  - `ai-compatibility-scorer.ts` - AI compatibility scoring
+  - `change-impact-analyzer.ts` - Change impact analysis (`analyzeToolChangeImpact`, `analyzeDiffImpact`, `isBreakingChange`, etc.)
+  - `deprecation-tracker.ts` - Deprecation tracking (`checkDeprecations`, `markAsDeprecated`, `getDeprecatedTools`, etc.)
+  - `health-scorer.ts` - Health scoring (`calculateHealthScore`, `formatHealthScore`, `HEALTH_SCORING`, etc.)
+  - `migration-generator.ts` - Migration guide generation (`generateMigrationGuide`, `formatMigrationGuideMarkdown`, etc.)
+  - `pr-comment-generator.ts` - PR comment generation (`generatePRComment`, `generateCompactPRComment`, etc.)
+  - `risk-scorer.ts` - Risk scoring (`calculateRiskScore`, `generateRiskScoreMarkdown`, etc.)
+  - `scenario-generator.ts` - Auto scenario generation (`generateToolScenarios`, `generateBaselineScenarios`, etc.)
+  - `schema-evolution.ts` - Schema evolution timeline (`buildServerTimeline`, `getSchemaChanges`, etc.)
+  - `test-pruner.ts` - Test pruning (`calculatePruningDecisions`, `prioritizeTools`, etc.)
+  - `cloud-types.ts` - Cloud type definitions
+  - `constants/cloud.ts` - Cloud constants
+- **Renamed baseline function**: `createCloudBaseline()` renamed to `createBaselineFromInterview()`
+- **Removed `PERFORMANCE` constant export** from `performance-tracker.ts`
+
+### Added
+
+- **Deterministic prompt testing**: New `prompt-test-generator.ts` for generating deterministic tests for MCP prompts without requiring LLM calls
+- **Deterministic resource testing**: New `resource-test-generator.ts` for generating deterministic tests for MCP resources
+- **Interview insights module**: New `insights.ts` module with `buildInterviewInsights()` for deriving semantic inferences, schema evolution, and error analysis
+- **Baseline format types**: New `baseline-format.ts` with enhanced types:
+  - `PersonaInterview` and `PersonaFinding` for structured interview results
+  - `ResourceCapability` and `PromptCapability` for resource/prompt discovery
+  - Enhanced `ToolCapability` with observed schema tracking and security fingerprints
+  - `ResponseSchemaEvolution` and `DocumentationScoreSummary` types
+- **Registry constants**: New `constants/registry.ts` for MCP Registry integration
+- **Man pages**: Added `man/bellwether.1` and `man/bellwether.1.md` for Unix manual pages
+- **Explore report schema**: New `schemas/bellwether-explore.schema.json` for JSON report validation
+- **JSON schema embedding**: JSON reports now include `$schema` pointer for IDE validation
+- **Expanded behavior aspects**: `BehaviorAspect` type now includes `prompt`, `resource`, `server`, `capability`
+
+### Changed
+
+- **Simplified baseline system**: Removed cloud-specific baseline logic in favor of a single, self-contained format
+- **Enhanced schema comparison**: Expanded `schema-compare.ts` with improved property-level diff detection
+- **Improved comparator**: Enhanced `comparator.ts` with better change detection and categorization
+- **SSE transport improvements**: Refactored `sse-transport.ts` for better reliability and error handling
+- **Response cache enhancements**: Improved `response-cache.ts` with better TTL management
+- **Interview system refinements**: Updated `interviewer.ts` and `schema-test-generator.ts` for deterministic test merging
+- **Stateful test runner**: Enhanced `stateful-test-runner.ts` with improved state management
+
+### Documentation
+
+- Updated all CLI documentation for consistency
+- Added JSON schema validation pointers to output format docs
+- Updated GitHub Action examples to v2.0.0
+- Improved baseline and CI/CD documentation
+- Enhanced configuration guide with new options
+
+### Internal
+
+- Removed ~13,600 lines of cloud-related code
+- Added ~2,600 lines of deterministic testing and baseline improvements
+- Consolidated test files, removing 12 test files for deleted modules
+- Added new tests for prompt/resource generators and enhanced schema comparison
+
+### Migration Guide
+
+If you were importing from the `@dotsetlabs/bellwether` library API:
+
+1. **Baseline functions**: Replace `createCloudBaseline()` with `createBaselineFromInterview()`
+
+2. **Removed exports**: The following modules are no longer available. If you depended on them, you'll need to implement alternatives:
+   - Health scoring, deprecation tracking, migration generation
+   - PR comment generation, risk scoring, scenario generation
+   - AI compatibility scoring, test pruning, schema evolution timeline
+
+3. **CLI users**: No changes required. The CLI interface remains fully compatible.
+
 ## [1.0.3] - 2026-02-02
 
 ### Added

package/README.md

@@ -75,6 +75,15 @@ jobs:
 | Parameter renamed | `path` to `file_path` | Breaking |
 | Description changed | Tool help text updated | Warning |
 | Performance regression | Latency increased >10% | Warning |
+| Tool annotations changed | `readOnlyHint` flipped to `false` | Warning |
+| Output schema changed | Return type structure modified | Warning |
+| Entity title changed | Tool/prompt/resource title updated | Info |
+| Task support changed | Execution mode switched to `async` | Warning |
+| Server instructions changed | Server-level instructions updated | Info |
+| Prompt added/removed | Prompt template appears or disappears | Breaking |
+| Resource changed | Resource URI or MIME type modified | Warning |
+
+Comparisons are **protocol-version-aware** — version-specific fields (annotations, titles, output schemas, etc.) are only compared when both baselines support the relevant MCP protocol version.
 
 ## Commands
 
@@ -105,11 +114,16 @@ Requires LLM (Ollama for free local, or OpenAI/Anthropic). Generates `AGENTS.md`
 | `explore` | LLM behavioral testing |
 | `baseline save` | Save test results as baseline |
 | `baseline compare` | Compare against baseline |
+| `baseline show` | Display baseline contents |
 | `baseline accept` | Accept drift as intentional |
 | `baseline diff` | Compare two baselines |
 | `discover` | Show server capabilities |
 | `watch` | Continuous checking on file changes |
 | `registry` | Search MCP Registry |
+| `golden` | Golden output regression testing |
+| `contract` | Contract validation (generate/validate/show) |
+| `auth` | Manage LLM provider API keys |
+| `validate-config` | Validate bellwether.yaml without running tests |
 
 ## CI/CD Exit Codes
 
@@ -120,13 +134,14 @@ Requires LLM (Ollama for free local, or OpenAI/Anthropic). Generates `AGENTS.md`
 | `2` | Warning-level changes | Warn |
 | `3` | Breaking changes | Fail |
 | `4` | Runtime error | Fail |
+| `5` | Low confidence metrics | Warn or fail |
 
 ## GitHub Action
 
 ```yaml
-- uses: dotsetlabs/bellwether@v1.0.2
+- uses: dotsetlabs/bellwether@v2.0.0
   with:
-    version: '1.0.2'
+    version: '2.0.0'
     server-command: 'npx @mcp/your-server'
     baseline-path: './bellwether-baseline.json'
     fail-on-severity: 'warning'

package/dist/auth/credentials.js

@@ -58,6 +58,8 @@ function readEnvFile(filePath, envVar, options) {
                 if (decrypted) {
                     return decrypted;
                 }
+                // Warn about decryption failure so users know their credential exists but can't be decrypted
+                console.warn(`[bellwether] Encrypted credential found for ${envVar} but decryption failed. Check your encryption key.`);
                 return undefined;
             }
             if (value) {

package/dist/baseline/accessors.d.ts

@@ -1,4 +1,4 @@
-import type { ToolCapability } from './cloud-types.js';
+import type { ToolCapability } from './baseline-format.js';
 import type { BehavioralBaseline, ToolFingerprint } from './types.js';
 export declare function getBaselineGeneratedAt(baseline: BehavioralBaseline): Date;
 export declare function getBaselineHash(baseline: BehavioralBaseline): string;

package/dist/baseline/accessors.js

@@ -57,9 +57,15 @@ export function toToolCapability(tool) {
         errorPatterns: tool.errorPatterns,
         baselineP50Ms: tool.baselineP50Ms,
         baselineP95Ms: tool.baselineP95Ms,
+        baselineP99Ms: tool.baselineP99Ms,
         baselineSuccessRate: tool.baselineSuccessRate,
         performanceConfidence: tool.performanceConfidence,
         securityFingerprint: tool.securityFingerprint,
+        title: tool.title,
+        outputSchema: tool.outputSchema,
+        outputSchemaHash: tool.outputSchemaHash,
+        annotations: tool.annotations,
+        execution: tool.execution,
     };
 }
 export function getToolFingerprints(baseline) {
@@ -73,9 +79,7 @@ export function getToolFingerprints(baseline) {
         const limitations = profile?.limitations ?? [];
         const description = tool.description || profile?.description || '';
         const schemaHash = tool.schemaHash || profile?.schemaHash || '';
-        const lastTestedAt = tool.lastTestedAt
-            ? new Date(tool.lastTestedAt)
-            : undefined;
+        const lastTestedAt = tool.lastTestedAt ? new Date(tool.lastTestedAt) : undefined;
         return {
             name: tool.name,
             description,
@@ -95,6 +99,12 @@ export function getToolFingerprints(baseline) {
             baselineSuccessRate: tool.baselineSuccessRate,
             performanceConfidence: tool.performanceConfidence,
             securityFingerprint: tool.securityFingerprint,
+            title: tool.title,
+            outputSchema: tool.outputSchema,
+            outputSchemaHash: tool.outputSchemaHash,
+            annotations: tool.annotations,
+            execution: tool.execution,
+            baselineP99Ms: tool.baselineP99Ms,
         };
     });
     if (fingerprints.length > 0) {

package/dist/baseline/baseline-format.d.ts

@@ -0,0 +1,335 @@
+/**
+ * Baseline types for Bellwether.
+ *
+ * These types define the canonical baseline format used for:
+ * - Local baseline storage
+ * - Baseline comparison and drift detection
+ * - Tool capability tracking
+ *
+ * Originally part of a hosted integration, now standalone for open-source use.
+ */
+import type { WorkflowSignature } from './types.js';
+import type { ResponseFingerprint, InferredSchema, ErrorPattern } from './response-fingerprint.js';
+import type { SecurityFingerprint } from '../security/types.js';
+/**
+ * Assertion type for baseline assertions.
+ * Maps to: expects (positive), requires (critical), warns (negative), notes (informational)
+ */
+export type BaselineAssertionType = 'expects' | 'requires' | 'warns' | 'notes';
+/**
+ * Severity level for assertions.
+ */
+export type BaselineAssertionSeverity = 'info' | 'low' | 'medium' | 'high' | 'critical';
+/**
+ * Behavioral assertion in baseline format.
+ */
+export interface BaselineAssertion {
+    /** Type of assertion */
+    type: BaselineAssertionType;
+    /** The condition/assertion statement */
+    condition: string;
+    /** Tool this assertion relates to (optional) */
+    tool?: string;
+    /** Severity level (optional) */
+    severity?: BaselineAssertionSeverity;
+}
+/**
+ * Baseline mode indicating how the baseline was generated.
+ * - 'check': Deterministic structural testing (no LLM required)
+ * - 'explore': LLM-powered behavioral exploration
+ */
+export type BaselineMode = 'check' | 'explore';
+/**
+ * Metadata about how the baseline was generated.
+ */
+export interface BaselineMetadata {
+    /** Baseline mode: 'check' = deterministic, 'explore' = LLM-powered */
+    mode: BaselineMode;
+    /** ISO timestamp when generated */
+    generatedAt: string;
+    /** CLI version that generated this baseline */
+    cliVersion: string;
+    /** Command used to start the server */
+    serverCommand: string;
+    /** Server name from MCP initialization */
+    serverName?: string;
+    /** Interview duration in milliseconds */
+    durationMs: number;
+    /** Personas used during interview (empty for check mode) */
+    personas: string[];
+    /** LLM model used ('none' for check mode) */
+    model: string;
+}
+/**
+ * Server fingerprint in baseline format.
+ */
+export interface BaselineServerFingerprint {
+    /** Server name */
+    name: string;
+    /** Server version */
+    version: string;
+    /** MCP protocol version */
+    protocolVersion: string;
+    /** Available capabilities */
+    capabilities: string[];
+    /** Server-provided instructions (MCP 2025-11-25) */
+    instructions?: string;
+}
+/**
+ * Tool capability from discovery.
+ */
+export interface ToolCapability {
+    /** Tool name */
+    name: string;
+    /** Tool description */
+    description: string;
+    /** Input schema */
+    inputSchema: Record<string, unknown>;
+    /** Hash of the schema for change detection */
+    schemaHash: string;
+    /** Human-readable title for the tool (MCP 2025-11-25) */
+    title?: string;
+    /** JSON Schema for the tool's output (MCP 2025-11-25 structured content) */
+    outputSchema?: Record<string, unknown>;
+    /** Hash of the output schema for drift detection */
+    outputSchemaHash?: string;
+    /** Behavioral annotations/hints (MCP 2025-11-25) */
+    annotations?: {
+        title?: string;
+        readOnlyHint?: boolean;
+        destructiveHint?: boolean;
+        idempotentHint?: boolean;
+        openWorldHint?: boolean;
+    };
+    /** Task execution configuration (MCP 2025-11-25) */
+    execution?: {
+        taskSupport?: string;
+    };
+    /** Hash of observed arguments schema (from actual calls) */
+    observedArgsSchemaHash?: string;
+    /** Consistency of observed argument schemas (0-1) */
+    observedArgsSchemaConsistency?: number;
+    /** Number of observed schema variations */
+    observedArgsSchemaVariations?: number;
+    /** Fingerprint of the tool's response structure */
+    responseFingerprint?: ResponseFingerprint;
+    /** Inferred JSON schema of the tool's output */
+    inferredOutputSchema?: InferredSchema;
+    /** Normalized error patterns observed during testing */
+    errorPatterns?: ErrorPattern[];
+    /** Baseline p50 latency in milliseconds */
+    baselineP50Ms?: number;
+    /** Baseline p95 latency in milliseconds */
+    baselineP95Ms?: number;
+    /** Baseline p99 latency in milliseconds */
+    baselineP99Ms?: number;
+    /** Baseline success rate (0-1) */
+    baselineSuccessRate?: number;
+    /** Response schema evolution metadata */
+    responseSchemaEvolution?: ResponseSchemaEvolution;
+    /** ISO timestamp of last time this tool was tested */
+    lastTestedAt?: string;
+    /** Schema hash captured at the last test time */
+    inputSchemaHashAtTest?: string;
+    /** Statistical confidence for performance baselines */
+    performanceConfidence?: {
+        sampleCount: number;
+        successfulSamples: number;
+        validationSamples: number;
+        totalTests: number;
+        standardDeviation: number;
+        coefficientOfVariation: number;
+        confidenceLevel: 'low' | 'medium' | 'high';
+        recommendation?: string;
+    };
+    /** Security testing fingerprint with findings and risk score */
+    securityFingerprint?: SecurityFingerprint;
+}
+/**
+ * Resource capability from discovery.
+ */
+export interface ResourceCapability {
+    /** Resource URI template */
+    uri: string;
+    /** Resource name */
+    name: string;
+    /** Resource description */
+    description?: string;
+    /** MIME type */
+    mimeType?: string;
+    /** Human-readable title (MCP 2025-11-25) */
+    title?: string;
+    /** Resource annotations (MCP 2025-11-25) */
+    annotations?: {
+        audience?: string[];
+        priority?: number;
+        lastModified?: string;
+    };
+    /** Resource size in bytes (MCP 2025-11-25) */
+    size?: number;
+}
+/**
+ * Resource template capability from discovery (MCP 2025-11-25).
+ */
+export interface ResourceTemplateCapability {
+    /** URI template (RFC 6570) */
+    uriTemplate: string;
+    /** Template name */
+    name: string;
+    /** Human-readable title */
+    title?: string;
+    /** Template description */
+    description?: string;
+    /** Expected MIME type */
+    mimeType?: string;
+}
+/**
+ * Prompt capability from discovery.
+ */
+export interface PromptCapability {
+    /** Prompt name */
+    name: string;
+    /** Prompt description */
+    description?: string;
+    /** Human-readable title (MCP 2025-11-25) */
+    title?: string;
+    /** Arguments the prompt accepts */
+    arguments?: Array<{
+        name: string;
+        description?: string;
+        required?: boolean;
+    }>;
+}
+/**
+ * Interview results for a single persona.
+ */
+export interface PersonaInterview {
+    /** Persona ID */
+    persona: string;
+    /** Number of tools interviewed */
+    toolsInterviewed: number;
+    /** Number of questions asked */
+    questionsAsked: number;
+    /** Findings from this persona */
+    findings: PersonaFinding[];
+}
+/**
+ * A finding from a persona interview.
+ */
+export interface PersonaFinding {
+    /** Tool this finding relates to */
+    tool: string;
+    /** Finding category */
+    category: 'behavior' | 'security' | 'reliability' | 'edge_case';
+    /** Severity level */
+    severity: 'info' | 'low' | 'medium' | 'high' | 'critical';
+    /** Description of the finding */
+    description: string;
+    /** Evidence supporting the finding */
+    evidence?: string;
+}
+/**
+ * Tool behavioral profile in baseline format.
+ */
+export interface BaselineToolProfile {
+    /** Tool name */
+    name: string;
+    /** Tool description */
+    description: string;
+    /** Hash of input schema */
+    schemaHash: string;
+    /** Behavioral assertions */
+    assertions: BaselineAssertion[];
+    /** Security notes */
+    securityNotes: string[];
+    /** Known limitations */
+    limitations: string[];
+    /** Behavioral notes */
+    behavioralNotes: string[];
+}
+/**
+ * Snapshot of accepted drift for a baseline.
+ */
+export interface AcceptedDiff {
+    toolsAdded: string[];
+    toolsRemoved: string[];
+    toolsModified: string[];
+    severity: 'none' | 'info' | 'warning' | 'breaking';
+    breakingCount: number;
+    warningCount: number;
+    infoCount: number;
+}
+/**
+ * Drift acceptance metadata attached to a baseline.
+ */
+export interface DriftAcceptance {
+    acceptedAt: string | Date;
+    acceptedBy?: string;
+    reason?: string;
+    acceptedDiff: AcceptedDiff;
+}
+/**
+ * Serializable schema evolution data for baselines.
+ */
+export interface ResponseSchemaEvolution {
+    currentHash: string;
+    history: Array<{
+        hash: string;
+        schema: InferredSchema;
+        observedAt: string | Date;
+        sampleCount: number;
+    }>;
+    isStable: boolean;
+    stabilityConfidence: number;
+    inconsistentFields: string[];
+    sampleCount: number;
+}
+/**
+ * Serializable documentation score summary for baseline storage.
+ */
+export interface DocumentationScoreSummary {
+    overallScore: number;
+    grade: string;
+    issueCount: number;
+    toolCount: number;
+}
+/**
+ * Canonical baseline format.
+ *
+ * This is the single baseline schema used by Bellwether CLI.
+ *
+ * Versioning: Uses CLI package version for compatibility checking.
+ * Baselines with the same CLI major version are compatible.
+ */
+export interface BellwetherBaseline {
+    /** CLI version that generated this baseline (e.g., '1.0.0') */
+    version: string;
+    /** Generation metadata */
+    metadata: BaselineMetadata;
+    /** Server fingerprint */
+    server: BaselineServerFingerprint;
+    /** Discovered capabilities */
+    capabilities: {
+        tools: ToolCapability[];
+        resources?: ResourceCapability[];
+        resourceTemplates?: ResourceTemplateCapability[];
+        prompts?: PromptCapability[];
+    };
+    /** Interview results by persona */
+    interviews: PersonaInterview[];
+    /** Tool behavioral profiles */
+    toolProfiles: BaselineToolProfile[];
+    /** Workflow results (if workflows were tested) */
+    workflows?: WorkflowSignature[];
+    /** Overall behavioral assertions */
+    assertions: BaselineAssertion[];
+    /** Summary of findings */
+    summary: string;
+    /** SHA-256 hash of content (first 16 chars) for integrity */
+    hash: string;
+    /** Drift acceptance metadata (optional) */
+    acceptance?: DriftAcceptance;
+    /** Optional documentation score summary */
+    documentationScore?: DocumentationScoreSummary;
+}
+//# sourceMappingURL=baseline-format.d.ts.map

package/dist/baseline/baseline-format.js

@@ -0,0 +1,12 @@
+/**
+ * Baseline types for Bellwether.
+ *
+ * These types define the canonical baseline format used for:
+ * - Local baseline storage
+ * - Baseline comparison and drift detection
+ * - Tool capability tracking
+ *
+ * Originally part of a hosted integration, now standalone for open-source use.
+ */
+export {};
+//# sourceMappingURL=baseline-format.js.map