@dotsetlabs/bellwether 0.11.0 → 1.0.0

package/CHANGELOG.md

@@ -2,6 +2,134 @@
 
 All notable changes to this project will be documented in this file.
 
+## [1.0.0] - 2026-01-27
+
+### Breaking Changes
+
+- **Removed cloud commands**: The following commands have been removed: `login`, `upload`, `projects`, `history`, `diff`, `link`, `teams`, `badge`
+- **Removed benchmark command**: The `benchmark` command and "Tested with Bellwether" certification program have been removed
+- **Removed cloud module**: All cloud integration code has been removed from the CLI
+
+### Changed
+
+- **Fully open source**: Bellwether is now a completely free, open-source tool with no cloud dependencies
+- **Simplified configuration**: Removed cloud-related settings from `bellwether.yaml` template
+- **Updated documentation**: Removed all cloud-related documentation
+
+### Migration Guide
+
+If you were using cloud features:
+
+1. **Baselines**: Store baselines in git instead of uploading to cloud
+   ```bash
+   bellwether baseline save
+   git add bellwether-baseline.json
+   git commit -m "Add baseline"
+   ```
+
+2. **CI/CD**: Use local baseline comparison instead of cloud upload
+   ```bash
+   # Old
+   bellwether upload --ci --fail-on-drift
+
+   # New
+   bellwether check --fail-on-drift
+   bellwether baseline compare ./bellwether-baseline.json
+   ```
+
+3. **Environment variables**: Remove `BELLWETHER_SESSION`, `BELLWETHER_API_URL`, `BELLWETHER_TEAM_ID` from your CI/CD configuration
+
+## [0.13.0] - 2026-01-27
+
+### Breaking Changes
+
+- **Renamed `bellwether verify` to `bellwether benchmark`**: The verification command has been renamed to better reflect its purpose
+  - Old: `bellwether verify <server-command>`
+  - New: `bellwether benchmark <server-command>`
+- **Renamed "Verified by Bellwether" to "Tested with Bellwether"**: Updated branding throughout the CLI and documentation
+  - Badge text now shows "Tested with Bellwether"
+  - Status values changed: `verified` → `passed`, `not_verified` → `not_tested`
+- **Config section renamed**: The `verify:` section in `bellwether.yaml` is now `benchmark:`
+  - Old: `verify: { timeout: 30000 }`
+  - New: `benchmark: { timeout: 30000 }`
+- **Output file renamed**: Default benchmark report file changed from `bellwether-verification.json` to `bellwether-benchmark.json`
+- **Cloud API changes**: Benchmark-related API endpoints have been renamed
+  - `/verifications` → `/benchmarks`
+  - Activity events: `verification.completed` → `benchmark.completed`, `verification.failed` → `benchmark.failed`
+
+### Changed
+
+- All CLI output messages updated to use "benchmark" terminology
+- Documentation updated throughout to reflect new naming
+- Badge command description updated to reference "benchmark badge"
+- Constants renamed: `VERIFICATION_TIERS` → `BENCHMARK_TIERS`, `DEFAULT_VERIFICATION_REPORT_FILE` → `DEFAULT_BENCHMARK_REPORT_FILE`
+
+### Migration Guide
+
+1. Update your `bellwether.yaml` config file:
+   ```yaml
+   # Old
+   verify:
+     timeout: 30000
+
+   # New
+   benchmark:
+     timeout: 30000
+   ```
+
+2. Update any CI/CD scripts:
+   ```bash
+   # Old
+   bellwether verify npx @mcp/server
+
+   # New
+   bellwether benchmark npx @mcp/server
+   ```
+
+3. Update any references to the output file:
+   - `bellwether-verification.json` → `bellwether-benchmark.json`
+
+## [0.12.0] - 2026-01-26
+
+### Features
+
+- **Streamable HTTP transport improvements**: Full compliance with [MCP Streamable HTTP specification](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports)
+  - Fixed Accept header to include both `application/json` and `text/event-stream` as required by spec
+  - Added automatic session ID capture from `Mcp-Session-Id` response header
+  - Session ID is automatically included in all subsequent requests after initialization
+  - Changed header name from `X-Session-Id` to `Mcp-Session-Id` per MCP specification
+- **False positive reduction**: Intelligent pattern detection to reduce false positives in automated testing
+  - **Operation-based tool detection**: Tools with `operation` enum + `args` object patterns now use flexible `either` outcome
+  - **Self-stateful tool detection**: Tools requiring prior state (session/chain/context) are handled appropriately
+  - **Complex array schema detection**: Arrays with nested objects containing required properties use flexible validation
+  - **Flexible semantic validation**: Semantic type tests now use `either` outcome by default, allowing tools to accept varied formats (e.g., dayjs, date-fns)
+- **Pattern detection metadata**: Test metadata now includes detection flags for transparency
+  - `operationBased`, `operationParam`, `argsParam` for operation-based tools
+  - `selfStateful`, `selfStatefulReason` for stateful tools
+  - `hasComplexArrays`, `complexArrayParams` for complex schema tools
+
+### Configuration
+
+- **New semantic validation option**: `check.flexibleSemanticTests` (default: `true`)
+  - When `true`, semantic validation tests use `either` outcome
+  - Set to `false` for strict format enforcement
+
+### Documentation
+
+- Updated remote-servers guide with correct streamable-http protocol details
+- Added MCP specification link for transport documentation
+- Clarified session ID behavior and Accept header requirements
+
+### Fixes
+
+- **Streamable HTTP session management**: Fixed session ID header to use MCP-compliant `Mcp-Session-Id`
+- **False positive tests**: Tests for operation-based, self-stateful, and complex array patterns no longer fail incorrectly
+
+### Tests
+
+- Added 17 HTTP transport tests including session ID capture verification
+- Added 11 new pattern detection tests for false positive reduction
+
 ## [0.11.0] - 2026-01-26
 
 ### Breaking Changes