@kryptosai/mcp-observatory 0.20.3 → 0.22.0

package/docs/project-case-study.md

@@ -0,0 +1,106 @@
+# MCP Observatory Case Study
+
+## One-Line Summary
+
+MCP Observatory is CI/security infrastructure for production MCP servers.
+
+## Problem
+
+MCP servers are becoming dependencies for AI agents. Teams need to know whether those servers still start, expose usable tools, keep schemas stable, avoid obvious security footguns, and behave consistently as agents depend on them.
+
+## Product
+
+MCP Observatory provides:
+
+- CLI checks for MCP servers
+- MCP server mode so agents can inspect other MCP servers
+- GitHub Action integration
+- PR comments and commit status checks
+- JSON, Markdown, HTML, JUnit, SARIF, and PR-comment reports
+- schema drift detection
+- record/replay/verify workflows
+- health score badges
+- static enterprise reports
+- telemetry intelligence for product and account-level learning
+
+## Architecture
+
+The project is a TypeScript/Node CLI with modular command handlers, MCP adapters, check runners, reporters, artifact schemas, and a GitHub Action wrapper. A Cloudflare Worker handles hosted artifact upload pilots, and a separate telemetry Worker stores private aggregate usage events in D1.
+
+## Technical Proof
+
+As of June 19, 2026:
+
+- 10k+ source lines in `src`
+- 40 test files
+- 321 passing tests
+- npm package published
+- GitHub Action available
+- MCP server mode available
+- telemetry export and company intelligence tooling available
+
+## Traction Snapshot
+
+Safe public and aggregate signals:
+
+- 10,278 telemetry events
+- 7,211 telemetry sessions
+- 5,368 external sessions after separating internal activity
+- 582 GitHub clones and 175 unique cloners in the visible June 2026 traffic window
+- 104 npm downloads during June 11-17, 2026
+
+These are early signals. Public social proof is still limited and should be improved through the certification campaign.
+
+## Security And Privacy Posture
+
+The project includes:
+
+- telemetry opt-out controls
+- privacy disclosure
+- security policy
+- token-based hosted artifact upload
+- private-network rejection for hosted scans
+- sanitized public reporting policy
+
+Public claims should use aggregate metrics and accepted public integrations, not raw telemetry.
+
+## Commercial Path
+
+The free OSS wedge is local testing and public repo CI. Paid value is production/private use:
+
+- hosted CI history
+- private repo reporting
+- recurring security reports
+- certification
+- support
+- fleet visibility
+- enterprise review
+
+Current pilot anchors:
+
+- Team: starts at `$299/month`
+- Business: starts at `$999/month`
+- Enterprise: starts at `$3k/month`
+- Strategic: `$250k+/year`
+
+## Job-Opportunity Value
+
+This project demonstrates ability across:
+
+- AI infrastructure
+- developer tooling
+- security tooling
+- MCP ecosystem work
+- CI/CD integrations
+- telemetry and product analytics
+- commercialization and enterprise packaging
+
+It is strongest as a portfolio asset when paired with public proof: accepted external PRs, badges in other repos, directory listings, and a short demo.
+
+## Next Milestones
+
+1. Publish latest package with `init-ci`.
+2. Open first certification PR wave.
+3. Capture accepted PRs as public proof.
+4. Publish recurring MCP safety reports.
+5. Convert serious users into paid pilots.

package/docs/proof.md

@@ -0,0 +1,68 @@
+# MCP Observatory Proof
+
+MCP Observatory is early, but it is already a working MCP testing/security stack rather than a landing page.
+
+## Current Public Surface
+
+- npm package: `@kryptosai/mcp-observatory`
+- GitHub Action: `KryptosAI/mcp-observatory/action@main`
+- CLI command count: scan, test, record, replay, verify, diff, watch, suggest, serve, lock, history, init-ci, ci-report, enterprise-report, score, badge, cloud
+- Test suite: 321 passing tests across 40 test files as of June 19, 2026
+- GitHub traffic snapshot: 582 clones and 175 unique cloners in the visible June 2026 traffic window
+- npm downloads snapshot: 104 downloads for June 11-17, 2026
+
+## Safe Aggregate Telemetry Snapshot
+
+Internal telemetry is used for product analytics and account-level outreach. Public reporting uses only aggregate or sanitized data.
+
+As of the latest local export on June 19, 2026:
+
+- 10,278 telemetry events
+- 7,211 unique sessions
+- 5,368 external sessions after separating internal/personal activity
+- 2,434 external CI sessions
+- 128 attributed company/org sessions
+- top external commands: `serve`, `run`, `diff`, `test`, `scan`, `history`
+
+Raw emails, hostnames, private URLs, tokens, and response bodies are not published.
+
+## Product Proof
+
+MCP Observatory can:
+
+- install as an npm CLI
+- run as an MCP server
+- test local-process and HTTP MCP targets
+- run in GitHub Actions
+- post PR comments
+- generate JSON, Markdown, HTML, JUnit, SARIF, and PR-comment reports
+- generate health badges
+- record/replay/verify MCP sessions
+- detect schema drift
+- run lightweight MCP security checks
+- create static enterprise reports from artifacts
+
+## Certification Proof
+
+The certification campaign is designed to create public proof through accepted maintainer PRs.
+
+Accepted third-party integrations will be tracked here:
+
+| Repo | PR | Check Added | Badge Added | Status |
+| --- | --- | --- | --- | --- |
+| _pending_ | | | | |
+
+## Commercial Proof
+
+Current paid positioning is pilot-first:
+
+- Team Pilot: starts at `$299/month`
+- Business Pilot: starts at `$999/month`
+- Enterprise Pilot: starts at `$3k/month`
+- Strategic Accounts: custom, `$250k+/year`
+
+Paid value is production use: hosted reporting, private repo CI history, security reports, certification, support, fleet visibility, and enterprise review.
+
+## Claims Policy
+
+Use public proof for public claims. Company-domain telemetry signals are private and should only be used internally for outreach unless a customer gives permission or there is independent public evidence.

package/docs/publish-readiness.md

@@ -0,0 +1,77 @@
+# Publish And Distribution Readiness
+
+Use this checklist before a commercialization push. The goal is to increase distribution and start paid conversations without locking the project into a permanent licensing or product shape.
+
+## Release Gate
+
+Run:
+
+```bash
+npm run lint
+npm run typecheck
+npm run build
+npm test
+npm run validate:artifacts
+npm audit --json
+```
+
+Confirm:
+
+- GitHub Action target scans support `deep`, `security`, and baseline drift failure.
+- `schemas/run-artifact.schema.json` and `schemas/diff-artifact.schema.json` validate current artifacts.
+- HTTP target examples use env references instead of inline tokens.
+- Security findings appear in artifact evidence as structured `findings`.
+- Hosted upload is available through `mcp-observatory cloud upload <artifact>` when `MCP_OBSERVATORY_CLOUD_TOKEN` is set.
+
+Known audit note:
+
+- `npm audit` may report `undici <=6.26.0` through the `npm@11.17.0` package bundled under `@semantic-release/npm`. `npm audit fix` updates the fixable `@actions/http-client` path, but the remaining `undici` copy is bundled inside npm release tooling and is not part of MCP Observatory runtime dependencies or the packed npm artifact. Recheck after npm publishes a newer package.
+
+## Public Distribution
+
+- Merge the health/commercialization PR.
+- Update the GitHub repo homepage to the README or commercial page.
+- Publish npm only after the release gate is green.
+- Refresh MCP directory listings with: “MCP Observatory helps teams test, secure, and monitor MCP servers before agents depend on them.”
+- Include “free for local OSS use; paid for hosted reporting, private repo CI, security reports, production monitoring, certification, support, and fleet visibility.”
+- Link production users to `COMMERCIAL.md` and `william@banksey.com`.
+- Submit or refresh listings on Glama, PulseMCP, Smithery, and relevant awesome-MCP lists with the tags: security, developer tools, CI/CD, testing, observability, schema drift.
+- Use the certification distribution loop to open helpful PRs against popular MCP server repos and convert accepted PRs into proof points.
+- Link public proof, the safety report, and directory listing copy from launch/outreach materials.
+
+## Sales Operating Loop
+
+1. Export telemetry rows privately.
+2. Run:
+
+```bash
+npx tsx scripts/telemetry-company-intelligence.ts --input telemetry-events.jsonl --out-dir reports
+```
+
+3. Review `reports/telemetry-company-intelligence.html`.
+4. Contact the top 25 high-confidence company/org candidates.
+5. For large or strategic domains, quote Strategic only and ask for a security/procurement path.
+6. Use hosted artifact upload or static enterprise reports before building a full SaaS dashboard.
+
+## Hosted Upload
+
+CLI:
+
+```bash
+export MCP_OBSERVATORY_CLOUD_TOKEN="pilot-token"
+export MCP_OBSERVATORY_ORG="customer.com"
+npx @kryptosai/mcp-observatory cloud upload .mcp-observatory/runs/<run>.json
+```
+
+Worker:
+
+- `POST /api/v1/artifacts` stores a run artifact behind bearer-token auth.
+- `GET /api/v1/artifacts/:org` returns the org artifact index behind the same auth.
+- Hosted scans reject localhost/private-network targets; use local CLI for internal MCP servers.
+
+## What Not To Do Yet
+
+- Do not change the MIT license until paid signal demands it.
+- Do not hard-paywall existing local OSS checks.
+- Do not build a full dashboard before a buyer asks for dashboard workflows.
+- Do not publish raw telemetry emails or individual PII in public materials.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@kryptosai/mcp-observatory",
-  "version": "0.20.3",
-  "description": "Test your MCP servers for breaking changes. Checks capabilities, invokes tools, detects schema drift between versions.",
+  "version": "0.22.0",
+  "description": "Test, secure, and monitor MCP servers before agents depend on them.",
   "mcpName": "io.github.KryptosAI/mcp-observatory",
   "license": "MIT",
   "type": "module",
@@ -31,7 +31,11 @@
   "files": [
     "dist/src",
     "README.md",
+    "COMMERCIAL.md",
+    "PRIVACY.md",
+    "TERMS.md",
     "LICENSE",
+    "docs",
     "schemas"
   ],
   "scripts": {
@@ -43,11 +47,15 @@
     "prepack": "npm run build",
     "proof:refresh": "tsx scripts/refresh-proof-artifacts.ts",
     "release:prep": "node scripts/release.mjs",
+    "certification:pr-body": "tsx scripts/print-certification-pr-body.ts",
     "typecheck": "tsc --noEmit -p tsconfig.json",
     "test": "vitest run",
     "validate:artifacts": "tsx scripts/validate-artifacts.ts",
     "verify:packed-install": "node scripts/verify-packed-install.mjs",
-    "smoke": "npm run cli -- run --target tests/fixtures/sample-target-config.json && npm run cli -- diff tests/fixtures/sample-run-a.json tests/fixtures/sample-run-b.json"
+    "smoke": "npm run cli -- run --target tests/fixtures/sample-target-config.json && npm run cli -- diff tests/fixtures/sample-run-a.json tests/fixtures/sample-run-b.json",
+    "telemetry:export": "tsx scripts/export-telemetry-d1.ts",
+    "telemetry:intelligence": "tsx scripts/telemetry-company-intelligence.ts --input telemetry-exports/events-flat-full.json --out-dir reports",
+    "telemetry:external": "tsx scripts/telemetry-company-intelligence.ts --input telemetry-exports/events-flat-full.json --out-dir reports"
   },
   "keywords": [
     "mcp",
@@ -66,31 +74,38 @@
     "vcr",
     "mcp-testing",
     "security",
+    "mcp-security",
     "ci-cd",
+    "mcp-ci",
     "github-action",
+    "production-monitoring",
+    "enterprise",
+    "enterprise-report",
+    "feishu",
+    "lark",
     "schema-drift"
   ],
   "dependencies": {
-    "@modelcontextprotocol/sdk": "^1.27.1",
+    "@modelcontextprotocol/sdk": "^1.29.0",
     "ci-info": "^4.4.0",
-    "commander": "14.0.2",
+    "commander": "15.0.0",
     "update-notifier": "^7.3.1",
-    "zod": "4.1.12"
+    "zod": "4.4.3"
   },
   "devDependencies": {
-    "@eslint/js": "9.39.1",
+    "@eslint/js": "10.0.1",
     "@semantic-release/changelog": "^6.0.3",
     "@semantic-release/git": "^10.0.1",
-    "@types/node": "24.10.1",
-    "@typescript-eslint/eslint-plugin": "8.46.3",
-    "@typescript-eslint/parser": "8.46.3",
-    "ajv": "8.17.1",
-    "eslint": "9.39.1",
-    "globals": "16.5.0",
-    "semantic-release": "^25.0.3",
-    "tsx": "4.20.6",
-    "typescript": "5.9.3",
-    "typescript-eslint": "8.46.3",
-    "vitest": "4.0.8"
+    "@types/node": "25.9.3",
+    "@typescript-eslint/eslint-plugin": "8.61.1",
+    "@typescript-eslint/parser": "8.61.1",
+    "ajv": "8.20.0",
+    "eslint": "10.5.0",
+    "globals": "17.6.0",
+    "semantic-release": "^25.0.5",
+    "tsx": "4.22.4",
+    "typescript": "6.0.3",
+    "typescript-eslint": "8.61.1",
+    "vitest": "4.1.9"
   }
 }

package/schemas/diff-artifact.schema.json

@@ -19,113 +19,109 @@
     "removed"
   ],
   "properties": {
-    "artifactType": {
-      "const": "diff"
-    },
-    "schemaVersion": {
-      "const": "1.0.0"
-    },
-    "gate": {
-      "$ref": "#/definitions/gate"
-    },
-    "baseRunId": {
-      "type": "string"
-    },
-    "headRunId": {
-      "type": "string"
-    },
-    "createdAt": {
-      "type": "string"
-    },
-    "summary": {
-      "$ref": "#/definitions/diffSummary"
-    },
+    "artifactType": { "const": "diff" },
+    "schemaVersion": { "const": "1.0.0" },
+    "gate": { "$ref": "#/definitions/gate" },
+    "baseRunId": { "type": "string" },
+    "headRunId": { "type": "string" },
+    "createdAt": { "type": "string" },
+    "summary": { "$ref": "#/definitions/diffSummary" },
     "regressions": {
       "type": "array",
-      "items": {
-        "$ref": "#/definitions/diffEntry"
-      }
+      "items": { "$ref": "#/definitions/diffEntry" }
     },
     "recoveries": {
       "type": "array",
-      "items": {
-        "$ref": "#/definitions/diffEntry"
-      }
+      "items": { "$ref": "#/definitions/diffEntry" }
     },
     "unchanged": {
       "type": "array",
-      "items": {
-        "$ref": "#/definitions/diffEntry"
-      }
+      "items": { "$ref": "#/definitions/diffEntry" }
     },
     "added": {
       "type": "array",
-      "items": {
-        "$ref": "#/definitions/diffEntry"
-      }
+      "items": { "$ref": "#/definitions/diffEntry" }
     },
     "removed": {
       "type": "array",
-      "items": {
-        "$ref": "#/definitions/diffEntry"
-      }
+      "items": { "$ref": "#/definitions/diffEntry" }
+    },
+    "schemaDrift": {
+      "type": "array",
+      "items": { "$ref": "#/definitions/schemaDriftEntry" }
+    },
+    "responseChanges": {
+      "type": "array",
+      "items": { "$ref": "#/definitions/responseChangeEntry" }
     }
   },
   "definitions": {
-    "gate": {
-      "type": "string",
-      "enum": ["pass", "fail"]
-    },
+    "gate": { "type": "string", "enum": ["pass", "fail"] },
     "checkStatus": {
       "type": "string",
       "enum": ["pass", "fail", "partial", "unsupported", "flaky", "skipped"]
     },
     "checkId": {
       "type": "string",
-      "enum": ["tools", "prompts", "resources"]
+      "enum": [
+        "tools",
+        "prompts",
+        "resources",
+        "tools-invoke",
+        "security",
+        "security-lite",
+        "conformance",
+        "schema-quality"
+      ]
     },
     "diffEntry": {
       "type": "object",
       "additionalProperties": false,
       "required": ["id", "capability", "message"],
       "properties": {
-        "id": {
-          "$ref": "#/definitions/checkId"
-        },
-        "capability": {
-          "$ref": "#/definitions/checkId"
-        },
-        "fromStatus": {
-          "$ref": "#/definitions/checkStatus"
-        },
-        "toStatus": {
-          "$ref": "#/definitions/checkStatus"
-        },
-        "message": {
-          "type": "string"
+        "id": { "$ref": "#/definitions/checkId" },
+        "capability": { "$ref": "#/definitions/checkId" },
+        "fromStatus": { "$ref": "#/definitions/checkStatus" },
+        "toStatus": { "$ref": "#/definitions/checkStatus" },
+        "message": { "type": "string" }
+      }
+    },
+    "schemaDriftEntry": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["capability", "name", "changes"],
+      "properties": {
+        "capability": { "$ref": "#/definitions/checkId" },
+        "name": { "type": "string" },
+        "changes": {
+          "type": "array",
+          "items": { "type": "string" }
         }
       }
     },
+    "responseChangeEntry": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["capability", "name", "change"],
+      "properties": {
+        "capability": { "$ref": "#/definitions/checkId" },
+        "name": { "type": "string" },
+        "change": { "type": "string" }
+      }
+    },
     "diffSummary": {
       "type": "object",
       "additionalProperties": false,
-      "required": [
-        "regressions",
-        "recoveries",
-        "unchanged",
-        "added",
-        "removed",
-        "gate"
-      ],
+      "required": ["regressions", "recoveries", "unchanged", "added", "removed", "gate"],
       "properties": {
         "regressions": { "type": "integer", "minimum": 0 },
         "recoveries": { "type": "integer", "minimum": 0 },
         "unchanged": { "type": "integer", "minimum": 0 },
         "added": { "type": "integer", "minimum": 0 },
         "removed": { "type": "integer", "minimum": 0 },
-        "gate": {
-          "$ref": "#/definitions/gate"
-        }
+        "schemaDriftCount": { "type": "integer", "minimum": 0 },
+        "responseChangeCount": { "type": "integer", "minimum": 0 },
+        "gate": { "$ref": "#/definitions/gate" }
       }
     }
   }