@honua/mcp-server 0.0.9-alpha.0 → 0.0.20-alpha.0

package/README.md

@@ -40,3 +40,170 @@ HONUA_BASE_URL="https://honua.example.com" HONUA_TRANSPORT="grpc-web" node dist/
 
 - `honua://services`
 - `honua://services/{encodedServiceId}/layers/{layerId}`
+
+## Certification
+
+The package ships a **deterministic MCP certification harness** that proves the
+advertised MCP surface is well-formed and conformant to the open
+[`geospatial-mcp`](https://github.com/honua-io/geospatial-mcp) standard. It is
+fully offline — no model/API calls, no network in the default path — and is the
+evidence document for the WS-H "Provability" workstream.
+
+For each tool the server advertises (over an in-memory MCP transport) it:
+
+1. Enumerates `tools/list`, `resources/list`, and `prompts/list`.
+2. Validates each `inputSchema` (and any `structuredContent` output schema) is
+   well-formed JSON Schema, accepting both the draft-07 dialect emitted by
+   zod-to-json-schema and the draft 2020-12 dialect of the standard.
+3. Where a vendored standard schema matches the advertised tool (by the
+   standard's `referenceToolName`), checks **conformance** — every standard
+   `required` property must be accepted with a compatible type. Standard tools
+   that are not advertised, and advertised tools outside the standard, are
+   recorded as **known gaps**, not failures.
+4. **Round-trips** every read-only tool (`tools/call` with a fixture input),
+   validating the response. Write/destructive tools are never called.
+
+It emits a stable machine-readable JSON report plus a human-readable Markdown
+summary, and exits non-zero on any conformance/round-trip failure.
+
+```bash
+# Run the certifier against the offline fixture backend and write artifacts:
+npm run certify
+
+# CI entry points (also runnable locally):
+npm run test:certification            # gate: runs harness tests + certifier, exits non-zero on failure
+npm run test:certification:artifact   # evidence: writes artifacts, always exits 0
+```
+
+Artifacts are written to the package root as
+`mcp-certification-results.json` and `mcp-certification-results.md` (gitignored;
+uploaded by CI). To certify against a **live** honua-server, set `HONUA_BASE_URL`
+(and `HONUA_TRANSPORT`, `HONUA_MCP_SERVICE_ID`, `HONUA_MCP_LAYER_ID`); the
+harness then drives a real `HonuaClient` instead of the fixture.
+
+The standard schemas are vendored under
+`certification/geospatial-mcp-schemas/` (see that directory's `PROVENANCE.md`
+for the pinned source revision).
+
+## Transport-symmetric stdio proxy
+
+The honua server exposes one MCP catalog over streamable-HTTP/SSE at `/mcp`.
+Claude-Desktop-style clients speak **stdio**. Rather than reimplement that
+catalog (which is how the HTTP and stdio surfaces historically drifted apart),
+this package ships a **stdio proxy** (`honua-mcp-proxy`) that bridges a local
+stdio MCP client to the remote HTTP-SSE MCP server. It connects upstream as an
+MCP client and re-exposes the *same* catalog downstream over stdio — identical
+tools, identical input/output schemas, identical resources and prompts, and live
+`tools/list_changed` notifications. There is one source-of-truth catalog (the
+server's `/mcp`); the SDK proxies it, so the two transports are symmetric by
+construction. (Cross-repo: honua-io/honua-server#1950.)
+
+```bash
+# Bridge a stdio MCP client to a remote honua /mcp surface:
+HONUA_MCP_REMOTE_URL="https://demo.honua.io/mcp" honua-mcp-proxy
+```
+
+Environment variables:
+
+- `HONUA_MCP_REMOTE_URL` (required; alias `HONUA_MCP_URL`): the remote honua
+  `/mcp` endpoint to proxy.
+- `HONUA_MCP_AUTH_TOKEN` (optional): sent as `Authorization: Bearer <token>`.
+- `HONUA_API_KEY` (optional): sent as `x-api-key`.
+
+A parity test (`test/proxy.test.ts`) asserts the tool/resource/template catalog
+the downstream client sees is byte-identical to the upstream surface, that
+`tools/call` and resource reads round-trip identically, and that `list_changed`
+notifications are forwarded.
+
+## Cross-model workflow eval (provability)
+
+The package ships a **cross-model workflow eval** that proves the "any client →
+any workflow" claim: a held-out corpus of GIS workflows (`src/eval/corpus.ts`)
+is driven through the MCP surface by different client LLMs and graded for
+end-to-end success / clarification / edit rates per model. (Cross-repo:
+honua-io/honua-server#1956.)
+
+- **Deterministic control (offline, CI):** a scripted "ideal client" runs every
+  workflow's real `tools/call` round-trips against the offline fixture surface —
+  no model/API calls — and is graded identically to the live models. This is the
+  reproducible CI gate.
+- **Live cross-model (Claude + GPT):** when `ANTHROPIC_API_KEY` /
+  `OPENAI_API_KEY` are set, the Claude driver (latest Opus, `claude-opus-4-8`)
+  and the GPT driver (GA flagship, `gpt-5.5`, override with `OPENAI_MODEL`) run
+  the corpus through a real agentic tool-use loop over the identical catalog. The
+  `@anthropic-ai/sdk` / `openai` packages are imported dynamically (not
+  dependencies); keys come from the environment and are never hardcoded. Set
+  `HONUA_MCP_REMOTE_URL` to drive a live remote `/mcp` instead of the fixture.
+
+```bash
+npm run eval            # run the eval (live models join if their keys are set)
+npm run eval:offline    # force the deterministic control + fixture surface
+npm run test:eval       # gate: harness tests + offline eval, exits non-zero on failure
+npm run test:eval:artifact   # evidence: writes artifacts, always exits 0
+```
+
+Artifacts (`mcp-eval-results.json` / `mcp-eval-results.md`, gitignored, uploaded
+by CI) record the per-model scorecard. The CI gate asserts the deterministic
+control passes every scenario; live cross-model runs are recorded but
+informational.
+
+### Live lane (paid, manual only)
+
+The **operator corpus** (`src/eval/operator-corpus.ts`, 8 harder scenarios) is
+meant to run against a live honua-server **operator** `/mcp` surface with real
+models. This is a **billable** lane: it makes real Anthropic / OpenAI (or AWS
+Bedrock) calls. It never runs on push/PR — locally via `npm run eval:live`, or in
+CI via the manual **`MCP Live Cross-Model Eval`** workflow
+(`.github/workflows/mcp-eval-live.yml`, `workflow_dispatch`-only). The live model
+SDKs are not package dependencies; install them first for a live run:
+
+```bash
+npm install --no-save @anthropic-ai/sdk openai   # only needed for live runs
+```
+
+`eval:live` runs the operator corpus (`--corpus operator`) against the remote
+`/mcp` and always writes artifacts (`--artifact-only`). Authentication uses the
+same headers as the stdio proxy — **no dev-auth bypass**; the resolved auth mode
+(`bearer` / `api-key` / `anonymous`) is recorded in the artifact so the run
+proves it was authenticated. Set `HONUA_EVAL_REQUIRE_AUTH=1` to refuse an
+anonymous run outright.
+
+Shared env for every live run:
+
+- `HONUA_MCP_REMOTE_URL` (required): the operator `/mcp` endpoint, e.g. `https://demo.honua.io/mcp`.
+- `HONUA_MCP_AUTH_TOKEN`: sent as `Authorization: Bearer <token>` (preferred; ⇒ auth mode `bearer`).
+- `HONUA_API_KEY`: sent as `x-api-key` (⇒ auth mode `api-key`) when the deployment uses key auth instead.
+- `HONUA_EVAL_REQUIRE_AUTH=1` (recommended): fail fast if neither credential is present.
+
+```bash
+# Anthropic Claude (default claude-opus-4-8; override with HONUA_EVAL_ANTHROPIC_MODEL):
+HONUA_MCP_REMOTE_URL="https://demo.honua.io/mcp" \
+HONUA_MCP_AUTH_TOKEN="$HONUA_TOKEN" \
+HONUA_EVAL_REQUIRE_AUTH=1 \
+ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
+npm run eval:live -- --driver anthropic
+
+# OpenAI GPT (default gpt-5.5; override with OPENAI_MODEL, e.g. gpt-5.6-sol once you have access):
+HONUA_MCP_REMOTE_URL="https://demo.honua.io/mcp" \
+HONUA_MCP_AUTH_TOKEN="$HONUA_TOKEN" \
+HONUA_EVAL_REQUIRE_AUTH=1 \
+OPENAI_API_KEY="$OPENAI_API_KEY" \
+npm run eval:live -- --driver openai
+
+# Claude via Amazon Bedrock (AWS credential chain; default Sonnet 4.5,
+# override with HONUA_EVAL_BEDROCK_MODEL — most ids need a us.* inference profile):
+HONUA_MCP_REMOTE_URL="https://demo.honua.io/mcp" \
+HONUA_MCP_AUTH_TOKEN="$HONUA_TOKEN" \
+HONUA_EVAL_REQUIRE_AUTH=1 \
+HONUA_EVAL_BEDROCK=1 AWS_REGION=us-west-2 \
+HONUA_EVAL_BEDROCK_MODEL="us.anthropic.claude-sonnet-4-5-20250929-v1:0" \
+npm run eval:live -- --driver bedrock
+```
+
+Pass multiple drivers to compare them in one artifact:
+`npm run eval:live -- --driver anthropic,openai`. Every driver is graded against
+the **identical** operator corpus and catalog, and the artifact's
+`catalog.unresolvedRequiredTools` names any required tool the live surface did
+not advertise — so a scenario that fails does so for a real capability gap, not a
+silent tool name-resolution bug (the runner resolves required tools against the
+live `tools/list`, never against the vendored certification index).

package/certification/geospatial-mcp-schemas/PROVENANCE.md

@@ -0,0 +1,33 @@
+# Vendored geospatial-mcp JSON Schemas
+
+These JSON Schema files are a verbatim, vendored copy of the published
+machine-readable schemas from the open **geospatial-mcp** standard.
+
+- **Source repo:** https://github.com/honua-io/geospatial-mcp
+- **Source path:** `spec/schemas/`
+- **Source commit:** `54cbd49a6b8bf408024fe4d0316f79b8fa0534ab`
+  (`spec: align prose + served-family schemas to the honua-server /mcp grammar; harden strict manifest gate (#44)`)
+- **Schema index date:** `2026-07-01`
+- **Dialect:** JSON Schema draft 2020-12
+
+## Why vendored
+
+The MCP certification harness (`src/certification/`) must run deterministically
+in CI with **zero network access and zero model/API token spend**. Vendoring the
+published schemas pins the standard the Honua MCP surface is certified against to
+a known, reproducible revision and removes any cross-repo fetch at certify time.
+
+## Refreshing
+
+To re-pin to a newer published revision, re-copy `spec/schemas/` from the
+`geospatial-mcp` repo at the desired tag/commit and update the source commit and
+index date above. Do not hand-edit individual schema files; the standard is owned
+upstream.
+
+## Index
+
+`index.json` maps each standard tool's bare `standardName` (taxonomy.md name) to
+the `referenceToolName` the reference implementation (Honua `/mcp`) advertises,
+plus an `implementationStatus` (`implemented` | `known-gap`). The certifier reads
+this index to decide which advertised tools to conformance-check and which
+standard tools to record as known gaps.

package/certification/geospatial-mcp-schemas/README.md

@@ -0,0 +1,90 @@
+# Machine-Readable JSON Schemas
+
+**Status:** Draft
+**Date:** 2026-06-21
+**Scope:** Implementable JSON Schema (draft 2020-12) bindings for the geospatial MCP standard
+
+This directory makes the prose specification **implementable**. It publishes
+[JSON Schema](https://json-schema.org/) (draft 2020-12) documents describing:
+
+- each core MCP **tool** `inputSchema` (the argument shape an agent sends on a
+  `tools/call`), under [`tools/`](tools/);
+- each core MCP **resource** payload shape (the read-only projection a
+  `resources/read` returns), under [`resources/`](resources/).
+
+A machine-readable [`index.json`](index.json) maps every standard tool name and
+resource family URI to its schema file, so a harness can resolve a schema from
+a vocabulary key without hard-coding paths.
+
+A self-check **conformance fixture tree** lives under
+[`../../conformance/fixtures/`](../../conformance/fixtures/): example tool-call
+inputs and resource payloads that validate against these schemas. An
+implementer can run their JSON Schema validator over the fixtures to confirm the
+schemas load and that the examples conform, before testing their own server.
+
+## Relationship to the prose
+
+These schemas are **derived from**, and faithful to, the normative prose:
+
+- Tool families and names come from
+  [`spec/taxonomy.md` §MCP Tools to Workflow Family Mapping](../taxonomy.md#mcp-tools-to-workflow-family-mapping).
+- Resource families, URI grammar, and inspection fields come from
+  [`spec/resources.md`](../resources.md).
+- The canonical concept model (`AnalysisPlan`, `ArtifactRef`, `WorkspaceRef`,
+  `GeoprocessingError`, …) comes from
+  [`spec/taxonomy.md` §Canonical Concept Model](../taxonomy.md#canonical-concept-model).
+
+### Where the prose is intentionally upstream-owned
+
+The prose **defers concrete field spellings** for several canonical objects to
+`honua-server` (`MapPackage`, `AppPackage`, `PublishedService`, `Deployment`
+— see [`resources.md` §2.4 Deferred-Shape Fixtures](../conformance.md#24-deferred-shape-fixtures)
+and the responsibility-level projections in `resources.md`). For those, the
+prose deliberately gives **responsibilities**, not a frozen field table.
+
+Per the task of making the standard implementable, where the prose is ambiguous
+or upstream-owned, **these schemas adopt the shape the reference implementation
+(Honua's `/mcp`) already emits**, and mark it as such with a
+`x-honua-reference-shape: true` annotation and a `description` note. This keeps
+the standard constructible today while remaining honest that the concrete field
+names track the reference implementation until the upstream ticket lands.
+
+Resources whose concrete shape is still fully deferred upstream (e.g.
+`MapPackage`, `AppPackage`, `PublishedService`, `Deployment`,
+`PublishingResultPackage`) carry **responsibility-level** schemas: the stable
+identifier and `honua://` URI are pinned and required, additional properties are
+permitted (`additionalProperties: true`), and the responsibility fields are
+described but not over-constrained. This matches the conformance rubric, which
+scores those resources on stable identifiers, URI grammar, and
+responsibility-level projection while marking concrete-field assertions
+`not_applicable`.
+
+## Draft and dialect
+
+All schemas declare:
+
+```json
+"$schema": "https://json-schema.org/draft/2020-12/schema"
+```
+
+and a stable `$id` of the form
+`https://geospatial-mcp.honua.io/spec/schemas/{tools|resources}/{name}.schema.json`.
+
+## Tool-name prefixing
+
+The standard names tools without a vendor prefix (`plan_analysis`,
+`ground_candidates`, …; see `taxonomy.md`). A conformant server MAY advertise
+the tools under a vendor-namespaced name — the reference implementation
+advertises e.g. `honua_plan_analysis`. The `index.json` lists both the bare
+standard name and the reference implementation's advertised name so a harness
+can resolve either. The schemas validate the **argument object**, which is
+prefix-independent.
+
+## Files
+
+| File | Describes |
+|---|---|
+| [`index.json`](index.json) | Machine-readable tool/resource → schema map |
+| [`common/geoprocessing-error.schema.json`](common/geoprocessing-error.schema.json) | The canonical `GeoprocessingError` envelope (shared) |
+| [`tools/*.schema.json`](tools/) | One `inputSchema` per core tool family |
+| [`resources/*.schema.json`](resources/) | One payload schema per core resource family |

package/certification/geospatial-mcp-schemas/common/geoprocessing-error.schema.json

@@ -0,0 +1,45 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://geospatial-mcp.honua.io/spec/schemas/common/geoprocessing-error.schema.json",
+  "title": "GeoprocessingError",
+  "description": "Canonical error envelope reused verbatim by every MCP tool and resource failure path. See spec/resources.md §Error Model. No parallel MCP-local error taxonomy is permitted.",
+  "type": "object",
+  "required": ["kind", "message"],
+  "properties": {
+    "kind": {
+      "type": "string",
+      "description": "GeoprocessingErrorKind value.",
+      "enum": [
+        "ValidationFailed",
+        "AuthorizationDenied",
+        "UnknownDataset",
+        "UnknownProcess",
+        "ExecutionFailed",
+        "Timeout",
+        "Cancelled",
+        "OutputBindingFailed"
+      ]
+    },
+    "message": {
+      "type": "string",
+      "description": "Human-readable summary."
+    },
+    "stepId": {
+      "type": "string",
+      "description": "Plan step identifier when applicable."
+    },
+    "violations": {
+      "type": "array",
+      "description": "Structured validation failures.",
+      "items": {
+        "type": "object",
+        "required": ["code", "message"],
+        "properties": {
+          "code": { "type": "string" },
+          "message": { "type": "string" },
+          "fieldPath": { "type": "string" }
+        }
+      }
+    }
+  }
+}

package/certification/geospatial-mcp-schemas/conformance/manifest.schema.json

@@ -0,0 +1,73 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://geospatial-mcp.honua.io/spec/schemas/conformance/manifest.schema.json",
+  "title": "geospatial-mcp conformance manifest",
+  "description": "Static, server-emitted declaration of the MCP tool and resource surface an implementation advertises, used to check coverage of the geospatial-mcp standard vocabulary without running a live scenario harness. A conformant server produces this manifest from its own tools/list and resources/templates/list capability advertisement. See /CONFORMANCE.md and spec/conformance.md.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["specDate", "implementation", "tools", "resources"],
+  "properties": {
+    "specDate": {
+      "type": "string",
+      "description": "The Date header of the spec/conformance.md document this manifest was produced against (e.g. 2026-04-19), per spec/conformance.md §8 'Conformance spec identifier per run'.",
+      "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$"
+    },
+    "implementation": {
+      "type": "object",
+      "description": "Identity of the implementation under test.",
+      "additionalProperties": true,
+      "required": ["name"],
+      "properties": {
+        "name": { "type": "string", "description": "Implementation name (e.g. 'Honua /mcp')." },
+        "version": { "type": "string", "description": "Optional implementation version or build identifier." },
+        "isReferenceImplementation": {
+          "type": "boolean",
+          "description": "True only for the reference implementation named in /CONFORMANCE.md (Honua). Other implementations omit this or set it false."
+        }
+      }
+    },
+    "tools": {
+      "type": "array",
+      "description": "The tools the implementation advertises that map onto standard vocabulary.",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["standardName", "advertisedName"],
+        "properties": {
+          "standardName": {
+            "type": "string",
+            "description": "The bare taxonomy.md tool name this advertised tool implements (e.g. 'plan_analysis'). MUST match a tools[].standardName in spec/schemas/index.json."
+          },
+          "advertisedName": {
+            "type": "string",
+            "description": "The name the server advertises on tools/list (e.g. 'honua_plan_analysis'). MAY be vendor-prefixed."
+          },
+          "workflowFamily": {
+            "type": "string",
+            "description": "Optional workflow-family tag the server attaches to the tool, for coverage reporting."
+          }
+        }
+      }
+    },
+    "resources": {
+      "type": "array",
+      "description": "The resource families the implementation advertises that map onto standard vocabulary.",
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["family", "uriForm"],
+        "properties": {
+          "family": {
+            "type": "string",
+            "description": "Resource family name. MUST match a resources[].family in spec/schemas/index.json."
+          },
+          "uriForm": {
+            "type": "string",
+            "description": "The honua:// URI template the server advertises for the family (e.g. honua://results/{result_package_id}).",
+            "pattern": "^honua://"
+          }
+        }
+      }
+    }
+  }
+}