@cyanheads/mcp-ts-core 0.9.16 → 0.9.18

package/skills/api-mirror/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: api-mirror
+description: >
+  Stand up a persistent, self-refreshing local mirror of a bulk upstream dataset with the MirrorService (@cyanheads/mcp-ts-core/mirror). Use when a server wraps a large or slow API and should query a synced local index (embedded SQLite + FTS5) instead of paginating the live API per request.
+metadata:
+  author: cyanheads
+  version: "1.0"
+  audience: external
+  type: reference
+---
+
+## Context
+
+The MirrorService owns the source-agnostic half of a local mirror — the embedded store, the sync-state machine, the runner — so a server supplies only the two parts that are irreducibly per-source: the **ingester** (a `sync` generator) and the **schema**. It targets the embedded-SQLite tier (~10⁴–10⁷ rows). Node/Bun only: `bun:sqlite` is built-in on Bun, `better-sqlite3` is an optional peer dependency on Node; the store is unavailable on Workers (no SQLite, no persistent filesystem).
+
+Import from `@cyanheads/mcp-ts-core/mirror`.
+
+## The shape
+
+```ts
+import { defineMirror, sqliteMirrorStore } from '@cyanheads/mcp-ts-core/mirror';
+
+const papers = defineMirror({
+  name: 'arxiv-papers',
+  store: sqliteMirrorStore({
+    path: config.mirrorPath,
+    primaryKey: 'id',
+    columns: { id: 'TEXT', title: 'TEXT', authors: 'TEXT', abstract: 'TEXT', updated: 'TEXT' },
+    fts: ['title', 'authors', 'abstract'],          // opt-in FTS5 external-content index
+    indexes: [{ columns: ['updated'] }],
+  }),
+  // The ingester — the one part that is always server-specific.
+  async *sync({ mode, cursor, checkpoint, signal }) {
+    for await (const page of harvestPages({ resumeFrom: cursor, since: checkpoint, signal })) {
+      yield {
+        records: page.rows,             // objects keyed by declared column
+        tombstones: page.deletedIds,    // primary-key values to delete
+        cursor: page.token,             // volatile resume position (see below)
+        checkpoint: page.maxStamp,      // durable high-water mark (see below)
+      };
+    }
+  },
+});
+
+await papers.runSync({ mode: 'init', signal: AbortSignal.timeout(3_600_000) }); // full; resumes on interrupt
+await papers.runSync({ mode: 'refresh' });                                       // incremental
+const { rows, total } = await papers.query({ match: 'transformers', limit: 10, offset: 0 });
+const status = await papers.status();   // { status, ready, checkpoint, total, ... }
+```
+
+## cursor vs. checkpoint — the core distinction
+
+Two resume dimensions, deliberately separate. Conflating them silently corrupts resume for token-paged sources.
+
+| | `cursor` | `checkpoint` |
+|---|---|---|
+| Meaning | Volatile intra-run resume position (e.g. an OAI-PMH resumption token, a page token) | Durable incremental high-water mark (e.g. the max record datestamp) |
+| Lifetime | One run; may expire; **cleared on completion** | Persists; **advances monotonically, only on success** |
+| Used for | Resuming an interrupted `init` | Seeding the next `refresh` |
+
+Why they can't merge: during a from-scratch init the records aren't ordered by the high-water field, so the max-so-far is not a valid resume position — only the cursor is. After a completed init the cursor is meaningless, but the high-water mark is the correct refresh seed. The framework persists both per page and threads the right one back into `sync()` per mode. **The checkpoint must be lexicographically monotonic** (ISO 8601 works); the runner advances the stored checkpoint only when a page's value compares greater.
+
+## What you own vs. what the framework owns
+
+| Framework | Server |
+|---|---|
+| Cross-runtime SQLite handle, WAL + `busy_timeout` | The `sync` generator (the ingester) |
+| `mirror_sync_state` + cursor/checkpoint state machine | Translating your query syntax → FTS5 `match` |
+| `runSync({ init \| refresh })`, per-page persist, resume | Mapping upstream records → row objects |
+| Schema gen (columns + FTS + tokenizer + triggers) | Migration *content* (the `up` functions) |
+| `schema_version` + migration *runner* | Scheduling + init/refresh bootstrap (see below) |
+| Generic `query()` + the raw-handle escape hatch | Server-specific access paths via the raw handle |
+
+## Querying
+
+`query({ match?, filters?, sort?, limit, offset })` covers the common case:
+
+- `match` — an FTS5 `MATCH` expression (only when the store declares `fts` columns). Translate your own query grammar to FTS5 before calling.
+- `filters` — `[{ column, op, value }]`, AND-combined, over declared columns. `op` ∈ `eq|ne|gt|gte|lt|lte|in` (`in` takes an array).
+- `sort` — `{ column, direction }` or `'relevance'` (FTS bm25; requires `match`). Defaults to insertion order.
+
+For access paths the generic query can't express — junction tables for index-backed multi-value filtering, denormalized counters, bespoke `bm25` weighting — use the **raw handle**: `const db = await mirror.raw();` then run prepared statements against your own auxiliary tables (declare them via a migration). Add the auxiliary DDL in a `migrations` step; maintain it from your `sync` mapping or SQL triggers.
+
+## Readiness — key off the completion marker, not live status
+
+`status().ready` is `true` once a full sync has **ever completed** (`completedAt != null`), not when `status === 'complete'`. The dataset stays transactionally queryable during a refresh, so a mirror mid-refresh — or one whose last refresh failed — is still ready and should keep serving. Gate the mirror read path on `await mirror.ready()`; fall back to the live API only when it is `false` (cold, never-completed init).
+
+## Scheduling and bootstrap (server-owned)
+
+The service owns `runSync` + state; it does not schedule. Wire "self-refreshing" yourself:
+
+- **Refresh** — register `runSync({ mode: 'refresh' })` on a cron via `schedulerService` from `@cyanheads/mcp-ts-core/utils`, inside `setup()`. Gate on transport (HTTP) when stdio operators run it out-of-band.
+- **Init** — run out-of-band (a CLI script / one-shot), never on startup: a full init can take hours and must not block the server. It is idempotent and resumable — re-running after an interrupt continues from the persisted cursor.
+
+## Checklist
+
+- [ ] `defineMirror({ name, store, sync })`; the server holds the instance (one per mirror)
+- [ ] `sqliteMirrorStore` spec declares `primaryKey`, `columns`, and (if searching) `fts`
+- [ ] `sync` yields `{ records, tombstones?, cursor?, checkpoint? }` per page; checkpoint is lexicographically monotonic
+- [ ] Read path gated on `await mirror.ready()` with a live fallback when not ready
+- [ ] `better-sqlite3` added as a peer dependency for Node deployments; mirror disabled on Workers
+- [ ] Refresh wired via `schedulerService` in `setup()`; init runs out-of-band
+- [ ] `bun run devcheck` passes

package/skills/design-mcp-server/SKILL.md

@@ -4,7 +4,7 @@ description: >
   Design the tool surface, resources, and service layer for a new MCP server. Use when starting a new server, planning a major feature expansion, or when the user describes a domain/API they want to expose via MCP. Produces a design doc at docs/design.md that drives implementation.
 metadata:
   author: cyanheads
-  version: "2.14"
+  version: "2.15"
   audience: external
   type: workflow
 ---
@@ -350,6 +350,7 @@ output: z.object({
 
 - **Truncate large output with counts.** When a list exceeds a reasonable display size, show the top N and append "...and X more". Don't silently drop results.
 - **Spill big tabular results to a queryable surface.** When a tool's row set can exceed any reasonable context budget — paginated APIs, streamed exports, big query results — pair an inline preview with a `DataCanvas` table holding the full set, returned as a token the agent can SQL. Compute distributions or refinement hints across the full result, not the preview, so aggregate signal stays honest. See `api-canvas` for the `spillover()` helper.
+- **Mirror a bulk upstream instead of paginating it live.** When the server wraps a large or slow API whose corpus is queried far more than it changes, sync it once into a persistent local index and query that as the primary data path — not the live API per request. Match the backend to corpus size: ≲ tens of thousands of rows → an in-memory index (server-level, no primitive); ~10⁴–10⁷ → the `MirrorService` (embedded SQLite + FTS5; declare a schema + a `sync` ingester via `defineMirror`/`sqliteMirrorStore`, then `runSync`/`query`, see `api-mirror`); ≳ 10⁸ → an external store. Distinct lifecycle from DataCanvas: a mirror is long-lived and cross-session, refreshed on a schedule; canvas is ephemeral and per-session.
 - **`format()` is the markdown twin of `structuredContent` — make both content-complete.** Different MCP clients forward different surfaces to the model: some (e.g., Claude Code) read `structuredContent` from `output`, others (e.g., Claude Desktop) read `content[]` from `format()`. Both must carry the same data so every client sees the same picture — `format()` just dresses it up with markdown. A thin `format()` that returns only a count or title leaves `content[]`-only clients blind to data that `structuredContent` clients can see. Render all fields the LLM needs, with structured markdown (headers, bold labels, lists) for readability.
 - **Agent-facing context must reach both client surfaces — put it in `enrichment`.** `structuredContent` (from `output`) and `content[]` (from `format()`) are read by different clients. Empty-result notices, the query/filter as the server parsed it, and pagination totals — the context the agent *reasons with*, distinct from the domain payload — reach only `content[]` if hand-authored into `format()` text alone, leaving `structuredContent`-only clients (Claude Code) blind. (The reverse can't happen: `format-parity` drags every `output` field into `format()`, so `output`-authored context already reaches both.) An `enrichment` block — the success-path counterpart to `errors[]`, populated via `ctx.enrich(...)` — reaches both automatically: merged into `structuredContent`, advertised as `output.extend(enrichment)`, mirrored into a `content[]` trailer, no `format()` entry needed. How each field renders in that trailer is a per-tool call — a kind-tag (`notice`/`total`/`echo`/`delta`) when a canonical form fits, a domain key like `totalFound` otherwise, and an `enrichmentTrailer.render` for any structured (object/array) field so it doesn't ship as a JSON blob. See `add-tool`'s **Tool Response Design**.
 

package/dist/logs/combined.log

@@ -1,4 +0,0 @@
-{"level":40,"time":1780111591031,"env":"testing","version":"0.9.16","pid":58703,"transport":"http","requestId":"OIXA0-I9W20","timestamp":"2026-05-30T03:26:31.030Z","operation":"TransportManager.start","component":"HttpTransportSetup","msg":"MCP_ALLOWED_ORIGINS is not set — CORS is wildcard for CLI clients; browser Origin headers are restricted to loopback. Set MCP_ALLOWED_ORIGINS for production deployments accepting remote browser origins."}
-{"level":40,"time":1780111592750,"env":"testing","version":"0.9.16","pid":58703,"transport":"http","requestId":"OIXA0-I9W20","timestamp":"2026-05-30T03:26:31.030Z","operation":"TransportManager.start","component":"HttpTransportSetup","sessionId":"not-a-real-session-1780111592750","msg":"Session validation failed - invalid or hijacked session"}
-{"level":50,"time":1780111596553,"env":"testing","version":"0.0.0-test","pid":58776,"requestId":"P4PZW-06ZVZ","timestamp":"2026-05-30T03:26:36.552Z","operation":"HandleToolRequest","critical":false,"errorCode":-32005,"originalErrorType":"McpError","finalErrorType":"McpError","sessionId":"aeda50c721cd025136a1fe6ea032a77dee8f6213a87e45fcc702844bebf9347c","toolName":"scoped_echo","tenantId":"authz-tenant","auth":{"sub":"authz-user","scopes":["tool:other:read"],"clientId":"authz-client","tenantId":"authz-tenant","token":"[REDACTED]"},"errorData":{"sessionId":"aeda50c721cd025136a1fe6ea032a77dee8f6213a87e45fcc702844bebf9347c","toolName":"scoped_echo","requestId":"P4PZW-06ZVZ","timestamp":"2026-05-30T03:26:36.552Z","tenantId":"authz-tenant","operation":"HandleToolRequest","auth":{"sub":"authz-user","scopes":["tool:other:read"],"clientId":"authz-client","tenantId":"authz-tenant","token":"[REDACTED]"},"originalErrorName":"McpError","originalMessage":"Insufficient permissions.","originalStack":"McpError: Insufficient permissions.\n    at forbidden (/Users/casey/Developer/github/mcp-ts-core/dist/types-global/errors.js:84:58)\n    at withRequiredScopes (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/transports/auth/lib/authUtils.js:68:15)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/tools/utils/toolHandlerFactory.js:251:17)\n    at executeToolHandler (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:231:34)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:126:43)\n    at processTicksAndRejections (native:7:39)"},"stack":"McpError: Insufficient permissions.\n    at handleError (/Users/casey/Developer/github/mcp-ts-core/dist/utils/internal/error-handler/errorHandler.js:170:23)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/tools/utils/toolHandlerFactory.js:293:26)\n    at executeToolHandler (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:231:34)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:126:43)\n    at processTicksAndRejections (native:7:39)","msg":"Error in tool:scoped_echo: Insufficient permissions."}
-{"level":50,"time":1780111596560,"env":"testing","version":"0.0.0-test","pid":58776,"requestId":"MPTQ5-293E4","timestamp":"2026-05-30T03:26:36.560Z","operation":"HandleToolRequest","critical":false,"errorCode":-32005,"originalErrorType":"McpError","finalErrorType":"McpError","sessionId":"875421efd7fea0e9c733d54859c224a7844470f77beda25e25fbc9ffb79a7f56","toolName":"scoped_echo","tenantId":"authz-tenant","auth":{"sub":"authz-user","scopes":["openid","email","profile","offline_access"],"clientId":"authz-client","tenantId":"authz-tenant","token":"[REDACTED]"},"errorData":{"sessionId":"875421efd7fea0e9c733d54859c224a7844470f77beda25e25fbc9ffb79a7f56","toolName":"scoped_echo","requestId":"MPTQ5-293E4","timestamp":"2026-05-30T03:26:36.560Z","tenantId":"authz-tenant","operation":"HandleToolRequest","auth":{"sub":"authz-user","scopes":["openid","email","profile","offline_access"],"clientId":"authz-client","tenantId":"authz-tenant","token":"[REDACTED]"},"originalErrorName":"McpError","originalMessage":"Insufficient permissions.","originalStack":"McpError: Insufficient permissions.\n    at forbidden (/Users/casey/Developer/github/mcp-ts-core/dist/types-global/errors.js:84:58)\n    at withRequiredScopes (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/transports/auth/lib/authUtils.js:68:15)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/tools/utils/toolHandlerFactory.js:251:17)\n    at executeToolHandler (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:231:34)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:126:43)\n    at processTicksAndRejections (native:7:39)"},"stack":"McpError: Insufficient permissions.\n    at handleError (/Users/casey/Developer/github/mcp-ts-core/dist/utils/internal/error-handler/errorHandler.js:170:23)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/tools/utils/toolHandlerFactory.js:293:26)\n    at executeToolHandler (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:231:34)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:126:43)\n    at processTicksAndRejections (native:7:39)","msg":"Error in tool:scoped_echo: Insufficient permissions."}

package/dist/logs/error.log

@@ -1,2 +0,0 @@
-{"level":50,"time":1780111596553,"env":"testing","version":"0.0.0-test","pid":58776,"requestId":"P4PZW-06ZVZ","timestamp":"2026-05-30T03:26:36.552Z","operation":"HandleToolRequest","critical":false,"errorCode":-32005,"originalErrorType":"McpError","finalErrorType":"McpError","sessionId":"aeda50c721cd025136a1fe6ea032a77dee8f6213a87e45fcc702844bebf9347c","toolName":"scoped_echo","tenantId":"authz-tenant","auth":{"sub":"authz-user","scopes":["tool:other:read"],"clientId":"authz-client","tenantId":"authz-tenant","token":"[REDACTED]"},"errorData":{"sessionId":"aeda50c721cd025136a1fe6ea032a77dee8f6213a87e45fcc702844bebf9347c","toolName":"scoped_echo","requestId":"P4PZW-06ZVZ","timestamp":"2026-05-30T03:26:36.552Z","tenantId":"authz-tenant","operation":"HandleToolRequest","auth":{"sub":"authz-user","scopes":["tool:other:read"],"clientId":"authz-client","tenantId":"authz-tenant","token":"[REDACTED]"},"originalErrorName":"McpError","originalMessage":"Insufficient permissions.","originalStack":"McpError: Insufficient permissions.\n    at forbidden (/Users/casey/Developer/github/mcp-ts-core/dist/types-global/errors.js:84:58)\n    at withRequiredScopes (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/transports/auth/lib/authUtils.js:68:15)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/tools/utils/toolHandlerFactory.js:251:17)\n    at executeToolHandler (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:231:34)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:126:43)\n    at processTicksAndRejections (native:7:39)"},"stack":"McpError: Insufficient permissions.\n    at handleError (/Users/casey/Developer/github/mcp-ts-core/dist/utils/internal/error-handler/errorHandler.js:170:23)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/tools/utils/toolHandlerFactory.js:293:26)\n    at executeToolHandler (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:231:34)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:126:43)\n    at processTicksAndRejections (native:7:39)","msg":"Error in tool:scoped_echo: Insufficient permissions."}
-{"level":50,"time":1780111596560,"env":"testing","version":"0.0.0-test","pid":58776,"requestId":"MPTQ5-293E4","timestamp":"2026-05-30T03:26:36.560Z","operation":"HandleToolRequest","critical":false,"errorCode":-32005,"originalErrorType":"McpError","finalErrorType":"McpError","sessionId":"875421efd7fea0e9c733d54859c224a7844470f77beda25e25fbc9ffb79a7f56","toolName":"scoped_echo","tenantId":"authz-tenant","auth":{"sub":"authz-user","scopes":["openid","email","profile","offline_access"],"clientId":"authz-client","tenantId":"authz-tenant","token":"[REDACTED]"},"errorData":{"sessionId":"875421efd7fea0e9c733d54859c224a7844470f77beda25e25fbc9ffb79a7f56","toolName":"scoped_echo","requestId":"MPTQ5-293E4","timestamp":"2026-05-30T03:26:36.560Z","tenantId":"authz-tenant","operation":"HandleToolRequest","auth":{"sub":"authz-user","scopes":["openid","email","profile","offline_access"],"clientId":"authz-client","tenantId":"authz-tenant","token":"[REDACTED]"},"originalErrorName":"McpError","originalMessage":"Insufficient permissions.","originalStack":"McpError: Insufficient permissions.\n    at forbidden (/Users/casey/Developer/github/mcp-ts-core/dist/types-global/errors.js:84:58)\n    at withRequiredScopes (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/transports/auth/lib/authUtils.js:68:15)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/tools/utils/toolHandlerFactory.js:251:17)\n    at executeToolHandler (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:231:34)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:126:43)\n    at processTicksAndRejections (native:7:39)"},"stack":"McpError: Insufficient permissions.\n    at handleError (/Users/casey/Developer/github/mcp-ts-core/dist/utils/internal/error-handler/errorHandler.js:170:23)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/dist/mcp-server/tools/utils/toolHandlerFactory.js:293:26)\n    at executeToolHandler (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:231:34)\n    at <anonymous> (/Users/casey/Developer/github/mcp-ts-core/node_modules/@modelcontextprotocol/sdk/dist/esm/server/mcp.js:126:43)\n    at processTicksAndRejections (native:7:39)","msg":"Error in tool:scoped_echo: Insufficient permissions."}