proctor-mcp-server 0.1.0

package/README.md

@@ -0,0 +1,247 @@
+# Proctor MCP Server
+
+Haven't heard about MCP yet? The easiest way to keep up-to-date is to read our [weekly newsletter at PulseMCP](https://www.pulsemcp.com/).
+
+---
+
+This is an MCP ([Model Context Protocol](https://modelcontextprotocol.io/)) Server for running Proctor exams against MCP servers. It provides tools for executing tests, managing exam infrastructure, and tracking test results through direct integration with the [PulseMCP Proctor API](https://admin.pulsemcp.com).
+
+**Note**: This is an internal tool for the PulseMCP team. The source code is public for reference purposes, but the server requires API keys that are not publicly available.
+
+# Table of Contents
+
+- [Highlights](#highlights)
+- [Capabilities](#capabilities)
+- [Tool Groups](#tool-groups)
+- [Usage Tips](#usage-tips)
+- [Examples](#examples)
+- [Setup](#setup)
+  - [Cheatsheet](#cheatsheet)
+  - [Claude Desktop](#claude-desktop)
+    - [Manual Setup](#manual-setup)
+
+# Highlights
+
+**Exam Execution**: Run Proctor exams against MCP servers to test functionality and protocol compliance.
+
+**Result Management**: Save and retrieve exam results for comparison and regression testing.
+
+**Infrastructure Control**: List, manage, and clean up Fly.io machines used for exam execution.
+
+**Tool Groups**: Enable/disable tool groups via `TOOL_GROUPS` environment variable. Each group has a base variant (full access) and a `_readonly` variant (read-only access).
+
+**Streaming Results**: Real-time exam execution logs with NDJSON streaming.
+
+# Capabilities
+
+This server is built and tested on macOS with Claude Desktop. It should work with other MCP clients as well.
+
+| Tool Name              | Tool Group | Read/Write | Description                                              |
+| ---------------------- | ---------- | ---------- | -------------------------------------------------------- |
+| `get_proctor_metadata` | exams      | read       | Get available runtimes and exams for Proctor testing.    |
+| `run_exam`             | exams      | write      | Execute a Proctor exam against an MCP server.            |
+| `save_result`          | exams      | write      | Save exam results to the database for future comparison. |
+| `get_prior_result`     | exams      | read       | Retrieve a previous exam result for comparison.          |
+| `get_machines`         | machines   | read       | List active Fly.io machines used for Proctor exams.      |
+| `destroy_machine`      | machines   | write      | Delete a Fly.io machine.                                 |
+| `cancel_exam`          | machines   | write      | Cancel a running Proctor exam.                           |
+
+# Tool Groups
+
+This server organizes tools into groups that can be selectively enabled or disabled. Each group has two variants:
+
+- **Base group** (e.g., `exams`): Full read + write access
+- **Readonly group** (e.g., `exams_readonly`): Read-only access
+
+## Available Groups
+
+| Group               | Tools | Description                            |
+| ------------------- | ----- | -------------------------------------- |
+| `exams`             | 4     | Full exam execution (read + write)     |
+| `exams_readonly`    | 2     | Exam metadata and results (read only)  |
+| `machines`          | 3     | Full machine management (read + write) |
+| `machines_readonly` | 1     | Machine listing (read only)            |
+
+### Tools by Group
+
+- **exams** / **exams_readonly**:
+  - Read-only: `get_proctor_metadata`, `get_prior_result`
+  - Write: `run_exam`, `save_result`
+- **machines** / **machines_readonly**:
+  - Read-only: `get_machines`
+  - Write: `destroy_machine`, `cancel_exam`
+
+## Environment Variables
+
+| Variable          | Description                                 | Default                       |
+| ----------------- | ------------------------------------------- | ----------------------------- |
+| `PROCTOR_API_KEY` | API key for PulseMCP Proctor API (required) | -                             |
+| `PROCTOR_API_URL` | Base URL for Proctor API                    | `https://admin.pulsemcp.com`  |
+| `TOOL_GROUPS`     | Comma-separated list of enabled tool groups | `exams,machines` (all groups) |
+
+## Examples
+
+Enable all tools with full access (default):
+
+```bash
+# No TOOL_GROUPS needed - all base groups enabled
+```
+
+Enable only exam tools:
+
+```bash
+TOOL_GROUPS=exams
+```
+
+Enable machines with read-only access:
+
+```bash
+TOOL_GROUPS=machines_readonly
+```
+
+Enable all groups with read-only access:
+
+```bash
+TOOL_GROUPS=exams_readonly,machines_readonly
+```
+
+Mix full and read-only access per group:
+
+```bash
+# Full exam access, read-only machines
+TOOL_GROUPS=exams,machines_readonly
+```
+
+# Usage Tips
+
+- Use `get_proctor_metadata` first to discover available runtimes and exam types
+- The `mcp_config` parameter for `run_exam` must be a valid JSON string
+- Save results with `save_result` to enable future comparisons with `get_prior_result`
+- Use `get_machines` to monitor active exam infrastructure
+- Clean up machines with `destroy_machine` when no longer needed
+- Use `cancel_exam` to stop a stuck or slow exam before destroying the machine
+
+# Examples
+
+## Get Available Exams
+
+```
+User: What exams can I run with Proctor?
+Assistant: I'll check what exams are available.
+
+[Calls get_proctor_metadata]
+
+Here are the available exams:
+
+**Runtimes:**
+- Proctor v0.0.37 (id: v0.0.37)
+
+**Exams:**
+- Auth Check (id: proctor-mcp-client-auth-check) - Verifies authentication mechanisms
+- Init Tools List (id: proctor-mcp-client-init-tools-list) - Tests initialization and tool listing
+```
+
+## Run an Exam
+
+```
+User: Run the init-tools-list exam against my MCP server at https://example.com/mcp
+Assistant: I'll run the exam against your server.
+
+[Calls run_exam with runtime_id="v0.0.37", exam_id="proctor-mcp-client-init-tools-list", mcp_config=...]
+
+The exam completed successfully. Here are the results:
+
+**Status:** Success
+**Tests Passed:** 5/5
+- Initialization: Passed
+- Tool listing: Passed
+- Tool execution: Passed
+...
+```
+
+## Monitor Infrastructure
+
+```
+User: What machines are currently running?
+Assistant: I'll check the active Fly machines.
+
+[Calls get_machines]
+
+There are 2 active machines:
+
+1. **machine-abc123** - Running in sjc region (created 10 minutes ago)
+2. **machine-def456** - Stopped in iad region (created 1 hour ago)
+```
+
+# Setup
+
+## Cheatsheet
+
+Quick setup:
+
+```bash
+# Install dependencies
+npm run install-all
+
+# Build the server
+npm run build
+
+# Set your API key
+export PROCTOR_API_KEY="your-api-key-here"
+
+# Run the server
+cd local && npm start
+```
+
+## Claude Desktop
+
+Add to your Claude Desktop configuration:
+
+### macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
+
+### Windows: `%APPDATA%\Claude\claude_desktop_config.json`
+
+```json
+{
+  "mcpServers": {
+    "proctor": {
+      "command": "node",
+      "args": ["/path/to/proctor/local/build/index.js"],
+      "env": {
+        "PROCTOR_API_KEY": "your-api-key-here",
+        "TOOL_GROUPS": "exams,machines"
+      }
+    }
+  }
+}
+```
+
+For read-only access:
+
+```json
+{
+  "mcpServers": {
+    "proctor-readonly": {
+      "command": "node",
+      "args": ["/path/to/proctor/local/build/index.js"],
+      "env": {
+        "PROCTOR_API_KEY": "your-api-key-here",
+        "TOOL_GROUPS": "exams_readonly,machines_readonly"
+      }
+    }
+  }
+}
+```
+
+### Manual Setup
+
+If you prefer to run the server manually:
+
+```bash
+cd /path/to/proctor/local
+PROCTOR_API_KEY="your-api-key-here" node build/index.js
+```
+
+## License
+
+MIT

package/build/index.integration-with-mock.js

@@ -0,0 +1,143 @@
+#!/usr/bin/env node
+/**
+ * Integration test entry point with mock client
+ * This file is used for testing the MCP server with mocked external API calls
+ */
+import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
+import { createMCPServer, logServerStart, logError } from '../shared/index.js';
+/**
+ * Integration mock implementation of IProctorClient
+ */
+class IntegrationMockProctorClient {
+    priorResults = new Map();
+    savedResultId = 0;
+    async getMetadata() {
+        return {
+            runtimes: [
+                {
+                    id: 'v0.0.37',
+                    name: 'Proctor v0.0.37',
+                    image: 'registry.fly.io/proctor:v0.0.37',
+                },
+                {
+                    id: 'v0.0.36',
+                    name: 'Proctor v0.0.36',
+                    image: 'registry.fly.io/proctor:v0.0.36',
+                },
+            ],
+            exams: [
+                {
+                    id: 'proctor-mcp-client-auth-check',
+                    name: 'Auth Check',
+                    description: 'Verifies authentication mechanisms',
+                },
+                {
+                    id: 'proctor-mcp-client-init-tools-list',
+                    name: 'Init Tools List',
+                    description: 'Tests initialization and tool listing',
+                },
+            ],
+        };
+    }
+    async *runExam(params) {
+        yield {
+            type: 'log',
+            data: { time: new Date().toISOString(), message: 'Starting exam...' },
+        };
+        yield {
+            type: 'log',
+            data: { time: new Date().toISOString(), message: `Using runtime: ${params.runtime_id}` },
+        };
+        yield {
+            type: 'log',
+            data: { time: new Date().toISOString(), message: `Running exam: ${params.exam_id}` },
+        };
+        yield {
+            type: 'log',
+            data: { time: new Date().toISOString(), message: 'Exam completed successfully' },
+        };
+        yield {
+            type: 'result',
+            data: {
+                status: 'success',
+                input: {
+                    'mcp.json': JSON.parse(params.mcp_config),
+                },
+                tests: [
+                    { name: 'initialization', passed: true },
+                    { name: 'tool_listing', passed: true },
+                ],
+            },
+        };
+    }
+    async saveResult(params) {
+        this.savedResultId++;
+        const result = {
+            success: true,
+            id: this.savedResultId,
+        };
+        const key = `${params.mirror_id}-${params.exam_id}`;
+        this.priorResults.set(key, {
+            id: this.savedResultId,
+            datetime_performed: new Date().toISOString(),
+            results: typeof params.results === 'string' ? JSON.parse(params.results) : params.results,
+            runtime_image: params.runtime_id,
+            match_type: 'exact',
+        });
+        return result;
+    }
+    async getPriorResult(params) {
+        const key = `${params.mirror_id}-${params.exam_id}`;
+        const result = this.priorResults.get(key);
+        if (!result) {
+            throw new Error('No prior result found');
+        }
+        return result;
+    }
+    async getMachines() {
+        return {
+            machines: [
+                {
+                    id: 'machine-123',
+                    name: 'proctor-exam-1',
+                    state: 'running',
+                    region: 'sjc',
+                    created_at: '2024-01-15T10:30:00Z',
+                },
+                {
+                    id: 'machine-456',
+                    name: 'proctor-exam-2',
+                    state: 'stopped',
+                    region: 'iad',
+                    created_at: '2024-01-15T09:00:00Z',
+                },
+            ],
+        };
+    }
+    async destroyMachine(_machineId) {
+        return { success: true };
+    }
+    async cancelExam(_params) {
+        return {
+            success: true,
+            message: 'Exam cancelled successfully',
+        };
+    }
+}
+async function main() {
+    // Create server using factory
+    const { server, registerHandlers } = createMCPServer();
+    // Create mock client for testing
+    const mockClient = new IntegrationMockProctorClient();
+    // Register all handlers with mock client
+    await registerHandlers(server, () => mockClient);
+    // Start server
+    const transport = new StdioServerTransport();
+    await server.connect(transport);
+    logServerStart('proctor-mcp-server (integration-mock)');
+}
+// Run the server
+main().catch((error) => {
+    logError('main', error);
+    process.exit(1);
+});

package/build/index.js

@@ -0,0 +1,57 @@
+#!/usr/bin/env node
+import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
+import { createMCPServer } from '../shared/index.js';
+import { logServerStart, logError } from '../shared/logging.js';
+// Validate required environment variables before starting
+function validateEnvironment() {
+    const required = [
+        {
+            name: 'PROCTOR_API_KEY',
+            description: 'API key for PulseMCP Proctor API authentication',
+        },
+    ];
+    const optional = [
+        {
+            name: 'PROCTOR_API_URL',
+            description: 'Base URL for Proctor API (default: https://admin.pulsemcp.com)',
+        },
+        {
+            name: 'TOOL_GROUPS',
+            description: 'Comma-separated list of enabled tool groups (default: all)',
+        },
+    ];
+    const missing = required.filter(({ name }) => !process.env[name]);
+    if (missing.length > 0) {
+        logError('validateEnvironment', 'Missing required environment variables:');
+        missing.forEach(({ name, description }) => {
+            console.error(`  - ${name}: ${description}`);
+        });
+        if (optional.length > 0) {
+            console.error('\nOptional environment variables:');
+            optional.forEach(({ name, description }) => {
+                console.error(`  - ${name}: ${description}`);
+            });
+        }
+        console.error('\nPlease set the required environment variables and try again.');
+        console.error('Example:');
+        console.error('  export PROCTOR_API_KEY="your-api-key"');
+        process.exit(1);
+    }
+}
+async function main() {
+    // Validate environment variables first
+    validateEnvironment();
+    // Create server using factory
+    const { server, registerHandlers } = createMCPServer();
+    // Register all handlers (resources and tools)
+    await registerHandlers(server);
+    // Start server
+    const transport = new StdioServerTransport();
+    await server.connect(transport);
+    logServerStart('proctor-mcp-server');
+}
+// Run the server
+main().catch((error) => {
+    logError('main', error);
+    process.exit(1);
+});

package/package.json

@@ -0,0 +1,43 @@
+{
+  "name": "proctor-mcp-server",
+  "version": "0.1.0",
+  "description": "Local implementation of Proctor MCP server",
+  "main": "build/index.js",
+  "type": "module",
+  "bin": {
+    "proctor-mcp-server": "./build/index.js"
+  },
+  "files": [
+    "build/**/*.js",
+    "build/**/*.d.ts",
+    "shared/**/*.js",
+    "shared/**/*.d.ts",
+    "README.md"
+  ],
+  "scripts": {
+    "build": "tsc && npm run build:integration",
+    "build:integration": "tsc -p tsconfig.integration.json",
+    "start": "node build/index.js",
+    "dev": "tsx src/index.ts",
+    "predev": "cd ../shared && npm run build && cd ../local && node setup-dev.js",
+    "prebuild": "cd ../shared && npm run build && cd ../local && node setup-dev.js",
+    "prepublishOnly": "node prepare-publish.js && node ../scripts/prepare-npm-readme.js",
+    "lint": "eslint . --ext .ts,.tsx",
+    "lint:fix": "eslint . --ext .ts,.tsx --fix",
+    "format": "prettier --write .",
+    "format:check": "prettier --check .",
+    "stage-publish": "npm version"
+  },
+  "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.19.1",
+    "zod": "^3.24.1"
+  },
+  "devDependencies": {
+    "@types/node": "^22.10.6",
+    "tsx": "^4.19.4",
+    "typescript": "^5.7.3"
+  },
+  "keywords": [],
+  "author": "PulseMCP",
+  "license": "MIT"
+}

package/shared/index.d.ts

@@ -0,0 +1,7 @@
+export { createMCPServer, ProctorClient } from './server.js';
+export type { IProctorClient, ClientFactory } from './server.js';
+export { createRegisterTools, parseEnabledToolGroups } from './tools.js';
+export type { ToolGroup } from './tools.js';
+export { logServerStart, logError, logWarning, logDebug } from './logging.js';
+export type { ProctorRuntime, ProctorExam, ProctorMetadataResponse, ExamLogEntry, ExamStreamLog, ExamStreamResult, ExamStreamError, ExamStreamEntry, ExamResult, RunExamParams, SaveResultParams, SaveResultResponse, PriorResultParams, PriorResultResponse, FlyMachine, MachinesResponse, CancelExamParams, CancelExamResponse, ApiError, } from './types.js';
+//# sourceMappingURL=index.d.ts.map

package/shared/index.js

@@ -0,0 +1,4 @@
+// Main exports for the Proctor MCP server shared module
+export { createMCPServer, ProctorClient } from './server.js';
+export { createRegisterTools, parseEnabledToolGroups } from './tools.js';
+export { logServerStart, logError, logWarning, logDebug } from './logging.js';

package/shared/logging.d.ts

@@ -0,0 +1,20 @@
+/**
+ * Logging utilities for consistent output across MCP servers
+ */
+/**
+ * Log server startup message
+ */
+export declare function logServerStart(serverName: string, transport?: string): void;
+/**
+ * Log an error with context
+ */
+export declare function logError(context: string, error: unknown): void;
+/**
+ * Log a warning
+ */
+export declare function logWarning(context: string, message: string): void;
+/**
+ * Log debug information (only in development)
+ */
+export declare function logDebug(context: string, message: string): void;
+//# sourceMappingURL=logging.d.ts.map

package/shared/logging.js

@@ -0,0 +1,34 @@
+/**
+ * Logging utilities for consistent output across MCP servers
+ */
+/**
+ * Log server startup message
+ */
+export function logServerStart(serverName, transport = 'stdio') {
+    console.error(`MCP server ${serverName} running on ${transport}`);
+}
+/**
+ * Log an error with context
+ */
+export function logError(context, error) {
+    const message = error instanceof Error ? error.message : String(error);
+    const stack = error instanceof Error ? error.stack : undefined;
+    console.error(`[ERROR] ${context}: ${message}`);
+    if (stack) {
+        console.error(stack);
+    }
+}
+/**
+ * Log a warning
+ */
+export function logWarning(context, message) {
+    console.error(`[WARN] ${context}: ${message}`);
+}
+/**
+ * Log debug information (only in development)
+ */
+export function logDebug(context, message) {
+    if (process.env.NODE_ENV === 'development' || process.env.DEBUG) {
+        console.error(`[DEBUG] ${context}: ${message}`);
+    }
+}

package/shared/proctor-client/lib/cancel-exam.d.ts

@@ -0,0 +1,6 @@
+import type { CancelExamParams, CancelExamResponse } from '../../types.js';
+/**
+ * Cancel a running exam
+ */
+export declare function cancelExam(apiKey: string, baseUrl: string, params: CancelExamParams): Promise<CancelExamResponse>;
+//# sourceMappingURL=cancel-exam.d.ts.map

package/shared/proctor-client/lib/cancel-exam.js

@@ -0,0 +1,36 @@
+/**
+ * Cancel a running exam
+ */
+export async function cancelExam(apiKey, baseUrl, params) {
+    const url = new URL('/api/proctor/cancel_exam', baseUrl);
+    const response = await fetch(url.toString(), {
+        method: 'POST',
+        headers: {
+            'X-API-Key': apiKey,
+            'Content-Type': 'application/json',
+            Accept: 'application/json',
+        },
+        body: JSON.stringify({
+            machine_id: params.machine_id,
+            exam_id: params.exam_id,
+        }),
+    });
+    if (!response.ok) {
+        if (response.status === 401) {
+            throw new Error('Invalid API key');
+        }
+        if (response.status === 403) {
+            throw new Error('User lacks admin privileges or insufficient permissions');
+        }
+        if (response.status === 400) {
+            const errorData = (await response.json());
+            throw new Error(`Bad request: ${errorData.error || 'Invalid parameters'}`);
+        }
+        if (response.status === 422) {
+            const errorData = (await response.json());
+            throw new Error(`Service error: ${errorData.error || 'Unknown error'}`);
+        }
+        throw new Error(`Failed to cancel exam: ${response.status} ${response.statusText}`);
+    }
+    return (await response.json());
+}

package/shared/proctor-client/lib/destroy-machine.d.ts

@@ -0,0 +1,7 @@
+/**
+ * Delete a Fly.io machine
+ */
+export declare function destroyMachine(apiKey: string, baseUrl: string, machineId: string): Promise<{
+    success: boolean;
+}>;
+//# sourceMappingURL=destroy-machine.d.ts.map

package/shared/proctor-client/lib/destroy-machine.js

@@ -0,0 +1,31 @@
+/**
+ * Delete a Fly.io machine
+ */
+export async function destroyMachine(apiKey, baseUrl, machineId) {
+    const url = new URL(`/api/proctor/machines/${encodeURIComponent(machineId)}`, baseUrl);
+    const response = await fetch(url.toString(), {
+        method: 'DELETE',
+        headers: {
+            'X-API-Key': apiKey,
+            Accept: 'application/json',
+        },
+    });
+    if (!response.ok) {
+        if (response.status === 401) {
+            throw new Error('Invalid API key');
+        }
+        if (response.status === 403) {
+            throw new Error('User lacks admin privileges or insufficient permissions');
+        }
+        if (response.status === 400) {
+            const errorData = (await response.json());
+            throw new Error(`Bad request: ${errorData.error || 'Invalid machine ID format'}`);
+        }
+        if (response.status === 422) {
+            const errorData = (await response.json());
+            throw new Error(`Service error: ${errorData.error || 'Unknown error'}`);
+        }
+        throw new Error(`Failed to destroy machine: ${response.status} ${response.statusText}`);
+    }
+    return (await response.json());
+}

package/shared/proctor-client/lib/get-machines.d.ts

@@ -0,0 +1,6 @@
+import type { MachinesResponse } from '../../types.js';
+/**
+ * List active Fly.io machines for proctor exams
+ */
+export declare function getMachines(apiKey: string, baseUrl: string): Promise<MachinesResponse>;
+//# sourceMappingURL=get-machines.d.ts.map

package/shared/proctor-client/lib/get-machines.js

@@ -0,0 +1,27 @@
+/**
+ * List active Fly.io machines for proctor exams
+ */
+export async function getMachines(apiKey, baseUrl) {
+    const url = new URL('/api/proctor/machines', baseUrl);
+    const response = await fetch(url.toString(), {
+        method: 'GET',
+        headers: {
+            'X-API-Key': apiKey,
+            Accept: 'application/json',
+        },
+    });
+    if (!response.ok) {
+        if (response.status === 401) {
+            throw new Error('Invalid API key');
+        }
+        if (response.status === 403) {
+            throw new Error('User lacks admin privileges or insufficient permissions');
+        }
+        if (response.status === 422) {
+            const errorData = (await response.json());
+            throw new Error(`Service error: ${errorData.error || 'Unknown error'}`);
+        }
+        throw new Error(`Failed to get machines: ${response.status} ${response.statusText}`);
+    }
+    return (await response.json());
+}