@wix/pathgrade 1.0.6 → 1.0.8

package/README.md

@@ -1,6 +1,6 @@
 # Pathgrade
 
-**Evaluate AI coding agents with Vitest or Jest.** Write evals as normal `.eval.ts` files, run Claude Code, Codex, or Cursor in isolated sandboxes, and score the result with deterministic checks, rubric judges, and tool-usage assertions.
+**Evaluate AI coding agents with Vitest or Jest.** Write evals as normal `.eval.ts` files, run Claude Code, Codex, Cursor, or trusted-internal OpenCode fixtures in isolated workspaces, and score the result with deterministic checks, rubric judges, and tool-usage assertions.
 
 ## Why Pathgrade?
 
@@ -13,7 +13,7 @@
 
 ## Quick Start
 
-**Prerequisites**: Node.js 20.11+, Vitest 4+ or Jest 30+, and at least one configured agent runtime. Claude uses the bundled `@anthropic-ai/claude-agent-sdk` binary by default; Codex requires the `codex` CLI; Cursor requires the `cursor-agent` CLI.
+**Prerequisites**: Node.js 20.11+, Vitest 4+ or Jest 30+, and at least one configured agent runtime. Claude uses the bundled `@anthropic-ai/claude-agent-sdk` binary by default; Codex requires the `codex` CLI; Cursor requires the `cursor-agent` CLI. OpenCode requires the exact v1.18.18 macOS ARM64 or Linux ARM64 executable.
 
 ```bash
 yarn add -D @wix/pathgrade
@@ -40,6 +40,9 @@ By default, Pathgrade tries to reuse the agent CLI's native auth before falling
   - forwards `CURSOR_API_KEY` when set
   - macOS: reuses `cursor-agent login` OAuth tokens from the login Keychain
   - surfaces a clear error when neither is available (run `cursor-agent login` or set `CURSOR_API_KEY`)
+- **OpenCode**
+  - requires an explicit `ANTHROPIC_API_KEY` in `createAgent({ env })`; ambient credentials and login state are never read
+  - accepts an optional explicit HTTPS `ANTHROPIC_BASE_URL` ending in `/v1`
 
 If you set `ANTHROPIC_BASE_URL`, `OPENAI_BASE_URL`, or `CURSOR_API_BASE_URL`, set the matching API key too.
 
@@ -231,7 +234,24 @@ toolUsage('expected-workflow', [
 
 ## Conversations
 
-Pathgrade currently supports three agent backends: `claude`, `codex`, and `cursor`. Set the backend per test via `createAgent({ agent: 'claude' })`, or omit `agent` and use `PATHGRADE_AGENT` as the process-wide fallback.
+Pathgrade supports `claude`, `codex`, `cursor`, and `opencode`. Set the backend per test via `createAgent({ agent: 'claude' })`, or omit `agent` and use `PATHGRADE_AGENT` as the process-wide fallback.
+
+### OpenCode trusted-internal backend
+
+OpenCode v1 is only for fixtures, prompts, skills, and generated MCP mocks controlled by trusted Pathgrade maintainers. It is not a security sandbox: OpenCode and Bash children run with the host user's filesystem, process, and network authority and receive the provider credentials. Do not use it for external repositories, user-supplied prompts, or adversarial fixtures.
+
+```typescript
+const agent = await createAgent({
+    agent: 'opencode',
+    opencodeExecutable: '/absolute/path/to/opencode-v1.18.18',
+    env: {
+        ANTHROPIC_API_KEY: process.env.APP_ANTHROPIC_API_KEY!,
+        ANTHROPIC_BASE_URL: 'https://api.example.com/v1',
+    },
+});
+```
+
+The backend supports only `anthropic/claude-sonnet-5`, generated stdio `mcpMock` servers, and clean hosts without system-managed OpenCode configuration. `PATHGRADE_AGENT=opencode` selects the backend but does not supply the required executable or credentials.
 
 ### `agent.prompt()` - One shot
 
@@ -292,7 +312,8 @@ const result = await agent.runConversation({
 Pathgrade exposes a few useful features that are easy to miss from the basic examples:
 
 - `createAgent({ skillDir, workspace })` stages a real skill and a fixture workspace into the sandbox, which is how Pathgrade's skill examples are evaluated.
-- `createAgent({ debug: true })` preserves the final workspace under `pathgrade-debug/<test-name>/`; when you use `runConversation()`, it also writes `run-snapshot.json`.
+- `createAgent({ debug: true })` preserves the final workspace under the backward-compatible `pathgrade-debug/<test-name>/` path; when you use `runConversation()`, it also writes `run-snapshot.json`.
+- `createAgent({ debug: { retainRuns: 5 } })` opts into managed run retention under `pathgrade-debug/runs/<run-id>/<test-name>/`. Use `pathgrade clean --debug`, `--keep=N`, and `--dry-run` to clean marked, inactive debug runs safely.
 - `evaluate.fromSnapshot(snapshotPath, scorers)` re-runs grading against a saved snapshot without re-running the agent.
 - `previewReactions(messages, reactions)` lets you inspect which scripted reactions would fire offline.
 - `conversationWindow` on agents and personas keeps long transcripts bounded with summarization instead of sending the full conversation every turn.
@@ -409,13 +430,13 @@ Notes:
 
 | Variable | Purpose |
 |----------|---------|
-| `ANTHROPIC_API_KEY` | Claude auth and the required key when using `ANTHROPIC_BASE_URL` |
+| `ANTHROPIC_API_KEY` | Claude auth; also required explicitly in `AgentOptions.env` for OpenCode |
 | `OPENAI_API_KEY` | Codex auth and the required key when using `OPENAI_BASE_URL` |
 | `CURSOR_API_KEY` | Cursor auth and the required key when using `CURSOR_API_BASE_URL` |
 | `ANTHROPIC_BASE_URL` | Custom Anthropic-compatible endpoint |
 | `OPENAI_BASE_URL` | Custom OpenAI-compatible endpoint |
 | `CURSOR_API_BASE_URL` | Custom Cursor-compatible endpoint |
-| `PATHGRADE_AGENT` | Fallback agent for all tests (`claude`, `codex`, or `cursor`). `createAgent({ agent })` wins over this. |
+| `PATHGRADE_AGENT` | Fallback agent for all tests (`claude`, `codex`, `cursor`, or `opencode`). `createAgent({ agent })` wins over this. |
 | `PATHGRADE_CODEX_TRANSPORT` | Fallback Codex transport (`exec` or `app-server`). `createAgent({ transport })` wins over this. |
 | `PATHGRADE_VERBOSE` | `1` enables live per-turn streaming to stderr |
 | `PATHGRADE_DIAGNOSTICS` | `1` prints full diagnostics for passing evals too |

package/dist/agents/opencode-contract.d.ts

@@ -0,0 +1,24 @@
+import type { MockMcpServerDescriptor } from '../core/mcp-mock.types.js';
+import type { AgentName, AgentOptions } from '../sdk/types.js';
+export declare const OPENCODE_MODEL = "anthropic/claude-sonnet-5";
+export declare const OPENCODE_VERSION = "1.18.18";
+export interface OpenCodeRuntimeLockEntry {
+    version: typeof OPENCODE_VERSION;
+    executableSha256: string;
+}
+export declare const OPENCODE_RUNTIME_LOCK: {
+    readonly 'darwin-arm64': {
+        readonly version: "1.18.18";
+        readonly executableSha256: "4f5979c2dadb06fbff1335335afaaea274e58f92e79aa43cf2ed98618d555422";
+    };
+    readonly 'linux-arm64': {
+        readonly version: "1.18.18";
+        readonly executableSha256: "a63ef0c7271383e48ffe36c156b4146087bbbc97929dec40e8be95ed4f4d76ae";
+    };
+};
+export type OpenCodePlatformKey = keyof typeof OPENCODE_RUNTIME_LOCK;
+export declare function getOpenCodePlatformKey(platform: NodeJS.Platform, arch: string): OpenCodePlatformKey | undefined;
+export declare function currentOpenCodePlatformKey(): OpenCodePlatformKey | undefined;
+export declare function validateOpenCodeDeclaration(agent: AgentName, opts: AgentOptions): void;
+export declare function sanitizeOpenCodeToolName(value: string): string;
+export declare function collectOpenCodeMcpToolNames(declaration: MockMcpServerDescriptor | MockMcpServerDescriptor[] | undefined): string[];

package/dist/agents/opencode-contract.js

@@ -0,0 +1,84 @@
+import * as path from 'node:path';
+export const OPENCODE_MODEL = 'anthropic/claude-sonnet-5';
+export const OPENCODE_VERSION = '1.18.18';
+export const OPENCODE_RUNTIME_LOCK = {
+    'darwin-arm64': {
+        version: OPENCODE_VERSION,
+        executableSha256: '4f5979c2dadb06fbff1335335afaaea274e58f92e79aa43cf2ed98618d555422',
+    },
+    'linux-arm64': {
+        version: OPENCODE_VERSION,
+        executableSha256: 'a63ef0c7271383e48ffe36c156b4146087bbbc97929dec40e8be95ed4f4d76ae',
+    },
+};
+export function getOpenCodePlatformKey(platform, arch) {
+    const key = `${platform}-${arch}`;
+    return key in OPENCODE_RUNTIME_LOCK ? key : undefined;
+}
+export function currentOpenCodePlatformKey() {
+    return getOpenCodePlatformKey(process.platform, process.arch);
+}
+function hasOwn(record, key) {
+    return Object.prototype.hasOwnProperty.call(record, key);
+}
+function assertOpenCodeBaseUrl(value) {
+    let url;
+    try {
+        url = new URL(value);
+    }
+    catch {
+        throw new Error('OpenCode ANTHROPIC_BASE_URL must be an absolute HTTPS API root ending in /v1');
+    }
+    if (url.protocol !== 'https:' || url.username || url.password || url.search || url.hash || !url.pathname.endsWith('/v1')) {
+        throw new Error('OpenCode ANTHROPIC_BASE_URL must be an absolute HTTPS API root ending in /v1');
+    }
+}
+export function validateOpenCodeDeclaration(agent, opts) {
+    if (agent !== 'opencode') {
+        if (opts.opencodeExecutable !== undefined) {
+            throw new Error('opencodeExecutable is valid only for agent: "opencode"');
+        }
+        return;
+    }
+    if (!currentOpenCodePlatformKey()) {
+        throw new Error(`OpenCode is not supported on ${process.platform}-${process.arch}`);
+    }
+    if (!opts.opencodeExecutable || !path.isAbsolute(opts.opencodeExecutable)) {
+        throw new Error('OpenCode requires an absolute opencodeExecutable path');
+    }
+    if (opts.model !== undefined && opts.model !== OPENCODE_MODEL) {
+        throw new Error(`OpenCode v1 supports only model ${OPENCODE_MODEL}`);
+    }
+    const env = opts.env ?? {};
+    if (!hasOwn(env, 'ANTHROPIC_API_KEY') || !env.ANTHROPIC_API_KEY?.trim()) {
+        throw new Error('OpenCode requires a nonempty own env.ANTHROPIC_API_KEY');
+    }
+    if (hasOwn(env, 'ANTHROPIC_BASE_URL')) {
+        if (!env.ANTHROPIC_BASE_URL) {
+            throw new Error('OpenCode ANTHROPIC_BASE_URL must be a nonempty absolute HTTPS API root ending in /v1');
+        }
+        assertOpenCodeBaseUrl(env.ANTHROPIC_BASE_URL);
+    }
+    if (opts.copyFromHome !== undefined)
+        throw new Error('OpenCode does not support copyFromHome');
+    if (opts.mcpConfigFile !== undefined)
+        throw new Error('OpenCode v1 supports generated mcpMock only, not mcpConfigFile');
+    if (opts.mcpSafety !== undefined)
+        throw new Error('OpenCode v1 does not support mcpSafety');
+    if (opts.transport !== undefined)
+        throw new Error('OpenCode does not support transport overrides');
+    for (const key of Object.keys(env)) {
+        if (key === 'HOME' || key.startsWith('XDG_') || key.startsWith('OPENCODE_')) {
+            throw new Error(`OpenCode reserves env.${key}`);
+        }
+    }
+}
+export function sanitizeOpenCodeToolName(value) {
+    return value.replace(/[^A-Za-z0-9_-]/g, '_');
+}
+export function collectOpenCodeMcpToolNames(declaration) {
+    if (!declaration)
+        return [];
+    const descriptors = Array.isArray(declaration) ? declaration : [declaration];
+    return descriptors.flatMap((descriptor) => descriptor.config.tools.map((tool) => `${sanitizeOpenCodeToolName(descriptor.config.name)}_${sanitizeOpenCodeToolName(tool.name)}`));
+}

package/dist/agents/opencode.d.ts

@@ -0,0 +1,27 @@
+import { BaseAgent, type AgentCommandRunner, type AgentSession, type AgentSessionOptions, type AgentTurnResult, type EnvironmentHandle } from '../types.js';
+interface SpawnResult {
+    stdout: string;
+    exitCode: number | null;
+    signal: NodeJS.Signals | null;
+    overflow: boolean;
+    aborted: boolean;
+}
+interface SpawnOpenCodeOptions {
+    cwd: string;
+    env: Record<string, string>;
+    stdin: string;
+    signal?: AbortSignal;
+    outputCapBytes?: number;
+}
+export declare function spawnOpenCode(executable: string, args: string[], options: SpawnOpenCodeOptions): Promise<SpawnResult>;
+interface ParsedOpenCodeTurn {
+    result: AgentTurnResult;
+    sessionId: string;
+}
+export declare function parseOpenCodeOutput(stdout: string, processResult: Pick<SpawnResult, 'exitCode' | 'overflow' | 'aborted'>, mcpToolNames: ReadonlySet<string>): ParsedOpenCodeTurn;
+export declare function managedOpenCodeConfigPaths(platform?: NodeJS.Platform, username?: string): string[];
+export declare function assertCleanManagedOpenCodeHost(candidates?: string[]): Promise<void>;
+export declare class OpenCodeAgent extends BaseAgent {
+    createSession(runtime: EnvironmentHandle, _runCommand: AgentCommandRunner, options?: AgentSessionOptions): Promise<AgentSession>;
+}
+export {};