scorecard-ai-mcp 2.6.0 → 3.0.0-beta.0

package/LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2025 Scorecard
+   Copyright 2026 Scorecard
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

package/README.md

@@ -26,7 +26,7 @@ For clients with a configuration JSON, it might look something like this:
   "mcpServers": {
     "scorecard_ai_api": {
       "command": "npx",
-      "args": ["-y", "scorecard-ai-mcp", "--client=claude", "--tools=dynamic"],
+      "args": ["-y", "scorecard-ai-mcp"],
       "env": {
         "SCORECARD_API_KEY": "My API Key",
         "SCORECARD_ENVIRONMENT": "production"
@@ -56,113 +56,25 @@ If you use Claude Code, you can install the MCP server by running the command be
 environment variables in Claude Code's `.claude.json`, which can be found in your home directory.
 
 ```
-claude mcp add --transport stdio scorecard_ai_api --env SCORECARD_API_KEY="Your SCORECARD_API_KEY here." -- npx -y scorecard-ai-mcp
+claude mcp add scorecard_ai_mcp_api --env SCORECARD_API_KEY="Your SCORECARD_API_KEY here." -- npx -y scorecard-ai-mcp
 ```
 
-## Exposing endpoints to your MCP Client
+## Code Mode
 
-There are three ways to expose endpoints as tools in the MCP server:
+This MCP server is built on the "Code Mode" tool scheme. In this MCP Server,
+your agent will write code against the TypeScript SDK, which will then be executed in an
+isolated sandbox. To accomplish this, the server will expose two tools to your agent:
 
-1. Exposing one tool per endpoint, and filtering as necessary
-2. Exposing a set of tools to dynamically discover and invoke endpoints from the API
-3. Exposing a docs search tool and a code execution tool, allowing the client to write code to be executed against the TypeScript client
+- The first tool is a docs search tool, which can be used to generically query for
+  documentation about your API/SDK.
 
-### Filtering endpoints and tools
+- The second tool is a code tool, where the agent can write code against the TypeScript SDK.
+  The code will be executed in a sandbox environment without web or filesystem access. Then,
+  anything the code returns or prints will be returned to the agent as the result of the
+  tool call.
 
-You can run the package on the command line to discover and filter the set of tools that are exposed by the
-MCP Server. This can be helpful for large APIs where including all endpoints at once is too much for your AI's
-context window.
-
-You can filter by multiple aspects:
-
-- `--tool` includes a specific tool by name
-- `--resource` includes all tools under a specific resource, and can have wildcards, e.g. `my.resource*`
-- `--operation` includes just read (get/list) or just write operations
-
-### Dynamic tools
-
-If you specify `--tools=dynamic` to the MCP server, instead of exposing one tool per endpoint in the API, it will
-expose the following tools:
-
-1. `list_api_endpoints` - Discovers available endpoints, with optional filtering by search query
-2. `get_api_endpoint_schema` - Gets detailed schema information for a specific endpoint
-3. `invoke_api_endpoint` - Executes any endpoint with the appropriate parameters
-
-This allows you to have the full set of API endpoints available to your MCP Client, while not requiring that all
-of their schemas be loaded into context at once. Instead, the LLM will automatically use these tools together to
-search for, look up, and invoke endpoints dynamically. However, due to the indirect nature of the schemas, it
-can struggle to provide the correct properties a bit more than when tools are imported explicitly. Therefore,
-you can opt-in to explicit tools, the dynamic tools, or both.
-
-See more information with `--help`.
-
-All of these command-line options can be repeated, combined together, and have corresponding exclusion versions (e.g. `--no-tool`).
-
-Use `--list` to see the list of available tools, or see below.
-
-### Code execution
-
-If you specify `--tools=code` to the MCP server, it will expose just two tools:
-
-- `search_docs` - Searches the API documentation and returns a list of markdown results
-- `execute` - Runs code against the TypeScript client
-
-This allows the LLM to implement more complex logic by chaining together many API calls without loading
-intermediary results into its context window.
-
-The code execution itself happens in a Deno sandbox that has network access only to the base URL for the API.
-
-### Specifying the MCP Client
-
-Different clients have varying abilities to handle arbitrary tools and schemas.
-
-You can specify the client you are using with the `--client` argument, and the MCP server will automatically
-serve tools and schemas that are more compatible with that client.
-
-- `--client=<type>`: Set all capabilities based on a known MCP client
-
-  - Valid values: `openai-agents`, `claude`, `claude-code`, `cursor`
-  - Example: `--client=cursor`
-
-Additionally, if you have a client not on the above list, or the client has gotten better
-over time, you can manually enable or disable certain capabilities:
-
-- `--capability=<name>`: Specify individual client capabilities
-  - Available capabilities:
-    - `top-level-unions`: Enable support for top-level unions in tool schemas
-    - `valid-json`: Enable JSON string parsing for arguments
-    - `refs`: Enable support for $ref pointers in schemas
-    - `unions`: Enable support for union types (anyOf) in schemas
-    - `formats`: Enable support for format validations in schemas (e.g. date-time, email)
-    - `tool-name-length=N`: Set maximum tool name length to N characters
-  - Example: `--capability=top-level-unions --capability=tool-name-length=40`
-  - Example: `--capability=top-level-unions,tool-name-length=40`
-
-### Examples
-
-1. Filter for read operations on cards:
-
-```bash
---resource=cards --operation=read
-```
-
-2. Exclude specific tools while including others:
-
-```bash
---resource=cards --no-tool=create_cards
-```
-
-3. Configure for Cursor client with custom max tool name length:
-
-```bash
---client=cursor --capability=tool-name-length=40
-```
-
-4. Complex filtering with multiple criteria:
-
-```bash
---resource=cards,accounts --operation=read --tag=kyc --no-tool=create_cards
-```
+Using this scheme, agents are capable of performing very complex tasks deterministically
+and repeatably.
 
 ## Running remotely
 
@@ -189,126 +101,3 @@ A configuration JSON for this server might look like this, assuming the server i
   }
 }
 ```
-
-The command-line arguments for filtering tools and specifying clients can also be used as query parameters in the URL.
-For example, to exclude specific tools while including others, use the URL:
-
-```
-http://localhost:3000?resource=cards&resource=accounts&no_tool=create_cards
-```
-
-Or, to configure for the Cursor client, with a custom max tool name length, use the URL:
-
-```
-http://localhost:3000?client=cursor&capability=tool-name-length%3D40
-```
-
-## Importing the tools and server individually
-
-```js
-// Import the server, generated endpoints, or the init function
-import { server, endpoints, init } from "scorecard-ai-mcp/server";
-
-// import a specific tool
-import createProjects from "scorecard-ai-mcp/tools/projects/create-projects";
-
-// initialize the server and all endpoints
-init({ server, endpoints });
-
-// manually start server
-const transport = new StdioServerTransport();
-await server.connect(transport);
-
-// or initialize your own server with specific tools
-const myServer = new McpServer(...);
-
-// define your own endpoint
-const myCustomEndpoint = {
-  tool: {
-    name: 'my_custom_tool',
-    description: 'My custom tool',
-    inputSchema: zodToJsonSchema(z.object({ a_property: z.string() })),
-  },
-  handler: async (client: client, args: any) => {
-    return { myResponse: 'Hello world!' };
-  })
-};
-
-// initialize the server with your custom endpoints
-init({ server: myServer, endpoints: [createProjects, myCustomEndpoint] });
-```
-
-## Available Tools
-
-The following tools are available in this MCP server.
-
-### Resource `projects`:
-
-- `create_projects` (`write`): Create a new Project.
-- `list_projects` (`read`): Retrieve a paginated list of all Projects. Projects are ordered by creation date, with oldest Projects first.
-
-### Resource `testsets`:
-
-- `create_testsets` (`write`): Create a new Testset for a Project. The Testset will be created in the Project specified in the path.
-- `update_testsets` (`write`): Update a Testset. Only the fields provided in the request body will be updated.
-  If a field is provided, the new content will replace the existing content.
-  If a field is not provided, the existing content will remain unchanged.
-
-  When updating the schema:
-
-  - If field mappings are not provided and existing mappings reference fields that no longer exist, those mappings will be automatically removed
-  - To preserve all existing mappings, ensure all referenced fields remain in the updated schema
-  - For complete control, provide both schema and fieldMapping when updating the schema
-
-- `list_testsets` (`read`): Retrieve a paginated list of Testsets belonging to a Project.
-- `delete_testsets` (`write`): Delete Testset
-- `get_testsets` (`read`): Get Testset
-
-### Resource `testcases`:
-
-- `create_testcases` (`write`): Create multiple Testcases in the specified Testset.
-- `update_testcases` (`write`): Replace the data of an existing Testcase while keeping its ID.
-- `list_testcases` (`read`): Retrieve a paginated list of Testcases belonging to a Testset.
-- `delete_testcases` (`write`): Delete multiple Testcases by their IDs.
-- `get_testcases` (`read`): Retrieve a specific Testcase by ID.
-
-### Resource `runs`:
-
-- `create_runs` (`write`): Create a new Run.
-- `list_runs` (`read`): Retrieve a paginated list of all Runs for a Project. Runs are ordered by creation date, most recent first.
-- `get_runs` (`read`): Retrieve a specific Run by ID.
-
-### Resource `metrics`:
-
-- `create_metrics` (`write`): Create a new Metric for evaluating system outputs. The structure of a metric depends on the evalType and outputType of the metric.
-- `update_metrics` (`write`): Update an existing Metric. You must specify the evalType and outputType of the metric. The structure of a metric depends on the evalType and outputType of the metric.
-- `list_metrics` (`read`): List Metrics configured for the specified Project. Metrics are returned in reverse chronological order.
-- `delete_metrics` (`write`): Delete a specific Metric by ID. The metric will be removed from metric groups and monitors.
-- `get_metrics` (`read`): Retrieve a specific Metric by ID.
-
-### Resource `records`:
-
-- `create_records` (`write`): Create a new Record in a Run.
-- `list_records` (`read`): Retrieve a paginated list of Records for a Run, including all scores for each record.
-- `delete_records` (`write`): Delete a specific Record by ID.
-
-### Resource `scores`:
-
-- `upsert_scores` (`write`): Create or update a Score for a given Record and MetricConfig. If a Score with the specified Record ID and MetricConfig ID already exists, it will be updated. Otherwise, a new Score will be created. The score provided should conform to the schema defined by the MetricConfig; otherwise, validation errors will be reported.
-
-### Resource `systems`:
-
-- `update_systems` (`write`): Update an existing system. Only the fields provided in the request body will be updated.
-  If a field is provided, the new content will replace the existing content.
-  If a field is not provided, the existing content will remain unchanged.
-- `list_systems` (`read`): Retrieve a paginated list of all systems. Systems are ordered by creation date.
-- `delete_systems` (`write`): Delete a system definition by ID. This will not delete associated system versions.
-- `get_systems` (`read`): Retrieve a specific system by ID.
-- `upsert_systems` (`write`): Create a new system. If one with the same name in the project exists, it updates it instead.
-
-### Resource `systems.versions`:
-
-- `get_systems_versions` (`read`): Retrieve a specific system version by ID.
-- `upsert_systems_versions` (`write`): Create a new system version if it does not already exist. Does **not** set the created version to be the system's production version.
-
-  If there is already a system version with the same config, its name will be updated.

package/code-tool-types.d.mts

@@ -1,16 +1,13 @@
 import { ClientOptions } from 'scorecard-ai';
 export type WorkerInput = {
-    opts: ClientOptions;
+    project_name: string;
     code: string;
+    client_opts: ClientOptions;
 };
-export type WorkerSuccess = {
+export type WorkerOutput = {
+    is_error: boolean;
     result: unknown | null;
-    logLines: string[];
-    errLines: string[];
-};
-export type WorkerError = {
-    message: string | undefined;
-    logLines: string[];
-    errLines: string[];
+    log_lines: string[];
+    err_lines: string[];
 };
 //# sourceMappingURL=code-tool-types.d.mts.map

package/code-tool-types.d.mts.map

@@ -1 +1 @@
-{"version":3,"file":"code-tool-types.d.mts","sourceRoot":"","sources":["src/code-tool-types.ts"],"names":[],"mappings":"OAEO,EAAE,aAAa,EAAE,MAAM,cAAc;AAE5C,MAAM,MAAM,WAAW,GAAG;IACxB,IAAI,EAAE,aAAa,CAAC;IACpB,IAAI,EAAE,MAAM,CAAC;CACd,CAAC;AACF,MAAM,MAAM,aAAa,GAAG;IAC1B,MAAM,EAAE,OAAO,GAAG,IAAI,CAAC;IACvB,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,QAAQ,EAAE,MAAM,EAAE,CAAC;CACpB,CAAC;AACF,MAAM,MAAM,WAAW,GAAG;IACxB,OAAO,EAAE,MAAM,GAAG,SAAS,CAAC;IAC5B,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,QAAQ,EAAE,MAAM,EAAE,CAAC;CACpB,CAAC"}
+{"version":3,"file":"code-tool-types.d.mts","sourceRoot":"","sources":["src/code-tool-types.ts"],"names":[],"mappings":"OAEO,EAAE,aAAa,EAAE,MAAM,cAAc;AAE5C,MAAM,MAAM,WAAW,GAAG;IACxB,YAAY,EAAE,MAAM,CAAC;IACrB,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,EAAE,aAAa,CAAC;CAC5B,CAAC;AACF,MAAM,MAAM,YAAY,GAAG;IACzB,QAAQ,EAAE,OAAO,CAAC;IAClB,MAAM,EAAE,OAAO,GAAG,IAAI,CAAC;IACvB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,SAAS,EAAE,MAAM,EAAE,CAAC;CACrB,CAAC"}

package/code-tool-types.d.ts

@@ -1,16 +1,13 @@
 import { ClientOptions } from 'scorecard-ai';
 export type WorkerInput = {
-    opts: ClientOptions;
+    project_name: string;
     code: string;
+    client_opts: ClientOptions;
 };
-export type WorkerSuccess = {
+export type WorkerOutput = {
+    is_error: boolean;
     result: unknown | null;
-    logLines: string[];
-    errLines: string[];
-};
-export type WorkerError = {
-    message: string | undefined;
-    logLines: string[];
-    errLines: string[];
+    log_lines: string[];
+    err_lines: string[];
 };
 //# sourceMappingURL=code-tool-types.d.ts.map

package/code-tool-types.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"code-tool-types.d.ts","sourceRoot":"","sources":["src/code-tool-types.ts"],"names":[],"mappings":"OAEO,EAAE,aAAa,EAAE,MAAM,cAAc;AAE5C,MAAM,MAAM,WAAW,GAAG;IACxB,IAAI,EAAE,aAAa,CAAC;IACpB,IAAI,EAAE,MAAM,CAAC;CACd,CAAC;AACF,MAAM,MAAM,aAAa,GAAG;IAC1B,MAAM,EAAE,OAAO,GAAG,IAAI,CAAC;IACvB,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,QAAQ,EAAE,MAAM,EAAE,CAAC;CACpB,CAAC;AACF,MAAM,MAAM,WAAW,GAAG;IACxB,OAAO,EAAE,MAAM,GAAG,SAAS,CAAC;IAC5B,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,QAAQ,EAAE,MAAM,EAAE,CAAC;CACpB,CAAC"}
+{"version":3,"file":"code-tool-types.d.ts","sourceRoot":"","sources":["src/code-tool-types.ts"],"names":[],"mappings":"OAEO,EAAE,aAAa,EAAE,MAAM,cAAc;AAE5C,MAAM,MAAM,WAAW,GAAG;IACxB,YAAY,EAAE,MAAM,CAAC;IACrB,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,EAAE,aAAa,CAAC;CAC5B,CAAC;AACF,MAAM,MAAM,YAAY,GAAG;IACzB,QAAQ,EAAE,OAAO,CAAC;IAClB,MAAM,EAAE,OAAO,GAAG,IAAI,CAAC;IACvB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,SAAS,EAAE,MAAM,EAAE,CAAC;CACrB,CAAC"}

package/code-tool.d.mts

@@ -1,4 +1,4 @@
-import { Metadata, ToolCallResult } from "./tools/types.mjs";
+import { McpTool } from "./types.mjs";
 /**
  * A tool that runs code against a copy of the SDK.
  *
@@ -8,47 +8,5 @@ import { Metadata, ToolCallResult } from "./tools/types.mjs";
  *
  * @param endpoints - The endpoints to include in the list.
  */
-export declare function codeTool(): Promise<{
-    metadata: Metadata;
-    tool: {
-        inputSchema: {
-            [x: string]: unknown;
-            type: "object";
-            properties?: {
-                [x: string]: object;
-            } | undefined;
-            required?: string[] | undefined;
-        };
-        name: string;
-        description?: string | undefined;
-        outputSchema?: {
-            [x: string]: unknown;
-            type: "object";
-            properties?: {
-                [x: string]: object;
-            } | undefined;
-            required?: string[] | undefined;
-        } | undefined;
-        annotations?: {
-            title?: string | undefined;
-            readOnlyHint?: boolean | undefined;
-            destructiveHint?: boolean | undefined;
-            idempotentHint?: boolean | undefined;
-            openWorldHint?: boolean | undefined;
-        } | undefined;
-        execution?: {
-            taskSupport?: "optional" | "required" | "forbidden" | undefined;
-        } | undefined;
-        _meta?: {
-            [x: string]: unknown;
-        } | undefined;
-        icons?: {
-            src: string;
-            mimeType?: string | undefined;
-            sizes?: string[] | undefined;
-        }[] | undefined;
-        title?: string | undefined;
-    };
-    handler: (_: unknown, args: any) => Promise<ToolCallResult>;
-}>;
+export declare function codeTool(): McpTool;
 //# sourceMappingURL=code-tool.d.mts.map

package/code-tool.d.mts.map

@@ -1 +1 @@
-{"version":3,"file":"code-tool.d.mts","sourceRoot":"","sources":["src/code-tool.ts"],"names":[],"mappings":"OAEO,EAAE,QAAQ,EAAE,cAAc,EAAuB;AAIxD;;;;;;;;GAQG;AACH,wBAAsB,QAAQ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;iBAQF,OAAO,QAAQ,GAAG,KAAG,OAAO,CAAC,cAAc,CAAC;GAmCvE"}
+{"version":3,"file":"code-tool.d.mts","sourceRoot":"","sources":["src/code-tool.ts"],"names":[],"mappings":"OAEO,EAAE,OAAO,EAAgE;AA0BhF;;;;;;;;GAQG;AACH,wBAAgB,QAAQ,IAAI,OAAO,CAuDlC"}

package/code-tool.d.ts

@@ -1,4 +1,4 @@
-import { Metadata, ToolCallResult } from "./tools/types.js";
+import { McpTool } from "./types.js";
 /**
  * A tool that runs code against a copy of the SDK.
  *
@@ -8,47 +8,5 @@ import { Metadata, ToolCallResult } from "./tools/types.js";
  *
  * @param endpoints - The endpoints to include in the list.
  */
-export declare function codeTool(): Promise<{
-    metadata: Metadata;
-    tool: {
-        inputSchema: {
-            [x: string]: unknown;
-            type: "object";
-            properties?: {
-                [x: string]: object;
-            } | undefined;
-            required?: string[] | undefined;
-        };
-        name: string;
-        description?: string | undefined;
-        outputSchema?: {
-            [x: string]: unknown;
-            type: "object";
-            properties?: {
-                [x: string]: object;
-            } | undefined;
-            required?: string[] | undefined;
-        } | undefined;
-        annotations?: {
-            title?: string | undefined;
-            readOnlyHint?: boolean | undefined;
-            destructiveHint?: boolean | undefined;
-            idempotentHint?: boolean | undefined;
-            openWorldHint?: boolean | undefined;
-        } | undefined;
-        execution?: {
-            taskSupport?: "optional" | "required" | "forbidden" | undefined;
-        } | undefined;
-        _meta?: {
-            [x: string]: unknown;
-        } | undefined;
-        icons?: {
-            src: string;
-            mimeType?: string | undefined;
-            sizes?: string[] | undefined;
-        }[] | undefined;
-        title?: string | undefined;
-    };
-    handler: (_: unknown, args: any) => Promise<ToolCallResult>;
-}>;
+export declare function codeTool(): McpTool;
 //# sourceMappingURL=code-tool.d.ts.map

package/code-tool.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"code-tool.d.ts","sourceRoot":"","sources":["src/code-tool.ts"],"names":[],"mappings":"OAEO,EAAE,QAAQ,EAAE,cAAc,EAAuB;AAIxD;;;;;;;;GAQG;AACH,wBAAsB,QAAQ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;iBAQF,OAAO,QAAQ,GAAG,KAAG,OAAO,CAAC,cAAc,CAAC;GAmCvE"}
+{"version":3,"file":"code-tool.d.ts","sourceRoot":"","sources":["src/code-tool.ts"],"names":[],"mappings":"OAEO,EAAE,OAAO,EAAgE;AA0BhF;;;;;;;;GAQG;AACH,wBAAgB,QAAQ,IAAI,OAAO,CAuDlC"}

package/code-tool.js

@@ -2,8 +2,27 @@
 // File generated from our OpenAPI spec by Stainless. See CONTRIBUTING.md for details.
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.codeTool = codeTool;
-const types_1 = require("./tools/types.js");
+const types_1 = require("./types.js");
 const server_1 = require("./server.js");
+const prompt = `Runs JavaScript code to interact with the Scorecard API.
+
+You are a skilled programmer writing code to interface with the service.
+Define an async function named "run" that takes a single parameter of an initialized SDK client and it will be run.
+For example:
+
+\`\`\`
+async function run(client) {
+  const run = await client.runs.create('314', { metricIds: ['789', '101'], testsetId: '246' });
+
+  console.log(run.id);
+}
+\`\`\`
+
+You will be returned anything that your function returns, plus the results of any console.log statements.
+Do not add try-catch blocks for single API calls. The tool will handle errors for you.
+Do not add comments unless necessary for generating better code.
+Code will run in a container, and cannot interact with the network outside of the given SDK client.
+Variables will not persist between calls, so make sure to return or log any data you might need later.`;
 /**
  * A tool that runs code against a copy of the SDK.
  *
@@ -13,14 +32,14 @@ const server_1 = require("./server.js");
  *
  * @param endpoints - The endpoints to include in the list.
  */
-async function codeTool() {
+function codeTool() {
     const metadata = { resource: 'all', operation: 'write', tags: [] };
     const tool = {
         name: 'execute',
-        description: 'Runs JavaScript code to interact with the API.\n\nYou are a skilled programmer writing code to interface with the service.\nDefine an async function named "run" that takes a single parameter of an initialized SDK client and it will be run.\nWrite code within this template:\n\n```\nasync function run(client) {\n  // Fill this out\n}\n```\n\nYou will be returned anything that your function returns, plus the results of any console.log statements.\nIf any code triggers an error, the tool will return an error response, so you do not need to add error handling unless you want to output something more helpful than the raw error.\nIt is not necessary to add comments to code, unless by adding those comments you believe that you can generate better code.\nThis code will run in a container, and you will not be able to use fetch or otherwise interact with the network calls other than through the client you are given.\nAny variables you define won\'t live between successive uses of this call, so make sure to return or log any data you might need later.',
+        description: prompt,
         inputSchema: { type: 'object', properties: { code: { type: 'string' } } },
     };
-    const handler = async (_, args) => {
+    const handler = async (client, args) => {
         const code = args.code;
         // this is not required, but passing a Stainless API key for the matching project_name
         // will allow you to run code-mode queries against non-published versions of your SDK.
@@ -31,18 +50,31 @@ async function codeTool() {
             headers: {
                 ...(stainlessAPIKey && { Authorization: stainlessAPIKey }),
                 'Content-Type': 'application/json',
-                client_envs: JSON.stringify({ SCORECARD_API_KEY: (0, server_1.readEnv)('SCORECARD_API_KEY') }),
+                client_envs: JSON.stringify({
+                    SCORECARD_API_KEY: client.apiKey || (0, server_1.readEnvOrError)('SCORECARD_API_KEY'),
+                    SCORECARD_BASE_URL: client.baseURL || (0, server_1.readEnv)('SCORECARD_BASE_URL'),
+                }),
             },
             body: JSON.stringify({
                 project_name: 'scorecard',
-                client_opts: { environment: ((0, server_1.readEnv)('SCORECARD_ENVIRONMENT') || undefined) },
                 code,
+                client_opts: { environment: ((0, server_1.readEnv)('SCORECARD_ENVIRONMENT') || undefined) },
             }),
         });
         if (!res.ok) {
             throw new Error(`${res.status}: ${res.statusText} error when trying to contact Code Tool server. Details: ${await res.text()}`);
         }
-        return (0, types_1.asTextContentResult)((await res.json()));
+        const { is_error, result, log_lines, err_lines } = (await res.json());
+        const hasLogs = log_lines.length > 0 || err_lines.length > 0;
+        const output = {
+            result,
+            ...(log_lines.length > 0 && { log_lines }),
+            ...(err_lines.length > 0 && { err_lines }),
+        };
+        if (is_error) {
+            return (0, types_1.asErrorResult)(typeof result === 'string' && !hasLogs ? result : JSON.stringify(output, null, 2));
+        }
+        return (0, types_1.asTextContentResult)(output);
     };
     return { metadata, tool, handler };
 }

package/code-tool.js.map

@@ -1 +1 @@
-{"version":3,"file":"code-tool.js","sourceRoot":"","sources":["src/code-tool.ts"],"names":[],"mappings":";AAAA,sFAAsF;;AAetF,4BA2CC;AAxDD,4CAA8E;AAE9E,wCAAmC;AAEnC;;;;;;;;GAQG;AACI,KAAK,UAAU,QAAQ;IAC5B,MAAM,QAAQ,GAAa,EAAE,QAAQ,EAAE,KAAK,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE,EAAE,EAAE,CAAC;IAC7E,MAAM,IAAI,GAAS;QACjB,IAAI,EAAE,SAAS;QACf,WAAW,EACT,iiCAAiiC;QACniC,WAAW,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,UAAU,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,EAAE,EAAE;KAC1E,CAAC;IACF,MAAM,OAAO,GAAG,KAAK,EAAE,CAAU,EAAE,IAAS,EAA2B,EAAE;QACvE,MAAM,IAAI,GAAG,IAAI,CAAC,IAAc,CAAC;QAEjC,sFAAsF;QACtF,sFAAsF;QACtF,MAAM,eAAe,GAAG,IAAA,gBAAO,EAAC,mBAAmB,CAAC,CAAC;QACrD,MAAM,gBAAgB,GACpB,IAAA,gBAAO,EAAC,wBAAwB,CAAC,IAAI,4CAA4C,CAAC;QAEpF,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,gBAAgB,EAAE;YACxC,MAAM,EAAE,MAAM;YACd,OAAO,EAAE;gBACP,GAAG,CAAC,eAAe,IAAI,EAAE,aAAa,EAAE,eAAe,EAAE,CAAC;gBAC1D,cAAc,EAAE,kBAAkB;gBAClC,WAAW,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,iBAAiB,EAAE,IAAA,gBAAO,EAAC,mBAAmB,CAAC,EAAE,CAAC;aACjF;YACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;gBACnB,YAAY,EAAE,WAAW;gBACzB,WAAW,EAAE,EAAE,WAAW,EAAE,CAAC,IAAA,gBAAO,EAAC,uBAAuB,CAAC,IAAI,SAAS,CAAQ,EAAE;gBACpF,IAAI;aACL,CAAC;SACH,CAAC,CAAC;QAEH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;YACZ,MAAM,IAAI,KAAK,CACb,GAAG,GAAG,CAAC,MAAM,KACX,GAAG,CAAC,UACN,4DAA4D,MAAM,GAAG,CAAC,IAAI,EAAE,EAAE,CAC/E,CAAC;QACJ,CAAC;QAED,OAAO,IAAA,2BAAmB,EAAC,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAAkB,CAAC,CAAC;IAClE,CAAC,CAAC;IAEF,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC;AACrC,CAAC"}
+{"version":3,"file":"code-tool.js","sourceRoot":"","sources":["src/code-tool.ts"],"names":[],"mappings":";AAAA,sFAAsF;;AAqCtF,4BAuDC;AA1FD,sCAAgG;AAEhG,wCAAmD;AAInD,MAAM,MAAM,GAAG;;;;;;;;;;;;;;;;;;uGAkBwF,CAAC;AAExG;;;;;;;;GAQG;AACH,SAAgB,QAAQ;IACtB,MAAM,QAAQ,GAAa,EAAE,QAAQ,EAAE,KAAK,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE,EAAE,EAAE,CAAC;IAC7E,MAAM,IAAI,GAAS;QACjB,IAAI,EAAE,SAAS;QACf,WAAW,EAAE,MAAM;QACnB,WAAW,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,UAAU,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,EAAE,EAAE;KAC1E,CAAC;IACF,MAAM,OAAO,GAAG,KAAK,EAAE,MAAiB,EAAE,IAAS,EAA2B,EAAE;QAC9E,MAAM,IAAI,GAAG,IAAI,CAAC,IAAc,CAAC;QAEjC,sFAAsF;QACtF,sFAAsF;QACtF,MAAM,eAAe,GAAG,IAAA,gBAAO,EAAC,mBAAmB,CAAC,CAAC;QACrD,MAAM,gBAAgB,GACpB,IAAA,gBAAO,EAAC,wBAAwB,CAAC,IAAI,4CAA4C,CAAC;QAEpF,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,gBAAgB,EAAE;YACxC,MAAM,EAAE,MAAM;YACd,OAAO,EAAE;gBACP,GAAG,CAAC,eAAe,IAAI,EAAE,aAAa,EAAE,eAAe,EAAE,CAAC;gBAC1D,cAAc,EAAE,kBAAkB;gBAClC,WAAW,EAAE,IAAI,CAAC,SAAS,CAAC;oBAC1B,iBAAiB,EAAE,MAAM,CAAC,MAAM,IAAI,IAAA,uBAAc,EAAC,mBAAmB,CAAC;oBACvE,kBAAkB,EAAE,MAAM,CAAC,OAAO,IAAI,IAAA,gBAAO,EAAC,oBAAoB,CAAC;iBACpE,CAAC;aACH;YACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;gBACnB,YAAY,EAAE,WAAW;gBACzB,IAAI;gBACJ,WAAW,EAAE,EAAE,WAAW,EAAE,CAAC,IAAA,gBAAO,EAAC,uBAAuB,CAAC,IAAI,SAAS,CAAQ,EAAE;aAC/D,CAAC;SACzB,CAAC,CAAC;QAEH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;YACZ,MAAM,IAAI,KAAK,CACb,GAAG,GAAG,CAAC,MAAM,KACX,GAAG,CAAC,UACN,4DAA4D,MAAM,GAAG,CAAC,IAAI,EAAE,EAAE,CAC/E,CAAC;QACJ,CAAC;QAED,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,SAAS,EAAE,SAAS,EAAE,GAAG,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAAiB,CAAC;QACtF,MAAM,OAAO,GAAG,SAAS,CAAC,MAAM,GAAG,CAAC,IAAI,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC;QAC7D,MAAM,MAAM,GAAG;YACb,MAAM;YACN,GAAG,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,SAAS,EAAE,CAAC;YAC1C,GAAG,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,SAAS,EAAE,CAAC;SAC3C,CAAC;QACF,IAAI,QAAQ,EAAE,CAAC;YACb,OAAO,IAAA,qBAAa,EAAC,OAAO,MAAM,KAAK,QAAQ,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;QAC1G,CAAC;QACD,OAAO,IAAA,2BAAmB,EAAC,MAAM,CAAC,CAAC;IACrC,CAAC,CAAC;IAEF,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC;AACrC,CAAC"}

package/code-tool.mjs

@@ -1,6 +1,25 @@
 // File generated from our OpenAPI spec by Stainless. See CONTRIBUTING.md for details.
-import { asTextContentResult } from "./tools/types.mjs";
-import { readEnv } from "./server.mjs";
+import { asErrorResult, asTextContentResult } from "./types.mjs";
+import { readEnv, readEnvOrError } from "./server.mjs";
+const prompt = `Runs JavaScript code to interact with the Scorecard API.
+
+You are a skilled programmer writing code to interface with the service.
+Define an async function named "run" that takes a single parameter of an initialized SDK client and it will be run.
+For example:
+
+\`\`\`
+async function run(client) {
+  const run = await client.runs.create('314', { metricIds: ['789', '101'], testsetId: '246' });
+
+  console.log(run.id);
+}
+\`\`\`
+
+You will be returned anything that your function returns, plus the results of any console.log statements.
+Do not add try-catch blocks for single API calls. The tool will handle errors for you.
+Do not add comments unless necessary for generating better code.
+Code will run in a container, and cannot interact with the network outside of the given SDK client.
+Variables will not persist between calls, so make sure to return or log any data you might need later.`;
 /**
  * A tool that runs code against a copy of the SDK.
  *
@@ -10,14 +29,14 @@ import { readEnv } from "./server.mjs";
  *
  * @param endpoints - The endpoints to include in the list.
  */
-export async function codeTool() {
+export function codeTool() {
     const metadata = { resource: 'all', operation: 'write', tags: [] };
     const tool = {
         name: 'execute',
-        description: 'Runs JavaScript code to interact with the API.\n\nYou are a skilled programmer writing code to interface with the service.\nDefine an async function named "run" that takes a single parameter of an initialized SDK client and it will be run.\nWrite code within this template:\n\n```\nasync function run(client) {\n  // Fill this out\n}\n```\n\nYou will be returned anything that your function returns, plus the results of any console.log statements.\nIf any code triggers an error, the tool will return an error response, so you do not need to add error handling unless you want to output something more helpful than the raw error.\nIt is not necessary to add comments to code, unless by adding those comments you believe that you can generate better code.\nThis code will run in a container, and you will not be able to use fetch or otherwise interact with the network calls other than through the client you are given.\nAny variables you define won\'t live between successive uses of this call, so make sure to return or log any data you might need later.',
+        description: prompt,
         inputSchema: { type: 'object', properties: { code: { type: 'string' } } },
     };
-    const handler = async (_, args) => {
+    const handler = async (client, args) => {
         const code = args.code;
         // this is not required, but passing a Stainless API key for the matching project_name
         // will allow you to run code-mode queries against non-published versions of your SDK.
@@ -28,18 +47,31 @@ export async function codeTool() {
             headers: {
                 ...(stainlessAPIKey && { Authorization: stainlessAPIKey }),
                 'Content-Type': 'application/json',
-                client_envs: JSON.stringify({ SCORECARD_API_KEY: readEnv('SCORECARD_API_KEY') }),
+                client_envs: JSON.stringify({
+                    SCORECARD_API_KEY: client.apiKey || readEnvOrError('SCORECARD_API_KEY'),
+                    SCORECARD_BASE_URL: client.baseURL || readEnv('SCORECARD_BASE_URL'),
+                }),
             },
             body: JSON.stringify({
                 project_name: 'scorecard',
-                client_opts: { environment: (readEnv('SCORECARD_ENVIRONMENT') || undefined) },
                 code,
+                client_opts: { environment: (readEnv('SCORECARD_ENVIRONMENT') || undefined) },
             }),
         });
         if (!res.ok) {
             throw new Error(`${res.status}: ${res.statusText} error when trying to contact Code Tool server. Details: ${await res.text()}`);
         }
-        return asTextContentResult((await res.json()));
+        const { is_error, result, log_lines, err_lines } = (await res.json());
+        const hasLogs = log_lines.length > 0 || err_lines.length > 0;
+        const output = {
+            result,
+            ...(log_lines.length > 0 && { log_lines }),
+            ...(err_lines.length > 0 && { err_lines }),
+        };
+        if (is_error) {
+            return asErrorResult(typeof result === 'string' && !hasLogs ? result : JSON.stringify(output, null, 2));
+        }
+        return asTextContentResult(output);
     };
     return { metadata, tool, handler };
 }

package/code-tool.mjs.map

@@ -1 +1 @@
-{"version":3,"file":"code-tool.mjs","sourceRoot":"","sources":["src/code-tool.ts"],"names":[],"mappings":"AAAA,sFAAsF;OAE/E,EAA4B,mBAAmB,EAAE;OAEjD,EAAE,OAAO,EAAE;AAElB;;;;;;;;GAQG;AACH,MAAM,CAAC,KAAK,UAAU,QAAQ;IAC5B,MAAM,QAAQ,GAAa,EAAE,QAAQ,EAAE,KAAK,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE,EAAE,EAAE,CAAC;IAC7E,MAAM,IAAI,GAAS;QACjB,IAAI,EAAE,SAAS;QACf,WAAW,EACT,iiCAAiiC;QACniC,WAAW,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,UAAU,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,EAAE,EAAE;KAC1E,CAAC;IACF,MAAM,OAAO,GAAG,KAAK,EAAE,CAAU,EAAE,IAAS,EAA2B,EAAE;QACvE,MAAM,IAAI,GAAG,IAAI,CAAC,IAAc,CAAC;QAEjC,sFAAsF;QACtF,sFAAsF;QACtF,MAAM,eAAe,GAAG,OAAO,CAAC,mBAAmB,CAAC,CAAC;QACrD,MAAM,gBAAgB,GACpB,OAAO,CAAC,wBAAwB,CAAC,IAAI,4CAA4C,CAAC;QAEpF,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,gBAAgB,EAAE;YACxC,MAAM,EAAE,MAAM;YACd,OAAO,EAAE;gBACP,GAAG,CAAC,eAAe,IAAI,EAAE,aAAa,EAAE,eAAe,EAAE,CAAC;gBAC1D,cAAc,EAAE,kBAAkB;gBAClC,WAAW,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,iBAAiB,EAAE,OAAO,CAAC,mBAAmB,CAAC,EAAE,CAAC;aACjF;YACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;gBACnB,YAAY,EAAE,WAAW;gBACzB,WAAW,EAAE,EAAE,WAAW,EAAE,CAAC,OAAO,CAAC,uBAAuB,CAAC,IAAI,SAAS,CAAQ,EAAE;gBACpF,IAAI;aACL,CAAC;SACH,CAAC,CAAC;QAEH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;YACZ,MAAM,IAAI,KAAK,CACb,GAAG,GAAG,CAAC,MAAM,KACX,GAAG,CAAC,UACN,4DAA4D,MAAM,GAAG,CAAC,IAAI,EAAE,EAAE,CAC/E,CAAC;QACJ,CAAC;QAED,OAAO,mBAAmB,CAAC,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAAkB,CAAC,CAAC;IAClE,CAAC,CAAC;IAEF,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC;AACrC,CAAC"}
+{"version":3,"file":"code-tool.mjs","sourceRoot":"","sources":["src/code-tool.ts"],"names":[],"mappings":"AAAA,sFAAsF;OAE/E,EAAqC,aAAa,EAAE,mBAAmB,EAAE;OAEzE,EAAE,OAAO,EAAE,cAAc,EAAE;AAIlC,MAAM,MAAM,GAAG;;;;;;;;;;;;;;;;;;uGAkBwF,CAAC;AAExG;;;;;;;;GAQG;AACH,MAAM,UAAU,QAAQ;IACtB,MAAM,QAAQ,GAAa,EAAE,QAAQ,EAAE,KAAK,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE,EAAE,EAAE,CAAC;IAC7E,MAAM,IAAI,GAAS;QACjB,IAAI,EAAE,SAAS;QACf,WAAW,EAAE,MAAM;QACnB,WAAW,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,UAAU,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,EAAE,EAAE;KAC1E,CAAC;IACF,MAAM,OAAO,GAAG,KAAK,EAAE,MAAiB,EAAE,IAAS,EAA2B,EAAE;QAC9E,MAAM,IAAI,GAAG,IAAI,CAAC,IAAc,CAAC;QAEjC,sFAAsF;QACtF,sFAAsF;QACtF,MAAM,eAAe,GAAG,OAAO,CAAC,mBAAmB,CAAC,CAAC;QACrD,MAAM,gBAAgB,GACpB,OAAO,CAAC,wBAAwB,CAAC,IAAI,4CAA4C,CAAC;QAEpF,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,gBAAgB,EAAE;YACxC,MAAM,EAAE,MAAM;YACd,OAAO,EAAE;gBACP,GAAG,CAAC,eAAe,IAAI,EAAE,aAAa,EAAE,eAAe,EAAE,CAAC;gBAC1D,cAAc,EAAE,kBAAkB;gBAClC,WAAW,EAAE,IAAI,CAAC,SAAS,CAAC;oBAC1B,iBAAiB,EAAE,MAAM,CAAC,MAAM,IAAI,cAAc,CAAC,mBAAmB,CAAC;oBACvE,kBAAkB,EAAE,MAAM,CAAC,OAAO,IAAI,OAAO,CAAC,oBAAoB,CAAC;iBACpE,CAAC;aACH;YACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;gBACnB,YAAY,EAAE,WAAW;gBACzB,IAAI;gBACJ,WAAW,EAAE,EAAE,WAAW,EAAE,CAAC,OAAO,CAAC,uBAAuB,CAAC,IAAI,SAAS,CAAQ,EAAE;aAC/D,CAAC;SACzB,CAAC,CAAC;QAEH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;YACZ,MAAM,IAAI,KAAK,CACb,GAAG,GAAG,CAAC,MAAM,KACX,GAAG,CAAC,UACN,4DAA4D,MAAM,GAAG,CAAC,IAAI,EAAE,EAAE,CAC/E,CAAC;QACJ,CAAC;QAED,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,SAAS,EAAE,SAAS,EAAE,GAAG,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAAiB,CAAC;QACtF,MAAM,OAAO,GAAG,SAAS,CAAC,MAAM,GAAG,CAAC,IAAI,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC;QAC7D,MAAM,MAAM,GAAG;YACb,MAAM;YACN,GAAG,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,SAAS,EAAE,CAAC;YAC1C,GAAG,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,SAAS,EAAE,CAAC;SAC3C,CAAC;QACF,IAAI,QAAQ,EAAE,CAAC;YACb,OAAO,aAAa,CAAC,OAAO,MAAM,KAAK,QAAQ,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;QAC1G,CAAC;QACD,OAAO,mBAAmB,CAAC,MAAM,CAAC,CAAC;IACrC,CAAC,CAAC;IAEF,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC;AACrC,CAAC"}