scorecard-ai 2.6.0 → 3.0.0-beta.0

package/CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog
 
+## 3.0.0-beta.0 (2026-01-14)
+
+Full Changelog: [v2.6.0...v3.0.0-beta.0](https://github.com/scorecard-ai/scorecard-node/compare/v2.6.0...v3.0.0-beta.0)
+
+### ⚠ BREAKING CHANGES
+
+* **mcp:** remove deprecated tool schemes
+* **mcp:** **Migration:** To migrate, simply modify the command used to invoke the MCP server. Currently, the only supported tool scheme is code mode. Now, starting the server with just `node /path/to/mcp/server` or `npx package-name` will invoke code tools: changing your command to one of these is likely all you will need to do.
+
+### Features
+
+* Add telemetry wrappers around Anthropic and OpenAI SDKs ([#31](https://github.com/scorecard-ai/scorecard-node/issues/31)) ([bb4dfe5](https://github.com/scorecard-ai/scorecard-node/commit/bb4dfe590993bfb07d28a1db96622f86654366d6))
+* **mcp:** Use Scorecard API key from MCP client when executing code tool ([741fbdd](https://github.com/scorecard-ai/scorecard-node/commit/741fbdd38c9becc69f4e7a6de845dcff556434df))
+
+
+### Bug Fixes
+
+* **mcp:** correct code tool api output types ([c3983d7](https://github.com/scorecard-ai/scorecard-node/commit/c3983d734ccbe019061105f99ba6b2b8727e8b0f))
+* **mcp:** fix env parsing ([22c7368](https://github.com/scorecard-ai/scorecard-node/commit/22c73681d18660d972a9836fe20926b5d402b534))
+* **mcp:** fix options parsing ([c19c6e3](https://github.com/scorecard-ai/scorecard-node/commit/c19c6e3a2da944a83b55419acb2c72ae9a1e1f9d))
+* **mcp:** pass base url to code tool ([2a855d6](https://github.com/scorecard-ai/scorecard-node/commit/2a855d651ec9260455d9b1172cf543f9a334c6b8))
+* **mcp:** update code tool prompt ([cec30f5](https://github.com/scorecard-ai/scorecard-node/commit/cec30f5761b9af71a10bdbe0a5aacff277c4a04b))
+
+
+### Chores
+
+* break long lines in snippets into multiline ([6eb4de7](https://github.com/scorecard-ai/scorecard-node/commit/6eb4de7c08e0f14041714c8fb93aba4b1ab8de1a))
+* **internal:** codegen related update ([e1a1b97](https://github.com/scorecard-ai/scorecard-node/commit/e1a1b97a6516933282ba19368dfcd4abb6adecb6))
+* **internal:** codegen related update ([5c31e8f](https://github.com/scorecard-ai/scorecard-node/commit/5c31e8f81cbc97a6ade6567e9342231b79333eb9))
+* **internal:** codegen related update ([d638ee7](https://github.com/scorecard-ai/scorecard-node/commit/d638ee7d5c8d6c7a61e98f261a4aec42f57d1202))
+* **internal:** codegen related update ([8a98749](https://github.com/scorecard-ai/scorecard-node/commit/8a987491b973353fb77457075e3dbbd5d462c8bc))
+* **internal:** codegen related update ([32953e6](https://github.com/scorecard-ai/scorecard-node/commit/32953e6cf9066fb0952863950cd63575bf52435c))
+* **mcp:** remove deprecated tool schemes ([7175afa](https://github.com/scorecard-ai/scorecard-node/commit/7175afa508ca459c54ade311ecb970b316c1cc87))
+
+
+### Documentation
+
+* prominently feature MCP server setup in root SDK readmes ([2c59b3f](https://github.com/scorecard-ai/scorecard-node/commit/2c59b3f39f3d14b27b2f13f6183437446c25bb54))
+
 ## 2.6.0 (2025-12-11)
 
 Full Changelog: [v2.5.0...v2.6.0](https://github.com/scorecard-ai/scorecard-node/compare/v2.5.0...v2.6.0)

package/LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2025 Scorecard
+   Copyright 2026 Scorecard
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

package/README.md

@@ -8,6 +8,15 @@ The REST API documentation can be found on [docs.scorecard.io](https://docs.scor
 
 It is generated with [Stainless](https://www.stainless.com/).
 
+## MCP Server
+
+Use the Scorecard MCP Server to enable AI assistants to interact with this API, allowing them to explore endpoints, make test requests, and use documentation to help integrate this SDK into your application.
+
+[![Add to Cursor](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/en-US/install-mcp?name=scorecard-ai-mcp&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsInNjb3JlY2FyZC1haS1tY3AiXX0)
+[![Install in VS Code](https://img.shields.io/badge/_-Add_to_VS_Code-blue?style=for-the-badge&logo=data:image/svg%2bxml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGZpbGw9Im5vbmUiIHZpZXdCb3g9IjAgMCA0MCA0MCI+PHBhdGggZmlsbD0iI0VFRSIgZmlsbC1ydWxlPSJldmVub2RkIiBkPSJNMzAuMjM1IDM5Ljg4NGEyLjQ5MSAyLjQ5MSAwIDAgMS0xLjc4MS0uNzNMMTIuNyAyNC43OGwtMy40NiAyLjYyNC0zLjQwNiAyLjU4MmExLjY2NSAxLjY2NSAwIDAgMS0xLjA4Mi4zMzggMS42NjQgMS42NjQgMCAwIDEtMS4wNDYtLjQzMWwtMi4yLTJhMS42NjYgMS42NjYgMCAwIDEgMC0yLjQ2M0w3LjQ1OCAyMCA0LjY3IDE3LjQ1MyAxLjUwNyAxNC41N2ExLjY2NSAxLjY2NSAwIDAgMSAwLTIuNDYzbDIuMi0yYTEuNjY1IDEuNjY1IDAgMCAxIDIuMTMtLjA5N2w2Ljg2MyA1LjIwOUwyOC40NTIuODQ0YTIuNDg4IDIuNDg4IDAgMCAxIDEuODQxLS43MjljLjM1MS4wMDkuNjk5LjA5MSAxLjAxOS4yNDVsOC4yMzYgMy45NjFhMi41IDIuNSAwIDAgMSAxLjQxNSAyLjI1M3YuMDk5LS4wNDVWMzMuMzd2LS4wNDUuMDk1YTIuNTAxIDIuNTAxIDAgMCAxLTEuNDE2IDIuMjU3bC04LjIzNSAzLjk2MWEyLjQ5MiAyLjQ5MiAwIDAgMS0xLjA3Ny4yNDZabS43MTYtMjguOTQ3LTExLjk0OCA5LjA2MiAxMS45NTIgOS4wNjUtLjAwNC0xOC4xMjdaIi8+PC9zdmc+)](https://vscode.stainless.com/mcp/%7B%22name%22%3A%22scorecard-ai-mcp%22%2C%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22scorecard-ai-mcp%22%5D%7D)
+
+> Note: You may need to set environment variables in your MCP client.
+
 ## Installation
 
 ```sh

package/index.d.mts

@@ -6,4 +6,5 @@ export { PagePromise } from "./core/pagination.mjs";
 export { ScorecardError, APIError, APIConnectionError, APIConnectionTimeoutError, APIUserAbortError, NotFoundError, ConflictError, RateLimitError, BadRequestError, AuthenticationError, InternalServerError, PermissionDeniedError, UnprocessableEntityError, } from "./core/error.mjs";
 export { runAndEvaluate } from "./lib/runAndEvaluate.mjs";
 export { wrapAISDK } from "./lib/wrapAISDK.mjs";
+export { wrap, wrapOpenAI, wrapAnthropic } from "./lib/wrapLLMs.mjs";
 //# sourceMappingURL=index.d.mts.map

package/index.d.mts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.mts","sourceRoot":"","sources":["src/index.ts"],"names":[],"mappings":"OAEO,EAAE,SAAS,IAAI,OAAO,EAAE;OAExB,EAAE,KAAK,UAAU,EAAE,MAAM,EAAE;OAC3B,EAAE,UAAU,EAAE;OACd,EAAE,SAAS,EAAE,KAAK,aAAa,EAAE;OACjC,EAAE,WAAW,EAAE;OACf,EACL,cAAc,EACd,QAAQ,EACR,kBAAkB,EAClB,yBAAyB,EACzB,iBAAiB,EACjB,aAAa,EACb,aAAa,EACb,cAAc,EACd,eAAe,EACf,mBAAmB,EACnB,mBAAmB,EACnB,qBAAqB,EACrB,wBAAwB,GACzB;OAEM,EAAE,cAAc,EAAE;OAClB,EAAE,SAAS,EAAE"}
+{"version":3,"file":"index.d.mts","sourceRoot":"","sources":["src/index.ts"],"names":[],"mappings":"OAEO,EAAE,SAAS,IAAI,OAAO,EAAE;OAExB,EAAE,KAAK,UAAU,EAAE,MAAM,EAAE;OAC3B,EAAE,UAAU,EAAE;OACd,EAAE,SAAS,EAAE,KAAK,aAAa,EAAE;OACjC,EAAE,WAAW,EAAE;OACf,EACL,cAAc,EACd,QAAQ,EACR,kBAAkB,EAClB,yBAAyB,EACzB,iBAAiB,EACjB,aAAa,EACb,aAAa,EACb,cAAc,EACd,eAAe,EACf,mBAAmB,EACnB,mBAAmB,EACnB,qBAAqB,EACrB,wBAAwB,GACzB;OAEM,EAAE,cAAc,EAAE;OAClB,EAAE,SAAS,EAAE;OACb,EAAE,IAAI,EAAE,UAAU,EAAE,aAAa,EAAE"}

package/index.d.ts

@@ -6,4 +6,5 @@ export { PagePromise } from "./core/pagination.js";
 export { ScorecardError, APIError, APIConnectionError, APIConnectionTimeoutError, APIUserAbortError, NotFoundError, ConflictError, RateLimitError, BadRequestError, AuthenticationError, InternalServerError, PermissionDeniedError, UnprocessableEntityError, } from "./core/error.js";
 export { runAndEvaluate } from "./lib/runAndEvaluate.js";
 export { wrapAISDK } from "./lib/wrapAISDK.js";
+export { wrap, wrapOpenAI, wrapAnthropic } from "./lib/wrapLLMs.js";
 //# sourceMappingURL=index.d.ts.map

package/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["src/index.ts"],"names":[],"mappings":"OAEO,EAAE,SAAS,IAAI,OAAO,EAAE;OAExB,EAAE,KAAK,UAAU,EAAE,MAAM,EAAE;OAC3B,EAAE,UAAU,EAAE;OACd,EAAE,SAAS,EAAE,KAAK,aAAa,EAAE;OACjC,EAAE,WAAW,EAAE;OACf,EACL,cAAc,EACd,QAAQ,EACR,kBAAkB,EAClB,yBAAyB,EACzB,iBAAiB,EACjB,aAAa,EACb,aAAa,EACb,cAAc,EACd,eAAe,EACf,mBAAmB,EACnB,mBAAmB,EACnB,qBAAqB,EACrB,wBAAwB,GACzB;OAEM,EAAE,cAAc,EAAE;OAClB,EAAE,SAAS,EAAE"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["src/index.ts"],"names":[],"mappings":"OAEO,EAAE,SAAS,IAAI,OAAO,EAAE;OAExB,EAAE,KAAK,UAAU,EAAE,MAAM,EAAE;OAC3B,EAAE,UAAU,EAAE;OACd,EAAE,SAAS,EAAE,KAAK,aAAa,EAAE;OACjC,EAAE,WAAW,EAAE;OACf,EACL,cAAc,EACd,QAAQ,EACR,kBAAkB,EAClB,yBAAyB,EACzB,iBAAiB,EACjB,aAAa,EACb,aAAa,EACb,cAAc,EACd,eAAe,EACf,mBAAmB,EACnB,mBAAmB,EACnB,qBAAqB,EACrB,wBAAwB,GACzB;OAEM,EAAE,cAAc,EAAE;OAClB,EAAE,SAAS,EAAE;OACb,EAAE,IAAI,EAAE,UAAU,EAAE,aAAa,EAAE"}

package/index.js

@@ -4,7 +4,7 @@ exports = module.exports = function (...args) {
   return new exports.default(...args)
 }
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.wrapAISDK = exports.runAndEvaluate = exports.UnprocessableEntityError = exports.PermissionDeniedError = exports.InternalServerError = exports.AuthenticationError = exports.BadRequestError = exports.RateLimitError = exports.ConflictError = exports.NotFoundError = exports.APIUserAbortError = exports.APIConnectionTimeoutError = exports.APIConnectionError = exports.APIError = exports.ScorecardError = exports.PagePromise = exports.Scorecard = exports.APIPromise = exports.toFile = exports.default = void 0;
+exports.wrapAnthropic = exports.wrapOpenAI = exports.wrap = exports.wrapAISDK = exports.runAndEvaluate = exports.UnprocessableEntityError = exports.PermissionDeniedError = exports.InternalServerError = exports.AuthenticationError = exports.BadRequestError = exports.RateLimitError = exports.ConflictError = exports.NotFoundError = exports.APIUserAbortError = exports.APIConnectionTimeoutError = exports.APIConnectionError = exports.APIError = exports.ScorecardError = exports.PagePromise = exports.Scorecard = exports.APIPromise = exports.toFile = exports.default = void 0;
 var client_1 = require("./client.js");
 Object.defineProperty(exports, "default", { enumerable: true, get: function () { return client_1.Scorecard; } });
 var uploads_1 = require("./core/uploads.js");
@@ -33,4 +33,8 @@ var runAndEvaluate_1 = require("./lib/runAndEvaluate.js");
 Object.defineProperty(exports, "runAndEvaluate", { enumerable: true, get: function () { return runAndEvaluate_1.runAndEvaluate; } });
 var wrapAISDK_1 = require("./lib/wrapAISDK.js");
 Object.defineProperty(exports, "wrapAISDK", { enumerable: true, get: function () { return wrapAISDK_1.wrapAISDK; } });
+var wrapLLMs_1 = require("./lib/wrapLLMs.js");
+Object.defineProperty(exports, "wrap", { enumerable: true, get: function () { return wrapLLMs_1.wrap; } });
+Object.defineProperty(exports, "wrapOpenAI", { enumerable: true, get: function () { return wrapLLMs_1.wrapOpenAI; } });
+Object.defineProperty(exports, "wrapAnthropic", { enumerable: true, get: function () { return wrapLLMs_1.wrapAnthropic; } });
 //# sourceMappingURL=index.js.map

package/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["src/index.ts"],"names":[],"mappings":";AAAA,sFAAsF;;;AAEtF,sCAAgD;AAAvC,iGAAA,SAAS,OAAW;AAE7B,6CAAyD;AAA/B,iGAAA,MAAM,OAAA;AAChC,qDAAgD;AAAvC,yGAAA,UAAU,OAAA;AACnB,sCAAyD;AAAhD,mGAAA,SAAS,OAAA;AAClB,mDAAgD;AAAvC,yGAAA,WAAW,OAAA;AACpB,yCAcsB;AAbpB,uGAAA,cAAc,OAAA;AACd,iGAAA,QAAQ,OAAA;AACR,2GAAA,kBAAkB,OAAA;AAClB,kHAAA,yBAAyB,OAAA;AACzB,0GAAA,iBAAiB,OAAA;AACjB,sGAAA,aAAa,OAAA;AACb,sGAAA,aAAa,OAAA;AACb,uGAAA,cAAc,OAAA;AACd,wGAAA,eAAe,OAAA;AACf,4GAAA,mBAAmB,OAAA;AACnB,4GAAA,mBAAmB,OAAA;AACnB,8GAAA,qBAAqB,OAAA;AACrB,iHAAA,wBAAwB,OAAA;AAG1B,0DAAsD;AAA7C,gHAAA,cAAc,OAAA;AACvB,gDAA4C;AAAnC,sGAAA,SAAS,OAAA"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["src/index.ts"],"names":[],"mappings":";AAAA,sFAAsF;;;AAEtF,sCAAgD;AAAvC,iGAAA,SAAS,OAAW;AAE7B,6CAAyD;AAA/B,iGAAA,MAAM,OAAA;AAChC,qDAAgD;AAAvC,yGAAA,UAAU,OAAA;AACnB,sCAAyD;AAAhD,mGAAA,SAAS,OAAA;AAClB,mDAAgD;AAAvC,yGAAA,WAAW,OAAA;AACpB,yCAcsB;AAbpB,uGAAA,cAAc,OAAA;AACd,iGAAA,QAAQ,OAAA;AACR,2GAAA,kBAAkB,OAAA;AAClB,kHAAA,yBAAyB,OAAA;AACzB,0GAAA,iBAAiB,OAAA;AACjB,sGAAA,aAAa,OAAA;AACb,sGAAA,aAAa,OAAA;AACb,uGAAA,cAAc,OAAA;AACd,wGAAA,eAAe,OAAA;AACf,4GAAA,mBAAmB,OAAA;AACnB,4GAAA,mBAAmB,OAAA;AACnB,8GAAA,qBAAqB,OAAA;AACrB,iHAAA,wBAAwB,OAAA;AAG1B,0DAAsD;AAA7C,gHAAA,cAAc,OAAA;AACvB,gDAA4C;AAAnC,sGAAA,SAAS,OAAA;AAClB,8CAAiE;AAAxD,gGAAA,IAAI,OAAA;AAAE,sGAAA,UAAU,OAAA;AAAE,yGAAA,aAAa,OAAA"}

package/index.mjs

@@ -7,4 +7,5 @@ export { PagePromise } from "./core/pagination.mjs";
 export { ScorecardError, APIError, APIConnectionError, APIConnectionTimeoutError, APIUserAbortError, NotFoundError, ConflictError, RateLimitError, BadRequestError, AuthenticationError, InternalServerError, PermissionDeniedError, UnprocessableEntityError, } from "./core/error.mjs";
 export { runAndEvaluate } from "./lib/runAndEvaluate.mjs";
 export { wrapAISDK } from "./lib/wrapAISDK.mjs";
+export { wrap, wrapOpenAI, wrapAnthropic } from "./lib/wrapLLMs.mjs";
 //# sourceMappingURL=index.mjs.map

package/index.mjs.map

@@ -1 +1 @@
-{"version":3,"file":"index.mjs","sourceRoot":"","sources":["src/index.ts"],"names":[],"mappings":"AAAA,sFAAsF;OAE/E,EAAE,SAAS,IAAI,OAAO,EAAE;OAExB,EAAmB,MAAM,EAAE;OAC3B,EAAE,UAAU,EAAE;OACd,EAAE,SAAS,EAAsB;OACjC,EAAE,WAAW,EAAE;OACf,EACL,cAAc,EACd,QAAQ,EACR,kBAAkB,EAClB,yBAAyB,EACzB,iBAAiB,EACjB,aAAa,EACb,aAAa,EACb,cAAc,EACd,eAAe,EACf,mBAAmB,EACnB,mBAAmB,EACnB,qBAAqB,EACrB,wBAAwB,GACzB;OAEM,EAAE,cAAc,EAAE;OAClB,EAAE,SAAS,EAAE"}
+{"version":3,"file":"index.mjs","sourceRoot":"","sources":["src/index.ts"],"names":[],"mappings":"AAAA,sFAAsF;OAE/E,EAAE,SAAS,IAAI,OAAO,EAAE;OAExB,EAAmB,MAAM,EAAE;OAC3B,EAAE,UAAU,EAAE;OACd,EAAE,SAAS,EAAsB;OACjC,EAAE,WAAW,EAAE;OACf,EACL,cAAc,EACd,QAAQ,EACR,kBAAkB,EAClB,yBAAyB,EACzB,iBAAiB,EACjB,aAAa,EACb,aAAa,EACb,cAAc,EACd,eAAe,EACf,mBAAmB,EACnB,mBAAmB,EACnB,qBAAqB,EACrB,wBAAwB,GACzB;OAEM,EAAE,cAAc,EAAE;OAClB,EAAE,SAAS,EAAE;OACb,EAAE,IAAI,EAAE,UAAU,EAAE,aAAa,EAAE"}

package/lib/wrapLLMs.d.mts

@@ -0,0 +1,63 @@
+/**
+ * Configuration for wrapping LLM SDKs
+ */
+interface WrapConfig {
+    /**
+     * ID of the Scorecard project that traces should be associated with.
+     * Defaults to SCORECARD_PROJECT_ID environment variable.
+     */
+    projectId?: string | undefined;
+    /**
+     * Scorecard API key for authentication.
+     * Defaults to SCORECARD_API_KEY environment variable.
+     */
+    apiKey?: string | undefined;
+    /**
+     * Service name for telemetry.
+     * Defaults to "llm-app".
+     */
+    serviceName?: string | undefined;
+    /**
+     * OTLP endpoint for trace export.
+     * Defaults to "https://tracing.scorecard.io/otel/v1/traces".
+     */
+    endpoint?: string | undefined;
+    /**
+     * Maximum batch size of spans to be exported in a single request.
+     * Lower values provide faster feedback but more network requests.
+     * Higher values are more efficient but delay span visibility.
+     * @default 1
+     */
+    maxExportBatchSize?: number | undefined;
+}
+/**
+ * Wrap any LLM SDK (OpenAI or Anthropic) to automatically trace all API calls
+ *
+ * @example
+ * ```typescript
+ * import { wrap } from '@scorecard/node';
+ * import OpenAI from 'openai';
+ * import Anthropic from '@anthropic-ai/sdk';
+ *
+ * // Works with OpenAI
+ * const openai = wrap(new OpenAI({ apiKey: '...' }), {
+ *   apiKey: process.env.SCORECARD_API_KEY,
+ *   projectId: '123'
+ * });
+ *
+ * // Works with Anthropic
+ * const claude = wrap(new Anthropic({ apiKey: '...' }), {
+ *   apiKey: process.env.SCORECARD_API_KEY,
+ *   projectId: '123'
+ * });
+ *
+ * // Use normally - traces are automatically sent to Scorecard
+ * const response = await openai.chat.completions.create({...});
+ * const response2 = await claude.messages.create({...});
+ * ```
+ */
+export declare function wrap<T>(client: T, config?: WrapConfig): T;
+export declare const wrapOpenAI: typeof wrap;
+export declare const wrapAnthropic: typeof wrap;
+export {};
+//# sourceMappingURL=wrapLLMs.d.mts.map

package/lib/wrapLLMs.d.mts.map

@@ -0,0 +1 @@
+{"version":3,"file":"wrapLLMs.d.mts","sourceRoot":"","sources":["../src/lib/wrapLLMs.ts"],"names":[],"mappings":"AAcA;;GAEG;AACH,UAAU,UAAU;IAClB;;;OAGG;IACH,SAAS,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAE/B;;;OAGG;IACH,MAAM,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAE5B;;;OAGG;IACH,WAAW,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAEjC;;;OAGG;IACH,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAE9B;;;;;OAKG;IACH,kBAAkB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CACzC;AAmTD;;;;;;;;;;;;;;;;;;;;;;;;;GAyBG;AACH,wBAAgB,IAAI,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,MAAM,GAAE,UAAe,GAAG,CAAC,CAkG7D;AAGD,eAAO,MAAM,UAAU,aAAO,CAAC;AAC/B,eAAO,MAAM,aAAa,aAAO,CAAC"}

package/lib/wrapLLMs.d.ts

@@ -0,0 +1,63 @@
+/**
+ * Configuration for wrapping LLM SDKs
+ */
+interface WrapConfig {
+    /**
+     * ID of the Scorecard project that traces should be associated with.
+     * Defaults to SCORECARD_PROJECT_ID environment variable.
+     */
+    projectId?: string | undefined;
+    /**
+     * Scorecard API key for authentication.
+     * Defaults to SCORECARD_API_KEY environment variable.
+     */
+    apiKey?: string | undefined;
+    /**
+     * Service name for telemetry.
+     * Defaults to "llm-app".
+     */
+    serviceName?: string | undefined;
+    /**
+     * OTLP endpoint for trace export.
+     * Defaults to "https://tracing.scorecard.io/otel/v1/traces".
+     */
+    endpoint?: string | undefined;
+    /**
+     * Maximum batch size of spans to be exported in a single request.
+     * Lower values provide faster feedback but more network requests.
+     * Higher values are more efficient but delay span visibility.
+     * @default 1
+     */
+    maxExportBatchSize?: number | undefined;
+}
+/**
+ * Wrap any LLM SDK (OpenAI or Anthropic) to automatically trace all API calls
+ *
+ * @example
+ * ```typescript
+ * import { wrap } from '@scorecard/node';
+ * import OpenAI from 'openai';
+ * import Anthropic from '@anthropic-ai/sdk';
+ *
+ * // Works with OpenAI
+ * const openai = wrap(new OpenAI({ apiKey: '...' }), {
+ *   apiKey: process.env.SCORECARD_API_KEY,
+ *   projectId: '123'
+ * });
+ *
+ * // Works with Anthropic
+ * const claude = wrap(new Anthropic({ apiKey: '...' }), {
+ *   apiKey: process.env.SCORECARD_API_KEY,
+ *   projectId: '123'
+ * });
+ *
+ * // Use normally - traces are automatically sent to Scorecard
+ * const response = await openai.chat.completions.create({...});
+ * const response2 = await claude.messages.create({...});
+ * ```
+ */
+export declare function wrap<T>(client: T, config?: WrapConfig): T;
+export declare const wrapOpenAI: typeof wrap;
+export declare const wrapAnthropic: typeof wrap;
+export {};
+//# sourceMappingURL=wrapLLMs.d.ts.map

package/lib/wrapLLMs.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"wrapLLMs.d.ts","sourceRoot":"","sources":["../src/lib/wrapLLMs.ts"],"names":[],"mappings":"AAcA;;GAEG;AACH,UAAU,UAAU;IAClB;;;OAGG;IACH,SAAS,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAE/B;;;OAGG;IACH,MAAM,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAE5B;;;OAGG;IACH,WAAW,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAEjC;;;OAGG;IACH,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAE9B;;;;;OAKG;IACH,kBAAkB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CACzC;AAmTD;;;;;;;;;;;;;;;;;;;;;;;;;GAyBG;AACH,wBAAgB,IAAI,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,MAAM,GAAE,UAAe,GAAG,CAAC,CAkG7D;AAGD,eAAO,MAAM,UAAU,aAAO,CAAC;AAC/B,eAAO,MAAM,aAAa,aAAO,CAAC"}

package/lib/wrapLLMs.js

@@ -0,0 +1,386 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.wrapAnthropic = exports.wrapOpenAI = void 0;
+exports.wrap = wrap;
+const api_1 = require("@opentelemetry/api");
+const sdk_trace_node_1 = require("@opentelemetry/sdk-trace-node");
+const exporter_trace_otlp_http_1 = require("@opentelemetry/exporter-trace-otlp-http");
+const resources_1 = require("@opentelemetry/resources");
+const semantic_conventions_1 = require("@opentelemetry/semantic-conventions");
+const utils_1 = require("../internal/utils.js");
+const error_1 = require("../error.js");
+/**
+ * Custom exporter that wraps OTLP exporter and injects projectId from span attributes
+ * into the resource before export. This allows per-span projectId while keeping
+ * ResourceAttributes where the backend expects them.
+ */
+class ScorecardExporter extends exporter_trace_otlp_http_1.OTLPTraceExporter {
+    export(spans, resultCallback) {
+        // For each span, inject all scorecard.* attributes into the resource
+        spans.forEach((span) => {
+            // Collect all scorecard.* attributes from span attributes
+            const scorecardAttrs = Object.entries(span.attributes).reduce((acc, [key, value]) => {
+                if (key.startsWith('scorecard.')) {
+                    acc[key] = value;
+                }
+                return acc;
+            }, {});
+            if (Object.keys(scorecardAttrs).length > 0) {
+                // Merge all scorecard.* attributes into the resource
+                const newResource = span.resource.merge((0, resources_1.resourceFromAttributes)(scorecardAttrs));
+                // Directly assign the new resource (cast to any to bypass readonly)
+                span.resource = newResource;
+            }
+        });
+        // Call the parent exporter with modified spans
+        super.export(spans, resultCallback);
+    }
+}
+/**
+ * Composite processor that forwards span events to all registered processors.
+ * Allows dynamic addition of exporters after provider registration.
+ */
+class CompositeProcessor {
+    constructor() {
+        this.processors = new Map();
+    }
+    addProcessor(apiKey, endpoint, maxExportBatchSize) {
+        const key = `${apiKey}:${endpoint}`;
+        if (this.processors.has(key))
+            return;
+        const exporter = new ScorecardExporter({
+            url: endpoint,
+            headers: { Authorization: `Bearer ${apiKey}` },
+        });
+        const processor = new sdk_trace_node_1.BatchSpanProcessor(exporter, {
+            maxExportBatchSize,
+        });
+        this.processors.set(key, processor);
+    }
+    onStart(span, parentContext) {
+        for (const processor of this.processors.values()) {
+            processor.onStart(span, parentContext);
+        }
+    }
+    onEnd(span) {
+        for (const processor of this.processors.values()) {
+            processor.onEnd(span);
+        }
+    }
+    async forceFlush() {
+        await Promise.all(Array.from(this.processors.values()).map((p) => p.forceFlush()));
+    }
+    async shutdown() {
+        await Promise.all(Array.from(this.processors.values()).map((p) => p.shutdown()));
+    }
+}
+let globalProvider = null;
+let globalTracer = null;
+let compositeProcessor = null;
+/**
+ * Initialize OpenTelemetry provider for LLM SDK wrappers.
+ * Creates a single global provider for nesting support, with exporters
+ * added dynamically for each unique apiKey+endpoint combination.
+ */
+function initProvider(config) {
+    const apiKey = config.apiKey || (0, utils_1.readEnv)('SCORECARD_API_KEY');
+    if (!apiKey) {
+        throw new error_1.ScorecardError('Scorecard API key is required. Set SCORECARD_API_KEY environment variable or pass apiKey in config.');
+    }
+    const endpoint = config.endpoint || 'https://tracing.scorecard.io/otel/v1/traces';
+    const serviceName = config.serviceName || 'llm-app';
+    const projectId = config.projectId || (0, utils_1.readEnv)('SCORECARD_PROJECT_ID');
+    const maxExportBatchSize = config.maxExportBatchSize ?? 1;
+    // Create the global provider once (enables span nesting)
+    if (!globalProvider) {
+        compositeProcessor = new CompositeProcessor();
+        const resource = (0, resources_1.resourceFromAttributes)({
+            [semantic_conventions_1.ATTR_SERVICE_NAME]: serviceName,
+        });
+        globalProvider = new sdk_trace_node_1.NodeTracerProvider({
+            resource,
+            spanProcessors: [compositeProcessor],
+        });
+        globalProvider.register();
+        globalTracer = api_1.trace.getTracer('scorecard-llm');
+    }
+    // Add an exporter for this specific apiKey+endpoint (if not already added)
+    compositeProcessor?.addProcessor(apiKey, endpoint, maxExportBatchSize);
+    return projectId;
+}
+/**
+ * Detect which LLM provider the client is for
+ */
+function detectProvider(client) {
+    // Check for OpenAI SDK structure
+    if (client.chat?.completions) {
+        return 'openai';
+    }
+    // Check for Anthropic SDK structure
+    if (client.messages) {
+        return 'anthropic';
+    }
+    throw new error_1.ScorecardError('Unable to detect LLM provider. Client must be an OpenAI or Anthropic SDK instance.');
+}
+/**
+ * Handle OpenAI-specific response parsing
+ */
+function handleOpenAIResponse(span, result, params) {
+    span.setAttributes({
+        'gen_ai.response.id': result.id || 'unknown',
+        'gen_ai.response.model': result.model || params.model || 'unknown',
+        'gen_ai.response.finish_reason': result.choices?.[0]?.finish_reason || 'unknown',
+        'gen_ai.usage.prompt_tokens': result.usage?.prompt_tokens || 0,
+        'gen_ai.usage.completion_tokens': result.usage?.completion_tokens || 0,
+        'gen_ai.usage.total_tokens': result.usage?.total_tokens || 0,
+    });
+    if (result.choices?.[0]?.message) {
+        span.setAttribute('gen_ai.completion.choices', JSON.stringify([result.choices[0].message]));
+    }
+}
+/**
+ * Handle Anthropic-specific response parsing
+ */
+function handleAnthropicResponse(span, result, params) {
+    span.setAttributes({
+        'gen_ai.response.id': result.id || 'unknown',
+        'gen_ai.response.model': result.model || params.model || 'unknown',
+        'gen_ai.response.finish_reason': result.stop_reason || 'unknown',
+        'gen_ai.usage.prompt_tokens': result.usage?.input_tokens || 0,
+        'gen_ai.usage.completion_tokens': result.usage?.output_tokens || 0,
+        'gen_ai.usage.total_tokens': (result.usage?.input_tokens || 0) + (result.usage?.output_tokens || 0),
+    });
+    // Collect text from all text blocks (Anthropic can return multiple content blocks)
+    if (result.content) {
+        const completionTexts = result.content.filter((c) => c.text).map((c) => c.text);
+        if (completionTexts.length > 0) {
+            span.setAttribute('gen_ai.completion.choices', JSON.stringify([{ message: { role: 'assistant', content: completionTexts.join('\n') } }]));
+        }
+    }
+}
+/**
+ * Wrapper for async streams that collects metadata and ends span when consumed
+ */
+class StreamWrapper {
+    constructor(stream, span, provider, params) {
+        this.stream = stream;
+        this.span = span;
+        this.provider = provider;
+        this.params = params;
+        this.contentParts = [];
+        this.finishReason = null;
+        this.usageData = {};
+        this.responseId = null;
+        this.model = null;
+    }
+    async *[Symbol.asyncIterator]() {
+        try {
+            for await (const chunk of this.stream) {
+                this.processChunk(chunk);
+                yield chunk;
+            }
+        }
+        finally {
+            this.finalizeSpan();
+        }
+    }
+    processChunk(chunk) {
+        // OpenAI streaming format
+        if (this.provider === 'openai') {
+            if (!this.responseId && chunk.id) {
+                this.responseId = chunk.id;
+            }
+            if (!this.model && chunk.model) {
+                this.model = chunk.model;
+            }
+            if (chunk.choices?.[0]) {
+                const choice = chunk.choices[0];
+                if (choice.delta?.content) {
+                    this.contentParts.push(choice.delta.content);
+                }
+                if (choice.finish_reason) {
+                    this.finishReason = choice.finish_reason;
+                }
+            }
+            // OpenAI includes usage in the last chunk with stream_options
+            if (chunk.usage) {
+                this.usageData = {
+                    prompt_tokens: chunk.usage.prompt_tokens || 0,
+                    completion_tokens: chunk.usage.completion_tokens || 0,
+                    total_tokens: chunk.usage.total_tokens || 0,
+                };
+            }
+        }
+        // Anthropic streaming format
+        else if (this.provider === 'anthropic') {
+            if (chunk.type === 'message_start' && chunk.message) {
+                this.responseId = chunk.message.id;
+                this.model = chunk.message.model;
+                if (chunk.message.usage) {
+                    this.usageData['input_tokens'] = chunk.message.usage.input_tokens || 0;
+                }
+            }
+            else if (chunk.type === 'content_block_delta' && chunk.delta?.text) {
+                this.contentParts.push(chunk.delta.text);
+            }
+            else if (chunk.type === 'message_delta') {
+                if (chunk.delta?.stop_reason) {
+                    this.finishReason = chunk.delta.stop_reason;
+                }
+                if (chunk.usage?.output_tokens) {
+                    this.usageData['output_tokens'] = chunk.usage.output_tokens;
+                }
+            }
+        }
+    }
+    finalizeSpan() {
+        // Set response attributes
+        this.span.setAttributes({
+            'gen_ai.response.id': this.responseId || 'unknown',
+            'gen_ai.response.model': this.model || this.params.model || 'unknown',
+            'gen_ai.response.finish_reason': this.finishReason || 'unknown',
+        });
+        // Set usage data
+        if (Object.keys(this.usageData).length > 0) {
+            if (this.provider === 'openai') {
+                this.span.setAttributes({
+                    'gen_ai.usage.prompt_tokens': this.usageData['prompt_tokens'] || 0,
+                    'gen_ai.usage.completion_tokens': this.usageData['completion_tokens'] || 0,
+                    'gen_ai.usage.total_tokens': this.usageData['total_tokens'] || 0,
+                });
+            }
+            else if (this.provider === 'anthropic') {
+                const inputTokens = this.usageData['input_tokens'] || 0;
+                const outputTokens = this.usageData['output_tokens'] || 0;
+                this.span.setAttributes({
+                    'gen_ai.usage.prompt_tokens': inputTokens,
+                    'gen_ai.usage.completion_tokens': outputTokens,
+                    'gen_ai.usage.total_tokens': inputTokens + outputTokens,
+                });
+            }
+        }
+        // Set completion content if any was collected
+        if (this.contentParts.length > 0) {
+            const content = this.contentParts.join('');
+            this.span.setAttribute('gen_ai.completion.choices', JSON.stringify([{ message: { role: 'assistant', content } }]));
+        }
+        this.span.end();
+    }
+}
+/**
+ * Wrap any LLM SDK (OpenAI or Anthropic) to automatically trace all API calls
+ *
+ * @example
+ * ```typescript
+ * import { wrap } from '@scorecard/node';
+ * import OpenAI from 'openai';
+ * import Anthropic from '@anthropic-ai/sdk';
+ *
+ * // Works with OpenAI
+ * const openai = wrap(new OpenAI({ apiKey: '...' }), {
+ *   apiKey: process.env.SCORECARD_API_KEY,
+ *   projectId: '123'
+ * });
+ *
+ * // Works with Anthropic
+ * const claude = wrap(new Anthropic({ apiKey: '...' }), {
+ *   apiKey: process.env.SCORECARD_API_KEY,
+ *   projectId: '123'
+ * });
+ *
+ * // Use normally - traces are automatically sent to Scorecard
+ * const response = await openai.chat.completions.create({...});
+ * const response2 = await claude.messages.create({...});
+ * ```
+ */
+function wrap(client, config = {}) {
+    const projectId = initProvider(config);
+    if (!globalTracer) {
+        throw new error_1.ScorecardError('Failed to initialize tracer');
+    }
+    const tracer = globalTracer;
+    const provider = detectProvider(client);
+    // Track the path to determine if we should wrap this method
+    const createHandler = (target, path = []) => ({
+        get(target, prop) {
+            const value = target[prop];
+            // Check if this is a method we should wrap based on the path
+            const currentPath = [...path, prop.toString()];
+            const shouldWrap = (provider === 'openai' && currentPath.join('.') === 'chat.completions.create') ||
+                (provider === 'anthropic' &&
+                    (currentPath.join('.') === 'messages.create' || currentPath.join('.') === 'messages.stream'));
+            // Intercept specific LLM methods
+            if (shouldWrap && typeof value === 'function') {
+                return async function (...args) {
+                    const params = args[0] || {};
+                    // Streaming if: 1) stream param is true, or 2) using the 'stream' method
+                    const isStreaming = params.stream === true || prop === 'stream';
+                    // Start span in the current active context (enables nesting)
+                    const span = tracer.startSpan(`${provider}.request`, {}, api_1.context.active());
+                    // Set request attributes (common to both providers)
+                    span.setAttributes({
+                        'gen_ai.system': provider,
+                        'gen_ai.request.model': params.model || 'unknown',
+                        'gen_ai.operation.name': 'chat',
+                        ...(params.temperature !== undefined && { 'gen_ai.request.temperature': params.temperature }),
+                        ...(params.max_tokens !== undefined && { 'gen_ai.request.max_tokens': params.max_tokens }),
+                        ...(params.top_p !== undefined && { 'gen_ai.request.top_p': params.top_p }),
+                    });
+                    // Store projectId as span attribute - our custom exporter will inject it
+                    // into ResourceAttributes before export (where the backend expects it)
+                    if (projectId) {
+                        span.setAttribute('scorecard.project_id', projectId);
+                    }
+                    // Set prompt messages
+                    if (params.messages) {
+                        span.setAttribute('gen_ai.prompt.messages', JSON.stringify(params.messages));
+                    }
+                    // Execute within the span's context (enables nested spans to be children)
+                    return api_1.context.with(api_1.trace.setSpan(api_1.context.active(), span), async () => {
+                        try {
+                            const result = await value.apply(target, args);
+                            if (isStreaming) {
+                                // For streaming, wrap the stream to collect metadata and end span when consumed
+                                return new StreamWrapper(result, span, provider, params);
+                            }
+                            else {
+                                // For non-streaming, set response attributes immediately
+                                if (provider === 'openai') {
+                                    handleOpenAIResponse(span, result, params);
+                                }
+                                else if (provider === 'anthropic') {
+                                    handleAnthropicResponse(span, result, params);
+                                }
+                                return result;
+                            }
+                        }
+                        catch (error) {
+                            span.recordException(error);
+                            throw error;
+                        }
+                        finally {
+                            // Only end span for non-streaming (streaming ends in StreamWrapper)
+                            if (!isStreaming) {
+                                span.end();
+                            }
+                        }
+                    });
+                };
+            }
+            // Recursively proxy nested objects, passing the path along
+            if (value && typeof value === 'object') {
+                return new Proxy(value, createHandler(value, currentPath));
+            }
+            // Return functions and primitives as-is
+            if (typeof value === 'function') {
+                return value.bind(target);
+            }
+            return value;
+        },
+    });
+    return new Proxy(client, createHandler(client, []));
+}
+// Backwards compatibility aliases
+exports.wrapOpenAI = wrap;
+exports.wrapAnthropic = wrap;
+//# sourceMappingURL=wrapLLMs.js.map