scorecard-ai 0.6.1 → 1.0.0-alpha.1

package/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+## 1.0.0-alpha.1 (2025-04-25)
+
+Full Changelog: [v1.0.0-alpha.0...v1.0.0-alpha.1](https://github.com/scorecard-ai/scorecard-node/compare/v1.0.0-alpha.0...v1.0.0-alpha.1)
+
+### Features
+
+* **api:** api update ([f0968b6](https://github.com/scorecard-ai/scorecard-node/commit/f0968b69700ec4a8058b516cf58897ef212514c2))
+* **api:** api update ([b056fe5](https://github.com/scorecard-ai/scorecard-node/commit/b056fe595ba94de7267b4978dc1dc223d328ed9e))
+* **api:** api update ([70f6d38](https://github.com/scorecard-ai/scorecard-node/commit/70f6d3866dcce23550b7765286a93df225dc566b))
+* **api:** api update ([3150074](https://github.com/scorecard-ai/scorecard-node/commit/3150074e245f3f375a51f80912acf3bf6ef87c26))
+* **api:** api update ([6952b48](https://github.com/scorecard-ai/scorecard-node/commit/6952b48af87c0c27fea668c2a110f2f9177c1d2d))
+* **api:** api update ([4e2d7d8](https://github.com/scorecard-ai/scorecard-node/commit/4e2d7d8bf812491ca94adb4fc39c4cdaca118d52))
+* **api:** api update ([3a57bbd](https://github.com/scorecard-ai/scorecard-node/commit/3a57bbd8e0ff7b553431819c141a82c7f2dc8104))
+* **api:** api update ([0c18f83](https://github.com/scorecard-ai/scorecard-node/commit/0c18f8326fa85f2a14a996c330f496c367fa7c36))
+* **api:** api update ([676b10f](https://github.com/scorecard-ai/scorecard-node/commit/676b10f7b89806e6fec4b29e3f9692e0bdef0583))
+* **api:** api update ([c033583](https://github.com/scorecard-ai/scorecard-node/commit/c033583c79940151a8bfdbb02cef44e263715c5b))
+* **api:** fix bulk testcases delete endpoint ([22a5206](https://github.com/scorecard-ai/scorecard-node/commit/22a5206d3a408dc3436b432d926b2d9298387d1e))
+
+
+### Chores
+
+* **ci:** add timeout thresholds for CI jobs ([be60867](https://github.com/scorecard-ai/scorecard-node/commit/be60867f34e9c551db0498af5a0f0d461a807027))
+* **ci:** only use depot for staging repos ([6009e6a](https://github.com/scorecard-ai/scorecard-node/commit/6009e6ab7fd194f66ce990045e22922c12f90de2))
+* **client:** minor internal fixes ([28b041e](https://github.com/scorecard-ai/scorecard-node/commit/28b041ef0610a90076088b0447fd5ba8b8b67919))
+* go live ([4057bd9](https://github.com/scorecard-ai/scorecard-node/commit/4057bd9a512260710cd9cb6456175ebd6322964a))
+* **internal:** codegen related update ([8d29dbd](https://github.com/scorecard-ai/scorecard-node/commit/8d29dbd1d0737b35bc41b5e17aabcbe3e8e0c899))
+* **perf:** faster base64 decoding ([a3208e7](https://github.com/scorecard-ai/scorecard-node/commit/a3208e7461705c8525f4bf2bdf18d874266e85f9))
+* update SDK settings ([3881b0f](https://github.com/scorecard-ai/scorecard-node/commit/3881b0f941193cdc7711606190d2072af858066a))
+* update SDK settings ([fc0a269](https://github.com/scorecard-ai/scorecard-node/commit/fc0a269e21f52ba1dcc6f08df0e4f4c95b7e32a5))
+* update SDK settings ([affa597](https://github.com/scorecard-ai/scorecard-node/commit/affa5972d77874a20cd6ccd9c7a538bdd865f453))
+* update SDK settings ([7d9cb03](https://github.com/scorecard-ai/scorecard-node/commit/7d9cb03544331910b1973869575df2bb123a2385))
+* update SDK settings ([358f860](https://github.com/scorecard-ai/scorecard-node/commit/358f86027f87953c19ee5ff9dd6d00f99407ce1a))
+* update SDK settings ([08e6f1d](https://github.com/scorecard-ai/scorecard-node/commit/08e6f1d25f1f379ab8ff7083fad9d57ed6cedfc4))
+* update SDK settings ([5523808](https://github.com/scorecard-ai/scorecard-node/commit/55238082ca6d74ee46191fe1190155c11fbfe2a7))
+* update SDK settings ([8c98a2f](https://github.com/scorecard-ai/scorecard-node/commit/8c98a2f036724a0208106988465eafb1be898351))
+* update SDK settings ([288b6a1](https://github.com/scorecard-ai/scorecard-node/commit/288b6a16de6cf78b888420225b6b6b220c44d356))
+
+## 1.0.0-alpha.0 (2025-04-11)
+
+Full Changelog: [v0.0.1-alpha.0...v1.0.0-alpha.0](https://github.com/scorecard-ai/scorecard-node/compare/v0.0.1-alpha.0...v1.0.0-alpha.0)
+
+### Bug Fixes
+
+* update README examples ([86a1f18](https://github.com/scorecard-ai/scorecard-node/commit/86a1f188ac1f99612255b67682009256fe388ac1))
+
+
+### Chores
+
+* go live ([02f09a8](https://github.com/scorecard-ai/scorecard-node/commit/02f09a8378c8b08d8b805500dbfcc3d3167067cc))
+* sync repo ([b26456f](https://github.com/scorecard-ai/scorecard-node/commit/b26456fabe267c35e206d4340d059d4f5f0f9a4b))
+* update SDK settings ([fe6f241](https://github.com/scorecard-ai/scorecard-node/commit/fe6f241fcc96af91638fb176491d6187155d0894))

package/LICENSE

@@ -1,4 +1,4 @@
-                                Apache License
+                                 Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
 
@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright [yyyy] [name of copyright owner]
+   Copyright 2025 Scorecard
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

package/README.md

@@ -1,100 +1,378 @@
-# Scorecard AI Node Library
+# Scorecard TypeScript API Library
 
-[![npm shield](https://img.shields.io/npm/v/scorecard-ai)](https://www.npmjs.com/package/scorecard-ai)
-[![fern shield](https://img.shields.io/badge/%F0%9F%8C%BF-SDK%20generated%20by%20Fern-brightgreen)](https://github.com/fern-api/fern)
+[![NPM version](https://img.shields.io/npm/v/scorecard-ai.svg)](https://npmjs.org/package/scorecard-ai) ![npm bundle size](https://img.shields.io/bundlephobia/minzip/scorecard-ai)
+
+This library provides convenient access to the Scorecard REST API from server-side TypeScript or JavaScript.
+
+The REST API documentation can be found on [docs.scorecard.io](https://docs.scorecard.io). The full API of this library can be found in [api.md](api.md).
+
+It is generated with [Stainless](https://www.stainless.com/).
 
 ## Installation
 
-```bash
+```sh
 npm install scorecard-ai
-# or
-yarn add scorecard-ai
 ```
 
 ## Usage
-Instantiate our client and start making calls to the API. 
-The SDK uses fetch and is compatible in multiple environments
-such as Node, Vercel, Cloudflare Workers. 
-
-```typescript
-import { ScorecardClient } from "scorecard-ai";
-
-const scorecard = new ScorecardClient({
-  apiKey: "YOUR_API_KEY"
-})
-
-const testset = await scorecard.testset.create({
-  testsetId=1234, 
-  userQuery="Your prompt...", 
-  modelParams: {
-    "param": "value"
-  }
-})
-
-console.log("Created testset", testset)
-```
-
-## Running Tests
-The `runTests` method will load your test cases, 
-invoke your models with saved prompts, 
-and record success and failure. This is 
-available on both the sync and async client.  
-
-```typescript
-import { callModel } from "../app.ts";
-import { ScorecardClient } from "scorecard-ai";
-
-const scorecard = new ScorecardClient({
-  apiKey: "YOUR_API_KEY"
-})
-
-client.runTests({
-  inputTestsetId: 123,
-  scoringConfigId: 456,
-  // The model invocation that you would like to test
-  model_invocation: () => callModel(prompt),
-})
-```
-
-## Timeouts
-By default, the client is configured to have 
-a timeout of 60 seconds. You can customize 
-this value by passing in request options. 
-
-```typescript
-await scorecard.testset.create({
-    testsetId=1234, 
-    ...
-  }, 
-  {
-    timeoutInSeconds: 30, // override timeout to 30s
+
+The full API of this library can be found in [api.md](api.md).
+
+<!-- prettier-ignore -->
+```js
+import Scorecard from 'scorecard-ai';
+
+const client = new Scorecard({
+  bearerToken: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted
+  environment: 'staging', // or 'production' | 'local'; defaults to 'production'
+});
+
+async function main() {
+  const testset = await client.testsets.create('projectId', {
+    description: 'Testset for long context Q&A chatbot.',
+    fieldMapping: { inputs: ['question'], labels: ['idealAnswer'], metadata: ['string'] },
+    jsonSchema: { type: 'bar', properties: 'bar' },
+    name: 'Long Context Q&A',
   });
+
+  console.log(testset.id);
+}
+
+main();
 ```
 
-## Handling Exceptions
-All exceptions thrown by the SDK will 
-sublcass [ScorecardError](./src/errors/ScorecardError.ts). 
+### Request & Response types
+
+This library includes TypeScript definitions for all request params and response fields. You may import and use them like so:
+
+<!-- prettier-ignore -->
+```ts
+import Scorecard from 'scorecard-ai';
 
-```typescript
-import { Scorecard, ScorecardError } from "scorecard-ai";
+const client = new Scorecard({
+  bearerToken: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted
+  environment: 'staging', // or 'production' | 'local'; defaults to 'production'
+});
 
-try {
-  await client.testset.get({...});
-} catch(err) {
-  if (err instanceof Scorecard.UnprocessableEntityError) {
-    // handle unprocessable entity error
-  } else if (err instanceof ScorecardError) {
-    // handle any sdk related error
-  }
+async function main() {
+  const testset: Scorecard.Testset = await client.testsets.get('testsetId');
 }
+
+main();
 ```
 
-## Beta status
+Documentation for each method, request param, and response field are available in docstrings and will appear on hover in most modern editors.
 
-This SDK is in beta, and there may be breaking changes between versions without a major version update. Therefore, we recommend pinning the package version to a specific version in your pyproject.toml file. This way, you can install the same version each time without breaking changes unless you are intentionally looking for the latest version.
+## Handling errors
 
-## Contributing
+When the library is unable to connect to the API,
+or if the API returns a non-success status code (i.e., 4xx or 5xx response),
+a subclass of `APIError` will be thrown:
+
+<!-- prettier-ignore -->
+```ts
+async function main() {
+  const testset = await client.testsets.get('testsetId').catch(async (err) => {
+    if (err instanceof Scorecard.APIError) {
+      console.log(err.status); // 400
+      console.log(err.name); // BadRequestError
+      console.log(err.headers); // {server: 'nginx', ...}
+    } else {
+      throw err;
+    }
+  });
+}
+
+main();
+```
+
+Error codes are as followed:
+
+| Status Code | Error Type                 |
+| ----------- | -------------------------- |
+| 400         | `BadRequestError`          |
+| 401         | `AuthenticationError`      |
+| 403         | `PermissionDeniedError`    |
+| 404         | `NotFoundError`            |
+| 422         | `UnprocessableEntityError` |
+| 429         | `RateLimitError`           |
+| >=500       | `InternalServerError`      |
+| N/A         | `APIConnectionError`       |
+
+### Retries
+
+Certain errors will be automatically retried 2 times by default, with a short exponential backoff.
+Connection errors (for example, due to a network connectivity problem), 408 Request Timeout, 409 Conflict,
+429 Rate Limit, and >=500 Internal errors will all be retried by default.
+
+You can use the `maxRetries` option to configure or disable this:
+
+<!-- prettier-ignore -->
+```js
+// Configure the default for all requests:
+const client = new Scorecard({
+  maxRetries: 0, // default is 2
+});
+
+// Or, configure per-request:
+await client.testsets.get('testsetId', {
+  maxRetries: 5,
+});
+```
+
+### Timeouts
+
+Requests time out after 1 minute by default. You can configure this with a `timeout` option:
+
+<!-- prettier-ignore -->
+```ts
+// Configure the default for all requests:
+const client = new Scorecard({
+  timeout: 20 * 1000, // 20 seconds (default is 1 minute)
+});
+
+// Override per-request:
+await client.testsets.get('testsetId', {
+  timeout: 5 * 1000,
+});
+```
+
+On timeout, an `APIConnectionTimeoutError` is thrown.
+
+Note that requests which time out will be [retried twice by default](#retries).
+
+## Advanced Usage
+
+### Accessing raw Response data (e.g., headers)
+
+The "raw" `Response` returned by `fetch()` can be accessed through the `.asResponse()` method on the `APIPromise` type that all methods return.
+This method returns as soon as the headers for a successful response are received and does not consume the response body, so you are free to write custom parsing or streaming logic.
+
+You can also use the `.withResponse()` method to get the raw `Response` along with the parsed data.
+Unlike `.asResponse()` this method consumes the body, returning once it is parsed.
+
+<!-- prettier-ignore -->
+```ts
+const client = new Scorecard();
+
+const response = await client.testsets.get('testsetId').asResponse();
+console.log(response.headers.get('X-My-Header'));
+console.log(response.statusText); // access the underlying Response object
+
+const { data: testset, response: raw } = await client.testsets.get('testsetId').withResponse();
+console.log(raw.headers.get('X-My-Header'));
+console.log(testset.id);
+```
+
+### Logging
+
+> [!IMPORTANT]
+> All log messages are intended for debugging only. The format and content of log messages
+> may change between releases.
+
+#### Log levels
+
+The log level can be configured in two ways:
+
+1. Via the `SCORECARD_LOG` environment variable
+2. Using the `logLevel` client option (overrides the environment variable if set)
+
+```ts
+import Scorecard from 'scorecard-ai';
+
+const client = new Scorecard({
+  logLevel: 'debug', // Show all log messages
+});
+```
+
+Available log levels, from most to least verbose:
+
+- `'debug'` - Show debug messages, info, warnings, and errors
+- `'info'` - Show info messages, warnings, and errors
+- `'warn'` - Show warnings and errors (default)
+- `'error'` - Show only errors
+- `'off'` - Disable all logging
+
+At the `'debug'` level, all HTTP requests and responses are logged, including headers and bodies.
+Some authentication-related headers are redacted, but sensitive data in request and response bodies
+may still be visible.
+
+#### Custom logger
+
+By default, this library logs to `globalThis.console`. You can also provide a custom logger.
+Most logging libraries are supported, including [pino](https://www.npmjs.com/package/pino), [winston](https://www.npmjs.com/package/winston), [bunyan](https://www.npmjs.com/package/bunyan), [consola](https://www.npmjs.com/package/consola), [signale](https://www.npmjs.com/package/signale), and [@std/log](https://jsr.io/@std/log). If your logger doesn't work, please open an issue.
+
+When providing a custom logger, the `logLevel` option still controls which messages are emitted, messages
+below the configured level will not be sent to your logger.
+
+```ts
+import Scorecard from 'scorecard-ai';
+import pino from 'pino';
+
+const logger = pino();
 
-While we value open-source contributions to this SDK, this library is generated programmatically. Additions made directly to this library would have to be moved over to our generation code, otherwise they would be overwritten upon the next generated release. Feel free to open a PR as a proof of concept, but know that we will not be able to merge it as-is. We suggest opening an issue first to discuss with us!
+const client = new Scorecard({
+  logger: logger.child({ name: 'Scorecard' }),
+  logLevel: 'debug', // Send all messages to pino, allowing it to filter
+});
+```
+
+### Making custom/undocumented requests
+
+This library is typed for convenient access to the documented API. If you need to access undocumented
+endpoints, params, or response properties, the library can still be used.
+
+#### Undocumented endpoints
+
+To make requests to undocumented endpoints, you can use `client.get`, `client.post`, and other HTTP verbs.
+Options on the client, such as retries, will be respected when making these requests.
+
+```ts
+await client.post('/some/path', {
+  body: { some_prop: 'foo' },
+  query: { some_query_arg: 'bar' },
+});
+```
+
+#### Undocumented request params
+
+To make requests using undocumented parameters, you may use `// @ts-expect-error` on the undocumented
+parameter. This library doesn't validate at runtime that the request matches the type, so any extra values you
+send will be sent as-is.
+
+```ts
+client.foo.create({
+  foo: 'my_param',
+  bar: 12,
+  // @ts-expect-error baz is not yet public
+  baz: 'undocumented option',
+});
+```
+
+For requests with the `GET` verb, any extra params will be in the query, all other requests will send the
+extra param in the body.
+
+If you want to explicitly send an extra argument, you can do so with the `query`, `body`, and `headers` request
+options.
+
+#### Undocumented response properties
+
+To access undocumented response properties, you may access the response object with `// @ts-expect-error` on
+the response object, or cast the response object to the requisite type. Like the request params, we do not
+validate or strip extra properties from the response from the API.
+
+### Customizing the fetch client
+
+By default, this library expects a global `fetch` function is defined.
+
+If you want to use a different `fetch` function, you can either polyfill the global:
+
+```ts
+import fetch from 'my-fetch';
+
+globalThis.fetch = fetch;
+```
+
+Or pass it to the client:
+
+```ts
+import Scorecard from 'scorecard-ai';
+import fetch from 'my-fetch';
+
+const client = new Scorecard({ fetch });
+```
+
+### Fetch options
+
+If you want to set custom `fetch` options without overriding the `fetch` function, you can provide a `fetchOptions` object when instantiating the client or making a request. (Request-specific options override client options.)
+
+```ts
+import Scorecard from 'scorecard-ai';
+
+const client = new Scorecard({
+  fetchOptions: {
+    // `RequestInit` options
+  },
+});
+```
+
+#### Configuring proxies
+
+To modify proxy behavior, you can provide custom `fetchOptions` that add runtime-specific proxy
+options to requests:
+
+<img src="https://raw.githubusercontent.com/stainless-api/sdk-assets/refs/heads/main/node.svg" align="top" width="18" height="21"> **Node** <sup>[[docs](https://github.com/nodejs/undici/blob/main/docs/docs/api/ProxyAgent.md#example---proxyagent-with-fetch)]</sup>
+
+```ts
+import Scorecard from 'scorecard-ai';
+import * as undici from 'undici';
+
+const proxyAgent = new undici.ProxyAgent('http://localhost:8888');
+const client = new Scorecard({
+  fetchOptions: {
+    dispatcher: proxyAgent,
+  },
+});
+```
+
+<img src="https://raw.githubusercontent.com/stainless-api/sdk-assets/refs/heads/main/bun.svg" align="top" width="18" height="21"> **Bun** <sup>[[docs](https://bun.sh/guides/http/proxy)]</sup>
+
+```ts
+import Scorecard from 'scorecard-ai';
+
+const client = new Scorecard({
+  fetchOptions: {
+    proxy: 'http://localhost:8888',
+  },
+});
+```
+
+<img src="https://raw.githubusercontent.com/stainless-api/sdk-assets/refs/heads/main/deno.svg" align="top" width="18" height="21"> **Deno** <sup>[[docs](https://docs.deno.com/api/deno/~/Deno.createHttpClient)]</sup>
+
+```ts
+import Scorecard from 'npm:scorecard-ai';
+
+const httpClient = Deno.createHttpClient({ proxy: { url: 'http://localhost:8888' } });
+const client = new Scorecard({
+  fetchOptions: {
+    client: httpClient,
+  },
+});
+```
+
+## Frequently Asked Questions
+
+## Semantic versioning
+
+This package generally follows [SemVer](https://semver.org/spec/v2.0.0.html) conventions, though certain backwards-incompatible changes may be released as minor versions:
+
+1. Changes that only affect static types, without breaking runtime behavior.
+2. Changes to library internals which are technically public but not intended or documented for external use. _(Please open a GitHub issue to let us know if you are relying on such internals.)_
+3. Changes that we do not expect to impact the vast majority of users in practice.
+
+We take backwards-compatibility seriously and work hard to ensure you can rely on a smooth upgrade experience.
+
+We are keen for your feedback; please open an [issue](https://www.github.com/scorecard-ai/scorecard-node/issues) with questions, bugs, or suggestions.
+
+## Requirements
+
+TypeScript >= 4.9 is supported.
+
+The following runtimes are supported:
+
+- Web browsers (Up-to-date Chrome, Firefox, Safari, Edge, and more)
+- Node.js 18 LTS or later ([non-EOL](https://endoflife.date/nodejs)) versions.
+- Deno v1.28.0 or higher.
+- Bun 1.0 or later.
+- Cloudflare Workers.
+- Vercel Edge Runtime.
+- Jest 28 or greater with the `"node"` environment (`"jsdom"` is not supported at this time).
+- Nitro v2.6 or greater.
+
+Note that React Native is not supported at this time.
+
+If you are interested in other runtime environments, please open or upvote an issue on GitHub.
+
+## Contributing
 
-On the other hand, contributions to the README are always very welcome!
+See [the contributing documentation](./CONTRIBUTING.md).

package/api-promise.d.mts

@@ -0,0 +1,2 @@
+export * from "./core/api-promise.mjs";
+//# sourceMappingURL=api-promise.d.mts.map

package/api-promise.d.mts.map

@@ -0,0 +1 @@
+{"version":3,"file":"api-promise.d.mts","sourceRoot":"","sources":["src/api-promise.ts"],"names":[],"mappings":""}

package/api-promise.d.ts

@@ -0,0 +1,2 @@
+export * from "./core/api-promise.js";
+//# sourceMappingURL=api-promise.d.ts.map

package/api-promise.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"api-promise.d.ts","sourceRoot":"","sources":["src/api-promise.ts"],"names":[],"mappings":""}

package/api-promise.js

@@ -0,0 +1,19 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __exportStar = (this && this.__exportStar) || function(m, exports) {
+    for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+/** @deprecated Import from ./core/api-promise instead */
+__exportStar(require("./core/api-promise.js"), exports);
+//# sourceMappingURL=api-promise.js.map

package/api-promise.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"api-promise.js","sourceRoot":"","sources":["src/api-promise.ts"],"names":[],"mappings":";;;;;;;;;;;;;;;;AAAA,yDAAyD;AACzD,wDAAmC"}

package/api-promise.mjs

@@ -0,0 +1,2 @@
+export * from "./core/api-promise.mjs";
+//# sourceMappingURL=api-promise.mjs.map

package/api-promise.mjs.map

@@ -0,0 +1 @@
+{"version":3,"file":"api-promise.mjs","sourceRoot":"","sources":["src/api-promise.ts"],"names":[],"mappings":""}