@evalguard/openai 1.1.1 → 1.2.1

package/README.md

@@ -1,123 +1,235 @@
-# @evalguard/openai
-
-Drop-in OpenAI SDK wrapper that adds real-time guardrails, trace logging, and cost tracking via [EvalGuard](https://evalguard.ai).
-
-## Installation
-
-```bash
-npm install @evalguard/openai openai
-```
-
-## Quick Start
-
-```typescript
-import OpenAI from "openai";
-import { wrapOpenAI } from "@evalguard/openai";
-
-const openai = wrapOpenAI(new OpenAI(), {
-  apiKey: "eg_...",
-  projectId: "proj_...",
-});
-
-// Use exactly like the normal OpenAI SDK — guardrails are automatic
-const response = await openai.chat.completions.create({
-  model: "gpt-4o",
-  messages: [{ role: "user", content: "Hello, how are you?" }],
-});
-
-console.log(response.choices[0].message.content);
-```
-
-## Streaming
-
-Streaming works transparently. The wrapper intercepts chunks to log the assembled response without affecting stream behavior.
-
-```typescript
-const stream = await openai.chat.completions.create({
-  model: "gpt-4o",
-  messages: [{ role: "user", content: "Write a poem about AI safety" }],
-  stream: true,
-});
-
-for await (const chunk of stream) {
-  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
-}
-```
-
-## Configuration
-
-```typescript
-const openai = wrapOpenAI(new OpenAI(), {
-  // Required: your EvalGuard API key
-  apiKey: "eg_...",
-
-  // Optional: EvalGuard API base URL (default: https://evalguard.ai/api/v1)
-  baseUrl: "https://your-evalguard-instance.com/api/v1",
-
-  // Optional: block requests that fail guardrails (default: true)
-  blockOnViolation: true,
-
-  // Optional: log all requests to EvalGuard (default: true)
-  enableLogging: true,
-
-  // Optional: project ID for organizing traces
-  projectId: "proj_...",
-
-  // Optional: custom metadata attached to every trace
-  metadata: { environment: "production", service: "chatbot" },
-
-  // Optional: callback when a guardrail violation is detected
-  onViolation: (result) => {
-    console.warn("Guardrail violation:", result.violations);
-  },
-});
-```
-
-## What It Does
-
-| Phase | Action |
-|-------|--------|
-| **Pre-request** | Sends the prompt to EvalGuard's firewall for prompt injection detection, PII scanning, and toxicity checks |
-| **LLM call** | Passes through to the real OpenAI API unchanged |
-| **Post-response** | Logs model, tokens, latency, cost, and guardrail results as a trace to EvalGuard |
-
-## Fail-Open Design
-
-If EvalGuard is unreachable (network error, timeout, 5xx), the wrapper passes requests through to OpenAI directly. Your application never breaks because of EvalGuard downtime.
-
-## Error Handling
-
-When `blockOnViolation` is `true` (default) and a guardrail check fails:
-
-```typescript
-import { EvalGuardViolationError } from "@evalguard/openai";
-
-try {
-  const response = await openai.chat.completions.create({
-    model: "gpt-4o",
-    messages: [{ role: "user", content: "malicious prompt..." }],
-  });
-} catch (error) {
-  if (error instanceof EvalGuardViolationError) {
-    console.log("Blocked:", error.violations);
-    // [{ type: "prompt_injection", severity: "critical", message: "..." }]
-  }
-}
-```
-
-Set `blockOnViolation: false` to log violations without blocking:
-
-```typescript
-const openai = wrapOpenAI(new OpenAI(), {
-  apiKey: "eg_...",
-  blockOnViolation: false,
-  onViolation: (result) => {
-    // Log but don't block
-    analytics.track("guardrail_violation", result);
-  },
-});
-```
-
-## License
-
-MIT
+# @evalguard/openai
+
+Drop-in OpenAI SDK wrapper that adds real-time guardrails, trace logging, and cost tracking via [EvalGuard](https://evalguard.ai).
+
+## Installation
+
+```bash
+npm install @evalguard/openai openai
+```
+
+> ### ⚠️ ESM-only — requires `"type": "module"`
+>
+> `@evalguard/openai` ships **ES modules only** (`"type": "module"`, no CJS build). The
+> quickstart below **will not type-check or run** in a default CommonJS
+> TypeScript project — you get `TS1479` ("the referenced file is an ECMAScript
+> module and cannot be imported with `require`") and, because the peer SDK's
+> types then resolve under a different module mode, a confusing
+> `TS2345 … Property '#private' … refers to a different member` on the very
+> first call.
+>
+> **To use this package, your consuming project must be ESM:**
+>
+> ```jsonc
+> // package.json
+> { "type": "module" }
+> ```
+> ```jsonc
+> // tsconfig.json
+> { "compilerOptions": { "module": "node16", "moduleResolution": "node16" } }
+> ```
+>
+> **Staying on CommonJS?** A dynamic `import()` works from CJS — but it is
+> **not enough on its own**. `openai` must be loaded dynamically **too**:
+> a static `import OpenAI from "openai"` in a CJS file resolves the peer's
+> types under CJS rules while the wrapper's resolve under ESM rules, so the
+> two `OpenAI` classes are different types and the very first call still fails
+> with `TS2345 … separate declarations of a private property`. Both imports
+> must be dynamic:
+>
+> …and both `await`s must sit **inside an async function**. Top-level `await`
+> is an ESM-only feature, so a bare `await import(...)` at file scope in a CJS
+> module is `TS1309: The current file is a CommonJS module and cannot use
+> 'await' at the top level` — which is why this block is an `async function`
+> and not four loose statements:
+>
+> ```typescript
+> // ✅ compiles under module/moduleResolution "node16", no "type": "module"
+> async function main() {
+>   const { wrapOpenAI } = await import("@evalguard/openai");
+>   const { default: OpenAI } = await import("openai");
+>
+>   const openai = wrapOpenAI(new OpenAI(), {
+>     apiKey: "eg_...",
+>     projectId: "proj_...",
+>   });
+>
+>   // …then use `openai` exactly as the quickstart below does.
+>   return openai;
+> }
+>
+> void main();
+> ```
+>
+> ```typescript
+> // ❌ still fails — the peer import is static
+> import OpenAI from "openai";
+>
+> async function main() {
+>   const { wrapOpenAI } = await import("@evalguard/openai");
+>   wrapOpenAI(new OpenAI(), { apiKey: "eg_..." });
+> }
+> // error TS2345: Argument of type 'OpenAI' is not assignable to parameter of
+> //   type 'OpenAI'. Types have separate declarations of a private property.
+> ```
+>
+> Every other snippet in this README uses the static `import OpenAI from
+> "openai"` form for readability. Under CommonJS you must convert **both**
+> lines, not just the `@evalguard/openai` one.
+>
+> Node.js ≥ 22.12 can also `require()` an ESM module directly
+> (`require(esm)`), but TypeScript still type-checks the import under CJS
+> rules, so the dynamic-import form above is the supported path.
+
+## Quick Start
+
+```typescript
+import OpenAI from "openai";
+import { wrapOpenAI } from "@evalguard/openai";
+
+const openai = wrapOpenAI(new OpenAI(), {
+  apiKey: "eg_...",
+  projectId: "proj_...",
+});
+
+// Use exactly like the normal OpenAI SDK — guardrails are automatic
+const response = await openai.chat.completions.create({
+  model: "gpt-4o",
+  messages: [{ role: "user", content: "Hello, how are you?" }],
+});
+
+console.log(response.choices[0].message.content);
+```
+
+## Streaming
+
+Streaming works transparently. The wrapper intercepts chunks to log the assembled response without affecting stream behavior.
+
+```typescript
+const stream = await openai.chat.completions.create({
+  model: "gpt-4o",
+  messages: [{ role: "user", content: "Write a poem about AI safety" }],
+  stream: true,
+});
+
+for await (const chunk of stream) {
+  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
+}
+```
+
+## Configuration
+
+```typescript
+const openai = wrapOpenAI(new OpenAI(), {
+  // Required: your EvalGuard API key
+  apiKey: "eg_...",
+
+  // Optional: EvalGuard API base URL (default: https://evalguard.ai/api/v1)
+  baseUrl: "https://your-evalguard-instance.com/api/v1",
+
+  // Optional: block requests that fail guardrails (default: true)
+  blockOnViolation: true,
+
+  // Optional: log all requests to EvalGuard (default: true)
+  enableLogging: true,
+
+  // Optional: project ID for organizing traces
+  projectId: "proj_...",
+
+  // Optional: custom metadata attached to every trace
+  metadata: { environment: "production", service: "chatbot" },
+
+  // Optional: callback when a guardrail violation is detected
+  onViolation: (result) => {
+    console.warn("Guardrail violation:", result.violations);
+  },
+});
+```
+
+## What It Does
+
+| Phase | Action |
+|-------|--------|
+| **Pre-request** | Sends the prompt to EvalGuard's firewall for prompt injection detection, PII scanning, and toxicity checks |
+| **LLM call** | Passes through to the real OpenAI API unchanged |
+| **Post-response** | Logs model, tokens, latency, cost, and guardrail results as a trace to EvalGuard |
+
+## Fail-Closed by Default
+
+If EvalGuard is unreachable (network error, timeout, 5xx) **and** `blockOnViolation` is `true` (the default), the wrapper **throws `EvalGuardViolationError`** (`type: "guardrail_unavailable"`) and does **not** call OpenAI — it fails *closed*, so an EvalGuard outage cannot silently bypass your guardrails.
+
+To fail *open* instead (pass the request through to OpenAI on an EvalGuard outage), set `blockOnViolation: false`, or set the env var `EVALGUARD_GUARDRAIL_FAIL_OPEN_LEGACY=1` to restore the legacy global fail-open behavior.
+
+## Error Handling
+
+When `blockOnViolation` is `true` (default) and a guardrail check fails:
+
+```typescript
+import { EvalGuardViolationError } from "@evalguard/openai";
+
+try {
+  const response = await openai.chat.completions.create({
+    model: "gpt-4o",
+    messages: [{ role: "user", content: "malicious prompt..." }],
+  });
+} catch (error) {
+  if (error instanceof EvalGuardViolationError) {
+    console.log("Blocked:", error.violations);
+    // [{ type: "prompt_injection", severity: "critical", message: "..." }]
+  }
+}
+```
+
+Set `blockOnViolation: false` to log violations without blocking:
+
+```typescript
+const openai = wrapOpenAI(new OpenAI(), {
+  apiKey: "eg_...",
+  blockOnViolation: false,
+  onViolation: (result) => {
+    // Log but don't block
+    analytics.track("guardrail_violation", result);
+  },
+});
+```
+
+## Cost estimates: unpriced models are reported as unpriced
+
+`estimateCost()` used to invent a price for any model outside a ~60-row table:
+`estimateCost("gpt-5", 1000, 500)` and `estimateCost("totally-unknown-model",
+1000, 500)` both returned `0.0105` from a blended $0.003/$0.015-per-1k
+fallback — with no flag and no warning. A FinOps figure you cannot tell apart
+from a real vendor price is worse than no figure at all.
+
+Two things changed:
+
+- **Coverage** — 2,200+ model ids now resolve to a real, sourced rate
+  (generated from EvalGuard's own pricing database, which is synced from the
+  LiteLLM catalogue). `gpt-5` is priced correctly.
+- **Honesty** — a genuinely unknown model is now *visibly* unknown.
+
+```typescript
+import { estimateCostDetailed, isModelPriced } from "@evalguard/openai";
+
+estimateCostDetailed("gpt-5", 1000, 500);
+// { model: "gpt-5", costUsd: 0.00625, priced: true, pricingSource: "catalog" }
+
+estimateCostDetailed("totally-unknown-model", 1000, 500);
+// { model: "totally-unknown-model",
+//   costUsd: null,              // <- never a fabricated number
+//   priced: false,
+//   pricingSource: "unpriced",
+//   blendedFallbackUsd: 0.0105 } // <- opt-in rough figure, clearly labelled
+
+isModelPriced("totally-unknown-model"); // false
+```
+
+`estimateCost()` still returns a `number` for backwards compatibility, but it
+now emits a one-time `console.warn` naming the unpriced model. Traces carry
+`costPricingSource` alongside `cost`, and `cost` is `null` for an unpriced
+model rather than a guess, so your EvalGuard dashboard shows "unpriced" instead
+of a fake dollar amount.
+
+## License
+
+Apache-2.0

package/dist/cost.d.ts

@@ -1,2 +1,4 @@
 export { estimateCost } from "@evalguard/wrapper-core";
+export { estimateCostDetailed, isModelPriced } from "@evalguard/wrapper-core";
+export type { CostEstimate, PricingSource } from "@evalguard/wrapper-core";
 //# sourceMappingURL=cost.d.ts.map

package/dist/cost.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"cost.d.ts","sourceRoot":"","sources":["../src/cost.ts"],"names":[],"mappings":"AAIA,OAAO,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC"}
+{"version":3,"file":"cost.d.ts","sourceRoot":"","sources":["../src/cost.ts"],"names":[],"mappings":"AAIA,OAAO,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC;AAMvD,OAAO,EAAE,oBAAoB,EAAE,aAAa,EAAE,MAAM,yBAAyB,CAAC;AAC9E,YAAY,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,yBAAyB,CAAC"}

package/dist/cost.js

@@ -3,4 +3,9 @@
 // source of truth (and Gemini / future-provider pricing rolls in
 // without touching this package).
 export { estimateCost } from "@evalguard/wrapper-core";
+// 2026-07-30 (audit finding 5): `estimateCostDetailed` is re-exported alongside
+// it. The blended fallback above is no longer SILENT — an unpriced model now
+// reports `{ costUsd: null, priced: false }` instead of a number a customer
+// cannot tell apart from a real vendor price.
+export { estimateCostDetailed, isModelPriced } from "@evalguard/wrapper-core";
 //# sourceMappingURL=cost.js.map

package/dist/cost.js.map

@@ -1 +1 @@
-{"version":3,"file":"cost.js","sourceRoot":"","sources":["../src/cost.ts"],"names":[],"mappings":"AAAA,wEAAwE;AACxE,uEAAuE;AACvE,iEAAiE;AACjE,kCAAkC;AAClC,OAAO,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC"}
+{"version":3,"file":"cost.js","sourceRoot":"","sources":["../src/cost.ts"],"names":[],"mappings":"AAAA,wEAAwE;AACxE,uEAAuE;AACvE,iEAAiE;AACjE,kCAAkC;AAClC,OAAO,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC;AAEvD,gFAAgF;AAChF,6EAA6E;AAC7E,4EAA4E;AAC5E,8CAA8C;AAC9C,OAAO,EAAE,oBAAoB,EAAE,aAAa,EAAE,MAAM,yBAAyB,CAAC"}

package/dist/index.d.ts

@@ -1,8 +1,8 @@
 import type OpenAI from "openai";
 import { type EvalGuardConfig } from "@evalguard/wrapper-core";
 export { GuardrailClient } from "./guardrail-client.js";
-export { EvalGuardViolationError, EvalGuardOutputViolationError, estimateCost, } from "@evalguard/wrapper-core";
-export type { EvalGuardConfig, GuardrailCheckResult, GuardrailViolation, TraceLogData, OutputScoreResult, ByokKeyResolver, RetryConfig, CircuitConfig, EvalOnResponseConfig, } from "@evalguard/wrapper-core";
+export { EvalGuardViolationError, EvalGuardOutputViolationError, estimateCost, estimateCostDetailed, isModelPriced, } from "@evalguard/wrapper-core";
+export type { CostEstimate, PricingSource, EvalGuardConfig, GuardrailCheckResult, GuardrailViolation, TraceLogData, OutputScoreResult, ByokKeyResolver, RetryConfig, CircuitConfig, EvalOnResponseConfig, } from "@evalguard/wrapper-core";
 /**
  * Wrap an existing OpenAI client to add EvalGuard guardrails, logging, and
  * cost tracking. Returns a proxied client that is fully type-compatible with
@@ -10,7 +10,7 @@ export type { EvalGuardConfig, GuardrailCheckResult, GuardrailViolation, TraceLo
  *
  * ```ts
  * import OpenAI from "openai";
- * import { wrapOpenAI } from "evalguardai-openai";
+ * import { wrapOpenAI } from "@evalguard/openai";
  *
  * const openai = wrapOpenAI(new OpenAI(), {
  *   apiKey: "eg_...",

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,MAAM,MAAM,QAAQ,CAAC;AAEjC,OAAO,EAML,KAAK,eAAe,EAGrB,MAAM,yBAAyB,CAAC;AAWjC,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AACxD,OAAO,EACL,uBAAuB,EACvB,6BAA6B,EAC7B,YAAY,GACb,MAAM,yBAAyB,CAAC;AACjC,YAAY,EACV,eAAe,EACf,oBAAoB,EACpB,kBAAkB,EAClB,YAAY,EACZ,iBAAiB,EACjB,eAAe,EACf,WAAW,EACX,aAAa,EACb,oBAAoB,GACrB,MAAM,yBAAyB,CAAC;AA4BjC;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,wBAAgB,UAAU,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,eAAe,GAAG,MAAM,CA2B1E"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,MAAM,MAAM,QAAQ,CAAC;AAEjC,OAAO,EAQL,KAAK,eAAe,EAGrB,MAAM,yBAAyB,CAAC;AAWjC,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AACxD,OAAO,EACL,uBAAuB,EACvB,6BAA6B,EAC7B,YAAY,EACZ,oBAAoB,EACpB,aAAa,GACd,MAAM,yBAAyB,CAAC;AACjC,YAAY,EACV,YAAY,EACZ,aAAa,EACb,eAAe,EACf,oBAAoB,EACpB,kBAAkB,EAClB,YAAY,EACZ,iBAAiB,EACjB,eAAe,EACf,WAAW,EACX,aAAa,EACb,oBAAoB,GACrB,MAAM,yBAAyB,CAAC;AA8DjC;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,wBAAgB,UAAU,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,eAAe,GAAG,MAAM,CAyE1E"}