agentid-sdk 0.1.37 → 0.1.40

package/README.md

@@ -111,8 +111,100 @@ console.log(response.choices[0]?.message?.content ?? "");
 By default, official AgentID SDK integrations inherit `enable_sdk_pii_masking`
 from the dashboard/runtime config. You only need to set `piiMasking: true` in
 code if you want to force local masking on even when the dashboard policy is off.
-
-Wrapped OpenAI calls persist telemetry for both regular and streamed completions. For `stream: true`, logging happens when the stream finishes.
+Starting with `agentid-sdk@0.1.40`, fail-open dependency fallback keeps local
+deterministic PII and secret masking enabled when `/agent/config` or `/guard`
+is unreachable. Fail-open can preserve availability, but official wrappers must
+not treat it as permission to send raw sensitive text to the provider.
+
+When SDK-side masking is enabled, the wrapper now masks both classic PII and
+high-confidence secret material before the request leaves your process:
+
+- emails, phones, card numbers, IBANs, national IDs, person names
+- OpenAI / Anthropic / Google / AWS / GitHub / Slack / Stripe credentials
+- bearer tokens, JWTs, `x-api-key` headers
+- password / credential assignments, PEM private keys, Azure connection strings and SAS tokens
+
+The masked form is what gets sent to `/guard`, logged to AgentID ingest, and
+forwarded to the model provider. The wrapper also protects returned completion
+text before it is logged or returned from the wrapped call when SDK-side masking
+is enabled.
+
+Important: this applies only to the wrapped call. If your app sends raw prompt
+or raw chat history through a separate direct provider call, AgentID cannot
+protect that bypass.
+
+Correct:
+
+```ts
+const secured = agent.wrapOpenAI(openai, {
+  system_id: process.env.AGENTID_SYSTEM_ID!,
+});
+
+await secured.chat.completions.create({
+  model: "gpt-4o-mini",
+  messages: fullConversationHistory,
+});
+```
+
+Incorrect:
+
+```ts
+// Raw history reaches the provider.
+await openai.chat.completions.create({
+  model: "gpt-4o-mini",
+  messages: rawConversationHistory,
+});
+
+// Logging a masked copy later does not protect the model call above.
+await agent.log({ system_id: systemId, input: maskedInput, output: maskedOutput });
+```
+
+For chat apps and agent workflows, protect the full message history, not just
+the latest text field. If a previous user/assistant/tool/memory message contains
+raw PII, the model can still repeat it later.
+
+If you cannot use `wrapOpenAI()` and need a manual integration, call
+`protectMessageHistory()` on the exact history that will be sent to the
+provider. Then pass `protected.messages` to the provider, not the raw
+`body.messages`.
+
+```ts
+import { AgentID, protectMessageHistory } from "agentid-sdk";
+
+const agent = new AgentID();
+const protectedHistory = protectMessageHistory(body.messages, {
+  pii: true,
+  secrets: true,
+});
+
+const latestUserInput = extractLatestUserInput(protectedHistory.messages);
+const verdict = await agent.guard({
+  system_id: process.env.AGENTID_SYSTEM_ID!,
+  input: latestUserInput,
+  model: "gpt-4o-mini",
+  metadata: {
+    runtime_surface: "manual_provider_integration",
+    full_history_protected: true,
+    messages_count: Array.isArray(protectedHistory.messages)
+      ? protectedHistory.messages.length
+      : undefined,
+    protected_messages_count: Array.isArray(protectedHistory.messages)
+      ? protectedHistory.messages.length
+      : undefined,
+    prompt_text_parts_count: protectedHistory.textPartsCount,
+    transformed_prompt_text_parts_count:
+      protectedHistory.transformedTextPartsCount,
+  },
+});
+if (!verdict.allowed) throw new Error(`Blocked: ${verdict.reason}`);
+
+const response = await openai.chat.completions.create({
+  model: "gpt-4o-mini",
+  messages: protectedHistory.messages,
+});
+```
+
+Wrapped OpenAI calls persist telemetry for both regular and streamed completions. For `stream: true`, logging happens when the stream finishes.
 
 > Scope note: AgentID compliance/risk controls apply to the specific SDK-wrapped LLM calls (`guard()`, `wrapOpenAI()`, LangChain callback-wrapped flows). They do not automatically classify unrelated code paths in your whole monolithic application.
 
@@ -133,17 +225,30 @@ npm install agentid-sdk openai @langchain/core @langchain/openai
 ```
 
 ```ts
-import { AgentID } from "agentid-sdk";
-import { AgentIDCallbackHandler } from "agentid-sdk/langchain";
-import { ChatOpenAI } from "@langchain/openai";
-import { ChatPromptTemplate } from "@langchain/core/prompts";
-import { StringOutputParser } from "@langchain/core/output_parsers";
-
-const agent = new AgentID();
-const handler = new AgentIDCallbackHandler(agent, {
-  system_id: process.env.AGENTID_SYSTEM_ID!,
-  expected_languages: ["en"],
-});
+import {
+  AgentID,
+  createAgentIdCorrelationId,
+  createAgentIdTelemetryContext,
+} from "agentid-sdk";
+import { AgentIDCallbackHandler } from "agentid-sdk/langchain";
+import { ChatOpenAI } from "@langchain/openai";
+import { ChatPromptTemplate } from "@langchain/core/prompts";
+import { StringOutputParser } from "@langchain/core/output_parsers";
+
+const agent = new AgentID();
+const workflowRunId = createAgentIdCorrelationId();
+const handler = new AgentIDCallbackHandler(agent, {
+  system_id: process.env.AGENTID_SYSTEM_ID!,
+  expected_languages: ["en"],
+  telemetry: createAgentIdTelemetryContext({
+    workflowRunId,
+    workflowStepName: "answer_question",
+    toolName: "langchain.chat",
+    toolTargetType: "conversation",
+    eventCategory: "ai",
+    eventSubtype: "answer_generated",
+  }),
+});
 
 const prompt = ChatPromptTemplate.fromTemplate("Answer in one sentence: {question}");
 const model = new ChatOpenAI({
@@ -157,29 +262,134 @@ const result = await chain.invoke(
   { callbacks: [handler] }
 );
 console.log(result);
-```
-
-LangChain callbacks log on run completion. Token/cost telemetry for streamed chains depends on the provider exposing usage in the final LangChain result.
-
-### Raw Ingest API (Telemetry Only)
-
-```ts
-import { AgentID } from "agentid-sdk";
+```
+
+LangChain callbacks log on run completion. Constructor-level `telemetry` is copied
+to the guard request, local policy telemetry, and final ingest log. You can
+override or extend it per invocation with LangChain metadata:
+`{ metadata: { agentid_telemetry: { workflowStepName: "..." } } }`.
+Token/cost telemetry for streamed chains depends on the provider exposing usage
+in the final LangChain result.
+
+### Raw Ingest API (Telemetry Only)
+
+```ts
+import { AgentID } from "agentid-sdk";
 
 const agent = new AgentID();
 
-await agent.log({
-  system_id: process.env.AGENTID_SYSTEM_ID!,
-  event_type: "complete",
-  severity: "info",
-  model: "gpt-4o-mini",
-  input: "Raw telemetry prompt",
-  output: '{"ok": true}',
-  metadata: { agent_role: "batch-worker", channel: "manual_ingest" },
-});
-```
-
-### Transparency Badge (Article 50 UI Evidence)
+await agent.log({
+  system_id: process.env.AGENTID_SYSTEM_ID!,
+  event_type: "complete",
+  severity: "info",
+  model: "gpt-4o-mini",
+  input: "Raw telemetry prompt",
+  output: '{"ok": true}',
+  usage: {
+    prompt_tokens: 33,
+    completion_tokens: 9,
+    total_tokens: 42,
+  },
+  latency: 1450,
+  metadata: { agent_role: "batch-worker", channel: "manual_ingest" },
+});
+```
+
+For manual integrations, preserve provider usage. Without `usage` or
+normalized `tokens`, AgentID can store Activity but cannot compute token totals,
+`cost_usd`, Total Spend, or ROI. ROI also requires the system business context
+fields `human_hourly_rate` and `human_time_per_task_min`.
+
+### Agent workflow and tool events
+
+Use `logOperation()` when an agent calls tools or performs operational work outside the wrapped LLM call. Reuse the same `workflowRunId` across steps.
+
+```ts
+import {
+  AgentID,
+  createAgentIdCorrelationId,
+  createAgentIdTelemetryContext,
+} from "agentid-sdk";
+
+const agent = new AgentID();
+const workflowRunId = createAgentIdCorrelationId();
+
+await agent.logOperation({
+  system_id: process.env.AGENTID_SYSTEM_ID!,
+  telemetry: createAgentIdTelemetryContext({
+    workflowRunId,
+    workflowStepName: "screen_candidate",
+    toolName: "hr.cv_screen",
+    toolTargetType: "candidate",
+  }),
+  event_category: "tool",
+  event_status: "completed",
+});
+
+await agent.logOperation({
+  system_id: process.env.AGENTID_SYSTEM_ID!,
+  telemetry: createAgentIdTelemetryContext({
+    workflowRunId,
+    workflowStepName: "send_followup",
+    toolName: "email.send",
+    toolTargetType: "email",
+  }),
+  event_category: "delivery",
+  event_status: "completed",
+});
+```
+
+Tool, delivery, inbox, workflow, guard, and operational events are logged as separate audit rows. They are grouped in the dashboard by `workflow_run_id` and do not count as model-used or spend-bearing unless you explicitly provide model/usage data. Do not reuse one `client_event_id` for the whole workflow; use `workflowRunId` for grouping and let each event keep its own idempotency key.
+
+Dashboard behavior:
+
+- prompt/guard checks remain visible as standalone Activity rows with `View Details` and `View Prompt`
+- workflow summary rows open the grouped timeline with tools, delivery, inbox, workflow lifecycle, guard checks, and LLM steps
+- the workflow timeline is operational context; the standalone prompt row is the forensic prompt inspection surface
+- non-model workflow/tool/delivery rows show `Model: Not applicable` and are not spend-bearing unless model/cost metadata is explicitly present
+
+For full agent runs, prefer the workflow trail helper so each step gets a shared
+`workflow_step_id`, plus automatic `started/completed/failed` rows:
+
+```ts
+import {
+  AgentID,
+  createAgentIdCorrelationId,
+  createAgentIdTelemetryContext,
+  createAgentIdWorkflowTrail,
+} from "agentid-sdk";
+
+const agent = new AgentID({ apiKey: process.env.AGENTID_API_KEY! });
+const workflowRunId = createAgentIdCorrelationId();
+
+const trail = createAgentIdWorkflowTrail({
+  agent,
+  system_id: process.env.AGENTID_SYSTEM_ID!,
+  telemetry: createAgentIdTelemetryContext({
+    workflowRunId,
+    workflowName: "Candidate intake",
+  }),
+});
+
+await trail.runStep(
+  {
+    telemetry: createAgentIdTelemetryContext({
+      workflowStepName: "screen_candidate",
+      toolName: "hr.cv_screen",
+      toolTargetType: "candidate",
+      eventCategory: "tool",
+    }),
+  },
+  async () => screenCandidate(),
+  {
+    complete: {
+      metadata: { result_count: 4 },
+    },
+  }
+);
+```
+
+### Transparency Badge (Article 50 UI Evidence)
 
 When rendering disclosure UI, log proof-of-render telemetry so you can demonstrate the end-user actually saw the badge.
 
@@ -251,11 +461,25 @@ By default, AgentID is designed to keep your application running if the AgentID
 
 - `guard()` returns a verdict (`allowed`, `reason`); handle deny paths explicitly.
 - `wrapOpenAI()` and LangChain handlers throw `SecurityBlockError` when a prompt is blocked.
-- Backend `/guard` is the default authority for prompt injection, DB access, code execution, and PII leakage in SDK-wrapped flows.
-- `clientFastFail` / `client_fast_fail` is optional and disabled by default. Enable it only when you explicitly want local preflight before the backend call.
-- If backend guard is unreachable and the effective failure mode is `fail_close`, wrapped OpenAI/LangChain flows can run local fallback enforcement. Local hits still block; otherwise the request can continue with fallback telemetry attached.
-- If `strictMode` is not explicitly set in SDK code, runtime behavior follows the system configuration from AgentID (`strict_security_mode` / `failure_mode`).
-- Ingest retries transient failures (5xx/429) and logs warnings if persistence fails.
+- Backend `/guard` is the default authority for prompt injection, DB access, code execution, and PII leakage in SDK-wrapped flows.
+- `clientFastFail` / `client_fast_fail` is optional and disabled by default. Enable it only when you explicitly want local preflight before the backend call.
+- If backend guard is unreachable and the effective failure mode is `fail_close`, wrapped OpenAI/LangChain flows can run local fallback enforcement. Local hits still block; otherwise the request can continue with fallback telemetry attached.
+- If `strictMode` is not explicitly set in SDK code, runtime behavior follows the system configuration from AgentID (`strict_security_mode` / `failure_mode`).
+- Ingest retries transient failures (5xx/429) and logs warnings if persistence fails.
+
+### SDK-side masking scope
+
+If `enable_sdk_pii_masking=true` in AgentID runtime config, or if you force
+`piiMasking: true` in code, masking happens locally before `/guard` and before
+provider dispatch.
+
+- Default mode: backend-first enforcement, optional local masking
+- `clientFastFail=false`: no local prompt/code/db blocker, but local masking can still rewrite prompt text before network dispatch
+- `clientFastFail=true`: local prompt-injection scan and strict local enforcement can run before `/guard`
+
+This means SDK masking is useful even when you keep backend guard as the main
+policy authority: it reduces raw data exposure on the wire without changing the
+server-side decision model.
 
 ### Event Identity Model
 
@@ -263,7 +487,7 @@ For consistent lifecycle correlation in Activity/Prompts, use this model:
 
 - `client_event_id`: external correlation ID for one end-to-end action.
 - `guard_event_id`: ID of the preflight guard event returned by `guard()`.
-- `event_id` on `log()`: idempotency key for ingest. In the JS SDK it is canonicalized to `client_event_id` for stable one-row lifecycle updates.
+- `event_id` on `log()`: idempotency key for ingest. In `agentid-sdk` it is canonicalized to `client_event_id` for stable one-row lifecycle updates.
 
 SDK behavior:
 
@@ -289,7 +513,7 @@ SDK-managed metadata can include:
 
 ### Policy-Pack Runtime Telemetry
 
-When the backend uses compiled policy packs, runtime metadata includes:
+When the backend uses compiled policy packs, runtime metadata includes:
 
 - `policy_pack_version`: active compiled artifact version.
 - `policy_pack_fallback`: `true` means fallback detector path was used.
@@ -299,7 +523,23 @@ Latency interpretation:
 
 - Activity `Latency (ms)` maps to synchronous processing (`processing_time_ms`).
 - Async AI audit time is separate (`ai_audit_duration_ms`) and can be higher.
-- First request after warm-up boundaries can be slower than steady-state requests.
+- First request after warm-up boundaries can be slower than steady-state requests.
+
+### Secret and PII Masking Edge Cases
+
+SDK-side masking and the backend scanner include regression coverage for common
+boundary failures:
+
+- multiline PEM, certificate, and PGP private key blocks
+- natural-language password disclosures such as `my Password is Passwordk123`
+- environment-style assignments such as `DB_PASSWORD=...`
+- secret values with suffix punctuation such as `#`
+- high-entropy base64-like values with `=` / `==` padding
+- security-question answers where the value appears after `answer is`, `is`, or localized equivalents
+
+When local masking is enabled, these values are replaced before provider
+dispatch and before AgentID ingest. Placeholder mappings stay local to the SDK
+for reversible deanonymization.
 
 ### Monorepo QA Commands (Maintainers)
 
@@ -319,21 +559,27 @@ powershell -ExecutionPolicy Bypass -File .\scripts\qa\run-ai-label-audit-check.p
 
 ## 7. Security & Compliance
 
-- Backend `/guard` remains the primary enforcement authority by default.
-- Optional local PII masking and opt-in `clientFastFail` are available for edge cases.
-- Guard checks run pre-execution; ingest + finalize telemetry captures prompt/output lifecycle and SDK timing breakdowns.
-- Safe for server and serverless runtimes (including async completion flows).
-- Supports compliance and forensics workflows with durable event records.
-
-## 8. Support
+- Backend `/guard` remains the primary enforcement authority by default.
+- Optional local masking and opt-in `clientFastFail` are available for edge cases.
+- SDK-side masking can now cover both structured PII and high-confidence leaked secrets before provider dispatch.
+- Guard checks run pre-execution; ingest + finalize telemetry captures prompt/output lifecycle and SDK timing breakdowns.
+- Safe for server and serverless runtimes (including async completion flows).
+- Supports compliance and forensics workflows with durable event records.
 
-- Dashboard: `https://app.getagentid.com`
-- Repository: `https://github.com/ondrejsukac-rgb/agentid/tree/main/js-sdk`
-- Issues: `https://github.com/ondrejsukac-rgb/agentid/issues`
-
-## 9. Publishing Notes (NPM)
-
-NPM automatically renders `README.md` from the package root during `npm publish`.
+## 8. Support
+
+- Dashboard: `https://app.getagentid.com`
+- Documentation: `https://docs.getagentid.com/docs/node-typescript-sdk`
+- Repository: `https://github.com/ondrejsukac-rgb/agentid/tree/main/agentid-sdk`
+- Issues: `https://github.com/ondrejsukac-rgb/agentid/issues`
 
-- File location: next to `package.json` in `js-sdk/`.
-- No additional NPM config is required for README rendering.
+## 9. Publishing Notes (NPM)
+
+NPM automatically renders `README.md` from the package root during `npm publish`.
+
+- File location: next to `package.json` in `agentid-sdk/`.
+- No additional NPM config is required for README rendering.
+- Before publishing from the monorepo, run `npm run audit:all` and
+  `npm run qa:production-gate` from the repository root.
+- The production gate audits the root app, `agentid-sdk`, `agentid-vercel-sdk`, and
+  the browser extension package so package-local lockfile issues are not missed.