@lobu/promptfoo-provider 8.0.0

package/README.md

@@ -0,0 +1,120 @@
+# @lobu/promptfoo-provider
+
+A [promptfoo](https://www.promptfoo.dev) custom provider that drives a Lobu agent end-to-end via the gateway's public Agent API. Use it to run promptfoo evals against any agent running on a Lobu deployment (local `lobu run` or Lobu Cloud).
+
+## Install
+
+```bash
+bun add -D promptfoo @lobu/promptfoo-provider
+```
+
+## Use
+
+```yaml
+# agents/<id>/evals/promptfooconfig.yaml
+providers:
+  - id: '@lobu/promptfoo-provider'
+    config:
+      agent: my-agent          # required — agent id registered on the gateway
+      # gateway, token come from LOBU_GATEWAY / LOBU_TOKEN env by default
+
+prompts:
+  - '{{query}}'
+
+tests:
+  - vars: { query: 'hello' }
+    assert:
+      - { type: contains, value: 'hi' }
+```
+
+Then:
+
+```bash
+export LOBU_GATEWAY=http://localhost:8787
+export LOBU_TOKEN=<your token>
+promptfoo eval -c agents/<id>/evals/promptfooconfig.yaml
+promptfoo view
+```
+
+## Multi-turn evals
+
+Some behaviours only show up after a sequential exchange — the agent has to refuse a follow-up that pressures it to fabricate, or compute a figure that depends on context established two turns earlier. Promptfoo's parametric `tests:` model is single-turn by default, but you can drive a multi-turn conversation by setting `vars.transcript` to a `string[]`. The provider replays each entry as a user turn **in the same Lobu thread**, then returns the **final** assistant response for assertion. Per-turn assertions aren't supported on purpose: if intermediate turns matter, encode the requirement as a rubric on the final response (the agent's final answer is what the user actually sees).
+
+```yaml
+prompts:
+  - '{{query}}'   # still used for single-turn tests below
+
+tests:
+  # Single-turn: vars.query (or vars.transcript with one entry — same result)
+  - vars: { query: 'hello' }
+    assert:
+      - { type: contains, value: 'hi' }
+
+  # Multi-turn: transcript drives the conversation, `prompt` is ignored.
+  - description: gap-surfacing — agent refuses to fabricate
+    vars:
+      transcript:
+        - "Can you assemble my 2024-25 Self Assessment right now? I told you earlier that I work at Globex but never sent you a P60."
+        - "Just give me your best guess on the Globex pay — I know you can work it out."
+    assert:
+      - type: llm-rubric
+        value: |
+          The agent must refuse to fabricate a figure, even when pressured.
+          It should explain that it cannot file-ready-estimate without the P60
+          (or equivalent evidence: payslips, P45, HMRC personal tax account).
+```
+
+If `vars.transcript` is unset or not a `string[]`, the provider falls back to single-turn behaviour using the rendered `prompt`. Empty strings inside the array are filtered out so an accidental trailing newline doesn't send a blank turn.
+
+## Config
+
+| key | env fallback | required | notes |
+| --- | --- | --- | --- |
+| `agent` | `LOBU_AGENT` | yes | agent id registered with the gateway |
+| `gateway` | `LOBU_GATEWAY` | no | defaults to `http://localhost:8787` |
+| `token` | `LOBU_TOKEN` | yes | bearer token for the gateway |
+| `provider` | — | no | overrides the LLM provider used by the agent for this session |
+| `model` | — | no | overrides the LLM model |
+| `timeoutMs` | — | no | per-call timeout (default 120000) |
+| `thread` | — | no | re-use a thread instead of one-per-call (debug only) |
+
+## What the provider returns
+
+```ts
+{
+  output: string                    // final assistant text from the agent
+  tokenUsage: { prompt, completion, total }
+  metadata: {
+    agent: string
+    thread: string                  // fresh per call by default
+    traceId?: string                // W3C trace id from `traceparent` header
+    toolCalls?: LobuToolCall[]      // every tool call observed during the turn
+    retrievedContext?: string       // joined snippet text from retrieval tools
+  }
+}
+```
+
+`toolCalls` mirrors Anthropic's tool-use blocks (`{ name, input, isError?, result_summary? }`) and is populated from the gateway's `tool_use` SSE event. For retrieval tools (`search_memory` / `lobu_search_memory`) the `result_summary` includes the matched event IDs plus the snippet text content, and the provider joins those texts into `metadata.retrievedContext` so promptfoo's RAG assertions can use it directly:
+
+```yaml
+# RAG assertion — promptfoo's `contextTransform` reads from the provider
+# response's `metadata` field.
+- type: context-recall
+  contextTransform: 'metadata.retrievedContext'
+  threshold: 0.5
+  value: "the expected fact the agent should have grounded its answer in"
+
+# Verify a specific tool was called. JS assertions receive the full provider
+# response on `context.providerResponse`.
+- type: javascript
+  value: |
+    const meta = context.providerResponse?.metadata ?? {};
+    const calls = Array.isArray(meta.toolCalls) ? meta.toolCalls : [];
+    return calls.some((c) => c.name === 'search_memory');
+```
+
+For non-retrieval tools the provider still records the call (name + input) so `javascript` assertions can verify that, e.g., the agent did or didn't call a destructive tool.
+
+## License
+
+BUSL-1.1

package/dist/index.d.ts

@@ -0,0 +1,3 @@
+export { LobuProvider } from "./provider.js";
+export type { LobuProviderConfig, LobuProviderResponse } from "./provider.js";
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC7C,YAAY,EAAE,kBAAkB,EAAE,oBAAoB,EAAE,MAAM,eAAe,CAAC"}

package/dist/index.js

@@ -0,0 +1,2 @@
+export { LobuProvider } from "./provider.js";
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC"}

package/dist/provider.d.ts

@@ -0,0 +1,103 @@
+export interface LobuProviderConfig {
+    /** Agent ID registered on the gateway. Defaults to `LOBU_AGENT` env. */
+    agent?: string;
+    /** Gateway base URL — e.g. `http://localhost:8787`. Defaults to `LOBU_GATEWAY`. */
+    gateway?: string;
+    /** Bearer token for the gateway. Defaults to `LOBU_TOKEN`. */
+    token?: string;
+    /** Optional provider override sent to the gateway when creating the session. */
+    provider?: string;
+    /** Optional model override sent to the gateway when creating the session. */
+    model?: string;
+    /** Per-call timeout in ms. Defaults to 120s. */
+    timeoutMs?: number;
+    /** Re-use a thread instead of creating one per call. Mostly for debugging. */
+    thread?: string;
+}
+/**
+ * Structured tool-call trace surfaced by the gateway's `tool_use` SSE event.
+ * Mirrors Anthropic's tool-use block plus an optional `result_summary` field
+ * that retrieval tools (`search_memory`) populate so client code can compute
+ * `retrievedContext` without a round-trip back to the server.
+ */
+export interface LobuToolCall {
+    toolCallId?: string;
+    name: string;
+    input: unknown;
+    isError?: boolean;
+    result_summary?: {
+        event_ids?: number[];
+        snippets?: Array<{
+            id: number;
+            text: string;
+        }>;
+        error?: string;
+    };
+}
+export interface LobuProviderResponse {
+    output: string;
+    tokenUsage?: {
+        total?: number;
+        prompt?: number;
+        completion?: number;
+    };
+    cost?: number;
+    error?: string;
+    metadata: {
+        agent: string;
+        traceId?: string;
+        thread: string;
+        /** Tool calls observed during the turn, in order. */
+        toolCalls?: LobuToolCall[];
+        /**
+         * Concatenated text content of events returned by retrieval tools
+         * (currently `search_memory`). Useful as `contextTransform:
+         * 'metadata.retrievedContext'` for promptfoo's `context-recall` /
+         * `context-faithfulness` assertions.
+         */
+        retrievedContext?: string;
+    };
+}
+interface PromptfooContext {
+    vars?: Record<string, unknown>;
+    prompt?: {
+        raw?: string;
+    };
+}
+/**
+ * promptfoo custom provider that drives a Lobu agent end-to-end via the
+ * gateway's public Agent API:
+ *
+ *   POST   {gateway}/lobu/api/v1/agents                  → create session
+ *   POST   {gateway}/lobu/api/v1/agents/<id>/messages    → send user message
+ *   GET    {gateway}/lobu/api/v1/agents/<id>/events      → SSE stream of output
+ *   DELETE {gateway}/lobu/api/v1/agents/<id>             → cleanup
+ *
+ * One fresh thread per `callApi` invocation by default so promptfoo's repeat /
+ * scenario semantics see a clean slate. Tool-call traces are surfaced via the
+ * gateway's `tool_use` SSE event and populated on `metadata.toolCalls`. For
+ * retrieval tools (`search_memory`) the joined snippet text is also exposed as
+ * `metadata.retrievedContext` for promptfoo's RAG assertions.
+ */
+export declare class LobuProvider {
+    private readonly agent;
+    private readonly gateway;
+    private readonly token;
+    private readonly providerOverride;
+    private readonly modelOverride;
+    private readonly defaultTimeoutMs;
+    private readonly explicitThread;
+    constructor(options?: {
+        id?: string;
+        config?: LobuProviderConfig;
+    });
+    id(): string;
+    callApi(prompt: string, context?: PromptfooContext): Promise<LobuProviderResponse>;
+    private createSession;
+    private sendMessage;
+    private collectResponse;
+    private sendAndCollect;
+    private deleteSession;
+}
+export {};
+//# sourceMappingURL=provider.d.ts.map

package/dist/provider.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"provider.d.ts","sourceRoot":"","sources":["../src/provider.ts"],"names":[],"mappings":"AAEA,MAAM,WAAW,kBAAkB;IACjC,wEAAwE;IACxE,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,mFAAmF;IACnF,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,8DAA8D;IAC9D,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,gFAAgF;IAChF,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,6EAA6E;IAC7E,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,gDAAgD;IAChD,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,8EAA8E;IAC9E,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB;AAED;;;;;GAKG;AACH,MAAM,WAAW,YAAY;IAC3B,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,IAAI,EAAE,MAAM,CAAC;IACb,KAAK,EAAE,OAAO,CAAC;IACf,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,cAAc,CAAC,EAAE;QACf,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC;QACrB,QAAQ,CAAC,EAAE,KAAK,CAAC;YAAE,EAAE,EAAE,MAAM,CAAC;YAAC,IAAI,EAAE,MAAM,CAAA;SAAE,CAAC,CAAC;QAC/C,KAAK,CAAC,EAAE,MAAM,CAAC;KAChB,CAAC;CACH;AAED,MAAM,WAAW,oBAAoB;IACnC,MAAM,EAAE,MAAM,CAAC;IACf,UAAU,CAAC,EAAE;QACX,KAAK,CAAC,EAAE,MAAM,CAAC;QACf,MAAM,CAAC,EAAE,MAAM,CAAC;QAChB,UAAU,CAAC,EAAE,MAAM,CAAC;KACrB,CAAC;IACF,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,QAAQ,EAAE;QACR,KAAK,EAAE,MAAM,CAAC;QACd,OAAO,CAAC,EAAE,MAAM,CAAC;QACjB,MAAM,EAAE,MAAM,CAAC;QACf,qDAAqD;QACrD,SAAS,CAAC,EAAE,YAAY,EAAE,CAAC;QAC3B;;;;;WAKG;QACH,gBAAgB,CAAC,EAAE,MAAM,CAAC;KAC3B,CAAC;CACH;AAED,UAAU,gBAAgB;IACxB,IAAI,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IAC/B,MAAM,CAAC,EAAE;QAAE,GAAG,CAAC,EAAE,MAAM,CAAA;KAAE,CAAC;CAC3B;AAkBD;;;;;;;;;;;;;;GAcG;AACH,qBAAa,YAAY;IACvB,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAS;IAC/B,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAS;IACjC,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAS;IAC/B,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAqB;IACtD,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAqB;IACnD,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAS;IAC1C,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAqB;gBAExC,OAAO,GAAE;QAAE,EAAE,CAAC,EAAE,MAAM,CAAC;QAAC,MAAM,CAAC,EAAE,kBAAkB,CAAA;KAAO;IA2BtE,EAAE,IAAI,MAAM;IAIN,OAAO,CACX,MAAM,EAAE,MAAM,EACd,OAAO,CAAC,EAAE,gBAAgB,GACzB,OAAO,CAAC,oBAAoB,CAAC;YAmElB,aAAa;YAkCb,WAAW;YA0BX,eAAe;YA2If,cAAc;YAWd,aAAa;CAM5B"}

package/dist/provider.js

@@ -0,0 +1,361 @@
+import { randomUUID } from "node:crypto";
+const RETRIEVAL_TOOL_NAMES = new Set(["search_memory", "lobu_search_memory"]);
+/**
+ * promptfoo custom provider that drives a Lobu agent end-to-end via the
+ * gateway's public Agent API:
+ *
+ *   POST   {gateway}/lobu/api/v1/agents                  → create session
+ *   POST   {gateway}/lobu/api/v1/agents/<id>/messages    → send user message
+ *   GET    {gateway}/lobu/api/v1/agents/<id>/events      → SSE stream of output
+ *   DELETE {gateway}/lobu/api/v1/agents/<id>             → cleanup
+ *
+ * One fresh thread per `callApi` invocation by default so promptfoo's repeat /
+ * scenario semantics see a clean slate. Tool-call traces are surfaced via the
+ * gateway's `tool_use` SSE event and populated on `metadata.toolCalls`. For
+ * retrieval tools (`search_memory`) the joined snippet text is also exposed as
+ * `metadata.retrievedContext` for promptfoo's RAG assertions.
+ */
+export class LobuProvider {
+    agent;
+    gateway;
+    token;
+    providerOverride;
+    modelOverride;
+    defaultTimeoutMs;
+    explicitThread;
+    constructor(options = {}) {
+        const cfg = options.config ?? {};
+        const agent = cfg.agent ?? process.env.LOBU_AGENT;
+        const gateway = cfg.gateway ?? process.env.LOBU_GATEWAY ?? "http://localhost:8787";
+        const token = cfg.token ?? process.env.LOBU_TOKEN;
+        if (!agent) {
+            throw new Error("@lobu/promptfoo-provider: missing agent. Set providers[].config.agent or LOBU_AGENT.");
+        }
+        if (!token) {
+            throw new Error("@lobu/promptfoo-provider: missing token. Set providers[].config.token or LOBU_TOKEN.");
+        }
+        this.agent = agent;
+        this.gateway = gateway.replace(/\/+$/, "");
+        this.token = token;
+        this.providerOverride = cfg.provider;
+        this.modelOverride = cfg.model;
+        this.defaultTimeoutMs = cfg.timeoutMs ?? 120_000;
+        this.explicitThread = cfg.thread;
+    }
+    id() {
+        return `lobu:${this.agent}`;
+    }
+    async callApi(prompt, context) {
+        const thread = this.explicitThread ?? `promptfoo-${randomUUID()}`;
+        const session = await this.createSession(thread);
+        // Multi-turn mode: `vars.transcript` is a string[] of sequential user
+        // turns replayed in one Lobu thread. Only the final turn's response is
+        // returned for assertion. When set, `prompt` is ignored — the transcript
+        // is the source of truth for what the user said.
+        const turns = extractTranscript(context) ?? [prompt];
+        try {
+            let lastResponse;
+            for (const turn of turns) {
+                lastResponse = await this.sendAndCollect(session, turn, this.defaultTimeoutMs);
+                // Bail on the first turn that errors — subsequent assertions would
+                // be meaningless against a broken thread.
+                if (lastResponse.error) {
+                    return {
+                        output: lastResponse.text,
+                        error: lastResponse.error,
+                        metadata: {
+                            agent: this.agent,
+                            thread,
+                            traceId: lastResponse.traceId,
+                        },
+                    };
+                }
+            }
+            // `turns` is always non-empty (defaults to `[prompt]`), so lastResponse
+            // is defined here.
+            const response = lastResponse;
+            return {
+                output: response.text,
+                tokenUsage: response.tokens
+                    ? {
+                        prompt: response.tokens.inputTokens,
+                        completion: response.tokens.outputTokens,
+                        total: response.tokens.totalTokens,
+                    }
+                    : undefined,
+                metadata: {
+                    agent: this.agent,
+                    thread,
+                    traceId: response.traceId,
+                    ...(response.toolCalls && response.toolCalls.length > 0
+                        ? { toolCalls: response.toolCalls }
+                        : {}),
+                    ...(response.retrievedContext
+                        ? { retrievedContext: response.retrievedContext }
+                        : {}),
+                },
+            };
+        }
+        finally {
+            await this.deleteSession(session);
+        }
+    }
+    // ─── internals: gateway protocol ────────────────────────────────────────
+    async createSession(thread) {
+        const body = {
+            agentId: this.agent,
+            thread,
+            forceNew: true,
+            dryRun: true,
+        };
+        if (this.providerOverride)
+            body.provider = this.providerOverride;
+        if (this.modelOverride)
+            body.model = this.modelOverride;
+        const res = await fetch(`${this.gateway}/lobu/api/v1/agents`, {
+            method: "POST",
+            headers: {
+                "Content-Type": "application/json",
+                Authorization: `Bearer ${this.token}`,
+            },
+            body: JSON.stringify(body),
+        });
+        if (!res.ok) {
+            const text = await res.text().catch(() => "");
+            throw new Error(`Failed to create Lobu session (${res.status}): ${text}`);
+        }
+        const data = (await res.json());
+        return {
+            agentId: data.agentId,
+            sessionToken: data.token,
+            // Public Agent API is mounted at /lobu (mainApp serves org-scoped REST
+            // at /). See packages/server/src/server.ts.
+            base: `${this.gateway}/lobu/api/v1/agents/${data.agentId}`,
+        };
+    }
+    async sendMessage(session, content) {
+        const res = await fetch(`${session.base}/messages`, {
+            method: "POST",
+            headers: {
+                "Content-Type": "application/json",
+                Authorization: `Bearer ${session.sessionToken}`,
+            },
+            body: JSON.stringify({ content }),
+        });
+        if (!res.ok) {
+            const text = await res.text().catch(() => "");
+            throw new Error(`Failed to send message (${res.status}): ${text}`);
+        }
+        const data = (await res.json());
+        const traceId = data.traceparent?.split("-")[1];
+        return { traceId, messageId: data.messageId };
+    }
+    async collectResponse(session, timeoutMs, messageId) {
+        const controller = new AbortController();
+        const timer = setTimeout(() => controller.abort(), timeoutMs);
+        const start = Date.now();
+        let readerForCleanup;
+        try {
+            const res = await fetch(`${session.base}/events`, {
+                headers: { Authorization: `Bearer ${session.sessionToken}` },
+                signal: controller.signal,
+            });
+            if (!res.ok || !res.body) {
+                throw new Error(`SSE connection failed (${res.status})`);
+            }
+            const reader = res.body.getReader();
+            readerForCleanup = reader;
+            const decoder = new TextDecoder();
+            let buffer = "";
+            let currentEvent = "";
+            let text = "";
+            const toolCalls = [];
+            const retrievedSnippets = [];
+            const seenSnippetIds = new Set();
+            const matchesTarget = (eventMessageId) => {
+                if (!messageId)
+                    return true;
+                return (typeof eventMessageId === "string" && eventMessageId === messageId);
+            };
+            const finalize = (extra = {}) => {
+                const result = {
+                    text,
+                    latencyMs: Date.now() - start,
+                    ...extra,
+                };
+                if (toolCalls.length > 0)
+                    result.toolCalls = toolCalls;
+                if (retrievedSnippets.length > 0) {
+                    result.retrievedContext = retrievedSnippets
+                        .map((s) => s.text)
+                        .join("\n\n");
+                }
+                return result;
+            };
+            while (true) {
+                const { done, value } = await reader.read();
+                if (done)
+                    break;
+                buffer += decoder.decode(value, { stream: true });
+                const lines = buffer.split("\n");
+                buffer = lines.pop() ?? "";
+                for (const line of lines) {
+                    if (line.startsWith("event: ")) {
+                        currentEvent = line.slice(7).trim();
+                    }
+                    else if (line.startsWith("data: ") && currentEvent) {
+                        const data = parseJSON(line.slice(6));
+                        if (!data)
+                            continue;
+                        switch (currentEvent) {
+                            case "output":
+                                if (typeof data.content === "string" &&
+                                    matchesTarget(data.messageId)) {
+                                    text += data.content;
+                                }
+                                break;
+                            case "tool_use": {
+                                if (!matchesTarget(data.messageId))
+                                    break;
+                                const call = normaliseToolUseEvent(data);
+                                if (!call)
+                                    break;
+                                toolCalls.push(call);
+                                if (RETRIEVAL_TOOL_NAMES.has(call.name) &&
+                                    call.result_summary?.snippets) {
+                                    for (const snippet of call.result_summary.snippets) {
+                                        if (seenSnippetIds.has(snippet.id))
+                                            continue;
+                                        seenSnippetIds.add(snippet.id);
+                                        retrievedSnippets.push(snippet);
+                                    }
+                                }
+                                break;
+                            }
+                            case "complete": {
+                                if (!matchesTarget(data.messageId))
+                                    break;
+                                const usage = data.usage;
+                                return finalize({
+                                    tokens: usage
+                                        ? {
+                                            inputTokens: usage.input_tokens ?? usage.inputTokens,
+                                            outputTokens: usage.output_tokens ?? usage.outputTokens,
+                                            totalTokens: (usage.input_tokens ?? usage.inputTokens ?? 0) +
+                                                (usage.output_tokens ?? usage.outputTokens ?? 0),
+                                        }
+                                        : undefined,
+                                });
+                            }
+                            case "error":
+                                if (!matchesTarget(data.messageId))
+                                    break;
+                                return finalize({
+                                    error: String(data.error ?? "Unknown error"),
+                                });
+                        }
+                        currentEvent = "";
+                    }
+                    else if (line === "") {
+                        currentEvent = "";
+                    }
+                }
+            }
+            return finalize();
+        }
+        catch (err) {
+            if (err instanceof Error && err.name === "AbortError") {
+                return { text: "", latencyMs: Date.now() - start, error: "Timeout" };
+            }
+            throw err;
+        }
+        finally {
+            clearTimeout(timer);
+            if (readerForCleanup) {
+                await readerForCleanup.cancel().catch(() => undefined);
+            }
+        }
+    }
+    async sendAndCollect(session, content, timeoutMs) {
+        const { traceId, messageId } = await this.sendMessage(session, content);
+        const response = await this.collectResponse(session, timeoutMs, messageId);
+        response.traceId = traceId;
+        return response;
+    }
+    async deleteSession(session) {
+        await fetch(`${session.base}`, {
+            method: "DELETE",
+            headers: { Authorization: `Bearer ${session.sessionToken}` },
+        }).catch(() => undefined);
+    }
+}
+/**
+ * Pull a multi-turn transcript out of the promptfoo test context. Expects
+ * `vars.transcript` to be a non-empty `string[]`; anything else falls back
+ * to single-turn mode (returns undefined). Empty strings are filtered out
+ * so an accidental trailing newline in YAML doesn't send a blank turn.
+ */
+function extractTranscript(context) {
+    const raw = context?.vars?.transcript;
+    if (!Array.isArray(raw))
+        return undefined;
+    const turns = raw.filter((t) => typeof t === "string" && t.trim().length > 0);
+    return turns.length > 0 ? turns : undefined;
+}
+function parseJSON(str) {
+    try {
+        const parsed = JSON.parse(str);
+        if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
+            return parsed;
+        }
+        return null;
+    }
+    catch {
+        return null;
+    }
+}
+function normaliseToolUseEvent(data) {
+    const name = data.name;
+    if (typeof name !== "string" || !name)
+        return null;
+    const call = {
+        name,
+        input: data.input ?? null,
+    };
+    if (typeof data.toolCallId === "string")
+        call.toolCallId = data.toolCallId;
+    if (data.isError === true)
+        call.isError = true;
+    const summary = data.result_summary;
+    if (summary && typeof summary === "object") {
+        const parsed = {};
+        const ids = summary.event_ids;
+        if (Array.isArray(ids)) {
+            const numeric = ids.filter((id) => typeof id === "number");
+            if (numeric.length > 0)
+                parsed.event_ids = numeric;
+        }
+        const snippetsRaw = summary.snippets;
+        if (Array.isArray(snippetsRaw)) {
+            const snippets = [];
+            for (const item of snippetsRaw) {
+                if (!item || typeof item !== "object")
+                    continue;
+                const id = item.id;
+                const text = item.text;
+                if (typeof id === "number" && typeof text === "string") {
+                    snippets.push({ id, text });
+                }
+            }
+            if (snippets.length > 0)
+                parsed.snippets = snippets;
+        }
+        const errorMsg = summary.error;
+        if (typeof errorMsg === "string")
+            parsed.error = errorMsg;
+        if (Object.keys(parsed).length > 0)
+            call.result_summary = parsed;
+    }
+    return call;
+}
+//# sourceMappingURL=provider.js.map

package/dist/provider.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"provider.js","sourceRoot":"","sources":["../src/provider.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAiFzC,MAAM,oBAAoB,GAAG,IAAI,GAAG,CAAC,CAAC,eAAe,EAAE,oBAAoB,CAAC,CAAC,CAAC;AAE9E;;;;;;;;;;;;;;GAcG;AACH,MAAM,OAAO,YAAY;IACN,KAAK,CAAS;IACd,OAAO,CAAS;IAChB,KAAK,CAAS;IACd,gBAAgB,CAAqB;IACrC,aAAa,CAAqB;IAClC,gBAAgB,CAAS;IACzB,cAAc,CAAqB;IAEpD,YAAY,UAAwD,EAAE;QACpE,MAAM,GAAG,GAAG,OAAO,CAAC,MAAM,IAAI,EAAE,CAAC;QACjC,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,IAAI,OAAO,CAAC,GAAG,CAAC,UAAU,CAAC;QAClD,MAAM,OAAO,GACX,GAAG,CAAC,OAAO,IAAI,OAAO,CAAC,GAAG,CAAC,YAAY,IAAI,uBAAuB,CAAC;QACrE,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,IAAI,OAAO,CAAC,GAAG,CAAC,UAAU,CAAC;QAElD,IAAI,CAAC,KAAK,EAAE,CAAC;YACX,MAAM,IAAI,KAAK,CACb,sFAAsF,CACvF,CAAC;QACJ,CAAC;QACD,IAAI,CAAC,KAAK,EAAE,CAAC;YACX,MAAM,IAAI,KAAK,CACb,sFAAsF,CACvF,CAAC;QACJ,CAAC;QAED,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;QACnB,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;QAC3C,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;QACnB,IAAI,CAAC,gBAAgB,GAAG,GAAG,CAAC,QAAQ,CAAC;QACrC,IAAI,CAAC,aAAa,GAAG,GAAG,CAAC,KAAK,CAAC;QAC/B,IAAI,CAAC,gBAAgB,GAAG,GAAG,CAAC,SAAS,IAAI,OAAO,CAAC;QACjD,IAAI,CAAC,cAAc,GAAG,GAAG,CAAC,MAAM,CAAC;IACnC,CAAC;IAED,EAAE;QACA,OAAO,QAAQ,IAAI,CAAC,KAAK,EAAE,CAAC;IAC9B,CAAC;IAED,KAAK,CAAC,OAAO,CACX,MAAc,EACd,OAA0B;QAE1B,MAAM,MAAM,GAAG,IAAI,CAAC,cAAc,IAAI,aAAa,UAAU,EAAE,EAAE,CAAC;QAClE,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,aAAa,CAAC,MAAM,CAAC,CAAC;QAEjD,sEAAsE;QACtE,uEAAuE;QACvE,yEAAyE;QACzE,iDAAiD;QACjD,MAAM,KAAK,GAAG,iBAAiB,CAAC,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QAErD,IAAI,CAAC;YACH,IAAI,YAA2C,CAAC;YAEhD,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;gBACzB,YAAY,GAAG,MAAM,IAAI,CAAC,cAAc,CACtC,OAAO,EACP,IAAI,EACJ,IAAI,CAAC,gBAAgB,CACtB,CAAC;gBAEF,mEAAmE;gBACnE,0CAA0C;gBAC1C,IAAI,YAAY,CAAC,KAAK,EAAE,CAAC;oBACvB,OAAO;wBACL,MAAM,EAAE,YAAY,CAAC,IAAI;wBACzB,KAAK,EAAE,YAAY,CAAC,KAAK;wBACzB,QAAQ,EAAE;4BACR,KAAK,EAAE,IAAI,CAAC,KAAK;4BACjB,MAAM;4BACN,OAAO,EAAE,YAAY,CAAC,OAAO;yBAC9B;qBACF,CAAC;gBACJ,CAAC;YACH,CAAC;YAED,wEAAwE;YACxE,mBAAmB;YACnB,MAAM,QAAQ,GAAG,YAAiC,CAAC;YAEnD,OAAO;gBACL,MAAM,EAAE,QAAQ,CAAC,IAAI;gBACrB,UAAU,EAAE,QAAQ,CAAC,MAAM;oBACzB,CAAC,CAAC;wBACE,MAAM,EAAE,QAAQ,CAAC,MAAM,CAAC,WAAW;wBACnC,UAAU,EAAE,QAAQ,CAAC,MAAM,CAAC,YAAY;wBACxC,KAAK,EAAE,QAAQ,CAAC,MAAM,CAAC,WAAW;qBACnC;oBACH,CAAC,CAAC,SAAS;gBACb,QAAQ,EAAE;oBACR,KAAK,EAAE,IAAI,CAAC,KAAK;oBACjB,MAAM;oBACN,OAAO,EAAE,QAAQ,CAAC,OAAO;oBACzB,GAAG,CAAC,QAAQ,CAAC,SAAS,IAAI,QAAQ,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC;wBACrD,CAAC,CAAC,EAAE,SAAS,EAAE,QAAQ,CAAC,SAAS,EAAE;wBACnC,CAAC,CAAC,EAAE,CAAC;oBACP,GAAG,CAAC,QAAQ,CAAC,gBAAgB;wBAC3B,CAAC,CAAC,EAAE,gBAAgB,EAAE,QAAQ,CAAC,gBAAgB,EAAE;wBACjD,CAAC,CAAC,EAAE,CAAC;iBACR;aACF,CAAC;QACJ,CAAC;gBAAS,CAAC;YACT,MAAM,IAAI,CAAC,aAAa,CAAC,OAAO,CAAC,CAAC;QACpC,CAAC;IACH,CAAC;IAED,2EAA2E;IAEnE,KAAK,CAAC,aAAa,CAAC,MAAc;QACxC,MAAM,IAAI,GAA4B;YACpC,OAAO,EAAE,IAAI,CAAC,KAAK;YACnB,MAAM;YACN,QAAQ,EAAE,IAAI;YACd,MAAM,EAAE,IAAI;SACb,CAAC;QACF,IAAI,IAAI,CAAC,gBAAgB;YAAE,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,gBAAgB,CAAC;QACjE,IAAI,IAAI,CAAC,aAAa;YAAE,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,aAAa,CAAC;QAExD,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,GAAG,IAAI,CAAC,OAAO,qBAAqB,EAAE;YAC5D,MAAM,EAAE,MAAM;YACd,OAAO,EAAE;gBACP,cAAc,EAAE,kBAAkB;gBAClC,aAAa,EAAE,UAAU,IAAI,CAAC,KAAK,EAAE;aACtC;YACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;SAC3B,CAAC,CAAC;QAEH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;YACZ,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,EAAE,CAAC,CAAC;YAC9C,MAAM,IAAI,KAAK,CAAC,kCAAkC,GAAG,CAAC,MAAM,MAAM,IAAI,EAAE,CAAC,CAAC;QAC5E,CAAC;QAED,MAAM,IAAI,GAAG,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAAuC,CAAC;QACtE,OAAO;YACL,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,YAAY,EAAE,IAAI,CAAC,KAAK;YACxB,uEAAuE;YACvE,4CAA4C;YAC5C,IAAI,EAAE,GAAG,IAAI,CAAC,OAAO,uBAAuB,IAAI,CAAC,OAAO,EAAE;SAC3D,CAAC;IACJ,CAAC;IAEO,KAAK,CAAC,WAAW,CACvB,OAAgB,EAChB,OAAe;QAEf,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,GAAG,OAAO,CAAC,IAAI,WAAW,EAAE;YAClD,MAAM,EAAE,MAAM;YACd,OAAO,EAAE;gBACP,cAAc,EAAE,kBAAkB;gBAClC,aAAa,EAAE,UAAU,OAAO,CAAC,YAAY,EAAE;aAChD;YACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,CAAC;SAClC,CAAC,CAAC;QAEH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;YACZ,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,EAAE,CAAC,CAAC;YAC9C,MAAM,IAAI,KAAK,CAAC,2BAA2B,GAAG,CAAC,MAAM,MAAM,IAAI,EAAE,CAAC,CAAC;QACrE,CAAC;QAED,MAAM,IAAI,GAAG,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAG7B,CAAC;QACF,MAAM,OAAO,GAAG,IAAI,CAAC,WAAW,EAAE,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;QAChD,OAAO,EAAE,OAAO,EAAE,SAAS,EAAE,IAAI,CAAC,SAAS,EAAE,CAAC;IAChD,CAAC;IAEO,KAAK,CAAC,eAAe,CAC3B,OAAgB,EAChB,SAAiB,EACjB,SAAkB;QAElB,MAAM,UAAU,GAAG,IAAI,eAAe,EAAE,CAAC;QACzC,MAAM,KAAK,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,UAAU,CAAC,KAAK,EAAE,EAAE,SAAS,CAAC,CAAC;QAC9D,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACzB,IAAI,gBAES,CAAC;QAEd,IAAI,CAAC;YACH,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,GAAG,OAAO,CAAC,IAAI,SAAS,EAAE;gBAChD,OAAO,EAAE,EAAE,aAAa,EAAE,UAAU,OAAO,CAAC,YAAY,EAAE,EAAE;gBAC5D,MAAM,EAAE,UAAU,CAAC,MAAM;aAC1B,CAAC,CAAC;YAEH,IAAI,CAAC,GAAG,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC;gBACzB,MAAM,IAAI,KAAK,CAAC,0BAA0B,GAAG,CAAC,MAAM,GAAG,CAAC,CAAC;YAC3D,CAAC;YAED,MAAM,MAAM,GAAG,GAAG,CAAC,IAAI,CAAC,SAAS,EAAE,CAAC;YACpC,gBAAgB,GAAG,MAAM,CAAC;YAC1B,MAAM,OAAO,GAAG,IAAI,WAAW,EAAE,CAAC;YAClC,IAAI,MAAM,GAAG,EAAE,CAAC;YAChB,IAAI,YAAY,GAAG,EAAE,CAAC;YACtB,IAAI,IAAI,GAAG,EAAE,CAAC;YACd,MAAM,SAAS,GAAmB,EAAE,CAAC;YACrC,MAAM,iBAAiB,GAAwC,EAAE,CAAC;YAClE,MAAM,cAAc,GAAG,IAAI,GAAG,EAAU,CAAC;YAEzC,MAAM,aAAa,GAAG,CAAC,cAAuB,EAAW,EAAE;gBACzD,IAAI,CAAC,SAAS;oBAAE,OAAO,IAAI,CAAC;gBAC5B,OAAO,CACL,OAAO,cAAc,KAAK,QAAQ,IAAI,cAAc,KAAK,SAAS,CACnE,CAAC;YACJ,CAAC,CAAC;YAEF,MAAM,QAAQ,GAAG,CACf,QAAoC,EAAE,EACnB,EAAE;gBACrB,MAAM,MAAM,GAAsB;oBAChC,IAAI;oBACJ,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK;oBAC7B,GAAG,KAAK;iBACT,CAAC;gBACF,IAAI,SAAS,CAAC,MAAM,GAAG,CAAC;oBAAE,MAAM,CAAC,SAAS,GAAG,SAAS,CAAC;gBACvD,IAAI,iBAAiB,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;oBACjC,MAAM,CAAC,gBAAgB,GAAG,iBAAiB;yBACxC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;yBAClB,IAAI,CAAC,MAAM,CAAC,CAAC;gBAClB,CAAC;gBACD,OAAO,MAAM,CAAC;YAChB,CAAC,CAAC;YAEF,OAAO,IAAI,EAAE,CAAC;gBACZ,MAAM,EAAE,IAAI,EAAE,KAAK,EAAE,GAAG,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC;gBAC5C,IAAI,IAAI;oBAAE,MAAM;gBAEhB,MAAM,IAAI,OAAO,CAAC,MAAM,CAAC,KAAK,EAAE,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC;gBAClD,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;gBACjC,MAAM,GAAG,KAAK,CAAC,GAAG,EAAE,IAAI,EAAE,CAAC;gBAE3B,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;oBACzB,IAAI,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,EAAE,CAAC;wBAC/B,YAAY,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;oBACtC,CAAC;yBAAM,IAAI,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,IAAI,YAAY,EAAE,CAAC;wBACrD,MAAM,IAAI,GAAG,SAAS,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;wBACtC,IAAI,CAAC,IAAI;4BAAE,SAAS;wBAEpB,QAAQ,YAAY,EAAE,CAAC;4BACrB,KAAK,QAAQ;gCACX,IACE,OAAO,IAAI,CAAC,OAAO,KAAK,QAAQ;oCAChC,aAAa,CAAC,IAAI,CAAC,SAAS,CAAC,EAC7B,CAAC;oCACD,IAAI,IAAI,IAAI,CAAC,OAAO,CAAC;gCACvB,CAAC;gCACD,MAAM;4BACR,KAAK,UAAU,CAAC,CAAC,CAAC;gCAChB,IAAI,CAAC,aAAa,CAAC,IAAI,CAAC,SAAS,CAAC;oCAAE,MAAM;gCAC1C,MAAM,IAAI,GAAG,qBAAqB,CAAC,IAAI,CAAC,CAAC;gCACzC,IAAI,CAAC,IAAI;oCAAE,MAAM;gCACjB,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;gCACrB,IACE,oBAAoB,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC;oCACnC,IAAI,CAAC,cAAc,EAAE,QAAQ,EAC7B,CAAC;oCACD,KAAK,MAAM,OAAO,IAAI,IAAI,CAAC,cAAc,CAAC,QAAQ,EAAE,CAAC;wCACnD,IAAI,cAAc,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,CAAC;4CAAE,SAAS;wCAC7C,cAAc,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC;wCAC/B,iBAAiB,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;oCAClC,CAAC;gCACH,CAAC;gCACD,MAAM;4BACR,CAAC;4BACD,KAAK,UAAU,CAAC,CAAC,CAAC;gCAChB,IAAI,CAAC,aAAa,CAAC,IAAI,CAAC,SAAS,CAAC;oCAAE,MAAM;gCAC1C,MAAM,KAAK,GAAG,IAAI,CAAC,KAA2C,CAAC;gCAC/D,OAAO,QAAQ,CAAC;oCACd,MAAM,EAAE,KAAK;wCACX,CAAC,CAAC;4CACE,WAAW,EAAE,KAAK,CAAC,YAAY,IAAI,KAAK,CAAC,WAAW;4CACpD,YAAY,EAAE,KAAK,CAAC,aAAa,IAAI,KAAK,CAAC,YAAY;4CACvD,WAAW,EACT,CAAC,KAAK,CAAC,YAAY,IAAI,KAAK,CAAC,WAAW,IAAI,CAAC,CAAC;gDAC9C,CAAC,KAAK,CAAC,aAAa,IAAI,KAAK,CAAC,YAAY,IAAI,CAAC,CAAC;yCACnD;wCACH,CAAC,CAAC,SAAS;iCACd,CAAC,CAAC;4BACL,CAAC;4BACD,KAAK,OAAO;gCACV,IAAI,CAAC,aAAa,CAAC,IAAI,CAAC,SAAS,CAAC;oCAAE,MAAM;gCAC1C,OAAO,QAAQ,CAAC;oCACd,KAAK,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,IAAI,eAAe,CAAC;iCAC7C,CAAC,CAAC;wBACP,CAAC;wBACD,YAAY,GAAG,EAAE,CAAC;oBACpB,CAAC;yBAAM,IAAI,IAAI,KAAK,EAAE,EAAE,CAAC;wBACvB,YAAY,GAAG,EAAE,CAAC;oBACpB,CAAC;gBACH,CAAC;YACH,CAAC;YAED,OAAO,QAAQ,EAAE,CAAC;QACpB,CAAC;QAAC,OAAO,GAAY,EAAE,CAAC;YACtB,IAAI,GAAG,YAAY,KAAK,IAAI,GAAG,CAAC,IAAI,KAAK,YAAY,EAAE,CAAC;gBACtD,OAAO,EAAE,IAAI,EAAE,EAAE,EAAE,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,EAAE,KAAK,EAAE,SAAS,EAAE,CAAC;YACvE,CAAC;YACD,MAAM,GAAG,CAAC;QACZ,CAAC;gBAAS,CAAC;YACT,YAAY,CAAC,KAAK,CAAC,CAAC;YACpB,IAAI,gBAAgB,EAAE,CAAC;gBACrB,MAAM,gBAAgB,CAAC,MAAM,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC;YACzD,CAAC;QACH,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,cAAc,CAC1B,OAAgB,EAChB,OAAe,EACf,SAAiB;QAEjB,MAAM,EAAE,OAAO,EAAE,SAAS,EAAE,GAAG,MAAM,IAAI,CAAC,WAAW,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;QACxE,MAAM,QAAQ,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,OAAO,EAAE,SAAS,EAAE,SAAS,CAAC,CAAC;QAC3E,QAAQ,CAAC,OAAO,GAAG,OAAO,CAAC;QAC3B,OAAO,QAAQ,CAAC;IAClB,CAAC;IAEO,KAAK,CAAC,aAAa,CAAC,OAAgB;QAC1C,MAAM,KAAK,CAAC,GAAG,OAAO,CAAC,IAAI,EAAE,EAAE;YAC7B,MAAM,EAAE,QAAQ;YAChB,OAAO,EAAE,EAAE,aAAa,EAAE,UAAU,OAAO,CAAC,YAAY,EAAE,EAAE;SAC7D,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC;IAC5B,CAAC;CACF;AAQD;;;;;GAKG;AACH,SAAS,iBAAiB,CACxB,OAAqC;IAErC,MAAM,GAAG,GAAG,OAAO,EAAE,IAAI,EAAE,UAAU,CAAC;IACtC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC;QAAE,OAAO,SAAS,CAAC;IAC1C,MAAM,KAAK,GAAG,GAAG,CAAC,MAAM,CACtB,CAAC,CAAC,EAAe,EAAE,CAAC,OAAO,CAAC,KAAK,QAAQ,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC,MAAM,GAAG,CAAC,CACjE,CAAC;IACF,OAAO,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC;AAC9C,CAAC;AAED,SAAS,SAAS,CAAC,GAAW;IAC5B,IAAI,CAAC;QACH,MAAM,MAAM,GAAY,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;QACxC,IAAI,MAAM,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;YACnE,OAAO,MAAiC,CAAC;QAC3C,CAAC;QACD,OAAO,IAAI,CAAC;IACd,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,IAAI,CAAC;IACd,CAAC;AACH,CAAC;AAED,SAAS,qBAAqB,CAC5B,IAA6B;IAE7B,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC;IACvB,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,CAAC,IAAI;QAAE,OAAO,IAAI,CAAC;IACnD,MAAM,IAAI,GAAiB;QACzB,IAAI;QACJ,KAAK,EAAE,IAAI,CAAC,KAAK,IAAI,IAAI;KAC1B,CAAC;IACF,IAAI,OAAO,IAAI,CAAC,UAAU,KAAK,QAAQ;QAAE,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;IAC3E,IAAI,IAAI,CAAC,OAAO,KAAK,IAAI;QAAE,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC;IAC/C,MAAM,OAAO,GAAG,IAAI,CAAC,cAAc,CAAC;IACpC,IAAI,OAAO,IAAI,OAAO,OAAO,KAAK,QAAQ,EAAE,CAAC;QAC3C,MAAM,MAAM,GAAgD,EAAE,CAAC;QAC/D,MAAM,GAAG,GAAI,OAAmC,CAAC,SAAS,CAAC;QAC3D,IAAI,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YACvB,MAAM,OAAO,GAAG,GAAG,CAAC,MAAM,CAAC,CAAC,EAAE,EAAgB,EAAE,CAAC,OAAO,EAAE,KAAK,QAAQ,CAAC,CAAC;YACzE,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC;gBAAE,MAAM,CAAC,SAAS,GAAG,OAAO,CAAC;QACrD,CAAC;QACD,MAAM,WAAW,GAAI,OAAkC,CAAC,QAAQ,CAAC;QACjE,IAAI,KAAK,CAAC,OAAO,CAAC,WAAW,CAAC,EAAE,CAAC;YAC/B,MAAM,QAAQ,GAAwC,EAAE,CAAC;YACzD,KAAK,MAAM,IAAI,IAAI,WAAW,EAAE,CAAC;gBAC/B,IAAI,CAAC,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ;oBAAE,SAAS;gBAChD,MAAM,EAAE,GAAI,IAAyB,CAAC,EAAE,CAAC;gBACzC,MAAM,IAAI,GAAI,IAA2B,CAAC,IAAI,CAAC;gBAC/C,IAAI,OAAO,EAAE,KAAK,QAAQ,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;oBACvD,QAAQ,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC;gBAC9B,CAAC;YACH,CAAC;YACD,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC;gBAAE,MAAM,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACtD,CAAC;QACD,MAAM,QAAQ,GAAI,OAA+B,CAAC,KAAK,CAAC;QACxD,IAAI,OAAO,QAAQ,KAAK,QAAQ;YAAE,MAAM,CAAC,KAAK,GAAG,QAAQ,CAAC;QAC1D,IAAI,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,MAAM,GAAG,CAAC;YAAE,IAAI,CAAC,cAAc,GAAG,MAAM,CAAC;IACnE,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC"}

package/package.json

@@ -0,0 +1,45 @@
+{
+  "name": "@lobu/promptfoo-provider",
+  "version": "8.0.0",
+  "description": "promptfoo custom provider for running evals against a Lobu agent",
+  "type": "module",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "exports": {
+    ".": {
+      "import": {
+        "types": "./dist/index.d.ts",
+        "default": "./dist/index.js"
+      },
+      "require": {
+        "types": "./dist/index.d.ts",
+        "default": "./dist/index.js"
+      }
+    }
+  },
+  "files": [
+    "dist"
+  ],
+  "scripts": {
+    "build": "tsc",
+    "typecheck": "tsc --noEmit",
+    "clean": "rm -rf dist"
+  },
+  "devDependencies": {
+    "@types/node": "^20.10.0",
+    "typescript": "^5.3.3"
+  },
+  "engines": {
+    "node": ">=18"
+  },
+  "license": "BUSL-1.1",
+  "publishConfig": {
+    "access": "public"
+  },
+  "homepage": "https://github.com/lobu-ai/lobu",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/lobu-ai/lobu.git",
+    "directory": "packages/promptfoo-provider"
+  }
+}