katt 0.0.3 → 0.0.5

package/README.md

@@ -1,20 +1,44 @@
 # Katt
+[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/raphaelpor/katt/blob/main/LICENSE) [![npm version](https://img.shields.io/npm/v/katt.svg?style=flat)](https://www.npmjs.com/package/katt)
 
 Katt is a lightweight testing framework for running AI Evals, inspired by [Jest](https://github.com/jestjs/jest).
 
 <img src="https://raw.githubusercontent.com/raphaelpor/katt/main/docs/logo.png" alt="Katt logo" width="250" />
 
+## Table of Contents
+
+- [Overview](#overview)
+- [API Documentation](#api-documentation)
+- [Hello World - Example](#hello-world---example)
+- [Main Features](#main-features)
+- [Usage](#usage)
+- [Installation](#installation)
+- [Basic Usage](#basic-usage)
+- [Using promptFile](#using-promptfile)
+- [Specifying AI Models](#specifying-ai-models)
+- [Development](#development)
+- [Setup](#setup)
+- [Available Scripts](#available-scripts)
+- [Verification Process](#verification-process)
+- [Project Structure](#project-structure)
+- [How It Works](#how-it-works)
+- [Requirements](#requirements)
+- [License](#license)
+- [Contributing](#contributing)
+
 ## Overview
 
 Katt is designed to evaluate and validate the behavior of AI agents like **Claude Code**, **GitHub Copilot**, **OpenAI Codex** and more. It provides a simple, intuitive API for writing tests that interact with AI models and assert their responses.
 
 ## API Documentation
 
-For a complete list of features and usage examples, see [docs/api-documentation.md](docs/api-documentation.md).
+For a complete list of features and usage examples, see [docs/api-documentation.md](https://github.com/raphaelpor/katt/blob/main/docs/api-documentation.md).
 
 ## Hello World - Example
 
 ```typescript
+import { expect, prompt } from "katt";
+
 const result = await prompt("If you read this just say 'hello world'");
 expect(result).toContain("hello world");
 ```
@@ -22,6 +46,8 @@ expect(result).toContain("hello world");
 It also supports the familiar `describe` and `it` syntax for organizing tests:
 
 ```typescript
+import { describe, expect, it, prompt } from "katt";
+
 describe("Greeting agent", () => {
   it("should say hello world", async () => {
     const result = await prompt("If you read this just say 'hello world'");
@@ -37,13 +63,22 @@ describe("Greeting agent", () => {
 - **Classification Matcher**: Built-in `toBeClassifiedAs()` matcher to grade a response against a target label on a 1-5 scale
 - **Concurrent Execution**: Runs eval files concurrently for faster test execution
 - **Model Selection**: Support for specifying custom AI models
+- **Configurable Timeouts**: Override prompt wait time per test or via `katt.json`
 
 ## Usage
 
+### Installation
+
+```bash
+npm install -g katt
+```
+
 ### Basic Usage
 
 1. Create a file with the `.eval.ts` or `.eval.js` extension and write your tests.
 ```typescript
+import { expect, prompt } from "katt";
+
 const result = await prompt("If you read this just say 'hello world'");
 expect(result).toContain("hello world");
 ```
@@ -51,7 +86,7 @@ expect(result).toContain("hello world");
 2. Run Katt from your project directory:
 
 ```bash
-npx katt
+katt
 ```
 
 ### Using promptFile
@@ -60,6 +95,8 @@ Load prompts from external files:
 
 ```javascript
 // test.eval.js
+import { describe, expect, it, promptFile } from "katt";
+
 describe("Working with files", () => {
   it("should load the file and respond", async () => {
     const result = await promptFile("./myPrompt.md");
@@ -73,6 +110,8 @@ describe("Working with files", () => {
 You can specify a custom model for your prompts:
 
 ```javascript
+import { describe, expect, it, prompt } from "katt";
+
 describe("Model selection", () => {
   it("should use a specific model", async () => {
     const promptString = "You are a helpful agent. Say hi and ask what you could help the user with.";
@@ -89,6 +128,9 @@ You can also set a default model for the project by adding a `katt.json` file in
 {
   "copilot": {
     "model": "gpt-5-mini"
+  },
+  "prompt": {
+    "timeoutMs": 240000
   }
 }
 ```
@@ -97,6 +139,7 @@ When this file exists:
 
 - `prompt("...")` and `promptFile("...")` use `copilot.model` by default
 - `prompt("...", { model: "..." })` still overrides the config value
+- `prompt.timeoutMs` sets the default wait timeout for long-running prompts
 
 ## Development
 
@@ -162,7 +205,7 @@ MIT
 
 ## Contributing
 
-We welcome contributions from the community! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) guide for detailed information on how to contribute to Katt.
+We welcome contributions from the community! Please see our [CONTRIBUTING.md](https://github.com/raphaelpor/katt/blob/main/CONTRIBUTING.md) guide for detailed information on how to contribute to Katt.
 
 Quick start:
 1. Fork the repository
@@ -171,4 +214,4 @@ Quick start:
 4. Run the verification process
 5. Submit a pull request
 
-For detailed guidelines, development setup, coding standards, and more, check out our [contribution guide](CONTRIBUTING.md).
+For detailed guidelines, development setup, coding standards, and more, check out our [contribution guide](https://github.com/raphaelpor/katt/blob/main/CONTRIBUTING.md).

package/dist/index.js

@@ -1,183 +1,56 @@
-#!/usr/bin/env node
-import { AsyncLocalStorage as E } from "node:async_hooks";
-import { CopilotClient as et } from "@github/copilot-sdk";
-import { readFile as O, readdir as nt } from "node:fs/promises";
-import { resolve as F, dirname as N, isAbsolute as ot, basename as st, join as it } from "node:path";
-import { readFileSync as H, writeFileSync as R, mkdirSync as rt } from "node:fs";
-import { fileURLToPath as ct, pathToFileURL as at } from "node:url";
-const D = new E(), lt = {
-  describeStack: [],
-  itStack: [],
-  tokenUsageStack: [],
-  modelStack: []
-};
-let M = 0, G = 0;
-const A = [], $ = [];
-let T = 0;
-function u() {
-  return D.getStore() ?? lt;
-}
-function X(t) {
-  return {
-    describeStack: [...t.describeStack],
-    itStack: [...t.itStack],
-    tokenUsageStack: [...t.tokenUsageStack],
-    modelStack: [...t.modelStack]
-  };
-}
-function ut() {
-  return M += 1, `d${M}`;
-}
-function dt() {
-  return G += 1, `i${G}`;
-}
-function V(t, e) {
-  const n = e ?? X(u());
-  return D.run(n, t);
-}
-function P() {
-  return X(u());
-}
-function gt(t) {
-  u().describeStack.push({ id: ut(), description: t });
-}
-function y() {
-  u().describeStack.pop();
-}
-function Y() {
-  return u().describeStack.map((t) => t.description).join(" > ");
-}
-function ft(t) {
-  u().itStack.push({ id: dt(), description: t }), u().tokenUsageStack.push(0), u().modelStack.push(void 0);
-}
-function v() {
-  u().itStack.pop(), u().tokenUsageStack.pop(), u().modelStack.pop();
-}
-function z() {
-  return u().itStack.map((t) => t.description).join(" > ");
-}
-function pt(t) {
-  if (!Number.isFinite(t) || t <= 0)
-    return;
-  const e = u(), n = e.tokenUsageStack.length - 1;
-  n < 0 || (e.tokenUsageStack[n] += t);
-}
-function ht() {
-  const t = u(), e = t.tokenUsageStack.length - 1;
-  return e < 0 ? 0 : t.tokenUsageStack[e] ?? 0;
-}
-function It(t) {
-  if (t.length === 0)
-    return;
-  const e = u(), n = e.modelStack.length - 1;
-  n < 0 || (e.modelStack[n] = t);
-}
-function mt() {
-  const t = u(), e = t.modelStack.length - 1;
-  if (!(e < 0))
-    return t.modelStack[e];
-}
-function b(t) {
-  A.push(t);
-}
-function Ct() {
-  T += 1;
-}
-function St() {
-  return T;
-}
-function At() {
-  T = 0;
-}
-function $t(t) {
-  $.push(t);
-}
-function bt() {
-  return [...$];
-}
-function U() {
-  return $.length;
-}
-function kt() {
-  $.length = 0;
-}
-async function wt() {
-  const t = [];
-  for (; A.length > 0; ) {
-    const e = A.splice(0, A.length), n = await Promise.allSettled(e);
-    t.push(...n);
-  }
-  return t;
+import { r as E, c as N, a as O, p as I, b as m, l as d, d as w, g as F, e as U, f as C, s as _, h as R, i as D, j as z, k as B, m as W, n as K } from "./runCli-CDRmZ5hw.js";
+import { o as St } from "./runCli-CDRmZ5hw.js";
+import { CopilotClient as H } from "@github/copilot-sdk";
+import { readFile as A } from "node:fs/promises";
+import { resolve as j, dirname as T, isAbsolute as Y, basename as J, join as q } from "node:path";
+import { readFileSync as G, writeFileSync as S, mkdirSync as Q } from "node:fs";
+function vt(t, e) {
+  E(() => {
+    O(), I(t);
+    const n = F(), r = Date.now(), i = () => F() === n, o = () => Date.now() - r;
+    try {
+      const s = e();
+      if (s && typeof s.then == "function") {
+        m(
+          s.then(() => {
+            d(!0, o());
+          }).catch((a) => {
+            throw d(!1, o()), a;
+          }).finally(() => {
+            w();
+          })
+        );
+        return;
+      }
+    } catch (s) {
+      throw d(!1, o()), w(), s;
+    }
+    d(i(), o()), w();
+  }, N());
 }
-function yt(t, e) {
-  V(() => {
-    gt(t);
+function Tt(t, e) {
+  E(() => {
+    U(t);
     try {
       const n = e();
       if (n && typeof n.then == "function") {
-        b(
+        m(
           n.finally(() => {
-            y();
+            C();
           })
         );
         return;
       }
     } catch (n) {
-      throw y(), n;
+      throw C(), n;
     }
-    y();
-  }, P());
-}
-const vt = "\x1B[1;36m", Lt = "\x1B[33m", jt = "\x1B[38;5;208m", xt = "\x1B[1;38;5;208m", k = "\x1B[0m";
-function f(t) {
-  return `${vt}${t}${k}`;
-}
-function C(t) {
-  return `${Lt}${t}${k}`;
-}
-function W(t) {
-  return `${jt}${t}${k}`;
+    C();
+  }, N());
 }
-function Ft(t) {
-  return `${xt}${t}${k}`;
-}
-let x = "";
-function Nt() {
-  x = "";
-}
-function Tt({
-  suitePath: t,
-  casePath: e,
-  didPass: n,
-  durationMs: o,
-  model: i,
-  tokenUsage: s
-}) {
-  const r = t.length > 0 ? t : "(root)", a = e.length > 0 ? e : "(root)";
-  x !== r && (console.log(`Suite "${f(r)}"`), x = r);
-  const l = n ? "✅ Passed in" : "❌ Failed in", d = [
-    `Test "${f(a)}"`,
-    `- ${l} ${f(`${o}ms`)}`
-  ];
-  i && d.push(`- Model ${f(i)}`), (s ?? 0) > 0 && d.push(`- Tokens used ${f(String(s))}`), d.push("---"), console.log(d.join(`
-`));
-}
-function I(t, e, n = "(root)") {
-  const o = z();
-  Tt({
-    suitePath: Y(),
-    casePath: o.length > 0 ? o : n,
-    didPass: t,
-    durationMs: e,
-    model: mt(),
-    tokenUsage: ht()
-  });
-}
-const B = new E();
-function Bt(t, e) {
+function V(t, e) {
   return typeof t == "object" && t !== null && "code" in t && t.code === e;
 }
-function Jt(t) {
+function X(t) {
   try {
     const e = JSON.parse(t);
     return typeof e == "object" && e !== null ? e : void 0;
@@ -186,110 +59,133 @@ function Jt(t) {
     return;
   }
 }
-function Rt(t) {
-  const e = t?.copilot;
-  if (typeof e != "object" || e === null || Array.isArray(e))
-    return;
-  const n = {
-    ...e
-  }, o = n.model;
-  return (typeof o != "string" || o.length === 0) && delete n.model, Object.keys(n).length > 0 ? n : void 0;
-}
-async function Mt() {
-  const t = F(process.cwd(), "katt.json");
+async function Z() {
+  const t = j(process.cwd(), "katt.json");
   try {
-    const e = await O(t, "utf8");
-    return Rt(Jt(e));
+    const e = await A(t, "utf8");
+    return X(e);
   } catch (e) {
-    if (Bt(e, "ENOENT"))
+    if (V(e, "ENOENT"))
       return;
     console.warn(`Failed to read katt.json: ${String(e)}`);
     return;
   }
 }
-function Q(t) {
+function tt(t) {
+  const e = t?.copilot;
+  if (typeof e != "object" || e === null || Array.isArray(e))
+    return;
+  const n = {
+    ...e
+  }, r = n.model;
+  return (typeof r != "string" || r.length === 0) && delete n.model, Object.keys(n).length > 0 ? n : void 0;
+}
+function et(t) {
+  if (!(typeof t != "number" || !Number.isFinite(t)) && !(t <= 0))
+    return Math.floor(t);
+}
+function nt(t) {
+  const e = t?.prompt;
+  if (!(typeof e != "object" || e === null || Array.isArray(e)))
+    return et(e.timeoutMs);
+}
+async function ot() {
+  const t = await Z();
+  return {
+    copilot: tt(t),
+    promptTimeoutMs: nt(t)
+  };
+}
+const rt = 6e5;
+function P(t) {
   return typeof t == "string" && t.length > 0 ? t : void 0;
 }
-function L(t) {
+function v(t) {
   if (!t)
     return;
   const e = { ...t };
   if (e.model !== void 0) {
-    const n = Q(e.model);
+    const n = P(e.model);
     n ? e.model = n : delete e.model;
   }
   return Object.keys(e).length > 0 ? e : void 0;
 }
-function S(t) {
-  return !Number.isFinite(t) || (t ?? 0) <= 0 ? 0 : Math.floor(t ?? 0);
+function M(t) {
+  if (!(typeof t != "number" || !Number.isFinite(t)) && !(t <= 0))
+    return Math.floor(t);
 }
-function Gt(t) {
-  return S(t.inputTokens) + S(t.outputTokens) + S(t.cacheReadTokens) + S(t.cacheWriteTokens);
+function h(t) {
+  return !Number.isFinite(t) || (t ?? 0) <= 0 ? 0 : Math.floor(t ?? 0);
 }
-async function w(t, e = {}) {
-  const n = L(await Mt()), o = L(e), i = L({
-    ...n ?? {},
-    ...o ?? {}
-  }), s = Q(i?.model), r = new et({ useLoggedInUser: !0 });
-  let a, l, d = 0;
+function it(t) {
+  return h(t.inputTokens) + h(t.outputTokens) + h(t.cacheReadTokens) + h(t.cacheWriteTokens);
+}
+async function $(t, e = {}) {
+  const { timeoutMs: n, ...r } = e, i = await ot(), o = v(i.copilot), s = v(
+    r
+  ), a = v({
+    ...o ?? {},
+    ...s ?? {}
+  }), c = M(i.promptTimeoutMs), L = M(n) ?? c ?? rt, b = P(a?.model), g = new H({ useLoggedInUser: !0 });
+  let p, x, y = 0;
   try {
-    await r.start(), a = await r.createSession(i), l = a.on("assistant.usage", (c) => {
-      d += Gt(c.data);
+    await g.start(), p = await g.createSession(a), x = p.on("assistant.usage", (f) => {
+      y += it(f.data);
     });
-    const g = await a.sendAndWait({ prompt: t });
-    if (!g?.data?.content)
+    const l = await p.sendAndWait({ prompt: t }, L);
+    if (!l?.data?.content)
       throw new Error("Copilot did not return a response.");
-    return s && It(s), g.data.content;
+    return b && _(b), l.data.content;
   } finally {
-    const g = [];
-    if (l?.(), d > 0 && pt(d), a)
+    const l = [];
+    if (x?.(), y > 0 && R(y), p)
       try {
-        await a.destroy();
-      } catch (c) {
-        g.push(c);
+        await p.destroy();
+      } catch (f) {
+        l.push(f);
       }
     try {
-      const c = await r.stop();
-      g.push(...c);
-    } catch (c) {
-      g.push(c);
+      const f = await g.stop();
+      l.push(...f);
+    } catch (f) {
+      l.push(f);
     }
-    g.length > 0 && console.error(
-      `Copilot cleanup encountered ${g.length} error(s).`
+    l.length > 0 && console.error(
+      `Copilot cleanup encountered ${l.length} error(s).`
     );
   }
 }
-async function Ut(t, e = {}) {
-  const n = B.getStore(), o = n?.evalFile ? N(n.evalFile) : process.cwd(), i = ot(t) ? t : F(o, t), s = await O(i, "utf8");
-  return w(s, e);
+async function $t(t, e = {}) {
+  const n = D.getStore(), r = n?.evalFile ? T(n.evalFile) : process.cwd(), i = Y(t) ? t : j(r, t), o = await A(i, "utf8");
+  return $(o, e);
 }
-function p(t) {
-  $t({
-    describePath: Y(),
-    itPath: z(),
+function u(t) {
+  z({
+    describePath: W(),
+    itPath: B(),
     message: t
   });
 }
-async function Wt(t, e) {
-  const n = Date.now(), o = `expected '${t}' to satisfy '${e}'`, i = w(`Evaluate if the expectation is fulfiled in by the input.
+async function st(t, e) {
+  const n = Date.now(), r = `expected '${t}' to satisfy '${e}'`, i = $(`Evaluate if the expectation is fulfiled in by the input.
         Expectation: "${e}".
         Input:
         ---
         ${t}
         ---
         Important: Answer with "Yes" or "No" only, without any additional text.
-        `).then((s) => {
-    s.includes("Yes") ? I(!0, Date.now() - n, "promptCheck") : s.includes("No") ? p(o) : p(`failed to evaluate expectation '${e}'`);
+        `).then((o) => {
+    o.includes("Yes") ? d(!0, Date.now() - n, "promptCheck") : o.includes("No") ? u(r) : u(`failed to evaluate expectation '${e}'`);
   });
-  return b(i), i;
+  return m(i), i;
 }
-const Zt = 3;
-function Et(t) {
+const at = 3;
+function ct(t) {
   const e = t.match(/\b([1-5])\b/);
   return !e || !e[1] ? null : Number(e[1]);
 }
-async function Ot(t, e, n) {
-  const o = Date.now(), i = n?.threshold ?? Zt, s = w(
+async function ut(t, e, n) {
+  const r = Date.now(), i = n?.threshold ?? at, o = $(
     `Classify the input by how "${e}" it is on a scale of 1 to 5.
 1 means "not ${e}" and 5 means "very ${e}".
 Return only a single number: 1, 2, 3, 4, or 5.
@@ -299,89 +195,82 @@ Input:
 ${t}
 ---`,
     n?.model ? { model: n.model } : void 0
-  ).then((r) => {
-    const a = Et(r);
+  ).then((s) => {
+    const a = ct(s);
     if (a === null) {
-      p(
-        `failed to classify as '${e}'. Evaluator returned '${r}'`
+      u(
+        `failed to classify as '${e}'. Evaluator returned '${s}'`
       );
       return;
     }
-    const l = `expected response to be classified as '${e}' with score >= ${i}, got ${a}`;
+    const c = `expected response to be classified as '${e}' with score >= ${i}, got ${a}`;
     if (a < i) {
-      p(l);
+      u(c);
       return;
     }
-    I(
+    d(
       !0,
-      Date.now() - o,
+      Date.now() - r,
       "toBeClassifiedAs"
     );
   });
-  return b(s), s;
+  return m(o), o;
 }
-function Ht(t, e) {
+function lt(t, e) {
   const n = `expected '${t}' to include '${e}'`;
-  t.includes(e) || p(n);
+  t.includes(e) || u(n);
 }
-let K = !1;
-function Dt(t) {
-  K = t;
-}
-function Xt() {
-  return K;
-}
-function Vt(t) {
-  const n = st(t).replace(/\.eval\.[^./\\]+$/, "");
-  return it(
-    N(t),
+function ft(t) {
+  const n = J(t).replace(/\.eval\.[^./\\]+$/, "");
+  return q(
+    T(t),
     "__snapshots__",
     `${n}.snap.md`
   );
 }
-function Z(t) {
+function k(t) {
   return t.split(/\r?\n/);
 }
-function Pt(t, e) {
+function dt(t, e) {
   if (t === e)
     return "  (no diff)";
-  const n = Z(t), o = Z(e), i = Math.max(n.length, o.length), s = [];
-  for (let r = 0; r < i; r += 1) {
-    const a = n[r], l = o[r];
-    if (a !== l) {
-      if (a === void 0 && l !== void 0) {
-        s.push(`+ ${l}`);
+  const n = k(t), r = k(e), i = Math.max(n.length, r.length), o = [];
+  for (let s = 0; s < i; s += 1) {
+    const a = n[s], c = r[s];
+    if (a !== c) {
+      if (a === void 0 && c !== void 0) {
+        o.push(`+ ${c}`);
         continue;
       }
-      if (a !== void 0 && l === void 0) {
-        s.push(`- ${a}`);
+      if (a !== void 0 && c === void 0) {
+        o.push(`- ${a}`);
         continue;
       }
-      s.push(`- ${a ?? ""}`), s.push(`+ ${l ?? ""}`);
+      o.push(`- ${a ?? ""}`), o.push(`+ ${c ?? ""}`);
     }
   }
-  return s.join(`
+  return o.join(`
 `);
 }
-function Yt(t) {
-  const e = B.getStore()?.evalFile;
+function pt(t) {
+  const e = D.getStore()?.evalFile;
   if (!e) {
-    p(
+    u(
       "toMatchSnapshot can only be used while running an eval file."
     );
     return;
   }
-  const n = Vt(e);
+  const n = ft(e);
   try {
-    const o = H(n, "utf8");
-    if (o === t)
+    const r = G(n, "utf8");
+    if (r === t)
       return;
-    if (Xt()) {
-      R(n, t, "utf8");
+    if (K()) {
+      S(n, t, "utf8");
       return;
     }
-    const i = Pt(o, t);
-    p(
+    const i = dt(r, t);
+    u(
       [
         `Snapshot mismatch at ${n}`,
         "",
@@ -392,175 +281,43 @@ function Yt(t) {
       ].join(`
 `)
     );
-  } catch (o) {
-    if (o.code !== "ENOENT") {
-      p(
-        `Failed to read snapshot at ${n}: ${String(o)}`
+  } catch (r) {
+    if (r.code !== "ENOENT") {
+      u(
+        `Failed to read snapshot at ${n}: ${String(r)}`
       );
       return;
     }
     try {
-      rt(N(n), { recursive: !0 }), R(n, t, "utf8");
-    } catch (s) {
-      p(
-        `Failed to write snapshot at ${n}: ${String(s)}`
+      Q(T(n), { recursive: !0 }), S(n, t, "utf8");
+    } catch (o) {
+      u(
+        `Failed to write snapshot at ${n}: ${String(o)}`
       );
     }
   }
 }
-function zt(t) {
+function bt(t) {
   return {
     toContain: (e) => {
-      Ht(t, e);
+      lt(t, e);
     },
     toMatchSnapshot: () => {
-      Yt(t);
+      pt(t);
     },
     promptCheck: async (e) => {
-      await Wt(t, e);
+      await st(t, e);
     },
     toBeClassifiedAs: async (e, n) => {
-      await Ot(t, e, n);
+      await ut(t, e, n);
     }
   };
 }
-function Qt(t, e) {
-  V(() => {
-    Ct(), ft(t);
-    const n = U(), o = Date.now(), i = () => U() === n, s = () => Date.now() - o;
-    try {
-      const r = e();
-      if (r && typeof r.then == "function") {
-        b(
-          r.then(() => {
-            I(!0, s());
-          }).catch((a) => {
-            throw I(!1, s()), a;
-          }).finally(() => {
-            v();
-          })
-        );
-        return;
-      }
-    } catch (r) {
-      throw I(!1, s()), v(), r;
-    }
-    I(i(), s()), v();
-  }, P());
-}
-const Kt = /\.eval\.(js|ts)$/, _t = /* @__PURE__ */ new Set([".git", "node_modules"]);
-async function _(t) {
-  const e = await nt(t, { withFileTypes: !0 }), n = [];
-  return await Promise.all(
-    e.map(async (o) => {
-      const i = F(t, o.name);
-      if (o.isDirectory()) {
-        if (_t.has(o.name))
-          return;
-        n.push(...await _(i));
-        return;
-      }
-      o.isFile() && Kt.test(o.name) && n.push(i);
-    })
-  ), n;
-}
-const j = new URL("data:application/json;base64,ewogICJuYW1lIjogImthdHQiLAogICJ2ZXJzaW9uIjogIjAuMC4yIiwKICAiZGVzY3JpcHRpb24iOiAiQ0xJIHRvb2wgdGhhdCB0ZXN0cyB0aGUgb3V0cHV0IG9mIGFnZW50aWMgQUkgdG9vbHMiLAogICJrZXl3b3JkcyI6IFsKICAgICJjbGkiLAogICAgImFpIiwKICAgICJhZ2VudGljLWFpIiwKICAgICJ0ZXN0aW5nIiwKICAgICJldmFsdWF0aW9uIgogIF0sCiAgImF1dGhvciI6ICJSYXBoYWVsIFBvcnRvIChodHRwczovL2dpdGh1Yi5jb20vcmFwaGFlbHBvcikiLAogICJsaWNlbnNlIjogIk1JVCIsCiAgInR5cGUiOiAibW9kdWxlIiwKICAibWFpbiI6ICJkaXN0L2luZGV4LmpzIiwKICAiYmluIjogewogICAgImthdHQiOiAiZGlzdC9pbmRleC5qcyIKICB9LAogICJzY3JpcHRzIjogewogICAgImJ1aWxkIjogInZpdGUgYnVpbGQiLAogICAgImRldiI6ICJ0c3ggc3JjL2luZGV4LnRzIiwKICAgICJsaW50IjogImJpb21lIGxpbnQgLi9zcmMiLAogICAgImZvcm1hdCI6ICJiaW9tZSBmb3JtYXQgLS13cml0ZSAuL3NyYyIsCiAgICAidGVzdCI6ICJ2aXRlc3QiLAogICAgInR5cGVjaGVjayI6ICJ0c2MgLXAgdHNjb25maWcuanNvbiAtLW5vRW1pdCIsCiAgICAidGVzdDpidWlsZCI6ICJub2RlIC4vZGlzdC9pbmRleC5qcyIKICB9LAogICJ0eXBlcyI6ICJkaXN0L2luZGV4LmQudHMiLAogICJkZXZEZXBlbmRlbmNpZXMiOiB7CiAgICAiQGJpb21lanMvYmlvbWUiOiAiMS45LjQiLAogICAgIkB0eXBlcy9ub2RlIjogIjI1LjIuMCIsCiAgICAidHN4IjogIjQuMjEuMCIsCiAgICAidHlwZXNjcmlwdCI6ICI1LjguMiIsCiAgICAidml0ZSI6ICI3LjMuMSIsCiAgICAidml0ZS1wbHVnaW4tZHRzIjogIjQuNS40IiwKICAgICJ2aXRlc3QiOiAiMy4yLjQiLAogICAgInZzY29kZS1qc29ucnBjIjogIl44LjIuMSIKICB9LAogICJkZXBlbmRlbmNpZXMiOiB7CiAgICAiQGdpdGh1Yi9jb3BpbG90LXNkayI6ICJeMC4xLjIxIgogIH0sCiAgImJ1Z3MiOiB7CiAgICAidXJsIjogImh0dHBzOi8vZ2l0aHViLmNvbS9yYXBoYWVscG9yL2thdHQvaXNzdWVzIgogIH0sCiAgImhvbWVwYWdlIjogImh0dHBzOi8vZ2l0aHViLmNvbS9yYXBoYWVscG9yL2thdHQiCn0K", import.meta.url);
-let m;
-function qt() {
-  if (m !== void 0)
-    return m;
-  try {
-    const t = j.protocol === "data:" ? te(j) : H(ct(j), "utf8"), e = JSON.parse(t);
-    m = typeof e.version == "string" ? e.version : "unknown";
-  } catch {
-    m = "unknown";
-  }
-  return m;
-}
-function te(t) {
-  const e = t.pathname.indexOf(",");
-  if (e < 0)
-    throw new Error("Invalid data URL.");
-  const n = t.pathname.slice(0, e), o = t.pathname.slice(e + 1);
-  return n.includes(";base64") ? Buffer.from(o, "base64").toString("utf8") : decodeURIComponent(o);
-}
-function ee() {
-  const t = " ██╗  ██╗ █████╗ ████████╗████████╗", e = " ██║ ██╔╝██╔══██╗╚══██╔══╝╚══██╔══╝", n = " █████╔╝ ███████║   ██║      ██║", o = " ██╔═██╗ ██╔══██║   ██║      ██║", i = " ██║  ██╗██║  ██║   ██║      ██║", s = " ╚═╝  ╚═╝╚═╝  ╚═╝   ╚═╝      ╚═╝", r = `v${qt()}`, a = Math.max(
-    0,
-    Math.floor((t.length - r.length) / 2)
-  ), l = `${" ".repeat(a)}${r}`;
-  console.log(`
-${C(t)}
-${C(e)}
-${C(n)}
-${W(o)}
-${W(i)}
-${Ft(s)}
-${C(l)}
-`);
-}
-function ne(t) {
-  const e = String(t.getHours()).padStart(2, "0"), n = String(t.getMinutes()).padStart(2, "0"), o = String(t.getSeconds()).padStart(2, "0");
-  return `${e}:${n}:${o}`;
-}
-async function oe() {
-  const t = process.argv.slice(2), e = t.includes("--update-snapshots") || t.includes("-u");
-  Dt(e), ee();
-  const n = /* @__PURE__ */ new Date();
-  Nt(), kt(), At();
-  const o = await _(process.cwd());
-  if (o.length === 0)
-    return console.log("No .eval.js or .eval.ts files found."), 1;
-  const s = (await Promise.allSettled(
-    o.map(
-      (c) => B.run(
-        { evalFile: c },
-        () => import(at(c).href)
-      )
-    )
-  )).map((c, h) => ({ result: c, file: o[h] })).filter(({ result: c }) => c.status === "rejected");
-  if (s.length > 0) {
-    for (const c of s) {
-      const h = c.result.status === "rejected" ? c.result.reason : void 0;
-      console.error(`Error executing ${c.file}: ${String(h)}`);
-    }
-    return 1;
-  }
-  const a = (await wt()).filter(
-    (c) => c.status === "rejected"
-  );
-  if (a.length > 0) {
-    for (const c of a)
-      c.status === "rejected" && console.error(`Error executing async test: ${String(c.reason)}`);
-    return 1;
-  }
-  const l = bt();
-  if (l.length > 0) {
-    console.error("❌ Failed tests:");
-    for (const [c, h] of l.entries()) {
-      const J = [h.describePath, h.itPath].filter((tt) => tt.length > 0).join(" > "), q = J.length > 0 ? `${J}: ` : "";
-      console.error(`${c + 1}. ${q}${h.message}`);
-    }
-    return 1;
-  }
-  const d = St(), g = Date.now() - n.getTime();
-  return console.log(
-    [
-      "---",
-      `${f("Files")}  ${o.length} passed`,
-      `${f("Evals")}  ${d} passed`,
-      `${f("Start at")}  ${ne(n)}`,
-      `${f("Duration")}  ${g}ms`
-    ].join(`
-`)
-  ), 0;
-}
-Object.assign(globalThis, { describe: yt, it: Qt, expect: zt, prompt: w, promptFile: Ut });
-oe().then((t) => {
-  process.exit(t);
-}).catch((t) => {
-  console.error(`Unexpected error: ${String(t)}`), process.exit(1);
-});
 export {
-  oe as runCli
+  Tt as describe,
+  bt as expect,
+  vt as it,
+  $ as prompt,
+  $t as promptFile,
+  St as runCli
 };

package/dist/katt.js

@@ -0,0 +1,7 @@
+#!/usr/bin/env node
+import { o as r } from "./runCli-CDRmZ5hw.js";
+r().then((e) => {
+  process.exit(e);
+}).catch((e) => {
+  console.error(`Unexpected error: ${String(e)}`), process.exit(1);
+});

package/dist/runCli-CDRmZ5hw.js

@@ -0,0 +1,288 @@
+import { fileURLToPath as x, pathToFileURL as y } from "node:url";
+import { readdir as X } from "node:fs/promises";
+import { resolve as H } from "node:path";
+import { AsyncLocalStorage as j } from "node:async_hooks";
+import { readFileSync as M } from "node:fs";
+const R = /\.eval\.(js|ts)$/, W = /* @__PURE__ */ new Set([".git", "node_modules"]);
+async function v(t) {
+  const e = await X(t, { withFileTypes: !0 }), n = [];
+  return await Promise.all(
+    e.map(async (o) => {
+      const a = H(t, o.name);
+      if (o.isDirectory()) {
+        if (W.has(o.name))
+          return;
+        n.push(...await v(a));
+        return;
+      }
+      o.isFile() && R.test(o.name) && n.push(a);
+    })
+  ), n;
+}
+const J = new j(), G = {
+  describeStack: [],
+  itStack: [],
+  tokenUsageStack: [],
+  modelStack: []
+};
+let L = 0, $ = 0;
+const p = [], m = [];
+let k = 0;
+function s() {
+  return J.getStore() ?? G;
+}
+function B(t) {
+  return {
+    describeStack: [...t.describeStack],
+    itStack: [...t.itStack],
+    tokenUsageStack: [...t.tokenUsageStack],
+    modelStack: [...t.modelStack]
+  };
+}
+function U() {
+  return L += 1, `d${L}`;
+}
+function O() {
+  return $ += 1, `i${$}`;
+}
+function mt(t, e) {
+  const n = e ?? B(s());
+  return J.run(n, t);
+}
+function ht() {
+  return B(s());
+}
+function St(t) {
+  s().describeStack.push({ id: U(), description: t });
+}
+function At() {
+  s().describeStack.pop();
+}
+function Y() {
+  return s().describeStack.map((t) => t.description).join(" > ");
+}
+function kt(t) {
+  s().itStack.push({ id: O(), description: t }), s().tokenUsageStack.push(0), s().modelStack.push(void 0);
+}
+function bt() {
+  s().itStack.pop(), s().tokenUsageStack.pop(), s().modelStack.pop();
+}
+function Q() {
+  return s().itStack.map((t) => t.description).join(" > ");
+}
+function Lt(t) {
+  if (!Number.isFinite(t) || t <= 0)
+    return;
+  const e = s(), n = e.tokenUsageStack.length - 1;
+  n < 0 || (e.tokenUsageStack[n] += t);
+}
+function V() {
+  const t = s(), e = t.tokenUsageStack.length - 1;
+  return e < 0 ? 0 : t.tokenUsageStack[e] ?? 0;
+}
+function $t(t) {
+  if (t.length === 0)
+    return;
+  const e = s(), n = e.modelStack.length - 1;
+  n < 0 || (e.modelStack[n] = t);
+}
+function K() {
+  const t = s(), e = t.modelStack.length - 1;
+  if (!(e < 0))
+    return t.modelStack[e];
+}
+function wt(t) {
+  p.push(t);
+}
+function jt() {
+  k += 1;
+}
+function P() {
+  return k;
+}
+function z() {
+  k = 0;
+}
+function vt(t) {
+  m.push(t);
+}
+function E() {
+  return [...m];
+}
+function Jt() {
+  return m.length;
+}
+function D() {
+  m.length = 0;
+}
+async function _() {
+  const t = [];
+  for (; p.length > 0; ) {
+    const e = p.splice(0, p.length), n = await Promise.allSettled(e);
+    t.push(...n);
+  }
+  return t;
+}
+const q = "\x1B[1;36m", tt = "\x1B[33m", et = "\x1B[38;5;208m", nt = "\x1B[1;38;5;208m", h = "\x1B[0m";
+function c(t) {
+  return `${q}${t}${h}`;
+}
+function f(t) {
+  return `${tt}${t}${h}`;
+}
+function w(t) {
+  return `${et}${t}${h}`;
+}
+function ot(t) {
+  return `${nt}${t}${h}`;
+}
+let A = "";
+function it() {
+  A = "";
+}
+function st({
+  suitePath: t,
+  casePath: e,
+  didPass: n,
+  durationMs: o,
+  model: a,
+  tokenUsage: r
+}) {
+  const l = t.length > 0 ? t : "(root)", g = e.length > 0 ? e : "(root)";
+  A !== l && (console.log(`Suite "${c(l)}"`), A = l);
+  const I = n ? "✅ Passed in" : "❌ Failed in", d = [
+    `Test "${c(g)}"`,
+    `- ${I} ${c(`${o}ms`)}`
+  ];
+  a && d.push(`- Model ${c(a)}`), (r ?? 0) > 0 && d.push(`- Tokens used ${c(String(r))}`), d.push("---"), console.log(d.join(`
+`));
+}
+function Bt(t, e, n = "(root)") {
+  const o = Q();
+  st({
+    suitePath: Y(),
+    casePath: o.length > 0 ? o : n,
+    didPass: t,
+    durationMs: e,
+    model: K(),
+    tokenUsage: V()
+  });
+}
+const at = new j(), S = new URL("data:application/json;base64,ewogICJuYW1lIjogImthdHQiLAogICJ2ZXJzaW9uIjogIjAuMC40IiwKICAiZGVzY3JpcHRpb24iOiAiQ0xJIHRvb2wgdGhhdCB0ZXN0cyB0aGUgb3V0cHV0IG9mIGFnZW50aWMgQUkgdG9vbHMiLAogICJrZXl3b3JkcyI6IFsKICAgICJjbGkiLAogICAgImFpIiwKICAgICJhZ2VudGljLWFpIiwKICAgICJ0ZXN0aW5nIiwKICAgICJldmFsdWF0aW9uIgogIF0sCiAgImF1dGhvciI6ICJSYXBoYWVsIFBvcnRvIChodHRwczovL2dpdGh1Yi5jb20vcmFwaGFlbHBvcikiLAogICJsaWNlbnNlIjogIk1JVCIsCiAgInR5cGUiOiAibW9kdWxlIiwKICAibWFpbiI6ICJkaXN0L2luZGV4LmpzIiwKICAiZXhwb3J0cyI6IHsKICAgICIuIjogewogICAgICAidHlwZXMiOiAiLi9kaXN0L2luZGV4LmQudHMiLAogICAgICAiaW1wb3J0IjogIi4vZGlzdC9pbmRleC5qcyIKICAgIH0KICB9LAogICJiaW4iOiB7CiAgICAia2F0dCI6ICJkaXN0L2thdHQuanMiCiAgfSwKICAic2NyaXB0cyI6IHsKICAgICJidWlsZCI6ICJ2aXRlIGJ1aWxkIiwKICAgICJkZXYiOiAidHN4IHNyYy9pbmRleC50cyIsCiAgICAibGludCI6ICJiaW9tZSBsaW50IC4vc3JjIiwKICAgICJmb3JtYXQiOiAiYmlvbWUgZm9ybWF0IC0td3JpdGUgLi9zcmMiLAogICAgInRlc3QiOiAidml0ZXN0IiwKICAgICJ0eXBlY2hlY2siOiAidHNjIC1wIHRzY29uZmlnLmpzb24gLS1ub0VtaXQiLAogICAgInRlc3Q6YnVpbGQiOiAibm9kZSAuL2Rpc3Qva2F0dC5qcyIKICB9LAogICJ0eXBlcyI6ICJkaXN0L2luZGV4LmQudHMiLAogICJkZXZEZXBlbmRlbmNpZXMiOiB7CiAgICAiQGJpb21lanMvYmlvbWUiOiAiMS45LjQiLAogICAgIkB0eXBlcy9ub2RlIjogIjI1LjIuMCIsCiAgICAidHN4IjogIjQuMjEuMCIsCiAgICAidHlwZXNjcmlwdCI6ICI1LjguMiIsCiAgICAidml0ZSI6ICI3LjMuMSIsCiAgICAidml0ZS1wbHVnaW4tZHRzIjogIjQuNS40IiwKICAgICJ2aXRlc3QiOiAiMy4yLjQiLAogICAgInZzY29kZS1qc29ucnBjIjogIl44LjIuMSIKICB9LAogICJkZXBlbmRlbmNpZXMiOiB7CiAgICAiQGdpdGh1Yi9jb3BpbG90LXNkayI6ICJeMC4xLjIxIgogIH0sCiAgImJ1Z3MiOiB7CiAgICAidXJsIjogImh0dHBzOi8vZ2l0aHViLmNvbS9yYXBoYWVscG9yL2thdHQvaXNzdWVzIgogIH0sCiAgImhvbWVwYWdlIjogImh0dHBzOi8vZ2l0aHViLmNvbS9yYXBoYWVscG9yL2thdHQiCn0K", import.meta.url);
+let C;
+function ct() {
+  if (C !== void 0)
+    return C;
+  try {
+    const t = S.protocol === "data:" ? rt(S) : M(x(S), "utf8"), e = JSON.parse(t);
+    C = typeof e.version == "string" ? e.version : "unknown";
+  } catch {
+    C = "unknown";
+  }
+  return C;
+}
+function rt(t) {
+  const e = t.pathname.indexOf(",");
+  if (e < 0)
+    throw new Error("Invalid data URL.");
+  const n = t.pathname.slice(0, e), o = t.pathname.slice(e + 1);
+  return n.includes(";base64") ? Buffer.from(o, "base64").toString("utf8") : decodeURIComponent(o);
+}
+function lt() {
+  const t = " ██╗  ██╗ █████╗ ████████╗████████╗", e = " ██║ ██╔╝██╔══██╗╚══██╔══╝╚══██╔══╝", n = " █████╔╝ ███████║   ██║      ██║", o = " ██╔═██╗ ██╔══██║   ██║      ██║", a = " ██║  ██╗██║  ██║   ██║      ██║", r = " ╚═╝  ╚═╝╚═╝  ╚═╝   ╚═╝      ╚═╝", l = `v${ct()}`, g = Math.max(
+    0,
+    Math.floor((t.length - l.length) / 2)
+  ), I = `${" ".repeat(g)}${l}`;
+  console.log(`
+${f(t)}
+${f(e)}
+${f(n)}
+${w(o)}
+${w(a)}
+${ot(r)}
+${f(I)}
+`);
+}
+let Z = !1;
+function ut(t) {
+  Z = t;
+}
+function Zt() {
+  return Z;
+}
+function gt(t) {
+  const e = String(t.getHours()).padStart(2, "0"), n = String(t.getMinutes()).padStart(2, "0"), o = String(t.getSeconds()).padStart(2, "0");
+  return `${e}:${n}:${o}`;
+}
+async function Ft() {
+  const t = process.argv.slice(2), e = t.includes("--update-snapshots") || t.includes("-u");
+  ut(e), lt();
+  const n = /* @__PURE__ */ new Date();
+  it(), D(), z();
+  const o = await v(process.cwd());
+  if (o.length === 0)
+    return console.log("No .eval.js or .eval.ts files found."), 1;
+  const r = (await Promise.allSettled(
+    o.map(
+      (i) => at.run(
+        { evalFile: i },
+        () => import(y(i).href)
+      )
+    )
+  )).map((i, u) => ({ result: i, file: o[u] })).filter(({ result: i }) => i.status === "rejected");
+  if (r.length > 0) {
+    for (const i of r) {
+      const u = i.result.status === "rejected" ? i.result.reason : void 0;
+      console.error(`Error executing ${i.file}: ${String(u)}`);
+    }
+    return 1;
+  }
+  const g = (await _()).filter(
+    (i) => i.status === "rejected"
+  );
+  if (g.length > 0) {
+    for (const i of g)
+      i.status === "rejected" && console.error(`Error executing async test: ${String(i.reason)}`);
+    return 1;
+  }
+  const I = E();
+  if (I.length > 0) {
+    console.error("❌ Failed tests:");
+    for (const [i, u] of I.entries()) {
+      const b = [u.describePath, u.itPath].filter((T) => T.length > 0).join(" > "), N = b.length > 0 ? `${b}: ` : "";
+      console.error(`${i + 1}. ${N}${u.message}`);
+    }
+    return 1;
+  }
+  const d = P(), F = Date.now() - n.getTime();
+  return console.log(
+    [
+      "---",
+      `${c("Files")}  ${o.length} passed`,
+      `${c("Evals")}  ${d} passed`,
+      `${c("Start at")}  ${gt(n)}`,
+      `${c("Duration")}  ${F}ms`
+    ].join(`
+`)
+  ), 0;
+}
+export {
+  jt as a,
+  wt as b,
+  ht as c,
+  bt as d,
+  St as e,
+  At as f,
+  Jt as g,
+  Lt as h,
+  at as i,
+  vt as j,
+  Q as k,
+  Bt as l,
+  Y as m,
+  Zt as n,
+  Ft as o,
+  kt as p,
+  mt as r,
+  $t as s
+};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "katt",
-  "version": "0.0.3",
+  "version": "0.0.5",
   "description": "CLI tool that tests the output of agentic AI tools",
   "keywords": [
     "cli",
@@ -13,8 +13,14 @@
   "license": "MIT",
   "type": "module",
   "main": "dist/index.js",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    }
+  },
   "bin": {
-    "katt": "dist/index.js"
+    "katt": "dist/katt.js"
   },
   "scripts": {
     "build": "vite build",
@@ -23,7 +29,7 @@
     "format": "biome format --write ./src",
     "test": "vitest",
     "typecheck": "tsc -p tsconfig.json --noEmit",
-    "test:build": "node ./dist/index.js"
+    "test:build": "node ./dist/katt.js"
   },
   "types": "dist/index.d.ts",
   "devDependencies": {