katt 0.0.2

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 raphaelpor
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,174 @@
+# Katt
+
+Katt is a lightweight testing framework for running AI Evals, inspired by [Jest](https://github.com/jestjs/jest).
+
+<img src="docs/logo.png" alt="Katt logo" width="250" />
+
+## Overview
+
+Katt is designed to evaluate and validate the behavior of AI agents like **Claude Code**, **GitHub Copilot**, **OpenAI Codex** and more. It provides a simple, intuitive API for writing tests that interact with AI models and assert their responses.
+
+## API Documentation
+
+For a complete list of features and usage examples, see [docs/api-documentation.md](docs/api-documentation.md).
+
+## Hello World - Example
+
+```typescript
+const result = await prompt("If you read this just say 'hello world'");
+expect(result).toContain("hello world");
+```
+
+It also supports the familiar `describe` and `it` syntax for organizing tests:
+
+```typescript
+describe("Greeting agent", () => {
+  it("should say hello world", async () => {
+    const result = await prompt("If you read this just say 'hello world'");
+    expect(result).toContain("hello world");
+  });
+});
+```
+
+## Main Features
+
+- **Simple Testing API**: Familiar `describe` and `it` syntax for organizing tests
+- **AI Interaction and Verification**: Built-in `prompt()`, `promptFile()` and `promptCheck()` functions for running and analyzing prompts to AI agents
+- **Classification Matcher**: Built-in `toBeClassifiedAs()` matcher to grade a response against a target label on a 1-5 scale
+- **Concurrent Execution**: Runs eval files concurrently for faster test execution
+- **Model Selection**: Support for specifying custom AI models
+
+## Usage
+
+### Basic Usage
+
+1. Create a file with the `.eval.ts` or `.eval.js` extension and write your tests.
+```typescript
+const result = await prompt("If you read this just say 'hello world'");
+expect(result).toContain("hello world");
+```
+
+2. Run Katt from your project directory:
+
+```bash
+npx katt
+```
+
+### Using promptFile
+
+Load prompts from external files:
+
+```javascript
+// test.eval.js
+describe("Working with files", () => {
+  it("should load the file and respond", async () => {
+    const result = await promptFile("./myPrompt.md");
+    expect(result).toContain("expected response");
+  });
+});
+```
+
+### Specifying AI Models
+
+You can specify a custom model for your prompts:
+
+```javascript
+describe("Model selection", () => {
+  it("should use a specific model", async () => {
+    const promptString = "You are a helpful agent. Say hi and ask what you could help the user with.";
+    const result = await prompt(promptString, { model: "gpt-5.2" });
+
+    expect(result).promptCheck("It should be friendly and helpful");
+  });
+});
+```
+
+You can also set a default model for the project by adding a `katt.json` file in the project root:
+
+```json
+{
+  "copilot": {
+    "model": "gpt-5-mini"
+  }
+}
+```
+
+When this file exists:
+
+- `prompt("...")` and `promptFile("...")` use `copilot.model` by default
+- `prompt("...", { model: "..." })` still overrides the config value
+
+## Development
+
+### Setup
+
+```bash
+npm install
+```
+
+### Available Scripts
+
+- `npm run dev` - Run the CLI in development mode
+- `npm run build` - Build the project
+- `npm run test` - Run tests
+- `npm run typecheck` - Run TypeScript type checking
+- `npm run format` - Format code using Biome
+- `npm run lint` - Lint code using Biome
+- `npm run test:build` - Test the built CLI
+
+### Verification Process
+
+After making changes, run the following sequence:
+
+1. `npm run format`
+2. `npm run typecheck`
+3. `npm run test`
+4. `npm run build`
+5. `npm run test:build`
+
+## Project Structure
+
+```
+katt/
+├── src/              # Source code
+│   ├── cli/          # CLI implementation
+│   ├── lib/          # Core libraries (describe, it, expect, prompt)
+│   └── types/        # TypeScript type definitions
+├── examples/         # Example eval files
+├── specs/            # Markdown specifications
+├── package.json      # Package configuration
+└── tsconfig.json     # TypeScript configuration
+```
+
+## How It Works
+
+1. Katt searches the current directory recursively for `*.eval.js` and `*.eval.ts` files
+2. It skips `.git` and `node_modules` directories
+3. Found eval files are imported and executed concurrently
+4. Tests registered with `describe()` and `it()` are collected and run
+5. Each test duration is printed after execution
+6. A summary is displayed showing passed/failed tests and total duration
+7. Katt exits with code `0` on success or `1` on failure
+
+## Requirements
+
+- Node.js
+- GitHub Copilot CLI installed (see [GitHub Copilot CLI installation docs](https://docs.github.com/en/copilot/how-tos/copilot-cli/install-copilot-cli))
+- Access to AI models (e.g., OpenAI API key for Codex)
+
+## License
+
+MIT
+
+## Contributing
+
+We welcome contributions from the community! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) guide for detailed information on how to contribute to Katt.
+
+Quick start:
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Run the verification process
+5. Submit a pull request
+
+For detailed guidelines, development setup, coding standards, and more, check out our [contribution guide](CONTRIBUTING.md).

package/dist/index.js

@@ -0,0 +1,566 @@
+#!/usr/bin/env node
+import { AsyncLocalStorage as E } from "node:async_hooks";
+import { CopilotClient as et } from "@github/copilot-sdk";
+import { readFile as M, readdir as nt } from "node:fs/promises";
+import { resolve as x, dirname as T, isAbsolute as ot, basename as it, join as st } from "node:path";
+import { readFileSync as Y, writeFileSync as Z, mkdirSync as rt } from "node:fs";
+import { fileURLToPath as ct, pathToFileURL as at } from "node:url";
+const D = new E(), lt = {
+  describeStack: [],
+  itStack: [],
+  tokenUsageStack: [],
+  modelStack: []
+};
+let U = 0, W = 0;
+const w = [], A = [];
+let J = 0;
+function u() {
+  return D.getStore() ?? lt;
+}
+function H(t) {
+  return {
+    describeStack: [...t.describeStack],
+    itStack: [...t.itStack],
+    tokenUsageStack: [...t.tokenUsageStack],
+    modelStack: [...t.modelStack]
+  };
+}
+function ut() {
+  return U += 1, `d${U}`;
+}
+function dt() {
+  return W += 1, `i${W}`;
+}
+function X(t, e) {
+  const n = e ?? H(u());
+  return D.run(n, t);
+}
+function z() {
+  return H(u());
+}
+function gt(t) {
+  u().describeStack.push({ id: ut(), description: t });
+}
+function b() {
+  u().describeStack.pop();
+}
+function P() {
+  return u().describeStack.map((t) => t.description).join(" > ");
+}
+function ft(t) {
+  u().itStack.push({ id: dt(), description: t }), u().tokenUsageStack.push(0), u().modelStack.push(void 0);
+}
+function v() {
+  u().itStack.pop(), u().tokenUsageStack.pop(), u().modelStack.pop();
+}
+function V() {
+  return u().itStack.map((t) => t.description).join(" > ");
+}
+function pt(t) {
+  if (!Number.isFinite(t) || t <= 0)
+    return;
+  const e = u(), n = e.tokenUsageStack.length - 1;
+  n < 0 || (e.tokenUsageStack[n] += t);
+}
+function ht() {
+  const t = u(), e = t.tokenUsageStack.length - 1;
+  return e < 0 ? 0 : t.tokenUsageStack[e] ?? 0;
+}
+function It(t) {
+  if (t.length === 0)
+    return;
+  const e = u(), n = e.modelStack.length - 1;
+  n < 0 || (e.modelStack[n] = t);
+}
+function Ct() {
+  const t = u(), e = t.modelStack.length - 1;
+  if (!(e < 0))
+    return t.modelStack[e];
+}
+function $(t) {
+  w.push(t);
+}
+function mt() {
+  J += 1;
+}
+function St() {
+  return J;
+}
+function wt() {
+  J = 0;
+}
+function At(t) {
+  A.push(t);
+}
+function $t() {
+  return [...A];
+}
+function B() {
+  return A.length;
+}
+function kt() {
+  A.length = 0;
+}
+async function yt() {
+  const t = [];
+  for (; w.length > 0; ) {
+    const e = w.splice(0, w.length), n = await Promise.allSettled(e);
+    t.push(...n);
+  }
+  return t;
+}
+function bt(t, e) {
+  X(() => {
+    gt(t);
+    try {
+      const n = e();
+      if (n && typeof n.then == "function") {
+        $(
+          n.finally(() => {
+            b();
+          })
+        );
+        return;
+      }
+    } catch (n) {
+      throw b(), n;
+    }
+    b();
+  }, z());
+}
+const vt = "\x1B[1;36m", Lt = "\x1B[33m", jt = "\x1B[38;5;208m", Ft = "\x1B[1;38;5;208m", k = "\x1B[0m";
+function f(t) {
+  return `${vt}${t}${k}`;
+}
+function m(t) {
+  return `${Lt}${t}${k}`;
+}
+function G(t) {
+  return `${jt}${t}${k}`;
+}
+function xt(t) {
+  return `${Ft}${t}${k}`;
+}
+let F = "";
+function Tt() {
+  F = "";
+}
+function Jt({
+  suitePath: t,
+  casePath: e,
+  didPass: n,
+  durationMs: o,
+  model: s,
+  tokenUsage: i
+}) {
+  const r = t.length > 0 ? t : "(root)", a = e.length > 0 ? e : "(root)";
+  F !== r && (console.log(`Suite "${f(r)}"`), F = r);
+  const l = n ? "✅ Passed in" : "❌ Failed in", d = [
+    `Test "${f(a)}"`,
+    `- ${l} ${f(`${o}ms`)}`
+  ];
+  s && d.push(`- Model ${f(s)}`), (i ?? 0) > 0 && d.push(`- Tokens used ${f(String(i))}`), d.push("---"), console.log(d.join(`
+`));
+}
+function I(t, e, n = "(root)") {
+  const o = V();
+  Jt({
+    suitePath: P(),
+    casePath: o.length > 0 ? o : n,
+    didPass: t,
+    durationMs: e,
+    model: Ct(),
+    tokenUsage: ht()
+  });
+}
+const N = new E();
+function Nt(t, e) {
+  return typeof t == "object" && t !== null && "code" in t && t.code === e;
+}
+function Rt(t) {
+  try {
+    const e = JSON.parse(t);
+    return typeof e == "object" && e !== null ? e : void 0;
+  } catch (e) {
+    console.warn(`Failed to parse katt.json: ${String(e)}`);
+    return;
+  }
+}
+function Zt(t) {
+  const e = t?.copilot;
+  if (typeof e != "object" || e === null || Array.isArray(e))
+    return;
+  const n = {
+    ...e
+  }, o = n.model;
+  return (typeof o != "string" || o.length === 0) && delete n.model, Object.keys(n).length > 0 ? n : void 0;
+}
+async function Ut() {
+  const t = x(process.cwd(), "katt.json");
+  try {
+    const e = await M(t, "utf8");
+    return Zt(Rt(e));
+  } catch (e) {
+    if (Nt(e, "ENOENT"))
+      return;
+    console.warn(`Failed to read katt.json: ${String(e)}`);
+    return;
+  }
+}
+function K(t) {
+  return typeof t == "string" && t.length > 0 ? t : void 0;
+}
+function L(t) {
+  if (!t)
+    return;
+  const e = { ...t };
+  if (e.model !== void 0) {
+    const n = K(e.model);
+    n ? e.model = n : delete e.model;
+  }
+  return Object.keys(e).length > 0 ? e : void 0;
+}
+function S(t) {
+  return !Number.isFinite(t) || (t ?? 0) <= 0 ? 0 : Math.floor(t ?? 0);
+}
+function Wt(t) {
+  return S(t.inputTokens) + S(t.outputTokens) + S(t.cacheReadTokens) + S(t.cacheWriteTokens);
+}
+async function y(t, e = {}) {
+  const n = L(await Ut()), o = L(e), s = L({
+    ...n ?? {},
+    ...o ?? {}
+  }), i = K(s?.model), r = new et({ useLoggedInUser: !0 });
+  let a, l, d = 0;
+  try {
+    await r.start(), a = await r.createSession(s), l = a.on("assistant.usage", (c) => {
+      d += Wt(c.data);
+    });
+    const g = await a.sendAndWait({ prompt: t });
+    if (!g?.data?.content)
+      throw new Error("Copilot did not return a response.");
+    return i && It(i), g.data.content;
+  } finally {
+    const g = [];
+    if (l?.(), d > 0 && pt(d), a)
+      try {
+        await a.destroy();
+      } catch (c) {
+        g.push(c);
+      }
+    try {
+      const c = await r.stop();
+      g.push(...c);
+    } catch (c) {
+      g.push(c);
+    }
+    g.length > 0 && console.error(
+      `Copilot cleanup encountered ${g.length} error(s).`
+    );
+  }
+}
+async function Bt(t, e = {}) {
+  const n = N.getStore(), o = n?.evalFile ? T(n.evalFile) : process.cwd(), s = ot(t) ? t : x(o, t), i = await M(s, "utf8");
+  return y(i, e);
+}
+function p(t) {
+  At({
+    describePath: P(),
+    itPath: V(),
+    message: t
+  });
+}
+async function Gt(t, e) {
+  const n = Date.now(), o = `expected '${t}' to satisfy '${e}'`, s = y(`Evaluate if the expectation is fulfiled in by the input.
+        Expectation: "${e}".
+        Input:
+        ---
+        ${t}
+        ---
+        Important: Answer with "Yes" or "No" only, without any additional text.
+        `).then((i) => {
+    i.includes("Yes") ? I(!0, Date.now() - n, "promptCheck") : i.includes("No") ? p(o) : p(`failed to evaluate expectation '${e}'`);
+  });
+  return $(s), s;
+}
+const Ot = 3;
+function Et(t) {
+  const e = t.match(/\b([1-5])\b/);
+  return !e || !e[1] ? null : Number(e[1]);
+}
+async function Mt(t, e, n) {
+  const o = Date.now(), s = n?.threshold ?? Ot, i = y(
+    `Classify the input by how "${e}" it is on a scale of 1 to 5.
+1 means "not ${e}" and 5 means "very ${e}".
+Return only a single number: 1, 2, 3, 4, or 5.
+
+Input:
+---
+${t}
+---`,
+    n?.model ? { model: n.model } : void 0
+  ).then((r) => {
+    const a = Et(r);
+    if (a === null) {
+      p(
+        `failed to classify as '${e}'. Evaluator returned '${r}'`
+      );
+      return;
+    }
+    const l = `expected response to be classified as '${e}' with score >= ${s}, got ${a}`;
+    if (a < s) {
+      p(l);
+      return;
+    }
+    I(
+      !0,
+      Date.now() - o,
+      "toBeClassifiedAs"
+    );
+  });
+  return $(i), i;
+}
+function Yt(t, e) {
+  const n = `expected '${t}' to include '${e}'`;
+  t.includes(e) || p(n);
+}
+let _ = !1;
+function Dt(t) {
+  _ = t;
+}
+function Ht() {
+  return _;
+}
+function Xt(t) {
+  const n = it(t).replace(/\.eval\.[^./\\]+$/, "");
+  return st(
+    T(t),
+    "__snapshots__",
+    `${n}.snap.md`
+  );
+}
+function O(t) {
+  return t.split(/\r?\n/);
+}
+function zt(t, e) {
+  if (t === e)
+    return "  (no diff)";
+  const n = O(t), o = O(e), s = Math.max(n.length, o.length), i = [];
+  for (let r = 0; r < s; r += 1) {
+    const a = n[r], l = o[r];
+    if (a !== l) {
+      if (a === void 0 && l !== void 0) {
+        i.push(`+ ${l}`);
+        continue;
+      }
+      if (a !== void 0 && l === void 0) {
+        i.push(`- ${a}`);
+        continue;
+      }
+      i.push(`- ${a ?? ""}`), i.push(`+ ${l ?? ""}`);
+    }
+  }
+  return i.join(`
+`);
+}
+function Pt(t) {
+  const e = N.getStore()?.evalFile;
+  if (!e) {
+    p(
+      "toMatchSnapshot can only be used while running an eval file."
+    );
+    return;
+  }
+  const n = Xt(e);
+  try {
+    const o = Y(n, "utf8");
+    if (o === t)
+      return;
+    if (Ht()) {
+      Z(n, t, "utf8");
+      return;
+    }
+    const s = zt(o, t);
+    p(
+      [
+        `Snapshot mismatch at ${n}`,
+        "",
+        "Diff:",
+        s,
+        "",
+        "Run katt with --update-snapshots (or -u) to accept this change."
+      ].join(`
+`)
+    );
+  } catch (o) {
+    if (o.code !== "ENOENT") {
+      p(
+        `Failed to read snapshot at ${n}: ${String(o)}`
+      );
+      return;
+    }
+    try {
+      rt(T(n), { recursive: !0 }), Z(n, t, "utf8");
+    } catch (i) {
+      p(
+        `Failed to write snapshot at ${n}: ${String(i)}`
+      );
+    }
+  }
+}
+function Vt(t) {
+  return {
+    toContain: (e) => {
+      Yt(t, e);
+    },
+    toMatchSnapshot: () => {
+      Pt(t);
+    },
+    promptCheck: async (e) => {
+      await Gt(t, e);
+    },
+    toBeClassifiedAs: async (e, n) => {
+      await Mt(t, e, n);
+    }
+  };
+}
+function Kt(t, e) {
+  X(() => {
+    mt(), ft(t);
+    const n = B(), o = Date.now(), s = () => B() === n, i = () => Date.now() - o;
+    try {
+      const r = e();
+      if (r && typeof r.then == "function") {
+        $(
+          r.then(() => {
+            I(!0, i());
+          }).catch((a) => {
+            throw I(!1, i()), a;
+          }).finally(() => {
+            v();
+          })
+        );
+        return;
+      }
+    } catch (r) {
+      throw I(!1, i()), v(), r;
+    }
+    I(s(), i()), v();
+  }, z());
+}
+const _t = /\.eval\.(js|ts)$/, Qt = /* @__PURE__ */ new Set([".git", "node_modules"]);
+async function Q(t) {
+  const e = await nt(t, { withFileTypes: !0 }), n = [];
+  return await Promise.all(
+    e.map(async (o) => {
+      const s = x(t, o.name);
+      if (o.isDirectory()) {
+        if (Qt.has(o.name))
+          return;
+        n.push(...await Q(s));
+        return;
+      }
+      o.isFile() && _t.test(o.name) && n.push(s);
+    })
+  ), n;
+}
+const j = new URL("data:application/json;base64,ewogICJuYW1lIjogImthdHQiLAogICJ2ZXJzaW9uIjogIjAuMS4wIiwKICAiZGVzY3JpcHRpb24iOiAiQ0xJIHRvb2wgdGhhdCB0ZXN0cyB0aGUgb3V0cHV0IG9mIGFnZW50aWMgQUkgdG9vbHMiLAogICJrZXl3b3JkcyI6IFsKICAgICJjbGkiLAogICAgImFpIiwKICAgICJhZ2VudGljLWFpIiwKICAgICJ0ZXN0aW5nIiwKICAgICJldmFsdWF0aW9uIgogIF0sCiAgImF1dGhvciI6ICJSYXBoYWVsIFBvcnRvIChodHRwczovL2dpdGh1Yi5jb20vcmFwaGFlbHBvcikiLAogICJsaWNlbnNlIjogIk1JVCIsCiAgInByaXZhdGUiOiB0cnVlLAogICJ0eXBlIjogIm1vZHVsZSIsCiAgIm1haW4iOiAiZGlzdC9pbmRleC5qcyIsCiAgImJpbiI6IHsKICAgICJrYXR0IjogImRpc3QvaW5kZXguanMiCiAgfSwKICAic2NyaXB0cyI6IHsKICAgICJidWlsZCI6ICJ2aXRlIGJ1aWxkIiwKICAgICJkZXYiOiAidHN4IHNyYy9pbmRleC50cyIsCiAgICAibGludCI6ICJiaW9tZSBsaW50IC4vc3JjIiwKICAgICJmb3JtYXQiOiAiYmlvbWUgZm9ybWF0IC0td3JpdGUgLi9zcmMiLAogICAgInRlc3QiOiAidml0ZXN0IiwKICAgICJ0eXBlY2hlY2siOiAidHNjIC1wIHRzY29uZmlnLmpzb24gLS1ub0VtaXQiLAogICAgInRlc3Q6YnVpbGQiOiAibm9kZSAuL2Rpc3QvaW5kZXguanMiCiAgfSwKICAidHlwZXMiOiAiZGlzdC9pbmRleC5kLnRzIiwKICAiZGV2RGVwZW5kZW5jaWVzIjogewogICAgIkBiaW9tZWpzL2Jpb21lIjogIjEuOS40IiwKICAgICJAdHlwZXMvbm9kZSI6ICIyNS4yLjAiLAogICAgInRzeCI6ICI0LjIxLjAiLAogICAgInR5cGVzY3JpcHQiOiAiNS44LjIiLAogICAgInZpdGUiOiAiNy4zLjEiLAogICAgInZpdGUtcGx1Z2luLWR0cyI6ICI0LjUuNCIsCiAgICAidml0ZXN0IjogIjMuMi40IiwKICAgICJ2c2NvZGUtanNvbnJwYyI6ICJeOC4yLjEiCiAgfSwKICAiZGVwZW5kZW5jaWVzIjogewogICAgIkBnaXRodWIvY29waWxvdC1zZGsiOiAiXjAuMS4yMSIKICB9LAogICJidWdzIjogewogICAgInVybCI6ICJodHRwczovL2dpdGh1Yi5jb20vcmFwaGFlbHBvci9rYXR0L2lzc3VlcyIKICB9LAogICJob21lcGFnZSI6ICJodHRwczovL2dpdGh1Yi5jb20vcmFwaGFlbHBvci9rYXR0Igp9Cg==", import.meta.url);
+let C;
+function qt() {
+  if (C !== void 0)
+    return C;
+  try {
+    const t = j.protocol === "data:" ? te(j) : Y(ct(j), "utf8"), e = JSON.parse(t);
+    C = typeof e.version == "string" ? e.version : "unknown";
+  } catch {
+    C = "unknown";
+  }
+  return C;
+}
+function te(t) {
+  const e = t.pathname.indexOf(",");
+  if (e < 0)
+    throw new Error("Invalid data URL.");
+  const n = t.pathname.slice(0, e), o = t.pathname.slice(e + 1);
+  return n.includes(";base64") ? Buffer.from(o, "base64").toString("utf8") : decodeURIComponent(o);
+}
+function ee() {
+  const t = " ██╗  ██╗ █████╗ ████████╗████████╗", e = " ██║ ██╔╝██╔══██╗╚══██╔══╝╚══██╔══╝", n = " █████╔╝ ███████║   ██║      ██║", o = " ██╔═██╗ ██╔══██║   ██║      ██║", s = " ██║  ██╗██║  ██║   ██║      ██║", i = " ╚═╝  ╚═╝╚═╝  ╚═╝   ╚═╝      ╚═╝", r = `v${qt()}`, a = Math.max(
+    0,
+    Math.floor((t.length - r.length) / 2)
+  ), l = `${" ".repeat(a)}${r}`;
+  console.log(`
+${m(t)}
+${m(e)}
+${m(n)}
+${G(o)}
+${G(s)}
+${xt(i)}
+${m(l)}
+`);
+}
+function ne(t) {
+  const e = String(t.getHours()).padStart(2, "0"), n = String(t.getMinutes()).padStart(2, "0"), o = String(t.getSeconds()).padStart(2, "0");
+  return `${e}:${n}:${o}`;
+}
+async function oe() {
+  const t = process.argv.slice(2), e = t.includes("--update-snapshots") || t.includes("-u");
+  Dt(e), ee();
+  const n = /* @__PURE__ */ new Date();
+  Tt(), kt(), wt();
+  const o = await Q(process.cwd());
+  if (o.length === 0)
+    return console.log("No .eval.js or .eval.ts files found."), 1;
+  const i = (await Promise.allSettled(
+    o.map(
+      (c) => N.run(
+        { evalFile: c },
+        () => import(at(c).href)
+      )
+    )
+  )).map((c, h) => ({ result: c, file: o[h] })).filter(({ result: c }) => c.status === "rejected");
+  if (i.length > 0) {
+    for (const c of i) {
+      const h = c.result.status === "rejected" ? c.result.reason : void 0;
+      console.error(`Error executing ${c.file}: ${String(h)}`);
+    }
+    return 1;
+  }
+  const a = (await yt()).filter(
+    (c) => c.status === "rejected"
+  );
+  if (a.length > 0) {
+    for (const c of a)
+      c.status === "rejected" && console.error(`Error executing async test: ${String(c.reason)}`);
+    return 1;
+  }
+  const l = $t();
+  if (l.length > 0) {
+    console.error("❌ Failed tests:");
+    for (const [c, h] of l.entries()) {
+      const R = [h.describePath, h.itPath].filter((tt) => tt.length > 0).join(" > "), q = R.length > 0 ? `${R}: ` : "";
+      console.error(`${c + 1}. ${q}${h.message}`);
+    }
+    return 1;
+  }
+  const d = St(), g = Date.now() - n.getTime();
+  return console.log(
+    [
+      "---",
+      `${f("Files")}  ${o.length} passed`,
+      `${f("Evals")}  ${d} passed`,
+      `${f("Start at")}  ${ne(n)}`,
+      `${f("Duration")}  ${g}ms`
+    ].join(`
+`)
+  ), 0;
+}
+Object.assign(globalThis, { describe: bt, it: Kt, expect: Vt, prompt: y, promptFile: Bt });
+oe().then((t) => {
+  process.exit(t);
+}).catch((t) => {
+  console.error(`Unexpected error: ${String(t)}`), process.exit(1);
+});
+export {
+  oe as runCli
+};

package/package.json

@@ -0,0 +1,46 @@
+{
+  "name": "katt",
+  "version": "0.0.2",
+  "description": "CLI tool that tests the output of agentic AI tools",
+  "keywords": [
+    "cli",
+    "ai",
+    "agentic-ai",
+    "testing",
+    "evaluation"
+  ],
+  "author": "Raphael Porto (https://github.com/raphaelpor)",
+  "license": "MIT",
+  "type": "module",
+  "main": "dist/index.js",
+  "bin": {
+    "katt": "dist/index.js"
+  },
+  "scripts": {
+    "build": "vite build",
+    "dev": "tsx src/index.ts",
+    "lint": "biome lint ./src",
+    "format": "biome format --write ./src",
+    "test": "vitest",
+    "typecheck": "tsc -p tsconfig.json --noEmit",
+    "test:build": "node ./dist/index.js"
+  },
+  "types": "dist/index.d.ts",
+  "devDependencies": {
+    "@biomejs/biome": "1.9.4",
+    "@types/node": "25.2.0",
+    "tsx": "4.21.0",
+    "typescript": "5.8.2",
+    "vite": "7.3.1",
+    "vite-plugin-dts": "4.5.4",
+    "vitest": "3.2.4",
+    "vscode-jsonrpc": "^8.2.1"
+  },
+  "dependencies": {
+    "@github/copilot-sdk": "^0.1.21"
+  },
+  "bugs": {
+    "url": "https://github.com/raphaelpor/katt/issues"
+  },
+  "homepage": "https://github.com/raphaelpor/katt"
+}