@verica-app/cli 0.1.2 → 0.1.3

package/CHANGELOG.md

@@ -0,0 +1,66 @@
+# Changelog
+
+All notable changes to `@verica-app/cli` are documented here.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+This package is **pre-1.0**: while on `0.x`, a breaking change bumps the **minor**
+version and additive features/fixes bump the **patch** (see [Stability](./README.md#stability)).
+
+## [Unreleased]
+
+## [0.1.3] - 2026-06-18
+
+### Added
+
+- `--tools <file>` flag and `tools:` in `.verica.yml` to push a prompt's tool
+  definitions from the repo. The manifest accepts either a **path to a JSON file**
+  or an **inline array**. Each tool may use Verica's flat shape
+  (`{ name, description, parameters }`) **or** the OpenAI wrapper
+  (`{ type: "function", function: { … } }`), which is auto-unwrapped — paste your
+  real schemas as-is.
+
+### Changed
+
+- **Prompt push is now field-level.** `--prompt` (user template), `--system-prompt`,
+  and `--tools` are independent: push only the fields you changed and every omitted
+  field is **inherited from the eval's current prompt version**. A new version is
+  created only if the merged content differs.
+  - Previously, a push that included `--prompt` without `--system-prompt` produced a
+    version with **no system prompt** (it was dropped, not inherited). If your CI
+    relied on that, pass all the fields you intend to set.
+  - `--system-prompt` (or `--tools`) can now be pushed **on its own** — earlier the
+    CLI sent a prompt block only when `--prompt` was present.
+- Prompt templates reference dataset columns by their **bare name** (`{{ pais }}`),
+  and grader/judge prompts reference the model output via `{{ output.text }}` /
+  `{{ output.tool_calls }}`. The legacy `{{ item.* }}` / `{{ sample.* }}` forms still
+  resolve, so existing prompt files keep working.
+
+## [0.1.2] - 2026-06-18
+
+### Changed
+
+- Default to the hosted API (`https://verica.app`). `VERICA_BASE_URL` and `--base-url`
+  are now overrides for local dev / self-hosting only — clients no longer configure a URL.
+
+## [0.1.1] - 2026-06-18
+
+### Changed
+
+- Published under the `@verica-app` scope, with a tag-triggered release workflow
+  (`cli-v*` → `npm publish --provenance`). No behavior change.
+
+## [0.1.0] - 2026-06-18
+
+### Added
+
+- Initial release. The `run` command:
+  - `--eval <id>` (single) or `--manifest <file>` (`.verica.yml`, multi-prompt).
+  - Pushes prompt content (`--prompt` / `--system-prompt`), versioned by content equality.
+  - `--model` · `--sampling <file.json>` for execution config.
+  - `--wait` polls to a terminal status; the exit code reflects the gate.
+  - `--junit <file>` · `--junit-mode rows|gate` for a JUnit report; `--json` for
+    machine-readable results.
+  - `--threshold` · `--baseline-ref` · `--baseline-run` to override the gate per branch.
+  - Git provenance (`--git-sha` / `--git-ref`) auto-detected from common CI env vars.
+  - Exit codes: `0` passed · `1` gate failed · `2` validation/transport error.
+  - `VERICA_TOKEN` is the only required secret; provider keys stay in Verica (BYOK).

package/README.md

@@ -27,6 +27,8 @@ npm i -D @verica-app/cli
 verica run \
   --eval eval_8x2k9d \
   --prompt prompts/support-agent.txt \
+  --system-prompt prompts/support-agent.system.txt \  # optional
+  --tools prompts/support-agent.tools.json \          # optional
   --model gpt-4.1-mini \
   --wait \
   --junit verica-results.xml \
@@ -44,10 +46,16 @@ verica run --manifest .verica.yml --wait --junit report.xml
 evals:
   - id: eval_8x2k9d
     prompt: prompts/support-agent.txt
+    systemPrompt: prompts/support-agent.system.txt
+    tools: prompts/support-agent.tools.json   # a path to a JSON file…
     sampling: { temperature: 0.2, maxTokens: 512 }
     model: gpt-4.1-mini
   - id: eval_3p1m7q
     prompt: prompts/triage.txt
+    tools:                                      # …or an inline array
+      - name: get_order
+        description: Look up an order by id
+        parameters: { type: object, properties: { id: { type: string } }, required: [id] }
     model: claude-sonnet-4-6
 ```
 
@@ -63,7 +71,7 @@ default — you don't configure a URL.
 ## Key flags
 
 - `--eval <id>` / `--manifest <file>` — what to run.
-- `--prompt <file>` / `--system-prompt <file>` — prompt content to push (versioned by content).
+- `--prompt <file>` / `--system-prompt <file>` / `--tools <file>` — prompt content to push (versioned by content). See [Prompt content](#prompt-content-what-you-push).
 - `--model <model>` · `--sampling <file.json>` — execution config.
 - `--wait` — poll to completion; the exit code reflects the gate.
 - `--junit <file>` · `--junit-mode rows|gate` — JUnit report (default `rows`).
@@ -74,9 +82,62 @@ default — you don't configure a URL.
 > Local dev / self-hosting only: point the CLI at another instance with `--base-url`
 > (or the `VERICA_BASE_URL` env var). Clients never need this.
 
+## Prompt content (what you push)
+
+The repo owns the **prompt**: the user template (`--prompt`), the system prompt
+(`--system-prompt`), and the tool definitions (`--tools`). The **dataset, graders,
+gate, and any few-shot/simulated turns stay in Verica** — they're the test scenario,
+managed by whoever owns the eval.
+
+Each of the three prompt fields is **independent and optional**: push the ones you
+changed and every omitted field is **inherited from the current version**. A push
+creates a new prompt version only if the merged content actually differs.
+
+```bash
+verica run --eval eval_8x2k9d --system-prompt prompts/agent.system.txt --model gpt-4.1-mini --wait
+# user template + tools inherited; only the system prompt re-versions
+```
+
+Templates reference dataset columns by name — e.g. `What is the capital of {{ pais }}?`
+(the column is `pais`). Grader/judge prompts can also reference the model output via
+`{{ output.text }}` / `{{ output.tool_calls }}`.
+
+**Tools** are pushed as JSON — a `--tools <file>`, or a path / inline array under
+`tools:` in the manifest. Each entry may be Verica's flat shape **or** the OpenAI
+wrapper (auto-unwrapped), so you can paste your real schemas as-is:
+
+```json
+[
+  { "name": "get_order", "description": "Look up an order by id",
+    "parameters": { "type": "object", "properties": { "id": { "type": "string" } }, "required": ["id"] } },
+  { "type": "function", "function": { "name": "cancel_order", "description": "…", "parameters": { "type": "object" } } }
+]
+```
+
+Tools are never executed — the model's *decision* to call one (and with which
+arguments) is what the eval grades.
+
 ## Exit codes
 
 `0` passed · `1` gate failed · `2` validation/transport error.
 
+## Stability
+
+This CLI is **pre-1.0 (`0.x`)**. The command surface, the `--json` payload, the JUnit
+output, and the prompt-push behavior are still settling and may change. Exit codes
+(`0`/`1`/`2`) are stable.
+
+During `0.x` the **minor** version is the breaking lever, so pin accordingly:
+
+```jsonc
+// package.json
+"@verica-app/cli": "~0.1"   // >=0.1.0 <0.2.0 — gets patches, not breaking minors
+```
+
+We bump the **minor** for any breaking change (flags, output shapes, push behavior) and
+the **patch** for additive features and fixes. **1.0** will freeze the commands, flags,
+exit codes, and output shapes under standard semver. See
+[CHANGELOG.md](./CHANGELOG.md) for what changed in each release.
+
 MIT licensed. There's no IP in the client — the engine, graders, gate, and crypto all
 run server-side behind the token API.

package/dist/cli.js

@@ -4067,9 +4067,16 @@ var samplingParamsSchema = external_exports.object({
   reasoning: external_exports.boolean().optional()
 });
 var runRequestSchema = external_exports.object({
-  /** Omit to run the eval's current (UI-managed) prompt unchanged. */
+  /**
+   * Prompt content to push. Omit the whole block to run the eval's current
+   * (UI-managed) prompt unchanged. Each field is independent: supply only the
+   * ones you're changing — every omitted field (template / systemPrompt / tools)
+   * is inherited from the current version, and a new version is created only if
+   * the merged result differs (e.g. push just `systemPrompt` to re-version the
+   * system prompt while keeping the user template).
+   */
   prompt: external_exports.object({
-    template: external_exports.string(),
+    template: external_exports.string().optional(),
     systemPrompt: external_exports.string().optional(),
     tools: external_exports.array(toolDefinitionSchema).optional()
   }).optional(),
@@ -4289,6 +4296,7 @@ function parseManifest(raw, source = ".verica.yml") {
       id: e.id,
       prompt: typeof e.prompt === "string" ? e.prompt : void 0,
       systemPrompt: typeof e.systemPrompt === "string" ? e.systemPrompt : void 0,
+      tools: typeof e.tools === "string" || Array.isArray(e.tools) ? e.tools : void 0,
       model: typeof e.model === "string" ? e.model : void 0,
       sampling: e.sampling ?? void 0
     };
@@ -4298,6 +4306,27 @@ async function loadManifest(path) {
   return parseManifest(await readFile(path, "utf8"), path);
 }
 
+// src/tools.ts
+function unwrap(entry) {
+  if (entry !== null && typeof entry === "object" && entry.type === "function" && typeof entry.function === "object" && entry.function !== null) {
+    return entry.function;
+  }
+  return entry;
+}
+function normalizeToolDefinitions(raw) {
+  if (!Array.isArray(raw)) {
+    throw new Error("tools must be a JSON array of tool definitions.");
+  }
+  return raw.map((entry, i) => {
+    const parsed = toolDefinitionSchema.safeParse(unwrap(entry));
+    if (!parsed.success) {
+      const why = parsed.error.issues.map((issue) => issue.message).join("; ");
+      throw new Error(`tools[${i}] is not a valid tool definition: ${why}`);
+    }
+    return parsed.data;
+  });
+}
+
 // src/junit.ts
 function esc(s) {
   return s.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;").replace(/"/g, "&quot;").replace(/'/g, "&apos;");
@@ -4392,7 +4421,7 @@ async function runCommand(opts) {
       );
       err(`  ${accepted.resultUrl}`);
       if (!opts.wait) {
-        summaries.push({ evalId: entry.id, runId: accepted.runId, resultUrl: accepted.resultUrl });
+        summaries.push(buildSummary(entry.id, { status: "queued", accepted }));
         continue;
       }
       const run = await pollUntilTerminal(client, accepted.runId, {
@@ -4406,12 +4435,12 @@ async function runCommand(opts) {
           opts.junitMode === "rows" ? rowsSuite(entry.id, await client.getResults(accepted.runId)) : gateSuite(entry.id, run)
         );
       }
-      summaries.push({ evalId: entry.id, runId: accepted.runId, ...run });
+      summaries.push(buildSummary(entry.id, { status: "waited", runId: accepted.runId, run }));
     } catch (e) {
       worst = EXIT.error;
       const message = e instanceof Error ? e.message : String(e);
       err(`\u2717 ${entry.id}: ${message}`);
-      summaries.push({ evalId: entry.id, error: message });
+      summaries.push(buildSummary(entry.id, { status: "error", message }));
     }
   }
   if (opts.junitFile && suites.length > 0) {
@@ -4432,6 +4461,7 @@ async function resolveEntries(opts) {
       id: opts.evalId,
       prompt: opts.promptFile,
       systemPrompt: opts.systemPromptFile,
+      tools: opts.toolsFile,
       model: opts.model
     }
   ];
@@ -4442,18 +4472,39 @@ async function buildRequest(entry, ctx) {
   }
   const template = entry.prompt ? await readFile2(entry.prompt, "utf8") : void 0;
   const systemPrompt = entry.systemPrompt ? await readFile2(entry.systemPrompt, "utf8") : void 0;
+  const tools = await resolveTools(entry.tools);
   let sampling = entry.sampling;
   if (!sampling && ctx.samplingFile) {
     sampling = JSON.parse(await readFile2(ctx.samplingFile, "utf8"));
   }
+  const prompt = template !== void 0 || systemPrompt !== void 0 || tools !== void 0 ? {
+    ...template !== void 0 ? { template } : {},
+    ...systemPrompt !== void 0 ? { systemPrompt } : {},
+    ...tools !== void 0 ? { tools } : {}
+  } : void 0;
   return {
     model: entry.model,
-    ...template !== void 0 ? { prompt: { template, ...systemPrompt !== void 0 ? { systemPrompt } : {} } } : {},
+    ...prompt ? { prompt } : {},
     ...sampling ? { samplingParams: sampling } : {},
     ...ctx.git ? { git: ctx.git } : {},
     ...ctx.gate ? { gate: ctx.gate } : {}
   };
 }
+function buildSummary(evalId, outcome) {
+  switch (outcome.status) {
+    case "queued":
+      return { evalId, runId: outcome.accepted.runId, resultUrl: outcome.accepted.resultUrl };
+    case "waited":
+      return { evalId, runId: outcome.runId, ...outcome.run };
+    case "error":
+      return { evalId, error: outcome.message };
+  }
+}
+async function resolveTools(tools) {
+  if (tools === void 0) return void 0;
+  const raw = typeof tools === "string" ? JSON.parse(await readFile2(tools, "utf8")) : tools;
+  return normalizeToolDefinitions(raw);
+}
 function resolveGit(opts) {
   const sha = opts.gitSha ?? process.env.GITHUB_SHA ?? process.env.CI_COMMIT_SHA;
   const ref = opts.gitRef ?? process.env.GITHUB_REF ?? process.env.CI_COMMIT_REF_NAME;
@@ -4494,8 +4545,13 @@ Usage:
 
 Options:
   --eval <id>            Eval to run (or use --manifest for many).
-  --prompt <file>        Prompt template file to push (versioned by content).
-  --system-prompt <file> System-prompt file (optional).
+  --prompt <file>        User prompt (template) file to push (versioned by content).
+  --system-prompt <file> System-prompt file. Either prompt file is optional and
+                         independent: push one and the other is inherited from
+                         the current version (omit both to run it unchanged).
+  --tools <file>         Tool definitions to push: a JSON file (Verica-flat or
+                         OpenAI {type:function,\u2026} entries). Omit to inherit the
+                         current version's tools. Inline arrays: .verica.yml only.
   --model <model>        Model to sample under (overrides the manifest).
   --sampling <file>      JSON sampling params (temperature, maxTokens, \u2026).
   --manifest <file>      .verica.yml mapping prompts \u2192 eval IDs (multi-prompt).
@@ -4531,6 +4587,7 @@ async function main() {
       eval: { type: "string" },
       prompt: { type: "string" },
       "system-prompt": { type: "string" },
+      tools: { type: "string" },
       model: { type: "string" },
       sampling: { type: "string" },
       manifest: { type: "string" },
@@ -4567,6 +4624,7 @@ async function main() {
     evalId: values.eval,
     promptFile: values.prompt,
     systemPromptFile: values["system-prompt"],
+    toolsFile: values.tools,
     samplingFile: values.sampling,
     model: values.model,
     manifestFile: values.manifest,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@verica-app/cli",
-  "version": "0.1.2",
+  "version": "0.1.3",
   "private": false,
   "description": "Run a Verica eval from CI and block the merge on the result.",
   "license": "MIT",
@@ -12,18 +12,15 @@
     "prompt",
     "testing"
   ],
-  "repository": {
-    "type": "git",
-    "url": "git+https://github.com/mtn-labs/evals.git",
-    "directory": "packages/cli"
-  },
-  "homepage": "https://github.com/mtn-labs/evals/tree/main/packages/cli#readme",
+  "homepage": "https://verica.app",
   "type": "module",
   "bin": {
     "verica": "./dist/cli.js"
   },
   "files": [
-    "dist"
+    "dist",
+    "README.md",
+    "CHANGELOG.md"
   ],
   "publishConfig": {
     "access": "public"