@kortyx/cli 0.11.4 → 0.12.0

package/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## [0.12.0](https://github.com/kortyx-io/kortyx/compare/cli-v0.11.4...cli-v0.12.0) (2026-10-03)
+
+
+### Features
+
+* **evals:** run conversation suites from Studio and CLI ([#251](https://github.com/kortyx-io/kortyx/issues/251)) ([e653c91](https://github.com/kortyx-io/kortyx/commit/e653c9188d2fad4da0c75e79d090a8dedb639b92))
+
+
+### Dependencies
+
+* The following workspace dependencies were updated
+  * dependencies
+    * @kortyx/agent bumped to 0.29.0
+    * @kortyx/telemetry-contracts bumped to 0.13.0
+
 ## [0.11.4](https://github.com/kortyx-io/kortyx/compare/cli-v0.11.3...cli-v0.11.4) (2026-09-30)
 
 

package/README.md

@@ -248,6 +248,124 @@ profile referencing that project's read key—even if the deployment URL is the
 same. Account login and remote project/key administration are not implemented.
 VPN/private-network requirements remain in effect.
 
+## Eval suites and CI
+
+## Run locally without Studio
+
+Export the existing `createEvals` instance as `evals` (or the default export) from
+an application module such as `src/evals/index.ts`. Use a dedicated eval module
+that initializes the agent and suites without starting your HTTP server.
+TypeScript and JavaScript entries use the CLI's existing module loader and nearest
+`tsconfig.json` path aliases. The loader stays active for lazy workflow imports
+until execution finishes. Keep initialization synchronous; do asynchronous
+identity setup inside `createEvals.setup`.
+
+Add a script to your application's `package.json`:
+
+```json
+{
+  "scripts": {
+    "eval": "kortyx evals run --entry ./src/evals/index.ts"
+  }
+}
+```
+
+```sh
+pnpm eval
+pnpm eval --suite product-ambiguity --case choose-blue --repetitions 3
+pnpm exec kortyx evals list --entry ./src/evals/index.ts
+pnpm eval --suite product-ambiguity --concurrency 2 --json > eval-results.json
+```
+
+The command calls the exported instance's `run()` directly. Your existing setup,
+permission binding, custom executor, interrupt responders, cleanup and code judge
+all run in the application process. No Studio, consumer HTTP endpoint, target
+configuration or Studio key is required. Semantic criteria require a code judge;
+interaction and structured-output checks alone require no judge. Workflow models,
+tools and authentication still need their usual application configuration.
+
+The CLI loads `.env` then overlays `.env.local` before importing the entry.
+In a monorepo it loads defaults from the Git or pnpm workspace root down to the
+app directory, with nearer files taking precedence. Existing shell variables
+always win.
+To select other files, repeat `--env`; later files override earlier ones:
+
+```sh
+pnpm eval --env /private/development.env --env /private/eval.env
+```
+
+For a one-line package script without repeated flags, remember paths in a
+**gitignored** `.env.evals.json` in the app or workspace root:
+
+```json
+{
+  "envFiles": ["/private/development.env", "/private/eval.env"]
+}
+```
+
+Keep the profile and referenced files owner-only (`chmod 600`). Relative paths
+in the profile resolve from its directory. The nearest profile replaces default
+env loading; discovery stops at a Git or pnpm workspace boundary. Explicit
+`--env` flags replace both the profile and defaults, and resolve from the
+invocation directory. Missing configured files fail before loading the app;
+credentials are never printed. The CLI handles loading directly, so no launcher
+script or shell exports are required. Package scripts in monorepos can forward
+with `pnpm --filter my-agent eval`.
+
+Model credentials remain application-owned, for example `OPENROUTER_API_KEY`
+for an OpenRouter app judge. Studio's judge settings are not used by this command.
+
+Omitting `--suite` runs all configured suites in definition order. `--case` accepts
+space-separated case IDs and requires `--suite`. `--repetitions` and `--concurrency`
+override the SDK's run settings; omitted flags preserve them. `--export NAME`
+selects another named instance instead of `evals`/default.
+
+Interactive terminals show an animated suite progress bar, one live row per active
+attempt, elapsed time, and the actual setup, workflow, interrupt-response, judging
+and cleanup phases. Every completed step and case gets its own result, followed
+by a colored suite verdict and counts. Long calls keep animating; concurrent
+attempts remain distinct. The bar measures completed attempts, not estimated model
+completion. Application logs/warnings are preserved above the live display.
+Piped output is plain text with per-case start and step results; `--no-color` or
+`NO_COLOR` disables color/live progress. Explicit `--color` forces the interactive
+report, including when `NO_COLOR` is set. `--json` always disables the terminal UI.
+`--json` replaces the report with one JSON object containing `schemaVersion: 1`, `status`, aggregated
+`counts`, and full SDK `runs` (including observations). Keep application logs off
+stdout when consuming JSON. Configuration/import failures use stderr.
+
+Exit status is `0` when all selected suites pass, `1` for failed/errored runs or
+configuration errors, and `130` for cancellation. Ctrl+C forwards an abort signal,
+allows SDK cleanup and stops later suites. Cancellation remains cooperative: app
+code must honor its signal. Results are returned in the terminal/JSON; this local
+command does not save eval records to Studio. Agent telemetry, if configured by
+the application, continues to follow its existing settings.
+
+
+```bash
+kortyx studio evals suites list --connection staging --json
+kortyx studio evals suites get product-ambiguity --target catalog --connection staging --include-content --json
+kortyx studio evals runs start product-ambiguity --target catalog --connection staging --json
+kortyx studio evals runs list --connection staging --json
+kortyx studio evals runs get <eval-run-uuid-or-studio-url> --connection staging --json
+kortyx studio evals runs cancel <eval-run-uuid-or-studio-url> --connection staging --json
+```
+
+Eval commands reuse project connections and the Studio API. Discovery and reads
+need `studio:read`; start/cancel also require `eval:run`. The application's eval
+service key remains on the API server. Starting discovers the current revision,
+enqueues once and returns the run ID immediately, without waiting for a grade.
+Use `--case` repeatedly, `--repetitions` (1–20), `--concurrency` (1–4), and
+`--environment` to select execution; total attempts are capped at 100.
+
+Default detail exposes statuses, counts and criterion verdicts. Full definitions,
+observations and reasons require `--include-content`; credentials remain redacted.
+History is the latest 100 project records. Cancellation is cooperative. The CLI
+does not retry a POST automatically or treat an accepted request as a passing eval.
+
+See [CLI and post-deployment CI](https://github.com/kortyx-io/kortyx/blob/main/docs/evals/cli-and-ci.md)
+for a `continue-on-error` CI step that enqueues a suite after deployment and
+returns immediately, plus target configuration, permissions and output contracts.
+
 ## Push workflow topology to Studio
 
 Kortyx Studio should receive workflow topology as a build/deploy artifact, not only as best-effort runtime telemetry. Use `topology push` in local dev, release CI, or deployment pipelines: