cuttlely 1.0.3 → 1.2.0

package/README.md

@@ -12,10 +12,10 @@ Requires Node.js 20 or newer, and Docker.
 npx cuttlely start
 ```
 
-That pulls `ghcr.io/cuttlely/cuttlely:1.0.0` (the same version as this package), keeps data in the Docker volume `cuttlely-data`, publishes port **43117**, waits until `http://127.0.0.1:43117/api/v1/ping` answers `pong`, prints `http://localhost:43117`, and opens it.
+That pulls `ghcr.io/cuttlely/cuttlely`, tagged with the same version as the `cuttlely` package you ran, keeps data in the Docker volume `cuttlely-data`, publishes port **43117**, waits until `http://127.0.0.1:43117/api/v1/ping` answers `pong`, prints `http://localhost:43117`, and opens it.
 
 ```bash
-npx cuttlely start --port 8080 --tag 1.0.0 --env-file .env --no-open
+npx cuttlely start --port 8080 --tag 1.1.0 --env-file .env --no-open
 ```
 
 | Flag         | Meaning                                                                  |
@@ -50,13 +50,13 @@ export CUTTLELY_API_KEY=<key>
 npx cuttlely check --flow <flow id> --server http://127.0.0.1:43117 --timeout 300 --json check.json --fail-on-changed
 ```
 
-| Flag                | Meaning                                                                                                   |
-| ------------------- | --------------------------------------------------------------------------------------------------------- |
-| `--flow`            | Flow id from the canvas address bar. Required.                                                            |
-| `--server`          | Server origin. Default `http://127.0.0.1:43117`.                                                          |
-| `--timeout`         | Seconds to wait. Default 300.                                                                             |
-| `--json`            | Write the full check JSON to this path.                                                                   |
-| `--fail-on-changed` | Exit 1 when any answer is Changed. Without it, a finished check exits 0 and the summary still lists them. |
+| Flag                | Meaning                                                                                                                                 |
+| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
+| `--flow`            | Flow id from the canvas address bar. Required.                                                                                          |
+| `--server`          | Server origin. Default `http://127.0.0.1:43117`.                                                                                        |
+| `--timeout`         | Seconds to wait for the check. Default 300. The final confirmation read after a `--gate` pass has its own window of at most 30 seconds. |
+| `--json`            | Write the full check JSON to this path.                                                                                                 |
+| `--fail-on-changed` | Exit 1 when any answer is Changed. Without it, a finished check exits 0 and the summary still lists them.                               |
 
 `pnpm dev` serves the API on port 3000. `scripts/start-cuttlely.sh` and `npx cuttlely start` listen on port 43117.
 
@@ -70,6 +70,32 @@ A check the server marks stopped or failed also exits 2. If the draft is saved,
 
 The key needs both permissions for the flow: `chatflows:view` and `chatflows:update`, or `agentflows:view` and `agentflows:update`. Update starts the check. View is what reading the result uses. The guide is [Fail a build when answers change](https://github.com/cuttlely/cuttlely/blob/main/docs/how-to/ci-gate.md).
 
+## Score a test set
+
+`--dataset` or `--gate` asks a dataset instead of the four recent questions. The command exits on the server verdict. It does not score the answers itself.
+
+```bash
+npx cuttlely check --flow <flow id> --dataset <dataset id> --evaluator <evaluator id> --threshold 80
+npx cuttlely check --flow <flow id> --gate
+```
+
+| Flag          | Meaning                                                                                   |
+| ------------- | ----------------------------------------------------------------------------------------- |
+| `--dataset`   | Dataset id. Selects scored mode.                                                          |
+| `--evaluator` | Evaluator id. Repeat for another evaluator.                                               |
+| `--threshold` | Pass mark from 0 to 100. Omit it and the server uses 100.                                 |
+| `--gate`      | Dataset, evaluators, and threshold saved on the flow. Do not combine it with `--dataset`. |
+
+Do not combine those flags with `--fail-on-changed`. The gate covers Publish of the current draft only. Roll back, Promote, and `PUT /api/v1/chatflows/:id` can still change Live. A harness cannot store a passing test set. `--dataset` can score a harness for that run only. `--gate` on a harness exits 2 before any check.
+
+| Code | When                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| 0    | The server verdict is passed. With `--gate`, a second read of the flow still stores this check as passed with an empty `block` (`publishGate.enabled` is true, `publishGate.result.checkId` is this check, and `publishGate.result.verdict` is `passed`).                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| 1    | The server verdict is failed.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| 2    | The same failures as the question check, a missing verdict, a usage error, or a withheld dataset or evaluator id. `--gate` also exits 2 when the confirmation read fails: a non-empty block, a different checkId, or a missing passed. Turning the passing test set off during a `--gate` check, after the server verdict is `passed`, exits 2. The message is `The saved test set changed during the check. Run it again.` when the server sends no `block` text. Changing the dataset, the evaluators, or the pass mark during that check exits 2 the same way, and a `block` from the server is printed instead. A failed verdict exits 1 and does not read the flow again. |
+
+A 403 names the flow view and update permissions, plus `datasets:view` and `evaluators:view`. The summary prints the pass rate, the threshold, and the cases whose draft scores are not Pass. `--json` writes the full check, including each score.
+
 The container name is `cuttlely`. `start` leaves it running when it is already on the same image and port. `upgrade` always pulls and starts it again. The volume is `cuttlely-data`, mounted at `/var/cuttlely`.
 
 Compose uses the volume name `cuttlely_cuttlely-data`. This launcher uses `cuttlely-data`.

package/lib/check.js

@@ -12,6 +12,9 @@ export const EXIT_PASS = 0
 export const EXIT_CHANGED = 1
 export const EXIT_ERROR = 2
 
+// Flow, dataset, and evaluator ids share this shape.
+export const RESOURCE_ID_RE = /^[A-Za-z0-9][A-Za-z0-9_-]{0,127}$/
+
 const ANSWER_CLIP = 240
 
 function redact(text, secret) {
@@ -72,14 +75,52 @@ function errorCases(check) {
     return (check?.cases || []).filter((row) => row && (row.live?.error || row.draft?.error))
 }
 
+function verdictLine(verdict) {
+    const rate = verdict.passRate == null ? 'no scored answers' : `${verdict.passRate}%`
+    const mark = verdict.threshold == null ? 'n/a' : `${verdict.threshold}%`
+    const passed = Number(verdict.passed ?? 0)
+    const failed = Number(verdict.failed ?? 0)
+    const errored = Number(verdict.errored ?? 0)
+    return `Pass rate: ${rate} (threshold ${mark}). ${passed} passed, ${failed} failed, ${errored} errored.`
+}
+
+function failingCases(check) {
+    return (check?.cases || []).filter((row) => {
+        const scores = row?.draft?.scores
+        if (!Array.isArray(scores) || !scores.length) return false
+        return scores.some((score) => score && score.result !== 'Pass')
+    })
+}
+
+function failingLines(check) {
+    const rows = failingCases(check)
+    if (!rows.length) return []
+    const lines = ['', 'Failing']
+    for (const row of rows) {
+        lines.push(`  ${oneLine(row.question)}`)
+        for (const score of row.draft.scores) {
+            if (!score || score.result === 'Pass') continue
+            lines.push(`    ${score.name ? `${score.name}: ${score.result}` : score.result}`)
+        }
+        if (row.expected) lines.push(`    Expected: ${oneLine(row.expected)}`)
+        if (row.draft?.text) lines.push(`    Draft: ${oneLine(row.draft.text)}`)
+        else if (row.draft?.error) lines.push(`    ${oneLine(row.draft.error)}`)
+    }
+    return lines
+}
+
 export function summaryText(check) {
     const summary = check.summary || {}
     const lines = [
         `Check ${check.id} ${check.status}`,
-        `Questions: ${summary.ran ?? 0} ran, ${summary.same ?? 0} same, ${summary.changed ?? 0} changed, ${summary.errors ?? 0} errors`,
+        `Questions: ${summary.ran ?? 0} ran, ${summary.same ?? 0} same, ${summary.changed ?? 0} changed, ${summary.errors ?? 0} errors`
+    ]
+    // Present only when the server finished a dataset check. Diff mode has no verdict.
+    if (check.verdict && typeof check.verdict === 'object') lines.push(verdictLine(check.verdict))
+    lines.push(
         `Live: ${summary.liveTokens ?? 0} tokens, median ${formatMs(summary.liveMedianMs)}`,
         `Draft: ${summary.draftTokens ?? 0} tokens, median ${formatMs(summary.draftMedianMs)}`
-    ]
+    )
     // The check total, with each model priced once. It is not the sum of the per-answer figures.
     const liveEstimate = estimateLine('Live', summary.liveEstimatedCostUsd, summary.livePrice)
     const draftEstimate = estimateLine('Draft', summary.draftEstimatedCostUsd, summary.draftPrice)
@@ -106,11 +147,11 @@ export function summaryText(check) {
             lines.push(`    ${oneLine(why)}`)
         }
     }
+    if (check.verdict && typeof check.verdict === 'object') lines.push(...failingLines(check))
     return lines.join('\n')
 }
 
-// Replay is the only verdict this command runs. A later scored mode (pass or fail
-// from a test set) should return the same { code, label } and be selected in selectVerdict.
+// Replay is the question check. Scored mode reads the server verdict and does not rescore.
 export function replayVerdict(check, { failOnChanged }) {
     if (!check || typeof check !== 'object') return { code: EXIT_ERROR, label: 'error' }
     if (check.status === 'failed') return { code: EXIT_ERROR, label: 'failed' }
@@ -123,7 +164,21 @@ export function replayVerdict(check, { failOnChanged }) {
     return { code: EXIT_PASS, label: 'pass' }
 }
 
+// Exit 1 is the server's failed verdict. A missing verdict is an error, not a guess from the pass rate.
+export function scoredVerdict(check) {
+    if (!check || typeof check !== 'object') return { code: EXIT_ERROR, label: 'error' }
+    if (check.status === 'failed') return { code: EXIT_ERROR, label: 'failed' }
+    if (check.status === 'stopped') return { code: EXIT_ERROR, label: 'stopped' }
+    if (check.status !== 'done') return { code: EXIT_ERROR, label: 'incomplete' }
+    if (check.draftChanged) return { code: EXIT_ERROR, label: 'draft-changed' }
+    const result = check.verdict && check.verdict.result
+    if (result === 'passed') return { code: EXIT_PASS, label: 'passed' }
+    if (result === 'failed') return { code: EXIT_CHANGED, label: 'score-failed' }
+    return { code: EXIT_ERROR, label: 'no-verdict' }
+}
+
 export function selectVerdict(check, options) {
+    if (options && options.scored) return scoredVerdict(check)
     return replayVerdict(check, options)
 }
 
@@ -171,8 +226,98 @@ function isNoDraft(result) {
 
 // Starting a check needs update. Reading it needs view. A key with only one of those gets 403,
 // and the server message does not say which permission is missing.
-function permissionDeniedMessage() {
-    return 'The API key needs chatflows:view and chatflows:update, or agentflows:view and agentflows:update.'
+function permissionDeniedMessage(scored) {
+    const flow = 'The API key needs chatflows:view and chatflows:update, or agentflows:view and agentflows:update.'
+    if (!scored) return flow
+    return `${flow} Scored mode also needs datasets:view and evaluators:view.`
+}
+
+function scoreFailedLine(verdict) {
+    const rate = !verdict || verdict.passRate == null ? 'no scored answers' : `${verdict.passRate}%`
+    const mark = !verdict || verdict.threshold == null ? 'n/a' : `${verdict.threshold}%`
+    return `The test set failed at ${rate} (threshold ${mark}).`
+}
+
+const STALE_GATE = 'The saved test set changed during the check. Run it again.'
+// The check can use the whole --timeout. The confirmation read gets its own window, capped so a hung read cannot double a 3600s wait.
+const CONFIRM_READ_CAP_SECONDS = 30
+
+function confirmReadSeconds(timeoutSeconds) {
+    return Math.min(timeoutSeconds, CONFIRM_READ_CAP_SECONDS)
+}
+
+// An enabled gate always stores a dataset and at least one evaluator. A read that
+// omits them withheld the ids: datasets:view and evaluators:view are what reveal them.
+function withheldIdsMessage(datasetMissing, evaluatorsMissing) {
+    if (datasetMissing && evaluatorsMissing) {
+        return 'The response did not include the dataset id or the evaluator ids. The API key needs datasets:view and evaluators:view.'
+    }
+    if (datasetMissing) return 'The response did not include the dataset id. The API key needs datasets:view.'
+    return 'The response did not include the evaluator ids. The API key needs evaluators:view.'
+}
+
+function datasetIdMissing(gate) {
+    return typeof gate.datasetId !== 'string' || gate.datasetId.trim() === ''
+}
+
+function evaluatorIdsMissing(gate) {
+    return !Array.isArray(gate.evaluatorIds) || gate.evaluatorIds.length === 0
+}
+
+// --gate copies the dataset, evaluators, and threshold the flow already saved.
+function gateSettings(body) {
+    const gate = body && body.publishGate
+    if (!gate || gate.enabled !== true) {
+        return { ok: false, message: 'This flow has no saved test set. Turn the passing test set on, or pass --dataset.' }
+    }
+    const datasetMissing = datasetIdMissing(gate)
+    const evaluatorsMissing = evaluatorIdsMissing(gate)
+    if (datasetMissing || evaluatorsMissing) return { ok: false, message: withheldIdsMessage(datasetMissing, evaluatorsMissing) }
+    if (!RESOURCE_ID_RE.test(gate.datasetId)) {
+        return { ok: false, message: 'The saved test set has no dataset id.' }
+    }
+    const evaluatorIds = []
+    const seen = new Set()
+    for (const id of gate.evaluatorIds) {
+        if (typeof id !== 'string' || !RESOURCE_ID_RE.test(id)) {
+            return { ok: false, message: 'The saved test set has an evaluator id this command cannot use.' }
+        }
+        if (seen.has(id)) continue
+        seen.add(id)
+        evaluatorIds.push(id)
+    }
+    let threshold = null
+    if (gate.threshold != null) {
+        if (typeof gate.threshold !== 'number' || !Number.isInteger(gate.threshold) || gate.threshold < 0 || gate.threshold > 100) {
+            return { ok: false, message: 'The saved pass mark is not a whole number from 0 to 100.' }
+        }
+        threshold = gate.threshold
+    }
+    return { ok: true, datasetId: gate.datasetId, evaluatorIds, threshold }
+}
+
+// A passed --gate check is current only when Publish would still accept this run.
+function gateBlockMessage(gate) {
+    if (gate && typeof gate.block === 'string' && gate.block.trim()) return gate.block
+    return STALE_GATE
+}
+
+function gatePassStillCurrent(body, checkId) {
+    const gate = body && body.publishGate
+    const message = gateBlockMessage(gate)
+    if (!gate || gate.enabled !== true) return { ok: false, message }
+    if (typeof gate.block === 'string' && gate.block.trim()) return { ok: false, message }
+    const result = gate.result
+    if (!result || result.checkId !== checkId || result.verdict !== 'passed') return { ok: false, message }
+    return { ok: true }
+}
+
+function requestBody({ scored, datasetId, evaluatorIds, threshold }) {
+    if (!scored) return { count: DEFAULT_QUESTION_COUNT }
+    const body = { datasetId }
+    if (evaluatorIds && evaluatorIds.length) body.evaluatorIds = evaluatorIds
+    if (threshold != null) body.threshold = threshold
+    return body
 }
 
 function writeResult(writeFile, jsonPath, check, secret) {
@@ -208,7 +353,12 @@ export async function runCheck(options, io = {}) {
     const timeoutSeconds = options.timeoutSeconds ?? DEFAULT_TIMEOUT_SECONDS
     const flow = options.flow
     const failOnChanged = Boolean(options.failOnChanged)
-    const base = `${server}/api/v1/flow-drafts/${encodeURIComponent(flow)}/check`
+    const scored = Boolean(options.scored || options.gate)
+    let datasetId = options.datasetId
+    let evaluatorIds = options.evaluatorIds || []
+    let threshold = options.threshold ?? null
+    const flowUrl = `${server}/api/v1/flow-drafts/${encodeURIComponent(flow)}`
+    const base = `${flowUrl}/check`
     const headers = { Authorization: `Bearer ${secret}`, Accept: 'application/json' }
     const deadline = now() + timeoutSeconds * 1000
 
@@ -255,16 +405,34 @@ export async function runCheck(options, io = {}) {
     }
 
     try {
+        if (options.gate) {
+            const loaded = await callApi(fetchImpl, flowUrl, {
+                method: 'GET',
+                headers,
+                signal: deadlineSignal(deadline, now)
+            })
+            if (loaded.timeout) return fail(`Timed out after ${timeoutSeconds}s before the check started.`)
+            if (loaded.network) return fail(`Could not reach ${server}. ${loaded.message}`)
+            if (!loaded.ok) {
+                if (loaded.status === 403) return fail(permissionDeniedMessage(true))
+                return fail(`Check failed (${loaded.status}): ${loaded.message}`)
+            }
+            const parsed = gateSettings(loaded.json)
+            if (!parsed.ok) return fail(parsed.message)
+            datasetId = parsed.datasetId
+            evaluatorIds = parsed.evaluatorIds
+            threshold = parsed.threshold
+        }
         const started = await callApi(fetchImpl, base, {
             method: 'POST',
             headers: { ...headers, 'Content-Type': 'application/json' },
-            body: JSON.stringify({ count: DEFAULT_QUESTION_COUNT }),
+            body: JSON.stringify(requestBody({ scored, datasetId, evaluatorIds, threshold })),
             signal: deadlineSignal(deadline, now)
         })
         if (started.timeout) return fail(`Timed out after ${timeoutSeconds}s before the check started.`)
         if (started.network) return fail(`Could not reach ${server}. ${started.message}`)
         if (!started.ok) {
-            if (started.status === 403) return fail(permissionDeniedMessage())
+            if (started.status === 403) return fail(permissionDeniedMessage(scored))
             if (isNoDraft(started)) return fail(`No draft to check. ${started.message}`)
             return fail(`Check failed (${started.status}): ${started.message}`)
         }
@@ -293,7 +461,7 @@ export async function runCheck(options, io = {}) {
             })
             if (polled.timeout) return giveUp()
             if (polled.network) return failRunning(`Could not reach ${server}. ${polled.message}`)
-            if (polled.status === 403) return failRunning(permissionDeniedMessage())
+            if (polled.status === 403) return failRunning(permissionDeniedMessage(scored))
             if (!polled.ok) return failRunning(`Check failed (${polled.status}): ${polled.message}`)
             if (!polled.json || typeof polled.json !== 'object') return failRunning('The server did not return a check.')
             const previous = check.completed
@@ -316,7 +484,7 @@ export async function runCheck(options, io = {}) {
             return EXIT_ERROR
         }
         out(say(summaryText(check)))
-        const verdict = selectVerdict(check, { failOnChanged })
+        const verdict = selectVerdict(check, { failOnChanged, scored })
         if (verdict.label === 'draft-changed') {
             err('The draft changed while the check was running, so the result is out of date.')
         } else if (verdict.label === 'failed') {
@@ -326,10 +494,32 @@ export async function runCheck(options, io = {}) {
         } else if (verdict.label === 'changed') {
             const count = Number(check.summary?.changed ?? 0)
             err(count === 1 ? '1 answer changed.' : `${count} answers changed.`)
+        } else if (verdict.label === 'score-failed') {
+            err(scoreFailedLine(check.verdict))
+        } else if (verdict.label === 'no-verdict') {
+            err('The server did not return a verdict for this test set.')
         } else if (verdict.code === EXIT_ERROR) {
             err('The check did not finish.')
         }
         if (verdict.code === EXIT_ERROR) await stopIfStarted()
+        // --gate can exit 0 only when this run is still the stored pass. A threshold,
+        // row, or evaluator change during the check makes Publish refuse it.
+        if (options.gate && verdict.code === EXIT_PASS) {
+            const confirmSeconds = confirmReadSeconds(timeoutSeconds)
+            const again = await callApi(fetchImpl, flowUrl, {
+                method: 'GET',
+                headers,
+                signal: deadlineSignal(now() + confirmSeconds * 1000, now)
+            })
+            if (again.timeout) return fail(`Timed out after ${confirmSeconds}s before the saved test set could be read again.`)
+            if (again.network) return fail(`Could not reach ${server}. ${again.message}`)
+            if (!again.ok) {
+                if (again.status === 403) return fail(permissionDeniedMessage(true))
+                return fail(`Check failed (${again.status}): ${again.message}`)
+            }
+            const current = gatePassStillCurrent(again.json, check.id)
+            if (!current.ok) return fail(current.message)
+        }
         return verdict.code
     } catch (error) {
         await stopIfStarted()

package/lib/cli.js

@@ -4,7 +4,7 @@ import { readFileSync, statSync } from 'node:fs'
 import { spawn } from 'node:child_process'
 import { dirname, join } from 'node:path'
 import { fileURLToPath } from 'node:url'
-import { runCheck } from './check.js'
+import { RESOURCE_ID_RE, runCheck } from './check.js'
 import { spawnDocker } from './docker.js'
 
 const PKG = join(dirname(fileURLToPath(import.meta.url)), '..', 'package.json')
@@ -39,6 +39,8 @@ export function helpText() {
         '  cuttlely logs [--follow]',
         '  cuttlely upgrade [--port <n>] [--tag <tag>] [--env-file <path>] [--no-open]',
         '  cuttlely check --flow <id> [--server <url>] [--timeout <seconds>] [--json <path>] [--fail-on-changed]',
+        '  cuttlely check --flow <id> [--server <url>] [--timeout <seconds>] [--json <path>] --dataset <id> [--evaluator <id> ...] [--threshold <0-100>]',
+        '  cuttlely check --flow <id> [--server <url>] [--timeout <seconds>] [--json <path>] --gate',
         '  cuttlely --help',
         '  cuttlely --version',
         '',
@@ -54,13 +56,15 @@ export function helpText() {
         '  upgrade   Pull the image and start it again.',
         '  check     Run the Publish check on a flow draft. Reads CUTTLELY_API_KEY.',
         '            The key needs chatflows:view and chatflows:update, or agentflows:view and agentflows:update.',
+        '            --dataset or --gate also needs datasets:view and evaluators:view.',
         '',
         'Exit codes for check:',
-        '  0  The check finished. No answer was Changed, or --fail-on-changed was off.',
-        '  1  --fail-on-changed was set and at least one answer was Changed.',
+        '  0  The check finished. No answer was Changed, or --fail-on-changed was off. A scored check exits 0 when the server verdict is passed.',
+        '  1  --fail-on-changed was set and at least one answer was Changed. A scored check exits 1 when the server verdict is failed.',
         '  2  The check timed out, the server returned an error, CUTTLELY_API_KEY was missing, the flow has no draft, or the command was used wrong.',
         'A check the server marks stopped or failed also exits 2. If the draft is saved, published, or discarded while the check runs, the command exits 2, because the answers may be for a draft that is already gone. After the check has started, a timeout, a 403, or any other failure asks the server to stop it. That stop is best effort: if it fails, the exit code is still 2.',
         'Cancelling the command stops the server-side check.',
+        'A scored check (--dataset or --gate) exits on the server verdict and does not score the answers itself. --gate reads the dataset, evaluators, and threshold saved on the flow, and reads the flow again after a pass. Exit 0 only when that second read still stores this check as passed with no block. Otherwise exit 2. Do not combine --gate with --dataset, or those flags with --fail-on-changed.',
         '',
         `Install guide: ${INSTALL_DOCS}`
     ].join('\n')
@@ -82,7 +86,19 @@ function parseTag(value, exitCode = 1) {
 
 // These options take the next argv token as their value. A following `check` is that value, not the command.
 // Any other previous token, including an unknown flag, leaves `check` as the command.
-const VALUE_OPTIONS = new Set(['--flow', '--server', '--timeout', '--json', '--port', '-p', '--tag', '--env-file'])
+const VALUE_OPTIONS = new Set([
+    '--flow',
+    '--server',
+    '--timeout',
+    '--json',
+    '--dataset',
+    '--evaluator',
+    '--threshold',
+    '--port',
+    '-p',
+    '--tag',
+    '--env-file'
+])
 
 export function isCheckCommand(argv) {
     return argv.some((item, index) => {
@@ -97,15 +113,27 @@ export function commandFailureCode(argv) {
     return isCheckCommand(argv) ? 2 : 1
 }
 
-const FLOW_ID_RE = /^[A-Za-z0-9][A-Za-z0-9_-]{0,127}$/
-
 function parseFlow(value) {
-    if (!value || !FLOW_ID_RE.test(value)) {
+    if (!value || !RESOURCE_ID_RE.test(value)) {
         throw new UsageError('--flow needs the flow id from the canvas address bar.', 2)
     }
     return value
 }
 
+function parseResourceId(value, flag) {
+    if (!value || !RESOURCE_ID_RE.test(value)) {
+        throw new UsageError(`${flag} needs ${flag === '--dataset' ? 'a dataset id' : 'an evaluator id'}.`, 2)
+    }
+    return value
+}
+
+function parseThreshold(value) {
+    if (!/^(?:0|[1-9]\d?|100)$/.test(value || '')) {
+        throw new UsageError(`--threshold must be a whole number from 0 to 100, got ${value || '(missing)'}.`, 2)
+    }
+    return Number(value)
+}
+
 function parseServer(value) {
     let url
     try {
@@ -137,11 +165,13 @@ function parseJsonPath(value) {
     return value
 }
 
-function takeValue(argv, index, flag) {
+function takeValue(argv, index, flag, allowSigned = false) {
     const inline = argv[index].indexOf('=')
     if (inline !== -1) return { value: argv[index].slice(inline + 1), next: index }
     const value = argv[index + 1]
-    if (!value || value.startsWith('-')) throw new UsageError(`${flag} needs a value. Run cuttlely --help.`, commandFailureCode(argv))
+    if (!value || (!allowSigned && value.startsWith('-'))) {
+        throw new UsageError(`${flag} needs a value. Run cuttlely --help.`, commandFailureCode(argv))
+    }
     return { value, next: index + 1 }
 }
 
@@ -159,7 +189,12 @@ export function parseArgs(argv) {
         server: null,
         timeoutSeconds: null,
         jsonPath: null,
-        failOnChanged: false
+        failOnChanged: false,
+        datasetId: null,
+        evaluatorIds: [],
+        threshold: null,
+        gate: false,
+        scored: false
     }
     const words = []
     const dockerFlags = []
@@ -212,6 +247,24 @@ export function parseArgs(argv) {
         } else if (arg === '--fail-on-changed') {
             opts.failOnChanged = true
             checkFlags.push('--fail-on-changed')
+        } else if (arg === '--dataset' || arg.startsWith('--dataset=')) {
+            const taken = takeValue(argv, i, '--dataset')
+            opts.datasetId = taken.value
+            i = taken.next
+            checkFlags.push('--dataset')
+        } else if (arg === '--evaluator' || arg.startsWith('--evaluator=')) {
+            const taken = takeValue(argv, i, '--evaluator')
+            opts.evaluatorIds.push(taken.value)
+            i = taken.next
+            checkFlags.push('--evaluator')
+        } else if (arg === '--threshold' || arg.startsWith('--threshold=')) {
+            const taken = takeValue(argv, i, '--threshold', true)
+            opts.threshold = taken.value
+            i = taken.next
+            checkFlags.push('--threshold')
+        } else if (arg === '--gate') {
+            opts.gate = true
+            checkFlags.push('--gate')
         } else if (arg.startsWith('-')) {
             throw new UsageError(`Unknown option ${arg}. Run cuttlely --help.`, commandFailureCode(argv))
         } else words.push(arg)
@@ -243,6 +296,40 @@ export function parseArgs(argv) {
             }
             // An empty --json= is still the flag. Skipping it would run the check and write nothing.
             if (checkFlags.includes('--json')) opts.jsonPath = parseJsonPath(opts.jsonPath)
+            if (opts.gate && checkFlags.includes('--dataset')) {
+                throw new UsageError('--gate uses the saved test set. Do not pass --dataset with it.', checkExit)
+            }
+            if (opts.gate && (checkFlags.includes('--evaluator') || checkFlags.includes('--threshold'))) {
+                throw new UsageError(
+                    '--gate uses the saved dataset, evaluators, and threshold. Do not pass --evaluator or --threshold with it.',
+                    checkExit
+                )
+            }
+            const scoredFlag = ['--dataset', '--evaluator', '--threshold', '--gate'].some((flag) => checkFlags.includes(flag))
+            if (opts.failOnChanged && scoredFlag) {
+                throw new UsageError('--fail-on-changed cannot be combined with --dataset, --evaluator, --threshold, or --gate.', checkExit)
+            }
+            if (checkFlags.includes('--dataset')) opts.datasetId = parseResourceId(opts.datasetId, '--dataset')
+            if (checkFlags.includes('--evaluator')) {
+                const seen = new Set()
+                const unique = []
+                for (const id of opts.evaluatorIds) {
+                    const parsed = parseResourceId(id, '--evaluator')
+                    if (seen.has(parsed)) continue
+                    seen.add(parsed)
+                    unique.push(parsed)
+                }
+                opts.evaluatorIds = unique
+            }
+            if (checkFlags.includes('--threshold')) opts.threshold = parseThreshold(opts.threshold)
+            if (
+                !opts.gate &&
+                !checkFlags.includes('--dataset') &&
+                (checkFlags.includes('--evaluator') || checkFlags.includes('--threshold'))
+            ) {
+                throw new UsageError('--evaluator and --threshold need --dataset.', checkExit)
+            }
+            opts.scored = Boolean(opts.gate || checkFlags.includes('--dataset'))
         }
     } else if (checkFlags.length) {
         throw new UsageError(`Unknown option ${checkFlags[0]}. Run cuttlely --help.`)
@@ -576,7 +663,12 @@ export async function main(argv, io = {}) {
                     server: opts.server,
                     timeoutSeconds: opts.timeoutSeconds,
                     jsonPath: opts.jsonPath,
-                    failOnChanged: opts.failOnChanged
+                    failOnChanged: opts.failOnChanged,
+                    scored: opts.scored,
+                    gate: opts.gate,
+                    datasetId: opts.datasetId,
+                    evaluatorIds: opts.evaluatorIds,
+                    threshold: opts.threshold
                 },
                 ctx
             )

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "cuttlely",
-    "version": "1.0.3",
+    "version": "1.2.0",
     "description": "Start Cuttlely in Docker. The open-source agent platform for applications.",
     "type": "module",
     "bin": {