@agent-native/core 0.84.0 → 0.84.1

package/corpus/README.md

@@ -28,4 +28,4 @@ rg -n "defineAction|useActionQuery" node_modules/@agent-native/core/corpus
 ## Generated Counts
 
 - core files: 2043
-- template files: 4877
+- template files: 4895

package/corpus/core/CHANGELOG.md

@@ -1,5 +1,11 @@
 # @agent-native/core
 
+## 0.84.1
+
+### Patch Changes
+
+- 87806ef: Report intentionally gated eval cases as skipped, keep skipped suites from setting up agent runners, and avoid action/engine setup for eval cases that fully short-circuit with custom run handlers.
+
 ## 0.84.0
 
 ### Minor Changes

package/corpus/core/docs/content/evals.mdx

@@ -90,13 +90,14 @@ export default defineEval({
 
 An eval passes only when **every** scorer meets the threshold. Key `defineEval` fields:
 
-| Field       | Type                  | Notes                                                         |
-| ----------- | --------------------- | ------------------------------------------------------------- |
-| `name`      | string                | Required. Shown in the report.                                |
-| `input`     | `{ prompt, history }` | Required `prompt`; optional prior `{ role, text }` turns.     |
-| `scorers`   | `Scorer[]`            | Required, at least one.                                       |
-| `threshold` | number `0..1`         | Per-scorer pass bar. Default `0.5`; overridable from the CLI. |
-| `run`       | function              | Optional override for custom setup (seed data, multi-turn).   |
+| Field        | Type                  | Notes                                                                                                           |
+| ------------ | --------------------- | --------------------------------------------------------------------------------------------------------------- |
+| `name`       | string                | Required. Shown in the report.                                                                                  |
+| `input`      | `{ prompt, history }` | Required `prompt`; optional prior `{ role, text }` turns.                                                       |
+| `scorers`    | `Scorer[]`            | Required, at least one, unless `skipReason` is set.                                                             |
+| `threshold`  | number `0..1`         | Per-scorer pass bar. Default `0.5`; overridable from the CLI.                                                   |
+| `run`        | function              | Optional override for custom setup (seed data, multi-turn).                                                     |
+| `skipReason` | string                | Optional reason to skip this eval without running the agent; useful for manual/nightly gates that need secrets. |
 
 The agent run handed to scorers is small and transport-agnostic:
 
@@ -225,11 +226,32 @@ The command discovers `**/*.eval.ts` and `evals/*.ts` under the current app, run
 
 Exit codes:
 
-| Code | Meaning                                                         |
-| ---- | --------------------------------------------------------------- |
-| `0`  | All evals passed — _or_ no eval files were found (CI-friendly). |
-| `1`  | At least one eval scored below threshold, or the suite errored. |
-| `2`  | Bad arguments (e.g. `--threshold` outside `[0, 1]`).            |
+| Code | Meaning                                                                          |
+| ---- | -------------------------------------------------------------------------------- |
+| `0`  | All evals passed, evals were skipped, or no eval files were found (CI-friendly). |
+| `1`  | At least one eval scored below threshold, or the suite errored.                  |
+| `2`  | Bad arguments (e.g. `--threshold` outside `[0, 1]`).                             |
+
+### Skipping gated evals {#skipping}
+
+Use `skipReason` when an eval suite is intentionally gated behind manual,
+nightly, or secret-backed execution:
+
+```ts
+export default defineEval({
+  name: "provider-backed source sync",
+  input: { prompt: "Refresh the source and summarize the changes." },
+  skipReason: process.env.SOURCE_SYNC_EVALS
+    ? undefined
+    : "Skipped because SOURCE_SYNC_EVALS is unset",
+  scorers: process.env.SOURCE_SYNC_EVALS ? [contains("updated")] : [],
+});
+```
+
+Skipped evals do not run the agent or scorers, do not fail the suite, and are
+reported distinctly with `status: "skipped"` and `skipReason` in JSON output.
+Use this for expensive or credential-dependent evals; keep cheap deterministic
+coverage in normal PR CI.
 
 ### As a CI deploy gate {#ci}
 

package/corpus/core/docs/content/locales/ar-SA/evals.mdx

@@ -231,6 +231,13 @@ agent-native eval --threshold 0.8    # override every eval's pass threshold (0..
 | `1`   | تم تسجيل تقييم واحد على الأقل أقل من الحد الأدنى، أو حدث خطأ في المجموعة.           |
 | `2`   | وسيطات غير صالحة (على سبيل المثال، `--threshold` خارج `[0, 1]`).                    |
 
+### تخطي التقييمات المحكومة {#skipping}
+
+استخدم `skipReason` عندما تكون مجموعة التقييمات مخصصة عمدا للتشغيل اليدوي أو
+الليلي أو المعتمد على الأسرار. لا تشغل التقييمات المتخطاة الوكيل أو أدوات
+التسجيل، ولا تفشل المجموعة، وتظهر في JSON مع `status: "skipped"` و
+`skipReason`. أبق التغطية الحتمية الرخيصة في CI العادي لطلبات السحب.
+
 ### كبوابة نشر CI {#ci}
 
 قم بإضافته إلى المسار الذي يتم تشغيله قبل النشر:

package/corpus/core/docs/content/locales/de-DE/evals.mdx

@@ -231,6 +231,14 @@ Exit-Codes:
 | `1`  | Mindestens eine Bewertung lag unter dem Schwellenwert oder die Suite ist fehlerhaft.                |
 | `2`  | Ungültige Argumente (z. B. `--threshold` außerhalb von `[0, 1]`).                                   |
 
+### Gated Evals überspringen {#skipping}
+
+Verwenden Sie `skipReason`, wenn eine Eval-Suite absichtlich nur manuell,
+nachts oder mit Secrets ausgeführt werden soll. Übersprungene Evals führen
+weder Agent noch Scorer aus, lassen die Suite nicht fehlschlagen und erscheinen
+in der JSON-Ausgabe mit `status: "skipped"` und `skipReason`. Günstige
+deterministische Abdeckung bleibt im normalen PR-CI.
+
 ### Als CI-Deploy-Gate {#ci}
 
 Fügen Sie es der Pipeline hinzu, die vor einer Bereitstellung ausgeführt wird:

package/corpus/core/docs/content/locales/es-ES/evals.mdx

@@ -231,6 +231,14 @@ Códigos de salida:
 | `1`    | Al menos una evaluación obtuvo una puntuación inferior al umbral o la suite tuvo un error.                |
 | `2`    | Malos argumentos (por ejemplo, `--threshold` fuera de `[0, 1]`).                                          |
 
+### Omitir evaluaciones gated {#skipping}
+
+Use `skipReason` cuando una suite de evaluaciones esté reservada
+intencionalmente para ejecución manual, nocturna o con secretos. Las
+evaluaciones omitidas no ejecutan el agente ni los scorers, no hacen fallar la
+suite y aparecen en JSON con `status: "skipped"` y `skipReason`. Mantenga la
+cobertura determinista barata en el CI normal de PR.
+
 ### Como puerta de implementación de CI {#ci}
 
 Agréguelo al proceso que se ejecuta antes de una implementación:

package/corpus/core/docs/content/locales/fr-FR/evals.mdx

@@ -231,6 +231,14 @@ Codes de sortie :
 | `1`  | Au moins une évaluation a obtenu un score inférieur au seuil, ou la suite a commis une erreur.      |
 | `2`  | Mauvais arguments (par exemple `--threshold` en dehors de `[0, 1]`).                                |
 
+### Ignorer les évaluations gated {#skipping}
+
+Utilisez `skipReason` lorsqu'une suite d'évaluations est volontairement réservée
+à une exécution manuelle, nocturne ou dépendante de secrets. Les évaluations
+ignorées n'exécutent ni l'agent ni les scorers, ne font pas échouer la suite, et
+apparaissent dans la sortie JSON avec `status: "skipped"` et `skipReason`.
+Gardez la couverture déterministe peu coûteuse dans le CI normal des PR.
+
 ### En tant que porte de déploiement CI {#ci}
 
 Ajoutez-le au pipeline qui s'exécute avant un déploiement :

package/corpus/core/docs/content/locales/hi-IN/evals.mdx

@@ -229,6 +229,14 @@ agent-native eval --threshold 0.8    # override every eval's pass threshold (0..
 | `1` | कम से कम एक ईवल ने सीमा से नीचे स्कोर किया, या सुइट में त्रुटि हुई। |
 | `2` | खराब तर्क (उदाहरण के लिए `[0, 1]` के बाहर `--threshold`)।           |
 
+### gated evals छोड़ना {#skipping}
+
+जब कोई eval suite केवल manual, nightly, या secret-backed execution के लिए हो,
+तो `skipReason` का उपयोग करें। छोड़े गए evals agent या scorers नहीं चलाते,
+suite को fail नहीं करते, और JSON output में `status: "skipped"` और
+`skipReason` के साथ दिखते हैं। सस्ती deterministic coverage को सामान्य PR CI
+में रखें।
+
 ### सीआई परिनियोजन गेट के रूप में {#ci}
 
 इसे उस पाइपलाइन में जोड़ें जो परिनियोजन से पहले चलती है:

package/corpus/core/docs/content/locales/ja-JP/evals.mdx

@@ -231,6 +231,13 @@ agent-native eval --threshold 0.8    # override every eval's pass threshold (0..
 | `1`    | 少なくとも 1 つの評価スコアがしきい値を下回ったか、スイートでエラーが発生しました。             |
 | `2`    | 不正な引数 (例: `[0, 1]` の外の `--threshold`)。                                                |
 
+### gated eval をスキップする {#skipping}
+
+手動、夜間、または secret が必要な eval スイートには `skipReason` を使い
+ます。スキップされた eval はエージェントや scorer を実行せず、スイート
+を失敗させず、JSON 出力では `status: "skipped"` と `skipReason` とし
+て表示されます。安価で決定的なカバレッジは通常の PR CI に残します。
+
 ### CI デプロイゲートとして {#ci}
 
 デプロイ前に実行されるパイプラインに追加します。

package/corpus/core/docs/content/locales/ko-KR/evals.mdx

@@ -231,6 +231,13 @@ agent-native eval --threshold 0.8    # override every eval's pass threshold (0..
 | `1`  | 최소 하나의 평가가 임계값 미만의 점수를 얻었거나 제품군에 오류가 발생했습니다.   |
 | `2`  | 잘못된 인수(예: `[0, 1]` 외부의 `--threshold`).                                  |
 
+### 게이트된 평가 건너뛰기 {#skipping}
+
+수동, 야간 또는 시크릿이 필요한 평가 스위트에는 `skipReason`를 사용하세요.
+건너뛴 평가는 에이전트나 scorer를 실행하지 않고, 스위트를 실패시키지
+않으며, JSON 출력에 `status: "skipped"` 및 `skipReason`로 표시됩니다.
+저렴한 결정적 커버리지는 일반 PR CI에 유지하세요.
+
 ### CI 배포 게이트로 {#ci}
 
 배포 전에 실행되는 파이프라인에 추가하세요:

package/corpus/core/docs/content/locales/pt-BR/evals.mdx

@@ -231,6 +231,14 @@ Códigos de saída:
 | `1`    | Pelo menos uma avaliação com pontuação abaixo do limite ou o conjunto apresentou erro.                     |
 | `2`    | Argumentos incorretos (por exemplo, `--threshold` fora de `[0, 1]`).                                       |
 
+### Ignorar evals gated {#skipping}
+
+Use `skipReason` quando uma suíte de evals for intencionalmente reservada para
+execução manual, noturna ou dependente de secrets. Evals ignorados não executam
+o agente nem os scorers, não fazem a suíte falhar e aparecem no JSON com
+`status: "skipped"` e `skipReason`. Mantenha cobertura determinística barata no
+CI normal de PR.
+
 ### Como porta de implantação de CI {#ci}
 
 Adicione-o ao pipeline executado antes da implantação:

package/corpus/core/docs/content/locales/zh-CN/evals.mdx

@@ -229,6 +229,13 @@ agent-native eval --threshold 0.8    # override every eval's pass threshold (0..
 | `1`  | 至少有一项评估得分低于阈值，或者套件出错。            |
 | `2`  | 错误的参数（例如 `--threshold` 位于 `[0, 1]` 之外）。 |
 
+### 跳过受控评估 {#skipping}
+
+当某个评估套件只应在手动、夜间或具备密钥的环境中运行时，请使用
+`skipReason`。被跳过的评估不会运行代理或 scorer，不会让套件失败，并会
+在 JSON 输出中显示 `status: "skipped"` 和 `skipReason`。廉价的确定性覆
+盖应保留在普通 PR CI 中。
+
 ### 作为 CI 部署门 {#ci}
 
 将其添加到部署之前运行的管道：

package/corpus/core/docs/content/locales/zh-TW/evals.mdx

@@ -229,6 +229,13 @@ agent-native eval --threshold 0.8    # override every eval's pass threshold (0..
 | `1`    | 至少有一項評估得分低於閾值，或者套件出錯。            |
 | `2`    | 錯誤的參數（例如 `--threshold` 位於 `[0, 1]` 之外）。 |
 
+### 跳過受控評估 {#skipping}
+
+當某個評估套件只應在手動、夜間或具備密鑰的環境中執行時，請使用
+`skipReason`。被跳過的評估不會執行代理或 scorer，不會讓套件失敗，並會
+在 JSON 輸出中顯示 `status: "skipped"` 和 `skipReason`。廉價的確定性覆
+蓋應保留在普通 PR CI 中。
+
 ### 作為 CI 部署門 {#ci}
 
 將其新增到部署之前執行的管道：

package/corpus/core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-native/core",
-  "version": "0.84.0",
+  "version": "0.84.1",
   "description": "Framework for agent-native application development — where AI agents and UI share SQL state, actions, and context",
   "homepage": "https://github.com/BuilderIO/agent-native#readme",
   "bugs": {

package/corpus/core/src/eval/define-eval.ts

@@ -35,7 +35,10 @@ export function defineEval(spec: Eval): Eval {
   if (!spec.input || typeof spec.input.prompt !== "string") {
     throw new Error(`defineEval("${spec.name}"): \`input.prompt\` is required`);
   }
-  if (!Array.isArray(spec.scorers) || spec.scorers.length === 0) {
+  if (
+    (!Array.isArray(spec.scorers) || spec.scorers.length === 0) &&
+    !spec.skipReason
+  ) {
     throw new Error(
       `defineEval("${spec.name}"): at least one scorer is required`,
     );

package/corpus/core/src/eval/report.ts

@@ -27,13 +27,19 @@ export function formatReport(report: EvalRunReport): string {
   lines.push("  ─────");
 
   for (const row of report.results) {
-    const mark = row.passed ? "✓" : "✗";
+    const mark = row.status === "skipped" ? "-" : row.passed ? "✓" : "✗";
     lines.push("");
-    lines.push(
-      `  ${mark} ${row.eval}  (avg ${pct(row.avgScore)}, threshold ${pct(
-        row.threshold,
-      )})`,
-    );
+    if (row.status === "skipped") {
+      lines.push(`  ${mark} ${row.eval}  (skipped)`);
+      lines.push(`      reason: ${row.skipReason ?? "No reason provided"}`);
+      continue;
+    } else {
+      lines.push(
+        `  ${mark} ${row.eval}  (avg ${pct(row.avgScore)}, threshold ${pct(
+          row.threshold,
+        )})`,
+      );
+    }
     if (row.error) {
       lines.push(`      ⚠ run error: ${row.error}`);
     }
@@ -50,9 +56,18 @@ export function formatReport(report: EvalRunReport): string {
 
   lines.push("");
   lines.push("  ─────");
-  const verdict = report.failed === 0 ? "PASS" : "FAIL";
+  const skipped = report.skipped ?? 0;
+  const executedTotal = report.total - skipped;
+  const executedPassed = report.passed - skipped;
+  const verdict =
+    executedTotal === 0 && skipped > 0
+      ? "SKIPPED"
+      : report.failed === 0
+        ? "PASS"
+        : "FAIL";
   lines.push(
-    `  ${verdict}: ${report.passed}/${report.total} evals passed` +
+    `  ${verdict}: ${executedPassed}/${executedTotal} evals passed` +
+      (skipped > 0 ? `, ${skipped} skipped` : "") +
       (report.failed > 0 ? `, ${report.failed} below threshold` : ""),
   );
   lines.push("");

package/corpus/core/src/eval/runner.ts

@@ -78,6 +78,20 @@ export async function scoreEval(
   const threshold =
     opts.thresholdOverride ?? evalCase.threshold ?? DEFAULT_EVAL_THRESHOLD;
 
+  if (evalCase.skipReason) {
+    return {
+      eval: evalCase.name,
+      threshold,
+      scores: [],
+      status: "skipped",
+      skipReason: evalCase.skipReason,
+      // Keep the legacy boolean gate friendly: skipped rows do not fail CI.
+      passed: true,
+      avgScore: 0,
+      durationMs: 0,
+    };
+  }
+
   let run: AgentRunOutput;
   if (evalCase.run) {
     run = await evalCase.run({
@@ -104,6 +118,7 @@ export async function scoreEval(
     scores,
     // A run that errored, or any sub-threshold scorer, fails the case.
     passed: run.ok && scores.every((s) => s.passed),
+    status: run.ok && scores.every((s) => s.passed) ? "passed" : "failed",
     avgScore,
     durationMs: run.durationMs,
     error: run.ok ? undefined : run.error,
@@ -120,14 +135,18 @@ export async function runEvals(
   for (const evalCase of evals) {
     const row = await scoreEval(evalCase, runner, opts);
     results.push(row);
-    if (opts.persist) await persistEvalRow(row).catch(() => {});
+    if (opts.persist && row.status !== "skipped") {
+      await persistEvalRow(row).catch(() => {});
+    }
   }
 
   const passed = results.filter((r) => r.passed).length;
+  const skipped = results.filter((r) => r.status === "skipped").length;
   return {
     total: results.length,
     passed,
-    failed: results.length - passed,
+    failed: results.filter((r) => r.status !== "skipped" && !r.passed).length,
+    skipped,
     results,
   };
 }
@@ -301,12 +320,15 @@ export async function runEvalSuite(
     evals = loaded.evals;
   }
 
+  const needsRunner = evals.some((evalCase) => !evalCase.skipReason);
   const runner =
     opts.runner ??
-    (await createAgentRunner({
-      actions: opts.actions ?? (await discoverActions(cwd)),
-      systemPrompt: opts.systemPrompt,
-    }));
+    (needsRunner
+      ? await createAgentRunner({
+          actions: opts.actions ?? (await discoverActions(cwd)),
+          systemPrompt: opts.systemPrompt,
+        })
+      : createInertRunner());
 
   const report = await runEvals(evals, runner, {
     thresholdOverride: opts.thresholdOverride,
@@ -332,3 +354,16 @@ async function discoverActions(
     return {};
   }
 }
+
+function createInertRunner(): AgentRunner {
+  return {
+    engine: {} as AgentRunner["engine"],
+    model: "inert",
+    async runAgent() {
+      throw new Error("Eval unexpectedly requested the agent runner");
+    },
+    analyzeContext() {
+      throw new Error("Eval unexpectedly requested analyze context");
+    },
+  };
+}

package/corpus/core/src/eval/types.ts

@@ -133,6 +133,12 @@ export interface EvalRunContext {
 export interface Eval {
   name: string;
   input: EvalInput;
+  /**
+   * Skip this case without running the agent or scorers. Use for opt-in
+   * suites that need secrets, live provider credentials, or manual/nightly
+   * gates. Skipped rows are reported distinctly and do not fail the suite.
+   */
+  skipReason?: string;
   /**
    * Optional override for how the agent is run for this case. Defaults to the
    * runner's headless `runAgent`. Use this to do custom setup (seed data,
@@ -159,6 +165,10 @@ export interface EvalResultRow {
   eval: string;
   threshold: number;
   scores: ScorerResult[];
+  /** Explicit row status for reports. Older consumers can still read `passed`. */
+  status?: "passed" | "failed" | "skipped";
+  /** Human-readable reason when `status` is `skipped`. */
+  skipReason?: string;
   /** True only when every scorer passed. */
   passed: boolean;
   /** Mean of the scorer scores, for at-a-glance ranking. */
@@ -173,5 +183,6 @@ export interface EvalRunReport {
   total: number;
   passed: number;
   failed: number;
+  skipped?: number;
   results: EvalResultRow[];
 }

package/corpus/templates/content/AGENTS.md

@@ -121,7 +121,9 @@ cd templates/content && pnpm action <name> [args]
 | `validate-builder-source-execution`         | `--databaseId <id>` or `--documentId <id> --changeSetId <id> [--idempotencyKey <key>]`                                                                         | Validate/replay a prepared Builder execution gate locally as a dry run; never calls Builder                                             |
 | `execute-builder-source-execution`          | `--databaseId <id>` or `--documentId <id> --changeSetId <id> [--idempotencyKey <key>] [--pushModeConfirmation autosave\|draft\|publish]`                       | Execute a guarded live Builder write only when approved, validated, enabled, idempotent, and targeting `agent-native-blog-article-test` |
 | `add-database-item`                         | `--databaseId <id> [--title <text>] [--propertyValues <json>]`                                                                                                 | Add a page row to a database, optionally seeding property values                                                                        |
-| `duplicate-database-item`                   | `--itemId <id>` or `--documentId <id> [--title <text>]`                                                                                                        | Duplicate a database row page and its stored property values                                                                            |
+| `duplicate-database-item`                   | `--itemId <id>` or `--documentId <id> [--title <text>]`                                                                                                        | Duplicate exactly one database row page and its stored property values; for two or more rows, use `duplicate-database-items` once       |
+| `duplicate-database-items`                  | `--databaseId <id>` or `--documentId <databaseDocumentId> --itemIds <json-array>` or `--documentIds <json-array>`                                              | Preferred multi-row duplicate action; duplicate multiple database row pages atomically and return ordered duplicate item/document IDs   |
+| `delete-database-items`                     | `--databaseId <id>` or `--documentId <databaseDocumentId> --itemIds <json-array>` or `--documentIds <json-array>`                                              | Preferred multi-row delete action; delete multiple database row pages atomically while preserving `delete-document` admin semantics     |
 | `move-database-item`                        | `--itemId <id>` or `--documentId <id> --position <number>`                                                                                                     | Move a database row page to a new zero-based table position                                                                             |
 | `update-content-database-view`              | `--databaseId <id> --viewConfig <json>`                                                                                                                        | Persist database views, sorts, filters, hidden properties, and view settings                                                            |
 | `list-document-properties`                  | `--documentId <id> [--format json]`                                                                                                                            | List Notion-style property definitions and values for a document                                                                        |
@@ -568,11 +570,16 @@ multi-select, users can add a new board group from the board itself; this
 appends a new option to the grouped property definition.
 Use
 `create-content-database`, `get-content-database`,
-`add-database-item`, `duplicate-database-item`, `move-database-item`,
+`add-database-item`, `duplicate-database-item`, `duplicate-database-items`,
+`delete-database-items`, `move-database-item`,
 `update-content-database-view`, `list-document-properties`,
 `configure-document-property`, `set-document-property`,
 `duplicate-document-property`, and `delete-document-property`; do not edit
 property rows or view config via raw SQL when an action can do it.
+When targeting more than one database row, call `duplicate-database-items` or
+`delete-database-items` once with a native JSON array of `itemIds` or
+`documentIds`. Do not loop `duplicate-database-item` or `delete-document` for
+multi-row duplicate/delete requests.
 
 ## UI Components