@sanity/ailf-studio 0.1.17 → 0.1.18

package/README.md

@@ -57,6 +57,28 @@ This registers:
 - The `ailf.evalRequest` document type (evaluation request triggers)
 - The **AI Literacy Framework** dashboard tool in the Studio sidebar
 
+#### Document Actions
+
+The plugin registers two document actions for triggering evaluations directly
+from Studio:
+
+- **Run Task Eval** (on `ailf.task` documents) — evaluates a single task. Click
+  ▶ in the document actions menu to run all test cases for the task against the
+  current documentation. The button shows the score when complete (~10–15 min).
+  No secrets needed — it creates an `ailf.evalRequest` document that a
+  server-side webhook dispatches to the pipeline.
+
+- **Run AI Eval** (on content releases) — evaluates all tasks affected by a
+  content release. Appears in the release detail page's action bar. Answers "did
+  my doc changes help or hurt AI agent performance?" Shows score and delta vs
+  baseline when complete.
+
+Both actions use the same mechanism: they create an `ailf.evalRequest` document
+in the Content Lake with `status: "pending"`. A server-side Sanity webhook picks
+up the document and dispatches the pipeline via GitHub Actions. The Studio
+component polls for the resulting report and updates the button label with the
+score.
+
 ### 3. Alternative: tool-only installation
 
 If you only want the dashboard tool without the document schemas (e.g., the
@@ -103,6 +125,21 @@ export default defineConfig({
 })
 ```
 
+## Task Execution Workflows
+
+Tasks created in Studio are automatically included in every pipeline run — no
+registration step needed. There are four ways to execute tasks:
+
+| Method                         | Trigger                                                           | Scope                         |
+| ------------------------------ | ----------------------------------------------------------------- | ----------------------------- |
+| **Run Task Eval** action       | Click ▶ on any `ailf.task` document                               | Single task                   |
+| **Run AI Eval** release action | Click button on a content release page                            | Tasks affected by the release |
+| **CLI pipeline**               | `ailf pipeline` (with optional `--area`/`--task`/`--tag` filters) | All enabled tasks             |
+| **Scheduled pipeline**         | GitHub Actions cron (daily + weekly)                              | All enabled tasks             |
+
+See [CONTRIBUTING_TASKS.md](../../docs/CONTRIBUTING_TASKS.md#running-your-task)
+for the full execution flow and details on each method.
+
 ## Dashboard Views
 
 The plugin provides three tab views plus a detail drill-down, accessible from

package/dist/index.d.ts

@@ -436,10 +436,24 @@ declare const reportSchema: {
  * - A gold-standard implementation (reference solution)
  * - When/how the task runs (execution controls)
  *
+ * ## Execution paths
+ *
+ * Published tasks are automatically discovered by the pipeline — no
+ * registration step needed. There are four ways to execute a task:
+ *
+ * 1. **Run Task Eval** — click ▶ on any ailf.task document in Studio.
+ *    Creates an ailf.evalRequest scoped to this task. Webhook dispatches
+ *    the pipeline; score appears inline when complete (~10–15 min).
+ * 2. **Run AI Eval** — click on a content release page. Auto-scopes to
+ *    tasks whose canonical docs are in the release.
+ * 3. **CLI** — `ailf pipeline --task <id>` or `ailf pipeline --area <area>`.
+ * 4. **Scheduled** — GitHub Actions cron (daily baseline, weekly full).
+ *
  * Tasks can be authored natively in Studio or mirrored from external
  * repositories. Mirrored tasks have a read-only `origin` block that
  * tracks their source repo provenance.
  *
+ * @see docs/CONTRIBUTING_TASKS.md#running-your-task — full execution guide
  * @see docs/design-docs/tasks-as-content.md
  * @see docs/design-docs/tasks-as-content.md#decision-8-domain-specific-assertion-types-not-a-promptfoo-subset
  */

package/dist/index.js

@@ -2481,6 +2481,26 @@ var taskSchema = defineType5({
     }),
     // -----------------------------------------------------------------------
     // Execution controls
+    //
+    // Pipeline support status:
+    //   - `enabled`  → ✅ used in GROQ filter (execution.enabled != false)
+    //   - `blocking` → 🔮 forward-looking — schema reserved, not yet enforced
+    //                   by the pipeline. PR check workflow reports scores but
+    //                   does not gate merges on this flag.
+    //   - `trigger.branches`, `trigger.paths` → ✅ used by repo-based task
+    //                   threshold evaluator (RepoThresholdEvaluator)
+    //   - `trigger.labels`, `trigger.schedule` → 🔮 forward-looking — schema
+    //                   reserved, not yet read by the pipeline GROQ query or
+    //                   mapped to TaskDefinition.
+    //   - `threshold.score` → ✅ used by repo-based threshold evaluator
+    //   - `threshold.dimensions` → 🔮 forward-looking — schema reserved, not
+    //                   yet read by the pipeline GROQ query or mapped to
+    //                   TaskDefinition.
+    //
+    // When implementing a forward-looking field, update the GROQ query in
+    // content-lake-task-source.ts, the ContentLakeTask interface, the
+    // mapToTaskDefinition() mapping, and (if pipeline-relevant) the
+    // TaskDefinition port type in @sanity/ailf-core.
     // -----------------------------------------------------------------------
     defineField5({
       description: "Controls when and how this task runs in the pipeline. Tasks without execution controls use defaults: enabled, non-blocking, no filters, no threshold.",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sanity/ailf-studio",
-  "version": "0.1.17",
+  "version": "0.1.18",
   "description": "AI Literacy Framework — Sanity Studio dashboard plugin",
   "type": "module",
   "license": "MIT",