@desplega.ai/agent-swarm 1.172.0 → 2.0.0

package/templates/skills/qa-loom/content.md

@@ -0,0 +1,187 @@
+# QA loom
+
+A QA loom is a short screen recording of a real UI QA pass, with a voice-over and small one-line subtitles, embedded in the PR. The narrated mp4 plays inline on GitHub with audio. An optional subtitled GIF gives a silent preview.
+
+Two bundled files ship next to this skill:
+
+- `run-qa.sh` records the QA pass with `agent-browser` and writes `steps.json`.
+- `build-loom.mjs` turns `steps.json` and the raw recording into `qa-loom.mp4`.
+
+Copy both into your working directory before you edit them.
+
+## When to use
+
+- On every PR that changes visible UI, and on every UI QA pass.
+- Run your normal QA with the `qa` skill. Drive the browser with the `agent-browser` skill. This skill adds recording, narration and delivery on top.
+- Skip it when a PR has no visible UI change.
+
+## Requirements
+
+Check these first. If one is missing, use the fallback in section 7.
+
+| Requirement | Check |
+|---|---|
+| `ELEVENLABS_API_KEY` env var | `test -n "$ELEVENLABS_API_KEY"` (never echo the value) |
+| `agent-browser` with recording | `command -v agent-browser`; on errors, `agent-browser doctor` |
+| `ffmpeg` built with libass | `ffmpeg -hide_banner -filters 2>/dev/null \| grep -w ass` |
+| Node 18+ or Bun | `node --version` (the script needs global `fetch` and top-level await) |
+| `jq` and GNU `date` | `jq --version`; `date +%N` prints digits, not `N` |
+| A subtitle font | `fc-list \| grep -i "DejaVu Sans"`, or set `FONT` to an installed font |
+| `GITHUB_TOKEN` for delivery | a token that can upload user attachments to the repo (section 6) |
+
+Portability gotchas:
+
+- Some ffmpeg builds ship without `ffprobe`. Probe with `ffmpeg -hide_banner -i f.mp4 2>&1 | grep -E "Duration|Stream"`.
+- Some ffmpeg builds have no `drawtext` filter. Burn subtitles with the libass `ass` filter, as `build-loom.mjs` does.
+- libass ignores the alpha byte of `BackColour` for `BorderStyle=3`. The subtitle box is solid dark grey, not translucent.
+- Minimal containers may lack `bc`. Do time arithmetic in `jq`, as `run-qa.sh` does.
+
+## Style rules
+
+- **Speed 1.33x** on the raw footage (`SPEED=1.33`). 2x is too fast to follow.
+- **One short sentence per step**, present tense, describing what the viewer sees: "The editor opens a shared file." Aim for 6 to 12 words.
+- **Subtitles: one line, small, bottom centre.** 38 px on a 1080p frame, dark box, `MarginV 30`. Cues hold at most 6 words, split on punctuation, with no orphan cue under 3 words.
+- **Voice and subtitles start when the screen renders**, not when the click fires. That is why the step log carries `visualAt`.
+- **Letterbox, never stretch.** Mobile-viewport steps are fitted into 1920x1080 with padding.
+- **Keep it short.** 4 to 6 steps that prove the change, about 20 to 30 seconds final.
+
+## Context hygiene
+
+The pipeline handles large binary payloads. Keep them out of your context:
+
+- Run every ffmpeg call with `-loglevel error`, or send its stderr to a file and `tail -3` it only on failure.
+- Never print an ElevenLabs response. `audio_base64` is hundreds of KB. `build-loom.mjs` writes it straight to disk.
+- Never `cat` or read an mp4 or GIF. To look at the result, extract one PNG frame and view that.
+
+## 1. Record the QA pass with a step log
+
+Each step logs:
+
+- `start`: just before the action fires (`act`).
+- `visualAt`: after the resulting screen has rendered, following an `agent-browser wait <selector>` (`step`).
+- `verdict` and one `narration` sentence.
+- `end`: the next step's `start`. The last step ends when recording stops.
+
+All times are seconds from recording start. Replace the example steps in `run-qa.sh` with your QA steps, then run it:
+
+```bash
+BASE_URL=http://localhost:5274 OUT=./loom bash run-qa.sh
+```
+
+Notes:
+
+- Log in and seed data **before** `record start`, so the loom shows only the change.
+- For a mobile check, run `agent-browser set viewport 390 844` inside a step. The build letterboxes it.
+- After each step, wait about 800 ms so the viewer sees the result before the next action.
+- Take your normal QA screenshots as well. They are the fallback.
+
+## 2. Build the narrated mp4
+
+`build-loom.mjs` reads `steps.json` and the raw recording. For each step it:
+
+1. Calls ElevenLabs `POST /v1/text-to-speech/{voice}/with-timestamps` once and caches the mp3 and character alignment under `work/`. A re-run after a style tweak costs no characters.
+2. Cuts the step's segment, speeds it up, and letterboxes it to 1920x1080 at 30 fps.
+3. Freezes the last frame when the narration outlasts the segment.
+4. Builds subtitle cues from the same character alignment, offset by `visualAt`, so voice and subtitles cannot drift.
+
+Then it concatenates the segments with the concat demuxer, delays each step's audio to where its screen renders, mixes with `amix normalize=0`, burns the ASS subtitles, and writes `qa-loom.mp4` and `qa-loom.ass`.
+
+```bash
+node build-loom.mjs loom/steps.json loom/raw.webm loom/out > loom/build.log 2>&1 || tail -5 loom/build.log
+cat loom/build.log   # one JSON line: {"durationS":..,"steps":..,"cues":..,"chars":..}
+```
+
+Optional env: `SPEED` (default `1.33`), `VOICE` (default `21m00Tcm4TlvDq8ikWAM`, an ElevenLabs premade voice), `MODEL` (default `eleven_turbo_v2_5`), `FONT` (default `DejaVu Sans`).
+
+## 3. Check the result
+
+Look at one frame per cue, as PNG:
+
+```bash
+cd loom/out
+ffmpeg -hide_banner -i qa-loom.mp4 2>&1 | grep -E "Duration|Stream"          # 1920x1080, 30 fps, h264 + aac
+grep Dialogue qa-loom.ass                                                     # one short line per cue
+ffmpeg -hide_banner -i qa-loom.mp4 -vn -af volumedetect -f null - 2>&1 | grep mean_volume   # about -91 dB means silent
+ffmpeg -y -loglevel error -ss 2 -i qa-loom.mp4 -frames:v 1 check-2s.png       # view it: the subtitle matches the screen
+```
+
+If the loom feels slow or fast, change `SPEED` and re-run. TTS is cached, so the re-run is free.
+
+## 4. Optional GIF preview
+
+GitHub plays an mp4 inline only when it is a GitHub user attachment (section 6). The GIF is an optional silent preview with the subtitles already burned in.
+
+```bash
+ffmpeg -y -loglevel error -i qa-loom.mp4 \
+  -vf "fps=10,scale=960:-1:flags=lanczos,split[a][b];[a]palettegen=stats_mode=diff[p];[b][p]paletteuse=dither=bayer:bayer_scale=5:diff_mode=rectangle" \
+  -loop 0 qa-loom.gif 2>gif.err || tail -3 gif.err
+ffmpeg -hide_banner -i qa-loom.gif 2>&1 | grep -E "Duration|Stream"   # 960x540, 10 fps
+head -c 6 qa-loom.gif; echo                                           # GIF89a
+ffmpeg -y -loglevel error -ss 5 -i qa-loom.gif -frames:v 1 gif-check.png   # view it: subtitle readable
+```
+
+At these settings a GIF costs about 30 KB per second of loom. Keep it under 5 MB. If it is bigger, drop to `fps=8`, then to `scale=720:-1`. Below 720 px wide the subtitles become unreadable.
+
+## 5. Keep the originals
+
+Store `qa-loom.mp4` and `qa-loom.gif` wherever your team keeps QA evidence, next to the QA screenshots, under the same topic name. The GitHub attachment is the embed; your store is the durable original.
+
+## 6. Attach to the PR
+
+Upload both files to GitHub as user attachments. This uses GitHub's user-attachments upload endpoint, the one the web editor uses. It is not part of the documented REST API.
+
+```bash
+REPO=<owner>/<repo>
+REPO_ID=$(gh api "repos/$REPO" --jq .id)
+attach() {  # $1 local file, $2 content type; prints https://github.com/user-attachments/assets/<uuid>
+  q=$(jq -rn --arg name "$(basename "$1")" --arg content_type "$2" --arg repository_id "$REPO_ID" \
+    '$ARGS.named | to_entries | map("\(.key)=\(.value | @uri)") | join("&")')
+  curl --fail-with-body -sS -X POST "https://uploads.github.com/user-attachments/assets?$q" \
+    -H "Authorization: Bearer $GITHUB_TOKEN" -H 'Content-Type: application/octet-stream' \
+    --data-binary "@$1" | jq -er .url
+}
+MP4_URL=$(attach qa-loom.mp4 video/mp4)
+GIF_URL=$(attach qa-loom.gif image/gif)
+echo "$MP4_URL $GIF_URL"
+```
+
+Verification rules:
+
+- Both URLs must match `https://github.com/user-attachments/assets/<uuid>`.
+- A 401 means the token type is not accepted (GitHub App installation tokens are rejected; a personal access token works). A 404 means a wrong repository ID or a token without write access to the repo.
+- Do not embed presigned storage URLs. A presigned mp4 usually downloads instead of playing, and it expires.
+
+Put the loom in the PR body's `## Before / after` section. Write the section to a file:
+
+```bash
+cat > loom-section.md <<EOF
+QA loom: <one-line summary>. <n> steps, <verdicts>.
+
+$MP4_URL
+
+![QA loom preview (silent, subtitled)]($GIF_URL)
+EOF
+```
+
+`$MP4_URL` must sit on its own line, so GitHub renders a player with audio. The GIF line is optional. Merge the section into the body (`gh pr view <n> --json body`, then `gh pr edit <n> --body-file <file>`) and leave every other line unchanged.
+
+Then open the PR and confirm that the mp4 player and the GIF render. A successful upload response does not prove that the embed renders.
+
+## 7. Fallback: never block the PR
+
+If ElevenLabs fails (401, 429, quota, or a 5xx after one retry with backoff), ffmpeg fails, or a requirement is missing:
+
+1. Post the QA screenshots in the PR as GitHub user attachments.
+2. Add one line to the PR that says why there is no loom, for example "QA loom skipped: ElevenLabs 429 quota exceeded."
+3. Ship the PR. The loom is evidence, never a merge gate.
+
+A 4xx validation error is deterministic. Do not loop on it. Fix the input or fall back.
+
+## Deliverable checklist
+
+- [ ] `steps.json` has `start`, `visualAt`, `end`, `verdict` and `narration` for every step
+- [ ] `qa-loom.mp4` is 1920x1080 at 30 fps with h264 and aac, is not silent, and shows one short subtitle line at the bottom
+- [ ] `qa-loom.gif`, if made, is 960 px wide at 10 fps, under 5 MB, and starts with `GIF89a`
+- [ ] The originals are stored with the QA screenshots
+- [ ] The PR body's `## Before / after` section has `$MP4_URL` on its own line, and every URL is `https://github.com/user-attachments/assets/<uuid>`
+- [ ] You opened the PR and saw the player render

package/templates/skills/qa-loom/files/build-loom.mjs

@@ -0,0 +1,221 @@
+// build-loom.mjs: steps.json + raw recording -> narrated qa-loom.mp4 (+ qa-loom.ass, work/).
+// Usage: node build-loom.mjs <steps.json> <raw.webm> <outDir>   (bun works too)
+// Env: ELEVENLABS_API_KEY (required), SPEED=1.33, VOICE=21m00Tcm4TlvDq8ikWAM,
+//      MODEL=eleven_turbo_v2_5, FONT="DejaVu Sans"
+// Needs: Node 18+ (global fetch, top-level await) and ffmpeg built with libass (`ass` filter).
+import { execFileSync } from "node:child_process";
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import { join, resolve } from "node:path";
+
+const [stepsPath, raw, outArg] = process.argv.slice(2);
+if (!stepsPath || !raw || !outArg) {
+  throw new Error("usage: node build-loom.mjs <steps.json> <raw.webm> <outDir>");
+}
+const out = resolve(outArg);
+const SPEED = Number(process.env.SPEED ?? 1.33);
+const VOICE = process.env.VOICE ?? "21m00Tcm4TlvDq8ikWAM";
+const MODEL = process.env.MODEL ?? "eleven_turbo_v2_5";
+const FONT = process.env.FONT ?? "DejaVu Sans";
+const KEY = process.env.ELEVENLABS_API_KEY;
+if (!KEY) throw new Error("ELEVENLABS_API_KEY is not set");
+mkdirSync(join(out, "work"), { recursive: true });
+const W = (f) => join(out, "work", f);
+
+// -loglevel error keeps ffmpeg's banner and progress output out of the logs.
+const ff = (...args) => {
+  try {
+    execFileSync("ffmpeg", ["-y", "-hide_banner", "-loglevel", "error", ...args], {
+      stdio: ["ignore", "ignore", "pipe"],
+    });
+  } catch (e) {
+    throw new Error(`ffmpeg failed: ${String(e.stderr).slice(0, 400)}`);
+  }
+};
+const ENC = [
+  "-c:v",
+  "libx264",
+  "-preset",
+  "veryfast",
+  "-crf",
+  "23",
+  "-pix_fmt",
+  "yuv420p",
+  "-r",
+  "30",
+  "-video_track_timescale",
+  "15360",
+];
+// Keep the aspect ratio: letterbox or pillarbox into 1920x1080, never stretch (mobile viewports).
+const FIT =
+  "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2:color=white";
+
+const { steps } = JSON.parse(readFileSync(stepsPath, "utf8"));
+const concat = [];
+const cues = [];
+const audio = [];
+let T = 0; // running output timeline, in seconds
+
+for (const [i, s] of steps.entries()) {
+  const visualAt = s.visualAt ?? s.start;
+  const segLen = (s.end - s.start) / SPEED;
+  const v = (visualAt - s.start) / SPEED; // voice and subtitles start when the screen renders
+
+  // 1. TTS with character timestamps, one request per step. Cached per narration text,
+  //    so a re-run after a style tweak costs no ElevenLabs characters.
+  const mp3 = W(`audio-${i + 1}.mp3`);
+  const alPath = W(`audio-${i + 1}.json`);
+  let cached = existsSync(alPath) ? JSON.parse(readFileSync(alPath, "utf8")) : null;
+  if (cached?.text !== s.narration) {
+    const res = await fetch(
+      `https://api.elevenlabs.io/v1/text-to-speech/${VOICE}/with-timestamps`,
+      {
+        method: "POST",
+        headers: { "xi-api-key": KEY, "content-type": "application/json" },
+        body: JSON.stringify({ text: s.narration, model_id: MODEL }),
+      },
+    );
+    if (!res.ok) throw new Error(`ElevenLabs ${res.status}: ${(await res.text()).slice(0, 200)}`);
+    const tts = await res.json();
+    // audio_base64 is large: write it straight to disk, never print it.
+    writeFileSync(mp3, Buffer.from(tts.audio_base64, "base64"));
+    cached = { text: s.narration, alignment: tts.alignment };
+    writeFileSync(alPath, JSON.stringify(cached));
+  }
+  const al = cached.alignment;
+  const aDur = al.character_end_times_seconds.at(-1);
+
+  // 2. Sped-up segment. Freeze the last frame when the narration outlasts it.
+  const seg = W(`seg-${i + 1}.mp4`);
+  ff(
+    "-ss",
+    String(s.start),
+    "-t",
+    String(s.end - s.start),
+    "-i",
+    raw,
+    "-an",
+    "-vf",
+    `setpts=(PTS-STARTPTS)/${SPEED},${FIT},fps=30`,
+    ...ENC,
+    seg,
+  );
+  concat.push(seg);
+  const need = v + aDur + 0.35;
+  let len = segLen;
+  if (need > segLen) {
+    const png = W(`last-${i + 1}.png`);
+    ff("-sseof", "-0.2", "-i", seg, "-update", "1", "-frames:v", "1", png);
+    const pad = W(`pad-${i + 1}.mp4`);
+    const padLen = (need - segLen).toFixed(3);
+    ff("-loop", "1", "-framerate", "30", "-t", padLen, "-i", png, "-vf", FIT, ...ENC, pad);
+    concat.push(pad);
+    len = need;
+  }
+  audio.push({ mp3, delayMs: Math.round((T + v) * 1000) });
+
+  // 3. Subtitle cues from the same alignment, so voice and subtitles cannot drift.
+  //    At most 6 words per cue, break on punctuation, merge orphans under 3 words.
+  const words = [];
+  let cur = null;
+  al.characters.forEach((ch, k) => {
+    if (/\s/.test(ch)) {
+      if (cur) words.push(cur);
+      cur = null;
+      return;
+    }
+    if (!cur) cur = { text: "", t0: al.character_start_times_seconds[k] };
+    cur.text += ch;
+    cur.t1 = al.character_end_times_seconds[k];
+  });
+  if (cur) words.push(cur);
+  const groups = [];
+  let g = [];
+  for (const w of words) {
+    g.push(w);
+    if (g.length >= 6 || /[.,;:!?]$/.test(w.text)) {
+      groups.push(g);
+      g = [];
+    }
+  }
+  if (g.length) groups.push(g);
+  for (let k = groups.length - 1; k > 0; k--) {
+    if (groups[k].length < 3) {
+      groups[k - 1].push(...groups[k]);
+      groups.splice(k, 1);
+    }
+  }
+  for (const grp of groups) {
+    cues.push({
+      t0: T + v + grp[0].t0,
+      t1: T + v + grp.at(-1).t1 + 0.1,
+      text: grp.map((w) => w.text).join(" "),
+    });
+  }
+  T += len;
+}
+
+// 4. Concat with the concat demuxer. The concat filter in one big filtergraph can
+//    truncate the video stream.
+writeFileSync(W("concat.txt"), concat.map((f) => `file '${f}'`).join("\n"));
+ff("-f", "concat", "-safe", "0", "-i", W("concat.txt"), "-c", "copy", W("video.mp4"));
+
+// 5. ASS subtitles: one small line, bottom centre, dark box. libass ignores the
+//    BackColour alpha with BorderStyle=3, so the box is solid #303030.
+const ts = (t) => {
+  const cs = Math.round(t * 100);
+  const h = Math.floor(cs / 360000);
+  const m = String(Math.floor(cs / 6000) % 60).padStart(2, "0");
+  const sec = String(Math.floor(cs / 100) % 60).padStart(2, "0");
+  return `${h}:${m}:${sec}.${String(cs % 100).padStart(2, "0")}`;
+};
+for (let k = 0; k < cues.length - 1; k++) cues[k].t1 = Math.min(cues[k].t1, cues[k + 1].t0);
+const dialogue = cues
+  .map((c) => `Dialogue: 0,${ts(c.t0)},${ts(c.t1)},Default,,0,0,0,,${c.text}`)
+  .join("\n");
+const ass = `[Script Info]
+ScriptType: v4.00+
+PlayResX: 1920
+PlayResY: 1080
+
+[V4+ Styles]
+Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
+Style: Default,${FONT},38,&H00FFFFFF,&H00FFFFFF,&H303030,&H00000000,0,0,0,0,100,100,0,0,3,2,0,2,40,40,30,1
+
+[Events]
+Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
+${dialogue}
+`;
+const assPath = join(out, "qa-loom.ass");
+writeFileSync(assPath, ass);
+
+// 6. Mux: delay each step's audio to where its screen renders, mix, burn subtitles.
+//    normalize=0 stops amix from dividing the volume by the number of inputs.
+const inputs = audio.flatMap((a) => ["-i", a.mp3]);
+const delays = audio.map((a, k) => `[${k + 1}:a]adelay=${a.delayMs}:all=1[a${k}]`).join(";");
+const labels = audio.map((_, k) => `[a${k}]`).join("");
+const mix = `${labels}amix=inputs=${audio.length}:normalize=0:duration=longest[a]`;
+ff(
+  "-i",
+  W("video.mp4"),
+  ...inputs,
+  "-filter_complex",
+  `${delays};${mix};[0:v]ass=${assPath}[v]`,
+  "-map",
+  "[v]",
+  "-map",
+  "[a]",
+  ...ENC,
+  "-c:a",
+  "aac",
+  "-b:a",
+  "128k",
+  "-movflags",
+  "+faststart",
+  "-t",
+  T.toFixed(3),
+  join(out, "qa-loom.mp4"),
+);
+const chars = steps.reduce((n, s) => n + s.narration.length, 0);
+console.log(
+  JSON.stringify({ durationS: +T.toFixed(2), steps: steps.length, cues: cues.length, chars }),
+);

package/templates/skills/qa-loom/files/run-qa.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# run-qa.sh: record a UI QA pass with agent-browser and write steps.json.
+# Every step offset is in seconds from the moment recording starts.
+#
+# Usage: BASE_URL=http://localhost:5274 OUT=./loom bash run-qa.sh
+# Needs: agent-browser (with `record`), jq, GNU date (%N). No bc: time math runs in jq.
+#
+# Replace the two example steps with the steps that prove your change.
+set -euo pipefail
+BASE_URL=${BASE_URL:-http://localhost:5274}
+OUT=${OUT:-./loom}
+mkdir -p "$OUT"
+LOG="$OUT/steps.ndjson"
+: >"$LOG"
+
+now() { date +%s.%N; }
+# act: call right before the action fires. It marks the step start.
+act() { STEP_START=$(now); }
+# step <action> <observed> <verdict> <narration>: call AFTER the resulting screen has
+# rendered (after an `agent-browser wait <selector>`). That moment becomes visualAt.
+step() {
+  jq -nc --arg a "$1" --arg o "$2" --arg v "$3" --arg n "$4" \
+    --arg t0 "$T0" --arg s "$STEP_START" --arg va "$(now)" \
+    '{action:$a, observed:$o, verdict:$v, narration:$n,
+      start:(($s|tonumber)-($t0|tonumber)), visualAt:(($va|tonumber)-($t0|tonumber))}' >>"$LOG"
+}
+
+# Log in and seed data BEFORE recording starts.
+agent-browser set viewport 1920 1080 >/dev/null
+agent-browser open "$BASE_URL" >/dev/null
+agent-browser wait 1500 >/dev/null
+agent-browser record start "$OUT/raw.webm" --cursor --fps 30 >/dev/null
+T0=$(now)
+
+act
+agent-browser reload >/dev/null
+agent-browser wait h1 >/dev/null
+step "Load home page" "Heading and nav render" pass "The home page loads with its heading and nav."
+agent-browser wait 800 >/dev/null # dwell so the viewer sees the result before the next action
+
+act
+agent-browser open "$BASE_URL/settings" >/dev/null
+agent-browser wait h1 >/dev/null
+step "Open settings" "Settings page renders" pass "The settings page opens."
+agent-browser wait 800 >/dev/null
+
+END=$(jq -n --arg a "$(now)" --arg b "$T0" '($a|tonumber)-($b|tonumber)')
+agent-browser record stop >/dev/null
+
+# Each step ends where the next one starts; the last one ends at record stop.
+jq -s --argjson end "$END" '
+  {steps: ([to_entries[] | .value + {idx: (.key + 1)}]
+    | . as $s
+    | [range(0; length) as $i
+        | $s[$i] + {end: (if $i + 1 < ($s | length) then $s[$i + 1].start else $end end)}])}
+' "$LOG" >"$OUT/steps.json"
+jq -c '.steps[] | {idx, start, visualAt, end, verdict}' "$OUT/steps.json"

package/templates/skills/slack-interaction/content.md

@@ -48,3 +48,7 @@ When your heartbeat runbook says so, check for unaddressed requests older than o
 ## Scripts-only mode
 
 The named Slack tools are not registered. Use `script-run` with inline source: `ctx.swarm.slack_reply({ taskId, message })`. The task ID carries the thread context.
+
+## Status and blocker questions
+
+Answer for the scope of the place the question was asked. In a customer or project channel or thread, "current blockers" or "what's the status" means that customer's or project's work: its tracker, its PRs, its open asks. Answer swarm-wide only in a DM or an ops channel, or when the person asks for it. Name the scope you used in the first line.

package/templates/skills/swarm-extensions/content.md

@@ -47,10 +47,12 @@ curl -s "$MCP_BASE_URL/api/extensions/type-defs" \
 | `pre.task.create` | before any task is stored (REST, MCP, Slack, schedule, workflow, webhook, follow-up) | `block(reason)`, `modify({ priority, agentId, tags, description, ... })`, or nothing |
 | `pre.task.followUp` | before the lead follow-up task for a finished worker task | `block`, `modify({ description, priority, agentId })`, or nothing |
 | `pre.slack.route` | before a Slack message becomes a task | `modify({ target: { kind: "agent", agentId } })`, `{ kind: "lead" }`, `{ kind: "broadcast" }` |
+| `pre.chat.route` | before buffered Chat SDK messages (Telegram, Slack, Discord, Teams) become a task | `block(reason)`, or `modify({ target: { kind: "agent", agentId } })`, `{ kind: "lead" }`, `{ kind: "broadcast" }`, `{ kind: "workflow", workflowId }` |
 | `pre.heartbeat.remediate` | after the heartbeat sweep finds a stalled task | `modify({ proposedAction: "record" | "fail" | "supersede-resume" })` |
 | `pre.tool.call` | before an agent MCP tool call runs | `block(reason)` or `modify({ args })` |
 | `post.task.created`, `post.task.completed`, `post.task.failed`, `post.task.cancelled`, `post.task.superseded`, `post.task.progress` | after the change is committed | none |
 | `post.slack.message`, `post.tool.call` | after the message or tool call finished | none |
+| `post.chat.message` | after a Chat SDK message is received (before buffering) or sent; `direction` is `inbound` or `outbound` | none |
 | `post.approval.resolved` | after a person answers an approval request, workflow or standalone (not on timeout or cancel) | none |
 | `post.task.budgetRefused` | after a task is refused by a daily spend budget | none |
 | `post.email.received`, `post.kapso.message` | after an inbound AgentMail or Kapso message is received | none |

package/templates/skills/swarm-scripts/SKILL.md

@@ -76,7 +76,7 @@ export default async function (args: z.infer<typeof argsSchema>, ctx: ScriptCont
 
 - `ctx.swarm.*`: the swarm SDK. `task_get`, `task_send`, `task_storeProgress`, `task_action`, `task_list`, `slack_reply`, `memory_search`, `memory_store`, `kv_get`, `kv_getOrNull`, `kv_set`, `kv_delete`, `kv_incr`, `kv_list`, `swarm_get`, `agent_info`, `db_query`, and more. `kv_getOrNull` returns the entry, `null` on a missing key, and throws on other errors.
 - `ctx.swarm.task_send` requires `routingReason` and `routingNote` whenever `agentId` is present. The note must explain the worker choice in at least 10 characters after trimming whitespace (maximum 200 characters). Use `human_pinned` for an explicit/configured choice, `skill` for a role or specialization match, `continuity` for the same worker/session, `reroute_fault` for a fault handoff, or `overflow` for capacity/pool escalation. Omit both `agentId` and routing fields for the pool.
-- `ctx.swarm.config`: `apiKey`, `agentId`, `mcpBaseUrl`, and `ctx.swarm.config.get("KEY")` for user config values. All are `Redacted` wrappers that stringify to `<redacted>`. Never unwrap one into a return value, a log line, or a request body you build by hand.
+- `ctx.swarm.config`: `apiKey`, `agentId`, `mcpBaseUrl`, and `ctx.swarm.config.get("KEY")` for swarm config values, resolved for the agent running the script (agent scope over global, the same rows `get-config` returns for that agent). A key that is not set returns `undefined`, so `config.get("KEY") ?? fallback` works. All are `Redacted` wrappers that stringify to `<redacted>`. Never unwrap one into a return value, a log line, or a request body you build by hand.
 - `ctx.api.<slug>` and `ctx.mcp.<slug>`: typed clients for registered connections. They exist only for registered connections. Introspect with `Object.keys(ctx.api ?? {})` and `Object.keys(ctx.mcp ?? {})`.
 - `ctx.stdlib`: `fetch`, `fetchJson` (retries, 30 s timeout), `grep`, `glob`, `table`, `Redacted`.
 - `ctx.logger`: `log`, `warn`, `error`. Keep logs short.
@@ -107,11 +107,13 @@ export default async function main(args: { status?: string; limit?: number } | u
 
 ## Secrets
 
+A config row reaches your shell env only when its `exportToEnv` flag is on. If `$KEY` is empty in your shell but `get-config` or `list-config` shows the row marked `not in agent env`, the row stays server-side on purpose: do not copy the value into a file, an env var, or a command. Use it from a script instead, through one of the paths below. Harness, git, and agent-fs credentials stay exported. Keys only the API server reads (Slack app token and signing secret, GitHub App private key, webhook and OAuth client secrets) can never be exported. If a recipe genuinely needs the variable in a shell, ask the lead to turn `exportToEnv` on for that row (`set-config` with `exportToEnv: true`); it applies to your next task.
+
 Order of preference:
 
 1. A registered connection: `ctx.api.<slug>.<operationId>({ path, query, header, body })` (OpenAPI), `ctx.api.<slug>.graphql(query, variables)` (GraphQL), `ctx.mcp.<slug>.<toolName>(args)` (MCP, proxied server-side, returns the raw MCP envelope: read `res.structuredContent ?? res.content?.[0]?.text`). Credentials attach at egress. The script never sees them. OAuth tokens refresh on their own.
 2. A credential binding placeholder in a hand-written request: `Authorization: Bearer [REDACTED:GITHUB_TOKEN]`. The runtime substitutes the value at egress, only for the binding's allowed hosts.
-3. `ctx.swarm.config.get("KEY")` for a config value you pass to `ctx.stdlib.fetch` as a header value. Still `Redacted`; never log it.
+3. `ctx.swarm.config.get("KEY")` for a config value you pass to `ctx.stdlib.fetch` as a header value. Unwrap it with `ctx.stdlib.Redacted.value(...)` inside the header itself (a wrapped value sends the literal `<redacted>`), and never log it. Secret values are scrubbed from run output.
 
 A raw secret, or the output of `get-config` with `includeSecrets: true`, must not appear in script source, script `args`, a schedule's `taskTemplate` or `scriptArgs`, or a task description. Those are stored as plain text and replayed on every run.
 

package/templates/skills/swarm-scripts/config.json

@@ -5,7 +5,7 @@
   "slug": "swarm-scripts",
   "title": "Swarm Scripts",
   "description": "Bulk, repeat, fan-out, or data-heavy work: write and run swarm scripts (inline `script-run`, named `script-upsert`, durable `launch-script-run`). Covers the script-vs-tool rubric, the authoring contract (`args` first, `ctx` second), the seed catalog, connections and secrets, `db_query`, and exposing a script as an API.",
-  "version": "1.1.0",
+  "version": "1.2.0",
   "category": "skills",
   "placeholders": [],
   "runAllSeedersCandidate": true,

package/templates/skills/swarm-scripts/content.md

@@ -71,7 +71,7 @@ export default async function (args: z.infer<typeof argsSchema>, ctx: ScriptCont
 
 - `ctx.swarm.*`: the swarm SDK. `task_get`, `task_send`, `task_storeProgress`, `task_action`, `task_list`, `slack_reply`, `memory_search`, `memory_store`, `kv_get`, `kv_getOrNull`, `kv_set`, `kv_delete`, `kv_incr`, `kv_list`, `swarm_get`, `agent_info`, `db_query`, and more. `kv_getOrNull` returns the entry, `null` on a missing key, and throws on other errors.
 - `ctx.swarm.task_send` requires `routingReason` and `routingNote` whenever `agentId` is present. The note must explain the worker choice in at least 10 characters after trimming whitespace (maximum 200 characters). Use `human_pinned` for an explicit/configured choice, `skill` for a role or specialization match, `continuity` for the same worker/session, `reroute_fault` for a fault handoff, or `overflow` for capacity/pool escalation. Omit both `agentId` and routing fields for the pool.
-- `ctx.swarm.config`: `apiKey`, `agentId`, `mcpBaseUrl`, and `ctx.swarm.config.get("KEY")` for user config values. All are `Redacted` wrappers that stringify to `<redacted>`. Never unwrap one into a return value, a log line, or a request body you build by hand.
+- `ctx.swarm.config`: `apiKey`, `agentId`, `mcpBaseUrl`, and `ctx.swarm.config.get("KEY")` for swarm config values, resolved for the agent running the script (agent scope over global, the same rows `get-config` returns for that agent). A key that is not set returns `undefined`, so `config.get("KEY") ?? fallback` works. All are `Redacted` wrappers that stringify to `<redacted>`. Never unwrap one into a return value, a log line, or a request body you build by hand.
 - `ctx.api.<slug>` and `ctx.mcp.<slug>`: typed clients for registered connections. They exist only for registered connections. Introspect with `Object.keys(ctx.api ?? {})` and `Object.keys(ctx.mcp ?? {})`.
 - `ctx.stdlib`: `fetch`, `fetchJson` (retries, 30 s timeout), `grep`, `glob`, `table`, `Redacted`.
 - `ctx.logger`: `log`, `warn`, `error`. Keep logs short.
@@ -102,11 +102,13 @@ export default async function main(args: { status?: string; limit?: number } | u
 
 ## Secrets
 
+A config row reaches your shell env only when its `exportToEnv` flag is on. If `$KEY` is empty in your shell but `get-config` or `list-config` shows the row marked `not in agent env`, the row stays server-side on purpose: do not copy the value into a file, an env var, or a command. Use it from a script instead, through one of the paths below. Harness, git, and agent-fs credentials stay exported. Keys only the API server reads (Slack app token and signing secret, GitHub App private key, webhook and OAuth client secrets) can never be exported. If a recipe genuinely needs the variable in a shell, ask the lead to turn `exportToEnv` on for that row (`set-config` with `exportToEnv: true`); it applies to your next task.
+
 Order of preference:
 
 1. A registered connection: `ctx.api.<slug>.<operationId>({ path, query, header, body })` (OpenAPI), `ctx.api.<slug>.graphql(query, variables)` (GraphQL), `ctx.mcp.<slug>.<toolName>(args)` (MCP, proxied server-side, returns the raw MCP envelope: read `res.structuredContent ?? res.content?.[0]?.text`). Credentials attach at egress. The script never sees them. OAuth tokens refresh on their own.
 2. A credential binding placeholder in a hand-written request: `Authorization: Bearer [REDACTED:GITHUB_TOKEN]`. The runtime substitutes the value at egress, only for the binding's allowed hosts.
-3. `ctx.swarm.config.get("KEY")` for a config value you pass to `ctx.stdlib.fetch` as a header value. Still `Redacted`; never log it.
+3. `ctx.swarm.config.get("KEY")` for a config value you pass to `ctx.stdlib.fetch` as a header value. Unwrap it with `ctx.stdlib.Redacted.value(...)` inside the header itself (a wrapped value sends the literal `<redacted>`), and never log it. Secret values are scrubbed from run output.
 
 A raw secret, or the output of `get-config` with `includeSecrets: true`, must not appear in script source, script `args`, a schedule's `taskTemplate` or `scriptArgs`, or a task description. Those are stored as plain text and replayed on every run.
 

package/templates/skills/workflow-iterate/content.md

@@ -173,6 +173,44 @@ Afterwards, a `warnings` entry on `create-workflow` or a patch tool means the ke
 - Do not set `retry` or `validation.retry` on a `system-one-decision` node. It retries connection errors, 408, 429, and 5xx itself (`config.maxRetries`, 0 to 3, default 2) inside `config.timeoutMs` (default 30000). A 401, a 422, or an answer that fails validation ends after one attempt.
 - The node reads its provider's key (`TYPESAFE_API_KEY`, `OPENROUTER_API_KEY`, `LAYA_API_KEY`, `OPENAI_DECISIONS_API_KEY`, or `CLOUDFLARE_API_TOKEN`, plus `LAYA_URL` for `laya` and `CLOUDFLARE_ACCOUNT_ID` for `cloudflare`) server-side. The key never appears in the definition, the step output, or an error. A run whose key or `LAYA_URL` is missing fails before any node executes, and a 401 or 403 fails the step as `<KEY> was rejected`. An unresolved `{{token}}` fails the step before any request.
 
+#### Jev images (direct TypeSafe API)
+
+`system-one-decision` cannot send images today: its strict config has no `assets` field, its wire sends only `state`, `model`, and `questions`, and it does not send the image opt-in header. A URL or base64 string in `state` stays text. Use a `swarm-script` calling `POST https://api.typesafe.ai/v1/systemone` with the credential kept server-side until the node supports assets.
+
+- Send `TypeSafe-Danger-Zone-Images: true`. Both `jev-1.13.0` and `jev-1.14.0` accept images; Box questions require 1.14. Pin the model you tested.
+- Upload PNG or JPEG bytes, not a URL. Prefer multipart: first part `request` is the JSON with content type `application/json`; an asset such as `assets[uuid] = { type: "image", part: "photo-file" }` names its file part. For JSON-only calls, use the base64 shape below: standard padded base64, no whitespace or `data:` prefix.
+- Each image must be still, at most 16 MiB decoded, 256–1568 pixels on each side, and 78,400–2,458,624 pixels total (256×256 is too small). Request body, all processed images together, and the expanded request each have a 32 MiB limit. Limits reject the whole request; the API does not resize. Rotate pixels upright before sending because EXIF orientation is not applied.
+- Generate a fresh UUID reference per image. References are replaced wherever they occur in string values in `state`, `instructions`, or `criteria`; object keys are not replaced. Every asset must be referenced. Repeating the same reference inserts and bills another image copy.
+
+```json
+{
+  "model": "jev-1.13.0",
+  "state": { "photo": "d1369c6a-538e-4ab3-91cd-f76f2f67b651" },
+  "assets": {
+    "d1369c6a-538e-4ab3-91cd-f76f2f67b651": {
+      "type": "image", "format": "png", "data_base64": "<encoded PNG bytes>"
+    }
+  },
+  "questions": {
+    "ready": { "type": "noul", "instructions": "Is the product in `photo` fully assembled?" }
+  }
+}
+```
+
+Measured 2026-10-15 on `jev-1.13.0`, using an 850×850 RGB PNG with a red square on white. Cost is the complete request's `usage.input_tokens × $0.042 / 1,000,000`; output is free:
+
+| Placement | Questions | Input tokens | Request cost (USD) |
+|---|---:|---:|---:|
+| Once in `state` | 1 | 2,273 | $0.000095466 |
+| Once in `state`, questions use the `photo` path | 2 | 2,289 | $0.000096138 |
+| Twice in `state` | 1 | 4,237 | $0.000177954 |
+| Once in each question's `instructions` | 2 | 4,203 | $0.000176526 |
+| Once in `state`, resized to 425×425 | 1 | 863 | $0.000036246 |
+
+The roughly $0.0001 claim (about 10,000 one-image requests per dollar) holds for this example; the exact cost depends on image size and prompt. **Question count alone does not multiply a shared state image's tokens.** Put shared images in `state` once and have questions use its field path, not the asset UUID. Image duplication scales with repeated references, including one copy per question when embedded in each question's instructions. If several judgments form one classification, a single `choice` can avoid repeating question-local images. Downscale when the detail permits: halving both dimensions reduced this request's billed tokens from 2,273 to 863; recheck accuracy on your images.
+
+Contract and current pricing: [TypeSafe images](https://docs.typesafe.ai/images), [models](https://docs.typesafe.ai/models). Re-measure before budgeting a different model, resolution, or prompt.
+
 ### `human-in-the-loop` (how approvals render)
 
 The same rules apply to a standalone `request-human-input` call.