@themoltnet/node-red-contrib-core 0.7.1 → 0.9.0

package/README.md

@@ -213,12 +213,32 @@ model-specialization option.
 
 [`examples/ab-eval-with-judge.flow.json`](./examples/ab-eval-with-judge.flow.json)
 imports a reusable **A/B eval with judge** subflow plus a small demo tab. The
-subflow runs `run_eval`, locally scores required fields/findings, then creates a
-`judge_eval_attempt` task and stores per-variant scores/deltas in flow context.
+subflow runs one `run_eval` producer lane, records lightweight producer
+metadata, then creates one `judge_eval_attempt` task and stores the variant
+score/delta in flow context. The judge owns the rubric; the producer-side flow
+does not duplicate hidden criteria. The demo tab owns the reusable workflow
+runner pattern: initialize one correlation id, fan out configured variants in
+parallel, record successful or failed lanes, and emit a group result only once
+all expected variants have settled.
+
+The bundled seed uses `evals/moltnet-practices/dbos-after-commit` and compares
+`baseline-no-context` against `rendered-pack-dbos-rule`, a rendered-pack-style
+context excerpt that teaches the DBOS/Drizzle transaction boundary. Replace
+that inline context with a rendered MoltNet pack or skill content to evaluate a
+real candidate context source against the same hidden judge rubric.
 
 Fill the `moltnet-agent` config after import. Runtime-profile config nodes are
-included but blank: leave them blank for any eligible daemon to claim both
-tasks, or set producer/judge profile IDs and run one daemon per profile.
+included and default to the shared Gemma eval profile; clear them to let any
+eligible daemon claim both tasks, or set producer/judge profile IDs and run one
+daemon per profile. The example uses `maxAttempts=2` for producer and judge
+tasks so transient tool/model failures retry at the task layer before the lane
+is marked failed.
+Callers can also route per scenario or variant without editing the subflow:
+set `msg.evalRuntimeProfiles.producer`, `msg.evalRuntimeProfiles.judge`,
+`msg.evalScenario.runtimeProfiles.producer`, `msg.evalScenario.runtimeProfiles.judge`,
+or the variant-level `runtimeProfile` / `producerRuntimeProfile` /
+`judgeRuntimeProfile`. Each value may be a profile id string, `{ profileId }`,
+or a full `allowedProfiles` array.
 
 ## Freeform deep review workflow
 
@@ -309,6 +329,75 @@ in the Node-RED instance) to get a live workflow cockpit without any custom Vue
 widget. A starter flow is in [`examples/cockpit.flow.json`](./examples/cockpit.flow.json)
 (inject → `moltnet-workflow-status` → debug; add a `ui-table` to visualize).
 
+## Axiom alert triage workflow
+
+[`examples/axiom-alert-triage.flow.json`](./examples/axiom-alert-triage.flow.json)
+is an observability triage experiment with two trigger modes:
+
+`Axiom monitor history polling` → **parser/enricher task** →
+**codebase triage task** → **judge task**.
+
+When Axiom custom webhook notifiers are available, the same triage chain can also
+start from `POST /axiom/monitor`.
+
+The polling path calls Axiom monitor history with `AXIOM_MONITORING_TOKEN`
+(`AXIOM_API_TOKEN` is accepted as a fallback for local experiments), dedupes
+events by monitor/state/timestamp in flow context, starts work only for
+trigger-like events, builds task bodies through `task: build`, and chains tasks
+through `task: read` `artifactRef` values so large telemetry artifacts stay in
+MoltNet. Resolution/no-op events go to a debug lane instead of starting codebase
+triage.
+
+Polling environment:
+
+- `AXIOM_MONITORING_TOKEN`: Axiom token with monitor history/read access and,
+  for the current single-token experiment, telemetry query read access.
+- `AXIOM_MONITOR_IDS`: optional comma-separated monitor ids. Defaults to the
+  MoltNet monitors created by `infra/axiom/monitors/apply.mjs`.
+- `AXIOM_POLL_WINDOW_MINUTES`: optional lookback window, default `15`.
+
+To test the flow without a real Axiom trigger, use the **inject synthetic
+alert** node in the example. It builds the same normalized `msg.axiom` shape as
+the polling/webhook paths, marks it with `synthetic: true`, and routes it through
+the common monitor-events link so the parser, triage, artifact chaining, and
+judge stages all run.
+
+Optional Axiom custom webhook setup:
+
+- URL: `https://<node-red-host>/axiom/monitor`
+- Method: `POST`
+- Optional secret header: `x-axiom-webhook-secret: <secret>`
+- Node-RED env: `AXIOM_WEBHOOK_SECRET=<secret>` when that header is used
+
+Suggested custom webhook body:
+
+```json
+{
+  "action": "{{.Action}}",
+  "body": "{{.Body}}",
+  "endTime": "now",
+  "monitorId": "{{.MonitorID}}",
+  "monitorName": "{{.Title}}",
+  "service": "moltnet-rest-api",
+  "severity": "warning",
+  "startTime": "now-30m",
+  "value": "{{.Value}}"
+}
+```
+
+The parser/enricher task prompt asks the agent to use the repo-local
+`axiom-alert-triage` skill and produce a JSON artifact titled
+`telemetry-evidence`. The next task consumes that artifact, compares findings
+against the `getlarge/themoltnet` codebase, GitHub issues, and MoltNet entries,
+then writes `triage-report`. The final judge task writes `triage-judgment`.
+
+Before enabling live webhooks, refine Axiom monitors so broad auth/client noise
+(401/404) is separated from actionable alerts such as 5xx errors, latency
+regression, event loop pressure, and memory pressure. The companion repo-local
+`axiom-observability` skill and [`infra/axiom`](../../infra/axiom) document the
+monitor/dashboard maintenance conventions and provide idempotent apply scripts
+for the committed Axiom config.
+
 ## Build
 
 ```bash
@@ -369,7 +458,9 @@ export default {
 Open the editor, drag in `agent` + the task nodes, or import
 [`examples/issue-lifecycle.flow.json`](./examples/issue-lifecycle.flow.json) or
 [`examples/cockpit.flow.json`](./examples/cockpit.flow.json) or
-[`examples/deep-review-freeform.flow.json`](./examples/deep-review-freeform.flow.json),
+[`examples/deep-review-freeform.flow.json`](./examples/deep-review-freeform.flow.json)
+or
+[`examples/axiom-alert-triage.flow.json`](./examples/axiom-alert-triage.flow.json),
 then fill the agent's `clientId`/`clientSecret`.
 
 If Node-RED crashes in `@node-red/editor-api/lib/auth/tokens.js` with

package/dist/nodes/src.js

@@ -1927,7 +1927,7 @@ var findLatestRuntimeSlotForAttempt = (options) => (options.client ?? client).ge
 	...options
 });
 /**
-* Delete terminal tasks in bulk. Safe mode skips live, unauthorized, missing, and protected tasks.
+* Queue asynchronous deletion of terminal tasks in bulk. By default, live, unauthorized, missing, and protected tasks are skipped. Set force: true with a reason to delete protected terminal tasks.
 */
 var batchDeleteTasks = (options) => (options.client ?? client).delete({
 	security: [
@@ -14172,6 +14172,86 @@ var PRODUCER_TASK_TYPES = new Set([
 	"render_pack",
 	"run_eval"
 ]);
+function isNonEmptyString(value) {
+	return typeof value === "string" && value.length > 0;
+}
+function criterionWeight(criterion, index) {
+	if (typeof criterion.weight === "number") return criterion.weight;
+	if (typeof criterion.max_score === "number") return criterion.max_score / 100;
+	if (typeof criterion.maxScore === "number") return criterion.maxScore / 100;
+	throw new TaskBuildError([{
+		field: `successCriteria/rubric/criteria/${index}/weight`,
+		message: "criterion is missing weight or max_score"
+	}]);
+}
+/**
+* Normalize authoring-time rubric criteria to canonical MoltNet rubric
+* criteria. Accepts `{id,title,description,weight}` and
+* `{name,description,max_score}` style inputs, strips authoring-only fields,
+* and fills a default scoring mode.
+*/
+function normalizeRubricCriteria(criteria, options) {
+	const errors = [];
+	const normalized = criteria.map((criterion, index) => {
+		const id = criterion.id ?? criterion.name;
+		const description = criterion.description ?? criterion.title;
+		if (!isNonEmptyString(id)) errors.push({
+			field: `successCriteria/rubric/criteria/${index}/id`,
+			message: "criterion is missing id or name"
+		});
+		if (!isNonEmptyString(description)) errors.push({
+			field: `successCriteria/rubric/criteria/${index}/description`,
+			message: "criterion is missing description or title"
+		});
+		return {
+			id: id ?? "",
+			description: description ?? "",
+			weight: criterionWeight(criterion, index),
+			scoring: criterion.scoring ?? options?.scoring ?? "llm_score"
+		};
+	});
+	if (errors.length > 0) throw new TaskBuildError(errors);
+	return normalized;
+}
+/**
+* Build a canonical `SuccessCriteria` envelope from rubric/checklist-style
+* criteria. This keeps rubrics readable at the authoring boundary while
+* preserving the strict task schema on the wire.
+*/
+function buildRubricSuccessCriteria(options) {
+	const rubric = {
+		rubricId: options.rubricId,
+		version: options.version ?? "v1",
+		criteria: normalizeRubricCriteria(options.criteria, { scoring: options.scoring }),
+		...options.contentHash ? { contentHash: options.contentHash } : {},
+		...options.preamble ? { preamble: options.preamble } : {},
+		...options.scope ? { scope: options.scope } : {}
+	};
+	const weightError = validateRubricWeights(rubric);
+	if (weightError) throw new TaskBuildError([{
+		field: "successCriteria/rubric/criteria",
+		message: weightError
+	}]);
+	return {
+		version: 1,
+		rubric
+	};
+}
+function resolveJudgeEvalAttemptTarget(target) {
+	if ("judgeEvalTarget" in target && typeof target.judgeEvalTarget === "function") return target.judgeEvalTarget();
+	if ("targetTaskId" in target) return {
+		targetTaskId: target.targetTaskId,
+		targetAttemptN: target.targetAttemptN
+	};
+	if ("taskId" in target) return {
+		targetTaskId: target.taskId,
+		targetAttemptN: target.accepted?.attemptN ?? target.attemptN ?? 1
+	};
+	throw new TaskBuildError([{
+		field: "target",
+		message: "judge_eval_attempt target is missing task id"
+	}]);
+}
 /**
 * Fluent, network-free builder for a `tasks.create` body. Encodes the
 * non-obvious task schema (context arrays, success-criteria gates,
@@ -14622,6 +14702,20 @@ function buildJudgeEvalAttempt(input) {
 	return buildTask("judge_eval_attempt", input);
 }
 /**
+* Build a `judge_eval_attempt` task from an accepted `run_eval` result (or a
+* small target tuple) plus human-friendly rubric criteria.
+*
+* @param target - A `TaskResultReader` or `{targetTaskId,targetAttemptN}` tuple.
+* @param options - Rubric metadata and eval/checklist-style criteria.
+* @returns A typed {@link TaskBuilder}.
+*/
+function buildJudgeEvalAttemptForRunEval(target, options) {
+	return buildJudgeEvalAttempt({
+		...resolveJudgeEvalAttemptTarget(target),
+		successCriteria: buildRubricSuccessCriteria(options)
+	});
+}
+/**
 * Build a `pr_review` task. Requires `subject` + `successCriteria`. Note the
 * rubric criteria must use `boolean` scoring for this task type.
 *
@@ -14659,7 +14753,9 @@ var TaskResultReader = class {
 	accepted;
 	/** Token / cost usage for the accepted attempt, if reported. */
 	usage;
+	/** Task id for the task whose accepted attempt is being read. */
 	taskId;
+	/** CID of the accepted attempt output. */
 	outputCid;
 	constructor(task, attempt) {
 		const errors = [];
@@ -14752,6 +14848,19 @@ var TaskResultReader = class {
 		};
 	}
 	/**
+	* Return the target tuple required by `judge_eval_attempt`.
+	*
+	* This intentionally uses the accepted attempt number, not merely the
+	* attempt object passed to the reader, so a downstream judge is pinned to
+	* the producer output that the task accepted.
+	*/
+	judgeEvalTarget() {
+		return {
+			targetTaskId: this.taskId,
+			targetAttemptN: this.accepted.attemptN
+		};
+	}
+	/**
 	* Build a `TaskRef` that anchors a downstream task to this accepted output
 	* and points at one persistent task artifact by CID.
 	*
@@ -14889,6 +14998,7 @@ function createTasksNamespace(context) {
 		buildAssessBrief,
 		buildJudgePack,
 		buildJudgeEvalAttempt,
+		buildJudgeEvalAttemptForRunEval,
 		buildPrReview,
 		async readResult(taskOrId) {
 			const task = typeof taskOrId === "string" ? unwrapResult(await getTask({
@@ -17223,4 +17333,4 @@ if (!etc.sha512Sync) etc.sha512Sync = (...m) => {
 	return hash.digest();
 };
 //#endregion
-export { TaskResultError as a, TaskBuildError as i, createResultReader as n, buildTask as r, connect as t };
+export { TaskBuildError as a, buildTask as i, createResultReader as n, TaskResultError as o, buildJudgeEvalAttemptForRunEval as r, connect as t };

package/dist/nodes/task-builder.html

@@ -61,6 +61,8 @@
     >
     <select id="node-input-taskType">
       <option value="freeform">freeform</option>
+      <option value="run_eval">run_eval</option>
+      <option value="judge_eval_attempt">judge_eval_attempt</option>
     </select>
   </div>
   <div class="form-row">

package/dist/nodes/task-builder.js

@@ -1,10 +1,13 @@
-import { i as TaskBuildError, r as buildTask } from "./src.js";
+import { a as TaskBuildError, i as buildTask, r as buildJudgeEvalAttemptForRunEval } from "./src.js";
 //#region src/nodes/task-builder.ts
 /** Split a comma-separated config string into trimmed, non-empty values. */
 function parseCsv(raw) {
 	if (!raw) return [];
 	return raw.split(",").map((s) => s.trim()).filter(Boolean);
 }
+function isRuntimeProfileRef(value) {
+	return typeof value === "object" && value !== null && typeof value.profileId === "string";
+}
 /** Resolve a context mapping's raw value from the message / context stores / literal. */
 function resolveValue(RED, node, msg, m) {
 	switch (m.valueType) {
@@ -49,8 +52,12 @@ var init = (RED) => {
 		this.on("input", (msg, send, done) => {
 			try {
 				const payloadInput = msg.payload && typeof msg.payload === "object" ? msg.payload : {};
-				const brief = typeof payloadInput.input?.brief === "string" ? payloadInput.input.brief : def.brief ?? "";
-				const builder = buildTask(def.taskType?.trim() || "freeform", { brief });
+				const taskType = typeof payloadInput.taskType === "string" && payloadInput.taskType ? payloadInput.taskType : def.taskType?.trim() || "freeform";
+				const inputData = payloadInput.input && typeof payloadInput.input === "object" ? payloadInput.input : taskType === "freeform" || taskType === "fulfill_brief" ? { brief: def.brief ?? "" } : {};
+				const builder = taskType === "judge_eval_attempt" && payloadInput.judgeRubric && typeof payloadInput.judgeRubric === "object" ? buildJudgeEvalAttemptForRunEval({
+					targetTaskId: inputData.targetTaskId,
+					targetAttemptN: inputData.targetAttemptN
+				}, payloadInput.judgeRubric) : buildTask(taskType, inputData);
 				const teamId = resolveOverride(RED, this, msg, def.teamId, def.teamIdType) ?? payloadInput.teamId ?? agentNode?.teamId;
 				const diaryId = resolveOverride(RED, this, msg, def.diaryId, def.diaryIdType) ?? payloadInput.diaryId ?? agentNode?.diaryId;
 				if (teamId) builder.team(teamId);
@@ -81,6 +88,12 @@ var init = (RED) => {
 				if (title) builder.title(title);
 				const tags = Array.isArray(payloadInput.tags) ? payloadInput.tags : parseCsv(def.tags);
 				if (tags.length > 0) builder.tags(...tags);
+				if (typeof payloadInput.correlationId === "string" && payloadInput.correlationId) builder.correlationId(payloadInput.correlationId);
+				if (typeof payloadInput.maxAttempts === "number") builder.maxAttempts(payloadInput.maxAttempts);
+				if (Array.isArray(payloadInput.allowedProfiles)) {
+					const allowedProfiles = payloadInput.allowedProfiles.filter(isRuntimeProfileRef);
+					if (allowedProfiles.length > 0) builder.allowProfiles(...allowedProfiles);
+				}
 				const built = builder.build();
 				const out = RED.util.cloneMessage(msg);
 				out.payload = {

package/dist/nodes/task-reader.js

@@ -1,4 +1,4 @@
-import { a as TaskResultError, n as createResultReader } from "./src.js";
+import { n as createResultReader, o as TaskResultError } from "./src.js";
 //#region src/nodes/task-reader.ts
 var init = (RED) => {
 	function TaskReaderNode(def) {