clyops-jobs 0.9.0 → 0.11.0

package/README.md

@@ -27,6 +27,19 @@ and no `node_modules`. It includes clyops-tools. `require('clyops-jobs')` loads
 }
 ```
 
+**Editor support.** The config has a JSON Schema, so editors complete and check it. Name it in
+the file, from the site or from the installed package:
+
+```json
+{ "$schema": "https://clyops.io/schemas/clyops-jobs.json", "functions": {} }
+```
+
+It's `config.schema.json` in the package (`clyops-jobs/config.schema.json`, e.g.
+`"./node_modules/clyops-jobs/config.schema.json"`). It catches typos in function and step
+fields (`x-` fields are free for your own notes), `script` and `steps` together, and bad step
+names; that a branch names a later step is checked when the function is resolved. Options
+blocks are any object, since their keys come from each tool's schema.
+
 - `functions.NAME.script` is the tool, relative to a script root.
 - `key` names the options block (default: the function name). Options are
   `deepMerge(defaults[key], config[key])`, keyed like the tool's options (`out_dir`, `out-dir`)
@@ -35,7 +48,12 @@ and no `node_modules`. It includes clyops-tools. `require('clyops-jobs')` loads
 - `positionals` (optional) lists the tool's positional arguments explicitly; otherwise they're
   options keyed by argument name.
 - String values may use `${dot.path}` or `{{dot.path}}` templates against a context you build
-  (`job`, `paths`, whatever the trigger knows).
+  (`job`, `paths`, whatever the trigger knows). An unknown path is an error, so a typo fails
+  loudly, except under `params` (the caller's fields): an option or positional that names a
+  missing or null `params` field is left unset, so the tool's default applies. That holds for
+  `${params.x}` alone and inside a longer string (`v-${params.x}`), so a value is never half
+  filled. Elsewhere (`result`, `stdin`, `stdout`, artifact patterns) a missing field renders as
+  `''`. `strict_params: true` on a function makes a missing `params` field an error too.
 - `artifacts`: glob patterns (`*`, `?`, `**`) relative to `paths.work_dir`, copied into
   `paths.artifacts_dir`, with an optional `required: true` or `min: N`.
 - `stdin`: a template naming a file fed to the tool's stdin (`"${media.path}"`; relative to the
@@ -43,6 +61,73 @@ and no `node_modules`. It includes clyops-tools. `require('clyops-jobs')` loads
   (`"${paths.work_dir}/out.mp3"`; relative to `paths.work_dir`), which also counts as an artifact.
 - `result`: a template map added to the result record.
 
+### Steps
+
+A function can run several tools in order instead of one `script`, so a definition chains
+tools that already exist (fetch some JSON, download what it names, process the file):
+
+```json
+{
+  "functions": {
+    "fingerprint": {
+      "steps": [
+        { "name": "fetch",    "script": "tools/http-get-json" },
+        { "name": "download", "script": "tools/download", "stdout": "${job.id}/media" },
+        { "name": "fp",       "script": "tools/media-to-fp", "stdin": "${steps.download.stdout}" }
+      ],
+      "artifacts": { "include": ["${job.id}/*.fp"], "required": true },
+      "result": { "size": "${steps.fetch.result.size}" }
+    }
+  },
+  "fetch":    { "url": "https://media.example.com/recording/${params.uuid}", "token": "${secrets.token}" },
+  "download": { "url": "${steps.fetch.result.signedUrl}" }
+}
+```
+
+- Each step is a binding like a function: `name` (letters, digits, `_`, `-`), `script`, and
+  optionally `key` (its options block, default the step name), `positionals`,
+  `positionals_order`, `stdin`, `stdout` and `strict_params` (default the function's).
+  `artifacts` and `result` belong to the function, as do the caller-input rules: a caller
+  reaches a step only through `params` fields its options name.
+- Every step shares the context, `paths.work_dir` included, so files pass by path. A step's
+  outputs are `steps.NAME` in the templates after it: `exit_code`; `stdout`, the path of its
+  `stdout` file or else its collected output without trailing newlines; `result`, its
+  `result` role or else its collected stdout parsed as JSON; `meta`; `progress`. A
+  `${steps...}` path that isn't there, such as a later step or a field the JSON lacks, is an
+  error naming the step.
+- `previous` is the step that ran just before, on whichever path led here, with the same
+  fields plus its `name`: a handler several branches share reads `${previous.name}`,
+  `${previous.exit_code}`, `${previous.result.field}` or, after an `error` branch,
+  `${previous.error}`. The first step has none, so naming it there is an error.
+- After a step that succeeds the chain goes on to the next step, or ends after the last one.
+  A step that exits nonzero, times out or is stopped ends the chain; the run is that step's,
+  with `failedStep` naming it. A step that can't start (a template, stdin or schema error)
+  rejects the run. `continue_on_exit: [2]` on a step makes those codes a success, and the
+  chain then exits with the first of them (a partial result).
+- Branches change where a step leads: `succeed`, `fail` (a nonzero exit or timeout) and
+  `error` (it couldn't start) each name a later step, and `succeed: "$end"` ends the chain.
+  Branches only go forward, so a chain always ends. A step reached only through a branch,
+  such as a fallback or a cleanup, still sits in the list, so the step before it usually
+  needs a `succeed`. The chain's exit code is its last step's: a fallback that succeeds
+  makes the job succeed, and a cleanup that should leave it failed exits nonzero. A step
+  that took its `error` branch is `steps.NAME.error` (the message), with `exit_code: null`.
+  A cancelled chain, or one out of time, takes no branch.
+
+  ```json
+  "steps": [
+    { "name": "fetch",    "script": "tools/http-get-json", "fail": "mirror", "succeed": "process" },
+    { "name": "mirror",   "script": "tools/http-get-json" },
+    { "name": "process",  "script": "tools/media-to-fp" }
+  ]
+  ```
+- `timeoutMs` limits the whole chain and a step's `timeout_ms` that step; `signal` stops the
+  running step. `onProgress` gets each step's updates with `step` and a default `stage`.
+- The run is the last step's, plus `steps` (the outputs by name), `previous` (the last step)
+  and `stepRuns`. Render artifacts and the result with them through `runContext(context,
+  run)`, so a result map can use `${previous...}` whichever branch the chain ended on.
+- `functionParamNames(fn)` lists the `params` fields a function's templates use, steps
+  included, for describing its caller inputs.
+
 ```js
 import { resolveFunctionConfig, runScriptFunction, copyConfiguredArtifacts, buildResultRecord,
          newJobId, newJobRecord, transitionJobRecord, timestampIso } from 'clyops-jobs';
@@ -68,6 +153,7 @@ record = transitionJobRecord(record, { status: 'done', completedAt: timestampIso
 When the tool declares the well-known [roles](https://github.com/wankdanker/clyops/blob/main/spec/SPEC.md#14-roles-on-file-descriptors),
 the run wires them up: each `progress` object goes to `onProgress` (and the last one is the run's
 `progress`), `log` lines to `onLog`, and the `meta` document is parsed into the run's `meta`.
+The run's `result` is the `result` role parsed as JSON, or else the collected stdout when it is JSON.
 `updateJobProgress(record, update)` puts an update on a record as `progress` (and its `stage`, if
 any) without a history entry; the queue's tasks get it as `progress(update)`:
 

package/clyops-jobs.cjs

@@ -32,10 +32,13 @@ __export(index_exports, {
   deepMerge: () => deepMerge,
   ensureDir: () => ensureDir,
   findFilesByArtifactPattern: () => findFilesByArtifactPattern,
+  functionParamNames: () => functionParamNames,
+  functionTemplateOptions: () => functionTemplateOptions,
   globToRegex: () => globToRegex,
   isDirectoryPath: () => isDirectoryPath,
   isEnabled: () => isEnabled,
   isFilePath: () => isFilePath,
+  isUnsetTemplate: () => isUnsetTemplate,
   listFilesRecursive: () => listFilesRecursive,
   lookupTemplateValue: () => lookupTemplateValue,
   newJobId: () => newJobId,
@@ -51,9 +54,11 @@ __export(index_exports, {
   resolveConfigPath: () => resolveConfigPath,
   resolveFunctionConfig: () => resolveFunctionConfig,
   resolveScriptPath: () => resolveScriptPath,
+  runContext: () => runContext,
   runScriptFunction: () => runScriptFunction,
   shellQuote: () => shellQuote,
   statFollowingSymlinks: () => statFollowingSymlinks,
+  templateParamNames: () => templateParamNames,
   timestampIso: () => timestampIso,
   transitionJobRecord: () => transitionJobRecord,
   truthy: () => truthy,
@@ -135,40 +140,69 @@ function resolveConfigPath(root, value) {
 var resolveScriptPath = resolveConfigPath;
 
 // packages/jobs/src/template.ts
-function lookupTemplateValue(context, expression) {
+var OPTIONAL_ROOTS = ["params"];
+var TEMPLATE = /\$\{\s*([A-Za-z0-9_.-]+)\s*\}|\{\{\s*([A-Za-z0-9_.-]+)\s*\}\}/g;
+var MISSING = /* @__PURE__ */ Symbol("missing");
+function resolvePath(context, expression) {
   let current = context;
   for (const part of String(expression).split(".").filter(Boolean)) {
-    if (!current || typeof current !== "object" || !Object.prototype.hasOwnProperty.call(current, part)) {
-      throw new Error(`unknown template value: ${expression}`);
-    }
+    if (!current || typeof current !== "object" || !Object.prototype.hasOwnProperty.call(current, part)) return MISSING;
     current = current[part];
   }
+  return current;
+}
+function optionalRoot(expression, opts) {
+  return !opts?.strict && OPTIONAL_ROOTS.includes(String(expression).split(".").filter(Boolean)[0] ?? "");
+}
+function lookupTemplateValue(context, expression, opts) {
+  const current = resolvePath(context, expression);
+  if (current === MISSING) {
+    if (optionalRoot(expression, opts)) return "";
+    throw new Error(`unknown template value: ${expression}`);
+  }
   if (current === void 0 || current === null) return "";
   return typeof current === "object" ? JSON.stringify(current) : String(current);
 }
-function renderTemplateString(value, context) {
-  return String(value).replace(
-    /\$\{\s*([A-Za-z0-9_.-]+)\s*\}|\{\{\s*([A-Za-z0-9_.-]+)\s*\}\}/g,
-    (_, dollar, brace) => lookupTemplateValue(context, dollar || brace)
-  );
+function isUnsetTemplate(value, context, opts) {
+  if (typeof value !== "string") return false;
+  for (const [, dollar, brace] of value.matchAll(TEMPLATE)) {
+    const expression = dollar || brace;
+    if (!optionalRoot(expression, opts)) continue;
+    const current = resolvePath(context, expression);
+    if (current === MISSING || current === void 0 || current === null) return true;
+  }
+  return false;
+}
+function templateParamNames(value, names = /* @__PURE__ */ new Set()) {
+  if (typeof value === "string") {
+    for (const [, dollar, brace] of value.matchAll(TEMPLATE)) {
+      const [root, field] = String(dollar || brace).split(".").filter(Boolean);
+      if (OPTIONAL_ROOTS.includes(root ?? "") && field) names.add(field);
+    }
+  } else if (Array.isArray(value)) for (const item of value) templateParamNames(item, names);
+  else if (value && typeof value === "object") for (const item of Object.values(value)) templateParamNames(item, names);
+  return names;
+}
+function renderTemplateString(value, context, opts) {
+  return String(value).replace(TEMPLATE, (_, dollar, brace) => lookupTemplateValue(context, dollar || brace, opts));
 }
-function renderConfigValue(value, context) {
+function renderConfigValue(value, context, opts) {
   if (!context) return value;
-  if (typeof value === "string") return renderTemplateString(value, context);
-  if (Array.isArray(value)) return value.map((item) => renderConfigValue(item, context));
+  if (typeof value === "string") return renderTemplateString(value, context, opts);
+  if (Array.isArray(value)) return value.map((item) => renderConfigValue(item, context, opts));
   return value;
 }
-function renderResultValue(value, context) {
-  if (typeof value === "string") return renderTemplateString(value, context);
-  if (Array.isArray(value)) return value.map((item) => renderResultValue(item, context));
+function renderResultValue(value, context, opts) {
+  if (typeof value === "string") return renderTemplateString(value, context, opts);
+  if (Array.isArray(value)) return value.map((item) => renderResultValue(item, context, opts));
   if (value && typeof value === "object") {
-    return Object.fromEntries(Object.entries(value).map(([k, v]) => [k, renderResultValue(v, context)]));
+    return Object.fromEntries(Object.entries(value).map(([k, v]) => [k, renderResultValue(v, context, opts)]));
   }
   return value;
 }
-function renderResultMap(resultMap, context) {
+function renderResultMap(resultMap, context, opts) {
   if (!resultMap || typeof resultMap !== "object" || Array.isArray(resultMap)) return {};
-  return renderResultValue(resultMap, context);
+  return renderResultValue(resultMap, context, opts);
 }
 
 // packages/tools/src/schema.ts
@@ -236,6 +270,9 @@ function isPathValued(validation, type) {
 function isBareProgram(validation, value) {
   return validation === "file:executable" && value !== "" && !value.includes("/") && !(process.platform === "win32" && value.includes("\\"));
 }
+function oneJsonValue(validation, many) {
+  return (validation === "json" || validation.startsWith("json:")) && !many;
+}
 var TRUE = /^(true|1|yes|on)$/i;
 var FALSE = /^(false|0|no|off)$/i;
 function toArgv(schema, input, opts = {}) {
@@ -283,7 +320,7 @@ function toArgv(schema, input, opts = {}) {
       out.env[option.variableName] = text(found.value, isPathValued(option.validation, option.type), `--${option.name}`, option.validation);
       continue;
     }
-    for (const item of Array.isArray(found.value) ? found.value : [found.value]) {
+    for (const item of Array.isArray(found.value) && !oneJsonValue(option.validation, option.isArray) ? found.value : [found.value]) {
       out.argv.push(`--${option.name}`, text(item, isPathValued(option.validation, option.type), `--${option.name}`, option.validation));
     }
   }
@@ -309,7 +346,7 @@ function toArgv(schema, input, opts = {}) {
         positionals.push(text(gap.default, false));
       }
       skipped = [];
-      const values = Array.isArray(found.value) ? found.value : [found.value];
+      const values = Array.isArray(found.value) && !oneJsonValue(argument.validation, argument.isVariadic) ? found.value : [found.value];
       positionals.push(...values.map((v) => text(v, isPathValued(argument.validation), argument.name, argument.validation)));
     }
   }
@@ -591,31 +628,81 @@ var LOGGER = {
 function configureLogging(opts) {
   Object.assign(LOGGER, Object.fromEntries(Object.entries(opts).filter(([, v]) => v)));
 }
+var STEP_NAME = /^[A-Za-z0-9_-]+$/;
+var END = "$end";
 function resolveFunctionConfig(scriptRoot, routeConfig, functionName) {
   const definition = objectValue(objectValue(routeConfig.functions)[functionName]);
   if (Object.keys(definition).length === 0) return null;
   const key = String(definition.key || functionName);
+  if (definition.steps === void 0) return resolveBinding(scriptRoot, routeConfig, functionName, key, definition, `function ${functionName}`);
+  if (definition.script) throw new Error(`function ${functionName} defines both script and steps`);
+  if (!Array.isArray(definition.steps) || definition.steps.length === 0) throw new Error(`function ${functionName} defines no steps`);
+  const steps = [];
+  for (const [i, item] of definition.steps.entries()) {
+    const step = { strict_params: definition.strict_params, ...objectValue(item) };
+    const name = String(step.name ?? "");
+    if (!STEP_NAME.test(name)) throw new Error(`function ${functionName} step ${i + 1} needs a name of letters, digits, _ or -`);
+    if (steps.some((s) => s.stepName === name)) throw new Error(`function ${functionName} has two steps named ${name}`);
+    const where = `function ${functionName} step ${name}`;
+    steps.push({ ...resolveBinding(scriptRoot, routeConfig, functionName, String(step.key || name), step, where), stepName: name, definition: step });
+  }
+  for (const [i, step] of steps.entries()) {
+    for (const outcome of ["succeed", "fail", "error"]) {
+      const target = step.definition[outcome];
+      if (target === void 0 || outcome === "succeed" && target === END) continue;
+      const at = steps.findIndex((s) => s.stepName === target);
+      if (at < 0) throw new Error(`function ${functionName} step ${step.stepName}: ${outcome} names no step ${String(target)}`);
+      if (at <= i) throw new Error(`function ${functionName} step ${step.stepName}: ${outcome} must name a later step, not ${String(target)}`);
+    }
+  }
+  const config = Object.fromEntries(steps.map((s) => [s.stepName, s.config]));
+  return { functionName, key, script: steps[steps.length - 1].script, config, definition, steps };
+}
+function resolveBinding(scriptRoot, routeConfig, functionName, key, definition, where) {
   const script = String(definition.script || "");
-  if (!script) throw new Error(`function ${functionName} does not define a script`);
+  if (!script) throw new Error(`${where} does not define a script`);
   const config = deepMerge(objectValue(objectValue(routeConfig.defaults)[key]), objectValue(routeConfig[key]));
   return { functionName, key, script: resolveScriptPath(scriptRoot, script), config, definition };
 }
+function functionParamNames(fn) {
+  const names = /* @__PURE__ */ new Set();
+  for (const part of fn.steps ?? [fn]) {
+    const { positionals, stdin, stdout } = part.definition;
+    templateParamNames([part.config, positionals, stdin, stdout], names);
+  }
+  templateParamNames([fn.definition.artifacts, fn.definition.result], names);
+  return [...names].sort();
+}
+function functionTemplateOptions(fn) {
+  return { strict: fn.definition.strict_params === true };
+}
+function withoutUnsetParams(config, context, opts) {
+  const out = {};
+  for (const [key, value] of Object.entries(config)) {
+    if (isUnsetTemplate(value, context, opts)) continue;
+    out[key] = Array.isArray(value) ? value.filter((item) => !isUnsetTemplate(item, context, opts)) : value;
+  }
+  return out;
+}
 function explicitPositionals(fn, context) {
   if (!Array.isArray(fn.definition.positionals)) return void 0;
-  return fn.definition.positionals.map((item) => String(renderConfigValue(item, context))).filter(Boolean);
+  const templates = functionTemplateOptions(fn);
+  return fn.definition.positionals.filter((item) => !context || !isUnsetTemplate(item, context, templates)).map((item) => String(renderConfigValue(item, context, templates))).filter(Boolean);
 }
 async function buildFunctionCommand(fn, opts, schema) {
+  if (fn.steps) throw new Error(`function ${fn.functionName} runs steps; build each step's command`);
   schema ??= await loadSchema(fn.script);
-  const config = { ...fn.config };
+  const context = opts.context;
+  const templates = functionTemplateOptions(fn);
+  const config = context ? withoutUnsetParams(fn.config, context, templates) : { ...fn.config };
   const first = schema.arguments[0];
   if (opts.defaultPositional && first && !Array.isArray(fn.definition.positionals) && !inputKeys(first).some((k) => k in config)) {
     config[inputKeys(first)[0]] = opts.defaultPositional;
   }
-  const context = opts.context;
   const { argv, unknown, controlled } = toArgv(schema, config, {
     base: opts.configRoot,
     controlled: opts.controlled ?? ["help"],
-    render: context ? (s) => renderTemplateString(s, context) : void 0,
+    render: context ? (s) => renderTemplateString(s, context, templates) : void 0,
     positionals: explicitPositionals(fn, context),
     positionalsOrder: fn.definition.positionals_order
   });
@@ -632,19 +719,22 @@ function parseRoleJson(text) {
     return void 0;
   }
 }
-function streamPath(template, opts, base) {
+function streamPath(template, opts, base, templates) {
   if (!template) return void 0;
-  const rendered = opts.context ? renderTemplateString(template, opts.context) : template;
+  const rendered = opts.context ? renderTemplateString(template, opts.context, templates) : template;
   return (0, import_node_path3.isAbsolute)(rendered) ? rendered : (0, import_node_path3.resolve)(base, rendered);
 }
 async function runScriptFunction(fn, opts) {
+  return fn.steps ? runSteps(fn.steps, opts) : (await runTool(fn, opts)).run;
+}
+async function runTool(fn, opts) {
   const schema = await loadSchema(fn.script);
   const [file, ...argv] = await buildFunctionCommand(fn, opts, schema);
   const command = [file, ...redactArgv(schema, argv)];
   LOGGER.verbose("command: %s", shellQuote(command));
   const workDir = opts.context?.paths?.work_dir;
-  const stdinPath = streamPath(fn.definition.stdin, opts, opts.configRoot);
-  const stdoutPath = streamPath(fn.definition.stdout, opts, workDir ?? opts.configRoot);
+  const stdinPath = streamPath(fn.definition.stdin, opts, opts.configRoot, functionTemplateOptions(fn));
+  const stdoutPath = streamPath(fn.definition.stdout, opts, workDir ?? opts.configRoot, functionTemplateOptions(fn));
   let progress;
   const roles = {
     progress: {
@@ -656,8 +746,10 @@ async function runScriptFunction(fn, opts) {
       }
     },
     meta: {},
+    result: {},
     log: { onLine: opts.onLog }
   };
+  const stdin = stdinPath ? (0, import_node_fs4.createReadStream)(stdinPath, { fd: (0, import_node_fs4.openSync)(stdinPath, "r") }) : void 0;
   const started = start(file, argv, {
     cwd: opts.cwd,
     env: opts.env,
@@ -665,7 +757,7 @@ async function runScriptFunction(fn, opts) {
     killGraceMs: opts.killGraceMs,
     signal: opts.signal,
     onStderr: opts.onStderr,
-    stdin: stdinPath ? (0, import_node_fs4.createReadStream)(stdinPath) : void 0,
+    stdin,
     stdout: stdoutPath ? "stream" : "text",
     roles: declaredRoles({ schema }, roles)
   });
@@ -676,7 +768,70 @@ async function runScriptFunction(fn, opts) {
   }
   const [result] = await Promise.all([started.result, written]);
   const meta = parseRoleJson(result.roles?.meta);
-  return { ...result, command, stderrTail: tail(result.stderr), ...meta !== void 0 ? { meta } : {}, ...progress ? { progress } : {} };
+  const output = parseRoleJson(result.roles?.result) ?? (stdoutPath ? void 0 : parseRoleJson(result.stdout));
+  const run = {
+    ...result,
+    command,
+    stderrTail: tail(result.stderr),
+    ...meta !== void 0 ? { meta } : {},
+    ...progress ? { progress } : {},
+    ...output !== void 0 ? { result: output } : {}
+  };
+  return { run, stdoutPath };
+}
+async function runSteps(steps, opts) {
+  const deadline = opts.timeoutMs ? Date.now() + opts.timeoutMs : void 0;
+  const outputs = {};
+  const stepRuns = [];
+  const byName = new Map(steps.map((step, i) => [step.stepName, i]));
+  const after = (i, branch, fallthrough) => branch === END ? void 0 : branch ? byName.get(branch) : fallthrough && i + 1 < steps.length ? i + 1 : void 0;
+  let continued;
+  let last;
+  let previous;
+  for (let i = 0; i !== void 0; ) {
+    const step = steps[i];
+    const name = step.stepName;
+    const limits = [step.definition.timeout_ms, deadline && Math.max(1, deadline - Date.now())].filter((ms) => Number(ms) > 0);
+    const context = { ...opts.context, steps: { ...outputs }, ...previous ? { previous } : {} };
+    const onProgress = opts.onProgress && ((update) => opts.onProgress?.({ stage: name, ...update, step: name }));
+    let ran;
+    try {
+      ran = await runTool(step, { ...opts, context, onProgress, timeoutMs: limits.length ? Math.min(...limits) : void 0 });
+    } catch (error) {
+      const message = `step ${name}: ${error.message}`;
+      if (!step.definition.error) throw new Error(message, { cause: error });
+      outputs[name] = { exit_code: null, stdout: "", error: message };
+      previous = { name, ...outputs[name] };
+      i = byName.get(step.definition.error);
+      continue;
+    }
+    const { run, stdoutPath } = ran;
+    stepRuns.push({ ...run, step: name });
+    outputs[name] = {
+      exit_code: run.exitCode,
+      stdout: stdoutPath ?? run.stdout.replace(/\n+$/, ""),
+      ...run.result !== void 0 ? { result: run.result } : {},
+      ...run.meta !== void 0 ? { meta: run.meta } : {},
+      ...run.progress ? { progress: run.progress } : {}
+    };
+    previous = { name, ...outputs[name] };
+    last = { ...run, steps: outputs, previous, stepRuns, durationMs: stepRuns.reduce((ms, r) => ms + r.durationMs, 0) };
+    const ok = !run.timedOut && (run.exitCode === 0 || run.exitCode !== null && (step.definition.continue_on_exit ?? []).includes(run.exitCode));
+    if (ok) {
+      if (run.exitCode !== 0) continued ??= run.exitCode ?? void 0;
+      i = after(i, step.definition.succeed, true);
+      if (i === void 0) return { ...last, exitCode: run.exitCode || continued || 0 };
+      continue;
+    }
+    const stopped = opts.signal?.aborted || deadline !== void 0 && Date.now() >= deadline;
+    i = stopped ? void 0 : after(i, step.definition.fail, false);
+    if (i === void 0) return { ...last, failedStep: name };
+  }
+  if (!last) throw new Error("unreachable: a chain ends after a step that ran");
+  return last;
+}
+function runContext(context, run) {
+  return { ...context, ...run.steps ? { steps: run.steps } : {}, ...run.previous ? { previous: run.previous } : {} };
 }
 function buildResultRecord(p) {
   return {
@@ -688,7 +843,7 @@ function buildResultRecord(p) {
     artifact_keys: p.artifactKeys,
     completed_at: p.now,
     config: p.fn.config,
-    ...renderResultMap(p.fn.definition.result, p.context)
+    ...renderResultMap(p.fn.definition.result, p.context, functionTemplateOptions(p.fn))
   };
 }
 
@@ -734,8 +889,8 @@ function listFilesRecursive(dir, visited = /* @__PURE__ */ new Set()) {
   }
   return files;
 }
-function findFilesByArtifactPattern(pattern, context) {
-  const rendered = renderTemplateString(pattern, context);
+function findFilesByArtifactPattern(pattern, context, opts) {
+  const rendered = renderTemplateString(pattern, context, opts);
   const absolute = (0, import_node_path4.isAbsolute)(rendered) ? rendered : (0, import_node_path4.resolve)(context.paths.work_dir, rendered);
   if (!/[*?]/.test(absolute)) return isFilePath(absolute) ? [absolute] : [];
   const regex = globToRegex(absolute);
@@ -774,7 +929,7 @@ function copyConfiguredArtifacts(fn, context, opts = {}) {
   const seen = /* @__PURE__ */ new Set();
   const copied = [];
   for (const pattern of patterns) {
-    for (const source of findFilesByArtifactPattern(pattern, context)) {
+    for (const source of findFilesByArtifactPattern(pattern, context, { strict: fn.definition.strict_params === true })) {
       if (seen.has(source)) continue;
       seen.add(source);
       const destination = (0, import_node_path4.join)(dir, (0, import_node_path4.basename)(source));
@@ -924,10 +1079,13 @@ var JobQueue = class {
   deepMerge,
   ensureDir,
   findFilesByArtifactPattern,
+  functionParamNames,
+  functionTemplateOptions,
   globToRegex,
   isDirectoryPath,
   isEnabled,
   isFilePath,
+  isUnsetTemplate,
   listFilesRecursive,
   lookupTemplateValue,
   newJobId,
@@ -943,9 +1101,11 @@ var JobQueue = class {
   resolveConfigPath,
   resolveFunctionConfig,
   resolveScriptPath,
+  runContext,
   runScriptFunction,
   shellQuote,
   statFollowingSymlinks,
+  templateParamNames,
   timestampIso,
   transitionJobRecord,
   truthy,

package/config.schema.json

@@ -0,0 +1,113 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://clyops.io/schemas/clyops-jobs.json",
+  "title": "clyops-jobs config",
+  "description": "Functions bound to clyops tools, their default options and their option blocks. Every other top-level key is an options block, named by a function's or step's `key`.",
+  "type": "object",
+  "properties": {
+    "$schema": { "type": "string", "description": "This schema's URL, for editors: https://clyops.io/schemas/clyops-jobs.json" },
+    "functions": {
+      "type": "object",
+      "description": "The functions a trigger can run, by name.",
+      "additionalProperties": { "$ref": "#/$defs/function" }
+    },
+    "defaults": {
+      "type": "object",
+      "description": "Default options by block key, merged under the block of the same key.",
+      "additionalProperties": { "$ref": "#/$defs/options" }
+    }
+  },
+  "additionalProperties": { "$ref": "#/$defs/options" },
+  "$defs": {
+    "template": {
+      "type": "string",
+      "description": "Text with `${dot.path}` or `{{dot.path}}` templates rendered against the trigger's context (`job`, `paths`, `params`, `steps`, ...)."
+    },
+    "stepName": { "type": "string", "pattern": "^[A-Za-z0-9_-]+$" },
+    "options": {
+      "type": "object",
+      "description": "Tool options keyed like the tool's options (`out_dir` or `out-dir`) or arguments, mapped onto its command line through --help-json-schema. String values are templates; an option naming a missing `params` field is left unset."
+    },
+    "binding": {
+      "type": "object",
+      "properties": {
+        "key": { "type": "string", "description": "The options block (a top-level key and a `defaults` key); default: the function's or step's name." },
+        "script": { "type": "string", "description": "The tool, relative to the script root." },
+        "positionals": {
+          "type": "array",
+          "description": "The tool's positional arguments as templates, in order; `[]` for none. Without it, arguments are options keyed by argument name.",
+          "items": { "type": ["string", "number", "boolean"] }
+        },
+        "positionals_order": { "enum": ["first", "last"], "description": "Positionals before (default) or after the options." },
+        "stdin": { "$ref": "#/$defs/template", "description": "A template naming the file fed to the tool's stdin, relative to the config root." },
+        "stdout": { "$ref": "#/$defs/template", "description": "A template naming the file the tool's stdout is written to, relative to `paths.work_dir`." },
+        "strict_params": { "type": "boolean", "description": "A missing `params` field is an error instead of leaving the option unset (default false)." }
+      },
+      "patternProperties": { "^x-": { "description": "Free for triggers and comments; clyops-jobs ignores it." } }
+    },
+    "function": {
+      "description": "A function runs one tool (`script`) or several in order (`steps`).",
+      "type": "object",
+      "allOf": [{ "$ref": "#/$defs/binding" }],
+      "properties": {
+        "key": true, "script": true, "positionals": true, "positionals_order": true, "stdin": true, "stdout": true, "strict_params": true,
+        "steps": {
+          "type": "array",
+          "minItems": 1,
+          "description": "Tools run in order with one context; each step's outputs are `steps.NAME` (`exit_code`, `stdout`, `result`, `meta`, `progress`, `error`) in the templates after it, and the step that ran just before is `previous` (the same fields plus `name`).",
+          "items": { "$ref": "#/$defs/step" }
+        },
+        "artifacts": { "$ref": "#/$defs/artifacts" },
+        "result": { "type": "object", "description": "Templates added to the result record; nested objects and arrays are rendered too." }
+      },
+      "patternProperties": { "^x-": true },
+      "additionalProperties": false,
+      "oneOf": [
+        { "required": ["script"], "not": { "required": ["steps"] } },
+        { "required": ["steps"], "not": { "required": ["script"] } }
+      ]
+    },
+    "step": {
+      "description": "One tool in a function's `steps`.",
+      "type": "object",
+      "allOf": [{ "$ref": "#/$defs/binding" }],
+      "required": ["name", "script"],
+      "properties": {
+        "key": true, "script": true, "positionals": true, "positionals_order": true, "stdin": true, "stdout": true, "strict_params": true,
+        "name": { "$ref": "#/$defs/stepName", "description": "Names the step in `${steps.NAME...}` templates, branches and errors." },
+        "timeout_ms": { "type": "integer", "exclusiveMinimum": 0, "description": "This step's time limit; the whole chain also stays within the run's timeoutMs." },
+        "continue_on_exit": {
+          "type": "array",
+          "items": { "type": "integer" },
+          "description": "Exit codes besides 0 that count as success; the chain then reports the first of them, e.g. [2] for a partial result."
+        },
+        "succeed": {
+          "anyOf": [{ "$ref": "#/$defs/stepName" }, { "const": "$end" }],
+          "description": "The later step to run after this one succeeds, instead of the next; `$end` ends the chain."
+        },
+        "fail": { "$ref": "#/$defs/stepName", "description": "The later step to run when this one exits nonzero or times out, instead of ending the chain." },
+        "error": { "$ref": "#/$defs/stepName", "description": "The later step to run when this one can't start (a template, stdin or schema error), instead of failing the run." }
+      },
+      "patternProperties": { "^x-": true },
+      "additionalProperties": false
+    },
+    "artifacts": {
+      "description": "Files copied from `paths.work_dir` into `paths.artifacts_dir` after the run: glob patterns (`*`, `?`, `**`), or an object with them and a requirement.",
+      "anyOf": [
+        { "$ref": "#/$defs/template" },
+        { "type": "array", "items": { "$ref": "#/$defs/template" } },
+        {
+          "type": "object",
+          "properties": {
+            "include": { "anyOf": [{ "$ref": "#/$defs/template" }, { "type": "array", "items": { "$ref": "#/$defs/template" } }], "description": "Glob patterns relative to `paths.work_dir`." },
+            "includes": { "$ref": "#/$defs/artifacts/anyOf/2/properties/include", "description": "Same as include." },
+            "patterns": { "$ref": "#/$defs/artifacts/anyOf/2/properties/include", "description": "Same as include." },
+            "required": { "type": ["boolean", "string"], "description": "At least one file must match." },
+            "min": { "type": "integer", "minimum": 0, "description": "At least this many files must match." }
+          },
+          "additionalProperties": false
+        }
+      ]
+    }
+  }
+}

package/dist/artifacts.d.ts

@@ -1,5 +1,5 @@
 import type { ResolvedFunction } from './functions.js';
-import { type TemplateContext } from './template.js';
+import { type TemplateContext, type TemplateOptions } from './template.js';
 /** Compile an artifact glob into an anchored regular expression. */
 export declare function globToRegex(pattern: string): RegExp;
 /** Every file below `dir`, following symlinks without looping. */
@@ -9,7 +9,7 @@ export declare function findFilesByArtifactPattern(pattern: string, context: Tem
     paths: {
         work_dir: string;
     };
-}): string[];
+}, opts?: TemplateOptions): string[];
 /** A function's `artifacts` as a list of include patterns, plus its `stdout` file. */
 export declare function artifactIncludePatterns(fn: ResolvedFunction): string[];
 /** How many artifacts a function must produce: `min`, or 1 for `required: true`. */

package/dist/artifacts.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"artifacts.d.ts","sourceRoot":"","sources":["../src/artifacts.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,gBAAgB,CAAC;AACvD,OAAO,EAAwB,KAAK,eAAe,EAAE,MAAM,eAAe,CAAC;AAK3E,oEAAoE;AACpE,wBAAgB,WAAW,CAAC,OAAO,EAAE,MAAM,GAAG,MAAM,CAiBnD;AAWD,kEAAkE;AAClE,wBAAgB,kBAAkB,CAAC,GAAG,EAAE,MAAM,EAAE,OAAO,cAAoB,GAAG,MAAM,EAAE,CAarF;AAED,oFAAoF;AACpF,wBAAgB,0BAA0B,CAAC,OAAO,EAAE,MAAM,EAAE,OAAO,EAAE,eAAe,GAAG;IAAE,KAAK,EAAE;QAAE,QAAQ,EAAE,MAAM,CAAA;KAAE,CAAA;CAAE,GAAG,MAAM,EAAE,CAQhI;AAED,sFAAsF;AACtF,wBAAgB,uBAAuB,CAAC,EAAE,EAAE,gBAAgB,GAAG,MAAM,EAAE,CAatE;AAED,oFAAoF;AACpF,wBAAgB,mBAAmB,CAAC,EAAE,EAAE,gBAAgB,GAAG,MAAM,CAMhE;AAED,iFAAiF;AACjF,MAAM,WAAW,cAAc;IAC7B,mDAAmD;IACnD,MAAM,EAAE,MAAM,CAAC;IACf,qDAAqD;IACrD,GAAG,EAAE,MAAM,CAAC;CACb;AAED;;;;GAIG;AACH,wBAAgB,uBAAuB,CACrC,EAAE,EAAE,gBAAgB,EACpB,OAAO,EAAE,eAAe,GAAG;IAAE,KAAK,EAAE;QAAE,QAAQ,EAAE,MAAM,CAAC;QAAC,aAAa,EAAE,MAAM,CAAA;KAAE,CAAA;CAAE,EACjF,IAAI,GAAE;IAAE,SAAS,CAAC,EAAE,MAAM,CAAA;CAAO,GAChC,cAAc,EAAE,CAwBlB"}
+{"version":3,"file":"artifacts.d.ts","sourceRoot":"","sources":["../src/artifacts.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,gBAAgB,CAAC;AACvD,OAAO,EAAwB,KAAK,eAAe,EAAE,KAAK,eAAe,EAAE,MAAM,eAAe,CAAC;AAKjG,oEAAoE;AACpE,wBAAgB,WAAW,CAAC,OAAO,EAAE,MAAM,GAAG,MAAM,CAiBnD;AAWD,kEAAkE;AAClE,wBAAgB,kBAAkB,CAAC,GAAG,EAAE,MAAM,EAAE,OAAO,cAAoB,GAAG,MAAM,EAAE,CAarF;AAED,oFAAoF;AACpF,wBAAgB,0BAA0B,CAAC,OAAO,EAAE,MAAM,EAAE,OAAO,EAAE,eAAe,GAAG;IAAE,KAAK,EAAE;QAAE,QAAQ,EAAE,MAAM,CAAA;KAAE,CAAA;CAAE,EAAE,IAAI,CAAC,EAAE,eAAe,GAAG,MAAM,EAAE,CAQxJ;AAED,sFAAsF;AACtF,wBAAgB,uBAAuB,CAAC,EAAE,EAAE,gBAAgB,GAAG,MAAM,EAAE,CAatE;AAED,oFAAoF;AACpF,wBAAgB,mBAAmB,CAAC,EAAE,EAAE,gBAAgB,GAAG,MAAM,CAMhE;AAED,iFAAiF;AACjF,MAAM,WAAW,cAAc;IAC7B,mDAAmD;IACnD,MAAM,EAAE,MAAM,CAAC;IACf,qDAAqD;IACrD,GAAG,EAAE,MAAM,CAAC;CACb;AAED;;;;GAIG;AACH,wBAAgB,uBAAuB,CACrC,EAAE,EAAE,gBAAgB,EACpB,OAAO,EAAE,eAAe,GAAG;IAAE,KAAK,EAAE;QAAE,QAAQ,EAAE,MAAM,CAAC;QAAC,aAAa,EAAE,MAAM,CAAA;KAAE,CAAA;CAAE,EACjF,IAAI,GAAE;IAAE,SAAS,CAAC,EAAE,MAAM,CAAA;CAAO,GAChC,cAAc,EAAE,CAwBlB"}