@usearazzo/runner 1.0.1-alpha.20 → 1.0.1-alpha.21

package/CHANGELOG.md

@@ -3,6 +3,25 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+## [1.0.1-alpha.21](https://github.com/usearazzo/arazzo-toolkit/compare/v1.0.1-alpha.20...v1.0.1-alpha.21) (2026-11-03)
+
+### Features
+
+- **runner:** record criterion outcomes in step run records ([#302](https://github.com/usearazzo/arazzo-toolkit/issues/302)) ([513516b](https://github.com/usearazzo/arazzo-toolkit/commit/513516b80fc7edecae2d3fce74a53e000a494982)), closes [#231](https://github.com/usearazzo/arazzo-toolkit/issues/231)
+
+### BREAKING CHANGES
+
+- **runner:** CriterionEvaluator.evaluate returns a CriterionOutcome
+  ({ met, error? }) instead of a boolean. StepExecutor.evaluateCriteria
+  returns CriterionEvaluationRecord[] instead of a boolean, and
+  StepExecutor.selectActions returns { matchedActions, actionCriteria }
+  instead of the action list. CriterionPredicate gains a second `action`
+  argument and is called for every criterion of an action. StepRunRecord
+  and StepExecutionResult gain required `successCriteria` and
+  `actionCriteria` fields. Any error a criterion throws other than an
+  unevaluable condition, such as a malformed criterion, now aborts the run
+  even after an earlier unmet criterion.
+
 ## [1.0.1-alpha.20](https://github.com/usearazzo/arazzo-toolkit/compare/v1.0.1-alpha.19...v1.0.1-alpha.20) (2026-11-01)
 
 ### Bug Fixes

package/README.md

@@ -20,7 +20,7 @@
 
 An Arazzo document describes a multi-step API workflow: which operations to call, in what order, what to pass from one step to the next, and what counts as success. This package runs it.
 
-It loads the workflow and the OpenAPI documents its source descriptions point at. For each step it builds a real HTTP request, sends it to the live API, checks the step's success criteria against the response, and follows the control flow the document declares. When the run ends, you get the workflow's outputs and a trace of every step: whether it passed, which action was picked, how many attempts it took, and how long it ran.
+It loads the workflow and the OpenAPI documents its source descriptions point at. For each step it builds a real HTTP request, sends it to the live API, checks the step's success criteria against the response, and follows the control flow the document declares. When the run ends, you get the workflow's outputs and a trace of every step: whether it passed, which action was picked, how many attempts it took, how long it ran, and the outcome of each criterion it evaluated.
 
 Use it to test an API flow end to end in CI, to drive a workflow from your own service, or to let an agent run a workflow instead of improvising the calls. It handles:
 
@@ -44,7 +44,7 @@ A runner is four building blocks, composed bottom-up; `ArazzoRunner` composes th
   <img alt="Runner architecture: WorkflowExecutor runs steps through StepExecutor, which runs OpenAPI operations through OpenAPIOperationExecutor and the HTTPClient; DocumentRegistry supplies documents to all three executors" src="https://raw.githubusercontent.com/usearazzo/arazzo-toolkit/main/packages/runner/assets/architecture.svg">
 </p>
 
-A step whose criteria go unmet is a result, not an error: the run resolves with `status: 'failed'`. A criterion whose condition cannot be evaluated (an invalid regex, JSONPath, XPath or simple condition, or non-XML data under an `xpath` query) counts as unmet, as Arazzo 1.1.0 requires, rather than throwing. The type-specific criterion evaluators raise such a condition as `CriterionEvaluationError`, a `CriterionError` subclass, and every `CriterionError` carries a `reason` from `CriterionErrorReason`. Only a document that cannot be loaded or run as written, a request that cannot be built or sent, or a cancelled run throws, and every such error extends `ArazzoRunnerError`; a document that cannot be read or loaded (a missing file, an HTTP error, an unresolvable reference) throws `DocumentLoadError` with the underlying failure as its `cause`.
+A step whose criteria go unmet is a result, not an error: the run resolves with `status: 'failed'`. A criterion whose condition cannot be evaluated (an invalid regex, JSONPath, XPath or simple condition, or non-XML data under an `xpath` query) counts as unmet, as Arazzo 1.1.0 requires, rather than throwing. So does a criterion of a step that calls a workflow when it reads `$statusCode`, `$url`, `$method`, `$request` or `$response`: that step made no HTTP call. Each step record's `successCriteria` and `actionCriteria` list every criterion its last attempt evaluated, as a `CriterionEvaluationRecord` with the condition, whether it was met, the action's name for an action criterion and, when it could not be evaluated, a `diagnostic` with a `code` and `message`, never request or response data. The type-specific criterion evaluators raise an unevaluable condition as `CriterionEvaluationError`, a `CriterionError` subclass, and every `CriterionError` carries a `reason` from `CriterionErrorReason`. Only a document that cannot be loaded or run as written, a request that cannot be built or sent, or a cancelled run throws, and every such error extends `ArazzoRunnerError`; a document that cannot be read or loaded (a missing file, an HTTP error, an unresolvable reference) throws `DocumentLoadError` with the underlying failure as its `cause`.
 
 ## Installation
 

package/dist/arazzo-runner.browser.js

@@ -8960,10 +8960,11 @@ __webpack_require__.r(__webpack_exports__);
 
 
 /**
- * Determines whether a single criterion is met.
+ * Determines whether a single criterion of an action is met. Receives the
+ * action declaring the criterion, so a caller can attribute the outcome to it.
  *
  * Bridged to a criterion evaluator
- * (`(criterion) => criterionEvaluator.evaluate(criterion, resolve)`); an
+ * (`(criterion) => criterionEvaluator.evaluate(criterion, resolve).met`); an
  * unresolvable/failing criterion is `false`, not an error.
  * @public
  */
@@ -9095,6 +9096,7 @@ class ActionResolver {
         reason: 'malformed-criteria'
       });
     }
+    let met = true;
     for (const criterion of criteria) {
       if (!(0,_speclynx_apidom_ns_arazzo_1__WEBPACK_IMPORTED_MODULE_1__.isCriterionElement)(criterion)) {
         throw new _errors_ResolverError_ts__WEBPACK_IMPORTED_MODULE_2__["default"]("An action's `criteria` contains an entry that is not a Criterion Object", {
@@ -9103,9 +9105,11 @@ class ActionResolver {
           reason: 'malformed-criterion'
         });
       }
-      if (!isCriterionMet(criterion)) return false;
+      // every criterion is evaluated, not only up to the first unmet one, so
+      // a predicate that records outcomes sees the whole list.
+      if (!isCriterionMet(criterion, action)) met = false;
     }
-    return true;
+    return met;
   }
 }
 /* harmony default export */ const __WEBPACK_DEFAULT_EXPORT__ = (ActionResolver);
@@ -10034,6 +10038,11 @@ __webpack_require__.r(__webpack_exports__);
  * @public
  */
 
+/**
+ * The outcome of evaluating one criterion.
+ * @public
+ */
+
 /**
  * Options for the CriterionEvaluator.
  *
@@ -10044,7 +10053,7 @@ __webpack_require__.r(__webpack_exports__);
  */
 
 /**
- * Evaluates an Arazzo Criterion Object to a boolean.
+ * Evaluates an Arazzo Criterion Object to a {@link CriterionOutcome}.
  *
  * Dispatches on the criterion's `type` (defaulting to `simple` when omitted):
  * - `simple` — evaluates the condition directly, resolving the runtime
@@ -10064,12 +10073,13 @@ __webpack_require__.r(__webpack_exports__);
  * the 1.1.0 rule to every document, which accepts more but never evaluates a
  * query under the wrong dialect.
  *
- * A condition that cannot be evaluated fails the criterion (`false`), as
+ * A condition that cannot be evaluated fails the criterion (`met: false`), as
  * Arazzo 1.1.0 requires: a syntax error in the condition, a context the
  * condition cannot apply to (non-XML data for `xpath`), or an XPath evaluation
  * error. The type-specific evaluators report these as a
- * {@link CriterionEvaluationError}, which this facade absorbs; any other error
- * they raise propagates. A criterion that is malformed as written still throws: a
+ * {@link CriterionEvaluationError}, which this facade returns as the
+ * outcome's `error` so the caller can report it; any other error they raise
+ * propagates. A criterion that is malformed as written still throws: a
  * missing `condition`, an unknown or unreadable `type`, an unsupported
  * expression `version`, or a `context` that is missing or not a runtime
  * expression.
@@ -10093,7 +10103,8 @@ class CriterionEvaluator {
   }
 
   /**
-   * Evaluates a criterion element, returning whether the condition is met.
+   * Evaluates a criterion element, returning whether the condition is met and,
+   * when it cannot be evaluated, why.
    *
    * `resolve` turns a runtime expression into a value: for `simple` it resolves
    * the expressions embedded in the condition; for the other types it resolves
@@ -10115,23 +10126,28 @@ class CriterionEvaluator {
     }
     const type = this.#resolveType(criterion);
     this.#requireValidContext(criterion, type);
+    let met;
     try {
       switch (type) {
         case 'simple':
-          return this.#simple.evaluate(condition, resolve);
+          met = this.#simple.evaluate(condition, resolve);
+          break;
         case 'regex':
-          return this.#regex.evaluate(condition, this.#resolveContext(criterion, type, resolve));
+          met = this.#regex.evaluate(condition, this.#resolveContext(criterion, type, resolve));
+          break;
         case 'jsonpath':
           // only rfc9535 is supported, Arazzo 1.1.0's default, so an omitted
           // version is accepted. draft-goessner has different semantics and is
           // rejected rather than mis-evaluated.
           this.#requireVersion(criterion, type, condition, 'rfc9535');
-          return this.#jsonpath.evaluate(condition, this.#resolveContext(criterion, type, resolve));
+          met = this.#jsonpath.evaluate(condition, this.#resolveContext(criterion, type, resolve));
+          break;
         case 'xpath':
           // only XPath 3.1 is supported; xpath-31 is Arazzo 1.1.0's default, so
           // an omitted version is accepted. Older versions are rejected.
           this.#requireVersion(criterion, type, condition, 'xpath-31');
-          return this.#xpath.evaluate(condition, this.#resolveContext(criterion, type, resolve));
+          met = this.#xpath.evaluate(condition, this.#resolveContext(criterion, type, resolve));
+          break;
         default:
           throw new _errors_CriterionError_ts__WEBPACK_IMPORTED_MODULE_4__["default"](`Unknown criterion type "${type}"`, {
             condition,
@@ -10142,9 +10158,15 @@ class CriterionEvaluator {
     } catch (error) {
       // Arazzo 1.1.0: a condition that cannot be evaluated MUST evaluate to fail.
       // The structural checks throw a plain CriterionError and still propagate.
-      if (error instanceof _errors_CriterionEvaluationError_ts__WEBPACK_IMPORTED_MODULE_5__["default"]) return false;
+      if (error instanceof _errors_CriterionEvaluationError_ts__WEBPACK_IMPORTED_MODULE_5__["default"]) return {
+        met: false,
+        error
+      };
       throw error;
     }
+    return {
+      met
+    };
   }
 
   /**
@@ -10281,7 +10303,10 @@ class JSONPathCriterionEvaluator {
       return (0,_swaggerexpert_jsonpath__WEBPACK_IMPORTED_MODULE_0__.evaluate)(context, condition).length > 0;
     } catch (error) {
       if (error instanceof _swaggerexpert_jsonpath__WEBPACK_IMPORTED_MODULE_0__.JSONPathError) {
-        throw new _errors_CriterionEvaluationError_ts__WEBPACK_IMPORTED_MODULE_1__["default"](`Invalid jsonpath criterion condition "${condition}"`, {
+        // a parse error describes the query only; an evaluation error ran
+        // against the context, so its message is not surfaced.
+        const detail = error instanceof _swaggerexpert_jsonpath__WEBPACK_IMPORTED_MODULE_0__.JSONPathParseError ? `: ${error.message}` : '';
+        throw new _errors_CriterionEvaluationError_ts__WEBPACK_IMPORTED_MODULE_1__["default"](`Invalid jsonpath criterion condition "${condition}"${detail}`, {
           cause: error,
           condition,
           type: 'jsonpath',
@@ -10346,7 +10371,9 @@ class RegexCriterionEvaluator {
     try {
       regExp = new RegExp(condition);
     } catch (error) {
-      throw new _errors_CriterionEvaluationError_ts__WEBPACK_IMPORTED_MODULE_0__["default"](`Invalid regex criterion condition "${condition}"`, {
+      // the engine has seen only the pattern, never the context, so its
+      // message is safe to surface.
+      throw new _errors_CriterionEvaluationError_ts__WEBPACK_IMPORTED_MODULE_0__["default"](`Invalid regex criterion condition "${condition}": ${error.message}`, {
         cause: error,
         condition,
         type: 'regex',
@@ -10486,6 +10513,9 @@ class XPathCriterionEvaluator {
         language: evaluateXPath.XPATH_3_1_LANGUAGE
       });
     } catch (error) {
+      // no engine detail: the query has run against the context, and it can
+      // put context values into its own error (a dynamic error, or fn:error
+      // with a code and description of the query's choosing).
       throw new _errors_CriterionEvaluationError_ts__WEBPACK_IMPORTED_MODULE_2__["default"](`Invalid xpath criterion condition "${condition}"`, {
         cause: error,
         condition,
@@ -12300,6 +12330,12 @@ const STEP_TARGET_FIELDS = ['operationId', 'operationPath', 'workflowId'];
  * @public
  */
 
+/**
+ * The outcome of one criterion a step evaluated: one of its `successCriteria`,
+ * or one of the `criteria` of an action it considered.
+ * @public
+ */
+
 /**
  * The outcome of executing a single step.
  * @public
@@ -12446,16 +12482,22 @@ class StepExecutor {
     // evaluate criteria, outputs, and the next action against the post-request
     // context (with $request / $url / $method and $response / $statusCode).
     const postContext = state.toContext(this.#requestContext(response), this.#responseContext(response));
-    const successful = this.evaluateCriteria(step, postContext);
+    const successCriteria = this.evaluateCriteria(step, postContext);
+    const successful = successCriteria.every(criterion => criterion.met);
     const outputs = this.#outputResolver.resolve(step.outputs, expression => this.#evaluate(postContext, expression), scope);
-    const matchedActions = this.selectActions(step, successful, postContext);
+    const {
+      matchedActions,
+      actionCriteria
+    } = this.selectActions(step, successful, postContext);
     return {
       stepId,
       response,
       successful,
       outputs,
       action: matchedActions[0],
-      matchedActions
+      matchedActions,
+      successCriteria,
+      actionCriteria
     };
   }
 
@@ -12528,9 +12570,12 @@ class StepExecutor {
   }
 
   /**
-   * Evaluates the step's `successCriteria`; all criteria must pass. A step with
-   * no criteria succeeds on a received response. An unreadable
-   * `successCriteria` throws, see {@link StepExecutor.assertSuccessCriteria}.
+   * Evaluates the step's `successCriteria`, returning one record per criterion
+   * in list order; the step succeeds when every record is met. Every criterion
+   * is evaluated, so the records explain each unmet one, not only the first. A
+   * step with no criteria returns no records and succeeds on a received
+   * response. An unreadable `successCriteria` throws, see
+   * {@link StepExecutor.assertSuccessCriteria}.
    *
    * Public for the same reason as {@link StepExecutor.selectActions}: a step
    * targeting a `workflowId` is run by the workflow executor and never reaches
@@ -12540,15 +12585,16 @@ class StepExecutor {
   evaluateCriteria(step, context) {
     this.assertSuccessCriteria(step);
     const criteria = step.successCriteria;
-    if (criteria === undefined) return true;
+    if (criteria === undefined) return [];
 
     // assertSuccessCriteria above guarantees every entry is a Criterion Object
-    return [...criteria].every(criterion => this.#criterionEvaluator.evaluate(criterion, expression => this.#evaluate(context, expression)));
+    return [...criteria].map(criterion => this.#evaluateCriterion(criterion, context));
   }
 
   /**
    * Selects every matching `onSuccess` / `onFailure` action for the outcome, in
-   * list order, gating each action's criteria against the context.
+   * list order, gating each action's criteria against the context, and records
+   * the outcome of every action criterion evaluated.
    *
    * The step's own list is the whole story here: a step that declares none of
    * its own already carries the workflow-level defaults, inherited by
@@ -12567,25 +12613,65 @@ class StepExecutor {
    * the expression context through this one method rather than wiring their own.
    */
   selectActions(step, successful, context) {
-    const isCriterionMet = criterion => {
-      try {
-        return this.#criterionEvaluator.evaluate(criterion, expression => this.#evaluate(context, expression));
-      } catch (error) {
-        // an action's criteria decide whether it applies to this outcome, and
-        // actions are shared — inherited from the workflow, reused from
-        // components. A condition on an HTTP exchange cannot apply to a step
-        // that calls a workflow, so the action does not match; the run does not
-        // abort over a default written for the operation steps.
-        if (error instanceof _errors_RuntimeExpressionError_ts__WEBPACK_IMPORTED_MODULE_12__["default"] && error.reason === 'unavailable-expression') {
-          return false;
-        }
-        throw error;
-      }
+    const actionCriteria = [];
+    const isCriterionMet = (criterion, action) => {
+      const record = {
+        ...this.#evaluateCriterion(criterion, context),
+        actionName: (0,_speclynx_apidom_core__WEBPACK_IMPORTED_MODULE_0__["default"])(action.name)
+      };
+      actionCriteria.push(record);
+      return record.met;
     };
     const scope = {
       stepId: (0,_speclynx_apidom_core__WEBPACK_IMPORTED_MODULE_0__["default"])(step.stepId)
     };
-    return successful ? this.#actionResolver.resolveOnSuccess(step.onSuccess, isCriterionMet, scope) : this.#actionResolver.resolveOnFailure(step.onFailure, isCriterionMet, scope);
+    const matchedActions = successful ? this.#actionResolver.resolveOnSuccess(step.onSuccess, isCriterionMet, scope) : this.#actionResolver.resolveOnFailure(step.onFailure, isCriterionMet, scope);
+    return {
+      matchedActions,
+      actionCriteria
+    };
+  }
+
+  /**
+   * Evaluates one criterion into a {@link CriterionEvaluationRecord}. A
+   * condition that cannot be evaluated is unmet and records why; any other
+   * error propagates.
+   */
+  #evaluateCriterion(criterion, context) {
+    const condition = (0,_speclynx_apidom_core__WEBPACK_IMPORTED_MODULE_0__["default"])(criterion.condition);
+    try {
+      const {
+        met,
+        error
+      } = this.#criterionEvaluator.evaluate(criterion, expression => this.#evaluate(context, expression));
+      if (error === undefined) return {
+        condition,
+        met
+      };
+      return {
+        condition,
+        met,
+        diagnostic: {
+          code: error.reason,
+          message: error.message
+        }
+      };
+    } catch (error) {
+      // a step that calls a workflow made no HTTP call, so a condition on one
+      // fails like any unevaluable condition; the diagnostic keeps the
+      // authoring mistake visible.
+      if (error instanceof _errors_RuntimeExpressionError_ts__WEBPACK_IMPORTED_MODULE_12__["default"] && error.reason === 'unavailable-expression') {
+        return {
+          condition,
+          met: false,
+          diagnostic: {
+            code: error.reason,
+            message: error.message
+          }
+        };
+      }
+      throw error;
+    }
   }
 
   /**
@@ -13540,6 +13626,8 @@ class WorkflowExecutor {
         action,
         attempts,
         durationMs: this.#now() - stepStartedAt,
+        successCriteria: outcome.successCriteria,
+        actionCriteria: outcome.actionCriteria,
         ...(subWorkflows.length > 0 ? {
           subWorkflows
         } : {}),
@@ -14070,19 +14158,26 @@ class WorkflowExecutor {
       // apply on top — they are the author's assertion about this step, and
       // dropping them because the step happens to target a workflow would
       // silently discard it. They see the sub-run's own outputs (`{}` when it
-      // transferred) through `$outputs` and `$workflows`.
+      // transferred) through `$outputs` and `$workflows`, and are evaluated
+      // even when the sub-run failed, so the record explains them either way.
       // the criteria and actions are the *calling* step's, so they are
       // evaluated by the parent document's step executor — only the sub-run
       // itself switched documents.
       const stepExecutor = this.#stepExecutor.forDocument(document);
-      const successful = !this.#settledFailed(result) && stepExecutor.evaluateCriteria(step, context);
-      const matchedActions = stepExecutor.selectActions(step, successful, context);
+      const successCriteria = stepExecutor.evaluateCriteria(step, context);
+      const successful = !this.#settledFailed(result) && successCriteria.every(criterion => criterion.met);
+      const {
+        matchedActions,
+        actionCriteria
+      } = stepExecutor.selectActions(step, successful, context);
       return {
         stepId,
         successful,
         outputs,
         action: matchedActions[0],
-        matchedActions
+        matchedActions,
+        successCriteria,
+        actionCriteria
       };
     };
   }