@mastra/observability 1.12.0-alpha.4 → 1.13.0-alpha.0

package/CHANGELOG.md

@@ -1,5 +1,129 @@
 # @mastra/observability
 
+## 1.13.0-alpha.0
+
+### Minor Changes
+
+- Roll up token usage from internal MODEL_GENERATION spans onto the closest exported ancestor span. When `tracingPolicy.internal` filters a model call out of exported traces, its tokens used to disappear from both the trace UI and metrics. Now: ([#16434](https://github.com/mastra-ai/mastra/pull/16434))
+  - The visible ancestor (e.g. `PROCESSOR_RUN`, `AGENT_RUN`) gets an `internalUsage` attribute summing the tokens consumed by its hidden descendants — so a Mastra-owned processor that runs an internal agent (moderation, PII detector, structured output, etc.) shows its aggregate cost on the visible `PROCESSOR_RUN` span.
+  - Token / cost metrics still emit, but are attributed via labels to the visible ancestor instead of the hidden agent.
+
+  No action required — the rollup applies automatically whenever an internal `MODEL_GENERATION` ends inside a non-internal ancestor.
+
+### Patch Changes
+
+- Updated dependencies [[`8cdb86c`](https://github.com/mastra-ai/mastra/commit/8cdb86ceed1137bc2768e147dce85a0692b9fb26), [`eda90c5`](https://github.com/mastra-ai/mastra/commit/eda90c5bfd7de11805ecc9f4552716c895fbaf78), [`afc004f`](https://github.com/mastra-ai/mastra/commit/afc004f5cc7e30697809e7021820b9f5881e6719), [`408be73`](https://github.com/mastra-ai/mastra/commit/408be73449dfab92b51eab8c6623b6c443debc25)]:
+  - @mastra/core@1.36.0-alpha.1
+
+## 1.12.0
+
+### Minor Changes
+
+- `DefaultExporter` now notifies custom exporters and connected integrations when it cannot persist observability events, such as unsupported storage or retries being exceeded. ([#16111](https://github.com/mastra-ai/mastra/pull/16111))
+
+- Renamed two built-in observability exporters to clearer names. The originals are still exported (now deprecated) and continue to work unchanged, including their existing exporter `name` strings and error IDs, so monitoring rules and dashboards keep matching until you migrate. ([#16223](https://github.com/mastra-ai/mastra/pull/16223))
+  - `CloudExporter` → `MastraPlatformExporter`
+  - `DefaultExporter` → `MastraStorageExporter`
+
+  **Before**
+
+  ```ts
+  import { Observability, DefaultExporter, CloudExporter, SensitiveDataFilter } from '@mastra/observability';
+
+  new Observability({
+    configs: {
+      default: {
+        serviceName: 'my-app',
+        exporters: [new DefaultExporter(), new CloudExporter()],
+        spanOutputProcessors: [new SensitiveDataFilter()],
+      },
+    },
+  });
+  ```
+
+  **After**
+
+  ```ts
+  import {
+    Observability,
+    MastraStorageExporter,
+    MastraPlatformExporter,
+    SensitiveDataFilter,
+  } from '@mastra/observability';
+
+  new Observability({
+    configs: {
+      default: {
+        serviceName: 'my-app',
+        exporters: [new MastraStorageExporter(), new MastraPlatformExporter()],
+        spanOutputProcessors: [new SensitiveDataFilter()],
+      },
+    },
+  });
+  ```
+
+- Apply `SensitiveDataFilter` by default ([#16234](https://github.com/mastra-ai/mastra/pull/16234))
+
+  The `Observability` registry now auto-applies a `SensitiveDataFilter` span output processor to every configured instance, so secrets (API keys, tokens, passwords, etc.) are redacted before they reach exporters such as the Mastra cloud exporter. This protects against accidentally exporting sensitive data when the filter was not added manually.
+
+  A new top-level `sensitiveDataFilter` option on the `Observability` registry config controls this behavior:
+  - `true` (default): apply `SensitiveDataFilter` with default options.
+  - `false`: opt out of auto-applied filtering.
+  - a `SensitiveDataFilterOptions` object: customize the filter (sensitive fields, redaction token, redaction style).
+
+  If a config already includes a `SensitiveDataFilter` in `spanOutputProcessors`, the auto-applied filter is skipped to avoid double redaction. Pre-instantiated `ObservabilityInstance` values are not modified.
+
+  **Before:**
+
+  ```typescript
+  import { Observability, DefaultExporter, CloudExporter, SensitiveDataFilter } from '@mastra/observability';
+
+  new Observability({
+    configs: {
+      default: {
+        serviceName: 'mastra',
+        exporters: [new DefaultExporter(), new CloudExporter()],
+        spanOutputProcessors: [new SensitiveDataFilter()],
+      },
+    },
+  });
+  ```
+
+  **After:**
+
+  ```typescript
+  import { Observability, DefaultExporter, CloudExporter } from '@mastra/observability';
+
+  new Observability({
+    configs: {
+      default: {
+        serviceName: 'mastra',
+        exporters: [new DefaultExporter(), new CloudExporter()],
+      },
+    },
+    // Optional: customize or disable the auto-applied filter.
+    // sensitiveDataFilter: false,
+    // sensitiveDataFilter: { sensitiveFields: ['myCustomSecret'] },
+  });
+  ```
+
+- Added new `MODEL_INFERENCE` span type under `MODEL_STEP`, covering only the model provider call. Use it to measure model latency separately from input/output processors and tool executions. ([#16267](https://github.com/mastra-ai/mastra/pull/16267))
+
+### Patch Changes
+
+- Fixed cost estimation for OpenRouter models. The **Model Usage & Cost** panel now shows costs for OpenRouter `vendor/model` ids (e.g. `openai/gpt-5-mini-2025-08-07`, `xiaomi/mimo-v2-pro-20260318`) that previously rendered an empty cost column. ([#16206](https://github.com/mastra-ai/mastra/pull/16206))
+
+- Support `MASTRA_PLATFORM_ACCESS_TOKEN` as the preferred environment variable for `MastraPlatformExporter`, while retaining `MASTRA_CLOUD_ACCESS_TOKEN` as a fallback for backward compatibility. ([#16500](https://github.com/mastra-ai/mastra/pull/16500))
+
+- Score events now include scorer names and target entity types. ([#16185](https://github.com/mastra-ai/mastra/pull/16185))
+
+- Fixed `MODEL_INFERENCE` span timing so it measures pure model latency. ([#16357](https://github.com/mastra-ai/mastra/pull/16357))
+
+- Refreshed the embedded pricing data snapshot used for cost estimation in observability metrics with the latest provider rates. ([#16373](https://github.com/mastra-ai/mastra/pull/16373))
+
+- Updated dependencies [[`9f17410`](https://github.com/mastra-ai/mastra/commit/9f1741080def23d42ee50b39887a385ae316a3c6), [`7ad5585`](https://github.com/mastra-ai/mastra/commit/7ad55856406f1de398dc713f6a9eaa78b2784bb6), [`ac47842`](https://github.com/mastra-ai/mastra/commit/ac478427aa7a5f5fdaed633a911218689b438c60), [`cc189cc`](https://github.com/mastra-ai/mastra/commit/cc189cc0128eb7af233476b5e421ec6888bffde7), [`d1fdbd0`](https://github.com/mastra-ai/mastra/commit/d1fdbd012add5623cb7e6b7f882b605ab358bbb4), [`210ea7a`](https://github.com/mastra-ai/mastra/commit/210ea7af559791b73a44fc9c12179908aaa3183f), [`7c275a8`](https://github.com/mastra-ai/mastra/commit/7c275a810595e1a6c41ccc39720531ab65734700), [`bae019e`](https://github.com/mastra-ai/mastra/commit/bae019ecb6694da96909f7ec7b9eb3a0a33aa887), [`890b24c`](https://github.com/mastra-ai/mastra/commit/890b24cc7d32ed6aa4dfe253e54dc6bf4099f690), [`f984b4d`](https://github.com/mastra-ai/mastra/commit/f984b4d6c60bf2ae2a9b156f0e8c35a66fe96c91), [`6742347`](https://github.com/mastra-ai/mastra/commit/6742347d71955d7639adc9ddf6ff8282de7ee3ba), [`b59316f`](https://github.com/mastra-ai/mastra/commit/b59316ffa0f7688165b0f9c81ccdf85da461e5b2), [`0f48ebf`](https://github.com/mastra-ai/mastra/commit/0f48ebfc7ac7897b2092a189f45751924cf56d1c), [`37c0dc5`](https://github.com/mastra-ai/mastra/commit/37c0dc5697d343db98628bf867bf71ce6deec6d7), [`087e413`](https://github.com/mastra-ai/mastra/commit/087e4133e5d6efa36619e9556c16750e4179c047), [`83218c8`](https://github.com/mastra-ai/mastra/commit/83218c88b37773c9424fbe733b37be556e55e94d), [`ef6b584`](https://github.com/mastra-ai/mastra/commit/ef6b5847ac33c0a7e80af3a86e8801e2933dd3ee), [`c6eb39e`](https://github.com/mastra-ai/mastra/commit/c6eb39ea6dca381c6563cb240237fbe608e02f93), [`7b0ad1f`](https://github.com/mastra-ai/mastra/commit/7b0ad1f5c53dc118c6da12ae82ae2587037dc2b8), [`d91ebe2`](https://github.com/mastra-ai/mastra/commit/d91ebe28ee065d8f2ed6df741c3c07f58d359529), [`62666c3`](https://github.com/mastra-ai/mastra/commit/62666c367eaeac3941ead454b1d38810cc855721), [`33f5061`](https://github.com/mastra-ai/mastra/commit/33f5061cd1c0335020c3faae61ce96de822854fa), [`4af2160`](https://github.com/mastra-ai/mastra/commit/4af2160322f4718cac421930cce85641e9512389), [`087e413`](https://github.com/mastra-ai/mastra/commit/087e4133e5d6efa36619e9556c16750e4179c047), [`265ec9f`](https://github.com/mastra-ai/mastra/commit/265ec9f887b5c81255c873a76ff7796f16e4f99b), [`ce01024`](https://github.com/mastra-ai/mastra/commit/ce010242eee9bdfc09e4c26725b9d37998679a8d), [`6ce80bf`](https://github.com/mastra-ai/mastra/commit/6ce80bf4872a891e0bddf8b80561a80584efb14b), [`f984b4d`](https://github.com/mastra-ai/mastra/commit/f984b4d6c60bf2ae2a9b156f0e8c35a66fe96c91), [`136c959`](https://github.com/mastra-ai/mastra/commit/136c9592fb0eeb0cd212f28629d8a29b7557a2fc), [`9268531`](https://github.com/mastra-ai/mastra/commit/9268531e7ec4be98beeba3b3ae8be0a7ea380662), [`13ead79`](https://github.com/mastra-ai/mastra/commit/13ead79149486b88144db7e11e6ff551caef5be1), [`dccd8f1`](https://github.com/mastra-ai/mastra/commit/dccd8f1f8b8f1ad203b77556207e5529567c616d), [`4df7cc7`](https://github.com/mastra-ai/mastra/commit/4df7cc79342fd065fe7fdeef93c094db14b12bcd), [`f180e49`](https://github.com/mastra-ai/mastra/commit/f180e4990e71b04c9a475b523584071712f0048f), [`9260e01`](https://github.com/mastra-ai/mastra/commit/9260e015276fb1b500f7878ee452b47476bf1583), [`2f6c54e`](https://github.com/mastra-ai/mastra/commit/2f6c54e17c041cac1def54baaa6b771647836414), [`aca3121`](https://github.com/mastra-ai/mastra/commit/aca31211233dac25459f140ea4fcfb3a5af64c18), [`e06a159`](https://github.com/mastra-ai/mastra/commit/e06a1598ca07a6c3778aefc2a2d288363c6294ff), [`4dd900d`](https://github.com/mastra-ai/mastra/commit/4dd900d75dfe9be89f8c15188b368a8622aa1e18), [`b560d6f`](https://github.com/mastra-ai/mastra/commit/b560d6f88b9b904b15c10f75c949eb145bc27684), [`99869ec`](https://github.com/mastra-ai/mastra/commit/99869ecb1f2aa6dfcc44fa4e843e5ee0344efa64), [`900d086`](https://github.com/mastra-ai/mastra/commit/900d086bb737b9cf2fcf68f11b0389b801a2738c), [`4c0e286`](https://github.com/mastra-ai/mastra/commit/4c0e28637c9cfb4f416549b55e97ebfa13319dfc), [`55f1e2d`](https://github.com/mastra-ai/mastra/commit/55f1e2d65425b95a49ae788053b266f256e38c96), [`4ff5bdf`](https://github.com/mastra-ai/mastra/commit/4ff5bdfe170cba6dfb5260c6af0f4ba668430772), [`9cdf38e`](https://github.com/mastra-ai/mastra/commit/9cdf38e58506e1109c8b38f97cd7770978a4218e), [`087e413`](https://github.com/mastra-ai/mastra/commit/087e4133e5d6efa36619e9556c16750e4179c047), [`db34bc6`](https://github.com/mastra-ai/mastra/commit/db34bc6fb36cf125bda0c46be4d3fdc774b70cc4), [`990851e`](https://github.com/mastra-ai/mastra/commit/990851edcb0e30be5c2c18b6532f1a876cc2d335), [`bbcd93c`](https://github.com/mastra-ai/mastra/commit/bbcd93cf7d8aa1007d6d84bfd033b8015c912087), [`8373ff4`](https://github.com/mastra-ai/mastra/commit/8373ff46745d77af79f183c4470f80fa2727a6b2), [`d48a705`](https://github.com/mastra-ai/mastra/commit/d48a705ff3dfbdc7a996e07ecd8293b5effd9a2a), [`308bd07`](https://github.com/mastra-ai/mastra/commit/308bd074f35cef0c75d82fc1eb19382fe04ecf6f), [`6068a6c`](https://github.com/mastra-ai/mastra/commit/6068a6c42950fad3ebfc92346417896ba60803d2), [`36b3bbf`](https://github.com/mastra-ai/mastra/commit/36b3bbf5a8d59f7e23d47e29340e76c681b4929c), [`d86f031`](https://github.com/mastra-ai/mastra/commit/d86f031eb6b0b2570145afafea664e59bf688962), [`b275631`](https://github.com/mastra-ai/mastra/commit/b275631dc10541a482b2e2d4a3e3cfa843bd5fa1), [`00106be`](https://github.com/mastra-ai/mastra/commit/00106bede59b81e5b0e9cd6aad8d3b5dbc336387), [`bd36d8e`](https://github.com/mastra-ai/mastra/commit/bd36d8eb6de8c9a0310352649dbd4b06703c2299), [`11c1528`](https://github.com/mastra-ai/mastra/commit/11c152848c5d0ef227184853b5040f5b41ee7b1e), [`4999667`](https://github.com/mastra-ai/mastra/commit/49996678b68356cad7f088430009690406c50fbd), [`e2a079c`](https://github.com/mastra-ai/mastra/commit/e2a079cc3755b1895f7bd5dc36e9be81b11c7c22), [`8ac9141`](https://github.com/mastra-ai/mastra/commit/8ac9141439caa8fdd674944c4d84f29b3c730296), [`25184ff`](https://github.com/mastra-ai/mastra/commit/25184ffaf1293ec95119426eb1a1f8d38831b96c), [`534a456`](https://github.com/mastra-ai/mastra/commit/534a456a25e4df1e5407e7e632f4cb3b1fa14f9d), [`105e454`](https://github.com/mastra-ai/mastra/commit/105e454c95af06a7c741c15969d8f9b0f02463a7), [`aebde9c`](https://github.com/mastra-ai/mastra/commit/aebde9cfacf56592c6b6350cae721740fe090b8a), [`36bae07`](https://github.com/mastra-ai/mastra/commit/36bae07c0e70b1b3006f2fd20830e8883dcbd066), [`5688881`](https://github.com/mastra-ai/mastra/commit/5688881669c7ed157f31ac77f6fc5f8d95ceea32)]:
+  - @mastra/core@1.33.0
+
 ## 1.12.0-alpha.4
 
 ### Patch Changes

package/dist/features.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"features.d.ts","sourceRoot":"","sources":["../src/features.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAoCG;AAEH,eAAO,MAAM,qBAAqB,EAAE,WAAW,CAAC,MAAM,CAAqC,CAAC"}
+{"version":3,"file":"features.d.ts","sourceRoot":"","sources":["../src/features.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAoCG;AAEH,eAAO,MAAM,qBAAqB,EAAE,WAAW,CAAC,MAAM,CAA8D,CAAC"}

package/dist/index.cjs

@@ -19029,6 +19029,7 @@ var PricingRegistry = class _PricingRegistry {
   constructor(pricingModels) {
     this.pricingModels = pricingModels;
   }
+  pricingModels;
   static globalRegistry = null;
   static fromText(pricingModelText) {
     return new _PricingRegistry(parsePricingModelText(pricingModelText));
@@ -19432,6 +19433,9 @@ function emitTokenMetrics(span, metrics) {
   }
   emitUsageMetrics(attrs, attrs.usage, metrics);
 }
+function emitTokenMetricsForUsage(usage, provider, model, metrics) {
+  emitUsageMetrics({ provider, model }, usage, metrics);
+}
 function emitAutoExtractedMetrics(span, metrics) {
   emitDurationMetrics(span, metrics);
   emitTokenMetrics(span, metrics);
@@ -19593,6 +19597,46 @@ function extractUsageMetrics(usage, providerMetadata) {
 function sumDefinedValues(obj, keys) {
   return keys.reduce((sum, key) => sum + (obj[key] ?? 0), 0);
 }
+function addOptional(a, b) {
+  if (a === void 0 && b === void 0) return void 0;
+  return (a ?? 0) + (b ?? 0);
+}
+function mergeInputDetails(a, b) {
+  if (!a) return b ? { ...b } : void 0;
+  if (!b) return { ...a };
+  return {
+    text: addOptional(a.text, b.text),
+    cacheRead: addOptional(a.cacheRead, b.cacheRead),
+    cacheWrite: addOptional(a.cacheWrite, b.cacheWrite),
+    audio: addOptional(a.audio, b.audio),
+    image: addOptional(a.image, b.image)
+  };
+}
+function mergeOutputDetails(a, b) {
+  if (!a) return b ? { ...b } : void 0;
+  if (!b) return { ...a };
+  return {
+    text: addOptional(a.text, b.text),
+    reasoning: addOptional(a.reasoning, b.reasoning),
+    audio: addOptional(a.audio, b.audio),
+    image: addOptional(a.image, b.image)
+  };
+}
+function addUsageStats(a, b) {
+  if (!a) {
+    return {
+      ...b,
+      inputDetails: b.inputDetails ? { ...b.inputDetails } : void 0,
+      outputDetails: b.outputDetails ? { ...b.outputDetails } : void 0
+    };
+  }
+  return {
+    inputTokens: addOptional(a.inputTokens, b.inputTokens),
+    outputTokens: addOptional(a.outputTokens, b.outputTokens),
+    inputDetails: mergeInputDetails(a.inputDetails, b.inputDetails),
+    outputDetails: mergeOutputDetails(a.outputDetails, b.outputDetails)
+  };
+}
 
 // src/model-tracing.ts
 function supportsModelInference() {
@@ -21122,7 +21166,7 @@ var BaseObservabilityInstance = class extends base.MastraBase {
    * This ensures all spans emit events regardless of implementation
    */
   wireSpanLifecycle(span) {
-    if (!this.config.includeInternalSpans && span.isInternal) {
+    if (!this.config.includeInternalSpans && span.isInternal && span.type !== observability.SpanType.MODEL_GENERATION) {
       return;
     }
     const originalEnd = span.end.bind(span);
@@ -21132,7 +21176,11 @@ var BaseObservabilityInstance = class extends base.MastraBase {
         this.logger.warn(`End event is not available on event spans`);
         return;
       }
+      const rollupTarget = this.captureModelUsageRollup(span, options);
       originalEnd(options);
+      if (rollupTarget) {
+        this.applyUsageRollup(rollupTarget);
+      }
       this.emitSpanEnded(span);
     };
     span.update = (options) => {
@@ -21306,6 +21354,70 @@ var BaseObservabilityInstance = class extends base.MastraBase {
       this.emitTracingEvent(event);
     }
   }
+  /**
+   * When an internal MODEL_GENERATION span ends, capture the rollup payload
+   * (usage, provider, model, target ancestor) needed to attribute its cost
+   * to the closest exported ancestor span. Returns undefined when no rollup
+   * applies — non-MODEL_GENERATION spans, spans that will be exported, or
+   * spans whose usage isn't available at end time.
+   */
+  captureModelUsageRollup(span, endOptions) {
+    if (span.type !== observability.SpanType.MODEL_GENERATION) return void 0;
+    if (!span.isInternal || this.config.includeInternalSpans) return void 0;
+    const endAttrs = endOptions?.attributes ?? void 0;
+    const liveAttrs = span.attributes;
+    const usage = endAttrs?.usage ?? liveAttrs?.usage;
+    if (!usage) return void 0;
+    const ancestor = this.findExportedAncestor(span);
+    if (!ancestor) return void 0;
+    const provider = endAttrs?.provider ?? liveAttrs?.provider;
+    const model = endAttrs?.responseModel ?? endAttrs?.model ?? liveAttrs?.responseModel ?? liveAttrs?.model;
+    return { ancestor, usage, provider, model };
+  }
+  /**
+   * Accumulate usage onto the ancestor's `internalUsage` attribute (for trace
+   * UI visibility) and emit auto-extracted token metrics now, using the
+   * ancestor's metrics context so cost / token labels point at the visible
+   * span instead of the hidden agent that incurred them.
+   */
+  applyUsageRollup(target) {
+    const { ancestor, usage, provider, model } = target;
+    const attrs = ancestor.attributes;
+    attrs.internalUsage = addUsageStats(attrs.internalUsage, usage);
+    try {
+      emitTokenMetricsForUsage(usage, provider, model, this.getMetricsContext(ancestor));
+    } catch (err) {
+      this.logger.error("[Observability] Usage rollup metric emission error:", err);
+    }
+  }
+  /**
+   * Walk up the parent chain to find the closest ancestor that will actually
+   * reach exporters. Skips both internal-filtered ancestors and ancestors
+   * whose type matches `excludeSpanTypes`, so the rollup target is one whose
+   * mutated `internalUsage` attribute is visible in exported traces.
+   *
+   * Note: this does not preemptively run `spanFilter` — that filter can be
+   * async and have side effects, so the rare case of a `spanFilter`-dropped
+   * ancestor falls through.
+   */
+  findExportedAncestor(span) {
+    let ancestor = span.parent;
+    while (ancestor && this.isFilteredFromExport(ancestor)) {
+      ancestor = ancestor.parent;
+    }
+    return ancestor;
+  }
+  /**
+   * Returns true when a span would be dropped by `getSpanForExport` for a
+   * reason cheap to check up-front (internal-span filtering or
+   * `excludeSpanTypes`). Used by `findExportedAncestor` to skip rollup
+   * targets that would silently lose their `internalUsage` attribute.
+   */
+  isFilteredFromExport(span) {
+    if (span.isInternal && !this.config.includeInternalSpans) return true;
+    if (this.config.excludeSpanTypes?.includes(span.type)) return true;
+    return false;
+  }
   /**
    * Emit a tracing event through the bus.
    *
@@ -22256,7 +22368,7 @@ var Observability = class extends base.MastraBase {
 };
 
 // src/features.ts
-var observabilityFeatures = /* @__PURE__ */ new Set(["model-inference-span"]);
+var observabilityFeatures = /* @__PURE__ */ new Set(["model-inference-span", "internal-usage-rollup"]);
 
 // src/tracing-options.ts
 function buildTracingOptions(...updaters) {