@dbx-tools/appkit-mastra 0.1.4 → 0.1.12

package/README.md

@@ -195,14 +195,148 @@ intact.
 streaming-aware tools record built on top of the plugin's
 `exports().sendMessage` AsyncGenerator. Each Genie wire event
 (`FETCHING_METADATA`, `ASKING_AI`, `EXECUTING_QUERY`, attached SQL,
-`query_result` row counts, errors) is normalised into a `GenieProgress`
-payload and pushed mid-flight through Mastra's `ctx.writer`, surfacing
-as `tool-output` chunks the React client can render as inline status
-pills, SQL blocks, and row-count badges while the LLM is still waiting
-on the final `tool-result`. Tool ids are stable: `genie` for the
-default alias, `genie-<alias>` for additional aliases, and one shared
-`genie_get_conversation`. The LLM still receives a single clean final
-payload so the streaming UI doesn't leak into the assistant message.
+errors) is normalised into a `GenieProgress` payload and pushed
+mid-flight through Mastra's `ctx.writer`, surfacing as
+`tool-output` chunks the React client can render as inline status
+pills and SQL blocks while the LLM is still waiting on the final
+`tool-result`. Tool ids are stable: `genie` for the default alias,
+`genie-<alias>` for additional aliases, and one shared
+`genie_get_conversation`.
+
+Genie tool-result shape (LLM-bound):
+
+```
+{
+  conversationId?: string,        // pass back to continue the same Genie thread
+  genieAnswer?: string,           // Genie's prose answer; pass through verbatim
+  datasets?: [{                   // metadata only, one per executed SQL statement
+    chartId: string,              // embed [[chart:<chartId>]] to render inline
+    title?: string,
+    description?: string,
+    columns: string[],
+    rowCount: number,
+    sql?: string,
+  }],
+  suggestedFollowUps?: string[],  // UI shows as buttons; don't list in reply
+  error?: string,
+}
+```
+
+`datasets[]` is metadata only - column names, row count, the SQL
+Genie ran. The actual rows ride a separate `kind: "chart"` writer
+event so the LLM never has them in context (token cost stays flat
+regardless of dataset size). The model references each dataset by
+its `chartId` via the `[[chart:<chartId>]]` marker to display the
+chart inline; see the `render_data` section below for how the
+client resolves those markers.
+
+Genie data flow:
+
+- The writer emits `kind: "started" | "status" | "sql" |
+  "suggested" | "error"` events for the live loading pill. SQL
+  text is shown via a small Shiki-highlighted block.
+- The writer **also** emits two `kind: "chart"` events per
+  executed SQL statement, sharing the same `chartId`: the first
+  carries `{chartId, title, description?, data}` (rows converted
+  to objects keyed by column name with best-effort numeric
+  coercion); the second, when the chart-planner agent finishes,
+  carries `{chartId, option}` (the resolved Echarts spec). The
+  chat client's `<ChartSlot>` merges them by `chartId` and
+  renders inline at the matching `[[chart:<chartId>]]` marker.
+  This is the exact same wire format as the `render_data` tool,
+  so Genie and hand-built charts are indistinguishable on the
+  client.
+- After a hard reload, `synthesizeToolEventsFromHistory` rebuilds
+  `suggested` events from the persisted tool-result; the SQL pill
+  and chart events are live-only and don't replay.
+
+### `render_data` (system-default ambient tool)
+
+`buildAgents` registers a system-level `render_data` tool on
+every agent so the model can submit any tabular dataset for
+inline charting. Users can shadow it by including a same-named
+tool in `config.tools` or in a per-agent `tools` map; otherwise
+it's just there.
+
+The tool is generic - not coupled to Genie or any particular
+upstream. Input is `{ title, description?, data: Row[] }` where
+`data` is an array of objects keyed by column name (a SQL row
+set, an API response, a hand-built array, etc.).
+
+#### How it works (shared pipeline with Genie)
+
+Both `render_data` and Genie's `drainGenieStream` route through
+one helper, `emitChartWithPlanning`, so the wire format and
+trace shape are consistent everywhere:
+
+1. Mint a short `chartId` (8 hex chars).
+2. Push a `{ kind: "chart", chartId, title, description?, data }`
+   event to `ctx.writer` immediately. The chat client mounts a
+   `<ChartSlot>` showing a "Rendering chart" skeleton.
+3. Kick off the chart-planner agent
+   (`modelForTier(ModelTier.Fast)`) with the dataset in the
+   background. The agent's `structuredOutput` schema is a compact
+   "chart plan" (chart type + axis labels + categories +
+   series); the helper expands the plan into an Echarts
+   `EChartsOption` JSON.
+4. When the planner resolves, push a follow-up
+   `{ kind: "chart", chartId, option }` event. The client's
+   `<ChartSlot>` merges it with the dataset event (last write
+   wins per field) and swaps in `<ReactECharts>`.
+5. If the planner fails, no follow-up event fires. Once the
+   parent tool finishes, the slot transitions to a "couldn't
+   render chart" fallback frame.
+
+The parent tool (`render_data` or Genie) `await`s the planner
+promise(s) before its `execute` returns, so chart latency shows
+up under the tool's trace span. The dataset event fires
+**immediately**, though, so the calling LLM gets back the
+`chartId` synchronously and the chat client has a layout slot
+ready before the planner resolves.
+
+The LLM-bound payload is just `{ chartId }` (for `render_data`)
+or `datasets[]` metadata (for Genie); row data and option JSON
+never reach the LLM, so token cost stays flat regardless of
+dataset size.
+
+#### Inline placement contract
+
+The model embeds `[[chart:<chartId>]]` on its own line in its
+markdown reply at the position where the chart should appear:
+
+```markdown
+## Audit Score
+
+Audit Score is stable at ~94%, hovering between 93.5 and 95.0.
+
+[[chart:a3f9c1d2]]
+
+## Service Time
+
+Service time is the outlier at 162.5s, up from a target of 150s.
+
+[[chart:b7e2d4f1]]
+```
+
+The chat client splits the assistant text on these markers and
+drops a `<ChartSlot>` in at each spot. A marker that arrives
+before its `chart` event (rare; only during fast streaming) shows
+a "Queueing chart" skeleton; a chart whose marker the model
+forgot to place falls through to the end of the reply as a
+fallback. All three states (queueing, rendering, rendered) share
+the same fixed-height frame so the layout doesn't jump as
+charts resolve.
+
+#### Trade-offs
+
+- Both events ride the writer, not the persisted tool-result, so
+  charts don't re-render after a hard reload. The model can call
+  `render_data` again (or re-ask Genie) on the next turn if the
+  user wants the chart back.
+- The chart-planner is a separate model call per dataset (fast
+  tier, but still ~1-3s each). For an N-chart turn, latency is
+  `Genie + max(planners)` since the planners run concurrently
+  with the rest of Genie's stream and with each other.
 
 Plugins that aren't registered (or don't implement the toolkit
 interface) resolve to `undefined` at runtime, so guard with `?.` /
@@ -256,8 +390,9 @@ const saveDoc = createTool({
 ```
 
 When `tool()`'s `id` is omitted it's auto-derived from a slugified
-description plus a 6-char SHA-1 prefix - stable across runs so traces
-stay readable. Pass an explicit `id` when you want to pin one.
+description plus a 6-char FNV-1a base-32 suffix - stable across runs
+so traces stay readable. Pass an explicit `id` when you want to pin
+one.
 
 Reach for `createTool` when you need Mastra-only fields (`outputSchema`,
 `suspendSchema`, `requireApproval`, `mcp`, etc.).

package/dist/index.d.ts

@@ -13,6 +13,8 @@ export * from "./src/plugin.js";
 export * from "@dbx-tools/appkit-mastra-shared";
 export * from "./src/config.js";
 export * from "./src/agents.js";
+export * from "./src/chart.js";
 export * from "./src/genie.js";
+export * from "./src/tools/email.js";
 export { clearServingEndpointsCache, extractModelOverride, listServingEndpoints, MASTRA_MODEL_OVERRIDE_KEY, MODEL_OVERRIDE_BODY_FIELDS, MODEL_OVERRIDE_HEADER, MODEL_OVERRIDE_QUERY, resolveModelId, type ResolvedModel, type ResolveModelOptions, type ServingEndpointSummary, } from "./src/serving.js";
 export { FALLBACK_MODEL_IDS, MODEL_CATALOG, modelForTier, modelsForTier, ModelTier, } from "./src/model.js";

package/dist/index.js

@@ -13,6 +13,8 @@ export * from "./src/plugin.js";
 export * from "@dbx-tools/appkit-mastra-shared";
 export * from "./src/config.js";
 export * from "./src/agents.js";
+export * from "./src/chart.js";
 export * from "./src/genie.js";
+export * from "./src/tools/email.js";
 export { clearServingEndpointsCache, extractModelOverride, listServingEndpoints, MASTRA_MODEL_OVERRIDE_KEY, MODEL_OVERRIDE_BODY_FIELDS, MODEL_OVERRIDE_HEADER, MODEL_OVERRIDE_QUERY, resolveModelId, } from "./src/serving.js";
 export { FALLBACK_MODEL_IDS, MODEL_CATALOG, modelForTier, modelsForTier, ModelTier, } from "./src/model.js";

package/dist/src/agents.d.ts

@@ -284,7 +284,7 @@ export declare const FALLBACK_AGENT_ID = "default";
  * Override globally via {@link MastraPluginConfig.styleInstructions}
  * (pass `false` to disable entirely, or a string to replace).
  */
-export declare const DEFAULT_STYLE_INSTRUCTIONS = "Output style:\n\n- Plain prose. Use hyphens (-) only. Never use em dashes (\u2014) or en dashes (\u2013).\n- Never use emojis.\n- Skip openers like \"Great question\", \"Absolutely\", \"I'd be happy to help\".\n- Skip closers like \"Let me know if you have any questions\".\n- Skip self-disclaimers (\"I should mention\", \"It's important to note\").\n- Answer directly. No preamble before the actual answer.\n- Use lists and headers only when they clarify a multi-part answer; not for short replies.\n- Quote numbers, code, identifiers, and tool output verbatim. Never paraphrase them.";
+export declare const DEFAULT_STYLE_INSTRUCTIONS: string;
 /**
  * Resolve every entry in `config.agents` into a Mastra `Agent`
  * instance. When `config.agents` is omitted the plugin registers a

package/dist/src/agents.js

@@ -16,8 +16,10 @@ import { genie } from "@databricks/appkit";
 import { logUtils, pluginUtils, stringUtils } from "@dbx-tools/appkit-shared";
 import { Agent } from "@mastra/core/agent";
 import { createTool } from "@mastra/core/tools";
+import { buildRenderDataTool } from "./chart.js";
 import { buildGenieProvider } from "./genie.js";
 import { buildModel } from "./model.js";
+import { stripStaleChartsProcessor } from "./processors/strip-stale-charts.js";
 /** Re-export of Mastra's native `createTool` for full-feature access. */
 export { createTool } from "@mastra/core/tools";
 /**
@@ -63,8 +65,8 @@ export function tool(opts) {
 /**
  * Build a deterministic Mastra tool id from a description.
  * Delegates to {@link stringUtils.toUniqueSlug}: slug + always-on
- * SHA-1 suffix so two tools with the same leading words don't
- * collide in traces. Stable across runs.
+ * 6-char FNV-1a base-32 suffix so two tools with the same leading
+ * words don't collide in traces. Stable across runs.
  */
 function deriveToolId(description) {
     return stringUtils.toUniqueSlug(description, { fallbackPrefix: "tool" });
@@ -110,16 +112,21 @@ Rules:
  * Override globally via {@link MastraPluginConfig.styleInstructions}
  * (pass `false` to disable entirely, or a string to replace).
  */
-export const DEFAULT_STYLE_INSTRUCTIONS = `Output style:
-
-- Plain prose. Use hyphens (-) only. Never use em dashes (—) or en dashes (–).
-- Never use emojis.
-- Skip openers like "Great question", "Absolutely", "I'd be happy to help".
-- Skip closers like "Let me know if you have any questions".
-- Skip self-disclaimers ("I should mention", "It's important to note").
-- Answer directly. No preamble before the actual answer.
-- Use lists and headers only when they clarify a multi-part answer; not for short replies.
-- Quote numbers, code, identifiers, and tool output verbatim. Never paraphrase them.`;
+export const DEFAULT_STYLE_INSTRUCTIONS = [
+    "Output style:",
+    "",
+    "Use markdown formatting, including headings, lists, and code blocks.",
+    "Avoid lists and headers for short replies.",
+    "Plain prose.",
+    "Use hyphens (-) only. Never use em dashes or en dashes.",
+    "Never use emojis.",
+    "Skip openers like 'Great question', 'Absolutely', and 'I'd be happy to help'.",
+    "Skip closers like 'Let me know if you have any questions'.",
+    "Skip self-disclaimers like 'I should mention' and 'It's important to note'.",
+    "Answer directly.",
+    "Do not include a preamble before the actual answer.",
+    "Use lists and headers only when they clarify a multi-part answer.",
+].join("\n");
 /**
  * Resolve the style block to append to every agent's instructions.
  * Returns `null` when the caller opted out (`styleInstructions: false`).
@@ -159,9 +166,22 @@ export async function buildAgents(opts) {
     const definitions = resolveDefinitions(config);
     const ids = Object.keys(definitions);
     const defaultAgentId = config.defaultAgent ?? ids[0] ?? FALLBACK_AGENT_ID;
-    const plugins = buildPluginsMap(context);
-    const ambientTools = config.tools ?? {};
+    const plugins = buildPluginsMap(config, context);
+    // System-default ambient tools every agent gets out of the box.
+    // Currently just `render_data` for inline visualizations; the
+    // user can shadow it by including a same-named tool in their own
+    // `config.tools` or per-agent `tools`. Order in {@link resolveTools}
+    // is `system -> user-ambient -> per-agent`, last write wins.
+    const systemTools = {
+        render_data: buildRenderDataTool(config),
+    };
+    const ambientTools = { ...systemTools, ...(config.tools ?? {}) };
     const style = resolveStyleInstructions(config);
+    // Default-on protection against the model copying turn-scoped
+    // chartIds from prior assistant tool results into the new
+    // turn's `[[chart:<id>]]` markers. Opt out per-plugin via
+    // `config.stripStaleCharts: false`.
+    const inputProcessors = config.stripStaleCharts === false ? [] : [stripStaleChartsProcessor];
     const agents = {};
     for (const [id, def] of Object.entries(definitions)) {
         const tools = await resolveTools(def.tools, plugins, ambientTools);
@@ -174,6 +194,7 @@ export async function buildAgents(opts) {
             model: resolveModel(config, def.model),
             tools,
             ...(memory ? { memory } : {}),
+            ...(inputProcessors.length > 0 ? { inputProcessors } : {}),
         });
     }
     if (!agents[defaultAgentId]) {
@@ -304,7 +325,7 @@ async function resolveTools(defTools, plugins, ambientTools) {
  * Mastra `ctx.writer`, so the UI gets `tool-output` chunks in real
  * time instead of staring at a spinner for the full Genie round-trip.
  */
-function buildPluginsMap(context) {
+function buildPluginsMap(config, context) {
     const cache = new Map();
     return new Proxy({}, {
         get(_target, propName) {
@@ -312,7 +333,7 @@ function buildPluginsMap(context) {
                 return undefined;
             if (cache.has(propName))
                 return cache.get(propName) ?? undefined;
-            const provider = resolveProvider(context, propName);
+            const provider = resolveProvider(config, context, propName);
             cache.set(propName, provider);
             return provider ?? undefined;
         },
@@ -322,14 +343,17 @@ function buildPluginsMap(context) {
  * Pick the right {@link MastraPluginToolkitProvider} for a sibling
  * plugin lookup. Returns the streaming-aware Genie adapter when the
  * caller asks for `genie`; falls back to the generic AppKit
- * `ToolProvider` adapter for every other plugin name.
+ * `ToolProvider` adapter for every other plugin name. `config` is
+ * threaded through so Genie's tool can run the chart planner
+ * inline against the same model resolver / fallback ladder the
+ * agents use.
  */
-function resolveProvider(context, propName) {
+function resolveProvider(config, context, propName) {
     if (propName === "genie") {
         const geniePlugin = pluginUtils.instance(context, genie);
         if (!geniePlugin)
             return null;
-        return buildGenieProvider(geniePlugin);
+        return buildGenieProvider(geniePlugin, { config });
     }
     const plugin = context?.getPlugins().get(propName);
     return adaptPluginToolkit(plugin);

package/dist/src/chart.d.ts

@@ -0,0 +1,170 @@
+/**
+ * Chart-rendering primitives.
+ *
+ * Three surfaces, one shared brain:
+ *
+ * - {@link buildRenderDataTool}: a Mastra tool the model calls
+ *   ("here is a dataset, render it as a chart"). The tool's
+ *   `execute` emits a `kind: "chart"` event with the raw rows to
+ *   `ctx.writer` synchronously, kicks off the chart-planner agent,
+ *   and `await`s the planner promise before returning so the
+ *   planner's latency is attributed to this tool's trace span.
+ *   The LLM-bound output is just `{ chartId }`, so its context
+ *   stays flat regardless of dataset size.
+ *
+ * - {@link emitChartWithPlanning}: the underlying helper that both
+ *   `render_data` and Genie's `drainGenieStream` call. Mints the
+ *   `chartId`, fires the dataset event immediately, runs the
+ *   planner in the background, and returns `{ chartId,
+ *   plannerPromise }` so callers can choose to await for trace
+ *   shape or fire-and-forget.
+ *
+ * - {@link runChartPlanner}: the chart-planner Agent + ECOption
+ *   expansion as a plain async function. Used internally by
+ *   {@link emitChartWithPlanning}; producers shouldn't reach for
+ *   it directly so chart events keep a single wire-format
+ *   contract.
+ *
+ * The model wires the chart into its reply by emitting the marker
+ * `[[chart:<chartId>]]` on its own line in markdown. The chat
+ * client splits the assistant text on these markers and drops a
+ * `<ChartSlot>` in at the position the model placed it; the slot
+ * shows a skeleton until the second `kind: "chart"` event (with
+ * the resolved `EChartsOption`) arrives, then swaps in the
+ * rendered Echarts visualisation.
+ */
+import type { RequestContext } from "@mastra/core/request-context";
+import { z } from "zod";
+import type { MastraPluginConfig } from "./config.js";
+/**
+ * Compact, model-friendly representation of an Echarts spec. The
+ * planner agent emits this; {@link planToEchartsOption} expands it
+ * into a real `EChartsOption` JSON. Two layers because letting the
+ * model fill in a fully-typed `EChartsOption` is brittle (hundreds
+ * of optional fields, deep unions, version-dependent shapes). A
+ * small "chart plan" schema is much more reliable for a fast model
+ * and keeps animation / tooltip / styling defaults consistent
+ * across charts.
+ */
+declare const chartPlanSchema: z.ZodObject<{
+    chartType: z.ZodEnum<{
+        bar: "bar";
+        line: "line";
+        area: "area";
+        scatter: "scatter";
+        pie: "pie";
+    }>;
+    title: z.ZodOptional<z.ZodString>;
+    xAxisLabel: z.ZodOptional<z.ZodString>;
+    yAxisLabel: z.ZodOptional<z.ZodString>;
+    categories: z.ZodOptional<z.ZodArray<z.ZodString>>;
+    series: z.ZodArray<z.ZodObject<{
+        name: z.ZodString;
+        data: z.ZodArray<z.ZodUnion<readonly [z.ZodNumber, z.ZodTuple<[z.ZodNumber, z.ZodNumber], null>, z.ZodObject<{
+            name: z.ZodString;
+            value: z.ZodNumber;
+        }, z.core.$strip>]>>;
+    }, z.core.$strip>>;
+}, z.core.$strip>;
+type ChartPlan = z.infer<typeof chartPlanSchema>;
+/** Inputs to {@link runChartPlanner}. */
+export interface RunChartPlannerOptions {
+    config: MastraPluginConfig;
+    requestContext?: RequestContext;
+    title: string;
+    description?: string;
+    data: ReadonlyArray<Record<string, unknown>>;
+}
+/** Output of {@link runChartPlanner}: a fully-formed Echarts spec. */
+export interface RunChartPlannerResult {
+    option: Record<string, unknown>;
+    chartType: ChartPlan["chartType"];
+}
+/**
+ * Run the chart planner against the given dataset and return a
+ * full Echarts `EChartsOption` JSON. Used by
+ * {@link emitChartWithPlanning}; tools and producers shouldn't
+ * call this directly (use the helper instead so chart events
+ * follow the same wire-format contract everywhere).
+ */
+export declare function runChartPlanner(opts: RunChartPlannerOptions): Promise<RunChartPlannerResult>;
+/**
+ * Minimal `ToolStream`-shaped writer surface. Defined locally so
+ * helpers can take any object with a `.write` method without
+ * importing Mastra's full `ToolStream` (which would also drag in
+ * agent / tool types this module doesn't otherwise need).
+ */
+interface MinimalWriter {
+    write: (chunk: unknown) => unknown;
+}
+/** Inputs to {@link emitChartWithPlanning}. */
+export interface EmitChartWithPlanningOptions {
+    /** Mastra `ctx.writer`; missing or closed writers are tolerated. */
+    writer?: MinimalWriter;
+    /** Plugin config; used to resolve the planner's model. */
+    config: MastraPluginConfig;
+    /** Per-request context (OBO auth). */
+    requestContext?: RequestContext;
+    /** Title shown above the rendered chart. Required. */
+    title: string;
+    /** Optional one-line intent biasing the planner. */
+    description?: string;
+    /** Tabular dataset to chart (one object per row). */
+    data: ReadonlyArray<Record<string, unknown>>;
+}
+/** Output of {@link emitChartWithPlanning}. */
+export interface EmitChartWithPlanningResult {
+    /** Short id matching the marker `[[chart:<chartId>]]`. */
+    chartId: string;
+    /**
+     * Promise that resolves once the planner has finished and the
+     * `kind: "chart"` event with the option has been emitted (or
+     * once the planner has failed silently). Callers that want
+     * trace observability should `await` this before returning
+     * from their tool's `execute`; callers that want pure
+     * fire-and-forget can ignore it.
+     */
+    plannerPromise: Promise<void>;
+}
+/**
+ * Shared chart-emission primitive used by both the `render_data`
+ * tool and Genie's `drainGenieStream`. Keeps both producers on
+ * one wire-format contract so the chat client only ever has to
+ * understand a single chart event shape.
+ *
+ * Behaviour:
+ *
+ * 1. Generates a short `chartId` (8 hex chars).
+ * 2. Immediately emits `{ kind: "chart", chartId, title,
+ *    description?, data }` via the writer so the chat client can
+ *    mount its `<ChartSlot>` with the rows in hand.
+ * 3. Kicks off the chart-planner agent in the background. On
+ *    success, emits a second `{ kind: "chart", chartId, option }`
+ *    event - same `chartId`, just the spec - so the client merges
+ *    the two into one rendered chart. On failure, no follow-up
+ *    event fires; the client falls back to whatever it can do
+ *    with the dataset alone (typically a "render failed" frame
+ *    after the parent tool finishes).
+ *
+ * Returns `chartId` synchronously so the caller can include it in
+ * the tool result (model uses it in `[[chart:<chartId>]]`
+ * markers), and `plannerPromise` so the caller can choose
+ * trace-spanning vs. snappy-return semantics.
+ */
+export declare function emitChartWithPlanning(opts: EmitChartWithPlanningOptions): Promise<EmitChartWithPlanningResult>;
+/**
+ * Build the `render_data` tool bound to the given plugin config.
+ *
+ * The tool is a thin wrapper around {@link emitChartWithPlanning}:
+ * a single `kind: "chart"` writer event ships the raw rows to
+ * the client immediately, the chart-planner agent runs alongside
+ * (so the calling LLM stays unblocked while the planner thinks),
+ * and a follow-up `kind: "chart"` event with the resolved
+ * `EChartsOption` lands when it's ready. The tool's `execute`
+ * awaits the planner promise so the planner work shows up under
+ * the tool's trace span; the LLM still gets back just
+ * `{ chartId }`, so its context stays small regardless of dataset
+ * size.
+ */
+export declare function buildRenderDataTool(config: MastraPluginConfig): import("@mastra/core/tools").Tool<any, any, any, any, import("@mastra/core/tools").ToolExecutionContext<any, any, unknown>, "render_data", unknown>;
+export {};