@mindstudio-ai/agent 0.1.163 → 0.1.165

package/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+What changed in `@mindstudio-ai/agent`, newest first, each version under its own heading. An entry says what's new, what's fixed, and what changed behavior, with what an app's code should be checked for. An entry whose version npm doesn't have yet is still open: a later change adds to it.
+
+Versions before 0.1.163 aren't recorded here.
+
+## 0.1.165
+
+### New
+
+- The CLI can log what each call costs. When `MINDSTUDIO_USAGE_LOG` names a file, every billed call from an action, `batch` (a line per step) or `ask` appends one JSON line to it: `{ ts, action, billingCost, billingEvents?, durationMs }`, with `billingCost` in nano-dollars. It's for a tool that runs the CLI and keeps its own record of spend, whatever happens to the command's output. App code isn't affected.
+
+## 0.1.164
+
+### New
+
+- The package carries this changelog, `CHANGELOG.md`, from this version on. Its entries start at 0.1.163.
+
+### Fixed
+
+- Filters on a column typed with a named type compile again. A column like `category: DocumentCategory` is stored as a json column. In 0.1.163, `===`, `!==` and `[...].includes(o.category)` with strings on it read the whole table into JS, and `o.category.includes('x')` failed the query with `malformed JSON` once a row held a plain string.
+- `o.field == null` on a json column also matches a row whose text is `null`, which reads back as null. That happens when the string `'null'` was written.
+
+## 0.1.163
+
+Published by release v1.0.2832.
+
+### Changed behavior
+
+A filter that compiles to SQL now selects exactly the rows its function would in JavaScript, NULL columns included. Before, some compiled to SQL that answered differently, and which answer an app got depended on whether the filter compiled.
+
+- `!==` and `!=` keep rows where the column is NULL, as in JS (`null !== 'archived'` is true). They used to drop them. That includes `removeAll`, which now deletes those rows too. Check: a `filter`, `count` or `removeAll` with `!==` on a column that can be empty, where empty rows should be left out. Add `o.status != null &&`.
+- `<`, `<=`, `>` and `>=` give JS's answer for a NULL column: `o.n < 5` keeps a row with no `n`, since `null < 5` is true in JS. So does `!(…)`: `!(o.status === 'a')` keeps empty rows. Check: comparisons on a nullable number column where empty rows should be left out.
+- `o.text.includes('x')` is case-sensitive, as JS's `includes` is, and `_` and `%` in the needle are literal characters. It used to be a case-insensitive SQL `LIKE`, in which a needle holding `_` or `%` matched nothing. Check: a search box written as `o.name.includes(q)` that relies on case-insensitive matching. Keep a lowercased copy of the field and compare `o.nameLower.includes(q.toLowerCase())` with `q` passed in bindings, or use `db.sql()` with `LIKE` (and `ESCAPE`).
+- `o.tags.includes('x')` on a json array column is array membership, as in JS. It used to be a substring of the column's JSON text, so `'x'` also matched `'xy'`.
+- A bare field reads by JS truthiness for its type: `o.name` is false for `''`, `o.count` for `0`, `!o.flag` is true for false and for an empty boolean. They used to compile to `= 1` and `= 0`, which was wrong for text and number columns.
+- A computed `sortBy` (anything but `o => o.field`, such as `o => o.meta.date` or `o => o.name.toLowerCase()`) sorts. It used to be ignored whenever `take()` or `skip()` applied, so `.sortBy(computed).take(10)` returned 10 arbitrary rows. The rows are now sorted in JS after the filter runs in SQL, so `take()` reads every matching row first, and on a large table that can reach `result_too_large`. Check: a computed `sortBy` with `take()` on a table that grows. Sort by a column (`o => o.createdAt`) where you can.
+- With no `sortBy`, `reverse()` and `last()` order by the reverse of the order rows were written in. They used to do nothing, so `last()` returned the first row.
+- A filter SQL can't answer exactly runs in JS over every row instead, with a warning in the logs: a value of another type than its column (`o.amount > '100'`), a relational or loose comparison inside a JSON field (`o.stats.score > 50`, `o.meta.kind == 'x'`), and `.length`. The answer is right; the cost is reading the whole table.
+
+### New
+
+- A comparison with a constant expression compiles: `o.createdAt > db.ago(db.days(30))`, `o.createdAt > Date.now() - 90 * 24 * 60 * 60 * 1000`, `o.amount > 50 * 100`. These used to read the whole table into JS.
+- Comparisons compile either way round: `5000 < o.amount`.
+- `.select(...columns)` on a query or table returns only those columns (`SELECT id, name`). Filters and sorts still see the whole row.
+- Reads a method starts together against one database, such as a `Promise.all` of queries, go to the platform as one round trip. Each gets its own results, and identical reads in flight share one answer. `db.batch()` is still the way to run reads and writes together, in order, as one transaction.
+- Every value in a compiled filter is a bound parameter, an `IN` list included, so a query's statement doesn't change with its values.
+
+### Fixed
+
+- Filters in a deployed method's bundled code compile. esbuild prints `5000` as `5e3` and `'Zürich'` as `"Z\xFCrich"`, which the compiler couldn't read: a round number of 1,000 or more read the whole table into JS, and a string with a non-ASCII character or an escape matched nothing at all.
+- `skip()` without `take()` failed with a SQL error.
+- `[null, 'a'].includes(o.x)` didn't match rows where the field is empty.

package/dist/cli.js

@@ -9,6 +9,31 @@ var __export = (target, all) => {
     __defProp(target, name, { get: all[name], enumerable: true });
 };
 
+// src/usageLog.ts
+import { appendFileSync } from "fs";
+function logUsage(entry) {
+  const file = process.env.MINDSTUDIO_USAGE_LOG;
+  if (!file || typeof entry.billingCost !== "number") {
+    return;
+  }
+  const line = {
+    ts: Date.now(),
+    action: entry.action,
+    billingCost: entry.billingCost,
+    ...entry.billingEvents?.length && { billingEvents: entry.billingEvents },
+    durationMs: Date.now() - entry.startedAt
+  };
+  try {
+    appendFileSync(file, JSON.stringify(line) + "\n");
+  } catch {
+  }
+}
+var init_usageLog = __esm({
+  "src/usageLog.ts"() {
+    "use strict";
+  }
+});
+
 // src/generated/metadata.ts
 var metadata_exports = {};
 __export(metadata_exports, {
@@ -2413,6 +2438,25 @@ function nestedComparison(node, negated, params) {
 function withNulls(core, field, nullsMatch) {
   return nullsMatch ? `(${core} OR ${field.sql} IS NULL)` : core;
 }
+function readsBackAsWritten(value) {
+  if (typeof value !== "string") {
+    return false;
+  }
+  try {
+    JSON.parse(value);
+    return false;
+  } catch {
+    return true;
+  }
+}
+function jsonColumnIn(field, values, negated, params) {
+  params.push(JSON.stringify(values.flatMap((v) => [v, JSON.stringify(v)])));
+  return withNulls(
+    `${field.sql} ${negated ? "NOT IN" : "IN"} (SELECT value FROM json_each(?))`,
+    field,
+    negated
+  );
+}
 function compileNode(node, negated, columns, params) {
   const column = (field) => field.nested ? void 0 : columns.find((c) => c.name === field.column);
   const stored = (field, value) => field.nested ? serializeColumnParam(value, "", []) : serializeColumnParam(value, field.column, columns);
@@ -2430,6 +2474,10 @@ function compileNode(node, negated, columns, params) {
         return nestedComparison(node, negated, params);
       }
       const type = column(node.field)?.type;
+      if (type === "json") {
+        const strictEquals = node.op === "===" ? !negated : node.op === "!==" ? negated : null;
+        return strictEquals !== null && readsBackAsWritten(node.value) ? jsonColumnIn(node.field, [node.value], !strictEquals, params) : null;
+      }
       if (!type || typeof node.value !== VALUE_KIND[type]) {
         return null;
       }
@@ -2450,10 +2498,16 @@ function compileNode(node, negated, columns, params) {
         return null;
       }
       const isNull = negated ? !node.isNull : node.isNull;
+      if (column(node.field)?.type === "json") {
+        return isNull ? `(${node.field.sql} IS NULL OR ${node.field.sql} = 'null')` : `(${node.field.sql} IS NOT NULL AND ${node.field.sql} <> 'null')`;
+      }
       return `${node.field.sql} ${isNull ? "IS NULL" : "IS NOT NULL"}`;
     }
     case "in": {
       const type = column(node.field)?.type;
+      if (type === "json") {
+        return node.values.every(readsBackAsWritten) ? jsonColumnIn(node.field, node.values, negated, params) : null;
+      }
       if (!type || node.values.some(
         (v) => v !== null && v !== void 0 && typeof v !== VALUE_KIND[type]
       )) {
@@ -2473,6 +2527,7 @@ function compileNode(node, negated, columns, params) {
     case "includes": {
       const type = column(node.field)?.type;
       if (type === "json") {
+        const col = node.field.sql;
         let element;
         if (typeof node.needle === "boolean") {
           element = `type = '${node.needle}'`;
@@ -2480,7 +2535,9 @@ function compileNode(node, negated, columns, params) {
           params.push(node.needle);
           element = `type IN ${JSON_TYPES[typeof node.needle]} AND value = ?`;
         }
-        return `(json_type(${node.field.sql}) = 'array' AND ${negated ? "NOT EXISTS" : "EXISTS"} (SELECT 1 FROM json_each(${node.field.sql}) WHERE ${element}))`;
+        const substring = `${negated ? "=" : ">"} 0`;
+        params.push(String(node.needle), String(node.needle));
+        return `(CASE WHEN json_valid(${col}) THEN CASE json_type(${col}) WHEN 'array' THEN ${negated ? "NOT EXISTS" : "EXISTS"} (SELECT 1 FROM json_each(${col}) WHERE ${element}) WHEN 'text' THEN instr(${col} ->> '$', ?) ${substring} ELSE 0 END ELSE instr(${col}, ?) ${substring} END)`;
       }
       if ((type === "text" || type === "user") && typeof node.needle === "string") {
         params.push(node.needle);
@@ -9275,6 +9332,7 @@ function summarizeInput(input) {
 async function cmdAsk(question, options) {
   try {
     let lineBuffer = "";
+    const startedAt = Date.now();
     const result = await runAsk(question, options, (event) => {
       switch (event.type) {
         case "text":
@@ -9304,6 +9362,12 @@ async function cmdAsk(question, options) {
     if (lineBuffer) {
       process.stderr.write(lineBuffer);
     }
+    logUsage({
+      action: "ask",
+      billingCost: result.billingCost,
+      billingEvents: result.billingEvents,
+      startedAt
+    });
     if (options.json) {
       process.stdout.write(
         JSON.stringify({
@@ -9328,6 +9392,7 @@ var init_ask = __esm({
   async "src/ask/index.ts"() {
     "use strict";
     await init_client();
+    init_usageLog();
     init_context2();
     init_sse();
     init_tools();
@@ -9402,7 +9467,7 @@ async function startMcpServer(options) {
           capabilities: { tools: {} },
           serverInfo: {
             name: "mindstudio-agent",
-            version: "0.1.163"
+            version: "0.1.165"
           },
           instructions: 'Welcome to MindStudio \u2014 a platform with 200+ AI models, 850+ third-party integrations, and pre-built agents.\n\nGetting started:\n1. Call `ask` with any question about the SDK \u2014 it knows every action, model, and connector and returns working code with real model IDs and config options. Examples: ask("generate an image with FLUX"), ask("what models support vision?"), ask("how do I send a Slack message?").\n2. Call `changeName` to set your display name \u2014 use your name or whatever your user calls you. This is how you\'ll appear in MindStudio request logs.\n3. If you have a profile picture, pass its public image URL to `changeProfilePicture`.\n4. For manual browsing, call `listActions` to discover all available actions.\n\nThen use the tools to generate text, images, video, audio, search the web, work with data sources, run agents, and more.\n\nImportant:\n- AI-powered actions (text generation, image generation, video, audio, etc.) cost money. Before running these, call `estimateActionCost` and confirm with the user before proceeding \u2014 unless they\'ve explicitly told you to go ahead.\n- Not all agents from `listAgents` are configured for API use. Do not try to run an agent just because it appears in the list \u2014 it will likely fail. Only run agents the user specifically asks you to run.'
         });
@@ -9853,6 +9918,7 @@ var init_mcp = __esm({
 });
 
 // src/cli.ts
+init_usageLog();
 import { parseArgs } from "util";
 import { execSync } from "child_process";
 import { readFileSync as readFileSync3 } from "fs";
@@ -10172,11 +10238,18 @@ async function cmdExec(method, input, options) {
 `);
     };
   }
+  const startedAt = Date.now();
   const result = await agent.executeStep(meta.stepType, input, {
     appId: options.appId,
     threadId: options.threadId,
     onLog
   });
+  logUsage({
+    action: camelToKebab(method),
+    billingCost: result.$billingCost,
+    billingEvents: result.$billingEvents,
+    startedAt
+  });
   if (options.outputKey) {
     const val = result[options.outputKey];
     if (typeof val === "string") {
@@ -10352,10 +10425,18 @@ Example: mindstudio batch '[{"stepType":"generateImage","step":{"prompt":"a cat"
     return { stepType: item.stepType, step: item.step };
   });
   const agent = await createAgent(options);
+  const startedAt = Date.now();
   const result = await agent.executeStepBatch(steps, {
     appId: options.appId,
     threadId: options.threadId
   });
+  for (const step of result.results) {
+    logUsage({
+      action: `batch:${step.stepType}`,
+      billingCost: step.billingCost,
+      startedAt
+    });
+  }
   if (options.noMeta) {
     process.stdout.write(JSON.stringify(result.results, null, 2) + "\n");
   } else {
@@ -10426,7 +10507,7 @@ function isNewerVersion(current, latest) {
   return false;
 }
 async function checkForUpdate() {
-  const currentVersion = "0.1.163";
+  const currentVersion = "0.1.165";
   if (!currentVersion || process.env.MINDSTUDIO_NO_UPDATE_CHECK) {
     return null;
   }
@@ -10461,7 +10542,7 @@ async function checkForUpdate() {
   }
 }
 function printUpdateNotice(latestVersion) {
-  const currentVersion = "0.1.163";
+  const currentVersion = "0.1.165";
   process.stderr.write(
     `
   ${ansi2.cyanBright("Update available")} ${ansi2.gray(currentVersion + " \u2192")} ${ansi2.cyanBold(latestVersion)}
@@ -10474,7 +10555,7 @@ function isStandaloneBinary() {
   return !argv1.includes("node_modules");
 }
 async function cmdUpdate() {
-  const currentVersion = "0.1.163";
+  const currentVersion = "0.1.165";
   process.stderr.write(
     `  ${ansi2.gray("Current version:")} ${currentVersion}
 `
@@ -10615,7 +10696,7 @@ async function cmdLogin(options) {
   process.stderr.write("\n");
   printLogo();
   process.stderr.write("\n");
-  const ver = "0.1.163";
+  const ver = "0.1.165";
   process.stderr.write(
     `  ${ansi2.bold("MindStudio Agent")} ${ver ? " " + ansi2.gray("v" + ver) : ""}
 `
@@ -10943,7 +11024,7 @@ async function main() {
   const updatePromise = command !== "mcp" && command !== "login" && command !== "update" ? checkForUpdate() : Promise.resolve(null);
   try {
     if (command === "version" || command === "-v") {
-      process.stdout.write("0.1.163\n");
+      process.stdout.write("0.1.165\n");
       return;
     }
     if (command === "login") {

package/dist/index.js

@@ -1594,6 +1594,25 @@ function nestedComparison(node, negated, params) {
 function withNulls(core, field, nullsMatch) {
   return nullsMatch ? `(${core} OR ${field.sql} IS NULL)` : core;
 }
+function readsBackAsWritten(value) {
+  if (typeof value !== "string") {
+    return false;
+  }
+  try {
+    JSON.parse(value);
+    return false;
+  } catch {
+    return true;
+  }
+}
+function jsonColumnIn(field, values, negated, params) {
+  params.push(JSON.stringify(values.flatMap((v) => [v, JSON.stringify(v)])));
+  return withNulls(
+    `${field.sql} ${negated ? "NOT IN" : "IN"} (SELECT value FROM json_each(?))`,
+    field,
+    negated
+  );
+}
 function compileNode(node, negated, columns, params) {
   const column = (field) => field.nested ? void 0 : columns.find((c) => c.name === field.column);
   const stored = (field, value) => field.nested ? serializeColumnParam(value, "", []) : serializeColumnParam(value, field.column, columns);
@@ -1611,6 +1630,10 @@ function compileNode(node, negated, columns, params) {
         return nestedComparison(node, negated, params);
       }
       const type = column(node.field)?.type;
+      if (type === "json") {
+        const strictEquals = node.op === "===" ? !negated : node.op === "!==" ? negated : null;
+        return strictEquals !== null && readsBackAsWritten(node.value) ? jsonColumnIn(node.field, [node.value], !strictEquals, params) : null;
+      }
       if (!type || typeof node.value !== VALUE_KIND[type]) {
         return null;
       }
@@ -1631,10 +1654,16 @@ function compileNode(node, negated, columns, params) {
         return null;
       }
       const isNull = negated ? !node.isNull : node.isNull;
+      if (column(node.field)?.type === "json") {
+        return isNull ? `(${node.field.sql} IS NULL OR ${node.field.sql} = 'null')` : `(${node.field.sql} IS NOT NULL AND ${node.field.sql} <> 'null')`;
+      }
       return `${node.field.sql} ${isNull ? "IS NULL" : "IS NOT NULL"}`;
     }
     case "in": {
       const type = column(node.field)?.type;
+      if (type === "json") {
+        return node.values.every(readsBackAsWritten) ? jsonColumnIn(node.field, node.values, negated, params) : null;
+      }
       if (!type || node.values.some(
         (v) => v !== null && v !== void 0 && typeof v !== VALUE_KIND[type]
       )) {
@@ -1654,6 +1683,7 @@ function compileNode(node, negated, columns, params) {
     case "includes": {
       const type = column(node.field)?.type;
       if (type === "json") {
+        const col = node.field.sql;
         let element;
         if (typeof node.needle === "boolean") {
           element = `type = '${node.needle}'`;
@@ -1661,7 +1691,9 @@ function compileNode(node, negated, columns, params) {
           params.push(node.needle);
           element = `type IN ${JSON_TYPES[typeof node.needle]} AND value = ?`;
         }
-        return `(json_type(${node.field.sql}) = 'array' AND ${negated ? "NOT EXISTS" : "EXISTS"} (SELECT 1 FROM json_each(${node.field.sql}) WHERE ${element}))`;
+        const substring = `${negated ? "=" : ">"} 0`;
+        params.push(String(node.needle), String(node.needle));
+        return `(CASE WHEN json_valid(${col}) THEN CASE json_type(${col}) WHEN 'array' THEN ${negated ? "NOT EXISTS" : "EXISTS"} (SELECT 1 FROM json_each(${col}) WHERE ${element}) WHEN 'text' THEN instr(${col} ->> '$', ?) ${substring} ELSE 0 END ELSE instr(${col}, ?) ${substring} END)`;
       }
       if ((type === "text" || type === "user") && typeof node.needle === "string") {
         params.push(node.needle);