troxy-cli 1.16.0 → 1.18.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "troxy-cli",
-  "version": "1.16.0",
+  "version": "1.18.0",
   "description": "Control layer for AI agents: check payments, emails, logins and destructive actions against your policies",
   "homepage": "https://troxy.io",
   "bugs": {

package/src/mcp-server.js

@@ -51,7 +51,27 @@ export async function runMcp() {
         // its own is the agent's own prediction, and the dashboard labels every
         // agent that never reports back as unverified, so the instruction has
         // to make the second call as non-optional as the first.
-        'You MUST also call the evaluate_model tool when you begin a substantial task and whenever you switch model or effort level partway through it, giving the exact model id and how many tokens you expect to use. Do not run the task without first receiving an ALLOW or NOTIFY decision; if BLOCK, pick a cheaper or smaller model and call evaluate_model again with the new id. Then, when the task finishes, you MUST call report_model_usage with the audit_id from that decision and the real token total from your model API. Both halves are required: without the second call Troxy has only your prediction, and the account owner is shown that this agent\'s costs are unverified. Report the true number even when it is far above your estimate.',
+        //
+        // The previous wording gated this on whether the task felt big
+        // enough, which was a judgment call, not a rule. Driven live: the
+        // same agent, same session, called evaluate_model to review two
+        // source files, then minutes later pulled real data from an actual
+        // connected business (orders, customer messages, an artifact) and
+        // called nothing. Both were real work by any reasonable reading. The
+        // old wording meant whatever the model felt like in the moment, and
+        // Token Optimization is supposed to be a precise record, not a mood.
+        // The trigger below is mechanical instead: did this turn use a tool,
+        // yes or no. That is a fact you can check, not a judgment you make.
+        'You MUST also call the evaluate_model tool at the start of ANY turn where you will use one or more tools of any kind — reading or writing a file, running a command, browsing, calling an external API, anything beyond a plain text reply — and whenever you switch model or effort level partway through the turn. This is not a judgment call about how the turn feels: if you are about to use a tool, call evaluate_model first, every time, regardless of how small the turn seems, and WAIT for the decision before proceeding. Give the exact model id and how many tokens you expect to use. Do not use any tool without first receiving an ALLOW or NOTIFY decision; if BLOCK, pick a cheaper or smaller model and call evaluate_model again with the new id; if ESCALATE, call wait_for_approval and do not proceed until it resolves. Then, at the end of that same turn, you MUST call report_model_usage with the audit_id from that decision and the real token total from your model API. Both halves are required: without the second call Troxy has only your prediction, and the account owner is shown that this agent\'s costs are unverified. Report the true number even when it is far above your estimate. ' +
+        // A turn that is PURE conversation, no tool use at all, still gets
+        // logged, just not gated. Blocking a chat reply on a network round
+        // trip protects against a risk that is vanishingly rare in practice
+        // (almost nobody writes a policy to block message content), and the
+        // cost of blocking anyway is a slower product on every single message.
+        // So chat calls the same two tools, in the same order, but treats the
+        // decision as a record rather than a gate: reply to the user first,
+        // report afterward, never hold the reply for Troxy's answer.
+        'A turn that is pure conversation, with no tool use at all, is different: you still call evaluate_model and report_model_usage for it, but you do NOT wait for or act on the decision before replying. Reply to the user first. Then, as the last step of that same turn (or the first step of your next turn if your environment does not let you act again after producing a reply), call evaluate_model with the model and your best token estimate, and call report_model_usage with the real total right after — same two tools as above, just not gating. If the decision comes back BLOCK or ESCALATE for a chat turn, do not retry with a different model and do not poll wait_for_approval: the reply has already been sent, there is nothing left to gate, and the decision is recorded for the account owner\'s visibility rather than enforced in the moment. Chat usage still counts toward every rolling spend and token limit, so it still shapes what the next tool-use call is allowed to do.',
     },
     { capabilities: { tools: {} } },
   );
@@ -268,8 +288,10 @@ export async function runMcp() {
         description:
           'Evaluate whether running a task on a given AI model should be allowed, blocked, or escalated ' +
           'based on your Troxy policies, and record what it is expected to cost. ' +
-          'Call this when you start a task and whenever you switch model or effort level mid-task. ' +
-          'Returns an audit_id. Keep it: you must pass it to report_model_usage when the task finishes.',
+          'Call this on every turn: at the start, before proceeding, if you will use a tool of any kind, or whenever you switch model or effort level mid-turn; ' +
+          'or after replying, without waiting for the decision, if the turn was pure conversation with no tool use. ' +
+          'Tool use gates on the decision; chat only records it. ' +
+          'Returns an audit_id. Keep it: you must pass it to report_model_usage when the turn finishes.',
         inputSchema: {
           type: 'object',
           required: ['model'],
@@ -309,8 +331,8 @@ export async function runMcp() {
       {
         name: 'report_model_usage',
         description:
-          'Report how many tokens a task ACTUALLY used, once it has finished. ' +
-          'Call this after every ALLOW or NOTIFY from evaluate_model, using the audit_id it returned. ' +
+          'Report how many tokens the turn ACTUALLY used, once it has finished. ' +
+          'Call this after every ALLOW or NOTIFY from evaluate_model, using the audit_id it returned — no judgment call, every approved evaluate_model gets a matching report. ' +
           'Without it Troxy only has your prediction, and the account owner is told, per agent, that its figures are unverified. ' +
           'Report the real number even when it is far above what you estimated: the gap is the point.',
         inputSchema: {
@@ -489,7 +511,7 @@ export async function runMcp() {
       // Every non-blocked branch repeats the audit_id and the instruction to
       // report back. Half the value of this checkpoint is the second call, and
       // an agent that is only told "approved" has no reason to make it.
-      const followUp = ` When the task finishes, call report_model_usage(audit_id="${audit_id}", actual_tokens=<real total>).`;
+      const followUp = ` When this turn finishes, call report_model_usage(audit_id="${audit_id}", actual_tokens=<real total>).`;
       let modelText;
       switch (decision) {
         case 'ALLOW':

package/src/tests/model-checkpoint.test.js

@@ -132,3 +132,116 @@ test('the version is read from package.json, not hardcoded', () => {
     'CLI_VERSION is not derived from package.json',
   );
 });
+
+test('the checkpoint trigger is mechanical, not a judgment call', () => {
+  // "When you begin a substantial task" was tried first and driven live: the
+  // same agent, same session, called evaluate_model to review two source
+  // files, then minutes later pulled real data from an actual connected
+  // business (orders, customer messages, an artifact) and called nothing.
+  // Both were real work. "Substantial" meant whatever the model felt like in
+  // the moment, which is not what a precise cost record can be built on.
+  // Sliced from the opening quote of the instructions STRING, not the
+  // instructions: field or file start, so the explanatory code comment
+  // above it (which deliberately quotes the retired word for context) does
+  // not trip this check.
+  const instructions = src.slice(src.indexOf("'You are connected to Troxy"), src.indexOf('capabilities'));
+  assert.ok(
+    !/substantial/i.test(instructions),
+    'the instructions still gate the checkpoint on "substantial", a judgment '
+    + 'call proven inconsistent on the same agent in the same session',
+  );
+  assert.ok(
+    /tool/i.test(instructions) && /every time/i.test(instructions),
+    'the instructions do not state a mechanical, checkable trigger (tool use) '
+    + 'to replace the judgment call that was removed',
+  );
+});
+
+test('evaluate_model\'s own tool description matches the mechanical trigger', () => {
+  // The top-level instructions are what an agent reads once; the tool
+  // description is what it sees every time it considers calling this
+  // specific tool. Both have to agree, or whichever one it reads last wins.
+  const schema = src.slice(src.indexOf("name: 'evaluate_model'"));
+  const desc = schema.slice(0, schema.indexOf('inputSchema'));
+  assert.ok(!/substantial/i.test(desc), 'the tool description still says "substantial"');
+  assert.ok(/tool/i.test(desc), 'the tool description does not mention the tool-use trigger');
+});
+
+test('pure chat still reports usage, it just does not gate on the decision', () => {
+  // The first version of this fix (tool-use = mechanical trigger) left chat
+  // unreported entirely, on the reasoning that chat isn't "work." That
+  // reasoning doesn't survive contact with the goal: chat still costs real
+  // tokens, and a page that silently misses all chat-only usage is not a
+  // precise record, it just moved the same gap somewhere less visible.
+  const instructions = src.slice(src.indexOf("'You are connected to Troxy"), src.indexOf('capabilities'));
+  const chatSeg = instructions.slice(instructions.indexOf('A turn that is pure conversation'));
+  assert.ok(chatSeg, 'no instruction exists for the pure-conversation case at all');
+  assert.ok(
+    /still call evaluate_model and report_model_usage/.test(chatSeg),
+    'chat turns are not instructed to call the checkpoint tools at all, so '
+    + 'chat-only usage is invisible on Token Optimization',
+  );
+});
+
+test('chat is told explicitly not to block the reply on the decision', () => {
+  // The whole point of treating chat differently from tool use is that a
+  // human is waiting on the reply. If the instructions still say "wait for
+  // the decision" for chat, this fix adds latency to every message for no
+  // behavioural gain over the tool-use path it was meant to avoid copying.
+  const instructions = src.slice(src.indexOf("'You are connected to Troxy"), src.indexOf('capabilities'));
+  const chatSeg = instructions.slice(instructions.indexOf('A turn that is pure conversation'));
+  assert.ok(
+    /do NOT wait for or act on the decision before replying/.test(chatSeg),
+    'chat instructions do not say to reply before waiting on the checkpoint',
+  );
+  assert.ok(
+    /Reply to the user first/.test(chatSeg),
+    'chat instructions do not establish reply-then-report ordering',
+  );
+});
+
+test('a BLOCK or ESCALATE on a chat turn does not trigger retry or polling', () => {
+  // Retrying with a cheaper model, or polling wait_for_approval, are the
+  // tool-use responses to BLOCK/ESCALATE. For chat the reply is already sent
+  // by the time the decision comes back, so both of those actions are
+  // meaningless and would just make the agent stall on a message that
+  // already went out.
+  const instructions = src.slice(src.indexOf("'You are connected to Troxy"), src.indexOf('capabilities'));
+  const chatSeg = instructions.slice(instructions.indexOf('A turn that is pure conversation'));
+  assert.ok(
+    /do not retry with a different model and do not poll wait_for_approval/.test(chatSeg),
+    'chat instructions do not rule out the tool-use recovery actions, which '
+    + 'would stall a reply that has already been sent',
+  );
+});
+
+test('chat usage still feeds the rolling limits that gate tool use', () => {
+  // Not gating chat in real time is only an acceptable trade if chat spend
+  // still counts toward spend_per_hour / tokens_per_day, so a chat-heavy
+  // agent still gets caught the moment it tries to use a tool. Otherwise
+  // "we don't block chat" quietly becomes "chat spend is uncapped."
+  const instructions = src.slice(src.indexOf("'You are connected to Troxy"), src.indexOf('capabilities'));
+  const chatSeg = instructions.slice(instructions.indexOf('A turn that is pure conversation'));
+  assert.ok(
+    /still counts toward every rolling spend and token limit/.test(chatSeg),
+    'nothing tells the agent that unblocked chat spend still accumulates '
+    + 'toward the limits a later tool-use call is checked against',
+  );
+});
+
+test('no new tool was introduced for the chat path', () => {
+  // The backend already supports this: report_model_usage looks up its row
+  // by id/user/agent, never by decision, so it works identically whether the
+  // row says ALLOW, BLOCK, or ESCALATE. Reusing evaluate_model +
+  // report_model_usage for chat means zero backend changes and zero new
+  // surface area to get wrong; a bespoke chat-logging tool would duplicate a
+  // path that is already tested end to end.
+  // Scoped to the tools array, not the whole file: the Server constructor
+  // also has a top-level `name: 'troxy'` for the server itself, which the
+  // unscoped regex was catching as a ninth "tool".
+  const toolsSeg = src.slice(src.indexOf('tools: ['), src.indexOf('server.setRequestHandler(CallToolRequestSchema'));
+  const toolNames = [...toolsSeg.matchAll(/name: '([a-z_]+)'/g)].map(m => m[1]);
+  const unique = new Set(toolNames);
+  assert.strictEqual(unique.size, 8, 'a tool was added or removed for this change');
+  assert.ok(unique.has('evaluate_model') && unique.has('report_model_usage'));
+});