agy-insights 0.8.1 → 0.8.2

package/README.es.md

@@ -36,7 +36,7 @@ npx agy-insights
 npx agy-insights@latest --text
 ```
 
-`@latest` hace que npx compruebe cada vez si hay una versión nueva. Sin él, una copia instalada globalmente (`npm install -g`) o en el proyecto actual tiene prioridad y no se actualiza nunca. Para fijar una versión concreta: `npx agy-insights@0.8.1`.
+`@latest` hace que npx compruebe cada vez si hay una versión nueva. Sin él, una copia instalada globalmente (`npm install -g`) o en el proyecto actual tiene prioridad y no se actualiza nunca. Para fijar una versión concreta: `npx agy-insights@0.8.2`.
 
 **Instalándolo globalmente:**
 

package/README.md

@@ -36,7 +36,7 @@ npx agy-insights
 npx agy-insights@latest --text
 ```
 
-`@latest` makes npx check for the newest version every time. Without it, a copy installed globally (`npm install -g`) or in the current project takes precedence and is never updated. To pin a version instead: `npx agy-insights@0.8.1`.
+`@latest` makes npx check for the newest version every time. Without it, a copy installed globally (`npm install -g`) or in the current project takes precedence and is never updated. To pin a version instead: `npx agy-insights@0.8.2`.
 
 **Globally:**
 

package/docs/how-it-works.md

@@ -35,7 +35,7 @@ It is undocumented; this is what has been observed in `agy` 1.4.x.
 - Table `steps`: one row per step. `step_payload` is a protobuf message. Tool calls store the name (`view_file`, `run_command`, `grep_search`…) followed by their JSON arguments, for example `{"AbsolutePath":"C:\\dev\\...\\X.cs","StartLine":301,"EndLine":380}`, preceded by a `call_NNN` id.
 - Each call takes two steps with the same id: the model response that requests it (`step_type` 15) and the result (`step_type` 132), which repeats name and arguments and adds the file content or command output. `lib/history.js` counts the call once per id and adds the size of both steps, which approximates what entered the context.
 - Table `gen_metadata`: one row per model call, in the same order as the transcript's `PLANNER_RESPONSE` lines. Each row holds the model's display name with the effort level in brackets (`Gemini 3.8 Flash (High)`) and its id (`gemini-3.8-flash-high`). `modelOf()` takes the model and effort from whole strings only, so prose in the system prompt that mentions a model is ignored, and the n-th row is matched to the n-th call to get tokens per model and effort. That pairing is only trusted when the counts match. Otherwise each call takes the model selected in the transcript (`USER_INPUT` lines carry "changed setting `Model Selection` from X to Y"), or the conversation's only model; what is left is "unknown". `modelKey()` merges the two spellings (`Gemini 3.8 Flash` / `gemini-3.8-flash`, `High` / `high`). `--debug-models` prints, per conversation, how many calls got their model from each source: counts only, safe to paste into an issue.
-- Next to the history, `../brain/<id>/.system_generated/logs/transcript.jsonl`: one JSON line per step. `PLANNER_RESPONSE` lines are model calls and carry `input_tokens`, `cache_read_tokens` and `output_tokens`; failed steps have `status: "ERROR"`. Subagents have their own conversation and transcript. `realUsage()` in `lib/history.js` adds those figures up (it only reads the file). The total printed by `agy -p --output-format json` does not include subagents; this one does.
+- Next to the history, `../brain/<id>/.system_generated/logs/transcript.jsonl`: one JSON line per step. `PLANNER_RESPONSE` lines are model calls and carry `input_tokens`, `cache_read_tokens` and `output_tokens`; failed steps have `status: "ERROR"`. Subagents have their own conversation and transcript. `realUsage()` in `lib/history.js` adds those figures up (it only reads the file). The total printed by `agy -p --output-format json` does not include subagents; this one does. Some logs (seen with the VS Code plugin) have `PLANNER_RESPONSE` lines without any token field; those are not counted as 0 tokens. The same figures are also in each `gen_metadata` row, at protobuf field 1 › 4 (2 = input, 3 = output, 5 = cache read; the model's display name is at 1 › 21). When the transcript has no counts, or there is no transcript, `usageFromMeta()` takes real usage from there, keeping the day and selected model of the transcript's n-th call when there is one.
 
 Each `transcript.jsonl` line also has `created_at` (ISO time). A conversation is dated by the first and last of those; without a transcript, by the `.db` file's last write. `--since` / `--days` keep a conversation when its last activity falls inside the period, and skip the rest before opening them. `../conversation_summaries.db` has one row per conversation: its short `title` (or `preview` in older ones) names each session in the report instead of its id; `last_modified_time` is not used. If the file is missing, sessions are shown by id. Each model call's `created_at` also puts its tokens on a local day (`r.daily`), which feeds the usage chart.
 

package/lib/history.js

@@ -114,6 +114,39 @@ function collect(blob) {
   return out;
 }
 
+// The top-level fields of one protobuf message: tag -> varint value or bytes (last one wins). Null if malformed.
+function fields(buf) {
+  const out = new Map();
+  let i = 0;
+  const varint = () => {
+    let v = 0, s = 0, b;
+    do { if (i >= buf.length) throw 0; b = buf[i++]; v += (b & 0x7f) * 2 ** s; s += 7; } while (b & 0x80);
+    return v;
+  };
+  try {
+    while (i < buf.length) {
+      const k = varint(), tag = Math.floor(k / 8), wt = k & 7;
+      if (wt === 0) out.set(tag, varint());
+      else if (wt === 1) i += 8;
+      else if (wt === 5) i += 4;
+      else if (wt === 2) { const len = varint(); if (i + len > buf.length) throw 0; out.set(tag, buf.subarray(i, i + len)); i += len; } else throw 0;
+    }
+    return out;
+  } catch { return null; }
+}
+
+// Token counts of one gen_metadata row (one model call): field 1 › 4 holds 2 = input, 3 = output,
+// 5 = cache read, the same figures the transcript logs. Null when the row has none: some logs (seen
+// with the VS Code plugin) carry no token counts, and this is then the only place that has them.
+function metaUsage(blob) {
+  const top = blob && fields(Buffer.from(blob));
+  const call = Buffer.isBuffer(top?.get(1)) && fields(top.get(1));
+  const u = Buffer.isBuffer(call?.get(4)) && fields(call.get(4));
+  if (!u || !(u.has(2) || u.has(3))) return null;
+  const n = (k) => (typeof u.get(k) === 'number' ? u.get(k) : 0);
+  return { input: n(2), output: n(3), cache: n(5) };
+}
+
 // A tool call is stored as the tool name followed (within a few strings) by its JSON arguments,
 // e.g. "view_file" + {"AbsolutePath":"C:\\dev\\...\\X.cs","StartLine":301,"EndLine":380}.
 // The call id ("call_5859111") just before the name identifies the call: agy stores each call twice,
@@ -242,7 +275,7 @@ function realUsage(dir, id) {
     if (!['input_tokens', 'cache_read_tokens', 'output_tokens'].some((k) => k in e)) {
       u.unmetered++;
       for (const k of Object.keys(e)) if (k.length < 40) u.unmeteredKeys.add(k);
-      u.perCall.push({ metered: false, model: selected });
+      u.perCall.push({ metered: false, model: selected, at });
       continue;
     }
     u.calls++;
@@ -261,6 +294,27 @@ function realUsage(dir, id) {
   return u;
 }
 
+// Real usage rebuilt from gen_metadata, one row per model call, for conversations whose log has no token
+// counts (or no log). Each row's call keeps the model and day of the log's n-th call when there is one.
+function usageFromMeta(blobs, log, end) {
+  const rows = blobs.map(metaUsage);
+  if (!rows.some(Boolean)) return null;
+  const u = { calls: 0, input: 0, cache: 0, output: 0, errors: log?.errors || 0, start: log?.start ?? null, end: log?.end ?? null,
+    days: new Map(), perCall: [], unmetered: 0, unmeteredKeys: new Set(), unrecognized: log?.unrecognized || new Set(), fromMeta: true };
+  rows.forEach((x, i) => {
+    const from = log?.perCall[i];
+    if (!x) { u.perCall.push({ metered: false, model: from?.model || null }); return; }
+    u.calls++;
+    u.input += x.input; u.cache += x.cache; u.output += x.output;
+    u.perCall.push({ metered: true, ...x, model: from?.model || null });
+    const k = dayKey(from?.at || log?.end || end);
+    const d = u.days.get(k) || { calls: 0, input: 0, cache: 0, output: 0 };
+    d.calls++; d.input += x.input; d.cache += x.cache; d.output += x.output;
+    u.days.set(k, d);
+  });
+  return u;
+}
+
 // --- Titles ---
 
 // The short title agy gives each conversation ("Analyzing Package Lock Dependencies"), from
@@ -329,8 +383,8 @@ async function analyze({ dir = DEFAULT_DIR, workspaces = [], customKinds = {}, s
       const log = realUsage(dir, id);
       const end = log?.end ?? fs.statSync(path.join(dir, f)).mtimeMs;
       const start = log?.start ?? end;
-      // Real usage only when the log has token counts; otherwise the session is shown with the estimate.
-      const usage = log?.calls ? log : null;
+      // Real usage from the log when it has token counts; else from gen_metadata (below), else the estimate.
+      let usage = log?.calls ? log : null;
       if (since && end < since) { r.olderSkipped++; continue; }
       let db;
       try { db = await openDb(path.join(dir, f), tmp); } catch { r.unreadable++; continue; }
@@ -390,6 +444,8 @@ async function analyze({ dir = DEFAULT_DIR, workspaces = [], customKinds = {}, s
             files.add(key(file), { ...fields, convs: once(seenFiles, key(file)) }, { abs: file, rel, root, project, kind: k });
           });
         }
+        const metaBlobs = db.all('select data from gen_metadata order by idx').map(([blob]) => blob);
+        if (!usage) usage = usageFromMeta(metaBlobs, log, end);
         if (usage) {
           r.usage.conversations++;
           for (const k of ['calls', 'input', 'cache', 'output', 'errors']) r.usage[k] += usage[k];
@@ -404,9 +460,9 @@ async function analyze({ dir = DEFAULT_DIR, workspaces = [], customKinds = {}, s
         // Model of each call. gen_metadata usually has one row per call, in the transcript's order; when the
         // counts match, the n-th row is the n-th call. When they don't, pairing by position would mislabel
         // calls, so the model selected in the transcript is used instead, or the conversation's only model.
-        const metaStrs = db.all('select data from gen_metadata order by idx').map(([blob]) => collect(blob));
+        const metaStrs = metaBlobs.map(collect);
         const metaRows = metaStrs.map(modelOf);
-        const calls = log ? log.perCall : metaRows.map(() => null);
+        const calls = usage?.fromMeta ? usage.perCall : log ? log.perCall : metaRows.map(() => null);
         const aligned = metaRows.length === calls.length;
         const rowModels = new Set(metaRows.filter((m) => m.model).map(modelKey));
         const onlyModel = rowModels.size === 1 ? metaRows.find((m) => m.model) : null;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agy-insights",
-  "version": "0.8.1",
+  "version": "0.8.2",
   "description": "Measures Antigravity CLI token usage and shows which context is sent repeatedly or needlessly, in English or Spanish. Run: npx agy-insights@latest",
   "license": "MIT",
   "keywords": [