@ascenda-one/history-import 0.1.14 → 0.1.16

package/README.md

@@ -30,8 +30,15 @@ command (same pattern as hooks pairing) and this CLI does the reading.
 4. **Metrics only by default.** Prompt/response text, thinking blocks and
    file contents never leave the machine. Content-level ingestion, if it ever
    ships, is a separate explicit opt-in — not this package's default path.
-5. **Aggregate before shipping.** Per-session / per-day events, not one event
-   per bubble — a single machine's stores hold tens of thousands of them.
+5. **Aggregate before shipping, unless a reader counts the rows.** Per-session
+   / per-day events, not one event per bubble — a single machine's stores hold
+   tens of thousands of them. The one deliberate exception is
+   `ai_tool_call_started`: the backend's work-demand rail derives
+   `toolCallCount` by counting rows of that type and reads no `toolCallCount`
+   key off metadata, so a session-level aggregate would ship, store, and be
+   counted by nothing. Per-call events also place the work in the right hour,
+   which a session spanning six of them cannot. Expect an order of magnitude
+   more events than a session-only import, and an `events.jsonl` to match.
 6. **Provenance is data.** Every event carries `historical_direct`,
    `historical_derived` or `historical_unparsed` — never the live
    `ai_work_telemetry` provenance — so no chart can pass history off as
@@ -46,6 +53,8 @@ command (same pattern as hooks pairing) and this CLI does the reading.
 | Staging/snapshot (copy-then-parse, WAL-aware, **torn down by the run that makes it**) | implemented |
 | `archive` (durable content-addressed copy, dedup, verify, restore, prune) | **implemented, verified on a real 4.1 GB store** |
 | **Claude Code extractor** (human-prompt/tool-result split, session folds incl. recursive subagent transcripts, after-hours, compaction, tool failures, context-window peak, human-corrected edits, correction cadence, gap-split active minutes, epoch marker) | **implemented, verified live** |
+| **Active-time split** (hands-on vs agent-supervising, per session, per local day and per project digest; autonomy bands off the transcript's own `permissionMode`) | **implemented, verified against a real 400-session store** |
+| **Tool-call counting, all three stores** (`tool_use` items / `toolFormerData` / `toolInvocationSerialized`, one `ai_tool_call_started` per call) | **implemented; exercised end to end against all three stores on a developer machine** |
 | **Batch shipper** (`POST /v1/tool-events/batch`, salted hashes, stable importKey) | **implemented, verified live** |
 | **Cursor extractor** (composerHeaders + bubble aggregation via SQL-side `json_extract`, prompt text never parsed into the process, subagent-composer folding, epoch marker) | **implemented, verified live** |
 | **VS Code extractor** (Timeline-history Chat-Edit day×workspace aggregation, Copilot chatSessions folding, workspace identity via `workspace.json` longest-prefix match, epoch marker) | **implemented, verified live** |
@@ -58,12 +67,84 @@ user-role transcript lines are tool-result round-trips, not typed prompts.
 Conflating the two inflates every prompt metric by roughly an order of
 magnitude, so its fixtures are the ones to keep green.
 
-## Gaps that block a real user running this twice
+## Active time is two figures, never one
 
-Both blockers are enforced server-side, and both are **still landing**. Until
-they are live in the deployed backend, this package stays unpublished — a
-client anyone could install must not be able to backfill history against a
-backend that has not yet gated it.
+`activeMinutes` answers "how much of this session was not idle": every
+known-line timestamp, main thread and subagents merged, gap-split at five
+minutes. It has always been the honest alternative to wall clock, and it is
+unchanged.
+
+It is not, on its own, an answer to "how long did this take me". One prompt can
+drive a forty-minute agent run, and forty minutes of an agent working is not
+forty minutes of a person at a keyboard. So the same material is also reported
+split:
+
+| Figure | What it is |
+|---|---|
+| `handsOnMinutes` | The interval immediately **preceding** a human prompt. The prompt at its end is the evidence: someone read the previous output and typed. |
+| `agentSupervisingMinutes` | Every other active interval. The agent produced the lines that bound it. |
+
+The two partition `activeMinutes` exactly and there is **no third key holding
+their sum**, at session, day or project scale. Adding them reconstructs
+`activeMinutes`, which already exists; a differently-named total would be the
+same number wearing a claim it cannot support. On a real 400-session store the
+split came out 1,074 hands-on minutes against 36,948 supervising — quoting the
+combined 38,035 as time spent is off by a factor of thirty-five for the half a
+person would recognise as their own.
+
+**`agentSupervisingMinutes` does not claim anyone was watching**, and nothing
+in a transcript could show that they were. It is time the agent was working
+which the person did not spend typing. Rendering it as attention is a
+fabrication the name invites and the data does not support; the honest gloss is
+"the agent was working".
+
+### Autonomy bands
+
+`permissionMode` is on the transcript's human-prompt lines and nowhere else —
+across 120 real stores it appears on 6.7% of `user` lines and on no
+`assistant`, `system` or `attachment` line. So posture is known at prompt
+boundaries and carried forward between them, and supervising minutes are banded
+by it through `autonomyBand`. Time before the first declaration lands in
+`unknown` and is never folded into a neighbouring band.
+
+The band map rides in the local handoff only (`autonomySplit`), never on the
+wire: banding is a reader's vocabulary derived from the stored token at query
+time, and storing the band would freeze a decision deliberately left open.
+
+### The counters
+
+Three diagnostics ship with the split, read by neither the backend nor the
+handoff on purpose:
+
+- `activeSplitInstants` — distinct timestamps the split ran over. Two minutes
+  off four instants and off four hundred are not the same measurement.
+- `activeSplitUndatedLines` — known lines whose `timestamp` would not parse.
+  They still move the session's wall clock by string comparison, so only this
+  says both active figures are short.
+- `activeSplitUnposturedInstants` — instants reached before any
+  `permissionMode` was declared. The posture blind spot as a count, not
+  inferred from the `unknown` band being present.
+
+### The defect this replaced
+
+The per-day slices used to gap-split the **prompt timestamps** while the
+session figure gap-split the **whole timeline**. The threshold was shared and
+commented as keeping one definition of "active"; the material was not. Across
+200 real sessions the prompts-only reading came to 2,730 minutes against
+18,938 — an 85.6% under-report, concentrated exactly on the sessions where an
+agent did the most work. Both now cross the call, and
+`tests/activeSplit.test.mjs` pins it against a real transcript.
+
+## What the backend enforces on a second run
+
+Both of the blockers this section used to list — the consent scope and
+idempotency — are enforced by the deployed backend, and have been since
+20 Aug 2026 (verified against a real backfill on 25 Aug 2026). They stopped
+gating publication then, and the package ships from the release tag like
+every other CLI here. Two things stand apart from them and are decided
+elsewhere: which tier the batch ship belongs to, and the consent surface that
+grants the lease. Without that grant every event of a backfill is rejected
+`consent_missing_or_expired` — the gate working, not a broken token.
 
 - **Consent scope.** `historical_import` is a distinct `ToolConsentScope`,
   separate from the lease granted for live IDE telemetry, and a backfill
@@ -119,7 +200,7 @@ ascenda-history-import archive         # the durable copy; --verify / --list / -
 ### Staging is scaffolding; `archive` is the copy
 
 `import` snapshots each store, extracts, and **deletes the snapshot in a
-`finally`** — success or failure — keeping only the ~10 MB `events.jsonl`.
+`finally`** — success or failure — keeping only `events.jsonl`.
 It also sweeps snapshots left by earlier runs. This is not tidiness: nineteen
 runs once left 254 GB on a 926 GB disk and took free space to 279 MB, and the
 first thing to notice was unrelated tooling failing with `ENOSPC`.

package/dist/activeSplit.js

@@ -0,0 +1,159 @@
+/**
+ * Active time, split by who was doing the work.
+ *
+ * `activeMinutes` already answers "how much of this session was not idle" by
+ * gap-splitting every known-line timestamp. It cannot answer the question the
+ * Reveal actually wants to ask, which is what that time *was*: one prompt can
+ * drive a forty-minute agent run, and forty minutes of watching an agent is
+ * not forty minutes of a person at a keyboard. A single figure reports them as
+ * the same thing.
+ *
+ * This module splits the same gap-split material in two:
+ *
+ *  - **hands-on** — the interval immediately *preceding* a human prompt. The
+ *    prompt at its end is the evidence: someone read the previous output and
+ *    typed. It is the only interval in a transcript where a person is
+ *    demonstrably present, because it is the only one a person signed.
+ *  - **agent-supervising** — every other active interval. The agent produced
+ *    the lines that bound it.
+ *
+ * **What "supervising" does not claim.** It does not claim the person was
+ * watching, and nothing in a transcript could show that they were. It is time
+ * the agent was working which the person did not spend typing — no more. A
+ * reader that renders it as attention is reading a fabrication into it; the
+ * name is the plan's, the bound is this paragraph. The honest gloss is
+ * "the agent was working", not "you were supervising".
+ *
+ * **Two figures, never one total.** Nothing here returns their sum. They
+ * partition the same active milliseconds exactly (pinned by test), so a caller
+ * *can* add them — but adding them reconstructs `activeMinutes`, which already
+ * exists and already has readers. A third number spelled like a new
+ * measurement would be the same number wearing a claim it cannot support, and
+ * the whole point of the split is that the two halves differ.
+ *
+ * ## Posture
+ *
+ * `permissionMode` is on the transcript's human prompt lines and nowhere else
+ * — checked across 120 real stores: 6.7% of `user` lines carry it, no
+ * `assistant`, `system` or `attachment` line ever does. So posture is known at
+ * prompt boundaries and interpolated between them: the mode declared by the
+ * most recent human prompt at or before an interval governs that interval,
+ * because that is when the person last told the runtime what it could do
+ * unasked.
+ *
+ * Before the first declaration there is nothing to carry forward, and those
+ * milliseconds land in `unknown` — never folded into a neighbouring band, for
+ * the reason `autonomyBand` states at length: a guess there would look exactly
+ * like a measurement. `unknown` being non-zero is normal and is not a defect.
+ *
+ * Only supervising time is banded. Posture describes how much latitude the
+ * agent had while working; applied to the interval where the person was typing
+ * it would describe nothing.
+ */
+import { autonomyBand } from "@ascenda-one/tool-kit";
+/**
+ * Upstream's `permissionMode` spellings, snake-cased — the same one
+ * transformation `AutonomyMode` documents, applied here because the transcript
+ * writes camelCase where the hook payloads write snake_case. Mirroring rather
+ * than translating: a mode this table has never seen is passed through
+ * verbatim, so it reaches `autonomyBand` as an unrecognised token and lands in
+ * `unknown` instead of being quietly mapped onto whichever rung looks close.
+ */
+export function snakeCasePermissionMode(mode) {
+    return mode.replace(/([a-z0-9])([A-Z])/g, "$1_$2").toLowerCase();
+}
+/**
+ * The active spans of a timeline, in order.
+ *
+ * **Ties are collapsed, not ordered.** A parallel tool batch writes several
+ * lines on one millisecond, and a human prompt can share an instant with the
+ * attachment lines that accompany it. Sorting such lines against each other
+ * would make the split depend on a within-millisecond order the store does not
+ * promise and the reader cannot see. So every line at one epoch millisecond
+ * becomes one instant, `human` if *any* of them was a human prompt and
+ * carrying whichever posture was declared there. Intervals between tied lines
+ * are zero-length and are not spans, so nothing is lost by it.
+ */
+export function activeSpans(points, options) {
+    const collapsed = new Map();
+    let undatedPoints = 0;
+    for (const point of points) {
+        if (!Number.isFinite(point.at)) {
+            undatedPoints += 1;
+            continue;
+        }
+        const existing = collapsed.get(point.at);
+        if (existing) {
+            existing.human = existing.human || point.human;
+            if (point.autonomyMode)
+                existing.autonomyMode = point.autonomyMode;
+        }
+        else {
+            collapsed.set(point.at, { human: point.human, autonomyMode: point.autonomyMode });
+        }
+    }
+    const instants = [...collapsed.keys()].sort((a, b) => a - b);
+    const report = {
+        spans: [],
+        undatedPoints,
+        instants: instants.length,
+        unposturedInstants: 0
+    };
+    if (instants.length === 0)
+        return report;
+    // The posture in force, carried forward from the last human prompt that
+    // declared one. Seeded from the first instant before the loop so a session
+    // whose opening prompt declares a mode is not credited with an unknown
+    // stretch it never had.
+    let posture = collapsed.get(instants[0])?.autonomyMode ?? null;
+    if (posture === null)
+        report.unposturedInstants += 1;
+    for (let i = 1; i < instants.length; i += 1) {
+        const from = instants[i - 1];
+        const to = instants[i];
+        const here = collapsed.get(to);
+        const gap = to - from;
+        if (gap > 0 && gap <= options.activeGapMs) {
+            // `autonomyBand` maps a null or unrecognised token to `unknown` on its
+            // own; passing the raw carried value keeps that decision in the one
+            // place that documents it.
+            report.spans.push({ from, to, handsOn: here.human, band: autonomyBand(posture) });
+        }
+        if (here.autonomyMode)
+            posture = here.autonomyMode;
+        else if (posture === null)
+            report.unposturedInstants += 1;
+    }
+    return report;
+}
+/**
+ * Splits a session's timeline into hands-on and agent-supervising time, by
+ * summing {@link activeSpans}.
+ */
+export function splitActiveTime(points, options) {
+    const report = activeSpans(points, options);
+    const split = {
+        handsOnMs: 0,
+        agentSupervisingMs: 0,
+        supervisingMsByBand: {},
+        undatedPoints: report.undatedPoints,
+        instants: report.instants,
+        unposturedInstants: report.unposturedInstants
+    };
+    for (const span of report.spans) {
+        const ms = span.to - span.from;
+        if (span.handsOn) {
+            split.handsOnMs += ms;
+        }
+        else {
+            split.agentSupervisingMs += ms;
+            split.supervisingMsByBand[span.band] = (split.supervisingMsByBand[span.band] ?? 0) + ms;
+        }
+    }
+    return split;
+}
+/** Whole minutes, rounded once at the edge. */
+export function minutesOf(ms) {
+    return Math.round(ms / 60_000);
+}
+//# sourceMappingURL=activeSplit.js.map

package/dist/activeSplit.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"activeSplit.js","sourceRoot":"","sources":["../src/activeSplit.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAmDG;AACH,OAAO,EAAE,YAAY,EAAqB,MAAM,uBAAuB,CAAC;AAExE;;;;;;;GAOG;AACH,MAAM,UAAU,uBAAuB,CAAC,IAAY;IAClD,OAAO,IAAI,CAAC,OAAO,CAAC,oBAAoB,EAAE,OAAO,CAAC,CAAC,WAAW,EAAE,CAAC;AACnE,CAAC;AAoGD;;;;;;;;;;;GAWG;AACH,MAAM,UAAU,WAAW,CACzB,MAAgC,EAChC,OAAqB;IAErB,MAAM,SAAS,GAAG,IAAI,GAAG,EAA2D,CAAC;IACrF,IAAI,aAAa,GAAG,CAAC,CAAC;IACtB,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;QAC3B,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,CAAC;YAC/B,aAAa,IAAI,CAAC,CAAC;YACnB,SAAS;QACX,CAAC;QACD,MAAM,QAAQ,GAAG,SAAS,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACzC,IAAI,QAAQ,EAAE,CAAC;YACb,QAAQ,CAAC,KAAK,GAAG,QAAQ,CAAC,KAAK,IAAI,KAAK,CAAC,KAAK,CAAC;YAC/C,IAAI,KAAK,CAAC,YAAY;gBAAE,QAAQ,CAAC,YAAY,GAAG,KAAK,CAAC,YAAY,CAAC;QACrE,CAAC;aAAM,CAAC;YACN,SAAS,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,EAAE,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,YAAY,EAAE,KAAK,CAAC,YAAY,EAAE,CAAC,CAAC;QACpF,CAAC;IACH,CAAC;IAED,MAAM,QAAQ,GAAG,CAAC,GAAG,SAAS,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IAC7D,MAAM,MAAM,GAAqB;QAC/B,KAAK,EAAE,EAAE;QACT,aAAa;QACb,QAAQ,EAAE,QAAQ,CAAC,MAAM;QACzB,kBAAkB,EAAE,CAAC;KACtB,CAAC;IACF,IAAI,QAAQ,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,MAAM,CAAC;IAEzC,wEAAwE;IACxE,2EAA2E;IAC3E,uEAAuE;IACvE,wBAAwB;IACxB,IAAI,OAAO,GAAkB,SAAS,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,EAAE,YAAY,IAAI,IAAI,CAAC;IAC9E,IAAI,OAAO,KAAK,IAAI;QAAE,MAAM,CAAC,kBAAkB,IAAI,CAAC,CAAC;IAErD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,QAAQ,CAAC,MAAM,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC;QAC5C,MAAM,IAAI,GAAG,QAAQ,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;QAC7B,MAAM,EAAE,GAAG,QAAQ,CAAC,CAAC,CAAC,CAAC;QACvB,MAAM,IAAI,GAAG,SAAS,CAAC,GAAG,CAAC,EAAE,CAAE,CAAC;QAChC,MAAM,GAAG,GAAG,EAAE,GAAG,IAAI,CAAC;QACtB,IAAI,GAAG,GAAG,CAAC,IAAI,GAAG,IAAI,OAAO,CAAC,WAAW,EAAE,CAAC;YAC1C,uEAAuE;YACvE,oEAAoE;YACpE,2BAA2B;YAC3B,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,EAAE,EAAE,OAAO,EAAE,IAAI,CAAC,KAAK,EAAE,IAAI,EAAE,YAAY,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC;QACpF,CAAC;QACD,IAAI,IAAI,CAAC,YAAY;YAAE,OAAO,GAAG,IAAI,CAAC,YAAY,CAAC;aAC9C,IAAI,OAAO,KAAK,IAAI;YAAE,MAAM,CAAC,kBAAkB,IAAI,CAAC,CAAC;IAC5D,CAAC;IAED,OAAO,MAAM,CAAC;AAChB,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,eAAe,CAC7B,MAAgC,EAChC,OAAqB;IAErB,MAAM,MAAM,GAAG,WAAW,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IAC5C,MAAM,KAAK,GAAgB;QACzB,SAAS,EAAE,CAAC;QACZ,kBAAkB,EAAE,CAAC;QACrB,mBAAmB,EAAE,EAAE;QACvB,aAAa,EAAE,MAAM,CAAC,aAAa;QACnC,QAAQ,EAAE,MAAM,CAAC,QAAQ;QACzB,kBAAkB,EAAE,MAAM,CAAC,kBAAkB;KAC9C,CAAC;IACF,KAAK,MAAM,IAAI,IAAI,MAAM,CAAC,KAAK,EAAE,CAAC;QAChC,MAAM,EAAE,GAAG,IAAI,CAAC,EAAE,GAAG,IAAI,CAAC,IAAI,CAAC;QAC/B,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;YACjB,KAAK,CAAC,SAAS,IAAI,EAAE,CAAC;QACxB,CAAC;aAAM,CAAC;YACN,KAAK,CAAC,kBAAkB,IAAI,EAAE,CAAC;YAC/B,KAAK,CAAC,mBAAmB,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,mBAAmB,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,EAAE,CAAC;QAC1F,CAAC;IACH,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,+CAA+C;AAC/C,MAAM,UAAU,SAAS,CAAC,EAAU;IAClC,OAAO,IAAI,CAAC,KAAK,CAAC,EAAE,GAAG,MAAM,CAAC,CAAC;AACjC,CAAC"}