copilot-metrics 0.1.0 → 0.1.2

package/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## 0.1.2 - 2026-05-31
+
+### Fixed
+
+- Reports now import Copilot CLI session-state `events.jsonl` files by default, so token statistics are collected after `init` and hooks install without requiring users to export telemetry environment variables.
+- Copilot session-state imports persist only shutdown usage records, while using prompt-bearing session events in memory for label extraction and context.
+- Hook-only report diagnostics now stay quiet when token-bearing Copilot session-state usage is available.
+
+## 0.1.1 - 2026-05-30
+
+### Added
+
+- Setup-once report flow: report commands automatically import configured VS Code, Copilot CLI, and hook JSONL sources before querying.
+- Idempotent import fingerprints so repeated reports or imports do not double-count previously ingested JSONL rows.
+- Complete label token reporting for input, output, cache read, cache creation, and reasoning tokens in human and JSON output.
+- Hook-only label status so attribution evidence from hooks remains visible without implying token-bearing usage.
+
+### Changed
+
+- `setup all` now persists the central data directory config, matching `init` for setup-once usage.
+- Hook commands support installed executable shims as well as checkout-local JavaScript entrypoints.
+
 ## 0.1.0 - 2026-05-30
 
 First local release candidate for `copilot-metrics`.

package/README.md

@@ -9,8 +9,8 @@ Costs are estimates, not official billing records. GitHub billing remains the so
 From npm:
 
 ```bash
-npx copilot-metrics@0.1.0 --help
-npx copilot-metrics@0.1.0 init
+npx copilot-metrics@0.1.2 --help
+npx copilot-metrics@0.1.2 init
 ```
 
 From this checkout:
@@ -38,24 +38,28 @@ export COPILOT_METRICS_HOME=/path/to/copilot-metrics-data
 Useful commands:
 
 ```bash
-npx copilot-metrics@0.1.0 init
-npx copilot-metrics@0.1.0 paths --json
+npx copilot-metrics@0.1.2 init
+npx copilot-metrics@0.1.2 paths --json
 ```
 
 ## Configure Telemetry
 
+For Copilot CLI, `init` plus hooks are enough for local token reporting. Reports import Copilot's local session-state `events.jsonl` files and extract shutdown usage totals without requiring telemetry environment variables.
+
 Print VS Code Insiders Copilot Chat OpenTelemetry settings:
 
 ```bash
-npx copilot-metrics@0.1.0 setup vscode
+npx copilot-metrics@0.1.2 setup vscode
 ```
 
 Print Copilot CLI OpenTelemetry environment exports:
 
 ```bash
-npx copilot-metrics@0.1.0 setup copilot-cli
+npx copilot-metrics@0.1.2 setup copilot-cli
 ```
 
+This is optional. Use it only when you also want Copilot CLI OTel JSONL output.
+
 Content capture is disabled by default. Do not enable richer prompt capture unless you explicitly accept the privacy tradeoff.
 
 ## Configure Hooks
@@ -63,50 +67,53 @@ Content capture is disabled by default. Do not enable richer prompt capture unle
 Preview repo-local hook config. The default `--surface both` emits the Copilot CLI lower camel case hook format:
 
 ```bash
-npx copilot-metrics@0.1.0 hooks preview --scope local --surface both
+npx copilot-metrics@0.1.2 hooks preview --scope local --surface both
 ```
 
 Install repo-local or user-global hook config:
 
 ```bash
-npx copilot-metrics@0.1.0 hooks install --scope local --surface both
-npx copilot-metrics@0.1.0 hooks install --scope global --surface both
+npx copilot-metrics@0.1.2 hooks install --scope local --surface both
+npx copilot-metrics@0.1.2 hooks install --scope global --surface both
 ```
 
 Local install writes `.github/hooks/copilot-metrics.json`. Global install updates `~/.copilot/settings.json` idempotently, replacing prior `copilot-metrics` hook entries while preserving other settings and hooks. Use `--surface vscode` for VS Code-only PascalCase events or `--surface copilot-cli` for CLI-native lower camel case events. The hook logger writes redacted JSONL metadata to the central data directory. It extracts Jira-style labels such as `DEMO-12345` from safe metadata and does not store full prompt text by default.
 
 ## Import Telemetry
 
-Initialize the local SQLite store and import JSONL files:
+Initialize the local SQLite store and import JSONL files manually:
 
 ```bash
-npx copilot-metrics@0.1.0 store init
-npx copilot-metrics@0.1.0 import --source vscode --file ~/.local/share/copilot-metrics/telemetry/vscode-copilot-otel.jsonl
-npx copilot-metrics@0.1.0 import --source copilot-cli --file ~/.local/share/copilot-metrics/telemetry/copilot-cli-otel.jsonl
-npx copilot-metrics@0.1.0 import --source hooks --file ~/.local/share/copilot-metrics/hooks/copilot-hooks.jsonl
+npx copilot-metrics@0.1.2 store init
+npx copilot-metrics@0.1.2 import --source vscode --file ~/.local/share/copilot-metrics/telemetry/vscode-copilot-otel.jsonl
+npx copilot-metrics@0.1.2 import --source copilot-cli --file ~/.local/share/copilot-metrics/telemetry/copilot-cli-otel.jsonl
+npx copilot-metrics@0.1.2 import --source copilot-session --file ~/.copilot/session-state/<session-id>/events.jsonl
+npx copilot-metrics@0.1.2 import --source hooks --file ~/.local/share/copilot-metrics/hooks/copilot-hooks.jsonl
 ```
 
-Imports persist raw records, normalized LLM usage records, hook events, label evidence, and import warnings.
+Imports persist raw records, normalized LLM usage records, hook events, label evidence, and import warnings. Re-importing the same JSONL rows is idempotent. For Copilot session-state files, only shutdown usage rows are persisted; prompt-bearing session events are used in memory for label extraction and context and are not stored as raw records.
 
 ## Reports
 
 Run local reports from the SQLite store:
 
 ```bash
-npx copilot-metrics@0.1.0 report labels
-npx copilot-metrics@0.1.0 report label DEMO-12345
-npx copilot-metrics@0.1.0 report label DEMO-12345 --detail
-npx copilot-metrics@0.1.0 report models
-npx copilot-metrics@0.1.0 report repos
-npx copilot-metrics@0.1.0 report unattributed
+npx copilot-metrics@0.1.2 report labels
+npx copilot-metrics@0.1.2 report label DEMO-12345
+npx copilot-metrics@0.1.2 report label DEMO-12345 --detail
+npx copilot-metrics@0.1.2 report models
+npx copilot-metrics@0.1.2 report repos
+npx copilot-metrics@0.1.2 report unattributed
 ```
 
 Every report supports `--json`:
 
 ```bash
-npx copilot-metrics@0.1.0 report labels --json
+npx copilot-metrics@0.1.2 report labels --json
 ```
 
+Report commands automatically import newly appended configured VS Code OTel, optional Copilot CLI OTel, Copilot CLI session-state, and hook JSONL files before querying. Label reports include input, output, cache read, cache creation, and reasoning token totals. Labels seen only in hooks remain visible as hook-only evidence with zero usage records, so attribution hints do not imply token-bearing usage.
+
 ## Attribution Model
 
 The default extractor finds Jira-style labels such as `DEMO-12345` from safe metadata including hook labels, branch names, cwd/path values, repo metadata, and task hints.
@@ -168,7 +175,7 @@ The manual prompt performs one harmless tool call so Copilot CLI hook execution
 ## Current Limits
 
 - Costs are estimates, not official billing records.
-- Official GitHub usage report reconciliation is not included in `0.1.0`.
-- Local OTLP collector mode is not included in `0.1.0`.
-- Richer prompt/content capture and redaction controls are not included in `0.1.0`.
+- Official GitHub usage report reconciliation is not included in `0.1.2`.
+- Local OTLP collector mode is not included in `0.1.2`.
+- Richer prompt/content capture and redaction controls are not included in `0.1.2`.
 - Dashboard views are deferred until the CLI/query model proves useful.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "copilot-metrics",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "description": "Local-first Copilot usage telemetry setup and reporting tools.",
   "type": "commonjs",
   "homepage": "https://github.com/nnexai/copilot-metrics#readme",

package/src/cli.js

@@ -12,7 +12,7 @@ const {
 } = require('./setup');
 const { appendHookEvent, readJsonFromStream } = require('./hook-logger');
 const { initStore } = require('./sqlite-store');
-const { ingestFile } = require('./ingest');
+const { autoImportConfiguredSources, ingestFile } = require('./ingest');
 const { MODEL_PRICES, PRICING_VERSION } = require('./pricing');
 const {
   labelOverview,
@@ -34,6 +34,10 @@ function parseFlags(args) {
   const rest = [];
   for (let i = 0; i < args.length; i += 1) {
     const arg = args[i];
+    if (arg === '--') {
+      rest.push(...args.slice(i + 1));
+      break;
+    }
     if (!arg.startsWith('--')) {
       rest.push(arg);
       continue;
@@ -72,7 +76,7 @@ Usage:
   copilot-metrics hooks install [--scope local|global] [--surface both|vscode|copilot-cli] [--json]
   copilot-metrics hook-log --event <name>
   copilot-metrics store init [--json]
-  copilot-metrics import --source vscode|copilot-cli|hooks --file <path> [--json]
+  copilot-metrics import --source vscode|copilot-cli|copilot-session|hooks --file <path> [--json]
   copilot-metrics report labels [--json]
   copilot-metrics report label <id> [--detail] [--json]
   copilot-metrics report models [--json]
@@ -112,10 +116,55 @@ function formatCopilotCli(env) {
     'Export these variables before running Copilot CLI:',
     shellExports(env),
     '',
+    'This is optional. Copilot CLI session-state usage is imported by reports without these exports.',
+    '',
     'Content capture is disabled by default.',
   ].join('\n');
 }
 
+function telemetryDiagnostics(importResults) {
+  const hookResult = importResults.find((result) => result.source === 'hooks' && result.raw_records > 0);
+  if (!hookResult) return [];
+  const sessionUsage = importResults.find((result) => result.source === 'copilot-session' && result.raw_records > 0);
+  if (sessionUsage) return [];
+
+  const cliTelemetry = importResults.find((result) => result.source === 'copilot-cli');
+  if (!cliTelemetry) return [];
+
+  if (cliTelemetry.skipped && cliTelemetry.reason === 'missing_file') {
+    return [{
+      code: 'missing_copilot_cli_otel',
+      message: `Hook evidence was found, but no token-bearing Copilot session-state or OTel usage was imported. Check that ${cliTelemetry.file} exists for optional OTel data, or that Copilot session-state files are available under the configured COPILOT_HOME.`,
+    }];
+  }
+
+  if (cliTelemetry.raw_records === 0) {
+    return [{
+      code: 'empty_copilot_cli_otel',
+      message: `Hook evidence was found, but Copilot CLI token telemetry is empty at ${cliTelemetry.file}. Run another Copilot session with OTel enabled.`,
+    }];
+  }
+
+  if (cliTelemetry.usage_records === 0) {
+    return [{
+      code: 'no_copilot_cli_usage_records',
+      message: `Copilot CLI telemetry was found at ${cliTelemetry.file}, but no token-bearing LLM spans were normalized from it.`,
+    }];
+  }
+
+  return [];
+}
+
+function appendDiagnostics(output, diagnostics) {
+  if (diagnostics.length === 0) return output;
+  return [
+    output,
+    '',
+    'Diagnostics:',
+    ...diagnostics.map((diagnostic) => `- ${diagnostic.message}`),
+  ].join('\n');
+}
+
 async function main(args, io) {
   const { flags, rest } = parseFlags(args);
   const json = flags.json === true;
@@ -213,8 +262,8 @@ async function main(args, io) {
         : source === 'hooks'
           ? paths.hookEventsJsonl
           : null);
-    if (!['vscode', 'copilot-cli', 'hooks'].includes(source)) {
-      throw new Error('import requires --source vscode|copilot-cli|hooks');
+    if (!['vscode', 'copilot-cli', 'copilot-session', 'hooks'].includes(source)) {
+      throw new Error('import requires --source vscode|copilot-cli|copilot-session|hooks');
     }
     if (!file) throw new Error('import requires --file <path>');
     ensureDataDirs(paths);
@@ -236,9 +285,16 @@ async function main(args, io) {
   }
 
   if (command === 'report') {
+    ensureDataDirs(paths);
+    const imports = await autoImportConfiguredSources(paths, {
+      cwd: io.cwd,
+      extractors: loadConfiguredExtractors(paths.configJson, io.cwd),
+    });
+    const diagnostics = telemetryDiagnostics(imports);
+
     if (subcommand === 'labels') {
       const rows = await labelOverview(paths.usageDb);
-      writeOutput(io.stdout, json ? { labels: rows } : formatLabels(rows), json);
+      writeOutput(io.stdout, json ? { labels: rows, diagnostics } : appendDiagnostics(formatLabels(rows), diagnostics), json);
       return;
     }
     if (subcommand === 'label') {
@@ -247,10 +303,10 @@ async function main(args, io) {
       const summary = await labelSummary(paths.usageDb, label);
       if (flags.detail === true) {
         const details = await labelDetails(paths.usageDb, label);
-        writeOutput(io.stdout, json ? { label: summary, details } : formatLabelSummary(summary, details), json);
+        writeOutput(io.stdout, json ? { label: summary, details, diagnostics } : appendDiagnostics(formatLabelSummary(summary, details), diagnostics), json);
         return;
       }
-      writeOutput(io.stdout, json ? { label: summary } : formatLabelSummary(summary), json);
+      writeOutput(io.stdout, json ? { label: summary, diagnostics } : appendDiagnostics(formatLabelSummary(summary), diagnostics), json);
       return;
     }
     if (subcommand === 'models') {
@@ -285,4 +341,5 @@ module.exports = {
   main,
   parseFlags,
   helpText,
+  telemetryDiagnostics,
 };

package/src/ingest.js

@@ -1,10 +1,14 @@
 'use strict';
 
+const crypto = require('node:crypto');
+const fs = require('node:fs');
+const path = require('node:path');
 const { readJsonl } = require('./jsonl');
-const { normalizePayload, normalizeHookEvent } = require('./otel');
+const { normalizePayload, normalizeHookEvent, normalizeCopilotSessionEvents } = require('./otel');
 const { estimateCost, PRICING_VERSION } = require('./pricing');
-const { insertImport } = require('./sqlite-store');
+const { existingRawFingerprints, insertImport } = require('./sqlite-store');
 const { attachUsageLabelEvidence, attachHookLabelEvidence } = require('./labels');
+const { loadConfiguredExtractors } = require('./label-extractors');
 
 function enrichCosts(records) {
   return records.map((record) => {
@@ -21,20 +25,56 @@ function enrichCosts(records) {
   });
 }
 
+function rawFingerprint(source, file, record) {
+  return crypto
+    .createHash('sha256')
+    .update(source)
+    .update('\0')
+    .update(path.resolve(file))
+    .update('\0')
+    .update(String(record.line))
+    .update('\0')
+    .update(JSON.stringify(record.value))
+    .digest('hex');
+}
+
+function isCopilotSessionUsageRecord(record) {
+  return record.value && record.value.type === 'session.shutdown';
+}
+
 async function ingestFile(options) {
   const { dbPath, file, source } = options;
   const parsed = readJsonl(file);
   const warnings = [...parsed.warnings];
+  const sourceFile = path.resolve(file);
+  const parsedRecords = parsed.records.map((record) => ({
+    ...record,
+    raw_fingerprint: rawFingerprint(source, sourceFile, record),
+  }));
+  const importableRecords = source === 'copilot-session'
+    ? parsedRecords.filter(isCopilotSessionUsageRecord)
+    : parsedRecords;
+  const existing = await existingRawFingerprints(
+    dbPath,
+    source,
+    sourceFile,
+    importableRecords.map((record) => record.raw_fingerprint),
+  );
+  const newRecords = importableRecords.filter((record) => !existing.has(record.raw_fingerprint));
   const usageRecords = [];
   const hookEvents = [];
 
-  for (const record of parsed.records) {
-    if (source === 'hooks') {
-      const event = normalizeHookEvent(record.value, source, record.line);
-      if (event) hookEvents.push(event);
-      continue;
+  if (source === 'copilot-session') {
+    usageRecords.push(...normalizeCopilotSessionEvents(newRecords, parsedRecords));
+  } else {
+    for (const record of newRecords) {
+      if (source === 'hooks') {
+        const event = normalizeHookEvent(record.value, source, record.line);
+        if (event) hookEvents.push(event);
+        continue;
+      }
+      usageRecords.push(...normalizePayload(record.value, source, record.line));
     }
-    usageRecords.push(...normalizePayload(record.value, source, record.line));
   }
 
   const extractorOptions = { extractors: options.extractors || [] };
@@ -50,13 +90,15 @@ async function ingestFile(options) {
     }
   }
 
-  await insertImport(dbPath, source, parsed.records, enrichedUsage, enrichedHooks, warnings);
+  await insertImport(dbPath, source, sourceFile, newRecords, enrichedUsage, enrichedHooks, warnings);
 
   return {
     source,
     file,
     dbPath,
-    raw_records: parsed.records.length,
+    raw_records: importableRecords.length,
+    new_raw_records: newRecords.length,
+    skipped_existing_records: importableRecords.length - newRecords.length,
     usage_records: enrichedUsage.length,
     hook_events: enrichedHooks.length,
     label_evidence: enrichedUsage.reduce((sum, usage) => sum + (usage.label_evidence || []).length, 0)
@@ -66,6 +108,64 @@ async function ingestFile(options) {
   };
 }
 
+function configuredSourceFiles(paths, config = {}) {
+  const sourceConfig = config.sources || {};
+  const telemetryConfig = config.telemetry || {};
+  const files = [
+    { source: 'vscode', file: sourceConfig.vscode?.telemetry || telemetryConfig.vscode || paths.vscodeOtelJsonl },
+    { source: 'hooks', file: sourceConfig.vscode?.hooks || paths.hookEventsJsonl },
+    { source: 'copilot-cli', file: sourceConfig.copilotCli?.telemetry || telemetryConfig.copilotCli || paths.copilotCliOtelJsonl },
+    { source: 'hooks', file: sourceConfig.copilotCli?.hooks || paths.hookEventsJsonl },
+    ...discoverCopilotSessionFiles(sourceConfig.copilotCli?.sessions || paths.copilotSessionStateDir),
+  ];
+  const seen = new Set();
+  return files
+    .filter((entry) => entry.file)
+    .map((entry) => ({ source: entry.source, file: path.resolve(entry.file) }))
+    .filter((entry) => {
+      const key = `${entry.source}\0${entry.file}`;
+      if (seen.has(key)) return false;
+      seen.add(key);
+      return true;
+    });
+}
+
+function discoverCopilotSessionFiles(sessionStateDir) {
+  if (!sessionStateDir || !fs.existsSync(sessionStateDir)) return [];
+  return fs.readdirSync(sessionStateDir, { withFileTypes: true })
+    .filter((entry) => entry.isDirectory())
+    .map((entry) => path.join(sessionStateDir, entry.name, 'events.jsonl'))
+    .filter((file) => fs.existsSync(file))
+    .map((file) => ({ source: 'copilot-session', file }));
+}
+
+function readConfig(configJson) {
+  if (!fs.existsSync(configJson)) return {};
+  return JSON.parse(fs.readFileSync(configJson, 'utf8'));
+}
+
+async function autoImportConfiguredSources(paths, options = {}) {
+  const config = readConfig(paths.configJson);
+  const extractors = options.extractors || loadConfiguredExtractors(paths.configJson, options.cwd || process.cwd());
+  const results = [];
+  for (const entry of configuredSourceFiles(paths, config)) {
+    if (!fs.existsSync(entry.file)) {
+      results.push({ ...entry, skipped: true, reason: 'missing_file' });
+      continue;
+    }
+    results.push(await ingestFile({
+      dbPath: paths.usageDb,
+      file: entry.file,
+      source: entry.source,
+      extractors,
+    }));
+  }
+  return results;
+}
+
 module.exports = {
+  autoImportConfiguredSources,
+  configuredSourceFiles,
+  discoverCopilotSessionFiles,
   ingestFile,
 };

package/src/otel.js

@@ -29,6 +29,23 @@ function number(attrs, keys) {
   return Number.isFinite(parsed) ? parsed : 0;
 }
 
+const LABEL_RE = /\b[A-Z][A-Z0-9]+-\d+\b/g;
+
+function collectLabels(value, labels = new Set()) {
+  if (typeof value === 'string') {
+    for (const match of value.matchAll(LABEL_RE)) labels.add(match[0]);
+    return labels;
+  }
+  if (Array.isArray(value)) {
+    for (const item of value) collectLabels(item, labels);
+    return labels;
+  }
+  if (value && typeof value === 'object') {
+    for (const item of Object.values(value)) collectLabels(item, labels);
+  }
+  return labels;
+}
+
 function flattenSpans(payload) {
   if (!payload || typeof payload !== 'object') return [];
   if (payload.name || payload.attributes || payload.spanId) return [payload];
@@ -102,6 +119,76 @@ function normalizePayload(payload, source, rawLine) {
     .filter(Boolean);
 }
 
+function normalizeCopilotSessionEvents(newRecords, allRecords) {
+  const session = {
+    labels: new Set(),
+    id: null,
+    cwd: null,
+    repo: null,
+    branch: null,
+    commit: null,
+    conversationId: null,
+  };
+
+  for (const record of allRecords) {
+    const event = record.value;
+    if (!event || typeof event !== 'object') continue;
+    if (event.type === 'session.start') {
+      const data = event.data || {};
+      const context = data.context || {};
+      session.id = data.sessionId || session.id;
+      session.cwd = context.cwd || session.cwd;
+      session.repo = context.gitRoot || context.repository || session.repo;
+      session.branch = context.branch || session.branch;
+      session.commit = context.headCommit || session.commit;
+      collectLabels(context, session.labels);
+    }
+    if (event.type === 'hook.start') {
+      collectLabels(event.data && event.data.input, session.labels);
+    }
+    if (event.type === 'assistant.message') {
+      const data = event.data || {};
+      session.conversationId = data.interactionId || session.conversationId;
+      collectLabels(data.content, session.labels);
+    }
+  }
+
+  const records = [];
+  for (const record of newRecords) {
+    const event = record.value;
+    if (!event || event.type !== 'session.shutdown') continue;
+    const data = event.data || {};
+    const modelMetrics = data.modelMetrics || {};
+    for (const [model, metrics] of Object.entries(modelMetrics)) {
+      const usage = metrics.usage || {};
+      records.push({
+        raw_line: record.line,
+        span_id: event.id || null,
+        trace_id: session.id || null,
+        parent_span_id: event.parentId || null,
+        timestamp: event.timestamp || null,
+        surface: 'copilot-cli-session',
+        conversation_id: session.conversationId,
+        session_id: session.id,
+        requested_model: model,
+        resolved_model: model,
+        repo: session.repo,
+        branch: session.branch,
+        cwd: session.cwd,
+        commit_sha: session.commit,
+        labels: Array.from(session.labels).sort(),
+        input_tokens: Number(usage.inputTokens || 0),
+        output_tokens: Number(usage.outputTokens || 0),
+        cache_read_tokens: Number(usage.cacheReadTokens || 0),
+        cache_creation_tokens: Number(usage.cacheWriteTokens || 0),
+        reasoning_tokens: Number(usage.reasoningTokens || 0),
+        warnings: [],
+      });
+    }
+  }
+  return records;
+}
+
 function normalizeHookEvent(payload, source, rawLine) {
   if (!payload || typeof payload !== 'object' || Array.isArray(payload)) return null;
   return {
@@ -122,5 +209,6 @@ module.exports = {
   flattenSpans,
   classifySpan,
   normalizePayload,
+  normalizeCopilotSessionEvents,
   normalizeHookEvent,
 };

package/src/paths.js

@@ -28,6 +28,7 @@ function resolvePaths(options = {}) {
   const storeDir = path.join(home, 'store');
   const skillsDir = path.join(home, 'skills');
   const copilotHome = env.COPILOT_HOME || path.join(os.homedir(), '.copilot');
+  const copilotSessionStateDir = path.join(copilotHome, 'session-state');
 
   return {
     home,
@@ -40,6 +41,8 @@ function resolvePaths(options = {}) {
     hookEventsJsonl: path.join(hooksDir, 'copilot-hooks.jsonl'),
     usageDb: path.join(storeDir, 'copilot-metrics.sqlite'),
     configJson: path.join(home, 'config.json'),
+    copilotHome,
+    copilotSessionStateDir,
     localHookConfig: path.join(cwd, '.github', 'hooks', 'copilot-metrics.json'),
     globalHookConfig: path.join(copilotHome, 'settings.json'),
   };

package/src/reports.js

@@ -1,6 +1,6 @@
 'use strict';
 
-const { queryRows } = require('./sqlite-store');
+const { initStore, queryRows } = require('./sqlite-store');
 const { canonicalLabel } = require('./label-extractors');
 
 function n(value) {
@@ -31,6 +31,7 @@ function table(headers, rows) {
 }
 
 async function labelOverview(dbPath) {
+  await initStore(dbPath);
   return queryRows(dbPath, `
 SELECT
   labels.label,
@@ -43,6 +44,10 @@ SELECT
   COALESCE((SELECT SUM(cache_creation_tokens) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS cache_creation_tokens,
   COALESCE((SELECT SUM(reasoning_tokens) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS reasoning_tokens,
   COALESCE((SELECT SUM(COALESCE(estimated_ai_credits, 0)) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS estimated_ai_credits,
+  CASE
+    WHEN (SELECT COUNT(DISTINCT usage_record_id) FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL) = 0 THEN 'hook-only'
+    ELSE 'token-bearing'
+  END AS token_status,
   (SELECT MIN(COALESCE(ur.timestamp, le.timestamp, le.imported_at)) FROM label_evidence le LEFT JOIN usage_records ur ON ur.id = le.usage_record_id WHERE le.label = labels.label) AS first_seen,
   (SELECT MAX(COALESCE(ur.timestamp, le.timestamp, le.imported_at)) FROM label_evidence le LEFT JOIN usage_records ur ON ur.id = le.usage_record_id WHERE le.label = labels.label) AS last_seen,
   (SELECT MAX(estimate_label) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)) AS estimate_label
@@ -51,6 +56,7 @@ ORDER BY estimated_ai_credits DESC, labels.label`);
 }
 
 async function labelSummary(dbPath, label) {
+  await initStore(dbPath);
   const rows = await queryRows(dbPath, `
 SELECT
   labels.label,
@@ -63,6 +69,10 @@ SELECT
   COALESCE((SELECT SUM(cache_creation_tokens) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS cache_creation_tokens,
   COALESCE((SELECT SUM(reasoning_tokens) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS reasoning_tokens,
   COALESCE((SELECT SUM(COALESCE(estimated_ai_credits, 0)) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS estimated_ai_credits,
+  CASE
+    WHEN (SELECT COUNT(DISTINCT usage_record_id) FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL) = 0 THEN 'hook-only'
+    ELSE 'token-bearing'
+  END AS token_status,
   (SELECT MIN(COALESCE(ur.timestamp, le.timestamp, le.imported_at)) FROM label_evidence le LEFT JOIN usage_records ur ON ur.id = le.usage_record_id WHERE le.label = labels.label) AS first_seen,
   (SELECT MAX(COALESCE(ur.timestamp, le.timestamp, le.imported_at)) FROM label_evidence le LEFT JOIN usage_records ur ON ur.id = le.usage_record_id WHERE le.label = labels.label) AS last_seen,
   (SELECT MAX(estimate_label) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)) AS estimate_label
@@ -71,6 +81,7 @@ FROM (SELECT DISTINCT label FROM label_evidence WHERE label = ?) labels`, [canon
 }
 
 async function labelDetails(dbPath, label) {
+  await initStore(dbPath);
   return queryRows(dbPath, `
 SELECT
   le.label,
@@ -87,6 +98,9 @@ SELECT
   ur.resolved_model,
   ur.input_tokens,
   ur.output_tokens,
+  ur.cache_read_tokens,
+  ur.cache_creation_tokens,
+  ur.reasoning_tokens,
   ur.estimated_ai_credits,
   ur.estimate_label,
   COALESCE(ur.timestamp, le.timestamp, le.imported_at) AS timestamp
@@ -97,6 +111,7 @@ ORDER BY timestamp, le.source_type, le.source_field`, [canonicalLabel(label)]);
 }
 
 async function modelReport(dbPath) {
+  await initStore(dbPath);
   return queryRows(dbPath, `
 SELECT
   COALESCE(resolved_model, requested_model, 'unknown') AS model,
@@ -114,6 +129,7 @@ ORDER BY estimated_ai_credits DESC, model`);
 }
 
 async function repoReport(dbPath) {
+  await initStore(dbPath);
   return queryRows(dbPath, `
 SELECT
   COALESCE(repo, 'unknown') AS repo,
@@ -130,6 +146,7 @@ ORDER BY estimated_ai_credits DESC, repo, cwd`);
 }
 
 async function unattributedReport(dbPath) {
+  await initStore(dbPath);
   return queryRows(dbPath, `
 SELECT
   ur.id,
@@ -156,13 +173,18 @@ ORDER BY ur.timestamp, ur.id`);
 function formatLabels(rows) {
   return [
     table(
-      ['Label', 'Sessions', 'Input', 'Output', 'Credits', 'Evidence', 'Last seen'],
+      ['Label', 'Sessions', 'Usage', 'Input', 'Output', 'Cache read', 'Cache create', 'Reasoning', 'Credits', 'Status', 'Evidence', 'Last seen'],
       rows.map((row) => [
         row.label,
         row.sessions,
+        row.usage_records,
         formatNumber(row.input_tokens),
         formatNumber(row.output_tokens),
+        formatNumber(row.cache_read_tokens),
+        formatNumber(row.cache_creation_tokens),
+        formatNumber(row.reasoning_tokens),
         formatCredits(row.estimated_ai_credits),
+        row.token_status,
         row.evidence_count,
         row.last_seen || '',
       ]),
@@ -176,18 +198,35 @@ function formatLabelSummary(summary, details = null) {
   if (!summary) return 'No usage found for label.';
   const lines = [
     table(
-      ['Label', 'Sessions', 'Input', 'Output', 'Credits', 'Evidence'],
-      [[summary.label, summary.sessions, formatNumber(summary.input_tokens), formatNumber(summary.output_tokens), formatCredits(summary.estimated_ai_credits), summary.evidence_count]],
+      ['Label', 'Sessions', 'Usage', 'Input', 'Output', 'Cache read', 'Cache create', 'Reasoning', 'Credits', 'Status', 'Evidence'],
+      [[
+        summary.label,
+        summary.sessions,
+        summary.usage_records,
+        formatNumber(summary.input_tokens),
+        formatNumber(summary.output_tokens),
+        formatNumber(summary.cache_read_tokens),
+        formatNumber(summary.cache_creation_tokens),
+        formatNumber(summary.reasoning_tokens),
+        formatCredits(summary.estimated_ai_credits),
+        summary.token_status,
+        summary.evidence_count,
+      ]],
     ),
   ];
   if (details) {
     lines.push('', table(
-      ['Source', 'Field', 'Session', 'Model', 'Credits', 'Value'],
+      ['Source', 'Field', 'Session', 'Model', 'Input', 'Output', 'Cache read', 'Cache create', 'Reasoning', 'Credits', 'Value'],
       details.map((row) => [
         row.source_type,
         row.source_field,
         row.session_id || '',
         row.resolved_model || '',
+        formatNumber(row.input_tokens),
+        formatNumber(row.output_tokens),
+        formatNumber(row.cache_read_tokens),
+        formatNumber(row.cache_creation_tokens),
+        formatNumber(row.reasoning_tokens),
         formatCredits(row.estimated_ai_credits),
         row.source_value || '',
       ]),

package/src/setup.js

@@ -45,6 +45,7 @@ function ensureDataDirs(paths) {
   if (!fs.existsSync(paths.configJson)) {
     writePrivateFile(paths.configJson, `${JSON.stringify({
       version: 1,
+      dataHome: paths.home,
       contentCapture: false,
       telemetry: {
         vscode: paths.vscodeOtelJsonl,
@@ -58,6 +59,7 @@ function ensureDataDirs(paths) {
         copilotCli: {
           telemetry: paths.copilotCliOtelJsonl,
           hooks: paths.hookEventsJsonl,
+          sessions: paths.copilotSessionStateDir,
         },
       },
       labelExtractors: [],
@@ -93,6 +95,11 @@ function packageBinCommand(cwd) {
   return path.join(cwd, 'bin', 'copilot-metrics.js');
 }
 
+function commandInvocation(command) {
+  const quoted = shellQuote(command);
+  return command.endsWith('.js') ? `node ${quoted}` : quoted;
+}
+
 function hookEventsForSurface(surface) {
   if (surface === 'copilot-cli' || surface === 'both') return COPILOT_CLI_HOOK_EVENTS;
   if (surface === 'vscode') return VSCODE_HOOK_EVENTS;
@@ -100,7 +107,7 @@ function hookEventsForSurface(surface) {
 }
 
 function hookCommand(command, event, metricsHome) {
-  return `COPILOT_METRICS_HOME=${shellQuote(metricsHome)} node ${shellQuote(command)} hook-log --event ${shellQuote(event)} --quiet`;
+  return `COPILOT_METRICS_HOME=${shellQuote(metricsHome)} ${commandInvocation(command)} hook-log --event ${shellQuote(event)} --quiet`;
 }
 
 function hookConfig(paths, options = {}) {
@@ -171,6 +178,7 @@ function installHook(paths, options = {}) {
 
 function setupSnapshot(options = {}) {
   const paths = resolvePaths(options);
+  ensureDataDirs(paths);
   return {
     paths,
     vscode: vscodeSettings(paths),

package/src/sqlite-store.js

@@ -29,6 +29,19 @@ function persistDatabase(dbPath, db) {
   }
 }
 
+function hasColumn(db, table, column) {
+  const result = db.exec(`PRAGMA table_info(${table})`);
+  if (!result.length) return false;
+  const nameIndex = result[0].columns.indexOf('name');
+  return result[0].values.some((row) => row[nameIndex] === column);
+}
+
+function addColumnIfMissing(db, table, column, definition) {
+  if (!hasColumn(db, table, column)) {
+    db.run(`ALTER TABLE ${table} ADD COLUMN ${column} ${definition}`);
+  }
+}
+
 async function initStore(dbPath) {
   const db = await openDatabase(dbPath);
   db.run(`
@@ -36,7 +49,9 @@ CREATE TABLE IF NOT EXISTS raw_records (
   id INTEGER PRIMARY KEY AUTOINCREMENT,
   imported_at TEXT NOT NULL,
   source TEXT NOT NULL,
+  source_file TEXT,
   line INTEGER NOT NULL,
+  raw_fingerprint TEXT,
   payload_json TEXT NOT NULL
 );
 CREATE TABLE IF NOT EXISTS usage_records (
@@ -105,6 +120,9 @@ CREATE TABLE IF NOT EXISTS import_warnings (
   message TEXT NOT NULL
 );
 `);
+  addColumnIfMissing(db, 'raw_records', 'source_file', 'TEXT');
+  addColumnIfMissing(db, 'raw_records', 'raw_fingerprint', 'TEXT');
+  db.run('CREATE UNIQUE INDEX IF NOT EXISTS idx_raw_records_fingerprint ON raw_records (source, source_file, raw_fingerprint)');
   persistDatabase(dbPath, db);
 }
 
@@ -147,7 +165,25 @@ function insertLabelEvidence(db, importedAt, evidenceRows) {
   );
 }
 
-async function insertImport(dbPath, source, rawRecords, usageRecords, hookEvents, warnings) {
+async function existingRawFingerprints(dbPath, source, sourceFile, fingerprints) {
+  await initStore(dbPath);
+  if (!fingerprints.length) return new Set();
+  const db = await openDatabase(dbPath);
+  const existing = new Set();
+  const statement = db.prepare('SELECT 1 FROM raw_records WHERE source = ? AND source_file = ? AND raw_fingerprint = ? LIMIT 1');
+  try {
+    for (const fingerprint of fingerprints) {
+      statement.bind([source, sourceFile, fingerprint]);
+      if (statement.step()) existing.add(fingerprint);
+      statement.reset();
+    }
+  } finally {
+    statement.free();
+  }
+  return existing;
+}
+
+async function insertImport(dbPath, source, sourceFile, rawRecords, usageRecords, hookEvents, warnings) {
   await initStore(dbPath);
   const db = await openDatabase(dbPath);
   const importedAt = new Date().toISOString();
@@ -156,8 +192,8 @@ async function insertImport(dbPath, source, rawRecords, usageRecords, hookEvents
   try {
     runPrepared(
       db,
-      'INSERT INTO raw_records (imported_at, source, line, payload_json) VALUES (?, ?, ?, ?)',
-      rawRecords.map((record) => [importedAt, source, record.line, JSON.stringify(record.value)]),
+      'INSERT OR IGNORE INTO raw_records (imported_at, source, source_file, line, raw_fingerprint, payload_json) VALUES (?, ?, ?, ?, ?, ?)',
+      rawRecords.map((record) => [importedAt, source, sourceFile, record.line, record.raw_fingerprint || null, JSON.stringify(record.value)]),
     );
 
     const labelEvidence = [];
@@ -283,6 +319,7 @@ async function queryRows(dbPath, sql, params = []) {
 }
 
 module.exports = {
+  existingRawFingerprints,
   initStore,
   insertImport,
   queryOne,