copilot-metrics 0.1.0 → 0.1.1

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.1.1 - 2026-05-30
+
+### Added
+
+- Setup-once report flow: report commands automatically import configured VS Code, Copilot CLI, and hook JSONL sources before querying.
+- Idempotent import fingerprints so repeated reports or imports do not double-count previously ingested JSONL rows.
+- Complete label token reporting for input, output, cache read, cache creation, and reasoning tokens in human and JSON output.
+- Hook-only label status so attribution evidence from hooks remains visible without implying token-bearing usage.
+
+### Changed
+
+- `setup all` now persists the central data directory config, matching `init` for setup-once usage.
+- Hook commands support installed executable shims as well as checkout-local JavaScript entrypoints.
+
 ## 0.1.0 - 2026-05-30
 
 First local release candidate for `copilot-metrics`.

package/README.md

@@ -9,8 +9,8 @@ Costs are estimates, not official billing records. GitHub billing remains the so
 From npm:
 
 ```bash
-npx copilot-metrics@0.1.0 --help
-npx copilot-metrics@0.1.0 init
+npx copilot-metrics@0.1.1 --help
+npx copilot-metrics@0.1.1 init
 ```
 
 From this checkout:
@@ -38,8 +38,8 @@ export COPILOT_METRICS_HOME=/path/to/copilot-metrics-data
 Useful commands:
 
 ```bash
-npx copilot-metrics@0.1.0 init
-npx copilot-metrics@0.1.0 paths --json
+npx copilot-metrics@0.1.1 init
+npx copilot-metrics@0.1.1 paths --json
 ```
 
 ## Configure Telemetry
@@ -47,13 +47,13 @@ npx copilot-metrics@0.1.0 paths --json
 Print VS Code Insiders Copilot Chat OpenTelemetry settings:
 
 ```bash
-npx copilot-metrics@0.1.0 setup vscode
+npx copilot-metrics@0.1.1 setup vscode
 ```
 
 Print Copilot CLI OpenTelemetry environment exports:
 
 ```bash
-npx copilot-metrics@0.1.0 setup copilot-cli
+npx copilot-metrics@0.1.1 setup copilot-cli
 ```
 
 Content capture is disabled by default. Do not enable richer prompt capture unless you explicitly accept the privacy tradeoff.
@@ -63,14 +63,14 @@ Content capture is disabled by default. Do not enable richer prompt capture unle
 Preview repo-local hook config. The default `--surface both` emits the Copilot CLI lower camel case hook format:
 
 ```bash
-npx copilot-metrics@0.1.0 hooks preview --scope local --surface both
+npx copilot-metrics@0.1.1 hooks preview --scope local --surface both
 ```
 
 Install repo-local or user-global hook config:
 
 ```bash
-npx copilot-metrics@0.1.0 hooks install --scope local --surface both
-npx copilot-metrics@0.1.0 hooks install --scope global --surface both
+npx copilot-metrics@0.1.1 hooks install --scope local --surface both
+npx copilot-metrics@0.1.1 hooks install --scope global --surface both
 ```
 
 Local install writes `.github/hooks/copilot-metrics.json`. Global install updates `~/.copilot/settings.json` idempotently, replacing prior `copilot-metrics` hook entries while preserving other settings and hooks. Use `--surface vscode` for VS Code-only PascalCase events or `--surface copilot-cli` for CLI-native lower camel case events. The hook logger writes redacted JSONL metadata to the central data directory. It extracts Jira-style labels such as `DEMO-12345` from safe metadata and does not store full prompt text by default.
@@ -80,33 +80,35 @@ Local install writes `.github/hooks/copilot-metrics.json`. Global install update
 Initialize the local SQLite store and import JSONL files:
 
 ```bash
-npx copilot-metrics@0.1.0 store init
-npx copilot-metrics@0.1.0 import --source vscode --file ~/.local/share/copilot-metrics/telemetry/vscode-copilot-otel.jsonl
-npx copilot-metrics@0.1.0 import --source copilot-cli --file ~/.local/share/copilot-metrics/telemetry/copilot-cli-otel.jsonl
-npx copilot-metrics@0.1.0 import --source hooks --file ~/.local/share/copilot-metrics/hooks/copilot-hooks.jsonl
+npx copilot-metrics@0.1.1 store init
+npx copilot-metrics@0.1.1 import --source vscode --file ~/.local/share/copilot-metrics/telemetry/vscode-copilot-otel.jsonl
+npx copilot-metrics@0.1.1 import --source copilot-cli --file ~/.local/share/copilot-metrics/telemetry/copilot-cli-otel.jsonl
+npx copilot-metrics@0.1.1 import --source hooks --file ~/.local/share/copilot-metrics/hooks/copilot-hooks.jsonl
 ```
 
-Imports persist raw records, normalized LLM usage records, hook events, label evidence, and import warnings.
+Imports persist raw records, normalized LLM usage records, hook events, label evidence, and import warnings. Re-importing the same JSONL rows is idempotent.
 
 ## Reports
 
 Run local reports from the SQLite store:
 
 ```bash
-npx copilot-metrics@0.1.0 report labels
-npx copilot-metrics@0.1.0 report label DEMO-12345
-npx copilot-metrics@0.1.0 report label DEMO-12345 --detail
-npx copilot-metrics@0.1.0 report models
-npx copilot-metrics@0.1.0 report repos
-npx copilot-metrics@0.1.0 report unattributed
+npx copilot-metrics@0.1.1 report labels
+npx copilot-metrics@0.1.1 report label DEMO-12345
+npx copilot-metrics@0.1.1 report label DEMO-12345 --detail
+npx copilot-metrics@0.1.1 report models
+npx copilot-metrics@0.1.1 report repos
+npx copilot-metrics@0.1.1 report unattributed
 ```
 
 Every report supports `--json`:
 
 ```bash
-npx copilot-metrics@0.1.0 report labels --json
+npx copilot-metrics@0.1.1 report labels --json
 ```
 
+Report commands automatically import newly appended configured VS Code, Copilot CLI, and hook JSONL files before querying. Label reports include input, output, cache read, cache creation, and reasoning token totals. Labels seen only in hooks remain visible as hook-only evidence with zero usage records, so attribution hints do not imply token-bearing usage.
+
 ## Attribution Model
 
 The default extractor finds Jira-style labels such as `DEMO-12345` from safe metadata including hook labels, branch names, cwd/path values, repo metadata, and task hints.
@@ -168,7 +170,7 @@ The manual prompt performs one harmless tool call so Copilot CLI hook execution
 ## Current Limits
 
 - Costs are estimates, not official billing records.
-- Official GitHub usage report reconciliation is not included in `0.1.0`.
-- Local OTLP collector mode is not included in `0.1.0`.
-- Richer prompt/content capture and redaction controls are not included in `0.1.0`.
+- Official GitHub usage report reconciliation is not included in `0.1.1`.
+- Local OTLP collector mode is not included in `0.1.1`.
+- Richer prompt/content capture and redaction controls are not included in `0.1.1`.
 - Dashboard views are deferred until the CLI/query model proves useful.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "copilot-metrics",
-  "version": "0.1.0",
+  "version": "0.1.1",
   "description": "Local-first Copilot usage telemetry setup and reporting tools.",
   "type": "commonjs",
   "homepage": "https://github.com/nnexai/copilot-metrics#readme",

package/src/cli.js

@@ -12,7 +12,7 @@ const {
 } = require('./setup');
 const { appendHookEvent, readJsonFromStream } = require('./hook-logger');
 const { initStore } = require('./sqlite-store');
-const { ingestFile } = require('./ingest');
+const { autoImportConfiguredSources, ingestFile } = require('./ingest');
 const { MODEL_PRICES, PRICING_VERSION } = require('./pricing');
 const {
   labelOverview,
@@ -236,6 +236,12 @@ async function main(args, io) {
   }
 
   if (command === 'report') {
+    ensureDataDirs(paths);
+    await autoImportConfiguredSources(paths, {
+      cwd: io.cwd,
+      extractors: loadConfiguredExtractors(paths.configJson, io.cwd),
+    });
+
     if (subcommand === 'labels') {
       const rows = await labelOverview(paths.usageDb);
       writeOutput(io.stdout, json ? { labels: rows } : formatLabels(rows), json);

package/src/ingest.js

@@ -1,10 +1,14 @@
 'use strict';
 
+const crypto = require('node:crypto');
+const fs = require('node:fs');
+const path = require('node:path');
 const { readJsonl } = require('./jsonl');
 const { normalizePayload, normalizeHookEvent } = require('./otel');
 const { estimateCost, PRICING_VERSION } = require('./pricing');
-const { insertImport } = require('./sqlite-store');
+const { existingRawFingerprints, insertImport } = require('./sqlite-store');
 const { attachUsageLabelEvidence, attachHookLabelEvidence } = require('./labels');
+const { loadConfiguredExtractors } = require('./label-extractors');
 
 function enrichCosts(records) {
   return records.map((record) => {
@@ -21,14 +25,39 @@ function enrichCosts(records) {
   });
 }
 
+function rawFingerprint(source, file, record) {
+  return crypto
+    .createHash('sha256')
+    .update(source)
+    .update('\0')
+    .update(path.resolve(file))
+    .update('\0')
+    .update(String(record.line))
+    .update('\0')
+    .update(JSON.stringify(record.value))
+    .digest('hex');
+}
+
 async function ingestFile(options) {
   const { dbPath, file, source } = options;
   const parsed = readJsonl(file);
   const warnings = [...parsed.warnings];
+  const sourceFile = path.resolve(file);
+  const parsedRecords = parsed.records.map((record) => ({
+    ...record,
+    raw_fingerprint: rawFingerprint(source, sourceFile, record),
+  }));
+  const existing = await existingRawFingerprints(
+    dbPath,
+    source,
+    sourceFile,
+    parsedRecords.map((record) => record.raw_fingerprint),
+  );
+  const newRecords = parsedRecords.filter((record) => !existing.has(record.raw_fingerprint));
   const usageRecords = [];
   const hookEvents = [];
 
-  for (const record of parsed.records) {
+  for (const record of newRecords) {
     if (source === 'hooks') {
       const event = normalizeHookEvent(record.value, source, record.line);
       if (event) hookEvents.push(event);
@@ -50,13 +79,15 @@ async function ingestFile(options) {
     }
   }
 
-  await insertImport(dbPath, source, parsed.records, enrichedUsage, enrichedHooks, warnings);
+  await insertImport(dbPath, source, sourceFile, newRecords, enrichedUsage, enrichedHooks, warnings);
 
   return {
     source,
     file,
     dbPath,
     raw_records: parsed.records.length,
+    new_raw_records: newRecords.length,
+    skipped_existing_records: parsed.records.length - newRecords.length,
     usage_records: enrichedUsage.length,
     hook_events: enrichedHooks.length,
     label_evidence: enrichedUsage.reduce((sum, usage) => sum + (usage.label_evidence || []).length, 0)
@@ -66,6 +97,53 @@ async function ingestFile(options) {
   };
 }
 
+function configuredSourceFiles(paths, config = {}) {
+  const sourceConfig = config.sources || {};
+  const telemetryConfig = config.telemetry || {};
+  const files = [
+    { source: 'vscode', file: sourceConfig.vscode?.telemetry || telemetryConfig.vscode || paths.vscodeOtelJsonl },
+    { source: 'hooks', file: sourceConfig.vscode?.hooks || paths.hookEventsJsonl },
+    { source: 'copilot-cli', file: sourceConfig.copilotCli?.telemetry || telemetryConfig.copilotCli || paths.copilotCliOtelJsonl },
+    { source: 'hooks', file: sourceConfig.copilotCli?.hooks || paths.hookEventsJsonl },
+  ];
+  const seen = new Set();
+  return files
+    .filter((entry) => entry.file)
+    .map((entry) => ({ source: entry.source, file: path.resolve(entry.file) }))
+    .filter((entry) => {
+      const key = `${entry.source}\0${entry.file}`;
+      if (seen.has(key)) return false;
+      seen.add(key);
+      return true;
+    });
+}
+
+function readConfig(configJson) {
+  if (!fs.existsSync(configJson)) return {};
+  return JSON.parse(fs.readFileSync(configJson, 'utf8'));
+}
+
+async function autoImportConfiguredSources(paths, options = {}) {
+  const config = readConfig(paths.configJson);
+  const extractors = options.extractors || loadConfiguredExtractors(paths.configJson, options.cwd || process.cwd());
+  const results = [];
+  for (const entry of configuredSourceFiles(paths, config)) {
+    if (!fs.existsSync(entry.file)) {
+      results.push({ ...entry, skipped: true, reason: 'missing_file' });
+      continue;
+    }
+    results.push(await ingestFile({
+      dbPath: paths.usageDb,
+      file: entry.file,
+      source: entry.source,
+      extractors,
+    }));
+  }
+  return results;
+}
+
 module.exports = {
+  autoImportConfiguredSources,
+  configuredSourceFiles,
   ingestFile,
 };

package/src/reports.js

@@ -1,6 +1,6 @@
 'use strict';
 
-const { queryRows } = require('./sqlite-store');
+const { initStore, queryRows } = require('./sqlite-store');
 const { canonicalLabel } = require('./label-extractors');
 
 function n(value) {
@@ -31,6 +31,7 @@ function table(headers, rows) {
 }
 
 async function labelOverview(dbPath) {
+  await initStore(dbPath);
   return queryRows(dbPath, `
 SELECT
   labels.label,
@@ -43,6 +44,10 @@ SELECT
   COALESCE((SELECT SUM(cache_creation_tokens) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS cache_creation_tokens,
   COALESCE((SELECT SUM(reasoning_tokens) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS reasoning_tokens,
   COALESCE((SELECT SUM(COALESCE(estimated_ai_credits, 0)) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS estimated_ai_credits,
+  CASE
+    WHEN (SELECT COUNT(DISTINCT usage_record_id) FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL) = 0 THEN 'hook-only'
+    ELSE 'token-bearing'
+  END AS token_status,
   (SELECT MIN(COALESCE(ur.timestamp, le.timestamp, le.imported_at)) FROM label_evidence le LEFT JOIN usage_records ur ON ur.id = le.usage_record_id WHERE le.label = labels.label) AS first_seen,
   (SELECT MAX(COALESCE(ur.timestamp, le.timestamp, le.imported_at)) FROM label_evidence le LEFT JOIN usage_records ur ON ur.id = le.usage_record_id WHERE le.label = labels.label) AS last_seen,
   (SELECT MAX(estimate_label) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)) AS estimate_label
@@ -51,6 +56,7 @@ ORDER BY estimated_ai_credits DESC, labels.label`);
 }
 
 async function labelSummary(dbPath, label) {
+  await initStore(dbPath);
   const rows = await queryRows(dbPath, `
 SELECT
   labels.label,
@@ -63,6 +69,10 @@ SELECT
   COALESCE((SELECT SUM(cache_creation_tokens) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS cache_creation_tokens,
   COALESCE((SELECT SUM(reasoning_tokens) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS reasoning_tokens,
   COALESCE((SELECT SUM(COALESCE(estimated_ai_credits, 0)) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)), 0) AS estimated_ai_credits,
+  CASE
+    WHEN (SELECT COUNT(DISTINCT usage_record_id) FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL) = 0 THEN 'hook-only'
+    ELSE 'token-bearing'
+  END AS token_status,
   (SELECT MIN(COALESCE(ur.timestamp, le.timestamp, le.imported_at)) FROM label_evidence le LEFT JOIN usage_records ur ON ur.id = le.usage_record_id WHERE le.label = labels.label) AS first_seen,
   (SELECT MAX(COALESCE(ur.timestamp, le.timestamp, le.imported_at)) FROM label_evidence le LEFT JOIN usage_records ur ON ur.id = le.usage_record_id WHERE le.label = labels.label) AS last_seen,
   (SELECT MAX(estimate_label) FROM usage_records WHERE id IN (SELECT DISTINCT usage_record_id FROM label_evidence WHERE label = labels.label AND usage_record_id IS NOT NULL)) AS estimate_label
@@ -71,6 +81,7 @@ FROM (SELECT DISTINCT label FROM label_evidence WHERE label = ?) labels`, [canon
 }
 
 async function labelDetails(dbPath, label) {
+  await initStore(dbPath);
   return queryRows(dbPath, `
 SELECT
   le.label,
@@ -87,6 +98,9 @@ SELECT
   ur.resolved_model,
   ur.input_tokens,
   ur.output_tokens,
+  ur.cache_read_tokens,
+  ur.cache_creation_tokens,
+  ur.reasoning_tokens,
   ur.estimated_ai_credits,
   ur.estimate_label,
   COALESCE(ur.timestamp, le.timestamp, le.imported_at) AS timestamp
@@ -97,6 +111,7 @@ ORDER BY timestamp, le.source_type, le.source_field`, [canonicalLabel(label)]);
 }
 
 async function modelReport(dbPath) {
+  await initStore(dbPath);
   return queryRows(dbPath, `
 SELECT
   COALESCE(resolved_model, requested_model, 'unknown') AS model,
@@ -114,6 +129,7 @@ ORDER BY estimated_ai_credits DESC, model`);
 }
 
 async function repoReport(dbPath) {
+  await initStore(dbPath);
   return queryRows(dbPath, `
 SELECT
   COALESCE(repo, 'unknown') AS repo,
@@ -130,6 +146,7 @@ ORDER BY estimated_ai_credits DESC, repo, cwd`);
 }
 
 async function unattributedReport(dbPath) {
+  await initStore(dbPath);
   return queryRows(dbPath, `
 SELECT
   ur.id,
@@ -156,13 +173,18 @@ ORDER BY ur.timestamp, ur.id`);
 function formatLabels(rows) {
   return [
     table(
-      ['Label', 'Sessions', 'Input', 'Output', 'Credits', 'Evidence', 'Last seen'],
+      ['Label', 'Sessions', 'Usage', 'Input', 'Output', 'Cache read', 'Cache create', 'Reasoning', 'Credits', 'Status', 'Evidence', 'Last seen'],
       rows.map((row) => [
         row.label,
         row.sessions,
+        row.usage_records,
         formatNumber(row.input_tokens),
         formatNumber(row.output_tokens),
+        formatNumber(row.cache_read_tokens),
+        formatNumber(row.cache_creation_tokens),
+        formatNumber(row.reasoning_tokens),
         formatCredits(row.estimated_ai_credits),
+        row.token_status,
         row.evidence_count,
         row.last_seen || '',
       ]),
@@ -176,18 +198,35 @@ function formatLabelSummary(summary, details = null) {
   if (!summary) return 'No usage found for label.';
   const lines = [
     table(
-      ['Label', 'Sessions', 'Input', 'Output', 'Credits', 'Evidence'],
-      [[summary.label, summary.sessions, formatNumber(summary.input_tokens), formatNumber(summary.output_tokens), formatCredits(summary.estimated_ai_credits), summary.evidence_count]],
+      ['Label', 'Sessions', 'Usage', 'Input', 'Output', 'Cache read', 'Cache create', 'Reasoning', 'Credits', 'Status', 'Evidence'],
+      [[
+        summary.label,
+        summary.sessions,
+        summary.usage_records,
+        formatNumber(summary.input_tokens),
+        formatNumber(summary.output_tokens),
+        formatNumber(summary.cache_read_tokens),
+        formatNumber(summary.cache_creation_tokens),
+        formatNumber(summary.reasoning_tokens),
+        formatCredits(summary.estimated_ai_credits),
+        summary.token_status,
+        summary.evidence_count,
+      ]],
     ),
   ];
   if (details) {
     lines.push('', table(
-      ['Source', 'Field', 'Session', 'Model', 'Credits', 'Value'],
+      ['Source', 'Field', 'Session', 'Model', 'Input', 'Output', 'Cache read', 'Cache create', 'Reasoning', 'Credits', 'Value'],
       details.map((row) => [
         row.source_type,
         row.source_field,
         row.session_id || '',
         row.resolved_model || '',
+        formatNumber(row.input_tokens),
+        formatNumber(row.output_tokens),
+        formatNumber(row.cache_read_tokens),
+        formatNumber(row.cache_creation_tokens),
+        formatNumber(row.reasoning_tokens),
         formatCredits(row.estimated_ai_credits),
         row.source_value || '',
       ]),

package/src/setup.js

@@ -45,6 +45,7 @@ function ensureDataDirs(paths) {
   if (!fs.existsSync(paths.configJson)) {
     writePrivateFile(paths.configJson, `${JSON.stringify({
       version: 1,
+      dataHome: paths.home,
       contentCapture: false,
       telemetry: {
         vscode: paths.vscodeOtelJsonl,
@@ -93,6 +94,11 @@ function packageBinCommand(cwd) {
   return path.join(cwd, 'bin', 'copilot-metrics.js');
 }
 
+function commandInvocation(command) {
+  const quoted = shellQuote(command);
+  return command.endsWith('.js') ? `node ${quoted}` : quoted;
+}
+
 function hookEventsForSurface(surface) {
   if (surface === 'copilot-cli' || surface === 'both') return COPILOT_CLI_HOOK_EVENTS;
   if (surface === 'vscode') return VSCODE_HOOK_EVENTS;
@@ -100,7 +106,7 @@ function hookEventsForSurface(surface) {
 }
 
 function hookCommand(command, event, metricsHome) {
-  return `COPILOT_METRICS_HOME=${shellQuote(metricsHome)} node ${shellQuote(command)} hook-log --event ${shellQuote(event)} --quiet`;
+  return `COPILOT_METRICS_HOME=${shellQuote(metricsHome)} ${commandInvocation(command)} hook-log --event ${shellQuote(event)} --quiet`;
 }
 
 function hookConfig(paths, options = {}) {
@@ -171,6 +177,7 @@ function installHook(paths, options = {}) {
 
 function setupSnapshot(options = {}) {
   const paths = resolvePaths(options);
+  ensureDataDirs(paths);
   return {
     paths,
     vscode: vscodeSettings(paths),

package/src/sqlite-store.js

@@ -29,6 +29,19 @@ function persistDatabase(dbPath, db) {
   }
 }
 
+function hasColumn(db, table, column) {
+  const result = db.exec(`PRAGMA table_info(${table})`);
+  if (!result.length) return false;
+  const nameIndex = result[0].columns.indexOf('name');
+  return result[0].values.some((row) => row[nameIndex] === column);
+}
+
+function addColumnIfMissing(db, table, column, definition) {
+  if (!hasColumn(db, table, column)) {
+    db.run(`ALTER TABLE ${table} ADD COLUMN ${column} ${definition}`);
+  }
+}
+
 async function initStore(dbPath) {
   const db = await openDatabase(dbPath);
   db.run(`
@@ -36,7 +49,9 @@ CREATE TABLE IF NOT EXISTS raw_records (
   id INTEGER PRIMARY KEY AUTOINCREMENT,
   imported_at TEXT NOT NULL,
   source TEXT NOT NULL,
+  source_file TEXT,
   line INTEGER NOT NULL,
+  raw_fingerprint TEXT,
   payload_json TEXT NOT NULL
 );
 CREATE TABLE IF NOT EXISTS usage_records (
@@ -105,6 +120,9 @@ CREATE TABLE IF NOT EXISTS import_warnings (
   message TEXT NOT NULL
 );
 `);
+  addColumnIfMissing(db, 'raw_records', 'source_file', 'TEXT');
+  addColumnIfMissing(db, 'raw_records', 'raw_fingerprint', 'TEXT');
+  db.run('CREATE UNIQUE INDEX IF NOT EXISTS idx_raw_records_fingerprint ON raw_records (source, source_file, raw_fingerprint)');
   persistDatabase(dbPath, db);
 }
 
@@ -147,7 +165,25 @@ function insertLabelEvidence(db, importedAt, evidenceRows) {
   );
 }
 
-async function insertImport(dbPath, source, rawRecords, usageRecords, hookEvents, warnings) {
+async function existingRawFingerprints(dbPath, source, sourceFile, fingerprints) {
+  await initStore(dbPath);
+  if (!fingerprints.length) return new Set();
+  const db = await openDatabase(dbPath);
+  const existing = new Set();
+  const statement = db.prepare('SELECT 1 FROM raw_records WHERE source = ? AND source_file = ? AND raw_fingerprint = ? LIMIT 1');
+  try {
+    for (const fingerprint of fingerprints) {
+      statement.bind([source, sourceFile, fingerprint]);
+      if (statement.step()) existing.add(fingerprint);
+      statement.reset();
+    }
+  } finally {
+    statement.free();
+  }
+  return existing;
+}
+
+async function insertImport(dbPath, source, sourceFile, rawRecords, usageRecords, hookEvents, warnings) {
   await initStore(dbPath);
   const db = await openDatabase(dbPath);
   const importedAt = new Date().toISOString();
@@ -156,8 +192,8 @@ async function insertImport(dbPath, source, rawRecords, usageRecords, hookEvents
   try {
     runPrepared(
       db,
-      'INSERT INTO raw_records (imported_at, source, line, payload_json) VALUES (?, ?, ?, ?)',
-      rawRecords.map((record) => [importedAt, source, record.line, JSON.stringify(record.value)]),
+      'INSERT OR IGNORE INTO raw_records (imported_at, source, source_file, line, raw_fingerprint, payload_json) VALUES (?, ?, ?, ?, ?, ?)',
+      rawRecords.map((record) => [importedAt, source, sourceFile, record.line, record.raw_fingerprint || null, JSON.stringify(record.value)]),
     );
 
     const labelEvidence = [];
@@ -283,6 +319,7 @@ async function queryRows(dbPath, sql, params = []) {
 }
 
 module.exports = {
+  existingRawFingerprints,
   initStore,
   insertImport,
   queryOne,