copilot-metrics 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,22 @@
+# Changelog
+
+## 0.1.0 - 2026-05-30
+
+First local release candidate for `copilot-metrics`.
+
+### Added
+
+- CLI setup helpers for central local data directories, VS Code Copilot OTel settings, Copilot CLI OTel environment exports, and local/global hook config.
+- Redacted hook logger that captures safe attribution metadata with content capture disabled by default.
+- Local SQLite-backed import for VS Code Copilot OTel JSONL, Copilot CLI OTel JSONL, and Copilot CLI hook JSONL.
+- LLM span normalization, root-agent double-count prevention, token extraction, model pricing estimates, AI Credit estimates, malformed-row warnings, and unknown-model warnings.
+- Jira-style label extraction with evidence-preserving attribution by source, field, session, repo, branch, cwd, and confidence.
+- Configurable custom label extractors loaded from local config without modifying package source.
+- CLI reports for label overview, single-label summary/detail, models, repos/directories, and unattributed usage, with human and JSON output.
+- Release smoke checks, package verification, GitHub Actions npm publishing workflow, MIT license, and release checklist.
+
+### Notes
+
+- Cost and AI Credit values are estimates only. GitHub billing remains the source of truth.
+- Prompt/content capture is disabled by default.
+- Official usage reconciliation, collector mode, richer privacy controls, and dashboard views are deferred.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 copilot-metrics contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,174 @@
+# Copilot Metrics
+
+`copilot-metrics` is a local-first CLI for estimating GitHub Copilot usage from local OpenTelemetry and hook metadata. It helps answer which Jira-style labels, repos, models, and Copilot surfaces are driving estimated AI Credit usage.
+
+Costs are estimates, not official billing records. GitHub billing remains the source of truth.
+
+## Install
+
+From npm:
+
+```bash
+npx copilot-metrics@0.1.0 --help
+npx copilot-metrics@0.1.0 init
+```
+
+From this checkout:
+
+```bash
+npm ci
+npm test
+npm run cli -- --help
+```
+
+## Data Directory
+
+By default, all metadata is stored in a user-level local folder:
+
+- Linux: `$XDG_DATA_HOME/copilot-metrics` or `~/.local/share/copilot-metrics`
+- macOS: `~/Library/Application Support/copilot-metrics`
+- Windows: `%LOCALAPPDATA%\\copilot-metrics`
+
+Override it with:
+
+```bash
+export COPILOT_METRICS_HOME=/path/to/copilot-metrics-data
+```
+
+Useful commands:
+
+```bash
+npx copilot-metrics@0.1.0 init
+npx copilot-metrics@0.1.0 paths --json
+```
+
+## Configure Telemetry
+
+Print VS Code Insiders Copilot Chat OpenTelemetry settings:
+
+```bash
+npx copilot-metrics@0.1.0 setup vscode
+```
+
+Print Copilot CLI OpenTelemetry environment exports:
+
+```bash
+npx copilot-metrics@0.1.0 setup copilot-cli
+```
+
+Content capture is disabled by default. Do not enable richer prompt capture unless you explicitly accept the privacy tradeoff.
+
+## Configure Hooks
+
+Preview repo-local hook config. The default `--surface both` emits the Copilot CLI lower camel case hook format:
+
+```bash
+npx copilot-metrics@0.1.0 hooks preview --scope local --surface both
+```
+
+Install repo-local or user-global hook config:
+
+```bash
+npx copilot-metrics@0.1.0 hooks install --scope local --surface both
+npx copilot-metrics@0.1.0 hooks install --scope global --surface both
+```
+
+Local install writes `.github/hooks/copilot-metrics.json`. Global install updates `~/.copilot/settings.json` idempotently, replacing prior `copilot-metrics` hook entries while preserving other settings and hooks. Use `--surface vscode` for VS Code-only PascalCase events or `--surface copilot-cli` for CLI-native lower camel case events. The hook logger writes redacted JSONL metadata to the central data directory. It extracts Jira-style labels such as `DEMO-12345` from safe metadata and does not store full prompt text by default.
+
+## Import Telemetry
+
+Initialize the local SQLite store and import JSONL files:
+
+```bash
+npx copilot-metrics@0.1.0 store init
+npx copilot-metrics@0.1.0 import --source vscode --file ~/.local/share/copilot-metrics/telemetry/vscode-copilot-otel.jsonl
+npx copilot-metrics@0.1.0 import --source copilot-cli --file ~/.local/share/copilot-metrics/telemetry/copilot-cli-otel.jsonl
+npx copilot-metrics@0.1.0 import --source hooks --file ~/.local/share/copilot-metrics/hooks/copilot-hooks.jsonl
+```
+
+Imports persist raw records, normalized LLM usage records, hook events, label evidence, and import warnings.
+
+## Reports
+
+Run local reports from the SQLite store:
+
+```bash
+npx copilot-metrics@0.1.0 report labels
+npx copilot-metrics@0.1.0 report label DEMO-12345
+npx copilot-metrics@0.1.0 report label DEMO-12345 --detail
+npx copilot-metrics@0.1.0 report models
+npx copilot-metrics@0.1.0 report repos
+npx copilot-metrics@0.1.0 report unattributed
+```
+
+Every report supports `--json`:
+
+```bash
+npx copilot-metrics@0.1.0 report labels --json
+```
+
+## Attribution Model
+
+The default extractor finds Jira-style labels such as `DEMO-12345` from safe metadata including hook labels, branch names, cwd/path values, repo metadata, and task hints.
+
+Attribution is stored as evidence with source, field, session, repo, branch, cwd, confidence, and related usage or hook record IDs. This makes the data useful for later analysis, such as deciding whether a label was the main task or a sidetrack.
+
+Full prompt content is not stored by default. Prompt-like fields are only used to extract labels and the stored source value is reduced to the matched label.
+
+## Custom Label Extractors
+
+Custom extractors are configured in the local `config.json`; you do not modify package source.
+
+After `copilot-metrics init`, add a module path:
+
+```json
+{
+  "labelExtractors": ["/absolute/path/to/my-extractor.cjs"]
+}
+```
+
+Relative paths are resolved from the current working directory when the CLI runs.
+
+The module should export a function, or an object with `extractLabels`. Each extractor receives:
+
+- `sourceType`: for example `usage` or `hook`
+- `sourceData`: safe metadata for that source
+
+It returns zero or more labels, either as strings or evidence objects:
+
+```js
+const extractor = (sourceType, sourceData) => {
+  if (sourceData.branch === 'main') return [];
+  return [{ label: 'TEAM-123', source_field: 'branch', source_type: sourceType, confidence: 0.5 }];
+};
+
+module.exports = extractor;
+```
+
+## Release Verification
+
+For a release candidate checkout:
+
+```bash
+npm test
+npm run check
+npm run smoke
+npm run verify:package
+```
+
+Manual Copilot CLI validation is local-only and not run in CI:
+
+```bash
+node scripts/manual-copilot-cli-flow.js --setup-only
+node scripts/manual-copilot-cli-flow.js --run-prompt --model gpt-5-mini
+```
+
+The manual prompt performs one harmless tool call so Copilot CLI hook execution can be validated; answer quality is not part of the check. During the prompt run, the helper temporarily adds generated hooks to `~/.copilot/settings.json` and restores the original settings afterward.
+
+## Current Limits
+
+- Costs are estimates, not official billing records.
+- Official GitHub usage report reconciliation is not included in `0.1.0`.
+- Local OTLP collector mode is not included in `0.1.0`.
+- Richer prompt/content capture and redaction controls are not included in `0.1.0`.
+- Dashboard views are deferred until the CLI/query model proves useful.

package/RELEASE.md

@@ -0,0 +1,74 @@
+# Release Checklist
+
+This project publishes `copilot-metrics` to npm through GitHub Actions. The human gate is creating the GitHub release or tag that triggers the workflow; the workflow runs verification and then `npm publish`.
+
+## Prerequisites
+
+- GitHub repository exists at `nnexai/copilot-metrics`.
+- npm Trusted Publishing is configured for package `copilot-metrics`:
+  - Publisher: GitHub Actions
+  - Repository: `nnexai/copilot-metrics`
+  - Workflow filename: `npm-publish.yml`
+  - Allowed action: `npm publish`
+- npm package name is available or owned by the publishing account.
+- Local working tree is clean before tagging.
+
+## Local Verification
+
+Run:
+
+```bash
+npm ci
+npm test
+npm run check
+npm run smoke
+npm run verify:package
+npm pack --dry-run --json
+```
+
+The package must not include `.planning/`, `.codex/`, `test/`, fixture data, local telemetry, or generated SQLite stores.
+
+## Manual Copilot CLI Validation
+
+This is not a CI step because it depends on a local authenticated Copilot CLI and can call external services.
+
+Run setup/dry-run first:
+
+```bash
+node scripts/manual-copilot-cli-flow.js --setup-only
+```
+
+Run the full local validation from this checkout:
+
+```bash
+node scripts/manual-copilot-cli-flow.js --run-prompt --model gpt-5-mini
+```
+
+The script creates an example workspace, configures `copilot-metrics`, installs repo-local hook config, temporarily applies user-level Copilot CLI hook settings for the prompt run, imports collected telemetry/hook JSONL, prints report output, and restores the original Copilot settings.
+
+## GitHub Actions Publish
+
+1. Confirm `package.json` version is `0.1.0`.
+2. Commit all release changes.
+3. Push `main`.
+4. Create a GitHub release for `v0.1.0`.
+5. Confirm the `Node.js Package` workflow passes and publishes to npm through Trusted Publishing.
+
+## Post-Publish Verification
+
+After the workflow publishes:
+
+```bash
+npm view copilot-metrics@0.1.0 version
+npm view copilot-metrics@0.1.0 dist.tarball
+npx copilot-metrics@0.1.0 --help
+npx copilot-metrics@0.1.0 paths --json
+```
+
+## Do Not Publish
+
+- Local telemetry JSONL files
+- Generated SQLite stores
+- `.planning/` artifacts
+- `.codex/` runtime files
+- Prompt content or hook logs

package/bin/copilot-metrics.js

@@ -0,0 +1,16 @@
+#!/usr/bin/env node
+'use strict';
+
+const { main } = require('../src/cli');
+
+main(process.argv.slice(2), {
+  stdin: process.stdin,
+  stdout: process.stdout,
+  stderr: process.stderr,
+  env: process.env,
+  cwd: process.cwd(),
+  commandPath: process.argv[1],
+}).catch((error) => {
+  process.stderr.write(`copilot-metrics: ${error.message}\n`);
+  process.exitCode = 1;
+});

package/package.json

@@ -0,0 +1,48 @@
+{
+  "name": "copilot-metrics",
+  "version": "0.1.0",
+  "description": "Local-first Copilot usage telemetry setup and reporting tools.",
+  "type": "commonjs",
+  "homepage": "https://github.com/nnexai/copilot-metrics#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+ssh://git@github.com/nnexai/copilot-metrics.git"
+  },
+  "bugs": {
+    "url": "https://github.com/nnexai/copilot-metrics/issues"
+  },
+  "bin": {
+    "copilot-metrics": "bin/copilot-metrics.js"
+  },
+  "files": [
+    "bin/",
+    "src/",
+    "skills/",
+    "scripts/manual-copilot-cli-flow.js",
+    "README.md",
+    "CHANGELOG.md",
+    "RELEASE.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "cli": "node bin/copilot-metrics.js",
+    "check": "node --check bin/copilot-metrics.js && node --check src/*.js",
+    "smoke": "node scripts/smoke.js",
+    "verify:package": "node scripts/verify-package.js",
+    "test": "node --test"
+  },
+  "keywords": [
+    "copilot",
+    "telemetry",
+    "opentelemetry",
+    "cli",
+    "ai-credits"
+  ],
+  "engines": {
+    "node": ">=20"
+  },
+  "license": "MIT",
+  "dependencies": {
+    "sql.js": "^1.14.1"
+  }
+}

package/scripts/manual-copilot-cli-flow.js

@@ -0,0 +1,134 @@
+#!/usr/bin/env node
+'use strict';
+
+const { execFileSync, spawnSync } = require('node:child_process');
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const { mergeGlobalSettingsHooks } = require('../src/setup');
+
+const root = path.join(__dirname, '..');
+const cli = path.join(root, 'bin', 'copilot-metrics.js');
+
+function hasFlag(name) {
+  return process.argv.includes(name);
+}
+
+function option(name, fallback) {
+  const index = process.argv.indexOf(name);
+  return index >= 0 && process.argv[index + 1] ? process.argv[index + 1] : fallback;
+}
+
+function run(command, args, options = {}) {
+  return execFileSync(command, args, {
+    encoding: 'utf8',
+    stdio: options.stdio || 'pipe',
+    cwd: options.cwd,
+    env: options.env,
+  });
+}
+
+function runCli(args, cwd, home) {
+  return run(process.execPath, [cli, ...args, '--home', home], { cwd });
+}
+
+const workspace = path.resolve(option('--workspace', fs.mkdtempSync(path.join(os.tmpdir(), 'copilot-metrics-example-'))));
+const metricsHome = path.resolve(option('--home', path.join(workspace, '.copilot-metrics-data')));
+const model = option('--model', 'gpt-5-mini');
+const prompt = option(
+  '--prompt',
+  'Run pwd once, then reply with exactly: copilot-metrics validation ok',
+);
+const runPrompt = hasFlag('--run-prompt');
+const setupOnly = hasFlag('--setup-only') || !runPrompt;
+
+fs.mkdirSync(workspace, { recursive: true });
+fs.writeFileSync(path.join(workspace, 'README.md'), '# copilot-metrics validation workspace\n', { flag: 'a' });
+if (!fs.existsSync(path.join(workspace, '.git'))) {
+  run('git', ['init'], { cwd: workspace });
+}
+try {
+  run('git', ['remote', 'get-url', 'origin'], { cwd: workspace });
+} catch {
+  run('git', ['remote', 'add', 'origin', 'https://github.com/nnexai/copilot-metrics.git'], { cwd: workspace });
+}
+
+runCli(['init', '--json'], workspace, metricsHome);
+const hookInstall = JSON.parse(runCli(['hooks', 'install', '--scope', 'local', '--surface', 'both', '--json'], workspace, metricsHome));
+const envConfig = JSON.parse(runCli(['setup', 'copilot-cli', '--json'], workspace, metricsHome));
+const paths = JSON.parse(runCli(['paths', '--json'], workspace, metricsHome));
+
+const envFile = path.join(workspace, '.copilot-metrics.env');
+fs.writeFileSync(envFile, `${Object.entries(envConfig).map(([key, value]) => `export ${key}=${JSON.stringify(value)}`).join('\n')}\n`);
+
+const result = {
+  workspace,
+  metricsHome,
+  envFile,
+  hookConfig: hookInstall.target,
+  temporaryGlobalSettings: path.join(process.env.COPILOT_HOME || path.join(os.homedir(), '.copilot'), 'settings.json'),
+  copilotCliOtelJsonl: paths.copilotCliOtelJsonl,
+  hookEventsJsonl: paths.hookEventsJsonl,
+  ranPrompt: false,
+  telemetryExists: false,
+  hooksExist: false,
+};
+
+if (setupOnly) {
+  process.stdout.write(`${JSON.stringify({ ...result, next: 'Re-run with --run-prompt to call Copilot CLI.' }, null, 2)}\n`);
+  process.exit(0);
+}
+
+const settingsExisted = fs.existsSync(result.temporaryGlobalSettings);
+const originalSettings = settingsExisted ? fs.readFileSync(result.temporaryGlobalSettings, 'utf8') : null;
+let copilot;
+try {
+  const settings = settingsExisted ? JSON.parse(originalSettings) : {};
+  fs.mkdirSync(path.dirname(result.temporaryGlobalSettings), { recursive: true });
+  fs.writeFileSync(
+    result.temporaryGlobalSettings,
+    `${JSON.stringify(mergeGlobalSettingsHooks(settings, hookInstall.config.hooks), null, 2)}\n`,
+  );
+
+  copilot = spawnSync('copilot', ['-p', prompt, '--yolo', '--model', model, '--no-auto-update', '--silent'], {
+    cwd: workspace,
+    env: {
+      ...process.env,
+      ...envConfig,
+      COPILOT_METRICS_HOME: metricsHome,
+    },
+    encoding: 'utf8',
+    maxBuffer: 1024 * 1024 * 8,
+  });
+} finally {
+  if (settingsExisted) fs.writeFileSync(result.temporaryGlobalSettings, originalSettings);
+  else if (fs.existsSync(result.temporaryGlobalSettings)) fs.rmSync(result.temporaryGlobalSettings);
+}
+
+result.ranPrompt = true;
+result.exitCode = copilot.status;
+result.stdout = (copilot.stdout || '').slice(0, 2000);
+result.stderr = (copilot.stderr || '').slice(0, 2000);
+
+if (copilot.status !== 0) {
+  process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
+  process.exit(copilot.status || 1);
+}
+
+result.telemetryExists = fs.existsSync(paths.copilotCliOtelJsonl) && fs.statSync(paths.copilotCliOtelJsonl).size > 0;
+result.hooksExist = fs.existsSync(paths.hookEventsJsonl) && fs.statSync(paths.hookEventsJsonl).size > 0;
+
+if (result.telemetryExists) {
+  result.import = JSON.parse(runCli(['import', '--source', 'copilot-cli', '--file', paths.copilotCliOtelJsonl, '--json'], workspace, metricsHome));
+  result.models = JSON.parse(runCli(['report', 'models', '--json'], workspace, metricsHome));
+}
+
+if (result.hooksExist) {
+  result.hooksImport = JSON.parse(runCli(['import', '--source', 'hooks', '--file', paths.hookEventsJsonl, '--json'], workspace, metricsHome));
+}
+
+process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
+
+if (!result.telemetryExists || !result.hooksExist) {
+  process.exitCode = 2;
+}

package/skills/copilot-metrics/SKILL.md

@@ -0,0 +1,36 @@
+---
+name: copilot-metrics
+description: Query local Copilot Metrics data through the CLI without reading sensitive raw prompt content by default.
+---
+
+# Copilot Metrics Skill
+
+Use this skill when the user asks which Jira labels, repos, models, or Copilot surfaces are driving estimated local Copilot usage.
+
+## Rules
+
+- Use the `copilot-metrics` CLI instead of reading raw telemetry files directly.
+- Start with `copilot-metrics paths --json` to find the local data directory.
+- If the data directory is missing, suggest `copilot-metrics init`.
+- Treat all costs as estimates, not official GitHub billing records.
+- Do not read or display full prompts unless the user explicitly says content capture is enabled and asks for that content.
+- Prefer machine-readable output when summarizing for another tool: use `--json` where available.
+
+## Useful Commands
+
+```bash
+copilot-metrics paths --json
+copilot-metrics setup vscode --json
+copilot-metrics setup copilot-cli --json
+copilot-metrics hooks preview --scope local --json
+```
+
+Future report commands should be preferred when present, for example:
+
+```bash
+copilot-metrics report labels --json
+copilot-metrics report label DEMO-12345 --json
+copilot-metrics report unattributed --json
+```
+
+If a future report command is unavailable, explain that ingestion and reporting have not been implemented in this checkout yet.