copilot-tracer 1.0.1

package/README.md

@@ -0,0 +1,239 @@
+# copilot-tracer
+
+Real-time tracing and monitoring tool for **GitHub Copilot CLI** and **VS Code Copilot extension**.
+
+Captures every prompt, response, token usage, AI credits, tool calls, skill invocations and duration — all in one place. Works via native **OpenTelemetry (OTLP)** integration built into GitHub Copilot. No wrapper, no binary replacement, no ACP proxy needed.
+
+---
+
+## Features
+
+- **Zero-intrusion capture** — uses Copilot's built-in OTel support. Set 2 env vars, done.
+- **Works everywhere** — captures both Copilot CLI (`copilot -p "..."`) and VS Code Copilot Chat
+- **Real-time web UI** — live dashboard at `http://localhost:4747` with dark theme
+- **Full prompt & response** — see exactly what you sent and what Copilot replied
+- **Token breakdown** — input, output, cached, reasoning, written tokens per request
+- **AI Credits tracking** — matches exactly what Copilot terminal reports (e.g. `2.59 cr`)
+- **Tool call visibility** — see every tool/skill/MCP invoked during a session
+- **Persistent storage** — SQLite at `~/.copilot-tracer/traces.db`, survives restarts
+- **Console + Web UI** — CLI table view or browser dashboard, your choice
+
+---
+
+## Prerequisites
+
+- Node.js v18+ (tested on v24)
+- GitHub Copilot CLI (`copilot` command available in terminal)
+- VS Code 1.99+ with built-in Copilot (no extension install needed)
+
+---
+
+## Install
+
+```bash
+npm install -g copilot-tracer
+```
+
+Requires Node.js v18+. The `copilot-tracer` command is available globally after install.
+
+> **Building from source**
+> ```bash
+> git clone https://github.com/chuongnd/copilot-tracer
+> cd copilot-tracer && npm install && npx tsc
+> npm link   # registers global command from local build
+> ```
+
+---
+
+## Setup (one-time)
+
+Run the auto-setup command. It detects your Copilot CLI and VS Code installation and injects the required config automatically:
+
+```bash
+copilot-tracer --setup
+```
+
+What it does:
+- Detects `copilot` CLI path and version
+- Detects VS Code version and confirms built-in Copilot
+- Patches `~/.zshrc` (or `~/.bashrc`) with OTEL env vars
+- Patches VS Code `settings.json` with `terminal.integrated.env.osx` block
+
+Example output:
+```
+✅ GitHub Copilot CLI detected — /usr/bin/copilot v1.0.77
+✅ Visual Studio Code detected — 1.131.0 (Built-in Copilot)
+✅ Shell profile patched: .zshrc
+✅ VS Code settings patched
+```
+
+Then apply the env vars:
+
+```bash
+source ~/.zshrc
+```
+
+Restart VS Code completely (Cmd+Q, then reopen).
+
+---
+
+## Manual Setup (alternative)
+
+If you prefer to configure manually, add these to `~/.zshrc`:
+
+```bash
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4747
+export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
+export COPILOT_OTEL_ENABLED=true
+```
+
+For VS Code, add to `~/Library/Application Support/Code/User/settings.json`:
+
+```json
+"terminal.integrated.env.osx": {
+  "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4747",
+  "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT": "true",
+  "COPILOT_OTEL_ENABLED": "true"
+}
+```
+
+---
+
+## Start Tracer
+
+```bash
+copilot-tracer --ui web --port 4747 --no-proxy
+```
+
+Open **http://localhost:4747** — shows "waiting for copilot CLI activity".
+
+---
+
+## Use Copilot Normally
+
+No change to how you use Copilot. Just run as usual:
+
+```bash
+# Copilot CLI
+copilot -p "how to convert microservice to modular" --allow-all-tools
+
+# Or use Copilot Chat in VS Code
+```
+
+Traces appear instantly in the web UI as each request completes.
+
+---
+
+## How It Works
+
+```
+copilot CLI  /  VS Code Copilot Chat
+         |
+         |  reads OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4747
+         |
+         ↓  POST /v1/traces  (OpenTelemetry OTLP JSON)
+  copilot-tracer OTLP receiver
+         |
+         ↓
+  Parse spans → extract prompt, response, tokens, credits, tool calls
+         |
+         ↓
+  SQLite DB  (~/.copilot-tracer/traces.db)
+         |
+         ↓
+  Web UI (Socket.io real-time)  +  Console table
+```
+
+Copilot has built-in OpenTelemetry instrumentation. When `OTEL_EXPORTER_OTLP_ENDPOINT` is set, it pushes all trace data to that endpoint automatically — both CLI and VS Code extension.
+
+---
+
+## Web UI
+
+Open http://localhost:4747 after starting the tracer.
+
+**Table columns:**
+| Date/Time | Prompt | AI Credits | Duration | Cached | Written | Reasoning | Skills | Agents | MCPs |
+
+**Interactive features:**
+- Click any row → detail panel: full prompt, full response, reasoning text, call graph
+- Click AI Credits → cost breakdown per token type
+- Click Reasoning count → full reasoning text
+- Click Skills / Agents / MCPs pill → filtered call list with input/output/duration
+- Real-time updates via Socket.io — no page refresh needed
+- Dark theme
+
+---
+
+## Console UI
+
+```bash
+copilot-tracer --ui console --no-proxy
+```
+
+Live updating table in terminal. Same columns as web UI. TOTALS row pinned at top.
+
+---
+
+## CLI Flags
+
+| Flag | Description |
+|------|-------------|
+| `--ui web` | Start web UI (default) |
+| `--ui console` | Start console table UI |
+| `--ui both` | Both web + console |
+| `--port 4747` | Web UI port (default: 4747) |
+| `--no-proxy` | Web/console only, no ACP proxy |
+| `--session <id>` | Custom session ID |
+| `--setup` | Auto-detect and configure env vars |
+| `--debug` | Verbose logging |
+
+---
+
+## Storage
+
+Traces persist to `~/.copilot-tracer/traces.db` (SQLite). Safe to keep across sessions.
+
+To test the web UI without running a live Copilot session:
+
+```bash
+node test-seed.mjs   # seeds 4 sample traces (source build only)
+copilot-tracer --ui web --no-proxy --session test-session-001
+```
+
+---
+
+## Build & Publish
+
+```bash
+npm install
+npx tsc          # compile to dist/
+```
+
+To publish a new release to npm:
+
+```bash
+# Bump patch version (1.0.0 → 1.0.1), build, publish, git-tag
+bash scripts/publish.sh
+
+# Bump minor version
+bash scripts/publish.sh minor
+
+# Publish a beta pre-release
+bash scripts/publish.sh --tag beta --pre beta
+```
+
+The script will:
+1. Check npm authentication (`npm login` required first)
+2. Verify git working tree is clean
+3. Type-check + build
+4. Verify `better-sqlite3` native module loads
+5. Show files that will be published (dry-run preview)
+6. Prompt for confirmation
+7. `npm publish`, commit the version bump, and create a git tag
+
+After publishing:
+
+```bash
+git push && git push origin v<new-version>
+```

package/dist/cli.js

@@ -0,0 +1,108 @@
+#!/usr/bin/env node
+import { program } from 'commander';
+import { spawn } from 'child_process';
+import { randomUUID } from 'crypto';
+import readline from 'readline';
+import fs from 'fs';
+import { createSession, getTraces, getSessionSummary } from './db.js';
+import { handleAcpMessage, traceEvents } from './proxy.js';
+import { renderConsoleTable } from './consoleUi.js';
+import { startWebServer } from './webServer.js';
+import { runSetup } from './setup.js';
+import open from 'open';
+program
+    .name('copilot-tracer')
+    .description('Real-time monitor and tracer for GitHub Copilot CLI')
+    .option('--ui <mode>', 'UI mode: console | web | both', 'both')
+    .option('--port <port>', 'Web UI port', '4747')
+    .option('--cmd <command>', 'Copilot CLI command to wrap', 'copilot')
+    .option('--no-proxy', 'Run UI only (no ACP proxy, read from DB)')
+    .option('--session <id>', 'Filter by session ID')
+    .option('--debug', 'Dump ALL raw ACP messages to stderr (use to discover real method names)')
+    .option('--setup', 'Auto-detect copilot CLI + VS Code and configure OTLP env vars')
+    .allowUnknownOption()
+    .parse();
+const opts = program.opts();
+const port = parseInt(opts.port);
+// Handle --setup before anything else
+if (opts.setup) {
+    runSetup(port);
+    process.exit(0);
+}
+const sessionId = opts.session || randomUUID();
+createSession(sessionId);
+console.log(`\n  🤖 Copilot Tracer  |  Session: ${sessionId}\n`);
+// Start Web UI
+if (opts.ui === 'web' || opts.ui === 'both') {
+    startWebServer(port, sessionId);
+    setTimeout(() => open(`http://localhost:${port}`), 1500);
+}
+// Start Console UI refresh loop
+if (opts.ui === 'console' || opts.ui === 'both') {
+    const refreshConsole = () => {
+        const entries = getTraces(sessionId, 50);
+        const summary = getSessionSummary(sessionId);
+        renderConsoleTable(entries, summary ?? undefined);
+    };
+    traceEvents.on('trace:update', refreshConsole);
+    traceEvents.on('trace:done', refreshConsole);
+    refreshConsole();
+}
+// ACP Proxy — wrap copilot CLI
+if (opts.proxy !== false) {
+    const copilotArgs = ['--acp', '--stdio', ...program.args];
+    const child = spawn(opts.cmd, copilotArgs, {
+        stdio: ['pipe', 'pipe', 'inherit'],
+    });
+    if (!child.pid) {
+        console.error(`\n  ❌ Failed to start: ${opts.cmd} ${copilotArgs.join(' ')}`);
+        console.error('  Make sure GitHub Copilot CLI is installed: npm install -g @github/copilot-cli\n');
+        process.exit(1);
+    }
+    // Parse newline-delimited JSON (NDJSON) from copilot
+    const stdinRl = readline.createInterface({ input: process.stdin });
+    const stdoutRl = readline.createInterface({ input: child.stdout });
+    const debug = opts.debug === true;
+    const logFile = debug ? fs.createWriteStream(`/tmp/copilot-tracer-${sessionId.slice(0, 8)}.ndjson`, { flags: 'a' }) : null;
+    function debugLog(direction, raw, parsed) {
+        if (!debug)
+            return;
+        const entry = JSON.stringify({ ts: new Date().toISOString(), dir: direction, raw, parsed });
+        process.stderr.write('[TRACER] ' + entry + '\n');
+        logFile?.write(entry + '\n');
+    }
+    // stdin → copilot (user → copilot = 'out' direction from user's perspective)
+    stdinRl.on('line', (line) => {
+        let parsed;
+        try {
+            parsed = JSON.parse(line);
+            handleAcpMessage(sessionId, parsed, 'out'); // outbound = user sending to copilot
+        }
+        catch (e) {
+            // Not JSON — plain text from terminal, not ACP
+            if (debug)
+                process.stderr.write(`[TRACER] stdin non-JSON: ${line}\n`);
+        }
+        debugLog('→ copilot', line, parsed);
+        child.stdin.write(line + '\n');
+    });
+    // copilot → stdout (copilot → user = 'in' direction from user's perspective)
+    stdoutRl.on('line', (line) => {
+        let parsed;
+        try {
+            parsed = JSON.parse(line);
+            handleAcpMessage(sessionId, parsed, 'in'); // inbound = copilot sending to user
+        }
+        catch (e) {
+            if (debug)
+                process.stderr.write(`[TRACER] stdout non-JSON: ${line}\n`);
+        }
+        debugLog('← copilot', line, parsed);
+        process.stdout.write(line + '\n');
+    });
+    child.on('exit', (code) => {
+        console.log(`\n  Copilot CLI exited (code ${code})\n`);
+        process.exit(code ?? 0);
+    });
+    process.on('SIGINT', () => { child.kill(); process.exit(0); });
+}

package/dist/consoleUi.js

@@ -0,0 +1,74 @@
+import chalk from 'chalk';
+import Table from 'cli-table3';
+import { format } from 'date-fns';
+function truncate(str, len) {
+    return str.length > len ? str.slice(0, len - 1) + '…' : str;
+}
+function fmtDuration(ms) {
+    if (ms < 1000)
+        return `${ms}ms`;
+    return `${(ms / 1000).toFixed(1)}s`;
+}
+function fmtCredits(c) {
+    return `${c.toFixed(2)} cr | $${(c * 0.01).toFixed(4)}`;
+}
+function statusColor(status, text) {
+    if (status === 'running')
+        return chalk.yellow(text);
+    if (status === 'error')
+        return chalk.red(text);
+    return chalk.green(text);
+}
+export function renderConsoleTable(entries, summary) {
+    console.clear();
+    const table = new Table({
+        head: [
+            chalk.cyan('Date / Time'),
+            chalk.cyan('Prompt'),
+            chalk.cyan('AI Credits'),
+            chalk.cyan('Duration'),
+            chalk.cyan('Tokens\nCached|Written|Reason'),
+            chalk.cyan('Skills'),
+            chalk.cyan('Agents'),
+            chalk.cyan('MCPs'),
+        ],
+        colWidths: [20, 40, 12, 10, 26, 8, 8, 8],
+        style: { head: [], border: ['grey'] },
+        wordWrap: true,
+    });
+    // TOTALS row (right after header)
+    if (summary) {
+        const t = summary.totalTokens;
+        table.push([
+            chalk.bold.white('TOTALS'),
+            chalk.bold.white(`${summary.totalEntries} prompts`),
+            chalk.bold.yellow(fmtCredits(summary.totalCredits)),
+            chalk.bold.white(fmtDuration(summary.totalDurationMs)),
+            chalk.bold.white(`${t.cached} | ${t.written} | ${t.reasoning}`),
+            chalk.bold.magenta(String(summary.totalSkillCalls)),
+            chalk.bold.blue(String(summary.totalAgentCalls)),
+            chalk.bold.cyan(String(summary.totalMcpCalls)),
+        ]);
+        // divider
+        table.push([{ colSpan: 8, content: chalk.grey('─'.repeat(130)) }]);
+    }
+    // Data rows
+    for (const e of entries) {
+        const dt = format(new Date(e.dateTime), 'MM-dd HH:mm:ss');
+        const tokenStr = `${e.tokens.cached} | ${e.tokens.written} | ${e.tokens.reasoning}`;
+        const tools = e.toolCalls.map(c => c.name).join(', ');
+        table.push([
+            statusColor(e.status, dt),
+            truncate(e.prompt, 38),
+            chalk.yellow(fmtCredits(e.aiCredits)),
+            fmtDuration(e.durationMs),
+            tokenStr + (tools ? chalk.grey(`\n[${truncate(tools, 22)}]`) : ''),
+            chalk.magenta(String(e.skillCount)),
+            chalk.blue(String(e.agentCount)),
+            chalk.cyan(String(e.mcpCount)),
+        ]);
+    }
+    console.log(chalk.bold.white('\n  🤖  COPILOT TRACER — Real-time Monitor\n'));
+    console.log(table.toString());
+    console.log(chalk.grey(`  Last updated: ${format(new Date(), 'HH:mm:ss')}  |  DB: ~/.copilot-tracer/traces.db\n`));
+}

package/dist/db.js

@@ -0,0 +1,158 @@
+import Database from 'better-sqlite3';
+import path from 'path';
+import os from 'os';
+import fs from 'fs';
+const DB_DIR = path.join(os.homedir(), '.copilot-tracer');
+const DB_PATH = path.join(DB_DIR, 'traces.db');
+if (!fs.existsSync(DB_DIR))
+    fs.mkdirSync(DB_DIR, { recursive: true });
+const db = new Database(DB_PATH);
+db.exec(`
+  CREATE TABLE IF NOT EXISTS sessions (
+    id TEXT PRIMARY KEY,
+    started_at TEXT NOT NULL,
+    ended_at TEXT
+  );
+
+  CREATE TABLE IF NOT EXISTS traces (
+    id TEXT PRIMARY KEY,
+    session_id TEXT NOT NULL,
+    date_time TEXT NOT NULL,
+    prompt TEXT NOT NULL,
+    response TEXT,
+    reasoning TEXT,
+    tokens_input INTEGER DEFAULT 0,
+    tokens_output INTEGER DEFAULT 0,
+    tokens_cached INTEGER DEFAULT 0,
+    tokens_reasoning INTEGER DEFAULT 0,
+    tokens_written INTEGER DEFAULT 0,
+    tokens_total INTEGER DEFAULT 0,
+    ai_credits REAL DEFAULT 0,
+    duration_ms INTEGER DEFAULT 0,
+    tool_calls TEXT DEFAULT '[]',
+    skill_count INTEGER DEFAULT 0,
+    agent_count INTEGER DEFAULT 0,
+    mcp_count INTEGER DEFAULT 0,
+    status TEXT DEFAULT 'running',
+    error TEXT,
+    FOREIGN KEY(session_id) REFERENCES sessions(id)
+  );
+`);
+export function createSession(id) {
+    db.prepare('INSERT OR REPLACE INTO sessions (id, started_at) VALUES (?, ?)').run(id, new Date().toISOString());
+}
+export function upsertTrace(entry) {
+    db.prepare(`
+    INSERT OR REPLACE INTO traces (
+      id, session_id, date_time, prompt, response, reasoning,
+      tokens_input, tokens_output, tokens_cached, tokens_reasoning, tokens_written, tokens_total,
+      ai_credits, duration_ms, tool_calls,
+      skill_count, agent_count, mcp_count, status, error
+    ) VALUES (
+      @id, @sessionId, @dateTime, @prompt, @response, @reasoning,
+      @tokensInput, @tokensOutput, @tokensCached, @tokensReasoning, @tokensWritten, @tokensTotal,
+      @aiCredits, @durationMs, @toolCalls,
+      @skillCount, @agentCount, @mcpCount, @status, @error
+    )
+  `).run({
+        id: entry.id,
+        sessionId: entry.sessionId,
+        dateTime: entry.dateTime,
+        prompt: entry.prompt,
+        response: entry.response ?? null,
+        reasoning: entry.reasoning ?? null,
+        tokensInput: entry.tokens.input,
+        tokensOutput: entry.tokens.output,
+        tokensCached: entry.tokens.cached,
+        tokensReasoning: entry.tokens.reasoning,
+        tokensWritten: entry.tokens.written,
+        tokensTotal: entry.tokens.total,
+        aiCredits: entry.aiCredits,
+        durationMs: entry.durationMs,
+        toolCalls: JSON.stringify(entry.toolCalls),
+        skillCount: entry.skillCount,
+        agentCount: entry.agentCount,
+        mcpCount: entry.mcpCount,
+        status: entry.status,
+        error: entry.error ?? null,
+    });
+}
+export function getTraces(sessionId, limit = 100) {
+    const rows = sessionId
+        ? db.prepare('SELECT * FROM traces WHERE session_id = ? ORDER BY date_time DESC LIMIT ?').all(sessionId, limit)
+        : db.prepare('SELECT * FROM traces ORDER BY date_time DESC LIMIT ?').all(limit);
+    return rows.map((r) => rowToEntry(r));
+}
+export function getTrace(id) {
+    const row = db.prepare('SELECT * FROM traces WHERE id = ?').get(id);
+    return row ? rowToEntry(row) : null;
+}
+export function getSessionSummary(sessionId) {
+    const session = db.prepare('SELECT * FROM sessions WHERE id = ?').get(sessionId);
+    if (!session)
+        return null;
+    const stats = db.prepare(`
+    SELECT
+      COUNT(*) as entries,
+      SUM(tokens_input) as input,
+      SUM(tokens_output) as output,
+      SUM(tokens_cached) as cached,
+      SUM(tokens_reasoning) as reasoning,
+      SUM(tokens_written) as written,
+      SUM(tokens_total) as total,
+      SUM(ai_credits) as credits,
+      SUM(duration_ms) as duration,
+      SUM(skill_count) as skills,
+      SUM(agent_count) as agents,
+      SUM(mcp_count) as mcps
+    FROM traces WHERE session_id = ?
+  `).get(sessionId);
+    const tokens = {
+        input: stats.input || 0,
+        output: stats.output || 0,
+        cached: stats.cached || 0,
+        reasoning: stats.reasoning || 0,
+        written: stats.written || 0,
+        total: stats.total || 0,
+    };
+    return {
+        sessionId,
+        startedAt: session.started_at,
+        totalEntries: stats.entries || 0,
+        totalTokens: tokens,
+        totalCredits: stats.credits || 0,
+        totalDurationMs: stats.duration || 0,
+        totalSkillCalls: stats.skills || 0,
+        totalAgentCalls: stats.agents || 0,
+        totalMcpCalls: stats.mcps || 0,
+    };
+}
+function rowToEntry(row) {
+    return {
+        id: row.id,
+        sessionId: row.session_id,
+        dateTime: row.date_time,
+        prompt: row.prompt,
+        response: row.response,
+        reasoning: row.reasoning,
+        tokens: {
+            input: row.tokens_input || 0,
+            output: row.tokens_output || 0,
+            cached: row.tokens_cached || 0,
+            reasoning: row.tokens_reasoning || 0,
+            written: row.tokens_written || 0,
+            total: row.tokens_total || 0,
+        },
+        aiCredits: row.ai_credits || 0,
+        durationMs: row.duration_ms || 0,
+        toolCalls: JSON.parse(row.tool_calls || '[]'),
+        skillCount: row.skill_count || 0,
+        agentCount: row.agent_count || 0,
+        mcpCount: row.mcp_count || 0,
+        status: row.status,
+        error: row.error,
+    };
+}
+export function deleteTrace(id) {
+    db.prepare('DELETE FROM traces WHERE id = ?').run(id);
+}