talk-to-copilot 1.0.0

package/README.md

@@ -0,0 +1,156 @@
+# talk-to-copilot
+
+A transparent PTY wrapper for [GitHub Copilot CLI](https://github.com/github/copilot-cli) that adds **voice input** and **screenshot attachment** — without changing how you use Copilot at all.
+
+Run `ttc` instead of `copilot`. Everything works identically, plus two new hotkeys.
+
+```
+Ctrl+R  →  Start / stop voice recording  (transcription injected as text)
+Ctrl+P  →  Interactive screenshot picker  (injected as @/path/to/file.png)
+```
+
+---
+
+## Installation
+
+### Homebrew (recommended — installs ffmpeg + whisper-cpp automatically)
+
+```bash
+brew tap Errr0rr404/ttc
+brew install ttc
+whisper-cpp-download-ggml-model base.en   # one-time: download speech model
+ttc --setup                               # verify everything is ready
+```
+
+### npm
+
+```bash
+npm install -g talk-to-copilot
+# You still need ffmpeg and whisper-cpp:
+brew install ffmpeg whisper-cpp
+whisper-cpp-download-ggml-model base.en
+ttc --setup
+```
+
+---
+
+## How it works
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  ttc (PTY wrapper)                                      │
+│                                                          │
+│  stdin ──► intercept Ctrl+R / Ctrl+P                    │
+│              │                           │               │
+│              ▼                           ▼               │
+│         voice recorder            screencapture -i       │
+│         ffmpeg + whisper-cli      saves PNG to /tmp      │
+│              │                           │               │
+│              └──────────┬────────────────┘               │
+│                         ▼                                │
+│                  inject text / @path                     │
+│                         │                                │
+│  copilot (PTY child) ◄──┘  (all other keystrokes pass   │
+│                             through unchanged)           │
+└─────────────────────────────────────────────────────────┘
+```
+
+Transcriptions are injected as raw text — **no Enter is pressed automatically** so you can review and edit before sending. Screenshots are injected as `@/tmp/copilot-screenshots/screenshot-<ts>.png` which Copilot CLI's `@` file-mention picks up.
+
+---
+
+## Prerequisites
+
+| Tool | Install |
+|------|---------|
+| [GitHub Copilot CLI](https://github.com/github/copilot-cli) | see their docs |
+| [ffmpeg](https://ffmpeg.org) | `brew install ffmpeg` |
+| [whisper.cpp](https://github.com/ggerganov/whisper.cpp) | `brew install whisper-cpp` |
+| A whisper model | `whisper-cpp-download-ggml-model base.en` |
+| Node.js ≥ 18 | `brew install node` |
+
+> **Apple Silicon note:** The `base.en` model runs in ~1–2 s on M1/M2/M3. Use `small.en` for better accuracy at ~3–4 s.
+
+---
+
+## Installation
+
+```bash
+git clone https://github.com/yourname/talk-to-copilot
+cd talk-to-copilot
+npm install
+npm link          # makes `ttc` available system-wide
+```
+
+Verify everything is wired up:
+
+```bash
+talk --setup
+```
+
+---
+
+## Usage
+
+```bash
+talk              # drop-in replacement for `copilot`
+talk --setup      # check dependencies and show config
+```
+
+Any flags you pass are forwarded to `copilot` directly:
+
+```bash
+talk --experimental
+talk --banner
+```
+
+### Voice recording
+
+1. Press **Ctrl+R** — the terminal title changes to `🎙 Recording…` and a macOS notification appears.
+2. Speak your prompt.
+3. Press **Ctrl+R** again — transcription begins (`⏳ Transcribing…`).
+4. The transcribed text appears in the Copilot input. Review it, then press **Enter** to send.
+5. Press **Ctrl+C** while recording to cancel without transcribing.
+
+### Screenshot
+
+1. Press **Ctrl+P** — the macOS screenshot overlay appears (same as ⌘⇧4).
+2. Draw a selection around the area you want to share.
+3. The path is injected as `@/tmp/copilot-screenshots/screenshot-<ts>.png`.
+4. Type any additional context, then press **Enter**.
+
+---
+
+## Configuration
+
+Config is stored at `~/.copilot/talk-to-copilot.json`:
+
+```json
+{
+  "modelPath": "/opt/homebrew/share/whisper.cpp/models/ggml-base.en.bin",
+  "audioDevice": ":0",
+  "autoSubmit": false
+}
+```
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `modelPath` | auto-detected | Path to your `.bin` whisper model |
+| `audioDevice` | `:0` | ffmpeg avfoundation mic index (run `ffmpeg -f avfoundation -list_devices true -i ""` to list) |
+| `autoSubmit` | `false` | Set to `true` to auto-press Enter after transcription |
+
+---
+
+## Troubleshooting
+
+**`Error: could not open input device`**  
+Grant microphone access: *System Settings → Privacy & Security → Microphone → Terminal*.
+
+**`No whisper model found`**  
+Run `whisper-cpp-download-ggml-model base.en`, then `talk --setup` to verify.
+
+**Transcription is empty or garbled**  
+Try a larger model: `whisper-cpp-download-ggml-model small.en`, then update `modelPath` in your config.
+
+**Wrong microphone is used**  
+Run `ffmpeg -f avfoundation -list_devices true -i ""` and set `audioDevice` in the config (e.g. `":1"`).

package/bin/ttc

@@ -0,0 +1,96 @@
+#!/usr/bin/env node
+'use strict';
+
+const { execFile } = require('child_process');
+const fs = require('fs');
+const cfg = require('../src/config');
+const CopilotWrapper = require('../src/wrapper');
+
+// ─── Setup mode ────────────────────────────────────────────────────────────────
+if (process.argv.includes('--setup')) {
+  runSetup(); // process exits inside runSetup via setTimeout
+  return;
+}
+
+// ─── Normal mode ───────────────────────────────────────────────────────────────
+const config = cfg.load();
+const args = process.argv.slice(2);
+
+const wrapper = new CopilotWrapper(args, config);
+wrapper.start();
+
+// ─── Setup helper ──────────────────────────────────────────────────────────────
+function runSetup() {
+  const config = cfg.load();
+
+  console.log('\n🛠  talk-to-copilot setup\n');
+  console.log('Checking dependencies…\n');
+
+  let allGood = true;
+
+  // Check copilot
+  execFile('which', ['copilot'], (err) => {
+    if (err) {
+      console.log('❌  copilot   — not found in PATH');
+      console.log('     Install: https://github.com/github/copilot-cli\n');
+      allGood = false;
+    } else {
+      console.log('✅  copilot   — found');
+    }
+  });
+
+  // Check ffmpeg
+  execFile('which', ['ffmpeg'], (err) => {
+    if (err) {
+      console.log('❌  ffmpeg    — not found');
+      console.log('     Install: brew install ffmpeg\n');
+      allGood = false;
+    } else {
+      console.log('✅  ffmpeg    — found');
+    }
+  });
+
+  // Check whisper-cli
+  execFile('which', ['whisper-cli'], (err) => {
+    if (err) {
+      console.log('❌  whisper-cli — not found');
+      console.log('     Install: brew install whisper-cpp');
+      console.log('     Model:   whisper-cpp-download-ggml-model base.en\n');
+      allGood = false;
+    } else {
+      console.log('✅  whisper-cli — found');
+
+      const model = cfg.findWhisperModel();
+      if (model) {
+        console.log(`✅  model        — ${model}`);
+      } else {
+        console.log('❌  model        — no model file found');
+        console.log('     Run: whisper-cpp-download-ggml-model base.en\n');
+        allGood = false;
+      }
+    }
+  });
+
+  // Print config & hotkey summary after a short delay (so async checks print first)
+  setTimeout(() => {
+    console.log('\n─────────────────────────────────────');
+    console.log('Config:', cfg.CONFIG_PATH);
+    console.log(`  modelPath:   ${config.modelPath || '(not set)'}`);
+    console.log(`  audioDevice: ${config.audioDevice}   (avfoundation mic index)`);
+    console.log(`  autoSubmit:  ${config.autoSubmit}   (auto-press Enter after transcription)`);
+    console.log('\nHotkeys (inside talk):');
+    console.log('  Ctrl+R  →  Start / stop voice recording');
+    console.log('  Ctrl+P  →  Take a screenshot (injected as @path)');
+    console.log('\nUsage:');
+    console.log('  talk              →  launch copilot with voice + screenshot support');
+    console.log('  talk --setup      →  show this screen');
+    console.log('  talk --help       →  pass --help through to copilot\n');
+
+    if (allGood) {
+      console.log('✅  All dependencies found. Run `talk` to start!\n');
+    } else {
+      console.log('⚠️   Fix the issues above, then re-run `talk --setup`.\n');
+    }
+    process.exit(0);
+  }, 400);
+}

package/package.json

@@ -0,0 +1,43 @@
+{
+  "name": "talk-to-copilot",
+  "version": "1.0.0",
+  "description": "Voice + screenshot input wrapper for GitHub Copilot CLI — use your mic and screen instead of typing",
+  "bin": {
+    "ttc": "bin/ttc"
+  },
+  "scripts": {
+    "setup": "node bin/ttc --setup",
+    "postinstall": "node scripts/postinstall.js"
+  },
+  "files": [
+    "bin/",
+    "src/",
+    "scripts/"
+  ],
+  "keywords": [
+    "copilot",
+    "github-copilot",
+    "voice",
+    "speech",
+    "whisper",
+    "cli",
+    "ai",
+    "screenshot"
+  ],
+  "author": "Errr0rr404",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Errr0rr404/talk-to-copilot.git"
+  },
+  "homepage": "https://github.com/Errr0rr404/talk-to-copilot#readme",
+  "bugs": {
+    "url": "https://github.com/Errr0rr404/talk-to-copilot/issues"
+  },
+  "dependencies": {
+    "node-pty": "^1.0.0"
+  },
+  "engines": {
+    "node": ">=18"
+  }
+}

package/scripts/postinstall.js

@@ -0,0 +1,32 @@
+'use strict';
+
+// Automatically fix node-pty spawn-helper permissions after npm install.
+// node-pty ships prebuilt binaries without the executable bit set on macOS,
+// which causes "posix_spawnp failed" at runtime without this fix.
+
+const fs = require('fs');
+const path = require('path');
+const os = require('os');
+
+if (os.platform() !== 'darwin' && os.platform() !== 'linux') process.exit(0);
+
+const prebuildDir = path.join(__dirname, '..', 'node_modules', 'node-pty', 'prebuilds');
+if (!fs.existsSync(prebuildDir)) process.exit(0);
+
+const platform = `${os.platform()}-${os.arch()}`;
+const targets = [
+  path.join(prebuildDir, platform, 'spawn-helper'),
+  path.join(prebuildDir, platform, 'pty.node'),
+];
+
+let fixed = 0;
+for (const t of targets) {
+  if (fs.existsSync(t)) {
+    fs.chmodSync(t, 0o755);
+    fixed++;
+  }
+}
+
+if (fixed > 0) {
+  console.log(`[talk-to-copilot] Fixed node-pty permissions for ${platform} (${fixed} files)`);
+}

package/src/config.js

@@ -0,0 +1,47 @@
+'use strict';
+
+const fs = require('fs');
+const path = require('path');
+const os = require('os');
+
+const CONFIG_PATH = path.join(os.homedir(), '.copilot', 'talk-to-copilot.json');
+
+const WHISPER_MODEL_CANDIDATES = [
+  path.join(os.homedir(), '.copilot', 'whisper-model.bin'),
+  path.join(__dirname, '..', 'models', 'ggml-base.en.bin'),
+  path.join(__dirname, '..', 'models', 'ggml-small.en.bin'),
+  path.join(__dirname, '..', 'models', 'ggml-tiny.en.bin'),
+  '/opt/homebrew/share/whisper.cpp/models/ggml-base.en.bin',
+  '/opt/homebrew/share/whisper.cpp/models/ggml-small.en.bin',
+  '/opt/homebrew/share/whisper.cpp/models/ggml-tiny.en.bin',
+  '/usr/local/share/whisper.cpp/models/ggml-base.en.bin',
+];
+
+function findWhisperModel() {
+  return WHISPER_MODEL_CANDIDATES.find(p => fs.existsSync(p)) || null;
+}
+
+function load() {
+  const defaults = {
+    modelPath: findWhisperModel(),
+    audioDevice: ':0',       // avfoundation default mic
+    autoSubmit: false,       // whether to press Enter after injecting transcription
+    recordKey: 'ctrl+r',
+    screenshotKey: 'ctrl+p',
+  };
+
+  if (!fs.existsSync(CONFIG_PATH)) return defaults;
+
+  try {
+    return Object.assign(defaults, JSON.parse(fs.readFileSync(CONFIG_PATH, 'utf8')));
+  } catch {
+    return defaults;
+  }
+}
+
+function save(config) {
+  fs.mkdirSync(path.dirname(CONFIG_PATH), { recursive: true });
+  fs.writeFileSync(CONFIG_PATH, JSON.stringify(config, null, 2));
+}
+
+module.exports = { load, save, findWhisperModel, CONFIG_PATH };

package/src/screenshot.js

@@ -0,0 +1,40 @@
+'use strict';
+
+const { spawn } = require('child_process');
+const fs = require('fs');
+const path = require('path');
+const os = require('os');
+
+const SCREENSHOTS_DIR = path.join(os.tmpdir(), 'copilot-screenshots');
+
+fs.mkdirSync(SCREENSHOTS_DIR, { recursive: true });
+
+/**
+ * Launch the macOS interactive screenshot picker.
+ * Resolves to the saved file path, or null if the user cancelled.
+ * @returns {Promise<string|null>}
+ */
+function capture() {
+  const filePath = path.join(SCREENSHOTS_DIR, `screenshot-${Date.now()}.png`);
+
+  return new Promise((resolve, reject) => {
+    const proc = spawn('screencapture', [
+      '-i',        // interactive selection
+      '-x',        // no shutter sound
+      filePath,
+    ]);
+
+    proc.on('error', reject);
+
+    proc.on('exit', code => {
+      // screencapture exits 0 even on ESC but doesn't create the file
+      if (code === 0 && fs.existsSync(filePath)) {
+        resolve(filePath);
+      } else {
+        resolve(null);
+      }
+    });
+  });
+}
+
+module.exports = { capture };

package/src/voice.js

@@ -0,0 +1,122 @@
+'use strict';
+
+const { spawn, execFile } = require('child_process');
+const fs = require('fs');
+const path = require('path');
+const os = require('os');
+
+class VoiceRecorder {
+  constructor(config) {
+    this.config = config;
+    this._proc = null;
+    this._audioFile = null;
+  }
+
+  get isRecording() {
+    return this._proc !== null;
+  }
+
+  /** Start recording from the microphone. Returns immediately. */
+  start() {
+    if (this._proc) return;
+
+    this._audioFile = path.join(os.tmpdir(), `copilot-voice-${Date.now()}.wav`);
+
+    this._proc = spawn('ffmpeg', [
+      '-f', 'avfoundation',
+      '-i', this.config.audioDevice,
+      '-ar', '16000',    // 16kHz — whisper requirement
+      '-ac', '1',        // mono
+      '-y',
+      this._audioFile,
+    ], {
+      stdio: ['pipe', 'ignore', 'ignore'],
+    });
+
+    this._proc.on('error', err => {
+      this._cleanup();
+      throw err;
+    });
+  }
+
+  /**
+   * Stop recording and transcribe. Returns the transcribed text, or '' if nothing was heard.
+   * @returns {Promise<string>}
+   */
+  async stopAndTranscribe() {
+    if (!this._proc) return '';
+
+    const audioFile = this._audioFile;
+    const proc = this._proc;
+    this._proc = null;
+    this._audioFile = null;
+
+    // Ask ffmpeg to stop gracefully; it finalises the WAV header before exit
+    await new Promise((resolve, reject) => {
+      proc.stdin.write('q');
+      proc.stdin.end();
+      const timer = setTimeout(() => { proc.kill('SIGTERM'); }, 3000);
+      proc.on('exit', () => { clearTimeout(timer); resolve(); });
+      proc.on('error', reject);
+    });
+
+    if (!fs.existsSync(audioFile)) {
+      throw new Error('Audio file was not created — is the microphone accessible?');
+    }
+
+    try {
+      return await this._transcribe(audioFile);
+    } finally {
+      fs.unlink(audioFile, () => {});
+    }
+  }
+
+  /** Cancel in-progress recording without transcribing. */
+  cancel() {
+    if (!this._proc) return;
+    this._proc.kill('SIGTERM');
+    this._cleanup();
+  }
+
+  _cleanup() {
+    this._proc = null;
+    if (this._audioFile) {
+      fs.unlink(this._audioFile, () => {});
+      this._audioFile = null;
+    }
+  }
+
+  /** @returns {Promise<string>} */
+  _transcribe(audioFile) {
+    const { modelPath } = this.config;
+
+    if (!modelPath) {
+      return Promise.reject(new Error(
+        'No whisper model found. Run: talk --setup'
+      ));
+    }
+
+    return new Promise((resolve, reject) => {
+      execFile('whisper-cli', [
+        '-m', modelPath,
+        '-f', audioFile,
+        '-np',   // no extra prints
+        '-nt',   // no timestamps
+      ], (err, stdout) => {
+        if (err) return reject(err);
+
+        const text = stdout
+          .split('\n')
+          .map(l => l.trim())
+          .filter(Boolean)
+          // whisper sometimes emits noise-only lines like "[BLANK_AUDIO]"
+          .filter(l => !l.startsWith('[') || !l.endsWith(']'))
+          .join(' ');
+
+        resolve(text);
+      });
+    });
+  }
+}
+
+module.exports = VoiceRecorder;

package/src/wrapper.js

@@ -0,0 +1,178 @@
+'use strict';
+
+const pty = require('node-pty');
+const { execFile, execFileSync } = require('child_process');
+
+const config = require('./config');
+const VoiceRecorder = require('./voice');
+const screenshot = require('./screenshot');
+
+// Byte sequences we intercept before forwarding to copilot
+const CTRL_R = '\x12';  // voice toggle
+const CTRL_P = '\x10';  // screenshot
+const CTRL_C = '\x03';
+
+/** Resolve the absolute path to a binary so node-pty's posix_spawnp can find it. */
+function resolveBin(name) {
+  try {
+    return execFileSync('which', [name], { encoding: 'utf8' }).trim();
+  } catch {
+    return name; // fall back to letting PATH sort it out
+  }
+}
+
+class CopilotWrapper {
+  constructor(args, cfg) {
+    this.args = args;
+    this.cfg = cfg;
+    this.voice = new VoiceRecorder(cfg);
+    this._busy = false; // prevent overlapping async operations
+  }
+
+  start() {
+    const shell = pty.spawn(resolveBin('copilot'), this.args, {
+      name: process.env.TERM || 'xterm-256color',
+      cols: process.stdout.columns || 80,
+      rows: process.stdout.rows || 24,
+      cwd: process.cwd(),
+      env: process.env,
+    });
+
+    this._shell = shell;
+
+    // PTY → real terminal
+    shell.onData(data => process.stdout.write(data));
+
+    shell.onExit(({ exitCode }) => process.exit(exitCode));
+
+    // Resize relay
+    process.stdout.on('resize', () => {
+      try { shell.resize(process.stdout.columns, process.stdout.rows); } catch {}
+    });
+
+    // Real terminal → PTY (with hotkey interception)
+    process.stdin.setRawMode(true);
+    process.stdin.resume();
+    process.stdin.on('data', data => this._handleInput(data));
+
+    // Graceful shutdown: stop any in-progress recording
+    process.on('exit', () => { if (this.voice.isRecording) this.voice.cancel(); });
+    process.on('SIGTERM', () => process.exit(0));
+  }
+
+  _handleInput(data) {
+    const key = data.toString();
+
+    if (key === CTRL_R) {
+      if (this.voice.isRecording) {
+        this._stopVoice();
+      } else {
+        this._startVoice();
+      }
+      return;
+    }
+
+    if (key === CTRL_P) {
+      this._doScreenshot();
+      return;
+    }
+
+    // Ctrl+C while recording cancels the recording; still forward to copilot
+    if (key === CTRL_C && this.voice.isRecording) {
+      this.voice.cancel();
+      this._setTitle('copilot');
+      this._notify('🚫 Recording cancelled', '');
+    }
+
+    this._shell.write(key);
+  }
+
+  // --- Voice ---
+
+  _startVoice() {
+    if (this._busy) return;
+    try {
+      this.voice.start();
+      this._setTitle('🎙 Recording… (Ctrl+R to stop, Ctrl+C to cancel)');
+      this._notify('🎙 Recording started', 'Press Ctrl+R to stop');
+    } catch (err) {
+      this._notify('❌ Could not start recording', err.message);
+    }
+  }
+
+  _stopVoice() {
+    if (this._busy) return;
+    this._busy = true;
+    this._setTitle('⏳ Transcribing…');
+    this._notify('⏳ Transcribing…', 'Please wait');
+
+    this.voice.stopAndTranscribe()
+      .then(text => {
+        this._setTitle('copilot');
+        if (text) {
+          this._shell.write(text + (this.cfg.autoSubmit ? '\r' : ''));
+          this._notify('✅ Done', text.length > 80 ? text.slice(0, 77) + '…' : text);
+        } else {
+          this._notify('⚠️ Nothing heard', 'Try speaking more clearly');
+        }
+      })
+      .catch(err => {
+        this._setTitle('copilot');
+        this._notify('❌ Transcription failed', err.message.slice(0, 80));
+      })
+      .finally(() => { this._busy = false; });
+  }
+
+  // --- Screenshot ---
+
+  _doScreenshot() {
+    if (this._busy) return;
+    this._busy = true;
+    this._setTitle('📸 Select area…');
+    this._notify('📸 Screenshot', 'Draw to select · Esc to cancel');
+
+    screenshot.capture()
+      .then(filePath => {
+        this._setTitle('copilot');
+        if (filePath) {
+          // Inject @path so the user can see it and optionally add a prompt before sending
+          this._shell.write(`@${filePath} `);
+          this._notify('✅ Screenshot attached', filePath);
+          // Force copilot TUI to repaint after screencapture overlay closes
+          this._nudgeResize();
+        } else {
+          this._notify('📸 Cancelled', '');
+        }
+      })
+      .catch(err => {
+        this._setTitle('copilot');
+        this._notify('❌ Screenshot failed', err.message.slice(0, 80));
+      })
+      .finally(() => { this._busy = false; });
+  }
+
+  // --- Helpers ---
+
+  /** Flicker the PTY size by ±1 to trigger a TUI repaint. */
+  _nudgeResize() {
+    const cols = process.stdout.columns || 80;
+    const rows = process.stdout.rows || 24;
+    try {
+      this._shell.resize(cols, rows + 1);
+      setTimeout(() => { try { this._shell.resize(cols, rows); } catch {} }, 60);
+    } catch {}
+  }
+
+  _setTitle(title) {
+    process.stdout.write(`\x1b]0;${title}\x07`);
+  }
+
+  _notify(title, subtitle) {
+    execFile('osascript', [
+      '-e',
+      `display notification ${JSON.stringify(subtitle)} with title ${JSON.stringify(title)}`,
+    ]).unref();
+  }
+}
+
+module.exports = CopilotWrapper;