jumpstart-mode 1.1.13 → 1.1.14

package/.github/skills/automate-this/SKILL.md

@@ -0,0 +1,309 @@
+---
+name: automate-this
+description: 'Analyze a screen recording of a manual process and produce targeted, working automation scripts. Extracts frames and audio narration from video files, reconstructs the step-by-step workflow, and proposes automation at multiple complexity levels using tools already installed on the user machine. Use this skill whenever the user mentions automating a repetitive task, has a screen recording of a manual process, wants to script away tedious work, asks to turn a workflow into a script, or describes a process they repeat daily/weekly and want to eliminate. Also use when the user says things like "automate this", "script this", "I do this every week", "make this a one-click thing", or "I recorded myself doing X".'
+---
+
+# Automate This
+
+Analyze a screen recording of a manual process and build working automation for it.
+
+The user records themselves doing something repetitive or tedious, hands you the video file, and you figure out what they're doing, why, and how to script it away.
+
+## Bundled Resources
+
+This skill includes reference material, templates, and examples. Load them as needed:
+
+- **`references/video-extraction.md`** — Frame extraction strategies, codec handling, Whisper model selection, cross-platform ffmpeg commands. Read when processing any video file.
+- **`references/environment-detection.md`** — OS fingerprinting, tool inventories for Windows/macOS/Linux, automation stack selection. Read during Phase 3.
+- **`references/automation-patterns.md`** — Ready-to-use code patterns by application type (browser, spreadsheet, email, file management, terminal, cross-app). Read during Phase 4 to select the right approach.
+- **`references/security-credentials.md`** — Credential handling by platform, secure temp files, dry-run patterns, input validation. Read when writing any automation script that touches auth or secrets.
+- **`templates/process-reconstruction.md`** — Output format for Phase 2 reconstruction. Use as a structural guide.
+- **`templates/automation-proposal.md`** — Output format for Phase 4 tier proposals including ROI calculation. Use as a structural guide.
+- **`templates/environment-fingerprint.md`** — Output format for Phase 3 environment detection.
+- **`examples/automation-examples.md`** — Four complete worked examples showing reconstruction and proposals for different workflow types.
+
+## Prerequisites Check
+
+Before analyzing any recording, verify the required tools are available. Adapt checks to the user's OS:
+
+**macOS/Linux:**
+```bash
+command -v ffmpeg >/dev/null 2>&1 && ffmpeg -version 2>/dev/null | head -1 || echo "NO_FFMPEG"
+command -v whisper >/dev/null 2>&1 || command -v whisper-cpp >/dev/null 2>&1 || echo "NO_WHISPER"
+```
+
+**Windows (PowerShell):**
+```powershell
+if (Get-Command ffmpeg -ErrorAction SilentlyContinue) { ffmpeg -version 2>&1 | Select-Object -First 1 } else { "NO_FFMPEG" }
+if ((Get-Command whisper -ErrorAction SilentlyContinue) -or (Get-Command whisper-cpp -ErrorAction SilentlyContinue)) { "whisper: yes" } else { "NO_WHISPER" }
+```
+
+- **ffmpeg is required.** If missing, tell the user: `brew install ffmpeg` (macOS), `winget install ffmpeg` or `choco install ffmpeg` (Windows), or `sudo apt install ffmpeg` (Ubuntu/Debian).
+- **Whisper is optional.** Only needed if the recording has narration. If missing AND the recording has an audio track, suggest: `pip install openai-whisper` or `brew install whisper-cpp`. If the user declines, proceed with visual analysis only.
+
+## Phase 1: Extract Content from the Recording
+
+Given a video file path (typically on `~/Desktop/`), extract both visual frames and audio:
+
+### Frame Extraction
+
+Extract frames at one frame every 2 seconds. This balances coverage with context window limits. See `references/video-extraction.md` for detailed guidance on frame rates, quality settings, and format handling.
+
+**macOS/Linux:**
+```bash
+WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/automate-this-XXXXXX")
+chmod 700 "$WORK_DIR"
+mkdir -p "$WORK_DIR/frames"
+ffmpeg -y -i "<VIDEO_PATH>" -vf "fps=0.5" -q:v 2 -loglevel warning "$WORK_DIR/frames/frame_%04d.jpg"
+ls "$WORK_DIR/frames/" | wc -l
+```
+
+**Windows (PowerShell):**
+```powershell
+$WorkDir = Join-Path $env:TEMP "automate-this-$(Get-Random)"
+New-Item -ItemType Directory -Path "$WorkDir\frames" -Force | Out-Null
+ffmpeg -y -i "<VIDEO_PATH>" -vf "fps=0.5" -q:v 2 -loglevel warning "$WorkDir\frames\frame_%04d.jpg"
+(Get-ChildItem "$WorkDir\frames").Count
+```
+
+Use `$WORK_DIR` / `$WorkDir` for all subsequent temp file paths in the session. The per-run directory with restrictive permissions ensures extracted frames are only readable by the current user.
+
+If the recording is longer than 5 minutes (more than 150 frames), increase the interval to one frame every 4 seconds to stay within context limits. Tell the user you're sampling less frequently for longer recordings.
+
+### Audio Extraction and Transcription
+
+Check if the video has an audio track:
+
+```bash
+ffprobe -i "<VIDEO_PATH>" -show_streams -select_streams a -loglevel error | head -5
+```
+
+If audio exists:
+
+```bash
+ffmpeg -y -i "<VIDEO_PATH>" -ac 1 -ar 16000 -loglevel warning "$WORK_DIR/audio.wav"
+
+# Use whichever whisper binary is available
+if command -v whisper >/dev/null 2>&1; then
+  whisper "$WORK_DIR/audio.wav" --model small --language en --output_format txt --output_dir "$WORK_DIR/"
+  cat "$WORK_DIR/audio.txt"
+elif command -v whisper-cpp >/dev/null 2>&1; then
+  whisper-cpp -m "$(brew --prefix 2>/dev/null)/share/whisper-cpp/models/ggml-small.bin" -l en -f "$WORK_DIR/audio.wav" -otxt -of "$WORK_DIR/audio"
+  cat "$WORK_DIR/audio.txt"
+else
+  echo "NO_WHISPER"
+fi
+```
+
+If neither whisper binary is available and the recording has audio, inform the user they're missing narration context and ask if they want to install Whisper (`pip install openai-whisper` or `brew install whisper-cpp`) or proceed with visual-only analysis.
+
+## Phase 2: Reconstruct the Process
+
+Analyze the extracted frames (and transcript, if available) to build a structured understanding of what the user did. See `templates/process-reconstruction.md` for the output format and `examples/automation-examples.md` for worked examples. Work through the frames sequentially and identify:
+
+1. **Applications used** — Which apps appear in the recording? (browser, terminal, Finder, mail client, spreadsheet, IDE, etc.)
+2. **Sequence of actions** — What did the user do, in order? Click-by-click, step-by-step.
+3. **Data flow** — What information moved between steps? (copied text, downloaded files, form inputs, etc.)
+4. **Decision points** — Were there moments where the user paused, checked something, or made a choice?
+5. **Repetition patterns** — Did the user do the same thing multiple times with different inputs?
+6. **Pain points** — Where did the process look slow, error-prone, or tedious? The narration often reveals this directly ("I hate this part," "this always takes forever," "I have to do this for every single one").
+
+Present this reconstruction to the user as a numbered step list and ask them to confirm it's accurate before proposing automation. This is critical — a wrong understanding leads to useless automation.
+
+Format:
+
+```
+Here's what I see you doing in this recording:
+
+1. Open Chrome and navigate to [specific URL]
+2. Log in with credentials
+3. Click through to the reporting dashboard
+4. Download a CSV export
+5. Open the CSV in Excel
+6. Filter rows where column B is "pending"
+7. Copy those rows into a new spreadsheet
+8. Email the new spreadsheet to [recipient]
+
+You repeated steps 3-8 three times for different report types.
+
+[If narration was present]: You mentioned that the export step is the slowest
+part and that you do this every Monday morning.
+
+Does this match what you were doing? Anything I got wrong or missed?
+```
+
+Do NOT proceed to Phase 3 until the user confirms the reconstruction is accurate.
+
+## Phase 3: Environment Fingerprint
+
+Before proposing automation, understand what the user actually has to work with. Adapt these checks to the detected OS. See `references/environment-detection.md` for full platform-specific tool inventories and the decision matrix for choosing automation approaches.
+
+**macOS/Linux:**
+```bash
+echo "=== OS ===" && uname -a
+echo "=== Shell ===" && echo $SHELL
+echo "=== Python ===" && { command -v python3 && python3 --version 2>&1; } || echo "not installed"
+echo "=== Node ===" && { command -v node && node --version 2>&1; } || echo "not installed"
+echo "=== Homebrew ===" && { command -v brew && echo "installed"; } || echo "not installed"
+echo "=== Common Tools ===" && for cmd in curl jq playwright selenium osascript automator crontab; do command -v $cmd >/dev/null 2>&1 && echo "$cmd: yes" || echo "$cmd: no"; done
+```
+
+**Windows (PowerShell):**
+```powershell
+Write-Output "=== OS ==="; [System.Environment]::OSVersion.VersionString
+Write-Output "=== PowerShell ==="; $PSVersionTable.PSVersion.ToString()
+Write-Output "=== Python ==="; try { python --version 2>&1 } catch { "not installed" }
+Write-Output "=== Node ==="; try { node --version 2>&1 } catch { "not installed" }
+Write-Output "=== Package Manager ==="
+@("winget","choco","scoop") | ForEach-Object { if (Get-Command $_ -EA SilentlyContinue) { "$_`: yes" } }
+Write-Output "=== Common Tools ==="
+@("curl","jq","git","ffmpeg","playwright") | ForEach-Object {
+    if (Get-Command $_ -EA SilentlyContinue) { "$_`: yes" } else { "$_`: no" }
+}
+```
+
+Use this to constrain proposals to tools the user already has. Never propose automation that requires installing five new things unless the simpler path genuinely doesn't work. See `templates/environment-fingerprint.md` for the output format.
+
+## Phase 4: Propose Automation
+
+Based on the reconstructed process and the user's environment, propose automation at up to three tiers. Not every process needs three tiers — use judgment. See `references/automation-patterns.md` for ready-to-use code patterns by application type, and `templates/automation-proposal.md` for the proposal output format including ROI calculations. For credential handling, consult `references/security-credentials.md`.
+
+### Tier Structure
+
+**Tier 1 — Quick Win (under 5 minutes to set up)**
+The smallest useful automation. A shell alias, a one-liner, a keyboard shortcut, an AppleScript snippet. Automates the single most painful step, not the whole process.
+
+**Tier 2 — Script (under 30 minutes to set up)**
+A standalone script (bash, Python, or Node — whichever the user has) that automates the full process end-to-end. Handles common errors. Can be run manually when needed.
+
+**Tier 3 — Full Automation (under 2 hours to set up)**
+The script from Tier 2, plus: scheduled execution (cron, launchd, or GitHub Actions), logging, error notifications, and any necessary integration scaffolding (API keys, auth tokens, etc.).
+
+### Proposal Format
+
+For each tier, provide:
+
+```
+## Tier [N]: [Name]
+
+**What it automates:** [Which steps from the reconstruction]
+**What stays manual:** [Which steps still need a human]
+**Time savings:** [Estimated time saved per run, based on the recording length and repetition count]
+**Prerequisites:** [Anything needed that isn't already installed — ideally nothing]
+
+**How it works:**
+[2-3 sentence plain-English explanation]
+
+**The code:**
+[Complete, working, commented code — not pseudocode]
+
+**How to test it:**
+[Exact steps to verify it works, starting with a dry run if possible]
+
+**How to undo:**
+[How to reverse any changes if something goes wrong]
+```
+
+### Application-Specific Automation Strategies
+
+Use these strategies based on which applications appear in the recording:
+
+**Browser-based workflows:**
+- First choice: Check if the website has a public API. API calls are 10x more reliable than browser automation. Search for API documentation.
+- Second choice: `curl` or `wget` for simple HTTP requests with known endpoints.
+- Third choice: Playwright or Selenium for workflows that require clicking through UI. Prefer Playwright — it's faster and less flaky.
+- Look for patterns: if the user is downloading the same report from a dashboard repeatedly, it's almost certainly available via API or direct URL with query parameters.
+
+**Spreadsheet and data workflows:**
+- Python with pandas for data filtering, transformation, and aggregation.
+- If the user is doing simple column operations in Excel, a 5-line Python script replaces the entire manual process.
+- `csvkit` for quick command-line CSV manipulation without writing code.
+- If the output needs to stay in Excel format, use openpyxl.
+
+**Email workflows:**
+- macOS: `osascript` can control Mail.app to send emails with attachments.
+- Cross-platform: Python `smtplib` for sending, `imaplib` for reading.
+- If the email follows a template, generate the body from a template file with variable substitution.
+
+**File management workflows:**
+- Shell scripts for move/copy/rename patterns.
+- `find` + `xargs` for batch operations.
+- `fswatch` or `watchman` for triggered-on-change automation.
+- If the user is organizing files into folders by date or type, that's a 3-line shell script.
+
+**Terminal/CLI workflows:**
+- Shell aliases for frequently typed commands.
+- Shell functions for multi-step sequences.
+- Makefiles for project-specific task sets.
+- If the user ran the same command with different arguments, that's a loop.
+
+**macOS-specific workflows:**
+- AppleScript/JXA for controlling native apps (Mail, Calendar, Finder, Preview, etc.).
+- Shortcuts.app for simple multi-app workflows that don't need code.
+- `automator` for file-based workflows.
+- `launchd` plist files for scheduled tasks (prefer over cron on macOS).
+
+**Windows-specific workflows:**
+- PowerShell for file operations, COM automation (Excel, Outlook, Word), and system tasks.
+- `pywinauto` for native Windows UI automation via UIA/Win32 accessibility.
+- `pyautogui` for cross-platform mouse/keyboard control and image-based clicking.
+- AutoHotkey for hotkeys, text expansion, and simple GUI scripting.
+- Task Scheduler for scheduled/triggered execution.
+- COM objects (`New-Object -ComObject`) for scripting Office apps without opening them visually.
+
+**Linux-specific workflows:**
+- `xdotool` for X11 window/keyboard/mouse control.
+- `wmctrl` for window management.
+- `ydotool` for Wayland-based environments.
+- `cron` or `systemd timers` for scheduled tasks.
+- `notify-send` for desktop notifications on completion.
+
+**Cross-application workflows (data moves between apps):**
+- Identify the data transfer points. Each transfer is an automation opportunity.
+- Clipboard-based transfers in the recording suggest the apps don't talk to each other — look for APIs, file-based handoffs, or direct integrations instead.
+- If the user copies from App A and pastes into App B, the automation should read from A's data source and write to B's input format directly.
+
+### Making Proposals Targeted
+
+Apply these principles to every proposal:
+
+1. **Automate the bottleneck first.** The narration and timing in the recording reveal which step is actually painful. A 30-second automation of the worst step beats a 2-hour automation of the whole process.
+
+2. **Match the user's skill level.** If the recording shows someone comfortable in a terminal, propose shell scripts. If it shows someone navigating GUIs, propose something with a simple trigger (double-click a script, run a Shortcut, or type one command).
+
+3. **Estimate real time savings.** Count the recording duration and multiply by how often they do it. "This recording is 4 minutes. You said you do this daily. That's 17 hours per year. Tier 1 cuts it to 30 seconds each time — you get 16 hours back."
+
+4. **Handle the 80% case.** The first version of the automation should cover the common path perfectly. Edge cases can be handled in Tier 3 or flagged for manual intervention.
+
+5. **Preserve human checkpoints.** If the recording shows the user reviewing or approving something mid-process, keep that as a manual step. Don't automate judgment calls.
+
+6. **Propose dry runs.** Every script should have a mode where it shows what it *would* do without doing it. `--dry-run` flags, preview output, or confirmation prompts before destructive actions.
+
+7. **Account for auth and secrets.** If the process involves logging in or using credentials, never hardcode them. Use environment variables, keychain access (macOS `security` command), or prompt for them at runtime.
+
+8. **Consider failure modes.** What happens if the website is down? If the file doesn't exist? If the format changes? Good proposals mention this and handle it.
+
+## Phase 5: Build and Test
+
+When the user picks a tier:
+
+1. Write the complete automation code to a file (suggest a sensible location — the user's project directory if one exists, or `~/Desktop/` otherwise).
+2. Walk through a dry run or test with the user watching.
+3. If the test works, show how to run it for real.
+4. If it fails, diagnose and fix — don't give up after one attempt.
+
+## Cleanup
+
+After analysis is complete (regardless of outcome), clean up extracted frames and audio:
+
+**macOS/Linux:**
+```bash
+rm -rf "$WORK_DIR"
+```
+
+**Windows (PowerShell):**
+```powershell
+Remove-Item -Recurse -Force $WorkDir
+```
+
+Tell the user you're cleaning up temporary files so they know nothing is left behind.

package/.github/skills/automate-this/evals/evals.json

@@ -0,0 +1,73 @@
+{
+  "skill_name": "automate-this",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "I just recorded myself doing my Monday morning routine — I download a CSV report from our internal dashboard, open it in Excel, filter out rows where the status column says 'completed', save the filtered version, and email it to my manager. The recording is at ~/Desktop/monday-report.mov. Can you automate this?",
+      "expected_output": "A structured process reconstruction confirming the workflow steps, followed by tiered automation proposals that use Python/pandas for CSV filtering and a script for email, with appropriate credential handling.",
+      "files": [],
+      "expectations": [
+        "The response presents a numbered step-by-step reconstruction of the workflow and asks the user to confirm before proposing automation.",
+        "The response proposes at least two tiers of automation with increasing completeness.",
+        "Tier 1 focuses on the single most painful step rather than automating the entire process.",
+        "Code provided is complete and runnable — not pseudocode or placeholder-heavy.",
+        "Credentials are never hardcoded; environment variables, keyring, or runtime prompts are used.",
+        "A dry-run or preview mode is included in at least one tier."
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "I recorded a 7-minute video of me renaming and sorting about 30 files in my Downloads folder into subfolders by type (PDFs go to Documents/PDFs, images to Pictures, spreadsheets to Documents/Spreadsheets). I do this every few days. The video is at C:\\Users\\me\\Desktop\\file-cleanup.mp4. I'm on Windows 11 and mostly use PowerShell. Help me automate this.",
+      "expected_output": "A reconstruction that identifies the file-sorting pattern, environment detection noting Windows/PowerShell, and proposals using PowerShell scripts with dry-run support.",
+      "files": [],
+      "expectations": [
+        "The response adapts commands and proposals for Windows (PowerShell) rather than defaulting to bash/macOS.",
+        "Environment detection uses PowerShell-appropriate commands (Get-Command, Test-Path) rather than unix commands.",
+        "The automation script includes a -WhatIf or --dry-run mode to preview changes before executing.",
+        "The proposal estimates time savings based on the stated frequency and recording duration.",
+        "File paths use Windows conventions (backslashes or proper PowerShell path handling)."
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "Here's a screen recording of me manually entering data from PDF invoices into our web-based accounting system. I open each PDF, read the vendor name, invoice number, amount, and date, then type them into form fields on the website. I do about 15 of these a day and it takes roughly 2 minutes each. Video is on my desktop: invoice-entry.mov",
+      "expected_output": "A reconstruction identifying the PDF-read → web-form-entry loop, proposals that use PDF extraction (pdfplumber or similar) combined with either API calls or browser automation, with ROI calculations showing significant time savings.",
+      "files": [],
+      "expectations": [
+        "The response identifies this as a repetitive data-entry workflow and reconstructs the loop pattern.",
+        "The proposal checks whether the accounting system has an API before defaulting to browser automation.",
+        "PDF text extraction is proposed using an appropriate library (pdfplumber, PyPDF2, or similar).",
+        "The time savings calculation is based on the stated frequency (15/day × 2 min each).",
+        "Human review is preserved as a checkpoint — the script shows extracted data for confirmation before submitting.",
+        "The response handles the case where PDF layouts vary between vendors."
+      ]
+    },
+    {
+      "id": 4,
+      "prompt": "I recorded my screen while doing a deployment. I pull from git, run tests, build the project, then scp the dist folder to our server and restart the service via ssh. Takes about 3 minutes of waiting and typing. Recording is at ~/Desktop/deploy-recording.mov. Can you make this a one-command thing?",
+      "expected_output": "A reconstruction of the deploy steps, with Tier 1 as a shell function or alias that chains the commands, and higher tiers adding error handling, rollback, and optional CI/CD suggestions.",
+      "files": [],
+      "expectations": [
+        "The response reconstructs the terminal-based workflow accurately from the description.",
+        "Tier 1 is genuinely simple — a shell function or alias, not an overengineered script.",
+        "The script uses set -e or equivalent to stop on first failure rather than continuing after errors.",
+        "At least one tier mentions or includes rollback capability (keeping the previous deployment).",
+        "The response does not propose installing complex CI/CD infrastructure for Tier 1 or Tier 2."
+      ]
+    },
+    {
+      "id": 5,
+      "prompt": "I screen-recorded myself copying data between two spreadsheets — I have a master inventory sheet and I need to update quantities from a supplier email attachment (also a spreadsheet). I open both, manually compare item codes, and update the quantities in the master. About 50 items each time, takes 20 minutes. Recording: inventory-update.mp4 on my Desktop. I'm not very technical — I mostly just use Excel and email.",
+      "expected_output": "A reconstruction acknowledging the dual-spreadsheet comparison workflow, proposals matched to a non-technical user's comfort level, with Tier 1 being as simple as possible (maybe an Excel formula or a one-click script).",
+      "files": [],
+      "expectations": [
+        "The response detects the user's comfort level from context cues ('not very technical', 'mostly use Excel') and adjusts proposal language accordingly.",
+        "Technical jargon is minimized or explained when used.",
+        "Tier 1 is extremely simple to use — ideally a double-click script or single command.",
+        "The automation uses the item code as a join/lookup key to match rows between spreadsheets.",
+        "The proposal preserves a backup of the master sheet before modifying it.",
+        "Instructions for running the automation are step-by-step and assume minimal terminal experience."
+      ]
+    }
+  ]
+}