@complexthings/superpowers-agent 9.0.0 → 9.0.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@complexthings/superpowers-agent",
-  "version": "9.0.0",
+  "version": "9.0.2",
   "description": "Superpowers agent CLI — skills system for AI coding agents",
   "type": "module",
   "bin": {
@@ -33,7 +33,7 @@
   },
   "repository": {
     "type": "git",
-    "url": "https://github.com/complexthings/superpowers.git"
+    "url": "git+https://github.com/complexthings/superpowers.git"
   },
   "bugs": {
     "url": "https://github.com/complexthings/superpowers/issues"

package/skills/collaboration/leveraging-cli-tools/SKILL.md

@@ -1,132 +1,111 @@
 ---
 name: leveraging-cli-tools
-description: Use when performing code searches, JSON parsing, file viewing, or file finding tasks - ensures agents verify and use high-performance CLI tools (rg, jq, fd, bat, ast-grep) instead of slower standard tools, reducing token costs and latency by 5-50x
+description: Use when performing code searches, JSON/YAML parsing, file finding, structural refactors, or data wrangling, or when a command floods context with verbose output like logs, CI, or test runs - ensures agents reach for high-performance CLI tools (rg, jq, fd, yq, ast-grep, gh, sd) over slower standard tools like grep/find/sed and use them with discipline, choosing the right output-reducing flags, composing pipelines so raw output never enters context, and using a bundled reducer for output that flags cannot shape, cutting token cost and latency 5-50x. Check availability and offer to install a tool when a relevant task arises.
+compatibility: The bundled scripts/slim.py needs python3 (standard on macOS/Linux). The CLI tools install via the system package manager.
 ---
 
 # Leveraging CLI Tools
 
-## Core Principle
+## Purpose
 
-**High-performance CLI tools reduce costs, latency, and cognitive load by 5-50x.**
+Reach for high-performance CLI tools over slower standard tools. The leverage is filtering and transforming with the right tool **before reading**, so tokens and time go to the answer, not the search. On a large tree `rg` is 10-50x faster than `grep` and returns far less noise; across a session that compounds into hours and tens of thousands of tokens saved.
 
-Modern tools have massive RL optimization in frontier models - use them.
+Picking the faster tool is only half of it. The other half is using it with discipline — two levers that keep raw output out of your context: **ask for less** (the right flags and selectors), and **shrink what you can't shape** (a reducer for inherently verbose output). A fast tool fed a lazy command still floods your context.
 
-## Tool Availability Protocol
+## When a relevant task arises
 
-**Session start (silent check, cache result):**
+1. Pick the right tool from the table below.
+2. Confirm it's installed before relying on it — e.g. `command -v rg`.
+3. If it's missing, name the tool and its payoff and **offer** to install it — don't install silently. Adapt the command to the user's package manager/OS (the table shows `brew`; substitute `apt install`, `dnf install`, `pacman -S`, `cargo install`, etc.). If they decline, fall back to the standard tool and move on.
 
-```bash
-command -v rg jq fd bat gh >/dev/null 2>&1 && echo "ready" || echo "missing"
-```
+Check only the tools the current task needs — no upfront session-wide scan.
 
-**If tools missing:** Offer installation when relevant task appears.
-**If tools ready:** Use optimal workflows throughout session.
+## Tools
 
-## Tool Selection
+| Rating | Tool | Replaces | Why | Install |
+|:------:|------|----------|-----|---------|
+| 10 | `rg` (ripgrep) | `grep`, `grep -r`, `ack` | Code/text search. Respects `.gitignore`, 10-50x faster than `grep` — the highest-leverage tool; filter before reading. | `brew install ripgrep` |
+| 10 | `jq` | `grep`/`sed`/`awk` on JSON | JSON query/transform. Turns API responses and config into exactly the fields you need; the pipe target for JSON. | `brew install jq` |
+| 9 | `fd` | `find` | File finding. Faster, saner syntax, parallel traversal, `.gitignore`-aware. | `brew install fd` |
+| 8 | `yq` | `grep`/`sed`/`awk` on YAML | `jq` for YAML/TOML/XML. Reads CI files, `docker-compose`, k8s manifests, frontmatter. | `brew install yq` |
+| 8 | `ast-grep` (`sg`) | `sed`/`grep` for refactors | Structural search/rewrite by AST, not regex. Safe codebase-wide refactors that `sed` would mangle. | `brew install ast-grep` |
+| 8 | `gh` | `curl` + GitHub API + tokens | GitHub from the shell — PRs, issues, CI, API. No hand-rolled `curl` + token juggling. | `brew install gh` |
+| 7 | `sd` | `sed -i`, `perl -pe` | Find/replace. Literal-string-safe, no regex-escaping footguns. | `brew install sd` |
+| 6 | `dasel` | `jq`+`yq`+`xq` (mixed formats) | Query *and modify* JSON/YAML/TOML/XML/CSV through one selector. Use when format is mixed or unknown. | `brew install dasel` |
+| 6 | `htmlq` | `grep`/`sed` on HTML | `jq` for HTML — CSS-selector extraction from fetched pages. | `brew install htmlq` |
+| 6 | `miller` (`mlr`) | `awk`/`cut`/`join`/`sort` on CSV/TSV | `awk`/`cut`/`join`/`sort` for CSV/TSV/JSON with *named* fields — no brittle column counting. | `brew install miller` |
+| 5 | `qsv` | `awk`/`cut`/`sort -u` on CSV, `csvkit` | High-perf CSV toolkit (maintained `xsv` successor). Stats, slice, join, dedup on big CSVs. | `brew install qsv` |
+| 5 | `hyperfine` | `time`, `for`-loop timing | Statistical benchmarking with warmups. Real before/after numbers, not `time` guesses. | `brew install hyperfine` |
+| 5 | `tokei` | `wc -l`, `find … \| wc`, `cloc` | Instant LOC/language breakdown. Orient in an unfamiliar repo before exploring. | `brew install tokei` |
+| 5 | `fzf` (`-f`) | manual fuzzy filtering | Non-interactive `-f`/`--filter` mode: fuzzy-rank a candidate list piped from `fd`/`rg`. | `brew install fzf` |
+| 4 | `watchexec` | `while`+`sleep`, `entr` | Run a command on file change. Useful in build/test loops; non-interactive unlike most watchers. | `brew install watchexec` |
 
-| Task | Use | Instead of | Speedup |
-|------|-----|-----------|---------|
-| Search code | **rg** | grep | 10-50x |
-| Parse JSON | **jq** | awk/sed | 5-20x |
-| Find files | **fd** | find | 5-10x |
-| View code | **bat** | cat | Better UX |
-| Transform code | **ast-grep** | sed | 3-10x |
-| GitHub ops | **gh** | curl+API | 2-5x |
-| Interactive select | **fzf** | manual | 10-100x |
+## Use them with discipline
 
-## Quick Reference
+Five habits separate a fast, low-noise result from a slow, context-flooding one. The token savings below are measured, not hypothetical.
 
-### Code Search: rg
-```bash
-rg "AuthError" --type typescript  # 10-50x faster than grep, respects .gitignore
-```
+**1 — Pick the tool that fits the data, not the one in muscle memory.** JSON → `jq`, YAML/TOML → `yq`, structural refactor → `ast-grep`, GitHub → `gh`. The reflex to resist is `grep`/`sed`/`awk` on structured data — they treat structure as flat text, so you pay in escaping bugs and noise the format-aware tool never produces.
 
-### JSON Parse: jq
-```bash
-jq '.results[] | select(.status=="error") | .error.code' api.json | sort -u
-```
+**2 — Ask for less: use the flag or selector that returns the answer's _shape_.** The biggest single win is never pulling raw output into context. Match the request to the question:
 
-### Find Files: fd
-```bash
-fd "\.test\.ts$"  # 5-10x faster than find, parallel traversal
-```
+| Question | Flag / selector | Returns |
+|----------|-----------------|---------|
+| Which files match? | `rg -l` | paths only, no lines |
+| How many files? | `rg -l \| wc -l` | a file count, not a line count |
+| How many hits? | `rg -c` (lines/file) / `rg --count-matches` (matches/file) | counts only |
+| Just the matched bit? | `rg -o` | the substring, not the whole line |
+| Enough to judge a hit? | `rg -A/-B/-C N`, `rg -m N` (cap per file) | bounded context |
+| Only certain fields? | `jq -r '.a, .b'`, `yq`, `mlr --opprint cut -f` | projected values, not the whole doc |
+| First N of a list? | pipe to `head`, or `fd --max-results N` | the slice you need |
 
-### View Code: bat
-```bash
-bat src/auth.ts  # Syntax highlighting, Git integration, line numbers
-```
+For the **projection tools** (`jq`, `yq`, `dasel`, `miller`, `htmlq`) the selector *is* the filter — `jq` turning a 2,000-record array into one summed number is a ~100% reduction on its own. Shape their output at the selector; never post-filter it (that's what habit 4 is *not* for).
 
-### Code Transform: ast-grep
+**3 — Compose so only the answer comes back.** Do the counting, dedup, and projection *in the pipeline*, not by reading raw output and reasoning over it. One pass keeps every intermediate result out of context:
 ```bash
-sg --pattern 'console.log($$$ARGS)' --rewrite 'logger.debug($$$ARGS)'
+rg -l '"error"' logs/ --type json | xargs jq -r 'select(.level=="error") | .code' | sort -u
 ```
-
-## Installation Protocol
-
-**If tool missing when needed:**
-
-1. Explain impact: "Using rg is 10-50x faster, reducing token costs"
-2. Install automatically for core tools (rg, jq, fd, bat, gh): `brew install ripgrep jq fd bat gh` (macOS) or equivalent
-3. Ask before installing: ast-grep, httpie, fzf
-
-**No exceptions:** If tool unavailable and user declines install, explain performance cost but use fallback.
-
-## Key Workflows
-
-### Filter Before Reading
+This is the lesson of programmatic tool calling (Amazon's PTC benchmarks: ~87-92% fewer tokens): when a task needs several tool calls plus processing, write **one** pipeline or script that does it all and returns only the result — the intermediate data never touches your context. So: filter before reading, and refactor by AST, not regex:
 ```bash
-# DON'T: Read all files blindly
-# DO: Filter first, read matches only
-rg "password.*hash" src/auth/ --type ts -l | xargs bat
+rg -l "password.*hash" src/auth/ --type ts | xargs rg "TODO"
+sg --pattern 'console.log($$$A)' --rewrite 'logger.debug($$$A)' --lang ts
 ```
 
-Under fatigue: filtering first reduces cognitive load.
-
-### Compose Tools
+**4 — Shrink what you can't shape: the bundled reducer for verbose emitters.** Some output has no projection flag — logs, CI output (`gh run view --log`), test runs, stack traces, status dumps. Flags get you part way, but the residual stays repetitive, wide, and noisy. Pipe it through `scripts/slim.py` (relative to this skill; use its absolute path if you're working elsewhere) and ask only for the cuts the task needs:
 ```bash
-rg "error.code" logs/ -o | jq -r . | sort -u  # Search + parse + dedup
+gh run view --log | python3 scripts/slim.py --errors --uniq --max-line 200
+cargo test 2>&1   | python3 scripts/slim.py --errors --dedup
+rg -n TODO --no-ignore | python3 scripts/slim.py --group-dir
 ```
+It does dedup (`--dedup`/`--uniq`, repeats collapse to `(xN)`), grouping (`--group-dir` → per-directory counts), truncation (`--head/--tail/--middle/--max-line`), and noise-stripping (`--comments/--errors/--grep`). On a real 1,236-line CI log, `rg` error-filtering alone cut 91% but left ~6,700 tokens; adding `slim` reached 97% — a third the residual. Run `python3 scripts/slim.py -h` for all modes. The cuts are **lossy and opt-in**: ask for what you need, know what you're discarding, and don't reach for it when a projection tool's own selector would do the job.
 
-## Performance Impact
+**5 — Read the output critically — it's a claim, not a fact.** A zero or suspiciously low result is the one to distrust: `rg` skips `.gitignore`d and hidden files by default, so a real match in `node_modules/`, `dist/`, or a dotfile is silently absent. Before concluding "none," re-run with `--no-ignore`, `-uu`, or `--hidden` — then decide whether those ignored hits (vendored deps, build output, generated mirrors) actually belong in the answer; surfacing them is the check, keeping them is a judgment call. Mind case (`-i`) and word boundaries (`-w`) so you neither miss `Subagent` nor over-match `tasks` — and when an identifier doubles as a common word (a `Task` tool vs. the word "task"), even `-w` isn't enough: glance at the context to confirm the hit is the symbol, not prose. When a count or list drives a decision, confirm a match means what you think before acting on it.
 
-| Task | grep/find/cat | rg/fd/bat | Savings |
-|------|---------------|-----------|---------|
-| Search 50k files | 45s | 0.8s | 56x, ~40k tokens |
-| Parse 10MB JSON | 12s (awk) | 2s (jq) | 6x, ~15k tokens |
-| Find in monorepo | 8s | 1s | 8x, ~10k tokens |
+In Claude Code the `Grep` and `Glob` tools are themselves built on ripgrep — prefer them for in-context searches, and reach for the CLI tools when you need piping, transforms, rewrites, or output reduction.
 
-## Red Flags
+## Red flags
 
-- Manually parsing JSON with awk/sed/grep (use jq)
-- Reading files without filtering first (use rg to filter)
-- Not checking tool availability at session start
-- Using bash grep when Grep tool exists (use Grep tool)
+- Parsing JSON/YAML with `awk`/`sed`/`grep` instead of `jq`/`yq`.
+- Reading files before filtering them with `rg`.
+- Piping a tool's full output into context to eyeball it, when `-l`/`-c`/`-o` or a `jq` projection would return just the answer.
+- Letting a verbose command (test run, CI log, `git`/`docker` status dump) land raw in context when `slim` would cut it 90%+.
+- Reaching for `slim` on a projection tool's output — shape it at the `jq`/`yq`/`mlr` selector instead.
+- Concluding "no matches" from a default `rg` run without re-checking `--no-ignore`/`--hidden` — the hit may be sitting in an ignored directory.
+- Hand-rolling `curl` against the GitHub API instead of `gh`.
+- Regex codemods with `sed` where `ast-grep` is structurally safe.
 
-## Common Rationalizations
+## Common rationalizations
 
 | Excuse | Reality |
 |--------|---------|
-| "grep works fine" | 50x slower, burns 40k tokens on 50k files |
-| "I don't know if they have jq" | Check once, install takes 30s, saves hours |
-| "Not worth setup overhead" | One install = 10-100x speedup on ALL future tasks |
-| "User didn't ask for optimization" | Faster task completion IS better completion |
-
-## When NOT to Use
-
-**Skip when:**
-- Small dataset (< 100 files, < 1MB) AND one-off task
-- User explicitly declines install
-- Teaching context where standard tools are the goal
-
-**Use when:**
-- Codebase search (always)
-- JSON operations (always)
-- Time pressure, large datasets, repeated operations
-
-## Summary
-
-**Protocol: Check → Use → Combine**
-
-Session start: Silent availability check. Task appears: Use optimal tool or install. Complex tasks: Compose tools.
-
-**Not using available tools burns resources.**
+| "grep works fine" | On a big tree it's 10-50x slower and floods context with noise `rg` would have filtered out. |
+| "I don't know if they have jq" | One `command -v jq` answers it; install is seconds and pays back across the whole session. |
+| "Not worth the setup" | One install = a speedup on every future task, not just this one. |
+| "The test/CI output is just long, I'll scroll it" | A 1,200-line log is ~75k tokens of mostly noise; `slim --errors --uniq` makes it ~2k without losing the failures. |
+| "User didn't ask for optimization" | Faster, lower-noise completion *is* better completion. |
+
+## When NOT to use
+
+- A tiny one-off (a handful of files, well under a megabyte) where the standard tool is already at hand.
+- The user declined the install — note the cost once, then use the fallback.
+- A teaching context where the standard tool is the point.
+- Output you already shaped with a selector — don't add a `slim` pass for its own sake.

package/skills/collaboration/leveraging-cli-tools/scripts/slim.py

@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+"""slim - reduce noisy command output to signal before it reaches your context.
+
+A stdin->stdout filter for output you CAN'T shape with a tool's own flags:
+logs, scan results, status dumps, stack traces, repeated lines. For tools that
+already project output (jq/yq/dasel/miller/htmlq), use their selectors instead;
+piping them through slim is redundant.
+
+Pipe a verbose command through it and ask only for the reductions you need:
+
+    cargo test 2>&1 | slim --errors --dedup
+    rg -n TODO --no-ignore | slim --group-dir
+    gh run view --log | slim --uniq --max-line 200 --middle 40
+    kubectl get events | slim --dedup --stats
+
+Reductions (compose freely):
+  --comments        drop comment-only lines (# // ; -- *) and blank lines
+  --errors          keep only lines matching error|warn|fail|fatal|panic|exception (-i)
+  --grep RE         keep only lines matching regex RE (use --invert to drop them)
+  --invert          invert --grep / --errors (drop matches instead of keeping)
+  --dedup           collapse ALL repeated lines, first-seen order, annotate "  (xN)"
+  --uniq            collapse only CONSECUTIVE repeats, annotate "  (xN)" (log-friendly)
+  --group-dir       turn a list of file paths into per-directory counts
+  --max-line N      truncate each line to N chars, append a horizontal ellipsis
+  --head N          keep first N lines
+  --tail N          keep last N lines
+  --middle N        keep first N and last N lines, replace the gap with a marker
+  --stats           print "lines A->B, bytes A->B (-P%)" to stderr (output unaffected)
+
+Always-on, lossless: trailing whitespace is stripped and runs of blank lines
+collapse to one. Everything lossy is opt-in. Order: filter -> transform ->
+dedup -> group -> blank-collapse -> truncate. Reductions are LOSSY by design;
+ask for what the task needs and know what you're discarding.
+"""
+import argparse
+import re
+import sys
+
+COMMENT_RE = re.compile(r"^\s*(#|//|;|--|\*|/\*|\*/)")
+ERROR_RE = re.compile(r"error|warn|fail|fatal|panic|exception|traceback", re.IGNORECASE)
+ELLIPSIS = "…"
+
+
+def annotate(line: str, n: int) -> str:
+    return line if n == 1 else f"{line}  (x{n})"
+
+
+def dedup_all(lines):
+    counts, order = {}, []
+    for ln in lines:
+        if ln not in counts:
+            counts[ln] = 0
+            order.append(ln)
+        counts[ln] += 1
+    return [annotate(ln, counts[ln]) for ln in order]
+
+
+def dedup_adjacent(lines):
+    out, prev, n = [], None, 0
+    for ln in lines:
+        if ln == prev:
+            n += 1
+        else:
+            if prev is not None:
+                out.append(annotate(prev, n))
+            prev, n = ln, 1
+    if prev is not None:
+        out.append(annotate(prev, n))
+    return out
+
+
+def group_dir(lines):
+    counts, order = {}, []
+    for ln in lines:
+        token = ln.split(":", 1)[0].strip()  # tolerate rg's path:line: prefix
+        d = token.rsplit("/", 1)[0] + "/" if "/" in token else "./"
+        if d not in counts:
+            counts[d] = 0
+            order.append(d)
+        counts[d] += 1
+    width = max((len(d) for d in order), default=0)
+    return [f"{d.ljust(width)}  ({counts[d]} file{'s' if counts[d] != 1 else ''})" for d in order]
+
+
+def collapse_blanks(lines):
+    out, blank = [], False
+    for ln in lines:
+        if ln.strip() == "":
+            if not blank:
+                out.append("")
+            blank = True
+        else:
+            out.append(ln)
+            blank = False
+    while out and out[-1] == "":
+        out.pop()
+    return out
+
+
+def main() -> int:
+    p = argparse.ArgumentParser(add_help=True, description="Reduce noisy stdout to signal.")
+    p.add_argument("--comments", action="store_true")
+    p.add_argument("--errors", action="store_true")
+    p.add_argument("--grep", metavar="RE")
+    p.add_argument("--invert", action="store_true")
+    p.add_argument("--dedup", action="store_true")
+    p.add_argument("--uniq", action="store_true")
+    p.add_argument("--group-dir", action="store_true")
+    p.add_argument("--max-line", type=int, metavar="N")
+    p.add_argument("--head", type=int, metavar="N")
+    p.add_argument("--tail", type=int, metavar="N")
+    p.add_argument("--middle", type=int, metavar="N")
+    p.add_argument("--stats", action="store_true")
+    args = p.parse_args()
+
+    raw = sys.stdin.read()
+    in_lines = raw.splitlines()
+    in_bytes = len(raw.encode("utf-8", "replace"))
+
+    lines = [ln.rstrip() for ln in in_lines]
+
+    if args.comments:
+        lines = [ln for ln in lines if ln.strip() and not COMMENT_RE.match(ln)]
+
+    if args.errors or args.grep:
+        pat = re.compile(args.grep) if args.grep else ERROR_RE
+        keep = (lambda ln: not pat.search(ln)) if args.invert else (lambda ln: bool(pat.search(ln)))
+        lines = [ln for ln in lines if keep(ln)]
+
+    if args.max_line and args.max_line > 0:
+        lines = [ln if len(ln) <= args.max_line else ln[: args.max_line] + ELLIPSIS for ln in lines]
+
+    if args.dedup:
+        lines = dedup_all(lines)
+    elif args.uniq:
+        lines = dedup_adjacent(lines)
+
+    if args.group_dir:
+        lines = group_dir(lines)
+
+    lines = collapse_blanks(lines)
+
+    if args.middle and args.middle > 0 and len(lines) > 2 * args.middle:
+        omitted = len(lines) - 2 * args.middle
+        lines = lines[: args.middle] + [f"{ELLIPSIS} {omitted} lines omitted {ELLIPSIS}"] + lines[-args.middle :]
+    if args.head and args.head > 0:
+        lines = lines[: args.head]
+    if args.tail and args.tail > 0:
+        lines = lines[-args.tail :]
+
+    out = "\n".join(lines)
+    if out:
+        out += "\n"
+    sys.stdout.write(out)
+
+    if args.stats:
+        out_bytes = len(out.encode("utf-8", "replace"))
+        pct = 0 if in_bytes == 0 else round(100 * (1 - out_bytes / in_bytes))
+        change = f"-{pct}%" if pct >= 0 else f"+{-pct}% larger"
+        sys.stderr.write(
+            f"slim: lines {len(in_lines)}->{len(lines)}, bytes {in_bytes}->{out_bytes} ({change})\n"
+        )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())