@complexthings/superpowers-agent 9.0.1 → 9.0.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@complexthings/superpowers-agent",
-  "version": "9.0.1",
+  "version": "9.0.2",
   "description": "Superpowers agent CLI — skills system for AI coding agents",
   "type": "module",
   "bin": {

package/skills/collaboration/leveraging-cli-tools/SKILL.md

@@ -1,6 +1,7 @@
 ---
 name: leveraging-cli-tools
-description: Use when performing code searches, JSON/YAML parsing, file finding, structural refactors, or data wrangling - ensures agents reach for high-performance CLI tools (rg, jq, fd, yq, ast-grep, gh, sd) over slower standard tools like grep/find/sed, cutting token cost and latency 5-50x. Check availability and offer to install a tool when a relevant task arises.
+description: Use when performing code searches, JSON/YAML parsing, file finding, structural refactors, or data wrangling, or when a command floods context with verbose output like logs, CI, or test runs - ensures agents reach for high-performance CLI tools (rg, jq, fd, yq, ast-grep, gh, sd) over slower standard tools like grep/find/sed and use them with discipline, choosing the right output-reducing flags, composing pipelines so raw output never enters context, and using a bundled reducer for output that flags cannot shape, cutting token cost and latency 5-50x. Check availability and offer to install a tool when a relevant task arises.
+compatibility: The bundled scripts/slim.py needs python3 (standard on macOS/Linux). The CLI tools install via the system package manager.
 ---
 
 # Leveraging CLI Tools
@@ -9,6 +10,8 @@ description: Use when performing code searches, JSON/YAML parsing, file finding,
 
 Reach for high-performance CLI tools over slower standard tools. The leverage is filtering and transforming with the right tool **before reading**, so tokens and time go to the answer, not the search. On a large tree `rg` is 10-50x faster than `grep` and returns far less noise; across a session that compounds into hours and tens of thousands of tokens saved.
 
+Picking the faster tool is only half of it. The other half is using it with discipline — two levers that keep raw output out of your context: **ask for less** (the right flags and selectors), and **shrink what you can't shape** (a reducer for inherently verbose output). A fast tool fed a lazy command still floods your context.
+
 ## When a relevant task arises
 
 1. Pick the right tool from the table below.
@@ -37,29 +40,56 @@ Check only the tools the current task needs — no upfront session-wide scan.
 | 5 | `fzf` (`-f`) | manual fuzzy filtering | Non-interactive `-f`/`--filter` mode: fuzzy-rank a candidate list piped from `fd`/`rg`. | `brew install fzf` |
 | 4 | `watchexec` | `while`+`sleep`, `entr` | Run a command on file change. Useful in build/test loops; non-interactive unlike most watchers. | `brew install watchexec` |
 
-## Core workflows
+## Use them with discipline
 
-**Filter before reading** — find the matches, then read only those.
-```bash
-rg -l "password.*hash" src/auth/ --type ts | xargs rg "TODO"
-```
+Five habits separate a fast, low-noise result from a slow, context-flooding one. The token savings below are measured, not hypothetical.
+
+**1 — Pick the tool that fits the data, not the one in muscle memory.** JSON → `jq`, YAML/TOML → `yq`, structural refactor → `ast-grep`, GitHub → `gh`. The reflex to resist is `grep`/`sed`/`awk` on structured data — they treat structure as flat text, so you pay in escaping bugs and noise the format-aware tool never produces.
+
+**2 — Ask for less: use the flag or selector that returns the answer's _shape_.** The biggest single win is never pulling raw output into context. Match the request to the question:
+
+| Question | Flag / selector | Returns |
+|----------|-----------------|---------|
+| Which files match? | `rg -l` | paths only, no lines |
+| How many files? | `rg -l \| wc -l` | a file count, not a line count |
+| How many hits? | `rg -c` (lines/file) / `rg --count-matches` (matches/file) | counts only |
+| Just the matched bit? | `rg -o` | the substring, not the whole line |
+| Enough to judge a hit? | `rg -A/-B/-C N`, `rg -m N` (cap per file) | bounded context |
+| Only certain fields? | `jq -r '.a, .b'`, `yq`, `mlr --opprint cut -f` | projected values, not the whole doc |
+| First N of a list? | pipe to `head`, or `fd --max-results N` | the slice you need |
 
-**Compose tools** — search, parse, dedup in one pass.
+For the **projection tools** (`jq`, `yq`, `dasel`, `miller`, `htmlq`) the selector *is* the filter — `jq` turning a 2,000-record array into one summed number is a ~100% reduction on its own. Shape their output at the selector; never post-filter it (that's what habit 4 is *not* for).
+
+**3 — Compose so only the answer comes back.** Do the counting, dedup, and projection *in the pipeline*, not by reading raw output and reasoning over it. One pass keeps every intermediate result out of context:
 ```bash
 rg -l '"error"' logs/ --type json | xargs jq -r 'select(.level=="error") | .code' | sort -u
 ```
-
-**Structural rewrite, not regex** — refactor by AST so syntax can't trip you.
+This is the lesson of programmatic tool calling (Amazon's PTC benchmarks: ~87-92% fewer tokens): when a task needs several tool calls plus processing, write **one** pipeline or script that does it all and returns only the result — the intermediate data never touches your context. So: filter before reading, and refactor by AST, not regex:
 ```bash
+rg -l "password.*hash" src/auth/ --type ts | xargs rg "TODO"
 sg --pattern 'console.log($$$A)' --rewrite 'logger.debug($$$A)' --lang ts
 ```
 
-In Claude Code the `Grep` and `Glob` tools are themselves built on ripgrep — prefer them for in-context searches, and reach for the CLI tools when you need piping, transforms, or rewrites.
+**4 — Shrink what you can't shape: the bundled reducer for verbose emitters.** Some output has no projection flag — logs, CI output (`gh run view --log`), test runs, stack traces, status dumps. Flags get you part way, but the residual stays repetitive, wide, and noisy. Pipe it through `scripts/slim.py` (relative to this skill; use its absolute path if you're working elsewhere) and ask only for the cuts the task needs:
+```bash
+gh run view --log | python3 scripts/slim.py --errors --uniq --max-line 200
+cargo test 2>&1   | python3 scripts/slim.py --errors --dedup
+rg -n TODO --no-ignore | python3 scripts/slim.py --group-dir
+```
+It does dedup (`--dedup`/`--uniq`, repeats collapse to `(xN)`), grouping (`--group-dir` → per-directory counts), truncation (`--head/--tail/--middle/--max-line`), and noise-stripping (`--comments/--errors/--grep`). On a real 1,236-line CI log, `rg` error-filtering alone cut 91% but left ~6,700 tokens; adding `slim` reached 97% — a third the residual. Run `python3 scripts/slim.py -h` for all modes. The cuts are **lossy and opt-in**: ask for what you need, know what you're discarding, and don't reach for it when a projection tool's own selector would do the job.
+
+**5 — Read the output critically — it's a claim, not a fact.** A zero or suspiciously low result is the one to distrust: `rg` skips `.gitignore`d and hidden files by default, so a real match in `node_modules/`, `dist/`, or a dotfile is silently absent. Before concluding "none," re-run with `--no-ignore`, `-uu`, or `--hidden` — then decide whether those ignored hits (vendored deps, build output, generated mirrors) actually belong in the answer; surfacing them is the check, keeping them is a judgment call. Mind case (`-i`) and word boundaries (`-w`) so you neither miss `Subagent` nor over-match `tasks` — and when an identifier doubles as a common word (a `Task` tool vs. the word "task"), even `-w` isn't enough: glance at the context to confirm the hit is the symbol, not prose. When a count or list drives a decision, confirm a match means what you think before acting on it.
+
+In Claude Code the `Grep` and `Glob` tools are themselves built on ripgrep — prefer them for in-context searches, and reach for the CLI tools when you need piping, transforms, rewrites, or output reduction.
 
 ## Red flags
 
 - Parsing JSON/YAML with `awk`/`sed`/`grep` instead of `jq`/`yq`.
 - Reading files before filtering them with `rg`.
+- Piping a tool's full output into context to eyeball it, when `-l`/`-c`/`-o` or a `jq` projection would return just the answer.
+- Letting a verbose command (test run, CI log, `git`/`docker` status dump) land raw in context when `slim` would cut it 90%+.
+- Reaching for `slim` on a projection tool's output — shape it at the `jq`/`yq`/`mlr` selector instead.
+- Concluding "no matches" from a default `rg` run without re-checking `--no-ignore`/`--hidden` — the hit may be sitting in an ignored directory.
 - Hand-rolling `curl` against the GitHub API instead of `gh`.
 - Regex codemods with `sed` where `ast-grep` is structurally safe.
 
@@ -70,6 +100,7 @@ In Claude Code the `Grep` and `Glob` tools are themselves built on ripgrep — p
 | "grep works fine" | On a big tree it's 10-50x slower and floods context with noise `rg` would have filtered out. |
 | "I don't know if they have jq" | One `command -v jq` answers it; install is seconds and pays back across the whole session. |
 | "Not worth the setup" | One install = a speedup on every future task, not just this one. |
+| "The test/CI output is just long, I'll scroll it" | A 1,200-line log is ~75k tokens of mostly noise; `slim --errors --uniq` makes it ~2k without losing the failures. |
 | "User didn't ask for optimization" | Faster, lower-noise completion *is* better completion. |
 
 ## When NOT to use
@@ -77,3 +108,4 @@ In Claude Code the `Grep` and `Glob` tools are themselves built on ripgrep — p
 - A tiny one-off (a handful of files, well under a megabyte) where the standard tool is already at hand.
 - The user declined the install — note the cost once, then use the fallback.
 - A teaching context where the standard tool is the point.
+- Output you already shaped with a selector — don't add a `slim` pass for its own sake.

package/skills/collaboration/leveraging-cli-tools/scripts/slim.py

@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+"""slim - reduce noisy command output to signal before it reaches your context.
+
+A stdin->stdout filter for output you CAN'T shape with a tool's own flags:
+logs, scan results, status dumps, stack traces, repeated lines. For tools that
+already project output (jq/yq/dasel/miller/htmlq), use their selectors instead;
+piping them through slim is redundant.
+
+Pipe a verbose command through it and ask only for the reductions you need:
+
+    cargo test 2>&1 | slim --errors --dedup
+    rg -n TODO --no-ignore | slim --group-dir
+    gh run view --log | slim --uniq --max-line 200 --middle 40
+    kubectl get events | slim --dedup --stats
+
+Reductions (compose freely):
+  --comments        drop comment-only lines (# // ; -- *) and blank lines
+  --errors          keep only lines matching error|warn|fail|fatal|panic|exception (-i)
+  --grep RE         keep only lines matching regex RE (use --invert to drop them)
+  --invert          invert --grep / --errors (drop matches instead of keeping)
+  --dedup           collapse ALL repeated lines, first-seen order, annotate "  (xN)"
+  --uniq            collapse only CONSECUTIVE repeats, annotate "  (xN)" (log-friendly)
+  --group-dir       turn a list of file paths into per-directory counts
+  --max-line N      truncate each line to N chars, append a horizontal ellipsis
+  --head N          keep first N lines
+  --tail N          keep last N lines
+  --middle N        keep first N and last N lines, replace the gap with a marker
+  --stats           print "lines A->B, bytes A->B (-P%)" to stderr (output unaffected)
+
+Always-on, lossless: trailing whitespace is stripped and runs of blank lines
+collapse to one. Everything lossy is opt-in. Order: filter -> transform ->
+dedup -> group -> blank-collapse -> truncate. Reductions are LOSSY by design;
+ask for what the task needs and know what you're discarding.
+"""
+import argparse
+import re
+import sys
+
+COMMENT_RE = re.compile(r"^\s*(#|//|;|--|\*|/\*|\*/)")
+ERROR_RE = re.compile(r"error|warn|fail|fatal|panic|exception|traceback", re.IGNORECASE)
+ELLIPSIS = "…"
+
+
+def annotate(line: str, n: int) -> str:
+    return line if n == 1 else f"{line}  (x{n})"
+
+
+def dedup_all(lines):
+    counts, order = {}, []
+    for ln in lines:
+        if ln not in counts:
+            counts[ln] = 0
+            order.append(ln)
+        counts[ln] += 1
+    return [annotate(ln, counts[ln]) for ln in order]
+
+
+def dedup_adjacent(lines):
+    out, prev, n = [], None, 0
+    for ln in lines:
+        if ln == prev:
+            n += 1
+        else:
+            if prev is not None:
+                out.append(annotate(prev, n))
+            prev, n = ln, 1
+    if prev is not None:
+        out.append(annotate(prev, n))
+    return out
+
+
+def group_dir(lines):
+    counts, order = {}, []
+    for ln in lines:
+        token = ln.split(":", 1)[0].strip()  # tolerate rg's path:line: prefix
+        d = token.rsplit("/", 1)[0] + "/" if "/" in token else "./"
+        if d not in counts:
+            counts[d] = 0
+            order.append(d)
+        counts[d] += 1
+    width = max((len(d) for d in order), default=0)
+    return [f"{d.ljust(width)}  ({counts[d]} file{'s' if counts[d] != 1 else ''})" for d in order]
+
+
+def collapse_blanks(lines):
+    out, blank = [], False
+    for ln in lines:
+        if ln.strip() == "":
+            if not blank:
+                out.append("")
+            blank = True
+        else:
+            out.append(ln)
+            blank = False
+    while out and out[-1] == "":
+        out.pop()
+    return out
+
+
+def main() -> int:
+    p = argparse.ArgumentParser(add_help=True, description="Reduce noisy stdout to signal.")
+    p.add_argument("--comments", action="store_true")
+    p.add_argument("--errors", action="store_true")
+    p.add_argument("--grep", metavar="RE")
+    p.add_argument("--invert", action="store_true")
+    p.add_argument("--dedup", action="store_true")
+    p.add_argument("--uniq", action="store_true")
+    p.add_argument("--group-dir", action="store_true")
+    p.add_argument("--max-line", type=int, metavar="N")
+    p.add_argument("--head", type=int, metavar="N")
+    p.add_argument("--tail", type=int, metavar="N")
+    p.add_argument("--middle", type=int, metavar="N")
+    p.add_argument("--stats", action="store_true")
+    args = p.parse_args()
+
+    raw = sys.stdin.read()
+    in_lines = raw.splitlines()
+    in_bytes = len(raw.encode("utf-8", "replace"))
+
+    lines = [ln.rstrip() for ln in in_lines]
+
+    if args.comments:
+        lines = [ln for ln in lines if ln.strip() and not COMMENT_RE.match(ln)]
+
+    if args.errors or args.grep:
+        pat = re.compile(args.grep) if args.grep else ERROR_RE
+        keep = (lambda ln: not pat.search(ln)) if args.invert else (lambda ln: bool(pat.search(ln)))
+        lines = [ln for ln in lines if keep(ln)]
+
+    if args.max_line and args.max_line > 0:
+        lines = [ln if len(ln) <= args.max_line else ln[: args.max_line] + ELLIPSIS for ln in lines]
+
+    if args.dedup:
+        lines = dedup_all(lines)
+    elif args.uniq:
+        lines = dedup_adjacent(lines)
+
+    if args.group_dir:
+        lines = group_dir(lines)
+
+    lines = collapse_blanks(lines)
+
+    if args.middle and args.middle > 0 and len(lines) > 2 * args.middle:
+        omitted = len(lines) - 2 * args.middle
+        lines = lines[: args.middle] + [f"{ELLIPSIS} {omitted} lines omitted {ELLIPSIS}"] + lines[-args.middle :]
+    if args.head and args.head > 0:
+        lines = lines[: args.head]
+    if args.tail and args.tail > 0:
+        lines = lines[-args.tail :]
+
+    out = "\n".join(lines)
+    if out:
+        out += "\n"
+    sys.stdout.write(out)
+
+    if args.stats:
+        out_bytes = len(out.encode("utf-8", "replace"))
+        pct = 0 if in_bytes == 0 else round(100 * (1 - out_bytes / in_bytes))
+        change = f"-{pct}%" if pct >= 0 else f"+{-pct}% larger"
+        sys.stderr.write(
+            f"slim: lines {len(in_lines)}->{len(lines)}, bytes {in_bytes}->{out_bytes} ({change})\n"
+        )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())