panopticon-cli 0.4.32 → 0.5.0

package/skills/crash-investigation/SKILL.md

@@ -0,0 +1,301 @@
+---
+name: crash-investigation
+description: >
+  Investigate system crashes, OOM kills, and unresponsive episodes. Analyzes
+  previous boot logs, identifies memory hogs, tallies per-process-group consumption,
+  checks agent and workspace state, and produces a recovery summary. Use after a
+  hard reset, freeze, or reboot caused by resource exhaustion.
+triggers:
+  - crash investigation
+  - investigate crash
+  - why did my system crash
+  - OOM investigation
+  - system froze
+  - out of memory
+  - system was unresponsive
+  - what happened to my system
+allowed-tools:
+  - Bash
+  - Read
+  - Grep
+  - Glob
+---
+
+# Crash Investigation
+
+Post-crash forensic analysis for developer workstations running AI agents, containers, and k8s workloads.
+
+## When to Use
+
+- System was unresponsive and required a hard reset
+- OOM killer was active (processes were killed unexpectedly)
+- CPU was pegged at 100% for an extended period
+- System rebooted unexpectedly
+- Agents were running overnight and the machine froze
+
+## Investigation Workflow
+
+Work through each phase in order. Collect all evidence before drawing conclusions.
+
+---
+
+### Phase 1: Establish Timeline
+
+Determine when the crash happened and how long the previous session ran.
+
+```bash
+# List recent boots — the previous boot (-1) is the crashed session
+journalctl --list-boots | tail -5
+
+# Check reboot history
+last -x reboot shutdown | head -10
+```
+
+Record:
+- **Previous boot start**: when the session began
+- **Previous boot end**: when the crash/reset occurred
+- **Duration**: how long it ran before crashing
+
+---
+
+### Phase 2: Check for OOM Kills
+
+OOM (Out of Memory) kills are the most common cause of system freezes on dev machines.
+
+```bash
+# All OOM events from the crashed boot
+journalctl -b -1 --no-pager | grep -E "oom-kill|Out of memory|Killed process" | head -30
+
+# Memory pressure and related failures
+journalctl -b -1 --since "<LAST_FEW_HOURS>" --no-pager | \
+  grep -iE "oom|out.of.memory|killed process|enomem|cannot allocate|memory pressure" | tail -50
+```
+
+If OOM kills are found, note:
+- **First OOM kill timestamp** — this is when trouble started
+- **Which processes were killed** and in what order
+- **Whether the right processes were killed** (OOM killer picks by `oom_score_adj`, not by actual memory usage)
+
+---
+
+### Phase 3: Identify Memory Hogs
+
+When the OOM killer fires, the kernel dumps a full process table. This is the most valuable evidence.
+
+```bash
+# Find the process table dump from the first OOM event
+# Look for lines with [pid] format — these are the process listings
+journalctl -b -1 --since "<FIRST_OOM_TIME>" --until "<FIRST_OOM_TIME + 1min>" --no-pager | \
+  grep "kernel:" | grep -P "\[\s*\d+\]" | head -80
+```
+
+#### Tally Memory by Process Group
+
+For each major process type, sum the RSS (resident memory) column. The OOM dump format is:
+
+```
+[  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
+```
+
+RSS is field `$11` when parsed with awk from the full journalctl line. Values are in **pages** (4 KB each).
+
+```bash
+# Tally for a specific process name (replace PROCNAME)
+journalctl -b -1 --since "<OOM_TIME>" --until "<OOM_TIME + 1min>" --no-pager | \
+  grep "kernel:" | grep -P "\bPROCNAME\b" | \
+  awk '{rss=$11; total+=rss} END {printf "%s: %d processes, RSS: %.1f GB\n", "PROCNAME", NR, total*4/1024/1024}'
+```
+
+Common process groups to check:
+- `claude` — Claude Code CLI processes
+- `node` — Node.js (includes vitest workers, MCP servers, dev servers)
+- `vitest` — Test runner workers (often named "node (vitest N)")
+- `java` — JVMs (Spring Boot, Minecraft, Solr, k3s pods)
+- `python` — Python services
+- `beam.smp` — Erlang/RabbitMQ
+- `grafana`, `prometheus` — Monitoring stack
+- `postgres` — Database
+
+Compare the total against system RAM:
+```bash
+# System RAM
+free -h
+
+# CPU count (vitest/jest default to this many workers)
+nproc
+```
+
+---
+
+### Phase 4: Analyze the Death Spiral
+
+Check the final minutes/hours of the crashed boot for cascading failures.
+
+```bash
+# Last 100 log entries before the crash
+journalctl -b -1 --since "<LAST_10_MINUTES>" --no-pager | tail -100
+```
+
+Look for these indicators:
+- **"Under memory pressure, flushing caches"** — system is thrashing
+- **"SYN_DROPPED event"** on input devices — kernel dropping mouse/keyboard events (explains unresponsiveness)
+- **"Can't keep up!"** from game servers or other real-time processes
+- **"Watchdog timeout"** on systemd services — services frozen
+- **Services crash-looping** (e.g., fwupd, journald restarting repeatedly)
+- **DNS timeouts** — networking breaking down under load
+
+---
+
+### Phase 5: Check Agent and Workspace State
+
+After understanding what caused the crash, assess what work was in progress.
+
+#### Agent States
+```bash
+# List all agents and their last state
+for agent_dir in ~/.panopticon/agents/*/; do
+  agent=$(basename "$agent_dir")
+  state=$(cat "$agent_dir/state.json" 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'{d.get(\"state\",\"unknown\")} @ {d.get(\"lastActivity\",\"?\")}')" 2>/dev/null || echo "no state")
+  echo "$agent: $state"
+done
+```
+
+#### Workspace Git Status
+```bash
+# For each workspace, check for uncommitted work and branch status
+for ws in /path/to/project/workspaces/*/; do
+  echo "=== $(basename $ws) ==="
+  for repo in "$ws"*/; do
+    if [ -d "$repo/.git" ]; then
+      echo "  $(basename $repo):"
+      echo "    Branch: $(git -C "$repo" branch --show-current 2>/dev/null)"
+      echo "    Uncommitted: $(git -C "$repo" status --short 2>/dev/null | wc -l) files"
+      echo "    Ahead of main: $(git -C "$repo" log --oneline origin/main..HEAD 2>/dev/null | wc -l) commits"
+    fi
+  done
+done
+```
+
+#### Planning State
+```bash
+# Check STATE.md for each workspace
+for ws in /path/to/project/workspaces/*/; do
+  state_file="$ws/.planning/STATE.md"
+  if [ -f "$state_file" ]; then
+    echo "=== $(basename $ws) ==="
+    head -5 "$state_file"
+    echo
+  fi
+done
+```
+
+---
+
+### Phase 6: Check Current System Health
+
+Verify the system is stable after reboot before resuming work.
+
+```bash
+# Current memory usage
+free -h
+
+# Top memory consumers right now
+ps aux --sort=-%mem | head -15
+
+# Any leftover problematic processes
+ps aux | grep -c '[c]laude'
+ps aux | grep -c '[v]itest'
+ps aux | grep -c '[n]ode'
+
+# Disk space (OOM can also be triggered by full /tmp or swap partition)
+df -h / /tmp /home
+```
+
+---
+
+### Phase 7: Produce Recovery Summary
+
+Compile findings into a structured summary. This format is designed to be handed to another AI agent to resume work.
+
+```markdown
+## Crash Summary
+
+**Crash time**: [timestamp]
+**Cause**: [OOM / CPU exhaustion / disk full / other]
+**Duration of instability**: [how long before reset]
+
+### Root Cause
+[1-2 sentences: what process(es) consumed what resource]
+
+### Memory Breakdown at Time of Crash
+| Process Group | Count | RSS | Notes |
+|---|---|---|---|
+| ... | ... | ... | ... |
+
+### OOM Kill Sequence
+1. [timestamp] - [process] killed ([size])
+2. ...
+
+### Agent States
+| Agent | State | Last Activity | What it was doing |
+|---|---|---|---|
+| ... | ... | ... | ... |
+
+### Workspace Status
+| Workspace | Status | Uncommitted Changes | Commits Ahead | Notes |
+|---|---|---|---|---|
+| ... | ... | ... | ... | ... |
+
+### What Was In-Flight
+- [list of tasks that were actively being worked on]
+
+### What Needs to Happen Now
+1. [ordered recovery steps]
+```
+
+---
+
+## Common Root Causes
+
+### Vitest/Jest Worker Explosion
+**Symptom**: Many `node (vitest N)` processes each using 1-3 GB.
+**Cause**: No `maxForks`/`maxThreads` configured; defaults to CPU count.
+**Fix**: Add to `vitest.config.ts`:
+```typescript
+pool: 'forks',
+poolOptions: {
+  forks: { maxForks: 4 },
+},
+```
+
+### Multiple Claude Code Agents
+**Symptom**: 10+ `claude` processes, each 200-800 MB RSS plus swap.
+**Cause**: Panopticon agents + specialist agents + sub-agents all running simultaneously.
+**Fix**: Limit concurrent agents in Panopticon config; ensure agents exit cleanly.
+
+### K8s Pod Memory Limits Not Set
+**Symptom**: Java/Go processes in k8s pods growing unbounded.
+**Cause**: No resource limits in pod specs; pods consume all available memory.
+**Fix**: Set `resources.limits.memory` in pod specs.
+
+### Swap Thrashing
+**Symptom**: System responsive but extremely slow; disk I/O at 100%.
+**Cause**: Total memory demand exceeds RAM but not RAM+swap, causing constant page faults.
+**Fix**: Either add RAM, reduce workload, or set `vm.swappiness=10` to prefer OOM-killing over thrashing.
+
+---
+
+## Prevention
+
+1. **Limit test parallelism** — Always set `maxForks`/`maxThreads` in vitest/jest configs
+2. **Set agent concurrency limits** — Don't run more than 3-4 agents simultaneously on 64 GB
+3. **Configure k8s resource limits** — Every pod should have memory limits
+4. **Monitor proactively** — Use `watch -n 5 free -h` or a Grafana dashboard
+5. **Set up earlyoom** — `sudo apt install earlyoom` kills the largest process before the system freezes
+
+## Related Skills
+
+- `/pan:rescue` — Recover uncommitted work from crashed agents
+- `/pan:session-health` — Clean up stuck Claude Code sessions
+- `/incident-response` — Structured approach to production incidents
+- `/pan:status` — Check current agent and system status

package/skills/github-cli/SKILL.md

@@ -0,0 +1,185 @@
+---
+name: github-cli
+description: GitHub CLI (gh) reference for issues, PRs, and API calls
+triggers:
+  - gh cli
+  - github cli
+  - github issues
+  - github pull requests
+  - gh commands
+allowed-tools:
+  - Bash
+  - Read
+---
+
+# GitHub CLI (`gh`) Reference
+
+Use `gh` CLI for all GitHub operations. Anthropic recommends `gh` CLI over GitHub MCP servers for Claude Code since you already have shell access.
+
+## Common Pitfalls
+
+### Invalid JSON fields cause errors
+`--json` only accepts specific fields per command. Using invalid fields (e.g., `stateReason`) causes the command to fail and may cancel sibling tool calls.
+
+**Always use fields from the valid lists below.** When in doubt, run `gh <command> --json` with no field names to see the valid list.
+
+### Non-interactive mode
+`gh` commands that prompt for input will hang in automated contexts. Always provide all required flags explicitly:
+- `gh pr create` needs `--title` and `--body`
+- `gh issue create` needs `--title`
+- `gh pr merge` needs a merge strategy flag (`--merge`, `--squash`, or `--rebase`)
+
+## Issue Commands
+
+### `gh issue list`
+```bash
+gh issue list --state open --limit 50
+gh issue list --label "bug" --assignee "@me"
+gh issue list --search "error sort:created-asc"
+gh issue list --json number,title,state,labels
+```
+
+**Flags:** `-s/--state {open|closed|all}`, `-l/--label`, `-a/--assignee`, `-A/--author`, `-m/--milestone`, `-S/--search`, `-L/--limit`, `--json`, `-R/--repo`
+
+**Valid `--json` fields:**
+`assignees`, `author`, `body`, `closed`, `closedAt`, `comments`, `createdAt`, `id`, `labels`, `milestone`, `number`, `projectCards`, `projectItems`, `reactionGroups`, `state`, `title`, `updatedAt`, `url`
+
+### `gh issue view`
+```bash
+gh issue view 123
+gh issue view 123 --json number,title,state,body
+gh issue view 123 --comments
+```
+
+**Valid `--json` fields:** Same as `gh issue list`.
+
+**NOT valid:** `stateReason`, `closedBy`, `reactions`, `timeline`
+
+### `gh issue create`
+```bash
+gh issue create --title "Bug: X" --body "Description"
+gh issue create --title "Feature" --label "enhancement" --assignee "@me"
+```
+
+**Flags:** `-t/--title`, `-b/--body`, `-F/--body-file`, `-l/--label`, `-a/--assignee`, `-m/--milestone`, `-p/--project`
+
+### `gh issue close`
+```bash
+gh issue close 123
+gh issue close 123 --reason "completed"
+gh issue close 123 --comment "Fixed in PR #456"
+```
+
+**Flags:** `-r/--reason {completed|not planned}`, `-c/--comment`
+
+### `gh issue edit`
+```bash
+gh issue edit 123 --title "New title"
+gh issue edit 123 --add-label "bug" --remove-label "triage"
+gh issue edit 123 --add-assignee "@me"
+```
+
+## Pull Request Commands
+
+### `gh pr list`
+```bash
+gh pr list --state open
+gh pr list --json number,title,state,headRefName
+gh pr list --head feature/pan-123
+```
+
+**Flags:** `-s/--state {open|closed|merged|all}`, `-B/--base`, `-H/--head`, `-l/--label`, `-a/--assignee`, `-A/--author`, `-S/--search`, `-L/--limit`, `--json`
+
+**Valid `--json` fields:**
+`additions`, `assignees`, `author`, `autoMergeRequest`, `baseRefName`, `body`, `changedFiles`, `closed`, `closedAt`, `comments`, `commits`, `createdAt`, `deletions`, `files`, `headRefName`, `headRefOid`, `headRepository`, `headRepositoryOwner`, `id`, `isCrossRepository`, `isDraft`, `labels`, `latestReviews`, `maintainerCanModify`, `mergeCommit`, `mergeStateStatus`, `mergeable`, `mergedAt`, `mergedBy`, `milestone`, `number`, `potentialMergeCommit`, `projectCards`, `projectItems`, `reactionGroups`, `reviewDecision`, `reviewRequests`, `reviews`, `state`, `statusCheckRollup`, `title`, `updatedAt`, `url`
+
+### `gh pr view`
+```bash
+gh pr view 456
+gh pr view 456 --json number,title,state,mergeStateStatus,reviews
+```
+
+**Valid `--json` fields:** Same as `gh pr list`.
+
+### `gh pr create`
+```bash
+gh pr create --title "feat: Add X" --body "## Summary\n- Added X\n\n## Test plan\n- [ ] Test Y"
+gh pr create --title "fix: Bug" --body "Fixes #123" --label "bug"
+gh pr create --fill  # auto-fill from commit messages
+```
+
+**Flags:** `-t/--title`, `-b/--body`, `-F/--body-file`, `-B/--base`, `-H/--head`, `-l/--label`, `-r/--reviewer`, `-a/--assignee`, `-m/--milestone`, `-d/--draft`, `-f/--fill`
+
+Use HEREDOC for multi-line body:
+```bash
+gh pr create --title "feat: X" --body "$(cat <<'EOF'
+## Summary
+- Change 1
+
+## Test plan
+- [ ] Verify X
+EOF
+)"
+```
+
+### `gh pr merge`
+```bash
+gh pr merge 456 --squash --delete-branch
+gh pr merge 456 --merge
+gh pr merge 456 --rebase
+```
+
+**Flags:** `-m/--merge`, `-s/--squash`, `-r/--rebase`, `-d/--delete-branch`, `--auto`, `-b/--body`, `-t/--subject`
+
+**Must specify one of:** `--merge`, `--squash`, or `--rebase`
+
+### `gh pr close`
+```bash
+gh pr close 456
+gh pr close 456 --delete-branch
+```
+
+### `gh pr checkout`
+```bash
+gh pr checkout 456
+```
+
+## API Commands
+
+### `gh api`
+For operations not covered by `gh issue` or `gh pr`, use the REST API directly:
+
+```bash
+# Get PR comments
+gh api repos/{owner}/{repo}/pulls/123/comments
+
+# Get PR review comments
+gh api repos/{owner}/{repo}/pulls/123/reviews
+
+# Get issue timeline events
+gh api repos/{owner}/{repo}/issues/123/timeline
+
+# Add a comment
+gh api repos/{owner}/{repo}/issues/123/comments -f body="Comment text"
+
+# Get repo info
+gh api repos/{owner}/{repo}
+```
+
+`{owner}` and `{repo}` are auto-filled from the current repo context.
+
+**Flags:** `-X/--method {GET|POST|PUT|PATCH|DELETE}`, `-f/--raw-field key=value`, `-F/--field key=value`, `--paginate`, `-q/--jq`
+
+### Filtering with jq
+```bash
+gh issue list --json number,title,labels --jq '.[] | select(.labels[].name == "bug")'
+gh pr list --json number,title,headRefName --jq '.[] | select(.headRefName | startswith("feature/"))'
+```
+
+## Repo-Scoped Operations
+
+When not in a git repo or targeting a different repo, use `-R`:
+```bash
+gh issue list -R eltmon/panopticon-cli
+gh pr view 456 -R eltmon/panopticon-cli
+```