feed-the-machine 1.3.0 → 1.3.1

package/ftm-diagram/SKILL.md

@@ -18,6 +18,7 @@ Two-level diagram system: a root subway map of modules and per-module street map
 
 ---
 
+
 ## Graph-Powered Mode (ftm-map integration)
 
 Before running the standard analysis, check if the project has a code knowledge graph:
@@ -225,7 +226,6 @@ Add `sendEmail` node with edges to `src/notifications/DIAGRAM.mmd`.
 **View current architecture:**
 > "show architecture"
 Read and display `ARCHITECTURE.mmd` + list available module diagrams.
-
 ---
 
 ### Auto-Invocation by ftm-executor

package/ftm-executor/SKILL.md

@@ -23,10 +23,10 @@ description: Autonomous plan execution engine. Takes any plan document and execu
 
 Before starting, load context from the blackboard:
 
-1. Read `/Users/kioja.kudumu/.claude/ftm-state/blackboard/context.json` — check current_task, recent_decisions, active_constraints
-2. Read `/Users/kioja.kudumu/.claude/ftm-state/blackboard/experiences/index.json` — filter entries by task_type matching plan tasks and tags overlapping with the plan domain
+1. Read `~/.claude/ftm-state/blackboard/context.json` — check current_task, recent_decisions, active_constraints
+2. Read `~/.claude/ftm-state/blackboard/experiences/index.json` — filter entries by task_type matching plan tasks and tags overlapping with the plan domain
 3. Load top 3-5 matching experience files for relevant lessons on agent performance and timing
-4. Read `/Users/kioja.kudumu/.claude/ftm-state/blackboard/patterns.json` — check execution_patterns for agent performance and timing accuracy patterns
+4. Read `~/.claude/ftm-state/blackboard/patterns.json` — check execution_patterns for agent performance and timing accuracy patterns
 
 If index.json is empty or no matches found, proceed normally without experience-informed shortcuts.
 
@@ -648,12 +648,12 @@ Use `ftm-executor` when: human says "just go" and trusts the plan.
 
 After completing, update the blackboard:
 
-1. Update `/Users/kioja.kudumu/.claude/ftm-state/blackboard/context.json`:
+1. Update `~/.claude/ftm-state/blackboard/context.json`:
    - Set current_task status to "complete"
    - Append decision summary to recent_decisions (cap at 10)
    - Update session_metadata.skills_invoked and last_updated
-2. Write an experience file to `/Users/kioja.kudumu/.claude/ftm-state/blackboard/experiences/YYYY-MM-DD_task-slug.json` capturing task_type, agent team used, wave count, audit outcomes, and lessons learned
-3. Update `/Users/kioja.kudumu/.claude/ftm-state/blackboard/experiences/index.json` with the new entry
+2. Write an experience file to `~/.claude/ftm-state/blackboard/experiences/YYYY-MM-DD_task-slug.json` capturing task_type, agent team used, wave count, audit outcomes, and lessons learned
+3. Update `~/.claude/ftm-state/blackboard/experiences/index.json` with the new entry
 4. Emit `task_completed` event
 
 ## Requirements

package/ftm-git/SKILL.md

@@ -14,6 +14,7 @@ description: Secret scanning and credential safety gate for git operations. Prev
 ### Listens To
 - `code_changed` — run a quick scan on modified files before they get staged
 - `code_committed` — verify the commit doesn't contain secrets (post-commit safety net)
+- `push_requested` — block push and run full scan if not already cleared
 
 ## Blackboard Read
 
@@ -38,7 +39,7 @@ Yesterday we pushed API keys to the repo. That's the kind of mistake that leads
 
 ## Phase -1: Install Git Hook (First Invocation Only)
 
-The first time ftm-git runs in a repo, install a pre-commit hook as a hard safety net. This hook runs independently of Claude — it's a shell script that blocks `git commit` if staged files contain Tier 1 secret patterns. Even if this skill is not invoked, or someone runs git directly from the terminal, the hook catches it.
+The first time ftm-git runs in a repo, install a pre-commit hook as a hard safety net. This hook runs independently of Claude — it's a shell script that blocks `git commit` if staged files contain Tier 1 secret patterns. Even if Claude forgets to invoke this skill, or someone runs git directly from the terminal, the hook catches it.
 
 **Check if the hook is already installed:**
 
@@ -87,30 +88,104 @@ Before scanning, figure out what needs scanning and why you were invoked.
 
 Scan the in-scope files using regex patterns. The goal is zero false negatives — a few false positives are acceptable and will be filtered in Phase 2.
 
-Read `references/patterns/SECRET-PATTERNS.md` for the full Tier 1 and Tier 2 pattern library, the false positive suppression list, severity classifications, and per-finding record format.
+### Tier 1: High-Confidence Patterns (almost certainly real secrets)
 
-**Core Tier 1 patterns** (the most common — memorize these, consult the reference for the full set):
+These patterns have distinctive prefixes or structures that make false positives rare:
 
 ```
-AKIA[0-9A-Z]{16}                           # AWS Access Key ID
-ghp_[A-Za-z0-9_]{36}                       # GitHub PAT (classic)
-sk_live_[0-9a-zA-Z]{24,}                   # Stripe secret key (live)
-AIza[0-9A-Za-z\-_]{35}                     # Google API key
-xoxb-[0-9]{10,13}-[0-9]{10,13}-[a-zA-Z0-9]{24}  # Slack bot token
------BEGIN (RSA|DSA|EC|OPENSSH|PGP) PRIVATE KEY-----  # Private keys
+# AWS
+AKIA[0-9A-Z]{16}                                          # AWS Access Key ID
+amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}  # AWS MWS
+
+# GitHub
+ghp_[A-Za-z0-9_]{36}                                      # GitHub PAT (classic)
+gho_[A-Za-z0-9_]{36}                                      # GitHub OAuth
+ghu_[A-Za-z0-9_]{36}                                      # GitHub user token
+ghs_[A-Za-z0-9_]{36}                                      # GitHub server token
+github_pat_[A-Za-z0-9_]{82}                                # GitHub fine-grained PAT
+
+# Slack
+xoxb-[0-9]{10,13}-[0-9]{10,13}-[a-zA-Z0-9]{24}           # Slack bot token
+xoxp-[0-9]{10,13}-[0-9]{10,13}-[a-zA-Z0-9]{24,34}        # Slack user token
+xoxa-[0-9]{10,13}-[0-9]{10,13}-[a-zA-Z0-9]{24,34}        # Slack app token
+xoxr-[0-9]{10,13}-[0-9]{10,13}-[a-zA-Z0-9]{24,34}        # Slack refresh token
+
+# Google
+AIza[0-9A-Za-z\-_]{35}                                    # Google API key
+
+# Stripe
+sk_live_[0-9a-zA-Z]{24,}                                  # Stripe secret key (live)
+sk_test_[0-9a-zA-Z]{24,}                                  # Stripe secret key (test)
+rk_live_[0-9a-zA-Z]{24,}                                  # Stripe restricted key
+
+# Other services
+SG\.[A-Za-z0-9\-_]{22}\.[A-Za-z0-9\-_]{43}               # SendGrid
+SK[0-9a-fA-F]{32}                                          # Twilio
+npm_[A-Za-z0-9]{36}                                        # npm token
+pypi-[A-Za-z0-9\-_]{100,}                                 # PyPI token
+glpat-[A-Za-z0-9\-_]{20,}                                 # GitLab PAT
+-----BEGIN (RSA|DSA|EC|OPENSSH|PGP) PRIVATE KEY-----      # Private keys
 ```
 
-Run Tier 1 patterns in parallel since they're independent. For Tier 2, check surrounding context before confirming.
+### Tier 2: Context-Dependent Patterns (need surrounding context to confirm)
+
+These match common assignment patterns. Check that the value isn't a placeholder, empty string, or env var reference before flagging:
+
+```
+# Generic key/secret assignments — flag if value looks real (not placeholder)
+(api_key|apikey|api-key)\s*[:=]\s*["']?[A-Za-z0-9\-_]{16,}["']?
+(secret|secret_key|client_secret)\s*[:=]\s*["']?[A-Za-z0-9\-_]{16,}["']?
+(password|passwd|pwd)\s*[:=]\s*["']?[^\s"']{8,}["']?
+(token|access_token|auth_token)\s*[:=]\s*["']?[A-Za-z0-9\-_.]{16,}["']?
+(database_url|db_url|connection_string)\s*[:=]\s*["']?[^\s"']{20,}["']?
+
+# Bearer tokens in code
+bearer\s+[A-Za-z0-9\-._~+/]{20,}
+
+# Webhook URLs with tokens
+https://hooks\.slack\.com/services/T[A-Z0-9]{8,}/B[A-Z0-9]{8,}/[a-zA-Z0-9]{24}
+```
+
+### What to Ignore (false positive suppression)
+
+Skip matches that are clearly not real secrets:
+
+- Values that are `""`, `''`, `None`, `null`, `undefined`, `TODO`, `CHANGEME`, `your-key-here`, `xxx`, `placeholder`, `example`, `test`, `dummy`, `fake`, `sample`
+- References to environment variables: `os.environ[`, `process.env.`, `ENV[`, `${`, `os.getenv(`
+- Lines that are comments (`#`, `//`, `/*`, `--`)
+- Files in `node_modules/`, `.git/`, `vendor/`, `__pycache__/`, `dist/`, `build/`
+- Files that are themselves `.env.example`, `.env.sample`, `.env.template`
+- Lock files (`package-lock.json`, `yarn.lock`, `Gemfile.lock`, `poetry.lock`)
+- Test fixtures where the "secret" is obviously fake (e.g., `test_api_key = "sk_test_abc123"` in a test file — but still flag `sk_live_*` in test files, those are real)
+
+### Running the Scan
+
+Use the Grep tool to search in-scope files for each pattern. Run Tier 1 patterns in parallel since they're independent. For Tier 2, check surrounding context before confirming.
+
+For each finding, record:
+- **file**: absolute path
+- **line**: line number
+- **pattern**: which pattern matched
+- **tier**: 1 or 2
+- **value_preview**: first 8 chars + `...` + last 4 chars (never log the full secret)
+- **context**: the surrounding code (with the secret value masked)
 
 ## Phase 2: Validate Findings
 
 For each Tier 2 match, read the surrounding context (5 lines before and after) and determine:
 
-1. **Is the value a real secret or a placeholder?** — Check against the ignore list in `references/patterns/SECRET-PATTERNS.md`.
+1. **Is the value a real secret or a placeholder?** — Check against the ignore list above.
 2. **Is it already using an env var?** — If the code does `key = os.environ.get("API_KEY", "sk_live_abc...")`, the hardcoded value is a fallback default. Still a finding — fallback defaults with real secrets are dangerous.
 3. **Is it in a file that should be gitignored?** — If the secret is in `.env` and `.env` is in `.gitignore`, it's fine. If `.env` is NOT in `.gitignore`, that's a separate finding.
 
-After validation, produce a findings list sorted by severity (CRITICAL → HIGH → MEDIUM → LOW). See `references/patterns/SECRET-PATTERNS.md` for the severity table.
+After validation, produce a findings list sorted by severity:
+
+| Severity | Meaning |
+|---|---|
+| **CRITICAL** | Tier 1 match (high-confidence secret) in a tracked or staged file |
+| **HIGH** | Tier 2 confirmed match in a tracked or staged file |
+| **MEDIUM** | `.env` file not in `.gitignore`, or secret in a fallback default |
+| **LOW** | Secret in a gitignored file but the gitignore rule might be fragile |
 
 If zero findings after validation: emit `secrets_clear` and proceed. The commit/push is safe.
 
@@ -120,23 +195,135 @@ If any CRITICAL or HIGH findings: **STOP. The commit/push is BLOCKED.** Say this
 ftm-git: BLOCKED — <N> secret(s) found. Commit/push halted. Attempting auto-remediation...
 ```
 
-Then proceed to Phase 3. The commit/push does NOT happen until Phase 3 completes and a re-scan comes back clean.
+Then proceed to Phase 3 to fix. The commit/push does NOT happen until Phase 3 completes and a re-scan in Phase 3 Step 5 comes back clean. This is non-negotiable — even if you can fix the secrets, the user needs to see that the operation was blocked and why.
 
 ## Phase 3: Auto-Remediate
 
-Read `references/protocols/REMEDIATION.md` for the full step-by-step remediation protocol, language-specific env var patterns, report formats (clean/remediated/blocked), and the Phase 5 git history deep scan procedure.
+For each finding, apply the appropriate fix automatically. The goal is to make the code safe without breaking functionality.
+
+### Step 1: Ensure .env infrastructure exists
 
-**Summary of steps:**
-1. Ensure `.env` and `.gitignore` infrastructure exists
-2. Extract each secret to `.env` with a SCREAMING_SNAKE_CASE var name
-3. Add placeholder to `.env.example`
-4. Refactor source files to reference the env var (match language pattern)
-5. Unstage `.env`, re-stage refactored source files
-6. Verify: re-run Phase 1 on refactored files — do not proceed until clean
+Check for a `.env` file in the project root. If it doesn't exist, create one with a header comment:
+
+```
+# Environment variables — DO NOT COMMIT THIS FILE
+# Copy .env.example for the template, fill in real values locally
+```
+
+Check `.gitignore` for `.env` coverage. If missing, add:
+```
+# Environment files with secrets
+.env
+.env.local
+.env.production
+.env.staging
+.env.*.local
+```
+
+### Step 2: Extract secrets to .env
+
+For each finding:
+
+1. **Choose an env var name** — derive it from the context. If the code says `STRIPE_API_KEY = "sk_live_..."`, the env var is `STRIPE_API_KEY`. If it says `api_key: "AIza..."`, infer from the file/service context (e.g., `GOOGLE_API_KEY`). Use SCREAMING_SNAKE_CASE.
+
+2. **Add to .env** — append `VAR_NAME=<actual-secret-value>` to `.env`. If the var already exists, don't duplicate it.
+
+3. **Add to .env.example** — create or update `.env.example` with `VAR_NAME=your-value-here` so other developers know the variable exists without seeing the real value.
+
+### Step 3: Refactor source files
+
+Replace the hardcoded secret with an env var reference. Match the language/framework:
+
+| Language | Pattern |
+|---|---|
+| Python | `os.environ["VAR_NAME"]` or `os.getenv("VAR_NAME")` (match existing style in file) |
+| JavaScript/TypeScript | `process.env.VAR_NAME` |
+| Ruby | `ENV["VAR_NAME"]` or `ENV.fetch("VAR_NAME")` |
+| Go | `os.Getenv("VAR_NAME")` |
+| Java | `System.getenv("VAR_NAME")` |
+| Shell/Bash | `$VAR_NAME` or `${VAR_NAME}` |
+| YAML/JSON config | `${VAR_NAME}` (if the framework supports interpolation) or add a comment pointing to the env var |
+
+If the file doesn't already import the env-reading module (e.g., `import os` in Python, `require('dotenv').config()` in Node), add the import. Check if the project uses `python-dotenv`, `dotenv` (Node), or similar — if so, use the project's existing pattern for loading env vars.
+
+### Step 4: Unstage remediated files
+
+After refactoring, make sure the `.env` file (with real secrets) is NOT staged:
+
+```bash
+git reset HEAD .env 2>/dev/null  # unstage if accidentally staged
+```
+
+Stage the refactored source files (which now reference env vars instead of hardcoded secrets):
+
+```bash
+git add <refactored-files>
+```
+
+### Step 5: Verify the fix
+
+Re-run Phase 1 scan on the refactored files to confirm the secrets are gone. If any remain, loop back and fix. Do not proceed until the scan is clean.
 
 ## Phase 4: Report
 
-After remediation or clean scan, produce the summary. Read `references/protocols/REMEDIATION.md` for the exact report formats.
+After remediation (or if the scan was clean from the start), produce a summary:
+
+**Clean scan:**
+```
+ftm-git: Clean scan. 0 secrets found in <N> files scanned. Safe to commit.
+```
+
+**After remediation:**
+```
+ftm-git: Found <N> hardcoded secrets. Auto-remediated:
+
+  CRITICAL: sk_live_**** in src/payments.py:42 -> STRIPE_SECRET_KEY
+  HIGH:     AIza**** in config/google.ts:18 -> GOOGLE_API_KEY
+  MEDIUM:   .env was not in .gitignore -> added
+
+Actions taken:
+  - Extracted <N> secrets to .env (gitignored)
+  - Created/updated .env.example with placeholder vars
+  - Refactored <N> source files to use env var references
+  - Updated .gitignore
+
+Verify the app still works with the new env var setup, then commit.
+```
+
+**Blocked (auto-fix not possible):**
+
+Some secrets can't be auto-fixed — for example, a private key embedded in a binary file, or a secret in a format the skill can't safely refactor. In these cases:
+
+```
+ftm-git: BLOCKED. Found secrets that require manual remediation:
+
+  CRITICAL: Private key in assets/cert.pem:1
+            -> Move this file outside the repo and reference via path env var
+
+  Action required: Fix the above manually, then run ftm-git again.
+```
+
+## Phase 5: Git History Check (Manual Invocation Only)
+
+When explicitly asked to do a deep scan (e.g., "scan the repo history for secrets"), also check past commits. This is expensive so it only runs on explicit request, not as part of the pre-commit gate.
+
+```bash
+git log --all --diff-filter=A --name-only --pretty=format:"%H" -- "*.env" "*.pem" "*.key" "*credentials*" "*secret*"
+```
+
+For each historically added sensitive file, check if it's still in the current tree. If it was added and later removed, warn that the secret is still in git history and suggest:
+
+1. Rotate the credential immediately (it's compromised)
+2. Use `git filter-repo` or BFG Repo Cleaner to purge from history if needed
+
+## Operating Principles
+
+1. **Block first, fix second.** Never let a secret through while figuring out the fix. The commit waits.
+2. **Zero false negatives over zero false positives.** It's better to flag something that turns out to be harmless than to miss a real key.
+3. **Never log full secrets.** In all output, mask secret values. Show only enough to identify which secret it is (first 8 + last 4 chars).
+4. **Env vars are the escape hatch.** The remediation pattern is always: secret goes to gitignored .env, code references the env var.
+5. **Existing patterns win.** If the project already uses dotenv, Vault, AWS Secrets Manager, or any other secret management system, match that pattern rather than introducing a new one.
+6. **Test files are not exempt.** A real `sk_live_*` key in a test file is just as dangerous as one in production code. Only `sk_test_*` with obviously fake values get a pass.
 
 ## Integration Points
 

package/ftm-inbox/bin/start.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 # Start ftm-inbox backend + pollers
-cd "$(dirname "$0")/.."
+cd "$(dirname "$0")/.." || exit
 PORT=${FTM_INBOX_PORT:-8042}
 echo "Starting ftm-inbox on port $PORT..."
 python3 -m uvicorn backend.main:app --host 0.0.0.0 --port $PORT &

package/ftm-inbox/bin/status.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-if [ -f /tmp/ftm-inbox.pid ] && kill -0 $(cat /tmp/ftm-inbox.pid) 2>/dev/null; then
+if [ -f /tmp/ftm-inbox.pid ] && kill -0 "$(cat /tmp/ftm-inbox.pid)" 2>/dev/null; then
     echo "ftm-inbox is running (PID: $(cat /tmp/ftm-inbox.pid))"
     # Show last poll times from DB if available
     CONFIG_DIR="$HOME/.claude/ftm-inbox"

package/ftm-inbox/bin/stop.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 if [ -f /tmp/ftm-inbox.pid ]; then
-    kill $(cat /tmp/ftm-inbox.pid) 2>/dev/null
+    kill "$(cat /tmp/ftm-inbox.pid)" 2>/dev/null
     rm /tmp/ftm-inbox.pid
     echo "ftm-inbox stopped."
 else

package/ftm-intent/SKILL.md

@@ -190,7 +190,6 @@ When updating after changes:
 4. Write updates — add missing entries, remove stale entries, update changed fields
 5. If new modules were added, create their INTENT.md and add rows to root module map
 6. Report: list of files updated, entries added, entries removed, entries modified
-
 ---
 
 ### Auto-Invocation by ftm-executor