xtrm-tools 0.7.17 → 0.7.18

package/.xtrm/skills/default/releasing/SKILL.md

@@ -1,94 +1,200 @@
 ---
 name: releasing
 description: >-
-  Cut a release with the canonical xt release prepare/publish flow. Use when the
-  operator wants to publish a new tag (vX.Y.Z). Prepare drafts CHANGELOG from xt
-  reports and performs deterministic release-file mutations; publish creates the
-  annotated tag, pushes commits/tags, and can create a GitHub release.
-version: 1.2.0
+  Cut a release end-to-end. Promotes the existing [Unreleased] CHANGELOG block
+  to a dated [vX.Y.Z] section, bumps the package version, rebuilds dist,
+  commits, tags, pushes, and publishes to npm. Optionally dispatches the
+  changelog-keeper specialist to fill gaps in [Unreleased] from xt reports
+  before promotion.
+version: 2.0.0
 ---
 
 # releasing
 
-Canonical release publication via `xt release prepare` and `xt release publish`.
+End-to-end release skill. Drives the release directly with bash; no `xt release`
+wrapper. Assumes `[Unreleased]` in `CHANGELOG.md` is already populated by the
+`session-close-report` skill (Step 6) across the contributing sessions.
 
 ## When to use
 
-The operator wants to cut a release. They say "release it", "ship vX.Y.Z", "cut a tag", or just "release".
+The operator says "release it", "ship vX.Y.Z", "cut a tag", or just "release".
 
-## How
+## Preconditions
 
-1. Determine target version. Default is patch bump from most recent semver tag. Operator may specify `--minor`, `--major`, or explicit version.
+- Working tree clean except whitelisted release artifacts.
+- `CHANGELOG.md` exists with an `[Unreleased]` block.
+- On the publish branch (usually `master`).
+- npm auth set up (`npm whoami` succeeds).
 
-2. Determine tag range. Default is `<latest-tag>..HEAD`. For backfills, operator names `--from` / `--to` explicitly.
+If the project has no `CHANGELOG.md`, stop and ask the operator.
 
-3. Prepare release files:
+## Flow
 
-   ```bash
-   xt release prepare --patch
-   # or: xt release prepare --minor --from <tag> --to HEAD
-   ```
+### 1. Decide version
 
-   `prepare` is the canonical path. It builds the xt report bundle, calls the specialists changelog drafting script (`sp script changelog-keeper`), updates release files, rebuilds dist, and enforces the release scope guard.
+```bash
+git tag --sort=-v:refname | head -3        # last release
+git log <last-tag>..HEAD --oneline | wc -l # commit count since
+```
 
-   Current blocker: until specialists issue `unitAI-dnmcg` lands, `prepare` can fail with `interactive specialists are not allowed` because the changelog drafting specialist is not yet script-compatible. If that happens, do a manual prepare using the same scope rules and then continue with `xt release publish`.
+Default is patch bump. Operator may say `--minor`, `--major`, or specify
+`vX.Y.Z` explicitly. If the in-range work added user-facing surface (new flags,
+new endpoints, new commands), bump minor unless operator says otherwise.
 
-4. Verify release diff before publishing.
+### 2. Inspect [Unreleased]
 
-   ```bash
-   git diff --stat HEAD~1 HEAD
-   git status --short
-   ```
+```bash
+sed -n '/## \[Unreleased\]/,/## \[/p' CHANGELOG.md
+```
 
-   Release diff must be limited to release artifacts such as:
-   - `CHANGELOG.md`
-   - package manifests / lockfile for version sync
-   - generated `cli/dist/**` or `dist/**`
+Decision:
 
-5. Publish:
+- **Populated and complete** (covers the user-facing changes since last tag):
+  proceed to Step 3.
+- **Populated but gappy or sparse** (some sessions skipped Step 6, or the
+  bullets do not match commits): dispatch `changelog-keeper` to reconcile from
+  xt reports, then re-inspect. See "Optional: changelog-keeper dispatch".
+- **Empty** but commits since last tag are pure-internal (refactors,
+  doc-only): proceed; the released section will be near-empty.
+- **Empty** but there are user-facing changes: dispatch `changelog-keeper`.
 
-   ```bash
-   xt release publish
-   # optional GitHub release:
-   xt release publish --gh-release
-   ```
+### 3. Promote [Unreleased] → [vX.Y.Z]
 
-   `publish` creates the annotated tag for the current package version, pushes commits and tags, and optionally creates the GitHub release.
+Edit `CHANGELOG.md`:
 
-6. Confirm:
+- Insert a new empty `## [Unreleased]` block at the top.
+- Rename the previous `[Unreleased]` to `## [vX.Y.Z] — YYYY-MM-DD` (today's
+  date).
+- Keep the section bodies (Added / Changed / Fixed / etc.) untouched.
 
-   ```bash
-   git tag --list 'v*' | tail -3
-   git log --oneline -1
-   git status --short --branch
-   ```
+### 4. Bump version
 
-## Why this design
+```bash
+# package.json: "version": "X.Y.Z"
+# package-lock.json: "version" appears at top and inside packages[""]
+```
+
+Edit both. Other lockfiles (bun.lock, pnpm-lock) generally do not encode the
+top-level version — skip unless the project does.
+
+### 5. Build
+
+```bash
+npm run build
+git status --short
+```
+
+Tracked `dist/**` may or may not change (often byte-identical when HEAD
+already had a fresh build).
+
+### 6. Verify release scope
+
+```bash
+git status --short
+git diff --stat
+```
+
+Allowed paths only:
+
+- `CHANGELOG.md`
+- `package.json`
+- `package-lock.json` (if tracked)
+- `dist/**` (if tracked)
+
+Anything else dirty → stop, fix scope, retry. Stash unrelated untracked files
+(e.g. downstream specialist leakage in `.specialists/user/`) before continuing.
+
+### 7. Commit, tag, push
+
+```bash
+git add CHANGELOG.md package.json package-lock.json dist/   # only what's tracked
+git commit -m "release: vX.Y.Z"
+git tag -a vX.Y.Z -m "Release vX.Y.Z"
+git push origin <branch>
+git push origin vX.Y.Z
+```
 
-- `xt` owns deterministic release mutation: changelog insertion, version bump, build, scope guard, commit/tag/push.
-- The specialist owns only changelog drafting from xt reports through a script-compatible, READ_ONLY surface.
-- xt reports are synthesis input, not raw git log + bd query. Reports are pre-curated, signal-rich, written in user-facing language.
-- `xt release publish` is intentionally separate so operators can inspect prepared release files before pushing the tag.
+### 8. Publish to npm
 
-## Manual fallback while unitAI-dnmcg is open
+```bash
+npm publish --access public    # --access public for scoped packages
+```
 
-If `xt release prepare` fails on the changelog script compatibility guard:
+### 9. Refresh global toolchain
 
-1. Draft the CHANGELOG section manually from `.xtrm/reports/` and recent commits.
-2. Bump package versions and lockfile.
-3. Run `npm run build`.
-4. Commit with `release: vX.Y.Z`.
-5. Run `xt release publish`.
+If this project ships a CLI the operator uses globally:
+
+```bash
+npm i -g <package>@X.Y.Z
+<cli> --version
+```
+
+### 10. Confirm
+
+```bash
+git tag --list 'v*' | tail -3
+git log --oneline -1
+git status --short
+npm view <package> version
+```
+
+### 11. Optional: GitHub release
+
+```bash
+gh release create vX.Y.Z --notes "$(sed -n '/## \[vX.Y.Z\]/,/## \[/p' CHANGELOG.md | head -n -1)"
+```
+
+## Optional: changelog-keeper dispatch
+
+When `[Unreleased]` is empty or sparse and user-facing work shipped, file a
+small bead and dispatch `changelog-keeper`:
+
+```text
+TITLE: Fill [Unreleased] from <prev-tag>..HEAD
+PROBLEM: [Unreleased] is missing entries for sessions <list> which shipped user-facing changes; release vX.Y.Z is being cut.
+SUCCESS: [Unreleased] reflects every user-facing change in the range, in Keep-a-Changelog format.
+SCOPE: CHANGELOG.md only.
+REFERENCES: .xtrm/reports/<prev-tag-date>..<today> (filtered to in-range), recent commit subjects.
+NON_GOALS: no version bump, no build, no commit, no tag — skill owns those.
+CONSTRAINTS: edit CHANGELOG.md only; one bullet per change; bead refs in parens; sections Added / Changed / Fixed / Removed / Deprecated / Security; do not invent entries not grounded in reports or commits.
+VALIDATION: diff shows only [Unreleased] body changed; bullets cover the report set.
+OUTPUT: updated CHANGELOG.md with populated [Unreleased].
+```
+
+```bash
+sp run changelog-keeper --bead <bead-id>
+```
+
+After it returns, re-inspect `[Unreleased]` and proceed to Step 3. Skill — not
+keeper — owns version bump, build, commit, tag, push, publish.
+
+## Why this design
 
-Do not broaden the release diff beyond release artifacts.
+- `session-close-report` already drafts user-facing prose per session and
+  appends to `[Unreleased]` (Step 6 of that skill). The release-time work is
+  therefore small: promote, bump, build, commit, tag, push, publish.
+- Keeping the deterministic mutations in bash (driven by this skill) avoids
+  three failure modes seen with the old `xt release` wrapper: hardcoded
+  workspace paths, wrong specialist invocation, and a regex template engine
+  that over-greedy-matches `$VAR` patterns inside report prose.
+- The `changelog-keeper` specialist is now CHANGELOG-only and only invoked on
+  demand to fill gaps. It does not own commits, tags, pushes, or builds.
 
 ## Parallel sessions
 
-Each orchestrator runs this skill in its own session. Specialist commits + tags + pushes atomically. If two sessions try same version, first push wins; second sees remote tag conflict and aborts cleanly. Operator picks next version and retries.
+Two operators racing the same version: first `git push origin vX.Y.Z` wins;
+second sees a non-fast-forward / tag-exists error and aborts. Operator picks
+the next version and retries.
 
 ## Don't
 
-- Don't call `sp release prepare` / `sp release publish` as the canonical path. They are deprecated aliases in specialists.
-- Don't bypass `xt release publish` for tag/push unless the command itself is broken.
-- Don't broaden release diffs with source/docs/config changes. File a separate bead for non-release work.
-- Don't pre-stage unrelated files. The release scope guard should see a clean tree except allowed release artifacts.
+- Don't call `xt release prepare` / `xt release publish` — retired. See
+  `unitAI-g29jv`.
+- Don't broaden the release diff with source/docs/config changes. File a
+  separate bead.
+- Don't pre-stage unrelated files. The release scope check should see a clean
+  tree except whitelisted release artifacts.
+- Don't invent CHANGELOG entries from `git log -p`. The reports + existing
+  `[Unreleased]` are the input.
+- Don't `git push --force` or rewrite tags. If a release ships wrong, cut a
+  patch.

package/.xtrm/skills/default/security-pipeline/SKILL.md

@@ -0,0 +1,192 @@
+---
+name: security-pipeline
+description: Bootstrap a complete security pipeline (Dependabot + OSV + Semgrep + gitleaks + pre-commit hooks + Codex review) on any GitHub repo. Designed for free user-private repos where GitHub Advanced Security is unavailable. Reusable across Python/TypeScript/Go/Rust stacks.
+---
+
+# Security Pipeline
+
+Wires a 4-layer security baseline onto any GitHub repo. Originally proven on
+the Mercury infra stack but the templates and bootstrap script are
+project-agnostic — adapt the allowlists and dependabot ecosystems per repo.
+
+## When to use
+
+- Setting up security on a new repo (any language)
+- Existing repo has zero/partial security checks
+- User says "set up security pipeline" / "wire dependabot + sast + secret scan"
+
+Do NOT use this skill if the repo already has a working `dependabot.yml` AND
+all three workflows (`osv-scanner.yml`, `semgrep.yml`, `gitleaks.yml`).
+
+## Architecture (4 layers)
+
+```
+git commit  ──► pre-commit  (gitleaks staged, ruff, hygiene)        ~1s
+git push    ──► pre-push    (semgrep diff-only, osv, anti-main)     ~30s
+PR opened   ──► CI          (osv-scanner, semgrep, gitleaks)        ~1m
+PR review   ──► Codex       (semantic AI review, optional)          ~2m
+PR merged   ──► Dependabot  (continuous vuln + version PRs)         async
+```
+
+Pre-existing debt is NEVER blocked by the push gate — only NEW findings vs
+`origin/main`. CI does the full-repo authoritative scan.
+
+## Why this stack on free user-private repos
+
+GitHub Advanced Security (CodeQL, Dependency Review) needs Org/Enterprise
++ ~$49/user/month. Free substitutes:
+
+| GHAS | Free substitute |
+|---|---|
+| CodeQL | Semgrep `p/security-audit` + `p/secrets` + ecosystem packs |
+| Dependency Review | `osv-scanner` action |
+| Secret scanning | Native (free for all repos since 2025) |
+| Push protection | Native (free for all repos since 2025) |
+| Branch protection enforcement | Pre-push hook + `gh pr merge --auto` |
+
+## Quickstart
+
+The skill ships with a bootstrap script that detects ecosystems and copies
+templates. From the source repo (where this skill is installed):
+
+```bash
+./scripts/security-bootstrap.sh /path/to/target/repo
+```
+
+The script:
+1. Detects ecosystems (`pip`, `pip-pyproject`, `npm`, `docker`, `gomod`,
+   `cargo`, `github-actions`)
+2. Generates a tailored `.github/dependabot.yml`
+3. Copies the 11 baseline files from `templates/`
+4. Opens a `feat(security)` PR
+5. Calls `gh api` to enable Dependabot/secret scanning/push protection
+
+## Manual follow-up after bootstrap
+
+The script CAN'T do these — operator walks them per target repo:
+
+1. **Codex Connector** (optional) — install at https://chatgpt.com/codex/cloud/settings/general
+2. **Branch protection rule** — Settings → Branches → classic rule for `main`
+   - Required checks: `OSV scan`, `Semgrep scan`, `Gitleaks scan`
+   - On free private repos rules don't enforce server-side; the pre-push hook fills the gap
+3. **`make install-hooks`** in the target clone (or run `git config core.hooksPath .githooks`)
+
+## Files in `templates/`
+
+| Template | Lands at | Purpose |
+|---|---|---|
+| `.github/workflows/osv-scanner.yml` | same path | Vuln scan via OSV.dev |
+| `.github/workflows/semgrep.yml` | same path | SAST (replaces CodeQL) |
+| `.github/workflows/gitleaks.yml` | same path | Secret scan |
+| `.gitleaks.toml` | same path | **Allowlist — adapt per project** (see below) |
+| `.semgrepignore` | same path | **Excludes — adapt per project** (see below) |
+| `.pre-commit-config.yaml` | same path | Two-stage local gate |
+| `.githooks/pre-push.template` | merge into existing `.githooks/pre-push` | Anti-main-push + pre-commit chain |
+| `scripts/semgrep-diff.sh` | same path | Diff-only semgrep for push |
+| `scripts/security-scan.sh` | same path | Local audit (informational) |
+
+`.github/dependabot.yml` is NOT in `templates/` — it's generated per-repo from
+detected ecosystems.
+
+## Adapting allowlists per project
+
+The shipped `.gitleaks.toml` and `.semgrepignore` contain Mercury-specific
+paths as **examples**. When applying to a non-Mercury repo, prune what
+doesn't apply.
+
+### `.gitleaks.toml` — common allowlist patterns
+
+```toml
+[allowlist]
+paths = [
+    '''^\.env$''',           # gitignored secrets (no-git scan walks fs)
+    '''^\.env\..*''',
+    # Project-specific machine-generated dirs (drop what doesn't apply):
+    '''^\.beads/.*''',       # Mercury-only — issue tracker exports
+    '''^\.specialists/.*''', # Mercury-only — specialist runtime state
+    '''^\.dolt/.*''',        # Mercury-only — Dolt SQL storage
+    # Add your own:
+    '''^vendor/.*''',        # Go vendoring
+    '''^node_modules/.*''',  # NPM (usually gitignored anyway)
+]
+```
+
+### `.semgrepignore` — common patterns
+
+```
+.env
+.env.*
+node_modules/
+vendor/
+**/__pycache__/
+**/test_fixtures/
+package-lock.json
+pnpm-lock.yaml
+yarn.lock
+poetry.lock
+Pipfile.lock
+go.sum
+Cargo.lock
+```
+
+Don't blanket-allowlist findings without a tracked issue explaining why.
+Acknowledged debt should be visible.
+
+## Local install (per-clone, after bootstrap merges)
+
+```bash
+pip3 install --user --break-system-packages pre-commit semgrep
+mkdir -p ~/.local/bin
+curl -sL https://github.com/gitleaks/gitleaks/releases/download/v8.21.2/gitleaks_8.21.2_linux_x64.tar.gz \
+  | tar -xz -C ~/.local/bin gitleaks
+curl -sL https://github.com/google/osv-scanner/releases/download/v2.0.2/osv-scanner_linux_amd64 \
+  -o ~/.local/bin/osv-scanner && chmod +x ~/.local/bin/osv-scanner
+git config core.hooksPath .githooks
+chmod +x .githooks/pre-commit .githooks/pre-push 2>/dev/null
+```
+
+Verify: `./scripts/security-scan.sh`.
+
+## Reading Codex feedback on a PR (if Codex is installed)
+
+```bash
+gh pr view <num> --json reviews,comments | python3 -c "
+import json, sys
+d = json.load(sys.stdin)
+for r in d.get('reviews', []):
+    if 'codex' in r.get('author',{}).get('login','').lower():
+        body = r.get('body', '')
+        print('👍 no suggestions' if 'automated review suggestions' in body and len(body) < 1500 else body[:1500])
+"
+```
+
+## Known pitfalls (encoded in the templates)
+
+- **Pre-commit can't install with `core.hooksPath` set** → templates chain
+  pre-commit from `.githooks/pre-commit` and `.githooks/pre-push` instead of
+  using `pre-commit install`.
+- **Semgrep's pre-commit env breaks on Python 3.13** (`pkg_resources` missing)
+  → templates use `language: system` pointing at globally installed semgrep.
+- **`semgrep ci --error` is invalid** → use `semgrep scan --error`.
+- **`actions/dependency-review-action` requires GHAS** → use `osv-scanner` instead.
+- **Gitleaks action needs `pull-requests: write`** to post leak summary on PRs.
+- **Full-repo semgrep at push stage flags pre-existing debt** → use
+  `scripts/semgrep-diff.sh` with `--baseline-commit=$(git merge-base HEAD origin/main)`.
+- **`.pre-commit-config.yaml` `default_stages: [pre-commit]`** → otherwise
+  ruff/hygiene hooks fire at push too.
+- **Squash-merging while iterating with `git commit --amend`** → verify
+  `git log --stat <merge-sha>` after merge; missing files require a follow-up PR.
+- **Auto-merge disabled** → fall back to `gh pr merge --squash --delete-branch`
+  after `gh pr checks --watch`.
+
+## Complementary tools (optional, second opinion)
+
+- `trivy fs` / `trivy image` — container + IaC scanning
+- `bandit` — Python-specific SAST (Semgrep `p/python` already covers most)
+- `actionlint` — GitHub Actions linter (Semgrep `p/github-actions` covers basics)
+
+## Reference doc
+
+Full pipeline narrative + UI screenshots + per-feature rationale lives in
+the Mercury reference at `mercury-infra/SECURITY-PIPELINE.md`, also mirrored
+in `~/second-mind/3-resources/github/SECURITY-PIPELINE.md`.