pi-dev 0.1.5 → 0.1.7

package/dist/manifest.js

@@ -11,6 +11,7 @@ export const SKILLS = [
     { name: "do", kind: "human", summary: "Do the engineering work end-to-end." },
     { name: "taste", kind: "human", summary: "View / update / onboard preferences." },
     { name: "where", kind: "human", summary: "Recall prior pi sessions for this cwd." },
+    { name: "improve-skill-flow", kind: "human", summary: "Audit pi session telemetry and propose evidence-based SKILL.md edits." },
     // Auto-invoked support skills
     { name: "migrate", kind: "support", summary: "Strict migration gate before /do can run." },
     { name: "setup", kind: "support", summary: "Scaffold issue-tracker / triage / domain docs." },

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-dev",
-  "version": "0.1.5",
+  "version": "0.1.7",
   "description": "An autonomous engineering skill framework for the pi runtime — built on Matt Pocock's skills.",
   "type": "module",
   "bin": {

package/skills/improve-skill-flow/SKILL.md

@@ -0,0 +1,302 @@
+---
+name: improve-skill-flow
+description: Analyse real pi-runtime session telemetry from a consumer repo to find where the engineering skills (especially /do's chain) drift from their stated contract, then propose evidence-anchored edits to the SKILL.md files. Use when the user wants to improve, audit, debug, or evolve the pi-dev skill framework itself based on what actually happened in real sessions ("스킬 개선하자", "do 가 왜 멈춰", "히스토리 보고 분석해서 스킬 고치자", "메타 스킬 작업", etc).
+---
+
+# /improve-skill-flow — Meta-skill for evidence-based skill improvement
+
+The pi-dev skills are pure markdown. They get better by reading what real sessions did, comparing that to what the SKILL.md said *should* happen, and editing the gap closed. This skill is the canonical loop for that.
+
+The point: **never edit a SKILL.md from gut feeling.** Edit because session N showed phase P violated predicate Q on M occasions, and here is the line that would have prevented it.
+
+## When to run
+
+- A consumer repo has accumulated at least one real day of pi sessions (≈ 3+ `.jsonl` files).
+- A specific skill is suspected of misbehaving ("why does `/do` keep stopping?").
+- After landing a skill change, to verify the next session(s) actually follow the new wording.
+- Periodically (every N releases) as a regression sweep across all human-facing skills.
+
+## What this skill is NOT
+
+- Not for analysing the *codebase* of the consumer repo — that is `improve-codebase-architecture`.
+- Not for shipping engineering work — it does not invoke `/do`. Findings turn into proposed SKILL.md diffs, which are committed via the normal release-please flow on `pi-dev`.
+- Not a real-time monitor — it reads completed session files.
+
+## Inputs
+
+- **Target repo path** (or its sessions directory). Defaults: the user names a repo; you resolve it.
+- Optional: a specific skill name to focus the audit on (`do`, `migrate`, `triage`, …).
+- Optional: a date range.
+- **Install scope** for any fixes that come out of the audit — `global` or `project`. Auto-detected (see Step 5.5); user can override per finding.
+
+## Install scopes
+
+pi-runtime today loads skill bodies from a single location: `~/.pi/agent/skills/<name>/SKILL.md`. The framework's 3-layer override is on **preferences**, not on SKILL bodies. So a finding lands in one of two places:
+
+| scope | lands in | reaches | propagation | when to pick |
+| --- | --- | --- | --- | --- |
+| **global** | `pi-dev`'s `skills/<name>/SKILL.md` | every consumer after the next `npx pi-dev update` | release-please → npm publish | the SKILL.md wording itself is wrong; gap shows up generically |
+| **project** | consumer repo's `docs/agents/preferences.md` (Project taboos / Diagnosis posture / Local-live playbook / Free notes — whichever section fits) | only this repo, on every `/do` bootstrap | regular consumer-repo commit | gap is the repo's domain / paths / conventions, not the SKILL.md |
+
+Notes:
+
+- A `global` apply is **always** mirrored into `~/.pi/agent/skills/<name>/` on the operator's machine so the next session picks it up immediately, without waiting for npm.
+- A `project` apply touches no pi-dev files. It is committed to the consumer repo only.
+- A single audit may produce a mix of global and project findings. Decide scope per finding, not per audit.
+
+## Session-data location & format
+
+pi-runtime persists every session as JSON Lines at:
+
+```
+~/.pi/agent/sessions/--<cwd-with-slashes-replaced-by-dashes>--/<ISO-ts>_<uuid>.jsonl
+```
+
+For example `cwd=/Users/jason/dev/sandbox/hugn` → `~/.pi/agent/sessions/--Users-jason-dev-sandbox-hugn--/`.
+
+Each line is one of these record types:
+
+| `type` | meaning |
+| --- | --- |
+| `session` | session header — has `cwd`, `timestamp`, `version`, `id` |
+| `model_change` | provider / modelId switch |
+| `thinking_level_change` | reasoning effort knob |
+| `message` | a user or assistant turn |
+
+`message.content` is a **list of blocks**, each block has a `type`:
+
+| block `type` | shape | what it represents |
+| --- | --- | --- |
+| `text` | `{text}` | plain text from either side |
+| `thinking` | `{thinking, thinkingSignature}` | model scratchpad (assistant only) |
+| `toolCall` | `{id, name, arguments}` | **pi uses this** — NOT Anthropic SDK's `tool_use` |
+| `toolResult` | `{id, output}` | **pi uses this** — NOT `tool_result` |
+
+Tool names are **lower-case** (`bash`, `read`, `edit`, `write`, `glob`, `grep`, etc.). Build any parser around `toolCall` / `toolResult` first, then fall back to Anthropic-shaped blocks for robustness.
+
+**Critical detail:** the pi runtime injects each skill's `SKILL.md` content into the conversation as a `<skill name="..." location="...">…</skill>` block embedded inside a **user-role** message (system-side injection, but the role is user). This is how you tell the classifier loaded a skill. Count these to see what `/do` actually picked.
+
+Timestamps on `message` records are ISO strings; some other record types use int millis. Handle both.
+
+## Process
+
+### 1 — Scope and load
+
+Ask the user (one round, only if not already specified):
+
+- target repo (or "all repos with sessions")
+- a skill to focus on, or "everything"
+- a date range or "all"
+
+Resolve the sessions directory. List the `.jsonl` files with size + line count so the user can see the input scale.
+
+### 2 — Build the raw signal table
+
+Run a single pass over every targeted `.jsonl` and tally:
+
+**Per session:**
+- user messages, assistant messages, tool calls, thinking blocks
+- duration (last ts − first ts)
+
+**Aggregates:**
+- tool-use frequency (`bash`, `read`, `edit`, `write`, …)
+- `<skill name="X">` injections per skill (this is the classifier's actual decision)
+- skills mentioned in user text vs. assistant text (request side vs. recall side)
+- top Edit / Write targets (hotspot files)
+- bash commands matching domain-specific danger / smell patterns
+
+**Skill-flow specific:**
+- After each `<skill name="do">` injection, did another skill get injected within the same session? **This measures `/do` chain depth.** Single-step chains are a red flag.
+- Count user messages shorter than ~80 chars — these are usually nudges ("진행해", "다음은?", "끝났어?"). High proportion = `/do` is handing the flow back too often.
+- Count user messages containing correction markers (`아니`, `wait`, `stop`, `취소`, `다시`, `그만`, `undo`, `revert`). These mark interventions.
+- Count `<!-- migrated: ... -->` marker date vs. any post-marker `docs/handoff/` / `.scratch/flow/` / `SESSION_*.md` writes. Drift = handoff lockout failed.
+- Count tracker writes (`gh issue create`, `gh pr create`) and check whether `generated by AI` (the `/triage` disclaimer) appears in the surrounding text. Missing disclaimer = `/to-issues` / `/triage` predicate violated.
+
+Use a deterministic Python or shell script you write once and check into `/tmp` for the duration of the run. Do not eyeball big JSONLs by hand.
+
+### 3 — Cross-reference with repo state
+
+For the same date range, pull:
+
+- `git log --since=<start> --pretty=format:"%h %ad %s"` — commit cadence vs. the predicate `auto-commit-per-slice`.
+- `gh issue list / pr list` (if GitHub) — slice/PR shape vs. `default-issue-style=vertical-slice`.
+- Existence of forbidden paths from `docs/agents/preferences.md` taboos (`docs/handoff/`, `.scratch/flow/`, etc.).
+- `gh label list` filtered against taboo-marked labels.
+
+Cross-reference each signal against:
+
+- The skill's **terminal predicate** in `do/SKILL.md` → "Phase contracts".
+- The repo's **`docs/agents/preferences.md`** taboos and `auto-*` settings.
+- The hard rules in the skill being audited.
+
+### 4 — Score the gaps
+
+Produce a small table per audited skill. Each row:
+
+```
+| signal                         | observed | expected (predicate / rule)        | severity |
+| ------------------------------ | -------- | ---------------------------------- | -------- |
+| /do → next-skill chain depth   | 1/5      | ≥1 per chain step (M phases)       | 🔴       |
+| AI disclaimer on slice issues  | 0/8      | every issue starts with disclaimer | 🔴       |
+| post-marker handoff writes     | 9        | 0                                  | 🟡       |
+| ...                            |          |                                    |          |
+```
+
+Severity rule of thumb:
+- 🔴 — predicate is violated repeatedly AND the rule is binding (in "Hard rules" or "Phase contracts").
+- 🟡 — taboo or preference violated but the skill's wording is soft / advisory.
+- 🟢 — observed behaviour matches the spec; do not propose a change.
+
+### 5 — Anchor each finding to an evidence excerpt
+
+For every 🔴 / 🟡 row, quote the smallest piece of evidence that makes the gap undeniable:
+
+- a user message timestamp + first 80 chars,
+- a tool-call command that violates a taboo,
+- a missing disclaimer in a created issue body.
+
+If a finding cannot be backed by an excerpt, it is not actionable yet — demote to a TODO and keep digging.
+
+### 5.5 — Decide install scope per finding (auto + user-overridable)
+
+For each 🔴 / 🟡 finding, pick a default scope using this two-step heuristic, then show the table to the user once and let them flip individual rows before applying.
+
+**Step A — detect operator context.** Run once at the start of this step:
+
+```bash
+origin=$(git -C "$PWD" remote get-url origin 2>/dev/null || echo "")
+pkg_name=$(jq -r '.name // empty' package.json 2>/dev/null)
+is_maintainer=false
+case "$origin" in *pi-dev*|*pi-dev.git) is_maintainer=true ;; esac
+[ "$pkg_name" = "pi-dev" ] && is_maintainer=true
+echo "operator_context=$([ \"$is_maintainer\" = true ] && echo maintainer || echo consumer)"
+```
+
+- `operator_context=maintainer` → cwd is the pi-dev repo itself; the release path is available.
+- `operator_context=consumer` → cwd is a downstream repo; no release path. `global` findings here become "draft a patch + open an upstream PR / issue" rather than "push and release".
+
+**Step B — score each finding.** Default to `global` if the finding matches **any** of:
+
+- Cites SKILL.md wording / phase / predicate / rule numbers.
+- The proposed fix is a generic anti-pattern string, a terminator literal, a runway line, or a lockout that any repo would benefit from.
+- The same gap would plausibly show up in two or more consumer repos.
+
+Default to `project` if the finding matches **any** of:
+
+- Cites a repo-specific path (`src/core/...`, `bin/...-smoke.ts`), brand, schema, table, or domain term.
+- The fix is a taboo, a smoke convention, an env / boot detail, or a glossary entry.
+- The fix would not apply (or would be wrong) in another consumer repo.
+
+Present the scope-decision table:
+
+```
+| # | finding (short)                          | default scope | target file                          | flip? |
+| - | ---------------------------------------- | ------------- | ------------------------------------ | ----- |
+| 1 | /do hands flow back between phases       | global        | pi-dev:skills/do/SKILL.md            |       |
+| 2 | docs/handoff/ resurrected after marker   | global        | pi-dev:skills/migrate/SKILL.md       |       |
+| 3 | retro-action-item label still alive      | project       | hugn:docs/agents/preferences.md      |       |
+| 4 | smoke command name changed in S058       | project       | hugn:docs/agents/preferences.md      |       |
+```
+
+Ask the user once: "OK to proceed with these scopes? Reply with row numbers to flip, or `go`." Apply their flips and move on. If `operator_context=consumer`, any rows still marked `global` get the suffix `(via upstream PR — cannot release locally)` and the apply step adjusts accordingly.
+
+### 6 — Propose edits (per-finding, scoped)
+
+For each 🔴 / 🟡 finding, draft the smallest possible edit that, **if it had been in place at session time, would have prevented the gap.** The shape of the draft depends on the scope from Step 5.5:
+
+**Global findings (target: pi-dev SKILL.md):**
+
+- Edit a **rule** or a **step**, not a flavour sentence. The model must be able to detect the constraint in its own draft output.
+- Prefer **explicit anti-pattern strings** ("Do not say 'shall I continue?'") over abstract injunctions ("be decisive"). The hugn-2026-05 audit showed that named anti-patterns work.
+- Prefer **terminal markers** ("the summary's last line must be one of these two literals: …") over qualitative descriptions of "good wrap-up".
+- Update **at most three skills per run.** More than that means findings aren't anchored well enough.
+
+**Project findings (target: consumer's `docs/agents/preferences.md`):**
+
+- Pick the *narrowest* existing section that fits before adding a new one. Mapping:
+
+  | finding flavour | preferences section |
+  | --- | --- |
+  | forbidden path / file / module / command | `## Project taboos` |
+  | label / state / triage rule | `## Side-effect gates` or `## Project taboos` |
+  | smoke / test / endpoint convention | `## Smoke / test conventions` |
+  | boot / env / probe change | `## Local-live playbook` |
+  | error taxonomy / 5-axes / domain rule | `## Diagnosis posture` |
+  | glossary / context term clarification | `## Glossary alignment` |
+  | rationale that doesn't fit elsewhere | `## Free notes` (one paragraph max, dated) |
+
+- One bullet per finding. Reference the evidence ticket ("S058 smoke name", "#103 missing disclaimer") so the line stays auditable.
+- Do **not** invent new top-level sections unless three findings legitimately share one.
+
+Show all drafts as one unified diff per target file before applying. Group by target file: pi-dev's `skills/<name>/SKILL.md` first (global), then consumer's `docs/agents/preferences.md` (project).
+
+### 7 — Apply, release, verify (branches on scope)
+
+Run both branches if the audit produced mixed-scope findings. Each branch has its own terminal state.
+
+**7a. Global branch** — only if any finding was approved as `global` **and** `operator_context=maintainer`:
+
+1. From the pi-dev checkout: `git add skills/<name>/SKILL.md && git commit -m "<conventional commit anchoring the evidence>"`. Commit body must cite the signal that motivated each change.
+2. `cp` each edited SKILL.md into `~/.pi/agent/skills/<name>/` so the **next** session anywhere picks up the change immediately (release-please takes a minute and a half).
+3. `git push origin main`; release-please opens the version-bump PR; merge it; npm publish runs automatically.
+4. Confirm `npm view pi-dev@latest version` matches the bumped tag.
+
+**7a' — Global findings when `operator_context=consumer`:** you cannot release. Instead:
+
+1. Stash the proposed diffs to `/tmp/pi-dev-upstream-<date>.patch` with one file per skill.
+2. Open an issue on `pi-dev` (or a PR if the operator has clone+push rights) with the evidence excerpts and the patch attached.
+3. As a hotfix for this machine only, optionally `cp` the edited bodies into `~/.pi/agent/skills/<name>/` and note in the issue that the next `pi-dev update` will overwrite them — which is the desired end state once the upstream change lands.
+
+**7b. Project branch** — only if any finding was approved as `project`:
+
+1. In the consumer repo: edit `docs/agents/preferences.md` per the drafts from Step 6. Keep the migration marker at the very end of the file undisturbed.
+2. Bump the `last-updated` line at the top of the file to today's UTC date.
+3. `git add docs/agents/preferences.md && git commit -m "docs(agents): <one-liner per finding>"`. Conventional Commits apply.
+4. Push per the repo's normal workflow. No release-please involvement — preferences are not packaged.
+
+**Verification (both branches).** After the next pi session in the affected repo:
+
+1. Re-run **this skill** scoped to the last 24 h.
+2. Confirm each previously-🔴 signal has moved (chain depth up, intervention rate down, taboo writes gone, disclaimer coverage at 100%, etc.).
+3. If a signal did not move, the fix wording was too weak — file a follow-up audit, do not re-write from scratch.
+
+## Terminal predicate
+
+This skill is done when **all four** are true:
+
+1. A signal table with severities and evidence excerpts has been presented.
+2. Each finding has an approved scope (`global` / `project` / `defer`) on record, defaulted by Step 5.5 and confirmed by the user.
+3. Either (a) zero 🔴 findings — flow is healthy, recorded as "no change this cycle", OR (b) each 🔴 finding has landed in its scope's target file (or been stashed + filed upstream when an operator-context mismatch prevents release).
+4. For any landed change: if `global` and `maintainer`, the npm version has bumped (`npm view pi-dev@latest version`); if `project`, the consumer repo has the commit on its push-stream. Either way, the next-session re-audit plan is stated.
+
+The summary's **last line** must be one of:
+
+```
+audit complete — no changes this cycle.
+```
+
+```
+audit complete — global v<X.Y.Z> released, project commit <sha>, next re-audit after the next session.
+```
+
+```
+audit complete — upstream issue <#N> filed, project commit <sha>, hotfix mirrored to ~/.pi.
+```
+
+## What this skill does not do
+
+- It does not modify a consumer repo's code, issues, or preferences. It only edits **pi-dev's own `skills/`**.
+- It does not invent gaps from first principles. Every finding must come from a session excerpt or a repo-state probe.
+- It does not run faster than the data allows — if there is only one session, run it but say so up front; the signal is noisy.
+
+## Heuristics
+
+- **One screenful of signals beats a dashboard.** Most of the time three or four numbers tell the whole story.
+- **Watch the gap between `/do` injection and next-skill injection.** That is the single highest-information ratio about whether the chain orchestrator is actually orchestrating.
+- **Short user messages are the cheapest interruption proxy.** Count them.
+- **A taboo without a `.gitignore` lockout will resurrect.** If the same taboo file shows up across two audits, the fix belongs in `/migrate`, not `/do`.
+- **Disclaimer / terminator / lockout / runway** are the four "tripwire" patterns that catch silent contract drift. Reach for them before inventing new ones.
+
+## Why this skill exists
+
+Skills are prose. Prose drifts. Without a feedback loop, the SKILL.md files become wishful thinking that the agent ignores in real sessions. This skill is the loop.