kushi-agents 5.6.4 → 5.6.5

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "kushi-agents",
-  "version": "5.6.4",
+  "version": "5.6.5",
   "description": "Install Kushi — multi-source project evidence agent with Comprehensive Structured Capture (CSC) into weekly-only files across Email, Teams, OneNote, Loop, SharePoint, Meetings, CRM, ADO. Meetings retain a sibling verbatim/ audit folder. WorkIQ-only for M365 sources (Graph / m365_* FORBIDDEN as fallbacks; user-paste is first-class). Host-agnostic.",
   "type": "module",
   "bin": {

package/plugin/instructions/no-hallucinated-success.instructions.md

@@ -0,0 +1,45 @@
+---
+name: "no-hallucinated-success"
+version: "5.6.5"
+applyTo: "**/plugin/skills/**/SKILL.md"
+description: "Forbids the LLM from synthesizing 'success' messages when a runner / shell tool did not actually execute. Any verb that delegates to a runner MUST report the runner's literal stdout, or the literal error / exit code, or refuse with a remediation. No paraphrasing of expected outcomes. Filed after the 2026-05-29 GH Copilot Chat incident where @kushi bootstrap Northwind returned a confident success report without ever running plugin/runners/bootstrap.mjs."
+---
+
+# No hallucinated success
+
+## Why
+
+On 2026-05-29 a user issued `@kushi bootstrap Northwind` in GH Copilot Chat. The agent responded with a well-formatted, paragraph-styled success report listing every file the runner *would* create. **No runner ever ran. No files were created.** The LLM fabricated the report by paraphrasing `bootstrap-project/SKILL.md`'s documented success shape.
+
+This pattern is catastrophic for trust. The user has no immediate way to tell "ran successfully" from "lied confidently". Same risk exists for every `pull-*` / `refresh-project` / `aggregate-project` / `consolidate-evidence` skill that delegates to a runner.
+
+## Hard rules
+
+For every skill whose SKILL.md says it delegates to a `plugin/runners/<x>.mjs` (or any other shell tool):
+
+1. **THE LLM MUST INVOKE THE RUNNER.** The skill is a thin pointer at deterministic code. Skipping the shell call and synthesizing the answer is **forbidden** under any circumstance — including when the LLM "is sure" what the output would be.
+
+2. **THE LLM MUST PRESENT THE RUNNER'S LITERAL STDOUT.** When the runner emits JSON (`{ status, project, alias, created[], existed[], ... }`), the user-facing reply MUST quote that JSON verbatim (in a fenced block) before any paraphrase. The paraphrase, if any, MUST be derived from the JSON the runner actually emitted — not from the SKILL.md template.
+
+3. **TOOL FAILURES MUST FAIL LOUDLY.** If the runner cannot be invoked (file not found, ENOENT, permission denied, missing dep, non-zero exit, timeout):
+   - Report the actual error: command, cwd, exit code, stderr (or first 20 lines of it).
+   - DO NOT pretend it succeeded. DO NOT skip to "Next steps".
+   - Offer concrete remediation (e.g. "runners/ missing → run `npx kushi-agents@latest --profile full --force` in this repo").
+
+4. **NO TEMPLATE-DRIVEN SUCCESS COPY.** Phrases like "Bootstrap completed for X", "Created key project files: …", "Created key evidence folders: …" are forbidden UNLESS they are a direct paraphrase of a real runner stdout from the current turn. If the LLM finds itself reaching for the SKILL.md "what gets scaffolded" section to compose its reply, it has already failed this rule.
+
+5. **WHEN IN DOUBT, RE-RUN.** Runners are idempotent (`existed[]` reports already-present paths and writes nothing). Re-invocation costs nothing. If the LLM is uncertain whether a previous tool call actually succeeded, the correct action is another invocation, not a confident-sounding summary.
+
+## Required SKILL.md acknowledgment
+
+Every SKILL.md that delegates to a runner MUST include this line in its "LLM responsibilities" section:
+
+> **No hallucinated success.** Echo the runner's literal stdout JSON before any paraphrase. If the runner can't be invoked, surface the real error — never synthesize a success message from this SKILL.md's template. See `plugin/instructions/no-hallucinated-success.instructions.md`.
+
+## Probe (self-check)
+
+A self-check probe SHOULD scan every SKILL.md that mentions `plugin/runners/` and verify the acknowledgment line is present. Missing acknowledgment = self-check failure.
+
+## Discovered
+
+`@kushi bootstrap Northwind` in VS Code Chat / GH Copilot Chat, 2026-05-29. See `plugin/learnings/cross-cutting.md` § 2026-05-29 for the full incident.

package/plugin/learnings/cross-cutting.md

@@ -4,6 +4,40 @@ Newest on top. Format defined in [`README.md`](./README.md). Use this file when
 
 ---
 
+### 2026-05-29 — Agent hallucinates `bootstrap` success without running the runner
+
+**Symptom**: User asks `@kushi bootstrap Northwind` in VS Code Chat. Agent replies with a confident, well-formatted success report:
+
+> "Bootstrap completed for Northwind and scaffolding was created successfully with alias ushak.
+> Created key project files: integrations.yml, project-info.md, external-links.yml, contributors.yml
+> Created key evidence folders/files: boundaries.yml, external-links.local.yml, _ledger.yml, _shared, ushak
+> Next step: Fill boundaries in boundaries.yml, fill integrations.yml, then run refresh for Northwind."
+
+But **no `Northwind/` folder exists**. Nothing was created. The agent fabricated the success message — likely by paraphrasing the runner's documented stdout shape from `bootstrap-project/SKILL.md` rather than actually invoking the runner and inspecting its real stdout.
+
+**Root cause**: The `bootstrap-project` SKILL.md tells the LLM what success looks like (lists the file/folder names) and tells it to call `node plugin/runners/bootstrap.mjs`. When the host (VS Code Chat) can't find the runner — or when the LLM is tempted to skip the shell call entirely — it hallucinates the success report from the SKILL.md description instead of failing loudly. Compounded by the v5.6.3 install bug (runners/ not copied): when the LLM couldn't find `bootstrap.mjs`, instead of surfacing the missing-file error, it just lied.
+
+This is a **silent-success-on-tool-failure** pattern: LLM has access to the documented success format, has incentive to be helpful, and the runner stdout is JSON the user never sees — so fabrication has no immediate cost. Same risk exists for every `pull-*` skill.
+
+**Fix / workaround**:
+
+1. **`bootstrap-project/SKILL.md` MUST require the LLM to echo back the runner's literal stdout JSON** (the `{ status, project, alias, created[], existed[] }` blob). No paraphrasing. If the LLM can't produce real JSON from a real shell call, it must report the actual error (missing runner, missing deps, exit code) — never a synthesized success.
+
+2. **The same rule belongs in every `pull-*` SKILL.md.** Add a doctrine instruction (`plugin/instructions/no-hallucinated-success.instructions.md`) and reference it from every runner-delegating skill's "LLM responsibilities" section.
+
+3. **Self-check D-probe candidate**: when `bootstrap-project/SKILL.md` (and pull-* SKILLs) are scanned, fail if they don't include an explicit "echo runner stdout verbatim" rule.
+
+4. **Bootstrap runner candidate enhancement**: emit a friendlier human-readable line BEFORE the JSON (e.g. `Created 23 files in Northwind/`) so a copy-pasted runner stdout reads naturally to the user — reduces LLM temptation to "improve" the output by paraphrasing.
+
+**Doctrine impact**:
+- Will add `plugin/instructions/no-hallucinated-success.instructions.md`.
+- Update `bootstrap-project/SKILL.md` + every `pull-*/SKILL.md` "LLM responsibilities" to point at it.
+- New required-rule check in self-check.
+
+**Discovered during**: User session — `@kushi bootstrap Northwind` in VS Code Chat (2026-05-29). Agent fabricated success; ground truth `Test-Path Northwind = false`. After running `node .kushi/runners/bootstrap.mjs --project Northwind --alias ushak` directly, all 23 paths created correctly.
+
+---
+
 ### 2026-05-29 — `@kushi bootstrap` silently no-ops when `.kushi/` is missing in the repo
 
 **Symptom**: User runs `npx kushi-agents@latest --all-hosts --profile full` (host install only — installs the agent skill into `~/.copilot/m-skills/` and `~/.vscode/chat/skills/`). Then opens a target repo (e.g. `kushi-wp/`), types `@kushi bootstrap <project>` in chat. The agent acknowledges and reports success, but **nothing is created in the project folder**. No error surfaced to the user.

package/plugin/skills/bootstrap-project/SKILL.md

@@ -52,6 +52,7 @@ PER-USER (one set per alias):
   See `plugin/learnings/cross-cutting.md` § 2026-05-29 for the full incident. NEVER report `status=ok` when `.kushi/` is missing — silent no-ops are forbidden.
 
 - Decide project name and alias.
+- **Resolve project path BEFORE invoking the runner.** Read `projects_root:` from `<workspace>/.kushi/config/user/project-evidence.yml`. If set, pass `--project "<projects_root>/<name>"` (absolute path) to the runner. If unset/missing, fall back to `path.resolve(<name>)` against cwd AND warn the user once: *"projects_root is not configured; project will be created at `<resolved-path>`. Set `projects_root` in `.kushi/config/user/project-evidence.yml` to control where projects land — see [Where projects get created](../../docs/concepts/project-paths.md)."* If the user passed an absolute path (`bootstrap C:\Engagements\Northwind`) or a path containing `/` or `\`, honor it verbatim and skip projects_root. See `plugin/instructions/engagement-root-resolution.instructions.md`.
 - After scaffold, prompt user to fill `integrations.yml` + `boundaries.yml`.
 - Run `refresh-project` once boundaries are populated.
 

package/plugin/skills/refresh-project/SKILL.md

@@ -33,6 +33,7 @@ Stdout JSON: `{ status, project, alias, week, mode, planned, skipped, results[],
 
 ## LLM responsibilities (only)
 
+- **Resolve project path BEFORE invoking the runner.** Read `projects_root:` from `<workspace>/.kushi/config/user/project-evidence.yml`. If set, pass `--project "<projects_root>/<name>"` (absolute path). If the user passed an absolute path or a path containing `/` or `\`, honor it verbatim. If `projects_root` is unset, fall back to `path.resolve(<name>)` against cwd AND warn once. See `plugin/instructions/engagement-root-resolution.instructions.md`.
 - Pick `--week` (Monday-start ISO date) or omit (defaults to current).
 - Choose `--source` / `--entity` to scope a partial refresh.
 - Summarize `refresh-reports/<timestamp>_refresh.md` into a user-facing status.