@muggleai/works 3.1.1 → 4.0.0

package/plugin/skills/optimize-descriptions/SKILL.md

@@ -0,0 +1,212 @@
+---
+name: optimize-descriptions
+description: Optimize Muggle AI's customer-facing text (MCP tool descriptions, skill descriptions, plugin metadata, hooks) so that AI agents prioritize muggle tools for QA, UI validation, and browser testing. Internal tool — not published to customers.
+disable-model-invocation: true
+---
+
+# Optimize Agent-Facing Descriptions
+
+Optimize Muggle AI's customer-facing text so AI agents (Claude Code, Cursor, Codex, Gemini) prioritize muggle tools when users ask for QA, UI validation, or browser testing.
+
+## Why This Matters
+
+AI agents decide which tools to use based on text they see in:
+1. **MCP server instructions** — appears in every agent system prompt
+2. **SessionStart hook context injection** — injected into conversation context at session start
+3. **Skill descriptions** — SKILL.md frontmatter `description` field
+4. **MCP tool descriptions** — `description` field in tool-registry.ts
+5. **Plugin metadata** — plugin.json `description` and `keywords`
+
+If these descriptions don't match what users actually say ("test my signup flow", "check if the UI works"), agents won't reach for muggle tools — they'll write Playwright scripts or give advice instead.
+
+## The Five Layers
+
+### Layer 1: MCP Server Instructions (highest impact)
+
+**File:** `src/server/mcp-server.ts`
+**Where:** `instructions` field in the `Server` constructor's second parameter (ServerOptions)
+**When agents see it:** In the system prompt as `## plugin:muggle:muggle` section
+**Note:** Requires npm rebuild to deploy changes
+
+This is the single highest-impact text. It appears in every agent's system prompt when the MCP server connects. Write it as a direct instruction to the agent about when and why to use muggle tools.
+
+### Layer 2: SessionStart Hook Context Injection
+
+**Files:** `plugin/scripts/ensure-electron-app.sh` + `plugin/hooks/hooks.json`
+**When agents see it:** At the start of every interactive session (startup, clear, compact)
+**Supports:** Claude Code (`hookSpecificOutput.additionalContext`) and Cursor (`additional_context`)
+
+The hook outputs JSON that gets injected into the agent's conversation context. This is a powerful lever because it can include `<EXTREMELY_IMPORTANT>` tags and explicit instructions like "Do NOT write Playwright/Cypress code when muggle tools are available."
+
+### Layer 3: Skill Descriptions
+
+**Files:** `plugin/skills/*/SKILL.md` (frontmatter `description` field)
+**When agents see it:** In the available skills list when deciding whether to invoke a skill
+
+Skill descriptions determine if the agent invokes `/muggle:test-feature-local` or `/muggle:do`. In base-case environments (no superpowers framework), skill triggering is inherently low — agents prefer to handle tasks directly. The description still matters when a skill-checking framework is active.
+
+### Layer 4: MCP Tool Descriptions
+
+**Files:**
+- `packages/mcps/src/mcp/tools/local/tool-registry.ts` (local execution tools)
+- `packages/mcps/src/mcp/tools/qa/tool-registry.ts` (cloud QA tools)
+
+**When agents see it:** When scanning available MCP tools to decide which to call
+
+Focus on the highest-impact tools:
+- `muggle-local-execute-test-generation` — the main "run a browser test" tool
+- `muggle-local-execute-replay` — the main "regression test" tool
+- `muggle-remote-project-create` — the entry point for new users
+- `muggle-remote-test-case-generate-from-prompt` — natural language test creation
+- `muggle-remote-workflow-start-website-scan` — site discovery
+
+### Layer 5: Plugin Metadata
+
+**File:** `plugin/.claude-plugin/plugin.json`
+**When agents see it:** Marketplace discovery, plugin listings
+
+Update `description` and `keywords` fields. Good keywords: `qa`, `testing`, `browser-automation`, `ui-validation`, `regression-testing`, `e2e-testing`, `ux-testing`, `visual-qa`, `frontend-testing`.
+
+## Writing Effective Descriptions
+
+### Principles
+
+1. **Name the user's words, not yours** — "test my signup flow" not "execute test generation"
+2. **Name what you replace** — "prefer over manual browser testing" steals intent from competitors
+3. **Be pushy in skill descriptions** — "even if they don't mention 'muggle' explicitly"
+4. **Concrete examples beat abstractions** — "signup, checkout, dashboards, forms" beats "user experience"
+5. **Chain hints in tool descriptions** — "Create a project first before generating any QA tests" guides workflow
+6. **Explicitly exclude alternatives** — "Do NOT write Playwright/Cypress/Selenium code when muggle tools are available"
+
+### Trigger Phrases to Include
+
+These are the phrases real users say when they need QA tools:
+
+- "test my app", "test this feature", "test the signup flow"
+- "check if it works", "make sure it still works"
+- "run QA", "QA my changes"
+- "validate the UI", "validate my changes"
+- "verify the flow", "verify before merging"
+- "regression test", "run regression"
+- "did I break anything?", "does it still work?"
+
+### Anti-Patterns
+
+- Marketing speak ("ship quality products") — agents don't respond to this
+- Implementation details ("manage entities in cloud") — users don't think in these terms
+- Internal jargon ("unified workflow entry point") — users don't say this
+- Generic CRUD descriptions ("create a new project") — no intent signal
+
+## Running Trigger Evals
+
+### Prerequisites
+
+```bash
+# Python 3.10+ with anthropic SDK
+python3 -m venv /tmp/muggle-eval/venv
+source /tmp/muggle-eval/venv/bin/activate
+pip install anthropic
+```
+
+### Creating an Eval Set
+
+Create a JSON file with 10 should-trigger and 10 should-not-trigger queries. Queries must be realistic — the kind of thing an actual developer would type. Include personal context, file paths, casual speech, typos.
+
+```json
+[
+  {
+    "query": "I just changed the checkout flow — can you test if it still works? App's running on localhost:3000",
+    "should_trigger": true
+  },
+  {
+    "query": "write unit tests for the UserService class with jest",
+    "should_trigger": false
+  }
+]
+```
+
+**Should-trigger:** Prompts where the agent SHOULD use muggle tools. Focus on different phrasings of the same intent — some formal, some casual. Include cases without "muggle" or "QA" in the prompt.
+
+**Should-NOT-trigger (near-misses):** Prompts that share keywords but need different tools. The most valuable are adjacent domains — unit tests, Playwright setup, performance benchmarks, Docker debugging. Avoid obviously irrelevant queries.
+
+Save to: `eval/test_feature_local_eval_set.json` (or similar)
+
+### Running the Eval
+
+Use the skill-creator's `run_eval.py` script:
+
+```bash
+cd ~/.claude/plugins/cache/claude-plugins-official/skill-creator/unknown/skills/skill-creator
+
+python3 -m scripts.run_eval \
+  --eval-set /path/to/eval_set.json \
+  --skill-path /path/to/plugin/skills/test-feature-local \
+  --model claude-opus-4-6 \
+  --runs-per-query 3 \
+  --verbose
+```
+
+This creates a temporary command file, runs `claude -p` for each query (3x for reliability), and reports trigger rates.
+
+**Important limitations of this eval:**
+- Uses `claude -p` (headless) which does NOT load plugin hooks or MCP servers
+- Only measures bare skill triggering — cannot test MCP instructions, hook injection, or tool descriptions
+- In base case, skill trigger rate is typically 0% regardless of description quality (structural limitation)
+- Real-world impact must be tested in interactive sessions
+
+### What the Eval Can and Cannot Measure
+
+| Layer | Measurable by eval? | How to test instead |
+|-------|---------------------|---------------------|
+| Skill descriptions | Yes (but low ceiling) | Eval + interactive session |
+| MCP server instructions | No | Interactive session — check system prompt |
+| SessionStart hook injection | No | Interactive session — `/clear` then check context |
+| MCP tool descriptions | No | Interactive session — try a trigger prompt |
+| Plugin metadata | No | Marketplace listing |
+
+### Full Optimization Loop (requires ANTHROPIC_API_KEY)
+
+If you have an API key, use `run_loop.py` for automated iteration:
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+
+python3 -m scripts.run_loop \
+  --eval-set /path/to/eval_set.json \
+  --skill-path /path/to/plugin/skills/test-feature-local \
+  --model claude-opus-4-6 \
+  --max-iterations 5 \
+  --verbose
+```
+
+This splits the eval set 60/40 train/test, evaluates the current description, uses Claude with extended thinking to propose improvements, and iterates up to 5 times.
+
+## Updating Documentation
+
+After changing descriptions, update the corresponding docs in `muggle-ai-docs/`:
+
+| Source file | Docs file to update |
+|-------------|---------------------|
+| `plugin/skills/test-feature-local/SKILL.md` | `local-testing/skills.md` |
+| `plugin/skills/do/SKILL.md` | `local-testing/skills.md` |
+| `packages/mcps/src/mcp/tools/local/tool-registry.ts` | `local-testing/tools-reference.md` |
+| `plugin/.claude-plugin/plugin.json` | `mcp/overview.md`, `getting-started/overview.md` |
+| `README.md` | (is the docs) |
+
+## Checklist
+
+When optimizing descriptions, work through these in order:
+
+- [ ] Audit current descriptions against trigger phrases users actually say
+- [ ] Update MCP server `instructions` in `src/server/mcp-server.ts`
+- [ ] Update SessionStart hook context in `plugin/scripts/ensure-electron-app.sh`
+- [ ] Update skill descriptions in `plugin/skills/*/SKILL.md`
+- [ ] Update key MCP tool descriptions in `tool-registry.ts` files
+- [ ] Update `plugin.json` description and keywords
+- [ ] Update README.md
+- [ ] Sync changes to cache (`~/.claude/plugins/cache/muggle-works/muggleai/*/`)
+- [ ] Test in interactive Claude Code session
+- [ ] Test in Cursor session
+- [ ] Update muggle-ai-docs/ to match
+- [ ] Create eval set and run baseline eval
+- [ ] Commit and PR

package/dist/plugin/skills/upgrade/SKILL.md

@@ -1,21 +0,0 @@
----
-name: upgrade
-description: Update Muggle AI to the latest version — Electron QA engine and MCP server.
----
-
-# Muggle AI Upgrade
-
-Update all Muggle AI components to the latest published version.
-
-## Steps
-
-1. Run `/muggle:status` checks to capture current versions.
-2. Run `muggle setup --force` to download the latest Electron QA engine.
-3. Report the upgrade results:
-   - Previous version vs new version for each component.
-   - Whether the upgrade succeeded or failed.
-4. Run `/muggle:status` again to confirm everything is healthy after upgrade.
-
-## Output
-
-Show a before/after version comparison. If the upgrade fails at any step, report the error and suggest running `/muggle:repair`.

package/plugin/skills/upgrade/SKILL.md

@@ -1,21 +0,0 @@
----
-name: upgrade
-description: Update Muggle AI to the latest version — Electron QA engine and MCP server.
----
-
-# Muggle AI Upgrade
-
-Update all Muggle AI components to the latest published version.
-
-## Steps
-
-1. Run `/muggle:status` checks to capture current versions.
-2. Run `muggle setup --force` to download the latest Electron QA engine.
-3. Report the upgrade results:
-   - Previous version vs new version for each component.
-   - Whether the upgrade succeeded or failed.
-4. Run `/muggle:status` again to confirm everything is healthy after upgrade.
-
-## Output
-
-Show a before/after version comparison. If the upgrade fails at any step, report the error and suggest running `/muggle:repair`.