safeword 0.2.2 → 0.2.4

package/.safeword/guides/llm-instruction-design.md

@@ -0,0 +1,239 @@
+# Writing Instructions for LLMs
+
+**Context:** When creating documentation that LLMs will read and follow (like AGENTS.md, CLAUDE.md, testing guides, coding standards), different best practices apply than when prompting an LLM directly.
+
+## Core Principles
+
+**1. MECE Principle (Mutually Exclusive, Collectively Exhaustive)**
+
+Decision trees and categorization must have no overlap and cover all cases. Research shows LLMs struggle with overlapping categories—McKinsey/BCG MECE framework ensures clear decision paths.
+
+```markdown
+❌ BAD - Not mutually exclusive:
+├─ Pure function?
+├─ Multiple components interacting?
+├─ Full user flow?
+
+Problem: A function with database calls could match both
+
+✅ GOOD - Sequential, mutually exclusive:
+1. AI content quality? → LLM Eval
+2. Requires real browser? → E2E test
+3. Multiple components? → Integration test
+4. Pure function? → Unit test
+
+Stops at first match, no ambiguity.
+```
+
+**2. Explicit Over Implicit**
+
+Never assume LLMs know what you mean. Define all terms, even "obvious" ones.
+
+```markdown
+❌ BAD: "Test at the lowest level"
+✅ GOOD: "Test with the fastest test type that can catch the bug"
+
+Examples needing definition:
+- "Critical paths" → Always critical: auth, payment. Rarely: UI polish, admin
+- "Browser" → Real browser (Playwright/Cypress), not jsdom
+- "Pure function" → Input → output, no I/O (define edge cases like Date.now())
+```
+
+**3. No Contradictions**
+
+Different sections must align. LLMs don't reconcile conflicting guidance. When updating, grep for related terms and update all references.
+
+```markdown
+❌ BAD:
+Section A: "Write E2E tests only for critical user paths"
+Section B: "All user-facing features have at least one E2E test"
+
+✅ GOOD:
+Section A: "Write E2E tests only for critical user paths"
+Section B: "All critical multi-page user flows have at least one E2E test"
++ Definition of "critical" with examples
+```
+
+**4. Concrete Examples Over Abstract Rules**
+
+Show, don't just tell. LLMs learn patterns from examples. For every rule, include 2-3 concrete examples showing good vs bad.
+
+```markdown
+❌ BAD: "Follow best practices for testing"
+
+✅ GOOD:
+// ❌ BAD - Testing business logic with E2E
+test('discount calculation', async ({ page }) => {
+  await page.goto('/checkout')
+  await page.fill('[name="price"]', '100')
+  await expect(page.locator('.total')).toContainText('80')
+})
+
+// ✅ GOOD - Unit test (runs in milliseconds)
+it('applies 20% discount', () => {
+  expect(calculateDiscount(100, 0.20)).toBe(80)
+})
+```
+
+**5. Edge Cases Must Be Explicit**
+
+What seems obvious to humans often isn't to LLMs. After stating a rule, add "Edge cases:" section with common confusing scenarios.
+
+```markdown
+❌ BAD: "Unit test pure functions"
+
+✅ GOOD: "Unit test pure functions"
+
+Edge cases:
+- Non-deterministic functions (Math.random(), Date.now()) → Unit test with mocked randomness/time
+- Environment dependencies (process.env) → Integration test
+- Mixed pure + I/O → Extract pure part, unit test separately
+```
+
+**6. Actionable Over Vague**
+
+Give LLMs concrete actions, not subjective guidance. Replace subjective terms (most/some/few) with optimization rules + red flags.
+
+```markdown
+❌ BAD: "Most tests: Fast, Some tests: Slow"
+
+✅ GOOD:
+- Write as many fast tests as possible
+- Write E2E tests only for critical paths requiring a browser
+- Red flag: If you have more E2E tests than integration tests, suite is too slow
+```
+
+**7. Decision Trees: Sequential Over Parallel**
+
+Structure decisions as ordered steps, not simultaneous checks. Sequential questions force the LLM through a deterministic decision path.
+
+```markdown
+❌ BAD - Parallel branches:
+├─ Pure function?
+├─ Multiple components?
+└─ Full user flow?
+
+✅ GOOD - Sequential (see Principle 1 example above)
+Answer questions IN ORDER. Stop at the first match.
+```
+
+**8. Tie-Breaking Rules**
+
+When multiple options could apply, tell LLMs how to choose.
+
+```markdown
+✅ GOOD:
+"If multiple test types can catch the bug, choose the fastest one."
+
+Reference in decision trees:
+"If multiple seem to apply, use the tie-breaking rule stated above: choose the faster one."
+```
+
+**9. Lookup Tables for Complex Decisions**
+
+When decision logic has 3+ branches, nested conditions, or multiple variables to consider, provide a reference table.
+
+```markdown
+| Bug Type | Unit? | Integration? | E2E? | Best Choice |
+|----------|-------|--------------|------|-------------|
+| Calculation error | ✅ | ✅ | ✅ | Unit (fastest) |
+| Database query bug | ❌ | ✅ | ✅ | Integration |
+| CSS layout broken | ❌ | ❌ | ✅ | E2E (only option) |
+```
+
+**10. Avoid Caveats in Tables**
+
+Keep patterns clean. Parentheticals break LLM pattern matching. Add separate rows for caveat cases.
+
+```markdown
+❌ BAD: | State management bug | ❌ NO (if mocked) | ✅ YES |
+✅ GOOD: | State management bug (Zustand, Redux) | ❌ NO | ✅ YES |
+```
+
+**11. Percentages: Context or None**
+
+Don't use percentages without adjustment guidance.
+
+```markdown
+❌ BAD: "70% unit tests, 20% integration, 10% E2E"
+
+✅ BETTER: "Baseline: 70/20/10. Adjust: Microservices → 60/30/10, UI-heavy → 60/20/20"
+
+✅ BEST: "Write as many fast tests as possible. Red flag: More E2E than integration = too slow."
+```
+
+**12. Specificity in Questions**
+
+Use precise technical terms, not general descriptions.
+
+```markdown
+❌ BAD: "Does this require seeing the UI?"
+✅ GOOD: "Does this require a real browser (Playwright/Cypress)?"
+
+Note: React Testing Library does NOT require a browser - that's integration testing.
+```
+
+**13. Re-evaluation Paths**
+
+When LLMs hit dead ends, provide concrete next steps.
+
+```markdown
+❌ BAD: "If none of the above apply, re-evaluate your approach"
+
+✅ GOOD: "If testing behavior that doesn't fit the categories:
+1. Break it down: Separate pure logic from I/O/UI concerns
+2. Test each piece: Pure → Unit, I/O → Integration, Multi-page → E2E
+3. Example: Login validation
+   - isValidEmail(email) → Unit test
+   - checkUserExists(email) → Integration test (database)
+   - Login form → Dashboard → E2E test (multi-page)"
+```
+
+## Anti-Patterns to Avoid
+
+❌ **Visual metaphors** - Pyramids, icebergs—LLMs don't process visual information well
+❌ **Undefined jargon** - "Technical debt", "code smell" need definitions
+❌ **Competing guidance** - Multiple decision frameworks that contradict each other
+❌ **Outdated references** - Remove concepts, but forget to update all mentions
+
+## Quality Checklist
+
+Before saving/committing LLM-consumable documentation:
+
+- [ ] Decision trees follow MECE principle (mutually exclusive, collectively exhaustive)
+- [ ] Technical terms explicitly defined
+- [ ] No contradictions between sections
+- [ ] Every rule has 2-3 concrete examples (good vs bad)
+- [ ] Edge cases explicitly covered
+- [ ] Vague terms replaced with actionable principles
+- [ ] Tie-breaking rules provided
+- [ ] Complex decisions (3+ branches) have lookup tables
+- [ ] Dead-end paths have re-evaluation steps with examples
+
+## Research-Backed Principles
+
+- **MECE (McKinsey):** Mutually exclusive, collectively exhaustive decision trees for reliable LLM decisions
+- **Prompt ambiguity (2025):** "Ambiguity is one of the most common causes of poor LLM output" (Zero-Shot Decision Tree Construction)
+- **Concrete examples (2025):** Structured approaches with concrete examples consistently improve performance over "act as" or "###" techniques
+
+## Example: Before and After
+
+**Before (ambiguous):**
+```markdown
+Follow the test pyramid: lots of unit tests, some integration tests, few E2E tests.
+```
+
+**After (LLM-optimized):**
+```markdown
+Answer these questions IN ORDER to choose test type:
+
+1. Pure function (input → output, no I/O)? → Unit test
+2. Multiple components/services interacting? → Integration test
+3. Requires real browser (Playwright)? → E2E test
+
+If multiple apply: choose the faster one.
+
+Edge cases:
+- React components with React Testing Library → Integration (not E2E, no real browser)
+- Non-deterministic functions (Date.now()) → Unit test with mocked time
+```

package/.safeword/guides/llm-prompting.md

@@ -0,0 +1,95 @@
+# LLM Prompting Best Practices
+
+This guide covers two related topics:
+
+**Part 1: Prompting LLMs** - How to structure prompts when actively using an LLM (API calls, chat interactions)
+
+**Part 2: Writing Instructions for LLMs** - How to write documentation that LLMs will read and follow (SAFEWORD.md, CLAUDE.md, testing guides, coding standards)
+
+---
+
+## Part 1: Prompting LLMs
+
+### Prompt Engineering Principles
+
+**Concrete Examples Over Abstract Rules:**
+- ✅ Good: Show "❌ BAD" vs "✅ GOOD" code examples
+- ❌ Bad: "Follow best practices" (too vague)
+
+**"Why" Over "What":**
+- Explain architectural trade-offs and reasoning
+- Include specific numbers (90% cost reduction, 3x faster)
+- Document gotchas with explanations
+
+**Structured Outputs:**
+- Use JSON mode for predictable LLM responses
+- Define explicit schemas with validation
+- Return structured data, not prose
+
+```typescript
+// ❌ BAD - Prose output
+"The user wants to create a campaign named 'Shadows' with 4 players"
+
+// ✅ GOOD - Structured JSON
+{ "intent": "create_campaign", "name": "Shadows", "playerCount": 4 }
+```
+
+### Cost Optimization
+
+**Prompt Caching (Critical for AI Agents):**
+- Static rules → System prompt with cache_control: ephemeral (caches for ~5 min, auto-expires)
+- Dynamic data (character state, user input) → User message (no caching)
+- Example: 468-line prompt costs $0.10 without caching, $0.01 with (90% reduction)
+- Cache invalidation: ANY change to cached blocks breaks ALL caches
+- Rule: Change system prompts sparingly; accept one-time cache rebuild cost
+
+**Message Architecture:**
+```typescript
+// ✅ GOOD - Cacheable system prompt
+systemPrompt: [
+  { text: STATIC_RULES, cache_control: { type: 'ephemeral' } },
+  { text: STATIC_EXAMPLES, cache_control: { type: 'ephemeral' } }
+]
+userMessage: `Character: ${dynamicState}\nAction: ${userInput}`
+
+// ❌ BAD - Uncacheable (character state in system prompt)
+systemPrompt: `Rules + Character: ${dynamicState}`
+```
+
+### Testing AI Outputs
+
+**LLM-as-Judge Pattern:**
+- Use LLM to evaluate nuanced qualities (narrative tone, reasoning quality)
+- Avoid brittle keyword matching for creative outputs
+- Define rubrics: EXCELLENT / ACCEPTABLE / POOR with criteria
+- Example: "Does the GM's response show collaborative tone?" vs checking for specific words
+
+**Evaluation Framework:**
+- Unit tests: Pure functions (parsing, validation)
+- Integration tests: Agent + real LLM calls (schema compliance)
+- LLM Evals: Judgment quality (position/effect reasoning, atmosphere)
+- Cost awareness: 30 scenarios ≈ $0.15-0.30 per run with caching
+
+---
+
+## Part 2: Writing Instructions for LLMs
+
+**Comprehensive framework:** See @.safeword/guides/llm-instruction-design.md
+
+**Quick summary:** When creating documentation that LLMs will read and follow (SAFEWORD.md, CLAUDE.md, testing guides, coding standards), apply 13 core principles:
+
+1. **MECE Principle** - Decision trees must be mutually exclusive and collectively exhaustive
+2. **Explicit Over Implicit** - Define all terms, never assume LLMs know what you mean
+3. **No Contradictions** - Different sections must align, LLMs don't reconcile conflicts
+4. **Concrete Examples Over Abstract Rules** - Show, don't just tell (2-3 examples per rule)
+5. **Edge Cases Must Be Explicit** - What seems obvious to humans often isn't to LLMs
+6. **Actionable Over Vague** - Replace subjective terms with optimization rules + red flags
+7. **Decision Trees: Sequential Over Parallel** - Ordered steps that stop at first match
+8. **Tie-Breaking Rules** - Tell LLMs how to choose when multiple options apply
+9. **Lookup Tables for Complex Decisions** - Provide reference tables for complex logic
+10. **Avoid Caveats in Tables** - Keep patterns clean, parentheticals break LLM pattern matching
+11. **Percentages: Context or None** - Include adjustment guidance or use principles instead
+12. **Specificity in Questions** - Use precise technical terms, not general descriptions
+13. **Re-evaluation Paths** - Provide concrete next steps when LLMs hit dead ends
+
+**Also includes:** Anti-patterns to avoid, quality checklist, research-backed principles, and before/after examples.