@baselane/packs 0.1.5 → 0.1.7

package/packs/full-harness/rendered/.claude/skills/agent-architecture-audit/SKILL.md

@@ -0,0 +1,254 @@
+---
+name: agent-architecture-audit
+description: Full-stack diagnostic for agent and LLM applications. Audits the 12-layer agent stack for wrapper regression, memory pollution, tool discipline failures, hidden repair loops, and rendering corruption. Produces severity-ranked findings with code-first fixes. Essential for developers building agent applications, autonomous loops, or any LLM-powered feature.
+---
+
+# Agent Architecture Audit
+
+A diagnostic workflow for agent systems that hide failures behind wrapper layers, stale memory, retry loops, or transport/rendering mutations.
+
+## When to Activate
+
+**MANDATORY for:**
+- Releasing any agent or LLM-powered application to production
+- Shipping features with tool calling, memory, or multi-step workflows
+- Agent behavior degrades after adding wrapper layers
+- User reports "the agent is getting worse" or "tools are flaky"
+- Same model works in playground but breaks inside your wrapper
+- Debugging agent behavior for more than 15 minutes without finding root cause
+
+**Especially critical when:**
+- You've added new prompt layers, tool definitions, or memory systems
+- Different agents in your system behave inconsistently
+- The model was fine yesterday but is hallucinating today
+- You suspect hidden repair/retry loops silently mutating responses
+
+**Do not use for:**
+- General code debugging — use `agent-introspection-debugging`
+- Code review — use language-specific reviewer agents
+- Security scanning — use `security-review` or `security-review/scan`
+- Agent performance benchmarking — use `agent-eval`
+- Writing new features — use the appropriate workflow skill
+
+## The 12-Layer Stack
+
+Every agent system has these layers. Any of them can corrupt the answer:
+
+| # | Layer | What Goes Wrong |
+|---|-------|----------------|
+| 1 | System prompt | Conflicting instructions, instruction bloat |
+| 2 | Session history | Stale context injection from previous turns |
+| 3 | Long-term memory | Pollution across sessions, old topics in new conversations |
+| 4 | Distillation | Compressed artifacts re-entering as pseudo-facts |
+| 5 | Active recall | Redundant re-summary layers wasting context |
+| 6 | Tool selection | Wrong tool routing, model skips required tools |
+| 7 | Tool execution | Hallucinated execution — claims to call but doesn't |
+| 8 | Tool interpretation | Misread or ignored tool output |
+| 9 | Answer shaping | Format corruption in final response |
+| 10 | Platform rendering | Transport-layer mutation (UI, API, CLI mutates valid answers) |
+| 11 | Hidden repair loops | Silent fallback/retry agents running second LLM pass |
+| 12 | Persistence | Expired state or cached artifacts reused as live evidence |
+
+## Common Failure Patterns
+
+### 1. Wrapper Regression
+
+The base model produces correct answers, but the wrapper layers make it worse.
+
+**Symptoms:**
+- Model works fine in playground or direct API call, breaks in your agent
+- Added a new prompt layer, existing behavior degraded
+- Agent sounds confident but is confidently wrong
+- "It was working before the last update"
+
+### 2. Memory Contamination
+
+Old topics leak into new conversations through history, memory retrieval, or distillation.
+
+**Symptoms:**
+- Agent brings up unrelated past topics
+- User corrections don't stick (old memory overwrites new)
+- Same-session artifacts re-enter as pseudo-facts
+- Memory grows without bound, degrading response quality over time
+
+### 3. Tool Discipline Failure
+
+Tools are declared in the prompt but not enforced in code. The model skips them or hallucinates execution.
+
+**Symptoms:**
+- "Must use tool X" in prompt, but model answers without calling it
+- Tool results look correct but were never actually executed
+- Different tools fight over the same responsibility
+- Model uses tool when it shouldn't, or skips it when it must
+
+### 4. Rendering/Transport Corruption
+
+The agent's internal answer is correct, but the platform layer mutates it during delivery.
+
+**Symptoms:**
+- Logs show correct answer, user sees broken output
+- Markdown rendering, JSON parsing, or streaming fragments corrupt valid responses
+- Hidden fallback agent quietly replaces the answer before delivery
+- Output differs between terminal and UI
+
+### 5. Hidden Agent Layers
+
+Silent repair, retry, summarization, or recall agents run without explicit contracts.
+
+**Symptoms:**
+- Output changes between internal generation and user delivery
+- "Auto-fix" loops run a second LLM pass the user doesn't know about
+- Multiple agents modify the same output without coordination
+- Answers get "smoothed" or "corrected" by invisible layers
+
+## Audit Workflow
+
+### Phase 1: Scope
+
+Define what you're auditing:
+
+- **Target system** — what agent application?
+- **Entrypoints** — how do users interact with it?
+- **Model stack** — which LLM(s) and providers?
+- **Symptoms** — what does the user report?
+- **Time window** — when did it start?
+- **Layers to audit** — which of the 12 layers apply?
+
+### Phase 2: Evidence Collection
+
+Gather evidence from the codebase:
+
+- **Source code** — agent loop, tool router, memory admission, prompt assembly
+- **Logs** — historical session traces, tool call records
+- **Config** — prompt templates, tool schemas, provider settings
+- **Memory files** — SOPs, knowledge bases, session archives
+
+Use `rg` to search for anti-patterns:
+
+```bash
+# Tool requirements expressed only in prompt text (not code)
+rg "must.*tool|必须.*工具|required.*call" --type md
+
+# Tool execution without validation
+rg "tool_call|toolCall|tool_use" --type py --type ts
+
+# Hidden LLM calls outside main agent loop
+rg "completion|chat\.create|messages\.create|llm\.invoke"
+
+# Memory admission without user-correction priority
+rg "memory.*admit|long.*term.*update|persist.*memory" --type py --type ts
+
+# Fallback loops that run additional LLM calls
+rg "fallback|retry.*llm|repair.*prompt|re-?prompt" --type py --type ts
+
+# Silent output mutation
+rg "mutate|rewrite.*response|transform.*output|shap" --type py --type ts
+```
+
+### Phase 3: Failure Mapping
+
+For each finding, document:
+
+- **Symptom** — what the user sees
+- **Mechanism** — how the wrapper causes it
+- **Source layer** — which of the 12 layers
+- **Root cause** — the deepest cause
+- **Evidence** — file:line or log:row reference
+- **Confidence** — 0.0 to 1.0
+
+### Phase 4: Fix Strategy
+
+Default fix order (code-first, not prompt-first):
+
+1. **Code-gate tool requirements** — enforce in code, not just prompt text
+2. **Remove or narrow hidden repair agents** — make fallback explicit with contracts
+3. **Reduce context duplication** — same info through prompt + history + memory + distillation
+4. **Tighten memory admission** — user corrections > agent assertions
+5. **Tighten distillation triggers** — don't compress what shouldn't be compressed
+6. **Reduce rendering mutation** — pass-through, don't transform
+7. **Convert to typed JSON envelopes** — structured internal flow, not freeform prose
+
+## Severity Model
+
+| Level | Meaning | Action |
+|-------|---------|--------|
+| `critical` | Agent can confidently produce wrong operational behavior | Fix before next release |
+| `high` | Agent frequently degrades correctness or stability | Fix this sprint |
+| `medium` | Correctness usually survives but output is fragile or wasteful | Plan for next cycle |
+| `low` | Mostly cosmetic or maintainability issues | Backlog |
+
+## Output Format
+
+Present findings to the user in this order:
+
+1. **Severity-ranked findings** (most critical first)
+2. **Architecture diagnosis** (which layer corrupted what, and why)
+3. **Ordered fix plan** (code-first, not prompt-first)
+
+Do not lead with compliments or summaries. If the system is broken, say so directly.
+
+## Quick Diagnostic Questions
+
+When auditing an agent system, answer these:
+
+| # | Question | If Yes → |
+|---|----------|----------|
+| 1 | Can the model skip a required tool and still answer? | Tool not code-gated |
+| 2 | Does old conversation content appear in new turns? | Memory contamination |
+| 3 | Is the same info in system prompt AND memory AND history? | Context duplication |
+| 4 | Does the platform run a second LLM pass before delivery? | Hidden repair loop |
+| 5 | Does the output differ between internal generation and user delivery? | Rendering corruption |
+| 6 | Are "must use tool X" rules only in prompt text? | Tool discipline failure |
+| 7 | Can the agent's own monologue become persistent memory? | Memory poisoning |
+
+## Anti-Patterns to Avoid
+
+- Avoid blaming the model before falsifying wrapper-layer regressions.
+- Avoid blaming memory without showing the contamination path.
+- Do not let a clean current state erase a dirty historical incident.
+- Do not treat markdown prose as a trustworthy internal protocol.
+- Do not accept "must use tool" in prompt text when code never enforces it.
+- Keep findings direct, evidence-backed, and severity-ranked.
+
+## Report Schema
+
+Audits should produce structured reports following this shape:
+
+```json
+{
+  "schema_version": "ecc.agent-architecture-audit.report.v1",
+  "executive_verdict": {
+    "overall_health": "high_risk",
+    "primary_failure_mode": "string",
+    "most_urgent_fix": "string"
+  },
+  "scope": {
+    "target_name": "string",
+    "model_stack": ["string"],
+    "layers_to_audit": ["string"]
+  },
+  "findings": [
+    {
+      "severity": "critical|high|medium|low",
+      "title": "string",
+      "mechanism": "string",
+      "source_layer": "string",
+      "root_cause": "string",
+      "evidence_refs": ["file:line"],
+      "confidence": 0.0,
+      "recommended_fix": "string"
+    }
+  ],
+  "ordered_fix_plan": [
+    { "order": 1, "goal": "string", "why_now": "string", "expected_effect": "string" }
+  ]
+}
+```
+
+## Related Skills
+
+- `agent-introspection-debugging` — Debug agent runtime failures (loops, timeouts, state errors)
+- `agent-eval` — Benchmark agent performance head-to-head
+- `security-review` — Security audit for code and configuration
+- `autonomous-agent-harness` — Set up autonomous agent operations
+- `agent-harness-construction` — Build agent harnesses from scratch

package/packs/full-harness/rendered/.claude/skills/agent-eval/SKILL.md

@@ -0,0 +1,143 @@
+---
+name: agent-eval
+description: Head-to-head comparison of coding agents (Claude Code, Aider, Codex, etc.) on custom tasks with pass rate, cost, time, and consistency metrics
+---
+
+# Agent Eval Skill
+
+A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it.
+
+## When to Activate
+
+- Comparing coding agents (Claude Code, Aider, Codex, etc.) on your own codebase
+- Measuring agent performance before adopting a new tool or model
+- Running regression checks when an agent updates its model or tooling
+- Producing data-backed agent selection decisions for a team
+
+## Installation
+
+> **Note:** Install agent-eval from its repository after reviewing the source.
+
+## Core Concepts
+
+### YAML Task Definitions
+
+Define tasks declaratively. Each task specifies what to do, which files to touch, and how to judge success:
+
+```yaml
+name: add-retry-logic
+description: Add exponential backoff retry to the HTTP client
+repo: ./my-project
+files:
+  - src/http_client.py
+prompt: |
+  Add retry logic with exponential backoff to all HTTP requests.
+  Max 3 retries. Initial delay 1s, max delay 30s.
+judge:
+  - type: pytest
+    command: pytest tests/test_http_client.py -v
+  - type: grep
+    pattern: "exponential_backoff|retry"
+    files: src/http_client.py
+commit: "abc1234"  # pin to specific commit for reproducibility
+```
+
+### Git Worktree Isolation
+
+Each agent run gets its own git worktree — no Docker required. This provides reproducibility isolation so agents cannot interfere with each other or corrupt the base repo.
+
+### Metrics Collected
+
+| Metric | What It Measures |
+|--------|-----------------|
+| Pass rate | Did the agent produce code that passes the judge? |
+| Cost | API spend per task (when available) |
+| Time | Wall-clock seconds to completion |
+| Consistency | Pass rate across repeated runs (e.g., 3/3 = 100%) |
+
+## Workflow
+
+### 1. Define Tasks
+
+Create a `tasks/` directory with YAML files, one per task:
+
+```bash
+mkdir tasks
+# Write task definitions (see template above)
+```
+
+### 2. Run Agents
+
+Execute agents against your tasks:
+
+```bash
+agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider --runs 3
+```
+
+Each run:
+1. Creates a fresh git worktree from the specified commit
+2. Hands the prompt to the agent
+3. Runs the judge criteria
+4. Records pass/fail, cost, and time
+
+### 3. Compare Results
+
+Generate a comparison report:
+
+```bash
+agent-eval report --format table
+```
+
+```
+Task: add-retry-logic (3 runs each)
+┌──────────────┬───────────┬────────┬────────┬─────────────┐
+│ Agent        │ Pass Rate │ Cost   │ Time   │ Consistency │
+├──────────────┼───────────┼────────┼────────┼─────────────┤
+│ claude-code  │ 3/3       │ $0.12  │ 45s    │ 100%        │
+│ aider        │ 2/3       │ $0.08  │ 38s    │  67%        │
+└──────────────┴───────────┴────────┴────────┴─────────────┘
+```
+
+## Judge Types
+
+### Code-Based (deterministic)
+
+```yaml
+judge:
+  - type: pytest
+    command: pytest tests/ -v
+  - type: command
+    command: npm run build
+```
+
+### Pattern-Based
+
+```yaml
+judge:
+  - type: grep
+    pattern: "class.*Retry"
+    files: src/**/*.py
+```
+
+### Model-Based (LLM-as-judge)
+
+```yaml
+judge:
+  - type: llm
+    prompt: |
+      Does this implementation correctly handle exponential backoff?
+      Check for: max retries, increasing delays, jitter.
+```
+
+## Best Practices
+
+- **Start with 3-5 tasks** that represent your real workload, not toy examples
+- **Run at least 3 trials** per agent to capture variance — agents are non-deterministic
+- **Pin the commit** in your task YAML so results are reproducible across days/weeks
+- **Include at least one deterministic judge** (tests, build) per task — LLM judges add noise
+- **Track cost alongside pass rate** — a 95% agent at 10x the cost may not be the right choice
+- **Version your task definitions** — they are test fixtures, treat them as code
+
+## Links
+
+- Repository: [github.com/joaquinhuigomez/agent-eval](https://github.com/joaquinhuigomez/agent-eval)

package/packs/full-harness/rendered/.claude/skills/agent-harness-construction/SKILL.md

@@ -0,0 +1,72 @@
+---
+name: agent-harness-construction
+description: Design and optimize AI agent action spaces, tool definitions, and observation formatting for higher completion rates.
+---
+
+# Agent Harness Construction
+
+Use this skill when you are improving how an agent plans, calls tools, recovers from errors, and converges on completion.
+
+## Core Model
+
+Agent output quality is constrained by:
+1. Action space quality
+2. Observation quality
+3. Recovery quality
+4. Context budget quality
+
+## Action Space Design
+
+1. Use stable, explicit tool names.
+2. Keep inputs schema-first and narrow.
+3. Return deterministic output shapes.
+4. Avoid catch-all tools unless isolation is impossible.
+
+## Granularity Rules
+
+- Use micro-tools for high-risk operations (deploy, migration, permissions).
+- Use medium tools for common edit/read/search loops.
+- Use macro-tools only when round-trip overhead is the dominant cost.
+
+## Observation Design
+
+Every tool response should include:
+- `status`: success|warning|error
+- `summary`: one-line result
+- `next_actions`: actionable follow-ups
+- `artifacts`: file paths / IDs
+
+## Error Recovery Contract
+
+For every error path, include:
+- root cause hint
+- safe retry instruction
+- explicit stop condition
+
+## Context Budgeting
+
+1. Keep system prompt minimal and invariant.
+2. Move large guidance into skills loaded on demand.
+3. Prefer references to files over inlining long documents.
+4. Compact at phase boundaries, not arbitrary token thresholds.
+
+## Architecture Pattern Guidance
+
+- ReAct: best for exploratory tasks with uncertain path.
+- Function-calling: best for structured deterministic flows.
+- Hybrid (recommended): ReAct planning + typed tool execution.
+
+## Benchmarking
+
+Track:
+- completion rate
+- retries per task
+- pass@1 and pass@3
+- cost per successful task
+
+## Anti-Patterns
+
+- Too many tools with overlapping semantics.
+- Opaque tool output with no recovery hints.
+- Error-only output without next steps.
+- Context overloading with irrelevant references.

package/packs/full-harness/rendered/.claude/skills/agent-introspection-debugging/SKILL.md

@@ -0,0 +1,152 @@
+---
+name: agent-introspection-debugging
+description: Structured self-debugging workflow for AI agent failures using capture, diagnosis, contained recovery, and introspection reports.
+---
+
+# Agent Introspection Debugging
+
+Use this skill when an agent run is failing repeatedly, consuming tokens without progress, looping on the same tools, or drifting away from the intended task.
+
+This is a workflow skill, not a hidden runtime. It teaches the agent to debug itself systematically before escalating to a human.
+
+## When to Activate
+
+- Maximum tool call / loop-limit failures
+- Repeated retries with no forward progress
+- Context growth or prompt drift that starts degrading output quality
+- File-system or environment state mismatch between expectation and reality
+- Tool failures that are likely recoverable with diagnosis and a smaller corrective action
+
+## Scope Boundaries
+
+Activate this skill for:
+- capturing failure state before retrying blindly
+- diagnosing common agent-specific failure patterns
+- applying contained recovery actions
+- producing a structured human-readable debug report
+
+Do not use this skill as the primary source for:
+- feature verification after code changes; use `verification-loop`
+- framework-specific debugging when a narrower ECC skill already exists
+- runtime promises the current harness cannot enforce automatically
+
+## Four-Phase Loop
+
+### Phase 1: Failure Capture
+
+Before trying to recover, record the failure precisely.
+
+Capture:
+- error type, message, and stack trace when available
+- last meaningful tool call sequence
+- what the agent was trying to do
+- current context pressure: repeated prompts, oversized pasted logs, duplicated plans, or runaway notes
+- current environment assumptions: cwd, branch, relevant service state, expected files
+
+Minimum capture template:
+
+```markdown
+## Failure Capture
+- Session / task:
+- Goal in progress:
+- Error:
+- Last successful step:
+- Last failed tool / command:
+- Repeated pattern seen:
+- Environment assumptions to verify:
+```
+
+### Phase 2: Root-Cause Diagnosis
+
+Match the failure to a known pattern before changing anything.
+
+| Pattern | Likely Cause | Check |
+| --- | --- | --- |
+| Maximum tool calls / repeated same command | loop or no-exit observer path | inspect the last N tool calls for repetition |
+| Context overflow / degraded reasoning | unbounded notes, repeated plans, oversized logs | inspect recent context for duplication and low-signal bulk |
+| `ECONNREFUSED` / timeout | service unavailable or wrong port | verify service health, URL, and port assumptions |
+| `429` / quota exhaustion | retry storm or missing backoff | count repeated calls and inspect retry spacing |
+| file missing after write / stale diff | race, wrong cwd, or branch drift | re-check path, cwd, git status, and actual file existence |
+| tests still failing after “fix” | wrong hypothesis | isolate the exact failing test and re-derive the bug |
+
+Diagnosis questions:
+- is this a logic failure, state failure, environment failure, or policy failure?
+- did the agent lose the real objective and start optimizing the wrong subtask?
+- is the failure deterministic or transient?
+- what is the smallest reversible action that would validate the diagnosis?
+
+### Phase 3: Contained Recovery
+
+Recover with the smallest action that changes the diagnosis surface.
+
+Safe recovery actions:
+- stop repeated retries and restate the hypothesis
+- trim low-signal context and keep only the active goal, blockers, and evidence
+- re-check the actual filesystem / branch / process state
+- narrow the task to one failing command, one file, or one test
+- switch from speculative reasoning to direct observation
+- escalate to a human when the failure is high-risk or externally blocked
+
+Do not claim unsupported auto-healing actions like “reset agent state” or “update harness config” unless you are actually doing them through real tools in the current environment.
+
+Contained recovery checklist:
+
+```markdown
+## Recovery Action
+- Diagnosis chosen:
+- Smallest action taken:
+- Why this is safe:
+- What evidence would prove the fix worked:
+```
+
+### Phase 4: Introspection Report
+
+End with a report that makes the recovery legible to the next agent or human.
+
+```markdown
+## Agent Self-Debug Report
+- Session / task:
+- Failure:
+- Root cause:
+- Recovery action:
+- Result: success | partial | blocked
+- Token / time burn risk:
+- Follow-up needed:
+- Preventive change to encode later:
+```
+
+## Recovery Heuristics
+
+Prefer these interventions in order:
+
+1. Restate the real objective in one sentence.
+2. Verify the world state instead of trusting memory.
+3. Shrink the failing scope.
+4. Run one discriminating check.
+5. Only then retry.
+
+Bad pattern:
+- retrying the same action three times with slightly different wording
+
+Good pattern:
+- capture failure
+- classify the pattern
+- run one direct check
+- change the plan only if the check supports it
+
+## Integration with ECC
+
+- Use `verification-loop` after recovery if code was changed.
+- Use `continuous-learning-v2` when the failure pattern is worth turning into an instinct or later skill.
+- Use `council` when the issue is not technical failure but decision ambiguity.
+- Use `workspace-surface-audit` if the failure came from conflicting local state or repo drift.
+
+## Output Standard
+
+When this skill is active, do not end with “I fixed it” alone.
+
+Always provide:
+- the failure pattern
+- the root-cause hypothesis
+- the recovery action
+- the evidence that the situation is now better or still blocked