@heyai-rules/pilo-masterkit 1.2.2 → 2.1.0

package/.agent/skills/agent-eval/SKILL.md

@@ -0,0 +1,145 @@
+---
+name: agent-eval
+description: Head-to-head comparison of coding agents (Claude Code, Aider, Codex, etc.) on custom tasks with pass rate, cost, time, and consistency metrics
+origin: ECC
+tools: Read, Write, Edit, Bash, Grep, Glob
+---
+
+# Agent Eval Skill
+
+A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it.
+
+## When to Activate
+
+- Comparing coding agents (Claude Code, Aider, Codex, etc.) on your own codebase
+- Measuring agent performance before adopting a new tool or model
+- Running regression checks when an agent updates its model or tooling
+- Producing data-backed agent selection decisions for a team
+
+## Installation
+
+> **Note:** Install agent-eval from its repository after reviewing the source.
+
+## Core Concepts
+
+### YAML Task Definitions
+
+Define tasks declaratively. Each task specifies what to do, which files to touch, and how to judge success:
+
+```yaml
+name: add-retry-logic
+description: Add exponential backoff retry to the HTTP client
+repo: ./my-project
+files:
+  - src/http_client.py
+prompt: |
+  Add retry logic with exponential backoff to all HTTP requests.
+  Max 3 retries. Initial delay 1s, max delay 30s.
+judge:
+  - type: pytest
+    command: pytest tests/test_http_client.py -v
+  - type: grep
+    pattern: "exponential_backoff|retry"
+    files: src/http_client.py
+commit: "abc1234"  # pin to specific commit for reproducibility
+```
+
+### Git Worktree Isolation
+
+Each agent run gets its own git worktree — no Docker required. This provides reproducibility isolation so agents cannot interfere with each other or corrupt the base repo.
+
+### Metrics Collected
+
+| Metric | What It Measures |
+|--------|-----------------|
+| Pass rate | Did the agent produce code that passes the judge? |
+| Cost | API spend per task (when available) |
+| Time | Wall-clock seconds to completion |
+| Consistency | Pass rate across repeated runs (e.g., 3/3 = 100%) |
+
+## Workflow
+
+### 1. Define Tasks
+
+Create a `tasks/` directory with YAML files, one per task:
+
+```bash
+mkdir tasks
+# Write task definitions (see template above)
+```
+
+### 2. Run Agents
+
+Execute agents against your tasks:
+
+```bash
+agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider --runs 3
+```
+
+Each run:
+1. Creates a fresh git worktree from the specified commit
+2. Hands the prompt to the agent
+3. Runs the judge criteria
+4. Records pass/fail, cost, and time
+
+### 3. Compare Results
+
+Generate a comparison report:
+
+```bash
+agent-eval report --format table
+```
+
+```
+Task: add-retry-logic (3 runs each)
+┌──────────────┬───────────┬────────┬────────┬─────────────┐
+│ Agent        │ Pass Rate │ Cost   │ Time   │ Consistency │
+├──────────────┼───────────┼────────┼────────┼─────────────┤
+│ claude-code  │ 3/3       │ $0.12  │ 45s    │ 100%        │
+│ aider        │ 2/3       │ $0.08  │ 38s    │  67%        │
+└──────────────┴───────────┴────────┴────────┴─────────────┘
+```
+
+## Judge Types
+
+### Code-Based (deterministic)
+
+```yaml
+judge:
+  - type: pytest
+    command: pytest tests/ -v
+  - type: command
+    command: npm run build
+```
+
+### Pattern-Based
+
+```yaml
+judge:
+  - type: grep
+    pattern: "class.*Retry"
+    files: src/**/*.py
+```
+
+### Model-Based (LLM-as-judge)
+
+```yaml
+judge:
+  - type: llm
+    prompt: |
+      Does this implementation correctly handle exponential backoff?
+      Check for: max retries, increasing delays, jitter.
+```
+
+## Best Practices
+
+- **Start with 3-5 tasks** that represent your real workload, not toy examples
+- **Run at least 3 trials** per agent to capture variance — agents are non-deterministic
+- **Pin the commit** in your task YAML so results are reproducible across days/weeks
+- **Include at least one deterministic judge** (tests, build) per task — LLM judges add noise
+- **Track cost alongside pass rate** — a 95% agent at 10x the cost may not be the right choice
+- **Version your task definitions** — they are test fixtures, treat them as code
+
+## Links
+
+- Repository: [github.com/joaquinhuigomez/agent-eval](https://github.com/joaquinhuigomez/agent-eval)

package/.agent/skills/agent-harness-construction/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: agent-harness-construction
+description: Design and optimize AI agent action spaces, tool definitions, and observation formatting for higher completion rates.
+origin: ECC
+---
+
+# Agent Harness Construction
+
+Use this skill when you are improving how an agent plans, calls tools, recovers from errors, and converges on completion.
+
+## Core Model
+
+Agent output quality is constrained by:
+1. Action space quality
+2. Observation quality
+3. Recovery quality
+4. Context budget quality
+
+## Action Space Design
+
+1. Use stable, explicit tool names.
+2. Keep inputs schema-first and narrow.
+3. Return deterministic output shapes.
+4. Avoid catch-all tools unless isolation is impossible.
+
+## Granularity Rules
+
+- Use micro-tools for high-risk operations (deploy, migration, permissions).
+- Use medium tools for common edit/read/search loops.
+- Use macro-tools only when round-trip overhead is the dominant cost.
+
+## Observation Design
+
+Every tool response should include:
+- `status`: success|warning|error
+- `summary`: one-line result
+- `next_actions`: actionable follow-ups
+- `artifacts`: file paths / IDs
+
+## Error Recovery Contract
+
+For every error path, include:
+- root cause hint
+- safe retry instruction
+- explicit stop condition
+
+## Context Budgeting
+
+1. Keep system prompt minimal and invariant.
+2. Move large guidance into skills loaded on demand.
+3. Prefer references to files over inlining long documents.
+4. Compact at phase boundaries, not arbitrary token thresholds.
+
+## Architecture Pattern Guidance
+
+- ReAct: best for exploratory tasks with uncertain path.
+- Function-calling: best for structured deterministic flows.
+- Hybrid (recommended): ReAct planning + typed tool execution.
+
+## Benchmarking
+
+Track:
+- completion rate
+- retries per task
+- pass@1 and pass@3
+- cost per successful task
+
+## Anti-Patterns
+
+- Too many tools with overlapping semantics.
+- Opaque tool output with no recovery hints.
+- Error-only output without next steps.
+- Context overloading with irrelevant references.

package/.agent/skills/agent-payment-x402/SKILL.md

@@ -0,0 +1,178 @@
+---
+name: agent-payment-x402
+description: Add x402 payment execution to AI agents — per-task budgets, spending controls, and non-custodial wallets via MCP tools. Use when agents need to pay for APIs, services, or other agents.
+origin: community
+---
+
+# Agent Payment Execution (x402)
+
+Enable AI agents to make autonomous payments with built-in spending controls. Uses the x402 HTTP payment protocol and MCP tools so agents can pay for external services, APIs, or other agents without custodial risk.
+
+## When to Use
+
+Use when: your agent needs to pay for an API call, purchase a service, settle with another agent, enforce per-task spending limits, or manage a non-custodial wallet. Pairs naturally with cost-aware-llm-pipeline and security-review skills.
+
+## How It Works
+
+### x402 Protocol
+x402 extends HTTP 402 (Payment Required) into a machine-negotiable flow. When a server returns `402`, the agent's payment tool automatically negotiates price, checks budget, signs a transaction, and retries — no human in the loop.
+
+### Spending Controls
+Every payment tool call enforces a `SpendingPolicy`:
+- **Per-task budget** — max spend for a single agent action
+- **Per-session budget** — cumulative limit across an entire session
+- **Allowlisted recipients** — restrict which addresses/services the agent can pay
+- **Rate limits** — max transactions per minute/hour
+
+### Non-Custodial Wallets
+Agents hold their own keys via ERC-4337 smart accounts. The orchestrator sets policy before delegation; the agent can only spend within bounds. No pooled funds, no custodial risk.
+
+## MCP Integration
+
+The payment layer exposes standard MCP tools that slot into any Claude Code or agent harness setup.
+
+> **Security note**: Always pin the package version. This tool manages private keys — unpinned `npx` installs introduce supply-chain risk.
+
+```json
+{
+  "mcpServers": {
+    "agentpay": {
+      "command": "npx",
+      "args": ["agentwallet-sdk@6.0.0"]
+    }
+  }
+}
+```
+
+### Available Tools (agent-callable)
+
+| Tool | Purpose |
+|------|---------|
+| `get_balance` | Check agent wallet balance |
+| `send_payment` | Send payment to address or ENS |
+| `check_spending` | Query remaining budget |
+| `list_transactions` | Audit trail of all payments |
+
+> **Note**: Spending policy is set by the **orchestrator** before delegating to the agent — not by the agent itself. This prevents agents from escalating their own spending limits. Configure policy via `set_policy` in your orchestration layer or pre-task hook, never as an agent-callable tool.
+
+## Examples
+
+### Budget enforcement in an MCP client
+
+When building an orchestrator that calls the agentpay MCP server, enforce budgets before dispatching paid tool calls.
+
+> **Prerequisites**: Install the package before adding the MCP config — `npx` without `-y` will prompt for confirmation in non-interactive environments, causing the server to hang: `npm install -g agentwallet-sdk@6.0.0`
+
+```typescript
+import { Client } from "@modelcontextprotocol/sdk/client/index.js";
+import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
+
+async function main() {
+  // 1. Validate credentials before constructing the transport.
+  //    A missing key must fail immediately — never let the subprocess start without auth.
+  const walletKey = process.env.WALLET_PRIVATE_KEY;
+  if (!walletKey) {
+    throw new Error("WALLET_PRIVATE_KEY is not set — refusing to start payment server");
+  }
+
+  // Connect to the agentpay MCP server via stdio transport.
+  // Whitelist only the env vars the server needs — never forward all of process.env
+  // to a third-party subprocess that manages private keys.
+  const transport = new StdioClientTransport({
+    command: "npx",
+    args: ["agentwallet-sdk@6.0.0"],
+    env: {
+      PATH: process.env.PATH ?? "",
+      NODE_ENV: process.env.NODE_ENV ?? "production",
+      WALLET_PRIVATE_KEY: walletKey,
+    },
+  });
+  const agentpay = new Client({ name: "orchestrator", version: "1.0.0" });
+  await agentpay.connect(transport);
+
+  // 2. Set spending policy before delegating to the agent.
+  //    Always verify success — a silent failure means no controls are active.
+  const policyResult = await agentpay.callTool({
+    name: "set_policy",
+    arguments: {
+      per_task_budget: 0.50,
+      per_session_budget: 5.00,
+      allowlisted_recipients: ["api.example.com"],
+    },
+  });
+  if (policyResult.isError) {
+    throw new Error(
+      `Failed to set spending policy — do not delegate: ${JSON.stringify(policyResult.content)}`
+    );
+  }
+
+  // 3. Use preToolCheck before any paid action
+  await preToolCheck(agentpay, 0.01);
+}
+
+// Pre-tool hook: fail-closed budget enforcement with four distinct error paths.
+async function preToolCheck(agentpay: Client, apiCost: number): Promise<void> {
+  // Path 1: Reject invalid input (NaN/Infinity bypass the < comparison)
+  if (!Number.isFinite(apiCost) || apiCost < 0) {
+    throw new Error(`Invalid apiCost: ${apiCost} — action blocked`);
+  }
+
+  // Path 2: Transport/connectivity failure
+  let result;
+  try {
+    result = await agentpay.callTool({ name: "check_spending" });
+  } catch (err) {
+    throw new Error(`Payment service unreachable — action blocked: ${err}`);
+  }
+
+  // Path 3: Tool returned an error (e.g., auth failure, wallet not initialised)
+  if (result.isError) {
+    throw new Error(
+      `check_spending failed — action blocked: ${JSON.stringify(result.content)}`
+    );
+  }
+
+  // Path 4: Parse and validate the response shape
+  let remaining: number;
+  try {
+    const parsed = JSON.parse(
+      (result.content as Array<{ text: string }>)[0].text
+    );
+    if (!Number.isFinite(parsed?.remaining)) {
+      throw new TypeError("missing or non-finite 'remaining' field");
+    }
+    remaining = parsed.remaining;
+  } catch (err) {
+    throw new Error(
+      `check_spending returned unexpected format — action blocked: ${err}`
+    );
+  }
+
+  // Path 5: Budget exceeded
+  if (remaining < apiCost) {
+    throw new Error(
+      `Budget exceeded: need $${apiCost} but only $${remaining} remaining`
+    );
+  }
+}
+
+main().catch((err) => {
+  console.error(err);
+  process.exitCode = 1;
+});
+```
+
+## Best Practices
+
+- **Set budgets before delegation**: When spawning sub-agents, attach a SpendingPolicy via your orchestration layer. Never give an agent unlimited spend.
+- **Pin your dependencies**: Always specify an exact version in your MCP config (e.g., `agentwallet-sdk@6.0.0`). Verify package integrity before deploying to production.
+- **Audit trails**: Use `list_transactions` in post-task hooks to log what was spent and why.
+- **Fail closed**: If the payment tool is unreachable, block the paid action — don't fall back to unmetered access.
+- **Pair with security-review**: Payment tools are high-privilege. Apply the same scrutiny as shell access.
+- **Test with testnets first**: Use Base Sepolia for development; switch to Base mainnet for production.
+
+## Production Reference
+
+- **npm**: [`agentwallet-sdk`](https://www.npmjs.com/package/agentwallet-sdk)
+- **Merged into NVIDIA NeMo Agent Toolkit**: [PR #17](https://github.com/NVIDIA/NeMo-Agent-Toolkit-Examples/pull/17) — x402 payment tool for NVIDIA's agent examples
+- **Protocol spec**: [x402.org](https://x402.org)

package/.agent/skills/agentic-engineering/SKILL.md

@@ -0,0 +1,63 @@
+---
+name: agentic-engineering
+description: Operate as an agentic engineer using eval-first execution, decomposition, and cost-aware model routing.
+origin: ECC
+---
+
+# Agentic Engineering
+
+Use this skill for engineering workflows where AI agents perform most implementation work and humans enforce quality and risk controls.
+
+## Operating Principles
+
+1. Define completion criteria before execution.
+2. Decompose work into agent-sized units.
+3. Route model tiers by task complexity.
+4. Measure with evals and regression checks.
+
+## Eval-First Loop
+
+1. Define capability eval and regression eval.
+2. Run baseline and capture failure signatures.
+3. Execute implementation.
+4. Re-run evals and compare deltas.
+
+## Task Decomposition
+
+Apply the 15-minute unit rule:
+- each unit should be independently verifiable
+- each unit should have a single dominant risk
+- each unit should expose a clear done condition
+
+## Model Routing
+
+- Haiku: classification, boilerplate transforms, narrow edits
+- Sonnet: implementation and refactors
+- Opus: architecture, root-cause analysis, multi-file invariants
+
+## Session Strategy
+
+- Continue session for closely-coupled units.
+- Start fresh session after major phase transitions.
+- Compact after milestone completion, not during active debugging.
+
+## Review Focus for AI-Generated Code
+
+Prioritize:
+- invariants and edge cases
+- error boundaries
+- security and auth assumptions
+- hidden coupling and rollout risk
+
+Do not waste review cycles on style-only disagreements when automated format/lint already enforce style.
+
+## Cost Discipline
+
+Track per task:
+- model
+- token estimate
+- retries
+- wall-clock time
+- success/failure
+
+Escalate model tier only when lower tier fails with a clear reasoning gap.

package/.agent/skills/ai-first-engineering/SKILL.md

@@ -0,0 +1,51 @@
+---
+name: ai-first-engineering
+description: Engineering operating model for teams where AI agents generate a large share of implementation output.
+origin: ECC
+---
+
+# AI-First Engineering
+
+Use this skill when designing process, reviews, and architecture for teams shipping with AI-assisted code generation.
+
+## Process Shifts
+
+1. Planning quality matters more than typing speed.
+2. Eval coverage matters more than anecdotal confidence.
+3. Review focus shifts from syntax to system behavior.
+
+## Architecture Requirements
+
+Prefer architectures that are agent-friendly:
+- explicit boundaries
+- stable contracts
+- typed interfaces
+- deterministic tests
+
+Avoid implicit behavior spread across hidden conventions.
+
+## Code Review in AI-First Teams
+
+Review for:
+- behavior regressions
+- security assumptions
+- data integrity
+- failure handling
+- rollout safety
+
+Minimize time spent on style issues already covered by automation.
+
+## Hiring and Evaluation Signals
+
+Strong AI-first engineers:
+- decompose ambiguous work cleanly
+- define measurable acceptance criteria
+- produce high-signal prompts and evals
+- enforce risk controls under delivery pressure
+
+## Testing Standard
+
+Raise testing bar for generated code:
+- required regression coverage for touched domains
+- explicit edge-case assertions
+- integration checks for interface boundaries