@heytherevibin/skillforge 0.2.1

package/skills/agent-eval/SKILL.md

@@ -0,0 +1,144 @@
+---
+name: agent-eval
+description: Head-to-head comparison of coding agents (Claude Code, Aider, Codex, etc.) on custom tasks with pass rate, cost, time, and consistency metrics
+tools: Read, Write, Edit, Bash, Grep, Glob
+---
+
+# Agent Eval Skill
+
+A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it.
+
+## When to Activate
+
+- Comparing coding agents (Claude Code, Aider, Codex, etc.) on your own codebase
+- Measuring agent performance before adopting a new tool or model
+- Running regression checks when an agent updates its model or tooling
+- Producing data-backed agent selection decisions for a team
+
+## Installation
+
+> **Note:** Install agent-eval from its repository after reviewing the source.
+
+## Core Concepts
+
+### YAML Task Definitions
+
+Define tasks declaratively. Each task specifies what to do, which files to touch, and how to judge success:
+
+```yaml
+name: add-retry-logic
+description: Add exponential backoff retry to the HTTP client
+repo: ./my-project
+files:
+  - src/http_client.py
+prompt: |
+  Add retry logic with exponential backoff to all HTTP requests.
+  Max 3 retries. Initial delay 1s, max delay 30s.
+judge:
+  - type: pytest
+    command: pytest tests/test_http_client.py -v
+  - type: grep
+    pattern: "exponential_backoff|retry"
+    files: src/http_client.py
+commit: "abc1234"  # pin to specific commit for reproducibility
+```
+
+### Git Worktree Isolation
+
+Each agent run gets its own git worktree — no Docker required. This provides reproducibility isolation so agents cannot interfere with each other or corrupt the base repo.
+
+### Metrics Collected
+
+| Metric | What It Measures |
+|--------|-----------------|
+| Pass rate | Did the agent produce code that passes the judge? |
+| Cost | API spend per task (when available) |
+| Time | Wall-clock seconds to completion |
+| Consistency | Pass rate across repeated runs (e.g., 3/3 = 100%) |
+
+## Workflow
+
+### 1. Define Tasks
+
+Create a `tasks/` directory with YAML files, one per task:
+
+```bash
+mkdir tasks
+# Write task definitions (see template above)
+```
+
+### 2. Run Agents
+
+Execute agents against your tasks:
+
+```bash
+agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider --runs 3
+```
+
+Each run:
+1. Creates a fresh git worktree from the specified commit
+2. Hands the prompt to the agent
+3. Runs the judge criteria
+4. Records pass/fail, cost, and time
+
+### 3. Compare Results
+
+Generate a comparison report:
+
+```bash
+agent-eval report --format table
+```
+
+```
+Task: add-retry-logic (3 runs each)
+┌──────────────┬───────────┬────────┬────────┬─────────────┐
+│ Agent        │ Pass Rate │ Cost   │ Time   │ Consistency │
+├──────────────┼───────────┼────────┼────────┼─────────────┤
+│ claude-code  │ 3/3       │ $0.12  │ 45s    │ 100%        │
+│ aider        │ 2/3       │ $0.08  │ 38s    │  67%        │
+└──────────────┴───────────┴────────┴────────┴─────────────┘
+```
+
+## Judge Types
+
+### Code-Based (deterministic)
+
+```yaml
+judge:
+  - type: pytest
+    command: pytest tests/ -v
+  - type: command
+    command: npm run build
+```
+
+### Pattern-Based
+
+```yaml
+judge:
+  - type: grep
+    pattern: "class.*Retry"
+    files: src/**/*.py
+```
+
+### Model-Based (LLM-as-judge)
+
+```yaml
+judge:
+  - type: llm
+    prompt: |
+      Does this implementation correctly handle exponential backoff?
+      Check for: max retries, increasing delays, jitter.
+```
+
+## Best Practices
+
+- **Start with 3-5 tasks** that represent your real workload, not toy examples
+- **Run at least 3 trials** per agent to capture variance — agents are non-deterministic
+- **Pin the commit** in your task YAML so results are reproducible across days/weeks
+- **Include at least one deterministic judge** (tests, build) per task — LLM judges add noise
+- **Track cost alongside pass rate** — a 95% agent at 10x the cost may not be the right choice
+- **Version your task definitions** — they are test fixtures, treat them as code
+
+## Links
+
+- Repository: [github.com/joaquinhuigomez/agent-eval](https://github.com/joaquinhuigomez/agent-eval)

package/skills/agent-harness-construction/SKILL.md

@@ -0,0 +1,72 @@
+---
+name: agent-harness-construction
+description: Design and optimize AI agent action spaces, tool definitions, and observation formatting for higher completion rates.
+---
+
+# Agent Harness Construction
+
+Use this skill when you are improving how an agent plans, calls tools, recovers from errors, and converges on completion.
+
+## Core Model
+
+Agent output quality is constrained by:
+1. Action space quality
+2. Observation quality
+3. Recovery quality
+4. Context budget quality
+
+## Action Space Design
+
+1. Use stable, explicit tool names.
+2. Keep inputs schema-first and narrow.
+3. Return deterministic output shapes.
+4. Avoid catch-all tools unless isolation is impossible.
+
+## Granularity Rules
+
+- Use micro-tools for high-risk operations (deploy, migration, permissions).
+- Use medium tools for common edit/read/search loops.
+- Use macro-tools only when round-trip overhead is the dominant cost.
+
+## Observation Design
+
+Every tool response should include:
+- `status`: success|warning|error
+- `summary`: one-line result
+- `next_actions`: actionable follow-ups
+- `artifacts`: file paths / IDs
+
+## Error Recovery Contract
+
+For every error path, include:
+- root cause hint
+- safe retry instruction
+- explicit stop condition
+
+## Context Budgeting
+
+1. Keep system prompt minimal and invariant.
+2. Move large guidance into skills loaded on demand.
+3. Prefer references to files over inlining long documents.
+4. Compact at phase boundaries, not arbitrary token thresholds.
+
+## Architecture Pattern Guidance
+
+- ReAct: best for exploratory tasks with uncertain path.
+- Function-calling: best for structured deterministic flows.
+- Hybrid (recommended): ReAct planning + typed tool execution.
+
+## Benchmarking
+
+Track:
+- completion rate
+- retries per task
+- pass@1 and pass@3
+- cost per successful task
+
+## Anti-Patterns
+
+- Too many tools with overlapping semantics.
+- Opaque tool output with no recovery hints.
+- Error-only output without next steps.
+- Context overloading with irrelevant references.

package/skills/agent-introspection-debugging/SKILL.md

@@ -0,0 +1,152 @@
+---
+name: agent-introspection-debugging
+description: Structured self-debugging workflow for AI agent failures using capture, diagnosis, contained recovery, and introspection reports.
+---
+
+# Agent Introspection Debugging
+
+Use this skill when an agent run is failing repeatedly, consuming tokens without progress, looping on the same tools, or drifting away from the intended task.
+
+This is a workflow skill, not a hidden runtime. It teaches the agent to debug itself systematically before escalating to a human.
+
+## When to Activate
+
+- Maximum tool call / loop-limit failures
+- Repeated retries with no forward progress
+- Context growth or prompt drift that starts degrading output quality
+- File-system or environment state mismatch between expectation and reality
+- Tool failures that are likely recoverable with diagnosis and a smaller corrective action
+
+## Scope Boundaries
+
+Activate this skill for:
+- capturing failure state before retrying blindly
+- diagnosing common agent-specific failure patterns
+- applying contained recovery actions
+- producing a structured human-readable debug report
+
+Do not use this skill as the primary source for:
+- feature verification after code changes; use `verification-loop`
+- framework-specific debugging when a narrower the toolset skill already exists
+- runtime promises the current harness cannot enforce automatically
+
+## Four-Phase Loop
+
+### Phase 1: Failure Capture
+
+Before trying to recover, record the failure precisely.
+
+Capture:
+- error type, message, and stack trace when available
+- last meaningful tool call sequence
+- what the agent was trying to do
+- current context pressure: repeated prompts, oversized pasted logs, duplicated plans, or runaway notes
+- current environment assumptions: cwd, branch, relevant service state, expected files
+
+Minimum capture template:
+
+```markdown
+## Failure Capture
+- Session / task:
+- Goal in progress:
+- Error:
+- Last successful step:
+- Last failed tool / command:
+- Repeated pattern seen:
+- Environment assumptions to verify:
+```
+
+### Phase 2: Root-Cause Diagnosis
+
+Match the failure to a known pattern before changing anything.
+
+| Pattern | Likely Cause | Check |
+| --- | --- | --- |
+| Maximum tool calls / repeated same command | loop or no-exit observer path | inspect the last N tool calls for repetition |
+| Context overflow / degraded reasoning | unbounded notes, repeated plans, oversized logs | inspect recent context for duplication and low-signal bulk |
+| `ECONNREFUSED` / timeout | service unavailable or wrong port | verify service health, URL, and port assumptions |
+| `429` / quota exhaustion | retry storm or missing backoff | count repeated calls and inspect retry spacing |
+| file missing after write / stale diff | race, wrong cwd, or branch drift | re-check path, cwd, git status, and actual file existence |
+| tests still failing after “fix” | wrong hypothesis | isolate the exact failing test and re-derive the bug |
+
+Diagnosis questions:
+- is this a logic failure, state failure, environment failure, or policy failure?
+- did the agent lose the real objective and start optimizing the wrong subtask?
+- is the failure deterministic or transient?
+- what is the smallest reversible action that would validate the diagnosis?
+
+### Phase 3: Contained Recovery
+
+Recover with the smallest action that changes the diagnosis surface.
+
+Safe recovery actions:
+- stop repeated retries and restate the hypothesis
+- trim low-signal context and keep only the active goal, blockers, and evidence
+- re-check the actual filesystem / branch / process state
+- narrow the task to one failing command, one file, or one test
+- switch from speculative reasoning to direct observation
+- escalate to a human when the failure is high-risk or externally blocked
+
+Do not claim unsupported auto-healing actions like “reset agent state” or “update harness config” unless you are actually doing them through real tools in the current environment.
+
+Contained recovery checklist:
+
+```markdown
+## Recovery Action
+- Diagnosis chosen:
+- Smallest action taken:
+- Why this is safe:
+- What evidence would prove the fix worked:
+```
+
+### Phase 4: Introspection Report
+
+End with a report that makes the recovery legible to the next agent or human.
+
+```markdown
+## Agent Self-Debug Report
+- Session / task:
+- Failure:
+- Root cause:
+- Recovery action:
+- Result: success | partial | blocked
+- Token / time burn risk:
+- Follow-up needed:
+- Preventive change to encode later:
+```
+
+## Recovery Heuristics
+
+Prefer these interventions in order:
+
+1. Restate the real objective in one sentence.
+2. Verify the world state instead of trusting memory.
+3. Shrink the failing scope.
+4. Run one discriminating check.
+5. Only then retry.
+
+Bad pattern:
+- retrying the same action three times with slightly different wording
+
+Good pattern:
+- capture failure
+- classify the pattern
+- run one direct check
+- change the plan only if the check supports it
+
+## Integration with the toolset
+
+- Use `verification-loop` after recovery if code was changed.
+- Use `continuous-learning-v2` when the failure pattern is worth turning into an instinct or later skill.
+- Use `council` when the issue is not technical failure but decision ambiguity.
+- Use `workspace-surface-audit` if the failure came from conflicting local state or repo drift.
+
+## Output Standard
+
+When this skill is active, do not end with “I fixed it” alone.
+
+Always provide:
+- the failure pattern
+- the root-cause hypothesis
+- the recovery action
+- the evidence that the situation is now better or still blocked

package/skills/agent-payment-x402/SKILL.md

@@ -0,0 +1,224 @@
+---
+name: agent-payment-x402
+description: Add x402 payment execution to AI agents with per-task budgets, spending controls, and non-custodial wallets. Supports Base through agentwallet-sdk and X Layer through OKX Payments / OKX Agent Payments Protocol.
+origin: community
+---
+
+# Agent Payment Execution (x402)
+
+Enable AI agents to make policy-gated payments with built-in spending controls. Uses the x402 HTTP payment protocol and MCP tools so agents can pay for external services, APIs, or other agents without custodial risk.
+
+## When to Use
+
+Use when: your agent needs to pay for an API call, purchase a service, settle with another agent, enforce per-task spending limits, or manage a non-custodial wallet. Pairs naturally with cost-aware-llm-pipeline and security-review skills.
+
+## Decision Tree
+
+Choose the integration path based on whether your agent is buying access to a paid API or charging others for one:
+
+| Need | Recommended path |
+|------|------------------|
+| Agent pays a 402-gated API on Base or another agentwallet-supported chain | Use `agentwallet-sdk` as an MCP payment server with strict spending policy |
+| Agent pays a 402-gated API on X Layer | Use OKX Agent Payments Protocol from `okx/onchainos-skills`; `okx-x402-payment` is a deprecated legacy alias |
+| TypeScript API charges agents | Use OKX Payments TypeScript seller SDK docs for Express, Hono, Fastify, or Next.js |
+| Go API charges agents | Use OKX Payments Go seller SDK docs for Gin, Echo, or `net/http` |
+| Rust API charges agents | Use OKX Payments Rust seller SDK docs for Axum |
+| Java API charges agents | Use OKX Payments Java seller SDK docs for Spring Boot 2/3, Java EE, or Jakarta |
+| Python API charges agents | Check the current OKX Payments repository before implementation; a Python seller guide may not be available |
+
+## Supported Networks
+
+- `agentwallet-sdk`: use the package docs to confirm current network coverage before production. Base Sepolia is the safest development default; Base mainnet is the production path called out by the original skill.
+- OKX Payments / X Layer: current seller docs target X Layer (`eip155:196`) and USDT0 settlement. Fetch current SDK docs before generating production code because payment packages and facilitator behavior can change quickly.
+
+## How It Works
+
+### x402 Protocol
+x402 extends HTTP 402 (Payment Required) into a machine-negotiable flow. When a server returns `402`, the agent's payment tool negotiates price, checks budget, signs a transaction, and retries only inside the policy and confirmation boundary set by the orchestrator.
+
+### Spending Controls
+Every payment tool call enforces a `SpendingPolicy`:
+- **Per-task budget** — max spend for a single agent action
+- **Per-session budget** — cumulative limit across an entire session
+- **Allowlisted recipients** — restrict which addresses/services the agent can pay
+- **Rate limits** — max transactions per minute/hour
+
+### Non-Custodial Wallets
+Agents hold their own keys via ERC-4337 smart accounts. The orchestrator sets policy before delegation; the agent can only spend within bounds. No pooled funds, no custodial risk.
+
+## MCP Integration
+
+The payment layer exposes standard MCP tools that slot into any Claude Code or agent harness setup.
+
+> **Security note**: Always pin the package version. This tool manages private keys — unpinned `npx` installs introduce supply-chain risk.
+
+### Option A: agentwallet-sdk (Base / multi-chain)
+
+```json
+{
+  "mcpServers": {
+    "agentpay": {
+      "command": "npx",
+      "args": ["agentwallet-sdk@6.0.0"]
+    }
+  }
+}
+```
+
+### Available Tools (agent-callable)
+
+| Tool | Purpose |
+|------|---------|
+| `get_balance` | Check agent wallet balance |
+| `send_payment` | Send payment to address or ENS |
+| `check_spending` | Query remaining budget |
+| `list_transactions` | Audit trail of all payments |
+
+> **Note**: Spending policy is set by the **orchestrator** before delegating to the agent — not by the agent itself. This prevents agents from escalating their own spending limits. Configure policy via `set_policy` in your orchestration layer or pre-task hook, never as an agent-callable tool.
+
+### Option B: OKX Agent Payments Protocol (X Layer)
+
+Use this path for X Layer x402, Multi-Party Payment (MPP), session payment, charge, and A2A charge flows.
+
+For buyer-side agent flows:
+
+1. Install or reference the current `okx/onchainos-skills` repository.
+2. Use `skills/okx-agent-payments-protocol/SKILL.md` as the dispatcher.
+3. Treat `skills/okx-x402-payment/SKILL.md` as a deprecated compatibility alias, not as the canonical skill.
+4. Require explicit user confirmation before wallet status checks or payment actions. Do not hide payment execution behind a generic tool call.
+
+For seller-side API flows, fetch the latest language-specific guide before generating code:
+
+| Runtime | Current guide |
+|---------|---------------|
+| TypeScript | `https://raw.githubusercontent.com/okx/payments/main/typescript/SELLER.md` |
+| Go | `https://raw.githubusercontent.com/okx/payments/main/go/x402/SELLER.md` |
+| Rust | `https://raw.githubusercontent.com/okx/payments/main/rust/x402/SELLER.md` |
+| Java | `https://raw.githubusercontent.com/okx/payments/main/java/SELLER.md` |
+
+Do not copy examples from older docs without checking the current OKX repository. Current OKX guidance uses `okx-agent-payments-protocol` as the dispatcher, and Java seller docs are now available.
+
+## Examples
+
+### Budget enforcement in an MCP client
+
+When building an orchestrator that calls the agentpay MCP server, enforce budgets before dispatching paid tool calls.
+
+> **Prerequisites**: Install the package before adding the MCP config — `npx` without `-y` will prompt for confirmation in non-interactive environments, causing the server to hang: `npm install -g agentwallet-sdk@6.0.0`
+
+```typescript
+import { Client } from "@modelcontextprotocol/sdk/client/index.js";
+import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
+
+async function main() {
+  // 1. Validate credentials before constructing the transport.
+  //    A missing key must fail immediately — never let the subprocess start without auth.
+  const walletKey = process.env.WALLET_PRIVATE_KEY;
+  if (!walletKey) {
+    throw new Error("WALLET_PRIVATE_KEY is not set — refusing to start payment server");
+  }
+
+  // Connect to the agentpay MCP server via stdio transport.
+  // Whitelist only the env vars the server needs — never forward all of process.env
+  // to a third-party subprocess that manages private keys.
+  const transport = new StdioClientTransport({
+    command: "npx",
+    args: ["agentwallet-sdk@6.0.0"],
+    env: {
+      PATH: process.env.PATH ?? "",
+      NODE_ENV: process.env.NODE_ENV ?? "production",
+      WALLET_PRIVATE_KEY: walletKey,
+    },
+  });
+  const agentpay = new Client({ name: "orchestrator", version: "1.0.0" });
+  await agentpay.connect(transport);
+
+  // 2. Set spending policy before delegating to the agent.
+  //    Always verify success — a silent failure means no controls are active.
+  const policyResult = await agentpay.callTool({
+    name: "set_policy",
+    arguments: {
+      per_task_budget: 0.50,
+      per_session_budget: 5.00,
+      allowlisted_recipients: ["api.example.com"],
+    },
+  });
+  if (policyResult.isError) {
+    throw new Error(
+      `Failed to set spending policy — do not delegate: ${JSON.stringify(policyResult.content)}`
+    );
+  }
+
+  // 3. Use preToolCheck before any paid action
+  await preToolCheck(agentpay, 0.01);
+}
+
+// Pre-tool hook: fail-closed budget enforcement with four distinct error paths.
+async function preToolCheck(agentpay: Client, apiCost: number): Promise<void> {
+  // Path 1: Reject invalid input (NaN/Infinity bypass the < comparison)
+  if (!Number.isFinite(apiCost) || apiCost < 0) {
+    throw new Error(`Invalid apiCost: ${apiCost} — action blocked`);
+  }
+
+  // Path 2: Transport/connectivity failure
+  let result;
+  try {
+    result = await agentpay.callTool({ name: "check_spending" });
+  } catch (err) {
+    throw new Error(`Payment service unreachable — action blocked: ${err}`);
+  }
+
+  // Path 3: Tool returned an error (e.g., auth failure, wallet not initialised)
+  if (result.isError) {
+    throw new Error(
+      `check_spending failed — action blocked: ${JSON.stringify(result.content)}`
+    );
+  }
+
+  // Path 4: Parse and validate the response shape
+  let remaining: number;
+  try {
+    const parsed = JSON.parse(
+      (result.content as Array<{ text: string }>)[0].text
+    );
+    if (!Number.isFinite(parsed?.remaining)) {
+      throw new TypeError("missing or non-finite 'remaining' field");
+    }
+    remaining = parsed.remaining;
+  } catch (err) {
+    throw new Error(
+      `check_spending returned unexpected format — action blocked: ${err}`
+    );
+  }
+
+  // Path 5: Budget exceeded
+  if (remaining < apiCost) {
+    throw new Error(
+      `Budget exceeded: need $${apiCost} but only $${remaining} remaining`
+    );
+  }
+}
+
+main().catch((err) => {
+  console.error(err);
+  process.exitCode = 1;
+});
+```
+
+## Best Practices
+
+- **Set budgets before delegation**: When spawning sub-agents, attach a SpendingPolicy via your orchestration layer. Never give an agent unlimited spend.
+- **Pin your dependencies**: Always specify an exact version in your MCP config (e.g., `agentwallet-sdk@6.0.0`). Verify package integrity before deploying to production.
+- **Audit trails**: Use `list_transactions` in post-task hooks to log what was spent and why.
+- **Fail closed**: If the payment tool is unreachable, block the paid action — don't fall back to unmetered access.
+- **Pair with security-review**: Payment tools are high-privilege. Apply the same scrutiny as shell access.
+- **Test with testnets first**: Use Base Sepolia for development; switch to Base mainnet for production.
+
+## Production Reference
+
+- **npm**: [`agentwallet-sdk`](https://www.npmjs.com/package/agentwallet-sdk)
+- **Merged into NVIDIA NeMo Agent Toolkit**: [PR #17](https://github.com/NVIDIA/NeMo-Agent-Toolkit-Examples/pull/17) — x402 payment tool for NVIDIA's agent examples
+- **Protocol spec**: [x402.org](https://x402.org)
+- **OKX Payments SDKs**: [`okx/payments`](https://github.com/okx/payments) — TypeScript, Go, Rust, and Java seller integrations for X Layer x402
+- **OKX Agent Payments Protocol skill**: [`okx/onchainos-skills`](https://github.com/okx/onchainos-skills/tree/main/skills/okx-agent-payments-protocol)
+- **OKX Payments overview**: [web3.okx.com/onchainos/dev-docs/payments/overview](https://web3.okx.com/onchainos/dev-docs/payments/overview)