@heytherevibin/skillforge 0.2.1

package/skills/error-handling/SKILL.md

@@ -0,0 +1,375 @@
+---
+name: error-handling
+description: Patterns for robust error handling across TypeScript, Python, and Go. Covers typed errors, error boundaries, retries, circuit breakers, and user-facing error messages.
+---
+
+# Error Handling Patterns
+
+Consistent, robust error handling patterns for production applications.
+
+## When to Activate
+
+- Designing error types or exception hierarchies for a new module or service
+- Adding retry logic or circuit breakers for unreliable external dependencies
+- Reviewing API endpoints for missing error handling
+- Implementing user-facing error messages and feedback
+- Debugging cascading failures or silent error swallowing
+
+## Core Principles
+
+1. **Fail fast and loudly** — surface errors at the boundary where they occur; don't bury them
+2. **Typed errors over string messages** — errors are first-class values with structure
+3. **User messages ≠ developer messages** — show friendly text to users, log full context server-side
+4. **Never swallow errors silently** — every `catch` block must either handle, re-throw, or log
+5. **Errors are part of your API contract** — document every error code a client may receive
+
+## TypeScript / JavaScript
+
+### Typed Error Classes
+
+```typescript
+// Define an error hierarchy for your domain
+export class AppError extends Error {
+  constructor(
+    message: string,
+    public readonly code: string,
+    public readonly statusCode: number = 500,
+    public readonly details?: unknown,
+  ) {
+    super(message)
+    this.name = this.constructor.name
+    // Maintain correct prototype chain in transpiled ES5 JavaScript.
+    // Required for `instanceof` checks (e.g., `error instanceof NotFoundError`)
+    // to work correctly when extending the built-in Error class.
+    Object.setPrototypeOf(this, new.target.prototype)
+  }
+}
+
+export class NotFoundError extends AppError {
+  constructor(resource: string, id: string) {
+    super(`${resource} not found: ${id}`, 'NOT_FOUND', 404)
+  }
+}
+
+export class ValidationError extends AppError {
+  constructor(message: string, details: { field: string; message: string }[]) {
+    super(message, 'VALIDATION_ERROR', 422, details)
+  }
+}
+
+export class UnauthorizedError extends AppError {
+  constructor(reason = 'Authentication required') {
+    super(reason, 'UNAUTHORIZED', 401)
+  }
+}
+
+export class RateLimitError extends AppError {
+  constructor(public readonly retryAfterMs: number) {
+    super('Rate limit exceeded', 'RATE_LIMITED', 429)
+  }
+}
+```
+
+### Result Pattern (no-throw style)
+
+For operations where failure is expected and common (parsing, external calls):
+
+```typescript
+type Result<T, E = AppError> =
+  | { ok: true; value: T }
+  | { ok: false; error: E }
+
+function ok<T>(value: T): Result<T> {
+  return { ok: true, value }
+}
+
+function err<E>(error: E): Result<never, E> {
+  return { ok: false, error }
+}
+
+// Usage
+async function fetchUser(id: string): Promise<Result<User>> {
+  try {
+    const user = await db.users.findUnique({ where: { id } })
+    if (!user) return err(new NotFoundError('User', id))
+    return ok(user)
+  } catch (e) {
+    return err(new AppError('Database error', 'DB_ERROR'))
+  }
+}
+
+const result = await fetchUser('abc-123')
+if (!result.ok) {
+  // TypeScript knows result.error here
+  logger.error('Failed to fetch user', { error: result.error })
+  return
+}
+// TypeScript knows result.value here
+console.log(result.value.email)
+```
+
+### API Error Handler (Next.js / Express)
+
+```typescript
+import { NextRequest, NextResponse } from 'next/server'
+
+function handleApiError(error: unknown): NextResponse {
+  // Known application error
+  if (error instanceof AppError) {
+    return NextResponse.json(
+      {
+        error: {
+          code: error.code,
+          message: error.message,
+          ...(error.details ? { details: error.details } : {}),
+        },
+      },
+      { status: error.statusCode },
+    )
+  }
+
+  // Zod validation error
+  if (error instanceof z.ZodError) {
+    return NextResponse.json(
+      {
+        error: {
+          code: 'VALIDATION_ERROR',
+          message: 'Request validation failed',
+          details: error.issues.map(i => ({
+            field: i.path.join('.'),
+            message: i.message,
+          })),
+        },
+      },
+      { status: 422 },
+    )
+  }
+
+  // Unexpected error — log details, return generic message
+  console.error('Unexpected error:', error)
+  return NextResponse.json(
+    { error: { code: 'INTERNAL_ERROR', message: 'An unexpected error occurred' } },
+    { status: 500 },
+  )
+}
+
+export async function POST(req: NextRequest) {
+  try {
+    // ... handler logic
+  } catch (error) {
+    return handleApiError(error)
+  }
+}
+```
+
+### React Error Boundary
+
+```typescript
+import { Component, ErrorInfo, ReactNode } from 'react'
+
+interface Props {
+  fallback: ReactNode
+  onError?: (error: Error, info: ErrorInfo) => void
+  children: ReactNode
+}
+
+interface State {
+  hasError: boolean
+  error: Error | null
+}
+
+export class ErrorBoundary extends Component<Props, State> {
+  state: State = { hasError: false, error: null }
+
+  static getDerivedStateFromError(error: Error): State {
+    return { hasError: true, error }
+  }
+
+  componentDidCatch(error: Error, info: ErrorInfo) {
+    this.props.onError?.(error, info)
+    console.error('Unhandled React error:', error, info)
+  }
+
+  render() {
+    if (this.state.hasError) return this.props.fallback
+    return this.props.children
+  }
+}
+
+// Usage
+<ErrorBoundary fallback={<p>Something went wrong. Please refresh.</p>}>
+  <MyComponent />
+</ErrorBoundary>
+```
+
+## Python
+
+### Custom Exception Hierarchy
+
+```python
+class AppError(Exception):
+    """Base application error."""
+    def __init__(self, message: str, code: str, status_code: int = 500):
+        super().__init__(message)
+        self.code = code
+        self.status_code = status_code
+
+class NotFoundError(AppError):
+    def __init__(self, resource: str, id: str):
+        super().__init__(f"{resource} not found: {id}", "NOT_FOUND", 404)
+
+class ValidationError(AppError):
+    def __init__(self, message: str, details: list[dict] | None = None):
+        super().__init__(message, "VALIDATION_ERROR", 422)
+        self.details = details or []
+```
+
+### FastAPI Global Exception Handler
+
+```python
+from fastapi import FastAPI, Request
+from fastapi.responses import JSONResponse
+
+app = FastAPI()
+
+@app.exception_handler(AppError)
+async def app_error_handler(request: Request, exc: AppError) -> JSONResponse:
+    return JSONResponse(
+        status_code=exc.status_code,
+        content={"error": {"code": exc.code, "message": str(exc)}},
+    )
+
+@app.exception_handler(Exception)
+async def generic_error_handler(request: Request, exc: Exception) -> JSONResponse:
+    # Log full details, return generic message
+    logger.exception("Unexpected error", exc_info=exc)
+    return JSONResponse(
+        status_code=500,
+        content={"error": {"code": "INTERNAL_ERROR", "message": "An unexpected error occurred"}},
+    )
+```
+
+## Go
+
+### Sentinel Errors and Error Wrapping
+
+```go
+package domain
+
+import "errors"
+
+// Sentinel errors for type-checking
+var (
+    ErrNotFound    = errors.New("not found")
+    ErrUnauthorized = errors.New("unauthorized")
+    ErrConflict     = errors.New("conflict")
+)
+
+// Wrap errors with context — never lose the original
+func (r *UserRepository) FindByID(ctx context.Context, id string) (*User, error) {
+    user, err := r.db.QueryRow(ctx, "SELECT * FROM users WHERE id = $1", id)
+    if errors.Is(err, sql.ErrNoRows) {
+        return nil, fmt.Errorf("user %s: %w", id, ErrNotFound)
+    }
+    if err != nil {
+        return nil, fmt.Errorf("querying user %s: %w", id, err)
+    }
+    return user, nil
+}
+
+// At the handler level, unwrap to determine response
+func (h *Handler) GetUser(w http.ResponseWriter, r *http.Request) {
+    user, err := h.service.GetUser(r.Context(), chi.URLParam(r, "id"))
+    if err != nil {
+        switch {
+        case errors.Is(err, domain.ErrNotFound):
+            writeError(w, http.StatusNotFound, "not_found", err.Error())
+        case errors.Is(err, domain.ErrUnauthorized):
+            writeError(w, http.StatusForbidden, "forbidden", "Access denied")
+        default:
+            slog.Error("unexpected error", "err", err)
+            writeError(w, http.StatusInternalServerError, "internal_error", "An unexpected error occurred")
+        }
+        return
+    }
+    writeJSON(w, http.StatusOK, user)
+}
+```
+
+## Retry with Exponential Backoff
+
+```typescript
+interface RetryOptions {
+  maxAttempts?: number
+  baseDelayMs?: number
+  maxDelayMs?: number
+  retryIf?: (error: unknown) => boolean
+}
+
+async function withRetry<T>(
+  fn: () => Promise<T>,
+  options: RetryOptions = {},
+): Promise<T> {
+  const {
+    maxAttempts = 3,
+    baseDelayMs = 500,
+    maxDelayMs = 10_000,
+    retryIf = () => true,
+  } = options
+
+  let lastError: unknown
+
+  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
+    try {
+      return await fn()
+    } catch (error) {
+      lastError = error
+      if (attempt === maxAttempts || !retryIf(error)) throw error
+
+      const jitter = Math.random() * baseDelayMs
+      const delay = Math.min(baseDelayMs * 2 ** (attempt - 1) + jitter, maxDelayMs)
+      await new Promise(resolve => setTimeout(resolve, delay))
+    }
+  }
+
+  throw lastError
+}
+
+// Usage: retry transient network errors, not 4xx
+const data = await withRetry(() => fetch('/api/data').then(r => r.json()), {
+  maxAttempts: 3,
+  retryIf: (error) => !(error instanceof AppError && error.statusCode < 500),
+})
+```
+
+## User-Facing Error Messages
+
+Map error codes to human-readable messages. Keep technical details out of user-visible text.
+
+```typescript
+const USER_ERROR_MESSAGES: Record<string, string> = {
+  NOT_FOUND: 'The requested item could not be found.',
+  UNAUTHORIZED: 'Please sign in to continue.',
+  FORBIDDEN: "You don't have permission to do that.",
+  VALIDATION_ERROR: 'Please check your input and try again.',
+  RATE_LIMITED: 'Too many requests. Please wait a moment and try again.',
+  INTERNAL_ERROR: 'Something went wrong on our end. Please try again later.',
+}
+
+export function getUserMessage(code: string): string {
+  return USER_ERROR_MESSAGES[code] ?? USER_ERROR_MESSAGES.INTERNAL_ERROR
+}
+```
+
+## Error Handling Checklist
+
+Before merging any code that touches error handling:
+
+- [ ] Every `catch` block handles, re-throws, or logs — no silent swallowing
+- [ ] API errors follow the standard envelope `{ error: { code, message } }`
+- [ ] User-facing messages contain no stack traces or internal details
+- [ ] Full error context is logged server-side
+- [ ] Custom error classes extend a base `AppError` with a `code` field
+- [ ] Async functions surface errors to callers — no fire-and-forget without fallback
+- [ ] Retry logic only retries retriable errors (not 4xx client errors)
+- [ ] React components are wrapped in `ErrorBoundary` for rendering errors

package/skills/eval-harness/SKILL.md

@@ -0,0 +1,269 @@
+---
+name: eval-harness
+description: Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles
+tools: Read, Write, Edit, Bash, Grep, Glob
+---
+
+# Eval Harness Skill
+
+A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.
+
+## When to Activate
+
+- Setting up eval-driven development (EDD) for AI-assisted workflows
+- Defining pass/fail criteria for Claude Code task completion
+- Measuring agent reliability with pass@k metrics
+- Creating regression test suites for prompt or agent changes
+- Benchmarking agent performance across model versions
+
+## Philosophy
+
+Eval-Driven Development treats evals as the "unit tests of AI development":
+- Define expected behavior BEFORE implementation
+- Run evals continuously during development
+- Track regressions with each change
+- Use pass@k metrics for reliability measurement
+
+## Eval Types
+
+### Capability Evals
+Test if Claude can do something it couldn't before:
+```markdown
+[CAPABILITY EVAL: feature-name]
+Task: Description of what Claude should accomplish
+Success Criteria:
+  - [ ] Criterion 1
+  - [ ] Criterion 2
+  - [ ] Criterion 3
+Expected Output: Description of expected result
+```
+
+### Regression Evals
+Ensure changes don't break existing functionality:
+```markdown
+[REGRESSION EVAL: feature-name]
+Baseline: SHA or checkpoint name
+Tests:
+  - existing-test-1: PASS/FAIL
+  - existing-test-2: PASS/FAIL
+  - existing-test-3: PASS/FAIL
+Result: X/Y passed (previously Y/Y)
+```
+
+## Grader Types
+
+### 1. Code-Based Grader
+Deterministic checks using code:
+```bash
+# Check if file contains expected pattern
+grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
+
+# Check if tests pass
+npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
+
+# Check if build succeeds
+npm run build && echo "PASS" || echo "FAIL"
+```
+
+### 2. Model-Based Grader
+Use Claude to evaluate open-ended outputs:
+```markdown
+[MODEL GRADER PROMPT]
+Evaluate the following code change:
+1. Does it solve the stated problem?
+2. Is it well-structured?
+3. Are edge cases handled?
+4. Is error handling appropriate?
+
+Score: 1-5 (1=poor, 5=excellent)
+Reasoning: [explanation]
+```
+
+### 3. Human Grader
+Flag for manual review:
+```markdown
+[HUMAN REVIEW REQUIRED]
+Change: Description of what changed
+Reason: Why human review is needed
+Risk Level: LOW/MEDIUM/HIGH
+```
+
+## Metrics
+
+### pass@k
+"At least one success in k attempts"
+- pass@1: First attempt success rate
+- pass@3: Success within 3 attempts
+- Typical target: pass@3 > 90%
+
+### pass^k
+"All k trials succeed"
+- Higher bar for reliability
+- pass^3: 3 consecutive successes
+- Use for critical paths
+
+## Eval Workflow
+
+### 1. Define (Before Coding)
+```markdown
+## EVAL DEFINITION: feature-xyz
+
+### Capability Evals
+1. Can create new user account
+2. Can validate email format
+3. Can hash password securely
+
+### Regression Evals
+1. Existing login still works
+2. Session management unchanged
+3. Logout flow intact
+
+### Success Metrics
+- pass@3 > 90% for capability evals
+- pass^3 = 100% for regression evals
+```
+
+### 2. Implement
+Write code to pass the defined evals.
+
+### 3. Evaluate
+```bash
+# Run capability evals
+[Run each capability eval, record PASS/FAIL]
+
+# Run regression evals
+npm test -- --testPathPattern="existing"
+
+# Generate report
+```
+
+### 4. Report
+```markdown
+EVAL REPORT: feature-xyz
+========================
+
+Capability Evals:
+  create-user:     PASS (pass@1)
+  validate-email:  PASS (pass@2)
+  hash-password:   PASS (pass@1)
+  Overall:         3/3 passed
+
+Regression Evals:
+  login-flow:      PASS
+  session-mgmt:    PASS
+  logout-flow:     PASS
+  Overall:         3/3 passed
+
+Metrics:
+  pass@1: 67% (2/3)
+  pass@3: 100% (3/3)
+
+Status: READY FOR REVIEW
+```
+
+## Integration Patterns
+
+### Pre-Implementation
+```
+/eval define feature-name
+```
+Creates eval definition file at `.claude/evals/feature-name.md`
+
+### During Implementation
+```
+/eval check feature-name
+```
+Runs current evals and reports status
+
+### Post-Implementation
+```
+/eval report feature-name
+```
+Generates full eval report
+
+## Eval Storage
+
+Store evals in project:
+```
+.claude/
+  evals/
+    feature-xyz.md      # Eval definition
+    feature-xyz.log     # Eval run history
+    baseline.json       # Regression baselines
+```
+
+## Best Practices
+
+1. **Define evals BEFORE coding** - Forces clear thinking about success criteria
+2. **Run evals frequently** - Catch regressions early
+3. **Track pass@k over time** - Monitor reliability trends
+4. **Use code graders when possible** - Deterministic > probabilistic
+5. **Human review for security** - Never fully automate security checks
+6. **Keep evals fast** - Slow evals don't get run
+7. **Version evals with code** - Evals are first-class artifacts
+
+## Example: Adding Authentication
+
+```markdown
+## EVAL: add-authentication
+
+### Phase 1: Define (10 min)
+Capability Evals:
+- [ ] User can register with email/password
+- [ ] User can login with valid credentials
+- [ ] Invalid credentials rejected with proper error
+- [ ] Sessions persist across page reloads
+- [ ] Logout clears session
+
+Regression Evals:
+- [ ] Public routes still accessible
+- [ ] API responses unchanged
+- [ ] Database schema compatible
+
+### Phase 2: Implement (varies)
+[Write code]
+
+### Phase 3: Evaluate
+Run: /eval check add-authentication
+
+### Phase 4: Report
+EVAL REPORT: add-authentication
+==============================
+Capability: 5/5 passed (pass@3: 100%)
+Regression: 3/3 passed (pass^3: 100%)
+Status: SHIP IT
+```
+
+## Product Evals (v1.8)
+
+Use product evals when behavior quality cannot be captured by unit tests alone.
+
+### Grader Types
+
+1. Code grader (deterministic assertions)
+2. Rule grader (regex/schema constraints)
+3. Model grader (LLM-as-judge rubric)
+4. Human grader (manual adjudication for ambiguous outputs)
+
+### pass@k Guidance
+
+- `pass@1`: direct reliability
+- `pass@3`: practical reliability under controlled retries
+- `pass^3`: stability test (all 3 runs must pass)
+
+Recommended thresholds:
+- Capability evals: pass@3 >= 0.90
+- Regression evals: pass^3 = 1.00 for release-critical paths
+
+### Eval Anti-Patterns
+
+- Overfitting prompts to known eval examples
+- Measuring only happy-path outputs
+- Ignoring cost and latency drift while chasing pass rates
+- Allowing flaky graders in release gates
+
+### Minimal Eval Artifact Layout
+
+- `.claude/evals/<feature>.md` definition
+- `.claude/evals/<feature>.log` run history
+- `docs/releases/<version>/eval-summary.md` release snapshot