@cregis-dev/cckit 0.4.0 → 0.5.0

package/templates/ecc/skills/eval-harness/SKILL.md

@@ -1,236 +0,0 @@
----
-name: eval-harness
-description: Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles
-origin: ECC
-tools: Read, Write, Edit, Bash, Grep, Glob
----
-
-# Eval Harness Skill
-
-A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.
-
-## When to Activate
-
-- Setting up eval-driven development (EDD) for AI-assisted workflows
-- Defining pass/fail criteria for Claude Code task completion
-- Measuring agent reliability with pass@k metrics
-- Creating regression test suites for prompt or agent changes
-- Benchmarking agent performance across model versions
-
-## Philosophy
-
-Eval-Driven Development treats evals as the "unit tests of AI development":
-- Define expected behavior BEFORE implementation
-- Run evals continuously during development
-- Track regressions with each change
-- Use pass@k metrics for reliability measurement
-
-## Eval Types
-
-### Capability Evals
-Test if Claude can do something it couldn't before:
-```markdown
-[CAPABILITY EVAL: feature-name]
-Task: Description of what Claude should accomplish
-Success Criteria:
-  - [ ] Criterion 1
-  - [ ] Criterion 2
-  - [ ] Criterion 3
-Expected Output: Description of expected result
-```
-
-### Regression Evals
-Ensure changes don't break existing functionality:
-```markdown
-[REGRESSION EVAL: feature-name]
-Baseline: SHA or checkpoint name
-Tests:
-  - existing-test-1: PASS/FAIL
-  - existing-test-2: PASS/FAIL
-  - existing-test-3: PASS/FAIL
-Result: X/Y passed (previously Y/Y)
-```
-
-## Grader Types
-
-### 1. Code-Based Grader
-Deterministic checks using code:
-```bash
-# Check if file contains expected pattern
-grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
-
-# Check if tests pass
-npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
-
-# Check if build succeeds
-npm run build && echo "PASS" || echo "FAIL"
-```
-
-### 2. Model-Based Grader
-Use Claude to evaluate open-ended outputs:
-```markdown
-[MODEL GRADER PROMPT]
-Evaluate the following code change:
-1. Does it solve the stated problem?
-2. Is it well-structured?
-3. Are edge cases handled?
-4. Is error handling appropriate?
-
-Score: 1-5 (1=poor, 5=excellent)
-Reasoning: [explanation]
-```
-
-### 3. Human Grader
-Flag for manual review:
-```markdown
-[HUMAN REVIEW REQUIRED]
-Change: Description of what changed
-Reason: Why human review is needed
-Risk Level: LOW/MEDIUM/HIGH
-```
-
-## Metrics
-
-### pass@k
-"At least one success in k attempts"
-- pass@1: First attempt success rate
-- pass@3: Success within 3 attempts
-- Typical target: pass@3 > 90%
-
-### pass^k
-"All k trials succeed"
-- Higher bar for reliability
-- pass^3: 3 consecutive successes
-- Use for critical paths
-
-## Eval Workflow
-
-### 1. Define (Before Coding)
-```markdown
-## EVAL DEFINITION: feature-xyz
-
-### Capability Evals
-1. Can create new user account
-2. Can validate email format
-3. Can hash password securely
-
-### Regression Evals
-1. Existing login still works
-2. Session management unchanged
-3. Logout flow intact
-
-### Success Metrics
-- pass@3 > 90% for capability evals
-- pass^3 = 100% for regression evals
-```
-
-### 2. Implement
-Write code to pass the defined evals.
-
-### 3. Evaluate
-```bash
-# Run capability evals
-[Run each capability eval, record PASS/FAIL]
-
-# Run regression evals
-npm test -- --testPathPattern="existing"
-
-# Generate report
-```
-
-### 4. Report
-```markdown
-EVAL REPORT: feature-xyz
-========================
-
-Capability Evals:
-  create-user:     PASS (pass@1)
-  validate-email:  PASS (pass@2)
-  hash-password:   PASS (pass@1)
-  Overall:         3/3 passed
-
-Regression Evals:
-  login-flow:      PASS
-  session-mgmt:    PASS
-  logout-flow:     PASS
-  Overall:         3/3 passed
-
-Metrics:
-  pass@1: 67% (2/3)
-  pass@3: 100% (3/3)
-
-Status: READY FOR REVIEW
-```
-
-## Integration Patterns
-
-### Pre-Implementation
-```
-/eval define feature-name
-```
-Creates eval definition file at `.claude/evals/feature-name.md`
-
-### During Implementation
-```
-/eval check feature-name
-```
-Runs current evals and reports status
-
-### Post-Implementation
-```
-/eval report feature-name
-```
-Generates full eval report
-
-## Eval Storage
-
-Store evals in project:
-```
-.claude/
-  evals/
-    feature-xyz.md      # Eval definition
-    feature-xyz.log     # Eval run history
-    baseline.json       # Regression baselines
-```
-
-## Best Practices
-
-1. **Define evals BEFORE coding** - Forces clear thinking about success criteria
-2. **Run evals frequently** - Catch regressions early
-3. **Track pass@k over time** - Monitor reliability trends
-4. **Use code graders when possible** - Deterministic > probabilistic
-5. **Human review for security** - Never fully automate security checks
-6. **Keep evals fast** - Slow evals don't get run
-7. **Version evals with code** - Evals are first-class artifacts
-
-## Example: Adding Authentication
-
-```markdown
-## EVAL: add-authentication
-
-### Phase 1: Define (10 min)
-Capability Evals:
-- [ ] User can register with email/password
-- [ ] User can login with valid credentials
-- [ ] Invalid credentials rejected with proper error
-- [ ] Sessions persist across page reloads
-- [ ] Logout clears session
-
-Regression Evals:
-- [ ] Public routes still accessible
-- [ ] API responses unchanged
-- [ ] Database schema compatible
-
-### Phase 2: Implement (varies)
-[Write code]
-
-### Phase 3: Evaluate
-Run: /eval check add-authentication
-
-### Phase 4: Report
-EVAL REPORT: add-authentication
-==============================
-Capability: 5/5 passed (pass@3: 100%)
-Regression: 3/3 passed (pass^3: 100%)
-Status: SHIP IT
-```

package/templates/ecc/skills/foundation-models-on-device/SKILL.md

@@ -1,243 +0,0 @@
----
-name: foundation-models-on-device
-description: Apple FoundationModels framework for on-device LLM — text generation, guided generation with @Generable, tool calling, and snapshot streaming in iOS 26+.
----
-
-# FoundationModels: On-Device LLM (iOS 26)
-
-Patterns for integrating Apple's on-device language model into apps using the FoundationModels framework. Covers text generation, structured output with `@Generable`, custom tool calling, and snapshot streaming — all running on-device for privacy and offline support.
-
-## When to Activate
-
-- Building AI-powered features using Apple Intelligence on-device
-- Generating or summarizing text without cloud dependency
-- Extracting structured data from natural language input
-- Implementing custom tool calling for domain-specific AI actions
-- Streaming structured responses for real-time UI updates
-- Need privacy-preserving AI (no data leaves the device)
-
-## Core Pattern — Availability Check
-
-Always check model availability before creating a session:
-
-```swift
-struct GenerativeView: View {
-    private var model = SystemLanguageModel.default
-
-    var body: some View {
-        switch model.availability {
-        case .available:
-            ContentView()
-        case .unavailable(.deviceNotEligible):
-            Text("Device not eligible for Apple Intelligence")
-        case .unavailable(.appleIntelligenceNotEnabled):
-            Text("Please enable Apple Intelligence in Settings")
-        case .unavailable(.modelNotReady):
-            Text("Model is downloading or not ready")
-        case .unavailable(let other):
-            Text("Model unavailable: \(other)")
-        }
-    }
-}
-```
-
-## Core Pattern — Basic Session
-
-```swift
-// Single-turn: create a new session each time
-let session = LanguageModelSession()
-let response = try await session.respond(to: "What's a good month to visit Paris?")
-print(response.content)
-
-// Multi-turn: reuse session for conversation context
-let session = LanguageModelSession(instructions: """
-    You are a cooking assistant.
-    Provide recipe suggestions based on ingredients.
-    Keep suggestions brief and practical.
-    """)
-
-let first = try await session.respond(to: "I have chicken and rice")
-let followUp = try await session.respond(to: "What about a vegetarian option?")
-```
-
-Key points for instructions:
-- Define the model's role ("You are a mentor")
-- Specify what to do ("Help extract calendar events")
-- Set style preferences ("Respond as briefly as possible")
-- Add safety measures ("Respond with 'I can't help with that' for dangerous requests")
-
-## Core Pattern — Guided Generation with @Generable
-
-Generate structured Swift types instead of raw strings:
-
-### 1. Define a Generable Type
-
-```swift
-@Generable(description: "Basic profile information about a cat")
-struct CatProfile {
-    var name: String
-
-    @Guide(description: "The age of the cat", .range(0...20))
-    var age: Int
-
-    @Guide(description: "A one sentence profile about the cat's personality")
-    var profile: String
-}
-```
-
-### 2. Request Structured Output
-
-```swift
-let response = try await session.respond(
-    to: "Generate a cute rescue cat",
-    generating: CatProfile.self
-)
-
-// Access structured fields directly
-print("Name: \(response.content.name)")
-print("Age: \(response.content.age)")
-print("Profile: \(response.content.profile)")
-```
-
-### Supported @Guide Constraints
-
-- `.range(0...20)` — numeric range
-- `.count(3)` — array element count
-- `description:` — semantic guidance for generation
-
-## Core Pattern — Tool Calling
-
-Let the model invoke custom code for domain-specific tasks:
-
-### 1. Define a Tool
-
-```swift
-struct RecipeSearchTool: Tool {
-    let name = "recipe_search"
-    let description = "Search for recipes matching a given term and return a list of results."
-
-    @Generable
-    struct Arguments {
-        var searchTerm: String
-        var numberOfResults: Int
-    }
-
-    func call(arguments: Arguments) async throws -> ToolOutput {
-        let recipes = await searchRecipes(
-            term: arguments.searchTerm,
-            limit: arguments.numberOfResults
-        )
-        return .string(recipes.map { "- \($0.name): \($0.description)" }.joined(separator: "\n"))
-    }
-}
-```
-
-### 2. Create Session with Tools
-
-```swift
-let session = LanguageModelSession(tools: [RecipeSearchTool()])
-let response = try await session.respond(to: "Find me some pasta recipes")
-```
-
-### 3. Handle Tool Errors
-
-```swift
-do {
-    let answer = try await session.respond(to: "Find a recipe for tomato soup.")
-} catch let error as LanguageModelSession.ToolCallError {
-    print(error.tool.name)
-    if case .databaseIsEmpty = error.underlyingError as? RecipeSearchToolError {
-        // Handle specific tool error
-    }
-}
-```
-
-## Core Pattern — Snapshot Streaming
-
-Stream structured responses for real-time UI with `PartiallyGenerated` types:
-
-```swift
-@Generable
-struct TripIdeas {
-    @Guide(description: "Ideas for upcoming trips")
-    var ideas: [String]
-}
-
-let stream = session.streamResponse(
-    to: "What are some exciting trip ideas?",
-    generating: TripIdeas.self
-)
-
-for try await partial in stream {
-    // partial: TripIdeas.PartiallyGenerated (all properties Optional)
-    print(partial)
-}
-```
-
-### SwiftUI Integration
-
-```swift
-@State private var partialResult: TripIdeas.PartiallyGenerated?
-@State private var errorMessage: String?
-
-var body: some View {
-    List {
-        ForEach(partialResult?.ideas ?? [], id: \.self) { idea in
-            Text(idea)
-        }
-    }
-    .overlay {
-        if let errorMessage { Text(errorMessage).foregroundStyle(.red) }
-    }
-    .task {
-        do {
-            let stream = session.streamResponse(to: prompt, generating: TripIdeas.self)
-            for try await partial in stream {
-                partialResult = partial
-            }
-        } catch {
-            errorMessage = error.localizedDescription
-        }
-    }
-}
-```
-
-## Key Design Decisions
-
-| Decision | Rationale |
-|----------|-----------|
-| On-device execution | Privacy — no data leaves the device; works offline |
-| 4,096 token limit | On-device model constraint; chunk large data across sessions |
-| Snapshot streaming (not deltas) | Structured output friendly; each snapshot is a complete partial state |
-| `@Generable` macro | Compile-time safety for structured generation; auto-generates `PartiallyGenerated` type |
-| Single request per session | `isResponding` prevents concurrent requests; create multiple sessions if needed |
-| `response.content` (not `.output`) | Correct API — always access results via `.content` property |
-
-## Best Practices
-
-- **Always check `model.availability`** before creating a session — handle all unavailability cases
-- **Use `instructions`** to guide model behavior — they take priority over prompts
-- **Check `isResponding`** before sending a new request — sessions handle one request at a time
-- **Access `response.content`** for results — not `.output`
-- **Break large inputs into chunks** — 4,096 token limit applies to instructions + prompt + output combined
-- **Use `@Generable`** for structured output — stronger guarantees than parsing raw strings
-- **Use `GenerationOptions(temperature:)`** to tune creativity (higher = more creative)
-- **Monitor with Instruments** — use Xcode Instruments to profile request performance
-
-## Anti-Patterns to Avoid
-
-- Creating sessions without checking `model.availability` first
-- Sending inputs exceeding the 4,096 token context window
-- Attempting concurrent requests on a single session
-- Using `.output` instead of `.content` to access response data
-- Parsing raw string responses when `@Generable` structured output would work
-- Building complex multi-step logic in a single prompt — break into multiple focused prompts
-- Assuming the model is always available — device eligibility and settings vary
-
-## When to Use
-
-- On-device text generation for privacy-sensitive apps
-- Structured data extraction from user input (forms, natural language commands)
-- AI-assisted features that must work offline
-- Streaming UI that progressively shows generated content
-- Domain-specific AI actions via tool calling (search, compute, lookup)