cokit-cli 1.0.8 → 1.2.0

package/templates/repo/.github/skills/context-engineering/references/evaluation.md

@@ -0,0 +1,89 @@
+# Evaluation
+
+Systematically assess agent performance and context engineering choices.
+
+## Key Finding: 95% Performance Variance
+
+- **Token usage**: 80% of variance
+- **Tool calls**: ~10% of variance
+- **Model choice**: ~5% of variance
+
+**Implication**: Token budgets matter more than model upgrades.
+
+## Multi-Dimensional Rubric
+
+| Dimension | Weight | Description |
+|-----------|--------|-------------|
+| Factual Accuracy | 30% | Ground truth verification |
+| Completeness | 25% | Coverage of requirements |
+| Tool Efficiency | 20% | Appropriate tool usage |
+| Citation Accuracy | 15% | Sources match claims |
+| Source Quality | 10% | Authority/credibility |
+
+## Evaluation Methods
+
+### LLM-as-Judge
+
+Beware biases:
+- **Position**: First position preferred
+- **Length**: Longer = higher score
+- **Self-enhancement**: Rating own outputs higher
+- **Verbosity**: Detailed = better
+
+**Mitigation**: Position swapping, anti-bias prompting
+
+### Pairwise Comparison
+
+```python
+score_ab = judge.compare(output_a, output_b)
+score_ba = judge.compare(output_b, output_a)
+consistent = (score_ab > 0.5) != (score_ba > 0.5)
+```
+
+### Probe-Based Testing
+
+| Probe | Tests | Example |
+|-------|-------|---------|
+| Recall | Facts | "What was the error?" |
+| Artifact | Files | "Which files modified?" |
+| Continuation | Planning | "What's next?" |
+| Decision | Reasoning | "Why chose X?" |
+
+## Test Set Design
+
+```python
+class TestSet:
+    def sample_stratified(self, n):
+        per_level = n // 3
+        return (
+            sample(self.simple, per_level) +
+            sample(self.medium, per_level) +
+            sample(self.complex, per_level)
+        )
+```
+
+## Production Monitoring
+
+```python
+class Monitor:
+    sample_rate = 0.01  # 1% sampling
+    alert_threshold = 0.85
+
+    def check(self, scores):
+        if avg(scores) < self.alert_threshold:
+            self.alert(f"Quality degraded: {avg(scores):.2f}")
+```
+
+## Guidelines
+
+1. Start with outcome evaluation, not step-by-step
+2. Use multi-dimensional rubrics
+3. Mitigate LLM-as-Judge biases
+4. Test with stratified complexity
+5. Implement continuous monitoring
+6. Focus on token efficiency (80% variance)
+
+## Related
+
+- [Context Compression](./context-compression.md)
+- [Tool Design](./tool-design.md)

package/templates/repo/.github/skills/context-engineering/references/memory-systems.md

@@ -0,0 +1,88 @@
+# Memory Systems
+
+Architectures for persistent context beyond the window.
+
+## Memory Layer Architecture
+
+| Layer | Scope | Persistence | Use Case |
+|-------|-------|-------------|----------|
+| L1: Working | Current window | None | Active reasoning |
+| L2: Short-Term | Session | Session | Task continuity |
+| L3: Long-Term | Cross-session | Persistent | User preferences |
+| L4: Entity | Per-entity | Persistent | Consistency |
+| L5: Temporal Graph | Time-aware | Persistent | Evolving facts |
+
+## Benchmark Performance (DMR Accuracy)
+
+| System | Accuracy | Approach |
+|--------|----------|----------|
+| Zep | 94.8% | Temporal knowledge graphs |
+| MemGPT | 93.4% | Hierarchical memory |
+| GraphRAG | 75-85% | Knowledge graphs |
+| Vector RAG | 60-70% | Embedding similarity |
+
+## Vector Store with Metadata
+
+```python
+class MetadataVectorStore:
+    def add(self, text, embedding, metadata):
+        doc = {
+            "text": text, "embedding": embedding,
+            "entities": metadata.get("entities", []),
+            "timestamp": metadata.get("timestamp")
+        }
+        self.index_by_entity(doc)
+
+    def search_by_entity(self, entity, k=5):
+        return self.entity_index.get(entity, [])[:k]
+```
+
+## Temporal Knowledge Graph
+
+```python
+class TemporalKnowledgeGraph:
+    def add_fact(self, subject, predicate, obj, valid_from, valid_to=None):
+        self.facts.append({
+            "triple": (subject, predicate, obj),
+            "valid_from": valid_from,
+            "valid_to": valid_to or "current"
+        })
+
+    def query_at_time(self, subject, predicate, timestamp):
+        for fact in self.facts:
+            if (fact["triple"][0] == subject and
+                fact["valid_from"] <= timestamp <= fact["valid_to"]):
+                return fact["triple"][2]
+```
+
+## Memory Retrieval Patterns
+
+| Pattern | Query | Use Case |
+|---------|-------|----------|
+| Semantic | "Similar to X" | General recall |
+| Entity-based | "About user John" | Consistency |
+| Temporal | "Valid on date" | Evolving facts |
+| Hybrid | Combine above | Production |
+
+## File-System-as-Memory
+
+```
+memory/
+├── sessions/{id}/summary.md
+├── entities/{id}.json
+└── facts/{timestamp}_{id}.json
+```
+
+## Guidelines
+
+1. Start with file-system-as-memory (simplest)
+2. Add vector search for scale
+3. Use entity indexing for consistency
+4. Add temporal awareness for evolving facts
+5. Implement consolidation for health
+6. Measure retrieval accuracy
+
+## Related
+
+- [Context Fundamentals](./context-fundamentals.md)
+- [Multi-Agent Patterns](./multi-agent-patterns.md)

package/templates/repo/.github/skills/context-engineering/references/multi-agent-patterns.md

@@ -0,0 +1,90 @@
+# Multi-Agent Patterns
+
+Distribute work across multiple context windows for isolation and scale.
+
+## Core Insight
+
+Sub-agents exist to **isolate context**, not anthropomorphize roles.
+
+## Token Economics
+
+| Architecture | Multiplier | Use Case |
+|--------------|------------|----------|
+| Single agent | 1x | Simple tasks |
+| Single + tools | ~4x | Moderate complexity |
+| Multi-agent | ~15x | Context isolation needed |
+
+**Key**: Token usage explains 80% of performance variance.
+
+## Patterns
+
+### Supervisor/Orchestrator
+
+```python
+class Supervisor:
+    def process(self, task):
+        subtasks = self.decompose(task)
+        results = [worker.execute(st, clean_context=True) for st in subtasks]
+        return self.aggregate(results)
+```
+
+**Pros**: Control, human-in-loop | **Cons**: Bottleneck, telephone game
+
+### Peer-to-Peer/Swarm
+
+```python
+def process_with_handoff(agent, task):
+    result = agent.process(task)
+    if "handoff" in result:
+        return process_with_handoff(select_agent(result["to"]), result["state"])
+    return result
+```
+
+**Pros**: No SPOF, scales | **Cons**: Complex coordination
+
+### Hierarchical
+
+Strategy → Planning → Execution layers
+**Pros**: Separation of concerns | **Cons**: Coordination overhead
+
+## Context Isolation Patterns
+
+| Pattern | Isolation | Use Case |
+|---------|-----------|----------|
+| Full delegation | None | Max capability |
+| Instruction passing | High | Simple tasks |
+| File coordination | Medium | Shared state |
+
+## Consensus Mechanisms
+
+```python
+def weighted_consensus(responses):
+    scores = {}
+    for r in responses:
+        weight = r["confidence"] * r["expertise"]
+        scores[r["answer"]] = scores.get(r["answer"], 0) + weight
+    return max(scores, key=scores.get)
+```
+
+## Failure Recovery
+
+| Failure | Mitigation |
+|---------|------------|
+| Bottleneck | Output schemas, checkpointing |
+| Overhead | Clear handoffs, batching |
+| Divergence | Boundaries, convergence checks |
+| Errors | Validation, circuit breakers |
+
+## Guidelines
+
+1. Use multi-agent for context isolation, not role-play
+2. Accept ~15x token cost for benefits
+3. Implement circuit breakers
+4. Use files for shared state
+5. Design clear handoffs
+6. Validate between agents
+
+## Related
+
+- [Context Optimization](./context-optimization.md)
+- [Evaluation](./evaluation.md)

package/templates/repo/.github/skills/context-engineering/references/project-development.md

@@ -0,0 +1,97 @@
+# Project Development
+
+Design and build LLM-powered projects from ideation to deployment.
+
+## Task-Model Fit
+
+**LLM-Suited**: Synthesis, subjective judgment, NL output, error-tolerant batches
+**LLM-Unsuited**: Precise computation, real-time, perfect accuracy, deterministic output
+
+## Manual Prototype First
+
+Test one example with target model before automation.
+
+## Pipeline Architecture
+
+```
+acquire → prepare → process → parse → render
+ (fetch)  (prompt)   (LLM)   (extract) (output)
+```
+
+Stages 1,2,4,5: Deterministic, cheap | Stage 3: Non-deterministic, expensive
+
+## File System as State
+
+```
+data/{id}/
+├── raw.json      # acquire done
+├── prompt.md     # prepare done
+├── response.md   # process done
+└── parsed.json   # parse done
+```
+
+```python
+def get_stage(id):
+    if exists(f"{id}/parsed.json"): return "render"
+    if exists(f"{id}/response.md"): return "parse"
+    # ... check backwards
+```
+
+**Benefits**: Idempotent, resumable, debuggable
+
+## Structured Output
+
+```markdown
+## SUMMARY
+[Overview]
+
+## KEY_FINDINGS
+- Finding 1
+
+## SCORE
+[1-5]
+```
+
+```python
+def parse(response):
+    return {
+        "summary": extract_section(response, "SUMMARY"),
+        "findings": extract_list(response, "KEY_FINDINGS"),
+        "score": extract_int(response, "SCORE")
+    }
+```
+
+## Cost Estimation
+
+```python
+def estimate(items, tokens_per, price_per_1k):
+    return len(items) * tokens_per / 1000 * price_per_1k * 1.1  # 10% buffer
+# 1000 items × 2000 tokens × $0.01/1k = $22
+```
+
+## Case Studies
+
+**Karpathy HN**: 930 items, $58, 1hr, 15 workers
+**Vercel d0**: 17→2 tools, 80%→100% success, 3.5x faster
+
+## Single vs Multi-Agent
+
+| Factor | Single | Multi |
+|--------|--------|-------|
+| Context | Fits window | Exceeds |
+| Tasks | Sequential | Parallel |
+| Tokens | Limited | 15x OK |
+
+## Guidelines
+
+1. Validate manually before automating
+2. Use 5-stage pipeline
+3. Track state via files
+4. Design structured output
+5. Estimate costs first
+6. Start single, add multi when needed
+
+## Related
+
+- [Context Optimization](./context-optimization.md)
+- [Multi-Agent Patterns](./multi-agent-patterns.md)

package/templates/repo/.github/skills/context-engineering/references/runtime-awareness.md

@@ -0,0 +1,202 @@
+# Runtime Awareness
+
+Monitor usage limits and context window utilization in real-time to optimize Claude Code sessions.
+
+## Overview
+
+Runtime awareness provides visibility into two critical metrics:
+1. **Usage Limits** - API quota consumption (5-hour and 7-day rolling windows)
+2. **Context Window** - Current token utilization within the 200K context limit
+
+## Architecture
+
+```
+┌─────────────────┐    ┌──────────────────────────┐
+│  statusline.cjs │───▶│  /tmp/ck-context-*.json  │
+│  (writes data)  │    │  (context window data)   │
+└─────────────────┘    └────────────┬─────────────┘
+                                    │
+                       ┌────────────▼─────────────┐
+                       │  usage-context-hook.cjs  │◀── PostToolUse
+                       │  - Reads context file    │
+                       │  - Fetches usage limits  │
+                       │  - Injects awareness     │
+                       └──────────────────────────┘
+```
+
+## Usage Limits API
+
+### Endpoint
+
+```
+GET https://api.anthropic.com/api/oauth/usage
+```
+
+### Authentication
+
+Requires OAuth Bearer token with `anthropic-beta: oauth-2025-04-20` header.
+
+### Credential Locations
+
+| Platform | Method | Location |
+|----------|--------|----------|
+| macOS | Keychain | `Claude Code-credentials` |
+| Windows | File | `%USERPROFILE%.copilot\.credentials.json` |
+| Linux | File | `~/.copilot/.credentials.json` |
+
+### Response Structure
+
+```json
+{
+  "five_hour": {
+    "utilization": 45,
+    "resets_at": "2025-01-15T18:00:00Z"
+  },
+  "seven_day": {
+    "utilization": 32,
+    "resets_at": "2025-01-22T00:00:00Z"
+  },
+  "seven_day_sonnet": {
+    "utilization": 11,
+    "resets_at": "2025-01-15T09:00:00Z"
+  }
+}
+```
+
+- `utilization`: Already a percentage (0-100), NOT a decimal
+- `resets_at`: ISO 8601 timestamp when quota resets
+- `seven_day_sonnet`: Model-specific limit (may be null)
+
+## Context Window Data
+
+### Source
+
+Statusline writes context data to `/tmp/ck-context-{session_id}.json`:
+
+```json
+{
+  "percent": 67,
+  "tokens": 134000,
+  "size": 200000,
+  "usage": {
+    "input_tokens": 80000,
+    "cache_creation_input_tokens": 30000,
+    "cache_read_input_tokens": 24000
+  },
+  "timestamp": 1705312000000
+}
+```
+
+### Token Calculation
+
+```
+total = input_tokens + cache_creation_input_tokens + cache_read_input_tokens
+percent = (total + AUTOCOMPACT_BUFFER) / context_window_size * 100
+```
+
+Where `AUTOCOMPACT_BUFFER = 45000` (22.5% reserved).
+
+## Hook Output
+
+The PostToolUse hook injects awareness data every 5 minutes:
+
+```xml
+<usage-awareness>
+Limits: 5h=45%, 7d=32%
+Context: 67%
+</usage-awareness>
+```
+
+### Warning Indicators
+
+| Level | Threshold | Indicator |
+|-------|-----------|-----------|
+| Normal | < 70% | Plain percentage |
+| Warning | 70-89% | `[WARNING]` |
+| Critical | ≥ 90% | `[CRITICAL]` |
+
+### Examples
+
+Normal state:
+```xml
+<usage-awareness>
+Limits: 5h=45%, 7d=32%
+Context: 67%
+</usage-awareness>
+```
+
+Warning state:
+```xml
+<usage-awareness>
+Limits: 5h=75% [WARNING], 7d=32%
+Context: 78% [WARNING - consider compaction]
+</usage-awareness>
+```
+
+Critical state:
+```xml
+<usage-awareness>
+Limits: 5h=92% [CRITICAL], 7d=65%
+Context: 91% [CRITICAL - compaction needed]
+</usage-awareness>
+```
+
+## Recommendations by Threshold
+
+### Context Window
+
+| Utilization | Action |
+|-------------|--------|
+| < 70% | Continue normally |
+| 70-80% | Plan compaction strategy |
+| 80-90% | Execute compaction |
+| > 90% | Immediate compaction or session reset |
+
+### Usage Limits
+
+| 5-Hour | Action |
+|--------|--------|
+| < 70% | Normal usage |
+| 70-90% | Reduce parallelization, delegate to subagents |
+| > 90% | Wait for reset or use lower-tier models |
+
+| 7-Day | Action |
+|-------|--------|
+| < 70% | Normal usage |
+| 70-90% | Monitor daily consumption |
+| > 90% | Limit usage to essential tasks |
+
+## Configuration
+
+### Hook Settings (`$HOME/.copilot/settings.json`)
+
+```json
+{
+  "hooks": {
+    "PostToolUse": [
+      {
+        "matcher": "*",
+        "hooks": [{
+          "type": "command",
+          "command": "node $HOME/.copilot/hooks/usage-context-awareness.cjs"
+        }]
+      }
+    ]
+  }
+}
+```
+
+### Throttling
+
+- **Injection interval**: 5 minutes (300,000ms)
+- **API cache TTL**: 60 seconds
+- **Context data freshness**: 30 seconds
+
+## Troubleshooting
+
+| Issue | Cause | Solution |
+|-------|-------|----------|
+| No usage limits shown | No OAuth token | Run `claude login` |
+| Stale context data | Statusline not updating | Check statusline config |
+| 401 Unauthorized | Expired token | Re-authenticate |
+| Hook not firing | Settings misconfigured | Verify PostToolUse matcher |

package/templates/repo/.github/skills/context-engineering/references/tool-design.md

@@ -0,0 +1,86 @@
+# Tool Design
+
+Design effective tools for agent systems.
+
+## Consolidation Principle
+
+Single comprehensive tools > multiple narrow tools. **Target**: 10-20 tools max.
+
+## Architectural Reduction Evidence
+
+| Metric | 17 Tools | 2 Tools | Improvement |
+|--------|----------|---------|-------------|
+| Time | 274.8s | 77.4s | 3.5x faster |
+| Success | 80% | 100% | +20% |
+| Tokens | 102k | 61k | 37% fewer |
+
+**Key**: Good documentation replaces tool sophistication.
+
+## When Reduction Works
+
+**Prerequisites**: High docs quality, capable model, navigable problem
+**Avoid when**: Messy systems, specialized domain, safety-critical
+
+## Description Engineering
+
+Answer four questions:
+1. **What** does the tool do?
+2. **When** should it be used?
+3. **What inputs** does it accept?
+4. **What** does it return?
+
+### Good Example
+
+```json
+{
+  "name": "get_customer",
+  "description": "Retrieve customer profile by ID. Use for order processing, support. Returns 404 if not found.",
+  "parameters": {
+    "customer_id": {"type": "string", "pattern": "^CUST-[0-9]{6}$"},
+    "format": {"enum": ["concise", "detailed"]}
+  }
+}
+```
+
+### Poor Example
+
+```json
+{"name": "search", "description": "Search for things", "parameters": {"q": {}}}
+```
+
+## Error Messages
+
+```python
+def format_error(code, message, resolution):
+    return {
+        "error": {"code": code, "message": message,
+                  "resolution": resolution, "retryable": code in RETRYABLE}
+    }
+# "Use YYYY-MM-DD format, e.g., '2024-01-05'"
+```
+
+## Response Formats
+
+Offer concise vs detailed:
+
+```python
+def get_data(id, format="concise"):
+    if format == "concise":
+        return {"name": data.name}
+    return data.full()  # Detailed
+```
+
+## Guidelines
+
+1. Consolidate tools (target 10-20)
+2. Answer all four questions
+3. Use full parameter names
+4. Design errors for recovery
+5. Offer concise/detailed formats
+6. Test with agents before deploy
+7. Start minimal, add when proven
+
+## Related
+
+- [Context Fundamentals](./context-fundamentals.md)
+- [Multi-Agent Patterns](./multi-agent-patterns.md)