cokit-cli 1.0.8 → 1.2.0

package/skills/context-engineering/SKILL.md

@@ -0,0 +1,107 @@
+---
+name: context-engineering
+description: >-
+  Check context usage limits, monitor time remaining, optimize token consumption, debug context failures.
+  Use when asking about context percentage, rate limits, usage warnings, context optimization, agent architectures, memory systems.
+version: 1.0.0
+---
+
+# Context Engineering
+
+Context engineering curates the smallest high-signal token set for LLM tasks. The goal: maximize reasoning quality while minimizing token usage.
+
+## When to Activate
+
+- Designing/debugging agent systems
+- Context limits constrain performance
+- Optimizing cost/latency
+- Building multi-agent coordination
+- Implementing memory systems
+- Evaluating agent performance
+- Developing LLM-powered pipelines
+
+## Core Principles
+
+1. **Context quality > quantity** - High-signal tokens beat exhaustive content
+2. **Attention is finite** - U-shaped curve favors beginning/end positions
+3. **Progressive disclosure** - Load information just-in-time
+4. **Isolation prevents degradation** - Partition work across sub-agents
+5. **Measure before optimizing** - Know your baseline
+
+**IMPORTANT:**
+- Sacrifice grammar for the sake of concision.
+- Ensure token efficiency while maintaining high quality.
+- Pass these rules to subagents.
+
+## Quick Reference
+
+| Topic | When to Use | Reference |
+|-------|-------------|-----------|
+| **Fundamentals** | Understanding context anatomy, attention mechanics | [context-fundamentals.md](./references/context-fundamentals.md) |
+| **Degradation** | Debugging failures, lost-in-middle, poisoning | [context-degradation.md](./references/context-degradation.md) |
+| **Optimization** | Compaction, masking, caching, partitioning | [context-optimization.md](./references/context-optimization.md) |
+| **Compression** | Long sessions, summarization strategies | [context-compression.md](./references/context-compression.md) |
+| **Memory** | Cross-session persistence, knowledge graphs | [memory-systems.md](./references/memory-systems.md) |
+| **Multi-Agent** | Coordination patterns, context isolation | [multi-agent-patterns.md](./references/multi-agent-patterns.md) |
+| **Evaluation** | Testing agents, LLM-as-Judge, metrics | [evaluation.md](./references/evaluation.md) |
+| **Tool Design** | Tool consolidation, description engineering | [tool-design.md](./references/tool-design.md) |
+| **Pipelines** | Project development, batch processing | [project-development.md](./references/project-development.md) |
+| **Runtime Awareness** | Usage limits, context window monitoring | [runtime-awareness.md](./references/runtime-awareness.md) |
+
+## Key Metrics
+
+- **Token utilization**: Warning at 70%, trigger optimization at 80%
+- **Token variance**: Explains 80% of agent performance variance
+- **Multi-agent cost**: ~15x single agent baseline
+- **Compaction target**: 50-70% reduction, <5% quality loss
+- **Cache hit target**: 70%+ for stable workloads
+
+## Four-Bucket Strategy
+
+1. **Write**: Save context externally (scratchpads, files)
+2. **Select**: Pull only relevant context (retrieval, filtering)
+3. **Compress**: Reduce tokens while preserving info (summarization)
+4. **Isolate**: Split across sub-agents (partitioning)
+
+## Anti-Patterns
+
+- Exhaustive context over curated context
+- Critical info in middle positions
+- No compaction triggers before limits
+- Single agent for parallelizable tasks
+- Tools without clear descriptions
+
+## Guidelines
+
+1. Place critical info at beginning/end of context
+2. Implement compaction at 70-80% utilization
+3. Use sub-agents for context isolation, not role-play
+4. Design tools with 4-question framework (what, when, inputs, returns)
+5. Optimize for tokens-per-task, not tokens-per-request
+6. Validate with probe-based evaluation
+7. Monitor KV-cache hit rates in production
+8. Start minimal, add complexity only when proven necessary
+
+## Runtime Awareness
+
+The system automatically injects usage awareness via PostToolUse hook:
+
+```xml
+<usage-awareness>
+Claude Usage Limits: 5h=45%, 7d=32%
+Context Window Usage: 67%
+</usage-awareness>
+```
+
+**Thresholds:**
+- 70%: WARNING - consider optimization/compaction
+- 90%: CRITICAL - immediate action needed
+
+**Data Sources:**
+- Usage limits: Anthropic OAuth API (`https://api.anthropic.com/api/oauth/usage`)
+- Context window: Statusline temp file (`/tmp/ck-context-{session_id}.json`)
+
+## Scripts
+
+- [context_analyzer.py](./scripts/context_analyzer.py) - Context health analysis, degradation detection
+- [compression_evaluator.py](./scripts/compression_evaluator.py) - Compression quality evaluation

package/skills/context-engineering/references/context-compression.md

@@ -0,0 +1,84 @@
+# Context Compression
+
+Strategies for long-running sessions exceeding context windows.
+
+## Core Insight
+
+Optimize **tokens-per-task** (total to completion), not tokens-per-request.
+Aggressive compression causing re-fetching costs more than better retention.
+
+## Compression Methods
+
+| Method | Compression | Quality | Best For |
+|--------|-------------|---------|----------|
+| **Anchored Iterative** | 98.6% | 3.70/5 | Best balance |
+| **Regenerative Full** | 98.7% | 3.44/5 | Readability |
+| **Opaque** | 99.3% | 3.35/5 | Max compression |
+
+## Anchored Iterative Summary Template
+
+```markdown
+## Session Intent
+Original goal: [preserved]
+
+## Files Modified
+- file.py: Changes made
+
+## Decisions Made
+- Key decisions with rationale
+
+## Current State
+Progress summary
+
+## Next Steps
+1. Next action items
+```
+
+**On compression**: Merge new content into existing sections, don't regenerate.
+
+## Compression Triggers
+
+| Strategy | Trigger | Use Case |
+|----------|---------|----------|
+| Fixed threshold | 70-80% utilization | General purpose |
+| Sliding window | Keep last N turns + summary | Conversations |
+| Task-boundary | At logical completion | Multi-step workflows |
+
+## Artifact Trail Problem
+
+Weakest dimension (2.2-2.5/5.0). Coding agents need explicit tracking of:
+- Files created/modified/read
+- Function/variable names, error messages
+
+**Solution**: Dedicated artifact section in summary.
+
+## Probe-Based Evaluation
+
+| Probe Type | Tests | Example |
+|------------|-------|---------|
+| Recall | Factual retention | "What was the error?" |
+| Artifact | File tracking | "Which files modified?" |
+| Continuation | Task planning | "What next?" |
+| Decision | Reasoning chains | "Why chose X?" |
+
+## Six Evaluation Dimensions
+
+1. **Accuracy** - Technical correctness
+2. **Context Awareness** - Conversation state
+3. **Artifact Trail** - File tracking (universally weak)
+4. **Completeness** - Coverage depth
+5. **Continuity** - Work continuation
+6. **Instruction Following** - Constraints
+
+## Guidelines
+
+1. Use anchored iterative for best quality/compression
+2. Maintain explicit artifact tracking section
+3. Trigger compression at 70% utilization
+4. Merge into sections, don't regenerate
+5. Evaluate with probes, not lexical metrics
+
+## Related
+
+- [Context Optimization](./context-optimization.md)
+- [Evaluation](./evaluation.md)

package/skills/context-engineering/references/context-degradation.md

@@ -0,0 +1,93 @@
+# Context Degradation Patterns
+
+Predictable degradation as context grows. Not binary - a continuum.
+
+## Degradation Patterns
+
+| Pattern | Cause | Detection |
+|---------|-------|-----------|
+| **Lost-in-Middle** | U-shaped attention | Critical info recall drops 10-40% |
+| **Context Poisoning** | Errors compound via reference | Persistent hallucinations despite correction |
+| **Context Distraction** | Irrelevant info overwhelms | Single distractor degrades performance |
+| **Context Confusion** | Multiple tasks mix | Wrong tool calls, mixed requirements |
+| **Context Clash** | Contradictory info | Conflicting outputs, inconsistent reasoning |
+
+## Lost-in-Middle Phenomenon
+
+- Information in middle gets 10-40% lower recall
+- Models allocate massive attention to first token (BOS sink)
+- As context grows, middle tokens fail to get sufficient attention
+- **Mitigation**: Place critical info at beginning/end
+
+```markdown
+[CURRENT TASK]              # Beginning - high attention
+- Critical requirements
+
+[DETAILED CONTEXT]          # Middle - lower attention
+- Supporting details
+
+[KEY FINDINGS]              # End - high attention
+- Important conclusions
+```
+
+## Context Poisoning
+
+**Entry points**:
+1. Tool outputs with errors/unexpected formats
+2. Retrieved docs with incorrect/outdated info
+3. Model-generated summaries with hallucinations
+
+**Detection symptoms**:
+- Degraded quality on previously successful tasks
+- Tool misalignment (wrong tools/parameters)
+- Persistent hallucinations
+
+**Recovery**:
+- Truncate to before poisoning point
+- Explicit note + re-evaluation request
+- Restart with clean context, preserve only verified info
+
+## Model Degradation Thresholds
+
+| Model | Degradation Onset | Severe Degradation |
+|-------|-------------------|-------------------|
+| GPT-5.2 | ~64K tokens | ~200K tokens |
+| Claude Opus 4.5 | ~100K tokens | ~180K tokens |
+| Claude Sonnet 4.5 | ~80K tokens | ~150K tokens |
+| Gemini 3 Pro | ~500K tokens | ~800K tokens |
+
+## Four-Bucket Mitigation
+
+1. **Write**: Save externally (scratchpads, files)
+2. **Select**: Pull only relevant (retrieval, filtering)
+3. **Compress**: Reduce tokens (summarization)
+4. **Isolate**: Split across sub-agents (partitioning)
+
+## Detection Heuristics
+
+```python
+def calculate_health(utilization, degradation_risk, poisoning_risk):
+    """Health score: 1.0 = healthy, 0.0 = critical"""
+    score = 1.0
+    score -= utilization * 0.5 if utilization > 0.7 else 0
+    score -= degradation_risk * 0.3
+    score -= poisoning_risk * 0.2
+    return max(0, score)
+
+# Thresholds: healthy >0.8, warning >0.6, degraded >0.4, critical <=0.4
+```
+
+## Guidelines
+
+1. Monitor context length vs performance correlation
+2. Place critical info at beginning/end
+3. Implement compaction before degradation
+4. Validate retrieved docs before adding
+5. Use versioning to prevent outdated clash
+6. Segment tasks to prevent confusion
+7. Design for graceful degradation
+
+## Related Topics
+
+- [Context Optimization](./context-optimization.md) - Mitigation techniques
+- [Multi-Agent Patterns](./multi-agent-patterns.md) - Isolation strategies

package/skills/context-engineering/references/context-fundamentals.md

@@ -0,0 +1,75 @@
+# Context Fundamentals
+
+Context = all input provided to LLM for task completion.
+
+## Anatomy of Context
+
+| Component | Purpose | Token Impact |
+|-----------|---------|--------------|
+| System Prompt | Identity, constraints, guidelines | Stable, cacheable |
+| Tool Definitions | Action specs with params/returns | Grows with capabilities |
+| Retrieved Docs | Domain knowledge, just-in-time | Variable, selective |
+| Message History | Conversation state, task progress | Accumulates over time |
+| Tool Outputs | Results from actions | 83.9% of typical context |
+
+## Attention Mechanics
+
+- **U-shaped curve**: Beginning/end get more attention than middle
+- **Attention budget**: n^2 relationships for n tokens depletes with growth
+- **Position encoding**: Interpolation allows longer sequences with degradation
+- **First-token sink**: BOS token absorbs large attention budget
+
+## System Prompt Structure
+
+```xml
+<BACKGROUND_INFORMATION>Domain knowledge, role definition</BACKGROUND_INFORMATION>
+<INSTRUCTIONS>Step-by-step procedures</INSTRUCTIONS>
+<TOOL_GUIDANCE>When/how to use tools</TOOL_GUIDANCE>
+<OUTPUT_DESCRIPTION>Format requirements</OUTPUT_DESCRIPTION>
+```
+
+## Progressive Disclosure Levels
+
+1. **Metadata** (~100 words) - Always in context
+2. **SKILL.md body** (<5k words) - When skill triggers
+3. **Bundled resources** (Unlimited) - As needed
+
+## Token Budget Allocation
+
+| Component | Typical Range | Notes |
+|-----------|---------------|-------|
+| System Prompt | 500-2000 | Stable, optimize once |
+| Tool Definitions | 100-500 per tool | Keep under 20 tools |
+| Retrieved Docs | 1000-5000 | Selective loading |
+| Message History | Variable | Summarize at 70% |
+| Reserved Buffer | 10-20% | For responses |
+
+## Document Management
+
+**Strong identifiers**: `customer_pricing_rates.json` not `data/file1.json`
+**Chunk at semantic boundaries**: Paragraphs, sections, not arbitrary lengths
+**Include metadata**: Source, date, relevance score
+
+## Message History Pattern
+
+```python
+# Summary injection every 20 messages
+if len(messages) % 20 == 0:
+    summary = summarize_conversation(messages[-20:])
+    messages.append({"role": "system", "content": f"Summary: {summary}"})
+```
+
+## Guidelines
+
+1. Treat context as finite with diminishing returns
+2. Place critical info at attention-favored positions
+3. Use file-system-based access for large documents
+4. Pre-load stable content, just-in-time load dynamic
+5. Design with explicit token budgets
+6. Monitor usage, implement compaction triggers at 70-80%
+
+## Related Topics
+
+- [Context Degradation](./context-degradation.md) - Failure patterns
+- [Context Optimization](./context-optimization.md) - Efficiency techniques
+- [Memory Systems](./memory-systems.md) - External storage

package/skills/context-engineering/references/context-optimization.md

@@ -0,0 +1,82 @@
+# Context Optimization
+
+Extend effective context capacity through strategic techniques.
+
+## Four Core Strategies
+
+| Strategy | Target | Reduction | When to Use |
+|----------|--------|-----------|-------------|
+| **Compaction** | Full context | 50-70% | Approaching limits |
+| **Observation Masking** | Tool outputs | 60-80% | Verbose outputs >80% |
+| **KV-Cache Optimization** | Repeated prefixes | 70%+ hit | Stable prompts |
+| **Context Partitioning** | Work distribution | N/A | Parallelizable tasks |
+
+## Compaction
+
+Summarize context when approaching limits.
+
+**Priority**: Tool outputs → Old turns → Retrieved docs → Never: System prompt
+
+```python
+if context_tokens / context_limit > 0.8:
+    context = compact_context(context)
+```
+
+**Preserve**: Key findings, decisions, commitments (remove supporting details)
+
+## Observation Masking
+
+Replace verbose tool outputs with compact references.
+
+```python
+if len(observation) > max_length:
+    ref_id = store_observation(observation)
+    return f"[Obs:{ref_id}. Key: {extract_key(observation)}]"
+```
+
+**Never mask**: Current task critical, most recent turn, active reasoning
+**Always mask**: Repeated outputs, boilerplate, already summarized
+
+## KV-Cache Optimization
+
+Reuse cached Key/Value tensors for identical prefixes.
+
+```python
+# Cache-friendly ordering (stable first)
+context = [system_prompt, tool_definitions]  # Cacheable
+context += [unique_content]                   # Variable last
+```
+
+**Tips**: Avoid timestamps in stable sections, consistent formatting, stable structure
+
+## Context Partitioning
+
+Split work across sub-agents with isolated contexts.
+
+```python
+result = await sub_agent.process(subtask, clean_context=True)
+coordinator.receive(result.summary)  # Only essentials
+```
+
+## Decision Framework
+
+| Dominant Component | Apply |
+|-------------------|-------|
+| Tool outputs | Observation masking |
+| Retrieved docs | Summarization or partitioning |
+| Message history | Compaction + summarization |
+| Multiple | Combine strategies |
+
+## Guidelines
+
+1. Measure before optimizing
+2. Apply compaction before masking
+3. Design for cache stability
+4. Partition before context problematic
+5. Monitor effectiveness over time
+6. Balance savings vs quality
+
+## Related
+
+- [Context Compression](./context-compression.md)
+- [Memory Systems](./memory-systems.md)

package/skills/context-engineering/references/evaluation.md

@@ -0,0 +1,89 @@
+# Evaluation
+
+Systematically assess agent performance and context engineering choices.
+
+## Key Finding: 95% Performance Variance
+
+- **Token usage**: 80% of variance
+- **Tool calls**: ~10% of variance
+- **Model choice**: ~5% of variance
+
+**Implication**: Token budgets matter more than model upgrades.
+
+## Multi-Dimensional Rubric
+
+| Dimension | Weight | Description |
+|-----------|--------|-------------|
+| Factual Accuracy | 30% | Ground truth verification |
+| Completeness | 25% | Coverage of requirements |
+| Tool Efficiency | 20% | Appropriate tool usage |
+| Citation Accuracy | 15% | Sources match claims |
+| Source Quality | 10% | Authority/credibility |
+
+## Evaluation Methods
+
+### LLM-as-Judge
+
+Beware biases:
+- **Position**: First position preferred
+- **Length**: Longer = higher score
+- **Self-enhancement**: Rating own outputs higher
+- **Verbosity**: Detailed = better
+
+**Mitigation**: Position swapping, anti-bias prompting
+
+### Pairwise Comparison
+
+```python
+score_ab = judge.compare(output_a, output_b)
+score_ba = judge.compare(output_b, output_a)
+consistent = (score_ab > 0.5) != (score_ba > 0.5)
+```
+
+### Probe-Based Testing
+
+| Probe | Tests | Example |
+|-------|-------|---------|
+| Recall | Facts | "What was the error?" |
+| Artifact | Files | "Which files modified?" |
+| Continuation | Planning | "What's next?" |
+| Decision | Reasoning | "Why chose X?" |
+
+## Test Set Design
+
+```python
+class TestSet:
+    def sample_stratified(self, n):
+        per_level = n // 3
+        return (
+            sample(self.simple, per_level) +
+            sample(self.medium, per_level) +
+            sample(self.complex, per_level)
+        )
+```
+
+## Production Monitoring
+
+```python
+class Monitor:
+    sample_rate = 0.01  # 1% sampling
+    alert_threshold = 0.85
+
+    def check(self, scores):
+        if avg(scores) < self.alert_threshold:
+            self.alert(f"Quality degraded: {avg(scores):.2f}")
+```
+
+## Guidelines
+
+1. Start with outcome evaluation, not step-by-step
+2. Use multi-dimensional rubrics
+3. Mitigate LLM-as-Judge biases
+4. Test with stratified complexity
+5. Implement continuous monitoring
+6. Focus on token efficiency (80% variance)
+
+## Related
+
+- [Context Compression](./context-compression.md)
+- [Tool Design](./tool-design.md)

package/skills/context-engineering/references/memory-systems.md

@@ -0,0 +1,88 @@
+# Memory Systems
+
+Architectures for persistent context beyond the window.
+
+## Memory Layer Architecture
+
+| Layer | Scope | Persistence | Use Case |
+|-------|-------|-------------|----------|
+| L1: Working | Current window | None | Active reasoning |
+| L2: Short-Term | Session | Session | Task continuity |
+| L3: Long-Term | Cross-session | Persistent | User preferences |
+| L4: Entity | Per-entity | Persistent | Consistency |
+| L5: Temporal Graph | Time-aware | Persistent | Evolving facts |
+
+## Benchmark Performance (DMR Accuracy)
+
+| System | Accuracy | Approach |
+|--------|----------|----------|
+| Zep | 94.8% | Temporal knowledge graphs |
+| MemGPT | 93.4% | Hierarchical memory |
+| GraphRAG | 75-85% | Knowledge graphs |
+| Vector RAG | 60-70% | Embedding similarity |
+
+## Vector Store with Metadata
+
+```python
+class MetadataVectorStore:
+    def add(self, text, embedding, metadata):
+        doc = {
+            "text": text, "embedding": embedding,
+            "entities": metadata.get("entities", []),
+            "timestamp": metadata.get("timestamp")
+        }
+        self.index_by_entity(doc)
+
+    def search_by_entity(self, entity, k=5):
+        return self.entity_index.get(entity, [])[:k]
+```
+
+## Temporal Knowledge Graph
+
+```python
+class TemporalKnowledgeGraph:
+    def add_fact(self, subject, predicate, obj, valid_from, valid_to=None):
+        self.facts.append({
+            "triple": (subject, predicate, obj),
+            "valid_from": valid_from,
+            "valid_to": valid_to or "current"
+        })
+
+    def query_at_time(self, subject, predicate, timestamp):
+        for fact in self.facts:
+            if (fact["triple"][0] == subject and
+                fact["valid_from"] <= timestamp <= fact["valid_to"]):
+                return fact["triple"][2]
+```
+
+## Memory Retrieval Patterns
+
+| Pattern | Query | Use Case |
+|---------|-------|----------|
+| Semantic | "Similar to X" | General recall |
+| Entity-based | "About user John" | Consistency |
+| Temporal | "Valid on date" | Evolving facts |
+| Hybrid | Combine above | Production |
+
+## File-System-as-Memory
+
+```
+memory/
+├── sessions/{id}/summary.md
+├── entities/{id}.json
+└── facts/{timestamp}_{id}.json
+```
+
+## Guidelines
+
+1. Start with file-system-as-memory (simplest)
+2. Add vector search for scale
+3. Use entity indexing for consistency
+4. Add temporal awareness for evolving facts
+5. Implement consolidation for health
+6. Measure retrieval accuracy
+
+## Related
+
+- [Context Fundamentals](./context-fundamentals.md)
+- [Multi-Agent Patterns](./multi-agent-patterns.md)