@vodailoc/kilo-kit-mcp 1.1.0 → 1.2.0

package/skills/engineering/context-engineering/references/evaluation.md

@@ -1,89 +1,89 @@
-# Evaluation
-
-Systematically assess agent performance and context engineering choices.
-
-## Key Finding: 95% Performance Variance
-
-- **Token usage**: 80% of variance
-- **Tool calls**: ~10% of variance
-- **Model choice**: ~5% of variance
-
-**Implication**: Token budgets matter more than model upgrades.
-
-## Multi-Dimensional Rubric
-
-| Dimension | Weight | Description |
-|-----------|--------|-------------|
-| Factual Accuracy | 30% | Ground truth verification |
-| Completeness | 25% | Coverage of requirements |
-| Tool Efficiency | 20% | Appropriate tool usage |
-| Citation Accuracy | 15% | Sources match claims |
-| Source Quality | 10% | Authority/credibility |
-
-## Evaluation Methods
-
-### LLM-as-Judge
-
-Beware biases:
-- **Position**: First position preferred
-- **Length**: Longer = higher score
-- **Self-enhancement**: Rating own outputs higher
-- **Verbosity**: Detailed = better
-
-**Mitigation**: Position swapping, anti-bias prompting
-
-### Pairwise Comparison
-
-```python
-score_ab = judge.compare(output_a, output_b)
-score_ba = judge.compare(output_b, output_a)
-consistent = (score_ab > 0.5) != (score_ba > 0.5)
-```
-
-### Probe-Based Testing
-
-| Probe | Tests | Example |
-|-------|-------|---------|
-| Recall | Facts | "What was the error?" |
-| Artifact | Files | "Which files modified?" |
-| Continuation | Planning | "What's next?" |
-| Decision | Reasoning | "Why chose X?" |
-
-## Test Set Design
-
-```python
-class TestSet:
-    def sample_stratified(self, n):
-        per_level = n // 3
-        return (
-            sample(self.simple, per_level) +
-            sample(self.medium, per_level) +
-            sample(self.complex, per_level)
-        )
-```
-
-## Production Monitoring
-
-```python
-class Monitor:
-    sample_rate = 0.01  # 1% sampling
-    alert_threshold = 0.85
-
-    def check(self, scores):
-        if avg(scores) < self.alert_threshold:
-            self.alert(f"Quality degraded: {avg(scores):.2f}")
-```
-
-## Guidelines
-
-1. Start with outcome evaluation, not step-by-step
-2. Use multi-dimensional rubrics
-3. Mitigate LLM-as-Judge biases
-4. Test with stratified complexity
-5. Implement continuous monitoring
-6. Focus on token efficiency (80% variance)
-
-## Related
-
-- [Context Compression](./context-compression.md)
-- [Tool Design](./tool-design.md)
+# Evaluation
+
+Systematically assess agent performance and context engineering choices.
+
+## Key Finding: 95% Performance Variance
+
+- **Token usage**: 80% of variance
+- **Tool calls**: ~10% of variance
+- **Model choice**: ~5% of variance
+
+**Implication**: Token budgets matter more than model upgrades.
+
+## Multi-Dimensional Rubric
+
+| Dimension | Weight | Description |
+|-----------|--------|-------------|
+| Factual Accuracy | 30% | Ground truth verification |
+| Completeness | 25% | Coverage of requirements |
+| Tool Efficiency | 20% | Appropriate tool usage |
+| Citation Accuracy | 15% | Sources match claims |
+| Source Quality | 10% | Authority/credibility |
+
+## Evaluation Methods
+
+### LLM-as-Judge
+
+Beware biases:
+- **Position**: First position preferred
+- **Length**: Longer = higher score
+- **Self-enhancement**: Rating own outputs higher
+- **Verbosity**: Detailed = better
+
+**Mitigation**: Position swapping, anti-bias prompting
+
+### Pairwise Comparison
+
+```python
+score_ab = judge.compare(output_a, output_b)
+score_ba = judge.compare(output_b, output_a)
+consistent = (score_ab > 0.5) != (score_ba > 0.5)
+```
+
+### Probe-Based Testing
+
+| Probe | Tests | Example |
+|-------|-------|---------|
+| Recall | Facts | "What was the error?" |
+| Artifact | Files | "Which files modified?" |
+| Continuation | Planning | "What's next?" |
+| Decision | Reasoning | "Why chose X?" |
+
+## Test Set Design
+
+```python
+class TestSet:
+    def sample_stratified(self, n):
+        per_level = n // 3
+        return (
+            sample(self.simple, per_level) +
+            sample(self.medium, per_level) +
+            sample(self.complex, per_level)
+        )
+```
+
+## Production Monitoring
+
+```python
+class Monitor:
+    sample_rate = 0.01  # 1% sampling
+    alert_threshold = 0.85
+
+    def check(self, scores):
+        if avg(scores) < self.alert_threshold:
+            self.alert(f"Quality degraded: {avg(scores):.2f}")
+```
+
+## Guidelines
+
+1. Start with outcome evaluation, not step-by-step
+2. Use multi-dimensional rubrics
+3. Mitigate LLM-as-Judge biases
+4. Test with stratified complexity
+5. Implement continuous monitoring
+6. Focus on token efficiency (80% variance)
+
+## Related
+
+- [Context Compression](./context-compression.md)
+- [Tool Design](./tool-design.md)

package/skills/engineering/context-engineering/references/memory-systems.md

@@ -1,88 +1,88 @@
-# Memory Systems
-
-Architectures for persistent context beyond the window.
-
-## Memory Layer Architecture
-
-| Layer | Scope | Persistence | Use Case |
-|-------|-------|-------------|----------|
-| L1: Working | Current window | None | Active reasoning |
-| L2: Short-Term | Session | Session | Task continuity |
-| L3: Long-Term | Cross-session | Persistent | User preferences |
-| L4: Entity | Per-entity | Persistent | Consistency |
-| L5: Temporal Graph | Time-aware | Persistent | Evolving facts |
-
-## Benchmark Performance (DMR Accuracy)
-
-| System | Accuracy | Approach |
-|--------|----------|----------|
-| Zep | 94.8% | Temporal knowledge graphs |
-| MemGPT | 93.4% | Hierarchical memory |
-| GraphRAG | 75-85% | Knowledge graphs |
-| Vector RAG | 60-70% | Embedding similarity |
-
-## Vector Store with Metadata
-
-```python
-class MetadataVectorStore:
-    def add(self, text, embedding, metadata):
-        doc = {
-            "text": text, "embedding": embedding,
-            "entities": metadata.get("entities", []),
-            "timestamp": metadata.get("timestamp")
-        }
-        self.index_by_entity(doc)
-
-    def search_by_entity(self, entity, k=5):
-        return self.entity_index.get(entity, [])[:k]
-```
-
-## Temporal Knowledge Graph
-
-```python
-class TemporalKnowledgeGraph:
-    def add_fact(self, subject, predicate, obj, valid_from, valid_to=None):
-        self.facts.append({
-            "triple": (subject, predicate, obj),
-            "valid_from": valid_from,
-            "valid_to": valid_to or "current"
-        })
-
-    def query_at_time(self, subject, predicate, timestamp):
-        for fact in self.facts:
-            if (fact["triple"][0] == subject and
-                fact["valid_from"] <= timestamp <= fact["valid_to"]):
-                return fact["triple"][2]
-```
-
-## Memory Retrieval Patterns
-
-| Pattern | Query | Use Case |
-|---------|-------|----------|
-| Semantic | "Similar to X" | General recall |
-| Entity-based | "About user John" | Consistency |
-| Temporal | "Valid on date" | Evolving facts |
-| Hybrid | Combine above | Production |
-
-## File-System-as-Memory
-
-```
-memory/
-├── sessions/{id}/summary.md
-├── entities/{id}.json
-└── facts/{timestamp}_{id}.json
-```
-
-## Guidelines
-
-1. Start with file-system-as-memory (simplest)
-2. Add vector search for scale
-3. Use entity indexing for consistency
-4. Add temporal awareness for evolving facts
-5. Implement consolidation for health
-6. Measure retrieval accuracy
-
-## Related
-
-- [Context Fundamentals](./context-fundamentals.md)
-- [Multi-Agent Patterns](./multi-agent-patterns.md)
+# Memory Systems
+
+Architectures for persistent context beyond the window.
+
+## Memory Layer Architecture
+
+| Layer | Scope | Persistence | Use Case |
+|-------|-------|-------------|----------|
+| L1: Working | Current window | None | Active reasoning |
+| L2: Short-Term | Session | Session | Task continuity |
+| L3: Long-Term | Cross-session | Persistent | User preferences |
+| L4: Entity | Per-entity | Persistent | Consistency |
+| L5: Temporal Graph | Time-aware | Persistent | Evolving facts |
+
+## Benchmark Performance (DMR Accuracy)
+
+| System | Accuracy | Approach |
+|--------|----------|----------|
+| Zep | 94.8% | Temporal knowledge graphs |
+| MemGPT | 93.4% | Hierarchical memory |
+| GraphRAG | 75-85% | Knowledge graphs |
+| Vector RAG | 60-70% | Embedding similarity |
+
+## Vector Store with Metadata
+
+```python
+class MetadataVectorStore:
+    def add(self, text, embedding, metadata):
+        doc = {
+            "text": text, "embedding": embedding,
+            "entities": metadata.get("entities", []),
+            "timestamp": metadata.get("timestamp")
+        }
+        self.index_by_entity(doc)
+
+    def search_by_entity(self, entity, k=5):
+        return self.entity_index.get(entity, [])[:k]
+```
+
+## Temporal Knowledge Graph
+
+```python
+class TemporalKnowledgeGraph:
+    def add_fact(self, subject, predicate, obj, valid_from, valid_to=None):
+        self.facts.append({
+            "triple": (subject, predicate, obj),
+            "valid_from": valid_from,
+            "valid_to": valid_to or "current"
+        })
+
+    def query_at_time(self, subject, predicate, timestamp):
+        for fact in self.facts:
+            if (fact["triple"][0] == subject and
+                fact["valid_from"] <= timestamp <= fact["valid_to"]):
+                return fact["triple"][2]
+```
+
+## Memory Retrieval Patterns
+
+| Pattern | Query | Use Case |
+|---------|-------|----------|
+| Semantic | "Similar to X" | General recall |
+| Entity-based | "About user John" | Consistency |
+| Temporal | "Valid on date" | Evolving facts |
+| Hybrid | Combine above | Production |
+
+## File-System-as-Memory
+
+```
+memory/
+├── sessions/{id}/summary.md
+├── entities/{id}.json
+└── facts/{timestamp}_{id}.json
+```
+
+## Guidelines
+
+1. Start with file-system-as-memory (simplest)
+2. Add vector search for scale
+3. Use entity indexing for consistency
+4. Add temporal awareness for evolving facts
+5. Implement consolidation for health
+6. Measure retrieval accuracy
+
+## Related
+
+- [Context Fundamentals](./context-fundamentals.md)
+- [Multi-Agent Patterns](./multi-agent-patterns.md)

package/skills/engineering/context-engineering/references/multi-agent-patterns.md

@@ -1,90 +1,90 @@
-# Multi-Agent Patterns
-
-Distribute work across multiple context windows for isolation and scale.
-
-## Core Insight
-
-Sub-agents exist to **isolate context**, not anthropomorphize roles.
-
-## Token Economics
-
-| Architecture | Multiplier | Use Case |
-|--------------|------------|----------|
-| Single agent | 1x | Simple tasks |
-| Single + tools | ~4x | Moderate complexity |
-| Multi-agent | ~15x | Context isolation needed |
-
-**Key**: Token usage explains 80% of performance variance.
-
-## Patterns
-
-### Supervisor/Orchestrator
-
-```python
-class Supervisor:
-    def process(self, task):
-        subtasks = self.decompose(task)
-        results = [worker.execute(st, clean_context=True) for st in subtasks]
-        return self.aggregate(results)
-```
-
-**Pros**: Control, human-in-loop | **Cons**: Bottleneck, telephone game
-
-### Peer-to-Peer/Swarm
-
-```python
-def process_with_handoff(agent, task):
-    result = agent.process(task)
-    if "handoff" in result:
-        return process_with_handoff(select_agent(result["to"]), result["state"])
-    return result
-```
-
-**Pros**: No SPOF, scales | **Cons**: Complex coordination
-
-### Hierarchical
-
-Strategy → Planning → Execution layers
-**Pros**: Separation of concerns | **Cons**: Coordination overhead
-
-## Context Isolation Patterns
-
-| Pattern | Isolation | Use Case |
-|---------|-----------|----------|
-| Full delegation | None | Max capability |
-| Instruction passing | High | Simple tasks |
-| File coordination | Medium | Shared state |
-
-## Consensus Mechanisms
-
-```python
-def weighted_consensus(responses):
-    scores = {}
-    for r in responses:
-        weight = r["confidence"] * r["expertise"]
-        scores[r["answer"]] = scores.get(r["answer"], 0) + weight
-    return max(scores, key=scores.get)
-```
-
-## Failure Recovery
-
-| Failure | Mitigation |
-|---------|------------|
-| Bottleneck | Output schemas, checkpointing |
-| Overhead | Clear handoffs, batching |
-| Divergence | Boundaries, convergence checks |
-| Errors | Validation, circuit breakers |
-
-## Guidelines
-
-1. Use multi-agent for context isolation, not role-play
-2. Accept ~15x token cost for benefits
-3. Implement circuit breakers
-4. Use files for shared state
-5. Design clear handoffs
-6. Validate between agents
-
-## Related
-
-- [Context Optimization](./context-optimization.md)
-- [Evaluation](./evaluation.md)
+# Multi-Agent Patterns
+
+Distribute work across multiple context windows for isolation and scale.
+
+## Core Insight
+
+Sub-agents exist to **isolate context**, not anthropomorphize roles.
+
+## Token Economics
+
+| Architecture | Multiplier | Use Case |
+|--------------|------------|----------|
+| Single agent | 1x | Simple tasks |
+| Single + tools | ~4x | Moderate complexity |
+| Multi-agent | ~15x | Context isolation needed |
+
+**Key**: Token usage explains 80% of performance variance.
+
+## Patterns
+
+### Supervisor/Orchestrator
+
+```python
+class Supervisor:
+    def process(self, task):
+        subtasks = self.decompose(task)
+        results = [worker.execute(st, clean_context=True) for st in subtasks]
+        return self.aggregate(results)
+```
+
+**Pros**: Control, human-in-loop | **Cons**: Bottleneck, telephone game
+
+### Peer-to-Peer/Swarm
+
+```python
+def process_with_handoff(agent, task):
+    result = agent.process(task)
+    if "handoff" in result:
+        return process_with_handoff(select_agent(result["to"]), result["state"])
+    return result
+```
+
+**Pros**: No SPOF, scales | **Cons**: Complex coordination
+
+### Hierarchical
+
+Strategy → Planning → Execution layers
+**Pros**: Separation of concerns | **Cons**: Coordination overhead
+
+## Context Isolation Patterns
+
+| Pattern | Isolation | Use Case |
+|---------|-----------|----------|
+| Full delegation | None | Max capability |
+| Instruction passing | High | Simple tasks |
+| File coordination | Medium | Shared state |
+
+## Consensus Mechanisms
+
+```python
+def weighted_consensus(responses):
+    scores = {}
+    for r in responses:
+        weight = r["confidence"] * r["expertise"]
+        scores[r["answer"]] = scores.get(r["answer"], 0) + weight
+    return max(scores, key=scores.get)
+```
+
+## Failure Recovery
+
+| Failure | Mitigation |
+|---------|------------|
+| Bottleneck | Output schemas, checkpointing |
+| Overhead | Clear handoffs, batching |
+| Divergence | Boundaries, convergence checks |
+| Errors | Validation, circuit breakers |
+
+## Guidelines
+
+1. Use multi-agent for context isolation, not role-play
+2. Accept ~15x token cost for benefits
+3. Implement circuit breakers
+4. Use files for shared state
+5. Design clear handoffs
+6. Validate between agents
+
+## Related
+
+- [Context Optimization](./context-optimization.md)
+- [Evaluation](./evaluation.md)