port-nuker 1.0.1 → 1.0.3

package/.agent/skills/skills/loki-mode/references/tool-orchestration.md

@@ -1,691 +0,0 @@
-# Tool Orchestration Patterns Reference
-
-Research-backed patterns inspired by NVIDIA ToolOrchestra, OpenAI Agents SDK, and multi-agent coordination research.
-
----
-
-## Overview
-
-Effective tool orchestration requires four key innovations:
-1. **Tracing Spans** - Hierarchical event tracking (OpenAI SDK pattern)
-2. **Efficiency Metrics** - Track computational cost per task
-3. **Reward Signals** - Outcome, efficiency, and preference rewards for learning
-4. **Dynamic Selection** - Adapt agent count and types based on task complexity
-
----
-
-## Tracing Spans Architecture (OpenAI SDK Pattern)
-
-### Span Types
-
-Every operation is wrapped in a typed span for observability:
-
-```yaml
-span_types:
-  agent_span:     # Wraps entire agent execution
-  generation_span: # Wraps LLM API calls
-  function_span:  # Wraps tool/function calls
-  guardrail_span: # Wraps validation checks
-  handoff_span:   # Wraps agent-to-agent transfers
-  custom_span:    # User-defined operations
-```
-
-### Hierarchical Trace Structure
-
-```json
-{
-  "trace_id": "trace_abc123def456",
-  "workflow_name": "implement_feature",
-  "group_id": "session_xyz789",
-  "spans": [
-    {
-      "span_id": "span_001",
-      "parent_id": null,
-      "type": "agent_span",
-      "agent_name": "orchestrator",
-      "started_at": "2026-01-07T10:00:00Z",
-      "ended_at": "2026-01-07T10:05:00Z",
-      "children": ["span_002", "span_003"]
-    },
-    {
-      "span_id": "span_002",
-      "parent_id": "span_001",
-      "type": "guardrail_span",
-      "guardrail_name": "input_validation",
-      "triggered": false,
-      "blocking": true
-    },
-    {
-      "span_id": "span_003",
-      "parent_id": "span_001",
-      "type": "handoff_span",
-      "from_agent": "orchestrator",
-      "to_agent": "backend-dev"
-    }
-  ]
-}
-```
-
-### Storage Location
-
-```
-.loki/traces/
-├── active/
-│   └── {trace_id}.json     # Currently running traces
-└── completed/
-    └── {date}/
-        └── {trace_id}.json # Archived traces
-```
-
-See `references/openai-patterns.md` for full tracing implementation.
-
----
-
-## Efficiency Metrics System
-
-### Why Track Efficiency?
-
-ToolOrchestra achieves 70% cost reduction vs GPT-5 by explicitly optimizing for efficiency. Loki Mode should track:
-
-- **Token usage** per task (input + output)
-- **Wall clock time** per task
-- **Agent spawns** per task
-- **Retry count** before success
-
-### Efficiency Tracking Schema
-
-```json
-{
-  "task_id": "task-2026-01-06-001",
-  "correlation_id": "session-abc123",
-  "started_at": "2026-01-06T10:00:00Z",
-  "completed_at": "2026-01-06T10:05:32Z",
-  "metrics": {
-    "wall_time_seconds": 332,
-    "agents_spawned": 3,
-    "total_agent_calls": 7,
-    "retry_count": 1,
-    "retry_reasons": ["test_failure"],
-    "recovery_rate": 1.0,
-    "model_usage": {
-      "haiku": {"calls": 4, "est_tokens": 12000},
-      "sonnet": {"calls": 2, "est_tokens": 8000},
-      "opus": {"calls": 1, "est_tokens": 6000}
-    }
-  },
-  "outcome": "success",
-  "outcome_reason": "tests_passed_after_fix",
-  "efficiency_score": 0.85,
-  "efficiency_factors": ["used_haiku_for_tests", "parallel_review"],
-  "quality_pillars": {
-    "tool_selection_correct": true,
-    "tool_reliability_rate": 0.95,
-    "memory_retrieval_relevant": true,
-    "goal_adherence": 1.0
-  }
-}
-```
-
-**Why capture these metrics?** (Based on multi-agent research)
-
-1. **Capture intent, not just actions** ([Hashrocket](https://hashrocket.substack.com/p/the-hidden-cost-of-well-fix-it-later))
-   - "UX debt turns into data debt" - recording actions without intent creates useless analytics
-
-2. **Track recovery rate** ([Assessment Framework, arXiv 2512.12791](https://arxiv.org/html/2512.12791v1))
-   - `recovery_rate = successful_retries / total_retries`
-   - Paper found "perfect tool sequencing but only 33% policy adherence" - surface metrics mask failures
-
-3. **Distributed tracing** ([Maxim AI](https://www.getmaxim.ai/articles/best-practices-for-building-production-ready-multi-agent-systems/))
-   - `correlation_id`: Links all tasks in a session for end-to-end tracing
-   - Essential for debugging multi-agent coordination failures
-
-4. **Tool reliability separate from selection** ([Stanford/Harvard](https://www.marktechpost.com/2025/12/24/this-ai-paper-from-stanford-and-harvard-explains-why-most-agentic-ai-systems-feel-impressive-in-demos-and-then-completely-fall-apart-in-real-use/))
-   - `tool_selection_correct`: Did we pick the right tool?
-   - `tool_reliability_rate`: Did the tool work as expected? (tools can fail even when correctly selected)
-   - Key insight: "Tool use reliability" is a primary demo-to-deployment gap
-
-5. **Quality pillars beyond outcomes** ([Assessment Framework](https://arxiv.org/html/2512.12791v1))
-   - `memory_retrieval_relevant`: Did episodic/semantic retrieval help?
-   - `goal_adherence`: Did we stay on task? (0.0-1.0 score)
-
-### Efficiency Score Calculation
-
-```python
-def calculate_efficiency_score(metrics, task_complexity):
-    """
-    Score from 0-1 where higher is more efficient.
-    Based on ToolOrchestra's efficiency reward signal.
-    """
-    # Baseline expectations by complexity
-    baselines = {
-        "trivial": {"time": 60, "agents": 1, "retries": 0},
-        "simple": {"time": 180, "agents": 2, "retries": 0},
-        "moderate": {"time": 600, "agents": 4, "retries": 1},
-        "complex": {"time": 1800, "agents": 8, "retries": 2},
-        "critical": {"time": 3600, "agents": 12, "retries": 3}
-    }
-
-    baseline = baselines[task_complexity]
-
-    # Calculate component scores (1.0 = at baseline, >1 = better, <1 = worse)
-    time_score = min(1.0, baseline["time"] / max(metrics["wall_time_seconds"], 1))
-    agent_score = min(1.0, baseline["agents"] / max(metrics["agents_spawned"], 1))
-    retry_score = 1.0 - (metrics["retry_count"] / (baseline["retries"] + 3))
-
-    # Weighted average (time matters most)
-    return (time_score * 0.5) + (agent_score * 0.3) + (retry_score * 0.2)
-```
-
-### Standard Reason Codes
-
-Use consistent codes to enable pattern analysis:
-
-```yaml
-outcome_reasons:
-  success:
-    - tests_passed_first_try
-    - tests_passed_after_fix
-    - review_approved
-    - spec_validated
-  partial:
-    - tests_partial_pass
-    - review_concerns_minor
-    - timeout_partial_work
-  failure:
-    - tests_failed
-    - review_blocked
-    - dependency_missing
-    - timeout_no_progress
-    - error_unrecoverable
-
-retry_reasons:
-  - test_failure
-  - lint_error
-  - type_error
-  - review_rejection
-  - rate_limit
-  - timeout
-  - dependency_conflict
-
-efficiency_factors:
-  positive:
-    - used_haiku_for_simple
-    - parallel_execution
-    - cached_result
-    - first_try_success
-    - spec_driven
-  negative:
-    - used_opus_for_simple
-    - sequential_when_parallel_possible
-    - multiple_retries
-    - missing_context
-    - unclear_requirements
-```
-
-### Storage Location
-
-```
-.loki/metrics/
-├── efficiency/
-│   ├── 2026-01-06.json      # Daily efficiency logs
-│   └── aggregate.json        # Running averages by task type
-└── rewards/
-    ├── outcomes.json         # Task success/failure records
-    └── preferences.json      # User preference signals
-```
-
----
-
-## Reward Signal Framework
-
-### Three Reward Types (ToolOrchestra Pattern)
-
-```
-+------------------------------------------------------------------+
-| 1. OUTCOME REWARD                                                 |
-|    - Did the task succeed? Binary + quality grade                 |
-|    - Signal: +1.0 (success), 0.0 (partial), -1.0 (failure)       |
-+------------------------------------------------------------------+
-| 2. EFFICIENCY REWARD                                              |
-|    - Did we use resources wisely?                                 |
-|    - Signal: 0.0 to 1.0 based on efficiency score                |
-+------------------------------------------------------------------+
-| 3. PREFERENCE REWARD                                              |
-|    - Did the user like the approach/result?                       |
-|    - Signal: Inferred from user actions (accept/reject/modify)   |
-+------------------------------------------------------------------+
-```
-
-### Outcome Reward Implementation
-
-```python
-def calculate_outcome_reward(task_result):
-    """
-    Outcome reward based on task completion status.
-    """
-    if task_result.status == "completed":
-        # Grade the quality of completion
-        if task_result.tests_passed and task_result.review_passed:
-            return 1.0  # Full success
-        elif task_result.tests_passed:
-            return 0.7  # Tests pass but review had concerns
-        else:
-            return 0.3  # Completed but with issues
-
-    elif task_result.status == "partial":
-        return 0.0  # Partial completion, no reward
-
-    else:  # failed
-        return -1.0  # Negative reward for failure
-```
-
-### Preference Reward Implementation
-
-```python
-def infer_preference_reward(task_result, user_actions):
-    """
-    Infer user preference from their actions after task completion.
-    Based on implicit feedback patterns.
-    """
-    signals = []
-
-    # Positive signals
-    if "commit" in user_actions:
-        signals.append(0.8)  # User committed our changes
-    if "deploy" in user_actions:
-        signals.append(1.0)  # User deployed our changes
-    if "no_edits" in user_actions:
-        signals.append(0.6)  # User didn't modify our output
-
-    # Negative signals
-    if "revert" in user_actions:
-        signals.append(-1.0)  # User reverted our changes
-    if "manual_fix" in user_actions:
-        signals.append(-0.5)  # User had to fix our work
-    if "retry_different" in user_actions:
-        signals.append(-0.3)  # User asked for different approach
-
-    # Neutral (no signal)
-    if not signals:
-        return None
-
-    return sum(signals) / len(signals)
-```
-
-### Reward Aggregation for Learning
-
-```python
-def aggregate_rewards(outcome, efficiency, preference):
-    """
-    Combine rewards into single learning signal.
-    Weights based on ToolOrchestra findings.
-    """
-    # Outcome is most important (must succeed)
-    # Efficiency secondary (once successful, optimize)
-    # Preference tertiary (align with user style)
-
-    weights = {
-        "outcome": 0.6,
-        "efficiency": 0.25,
-        "preference": 0.15
-    }
-
-    total = outcome * weights["outcome"]
-    total += efficiency * weights["efficiency"]
-
-    if preference is not None:
-        total += preference * weights["preference"]
-    else:
-        # Redistribute weight if no preference signal
-        total = total / (1 - weights["preference"])
-
-    return total
-```
-
----
-
-## Dynamic Agent Selection
-
-### Task Complexity Classification
-
-```python
-def classify_task_complexity(task):
-    """
-    Classify task to determine agent allocation.
-    Based on ToolOrchestra's tool selection flexibility.
-    """
-    complexity_signals = {
-        # File scope signals
-        "single_file": -1,
-        "few_files": 0,       # 2-5 files
-        "many_files": +1,     # 6-20 files
-        "system_wide": +2,    # 20+ files
-
-        # Change type signals
-        "typo_fix": -2,
-        "bug_fix": 0,
-        "feature": +1,
-        "refactor": +1,
-        "architecture": +2,
-
-        # Domain signals
-        "documentation": -1,
-        "tests_only": 0,
-        "frontend": 0,
-        "backend": 0,
-        "full_stack": +1,
-        "infrastructure": +1,
-        "security": +2,
-    }
-
-    score = 0
-    for signal, weight in complexity_signals.items():
-        if task.has_signal(signal):
-            score += weight
-
-    # Map score to complexity level
-    if score <= -2:
-        return "trivial"
-    elif score <= 0:
-        return "simple"
-    elif score <= 2:
-        return "moderate"
-    elif score <= 4:
-        return "complex"
-    else:
-        return "critical"
-```
-
-### Agent Allocation by Complexity
-
-```yaml
-# Agent allocation strategy
-# Model selection: Opus=planning, Sonnet=development, Haiku=unit tests/monitoring
-complexity_allocations:
-  trivial:
-    max_agents: 1
-    planning: null         # No planning needed
-    development: haiku
-    testing: haiku
-    review: skip           # No review needed for trivial
-    parallel: false
-
-  simple:
-    max_agents: 2
-    planning: null         # No planning needed
-    development: haiku
-    testing: haiku
-    review: single         # One quick review
-    parallel: false
-
-  moderate:
-    max_agents: 4
-    planning: sonnet       # Sonnet for moderate planning
-    development: sonnet
-    testing: haiku         # Unit tests always haiku
-    review: standard       # 3 parallel reviewers
-    parallel: true
-
-  complex:
-    max_agents: 8
-    planning: opus         # Opus ONLY for complex planning
-    development: sonnet    # Sonnet for implementation
-    testing: haiku         # Unit tests still haiku
-    review: deep           # 3 reviewers + devil's advocate
-    parallel: true
-
-  critical:
-    max_agents: 12
-    planning: opus         # Opus for critical planning
-    development: sonnet    # Sonnet for implementation
-    testing: sonnet        # Functional/E2E tests with sonnet
-    review: exhaustive     # Multiple review rounds
-    parallel: true
-    human_checkpoint: true # Pause for human review
-```
-
-### Dynamic Selection Algorithm
-
-```python
-def select_agents_for_task(task, available_agents):
-    """
-    Dynamically select agents based on task requirements.
-    Inspired by ToolOrchestra's configurable tool selection.
-    """
-    complexity = classify_task_complexity(task)
-    allocation = COMPLEXITY_ALLOCATIONS[complexity]
-
-    # 1. Identify required agent types
-    required_types = identify_required_agents(task)
-
-    # 2. Filter to available agents of required types
-    candidates = [a for a in available_agents if a.type in required_types]
-
-    # 3. Score candidates by past performance
-    for agent in candidates:
-        agent.selection_score = get_agent_performance_score(
-            agent,
-            task_type=task.type,
-            complexity=complexity
-        )
-
-    # 4. Select top N agents up to allocation limit
-    candidates.sort(key=lambda a: a.selection_score, reverse=True)
-    selected = candidates[:allocation["max_agents"]]
-
-    # 5. Assign models based on complexity
-    for agent in selected:
-        if agent.role == "reviewer":
-            agent.model = "opus"  # Always opus for reviews
-        else:
-            agent.model = allocation["model"]
-
-    return selected
-
-def get_agent_performance_score(agent, task_type, complexity):
-    """
-    Score agent based on historical performance on similar tasks.
-    Uses reward signals from previous executions.
-    """
-    history = load_agent_history(agent.id)
-
-    # Filter to similar tasks
-    similar = [h for h in history
-               if h.task_type == task_type
-               and h.complexity == complexity]
-
-    if not similar:
-        return 0.5  # Neutral score if no history
-
-    # Average past rewards
-    return sum(h.aggregate_reward for h in similar) / len(similar)
-```
-
----
-
-## Tool Usage Analytics
-
-### Track Tool Effectiveness
-
-```json
-{
-  "tool_analytics": {
-    "period": "2026-01-06",
-    "by_tool": {
-      "Grep": {
-        "calls": 142,
-        "success_rate": 0.89,
-        "avg_result_quality": 0.82,
-        "common_patterns": ["error handling", "function def"]
-      },
-      "Task": {
-        "calls": 47,
-        "success_rate": 0.94,
-        "avg_efficiency": 0.76,
-        "by_subagent_type": {
-          "general-purpose": {"calls": 35, "success": 0.91},
-          "Explore": {"calls": 12, "success": 1.0}
-        }
-      }
-    },
-    "insights": [
-      "Explore agent 100% success - use more for codebase search",
-      "Grep success drops to 0.65 for regex patterns - simplify searches"
-    ]
-  }
-}
-```
-
-### Continuous Improvement Loop
-
-```
-+------------------------------------------------------------------+
-| 1. COLLECT                                                        |
-|    Record every task: agents used, tools called, outcome          |
-+------------------------------------------------------------------+
-          |
-          v
-+------------------------------------------------------------------+
-| 2. ANALYZE                                                        |
-|    Weekly aggregation: What worked? What didn't?                  |
-|    Identify patterns in high-reward vs low-reward tasks           |
-+------------------------------------------------------------------+
-          |
-          v
-+------------------------------------------------------------------+
-| 3. ADAPT                                                          |
-|    Update selection algorithms based on analytics                 |
-|    Store successful patterns in semantic memory                   |
-+------------------------------------------------------------------+
-          |
-          v
-+------------------------------------------------------------------+
-| 4. VALIDATE                                                       |
-|    A/B test new selection strategies                              |
-|    Measure efficiency improvement                                 |
-+------------------------------------------------------------------+
-          |
-          +-----------> Loop back to COLLECT
-```
-
----
-
-## Integration with RARV Cycle
-
-The orchestration patterns integrate with RARV at each phase:
-
-```
-REASON:
-├── Check efficiency metrics for similar past tasks
-├── Classify task complexity
-└── Select appropriate agent allocation
-
-ACT:
-├── Dispatch agents according to allocation
-├── Track start time and resource usage
-└── Record tool calls and agent interactions
-
-REFLECT:
-├── Calculate outcome reward (did it work?)
-├── Calculate efficiency reward (resource usage)
-└── Log to metrics store
-
-VERIFY:
-├── Run verification checks
-├── If failed: negative outcome reward, retry with learning
-├── If passed: infer preference reward from user actions
-└── Update agent performance scores
-```
-
----
-
-## Key Metrics Dashboard
-
-Track these metrics in `.loki/metrics/dashboard.json`:
-
-```json
-{
-  "dashboard": {
-    "period": "rolling_7_days",
-    "summary": {
-      "tasks_completed": 127,
-      "success_rate": 0.94,
-      "avg_efficiency_score": 0.78,
-      "avg_outcome_reward": 0.82,
-      "avg_preference_reward": 0.71,
-      "avg_recovery_rate": 0.87,
-      "avg_goal_adherence": 0.93
-    },
-    "quality_pillars": {
-      "tool_selection_accuracy": 0.91,
-      "tool_reliability_rate": 0.93,
-      "memory_retrieval_relevance": 0.84,
-      "policy_adherence": 0.96
-    },
-    "trends": {
-      "efficiency": "+12% vs previous week",
-      "success_rate": "+3% vs previous week",
-      "avg_agents_per_task": "-0.8 (improving)",
-      "recovery_rate": "+5% vs previous week"
-    },
-    "top_performing_patterns": [
-      "Haiku for unit tests (0.95 success, 0.92 efficiency)",
-      "Explore agent for codebase search (1.0 success)",
-      "Parallel review with opus (0.98 accuracy)"
-    ],
-    "areas_for_improvement": [
-      "Complex refactors taking 2x expected time",
-      "Security review efficiency below baseline",
-      "Memory retrieval relevance below 0.85 target"
-    ]
-  }
-}
-```
-
----
-
-## Multi-Dimensional Evaluation
-
-Based on [Measurement Imbalance research (arXiv 2506.02064)](https://arxiv.org/abs/2506.02064):
-
-> "Technical metrics dominate assessments (83%), while human-centered (30%), safety (53%), and economic (30%) remain peripheral"
-
-**Loki Mode tracks four evaluation axes:**
-
-| Axis | Metrics | Current Coverage |
-|------|---------|------------------|
-| **Technical** | success_rate, efficiency_score, recovery_rate | Full |
-| **Human-Centered** | preference_reward, goal_adherence | Partial |
-| **Safety** | policy_adherence, quality_gates_passed | Full (via review system) |
-| **Economic** | model_usage, agents_spawned, wall_time | Full |
-
----
-
-## Sources
-
-**OpenAI Agents SDK:**
-- [Agents SDK Documentation](https://openai.github.io/openai-agents-python/) - Core primitives: agents, handoffs, guardrails, tracing
-- [Practical Guide to Building Agents](https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf) - Orchestration patterns
-- [Building Agents Track](https://developers.openai.com/tracks/building-agents/) - Official developer guide
-- [AGENTS.md Specification](https://agents.md/) - Standard for agent instructions
-- [Tracing Documentation](https://openai.github.io/openai-agents-python/tracing/) - Span types and observability
-
-**Efficiency & Orchestration:**
-- [NVIDIA ToolOrchestra](https://github.com/NVlabs/ToolOrchestra) - Multi-turn tool orchestration with RL
-- [ToolScale Dataset](https://huggingface.co/datasets/nvidia/ToolScale) - Training data synthesis
-
-**Evaluation Frameworks:**
-- [Assessment Framework for Agentic AI (arXiv 2512.12791)](https://arxiv.org/html/2512.12791v1) - Four-pillar evaluation model
-- [Measurement Imbalance in Agentic AI (arXiv 2506.02064)](https://arxiv.org/abs/2506.02064) - Multi-dimensional evaluation
-- [Adaptive Monitoring for Agentic AI (arXiv 2509.00115)](https://arxiv.org/abs/2509.00115) - AMDM algorithm
-
-**Best Practices:**
-- [Anthropic: Building Effective Agents](https://www.anthropic.com/research/building-effective-agents) - Simplicity, transparency, tool engineering
-- [Maxim AI: Production Multi-Agent Systems](https://www.getmaxim.ai/articles/best-practices-for-building-production-ready-multi-agent-systems/) - Orchestration patterns, distributed tracing
-- [UiPath: Agent Builder Best Practices](https://www.uipath.com/blog/ai/agent-builder-best-practices) - Single-responsibility, evaluations
-- [Stanford/Harvard: Demo-to-Deployment Gap](https://www.marktechpost.com/2025/12/24/this-ai-paper-from-stanford-and-harvard-explains-why-most-agentic-ai-systems-feel-impressive-in-demos-and-then-completely-fall-apart-in-real-use/) - Tool reliability as key failure mode
-
-**Safety & Reasoning:**
-- [Chain of Thought Monitoring](https://openai.com/index/chain-of-thought-monitoring/) - CoT monitorability for safety
-- [Agent Builder Safety](https://platform.openai.com/docs/guides/agent-builder-safety) - Human-in-loop patterns
-- [Agentic AI Foundation](https://openai.com/index/agentic-ai-foundation/) - Industry standards (MCP, AGENTS.md, goose)