@rubix0270/arboris 1.0.2 → 1.0.3

package/prisma/skills/agent-self-evaluation/references/hook-integration.md

@@ -1,64 +0,0 @@
-# Hook Integration for Session-Stop Self-Evaluation
-
-Add this hook to `hooks/hooks.json` to remind the agent to self-evaluate at the end of every session (the hook echoes a reminder; it does not run the evaluator automatically):
-
-```json
-{
-  "hooks": {
-    "Stop": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "echo '[Self-Eval] Session complete. Consider running agent-self-evaluation to rate your output.'"
-          }
-        ],
-        "description": "Remind agent to self-evaluate at session end"
-      }
-    ]
-  }
-}
-```
-
-`Stop` events do not require a `matcher` field (it is optional for `Stop`, `Notification`, `UserPromptSubmit`, and `SubagentStop` per `scripts/ci/validate-hooks.js`). If omitted, the hook object only needs `hooks` and metadata such as `description`.
-
-## Integration with the Python Evaluator
-
-The `scripts/evaluate.py` script can be used as a standalone tool:
-
-```bash
-# Pipe agent output directly
-echo "Your agent response here" | python3 skills/agent-self-evaluation/scripts/evaluate.py
-
-# From files
-python3 skills/agent-self-evaluation/scripts/evaluate.py --task task.txt --output response.txt
-```
-
-To integrate it into hooks, capture the last agent output to a file first, then run the evaluator. For lightweight reminders after shell-based verification, use a simple supported matcher string:
-
-```json
-{
-  "hooks": {
-    "PostToolUse": [
-      {
-        "matcher": "Bash",
-        "hooks": [
-          {
-            "type": "command",
-            "command": "echo '[Self-Eval] If this command completed verification for a non-trivial task, consider running agent-self-evaluation.'"
-          }
-        ],
-        "description": "Remind agent to self-evaluate after shell verification"
-      }
-    ]
-  }
-}
-```
-
-This avoids documenting unsupported command-expression matcher syntax. If your harness supports command-level matcher expressions, prefer a word-boundary regex such as `\b(pytest|npm test|go test)\b` rather than a broad `test` substring.
-
-These hooks are opt-in. Add them to your local `hooks/hooks.json` if you want automated evaluation prompts.
-
-## Manual Usage (Recommended)
-
-The most reliable approach is manual invocation — the agent runs self-evaluation as part of its workflow when the `agent-self-evaluation` skill is active, without requiring hook configuration. The skill's "When to Activate" section already covers trigger conditions (multi-file changes, debugging sessions, design documents).

package/prisma/skills/agent-self-evaluation/scripts/evaluate.py

@@ -1,408 +0,0 @@
-#!/usr/bin/env python3
-"""Standalone agent output evaluator using the 5-axis rubric.
-
-Reads a task description and agent output from stdin or files,
-scores each axis, and prints a structured evaluation report.
-
-Usage:
-    # Pipe output directly
-    echo "Task: Add retry logic" | evaluate.py --output response.txt
-
-    # From files
-    evaluate.py --task task.txt --output response.txt
-
-    # Interactive (reads task from prompt, output from stdin)
-    evaluate.py --interactive
-
-The evaluator uses keyword heuristics + structural checks as a first pass.
-For production use, pair with an LLM judge for semantic understanding.
-"""
-
-import argparse
-import re
-import sys
-from dataclasses import dataclass, field
-from typing import Optional
-
-# Tunable thresholds for evaluation heuristics
-WALL_OF_TEXT_WORDS = 200
-SUMMARY_CHECK_WORDS = 300
-SUMMARY_CHECK_FIRST_N = 100
-TASK_OUTPUT_RATIO_HIGH = 15
-TASK_OUTPUT_RATIO_MEDIUM = 8
-
-
-@dataclass
-class AxisScore:
-    name: str
-    score: int
-    evidence: list[str] = field(default_factory=list)
-    improvement: Optional[str] = None
-
-
-def count_words(text: str) -> int:
-    return len(text.split())
-
-
-def check_accuracy(text: str) -> AxisScore:
-    """Check for verifiable claims, tool output references, error signs."""
-    evidence = []
-    deductions = 0
-    score = 5
-
-    # Positive signals: verified claims
-    verified_patterns = [
-        (r"(?i)(tests?\s+pass|all\s+tests?\s+passing|\d+\s+passed)", "Tests passing"),
-        (r"(?i)(exit\s+code\s*[:=]?\s*0|exited\s+with\s+0)", "Clean exit code"),
-        (r"(?i)(lint.*clean|no\s+lint\s+errors|0\s+errors)", "Lint clean"),
-        (r"(?i)(verified|confirmed|validated)\s+(with|against|using|by)", "Explicit verification"),
-        (r"(?i)(grep|rg)\s+.*\b(found|matched|returned)", "Grep confirmed"),
-    ]
-    for pattern, label in verified_patterns:
-        if re.search(pattern, text):
-            evidence.append(f"+ {label}")
-
-    # Negative signals: unverified claims
-    danger_patterns = [
-        (r"(?i)(should\s+work|probably\s+fine|should\s+be\s+ok)", "Hedged claim without verification"),
-        (r"(?i)(I\s+think|I\s+believe|I\s+assume|might\s+be)", "Speculation without evidence"),
-        (r"(?i)(untested|not\s+tested|haven'?t\s+tested)", "Explicitly untested"),
-        (r"(?i)(TODO|FIXME|HACK|WORKAROUND)", "Unresolved TODO/FIXME"),
-    ]
-    for pattern, label in danger_patterns:
-        if re.search(pattern, text):
-            deductions += 1
-            evidence.append(f"- {label}")
-
-    if deductions >= 3:
-        score = 2
-    elif deductions == 2:
-        score = 3
-    elif deductions == 1:
-        score = 4
-
-    if not evidence:
-        evidence.append("No verification signals detected — score assumes correctness")
-
-    result = AxisScore(name="Accuracy", score=score, evidence=evidence)
-    if score < 5:
-        result.improvement = "Cite specific tool outputs (test results, exit codes, grep findings) to back claims"
-    return result
-
-
-def check_completeness(text: str) -> AxisScore:
-    """Check for requirement coverage, edge cases, error handling."""
-    evidence = []
-    score = 5
-
-    # Positive signals
-    completeness_signals = [
-        (r"(?i)(edge\s*cases?|corner\s*cases?)", "Edge cases addressed"),
-        (r"(?i)(error\s*handling|exception\s*handling|try/except|try\s*{)", "Error handling present"),
-        (r"(?i)(all\s+\w+\s+(methods|endpoints|routes))", "Full coverage claimed"),
-        (r"(?i)(verification|verified\s+that|confirmed\s+that)", "Verification step present"),
-    ]
-    for pattern, label in completeness_signals:
-        if re.search(pattern, text):
-            evidence.append(f"+ {label}")
-
-    # Gaps
-    gap_signals = [
-        (r"(?i)(not\s+covered|not\s+handled|out\s+of\s+scope)", "Explicit gap acknowledged"),
-        (r"(?i)(only\s+(works|handles|supports)\s+\w+)", "Limited scope noted"),
-        (r"(?i)(assume[sd]?\s+that|assuming\s+the)", "Assumption without verification"),
-    ]
-    deductions = 0
-    for pattern, label in gap_signals:
-        if re.search(pattern, text):
-            deductions += 1
-            evidence.append(f"- {label}")
-
-    if deductions >= 2:
-        score = 3
-    elif deductions == 1:
-        score = 4
-
-    if not evidence:
-        evidence.append("No completeness signals — unable to assess coverage")
-
-    result = AxisScore(name="Completeness", score=score, evidence=evidence)
-    if score < 5:
-        result.improvement = "List what was covered AND what was intentionally excluded, with reasoning"
-    return result
-
-
-def _check_jargon(text: str) -> tuple[int, list[str]]:
-    """Return clarity deductions for unexplained domain jargon."""
-    jargon = [
-        (r"\b(idempotent|race condition|deadlock|thundering herd)\b", "concurrency"),
-        (r"\b(exponential backoff|circuit breaker|bulkhead)\b", "resilience"),
-        (r"\b(ACID|CAP|eventual consistency|linearizability)\b", "database theory"),
-    ]
-    explanation_pattern = r"(?i)({domain}|means|refers to|i\.e\.|in other words)"
-    for pattern, domain in jargon:
-        has_term = re.search(pattern, text, re.IGNORECASE)
-        explains_term = re.search(explanation_pattern.format(domain=domain), text)
-        if has_term and not explains_term:
-            return 1, [f"- Domain term used without explanation ({domain})"]
-    return 0, []
-
-
-def _check_summary(text: str) -> tuple[int, list[str]]:
-    """Return clarity deduction when long output lacks an early summary."""
-    summary_terms = ["summary", "tldr", "overview", "in short"]
-    has_early_summary = any(term in ' '.join(text.split()[:SUMMARY_CHECK_FIRST_N]).lower() for term in summary_terms)
-    if not has_early_summary and count_words(text) > SUMMARY_CHECK_WORDS:
-        return 1, ["- No summary/TLDR in first 100 words (text is 300+ words)"]
-    return 0, []
-
-
-def check_clarity(text: str) -> AxisScore:
-    """Check for structure, readability, jargon handling."""
-    evidence = []
-    deductions = 0
-
-    if re.search(r"^#{1,3}\s+", text, re.MULTILINE):
-        evidence.append("+ Uses headings for structure")
-    if re.search(r"```", text):
-        evidence.append("+ Uses code blocks")
-    if re.search(r"^\s*[-*]\s+", text, re.MULTILINE):
-        evidence.append("+ Uses bullet points")
-
-    for paragraph in [p for p in text.split("\n\n") if p.strip()]:
-        if count_words(paragraph) > WALL_OF_TEXT_WORDS:
-            deductions += 1
-            evidence.append("- Wall-of-text paragraph (>200 words without break)")
-            break
-
-    jargon_deductions, jargon_evidence = _check_jargon(text)
-    summary_deductions, summary_evidence = _check_summary(text)
-    deductions += jargon_deductions + summary_deductions
-    evidence.extend(jargon_evidence + summary_evidence)
-
-    if deductions >= 3:
-        score = 2
-    elif deductions == 2:
-        score = 3
-    elif deductions == 1:
-        score = 4
-    else:
-        score = 5
-
-    if not evidence:
-        evidence.append("+ Well-structured with no clarity issues detected")
-
-    result = AxisScore(name="Clarity", score=score, evidence=evidence)
-    if score < 5:
-        result.improvement = "Add headings, break long paragraphs, define domain terms on first use"
-    return result
-
-
-def check_actionability(text: str) -> AxisScore:
-    """Check if the user can act on the output immediately."""
-    evidence = []
-    score = 5
-    deductions = 0
-
-    # Positive signals
-    actionable_signals = [
-        (r"(?i)(merge|PR|pull request).*?(created|ready|open)", "PR created"),
-        (r"(?i)(run|execute)\s+[`\"']?[\w./-]+", "Specific run command given"),
-        (r"(?i)(next\s+steps?|follow[- ]up|what\s+to\s+do)", "Next steps provided"),
-        (r"(?i)(file\s+(created|written|modified|updated)\s+at)", "File path specified"),
-    ]
-    for pattern, label in actionable_signals:
-        if re.search(pattern, text):
-            evidence.append(f"+ {label}")
-
-    # Negative signals
-    vague_signals = [
-        (r"(?i)(you\s+(should|could|might\s+want\s+to))\s+\w+", "Vague suggestion without specifics"),
-        (r"(?i)(consider|maybe|perhaps)\s+\w+ing", "Non-committal suggestion"),
-        (r"(?i)(figure\s+out|look\s+into|investigate)\s", "Defers work to user"),
-    ]
-    for pattern, label in vague_signals:
-        if re.search(pattern, text):
-            deductions += 1
-            evidence.append(f"- {label}")
-
-    if deductions >= 3:
-        score = 2
-    elif deductions == 2:
-        score = 3
-    elif deductions == 1:
-        score = 4
-
-    if not evidence:
-        evidence.append("No actionability signals — user may need to ask 'what now?'")
-
-    result = AxisScore(name="Actionability", score=score, evidence=evidence)
-    if score < 5:
-        result.improvement = "End with a single clear action: 'Merge this PR', 'Run ./deploy.sh', or 'Review the 3 changed files'"
-    return result
-
-
-def check_conciseness(text: str, task: Optional[str] = None) -> AxisScore:
-    """Check for redundancy, filler, information density."""
-    evidence = []
-    score = 5
-    wc = count_words(text)
-
-    # Heuristic: task-to-output ratio
-    if task:
-        task_wc = count_words(task)
-        ratio = wc / max(task_wc, 1)
-        if ratio > TASK_OUTPUT_RATIO_HIGH:
-            evidence.append(f"- Output is {ratio:.0f}x longer than task description (high ratio)")
-            score = min(score, 3)
-        elif ratio > TASK_OUTPUT_RATIO_MEDIUM:
-            evidence.append(f"- Output is {ratio:.0f}x longer than task description")
-            score = min(score, 4)
-
-    # Redundancy signals
-    redundancy_checks = [
-        (r"(?i)(as\s+(I|we)\s+(mentioned|said|noted|discussed)\s+(earlier|above|before))",
-         "Refers back to earlier statement (possible repetition)"),
-        (r"(?i)(to\s+summarize|in\s+summary|in\s+conclusion|to\s+conclude)",
-         "Has explicit summary (good if needed, flag if redundant)"),
-        (r"(?i)(let\s+me\s+(explain|break\s+this\s+down|walk\s+you\s+through))",
-         "Meta-commentary adds words without information"),
-    ]
-    redundant_count = 0
-    for pattern, label in redundancy_checks:
-        matches = re.findall(pattern, text)
-        if len(matches) > 2:
-            redundant_count += 1
-            evidence.append(f"- '{label}' appears {len(matches)} times")
-
-    if redundant_count >= 2:
-        score = min(score, 3)
-    elif redundant_count == 1:
-        score = min(score, 4)
-
-    if not evidence and score == 5:
-        evidence.append("+ No redundancy detected. Information density appears good.")
-
-    result = AxisScore(name="Conciseness", score=score, evidence=evidence)
-    if score < 5:
-        result.improvement = "Cut meta-commentary, remove repeated points, trim examples to one representative case"
-    return result
-
-
-def evaluate(task: Optional[str], output: str) -> list[AxisScore]:
-    """Run all 5 axis checks and return scored results."""
-    return [
-        check_accuracy(output),
-        check_completeness(output),
-        check_clarity(output),
-        check_actionability(output),
-        check_conciseness(output, task),
-    ]
-
-
-def format_report(scores: list[AxisScore]) -> str:
-    """Format scores into a readable evaluation report."""
-    avg = sum(s.score for s in scores) / len(scores)
-    lines = []
-    lines.append("=" * 60)
-    lines.append("AGENT SELF-EVALUATION REPORT")
-    lines.append("=" * 60)
-    lines.append(f"Summary: Overall score {avg:.1f}/5 across 5 quality axes.")
-    lines.append("")
-
-    for s in scores:
-        bar = "█" * s.score + "░" * (5 - s.score)
-        lines.append(f"  {s.name:<15} {bar} {s.score}/5")
-        lines.extend(f"    {e}" for e in s.evidence)
-        if s.improvement:
-            lines.append(f"    → {s.improvement}")
-        lines.append("")
-
-    lines.append(f"  {'OVERALL':<15} {avg:.1f}/5")
-    lines.append("")
-
-    # Critical issues (axes ≤ 2)
-    critical = [(s, s.improvement or "No improvement suggested") for s in scores if s.score <= 2]
-    lines.append("CRITICAL ISSUES (axes ≤ 2):")
-    if critical:
-        for s, imp in critical:
-            lines.append(f"  [{s.name}] Score {s.score}/5 — {imp}")
-    else:
-        lines.append("  None")
-
-    lines.append("")
-    lines.append("Self-check: Would the user agree with this assessment? [Yes/No + brief justification]")
-    lines.append("")
-
-    # Top improvements (axes scoring < 4, ranked by impact)
-    improvements = [(s, s.improvement) for s in scores if s.improvement and s.score < 4]
-    lines.append("TOP IMPROVEMENTS:")
-    if improvements:
-        for i, (s, imp) in enumerate(sorted(improvements, key=lambda x: x[0].score), 1):
-            lines.append(f"  {i}. [{s.name}] {imp}")
-    else:
-        lines.append("  No axes below 4. Strong output across all dimensions.")
-
-    lines.append("")
-
-    # Verdict
-    min_score = min(s.score for s in scores)
-    if min_score <= 2:
-        verdict = f"Redo with specific fixes. Weakest axis: {min(scores, key=lambda s: s.score).name} ({min_score}/5)."
-    elif any(s.score <= 3 for s in scores):
-        weak = [s.name for s in scores if s.score <= 3]
-        verdict = f"Fix {'/'.join(weak)} issues, then deliver."
-    elif avg >= 4.5:
-        verdict = "Deliver as-is. No changes needed."
-    else:
-        verdict = "Deliver as-is. Minor improvements noted above."
-    lines.append(f"VERDICT: {verdict}")
-
-    return "\n".join(lines)
-
-
-def _read_file_or_text(path: Optional[str], *, required: bool = False) -> Optional[str]:
-    """Read a file path or return inline text when allowed."""
-    if path is None:
-        return None
-    try:
-        with open(path) as f:
-            return f.read()
-    except FileNotFoundError:
-        if required:
-            print(f"Error: output file '{path}' not found", file=sys.stderr)
-            sys.exit(1)
-        return path
-
-
-def _read_input(args: argparse.Namespace) -> tuple[Optional[str], str]:
-    """Read task and output for interactive, file, or pipe mode."""
-    if args.interactive:
-        task = input("Task description: ").strip()
-        print("Paste agent output (Ctrl+D to finish):")
-        return task, sys.stdin.read()
-    if args.output:
-        return _read_file_or_text(args.task), _read_file_or_text(args.output, required=True) or ""
-    return _read_file_or_text(args.task), sys.stdin.read()
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Evaluate agent output against the 5-axis rubric"
-    )
-    parser.add_argument("--task", help="Task description (file path or inline text)")
-    parser.add_argument("--output", help="Agent output to evaluate (file path)")
-    parser.add_argument("--interactive", action="store_true", help="Prompt for task and read output from stdin")
-    args = parser.parse_args()
-
-    task, output = _read_input(args)
-    if not output:
-        print("Error: no output to evaluate", file=sys.stderr)
-        sys.exit(1)
-
-    scores = evaluate(task, output)
-    print(format_report(scores))
-
-
-if __name__ == "__main__":
-    main()

package/prisma/skills/agent-self-evaluation/templates/evaluation-report.md

@@ -1,86 +0,0 @@
-# Agent Self-Evaluation Report Template
-
-Copy this template and fill in after completing a task. The format matches `scripts/evaluate.py` output.
-
-```
-============================================================
-AGENT SELF-EVALUATION REPORT
-============================================================
-Summary: Overall score X.X/5 across 5 quality axes.
-
-  Accuracy         █████ 5/5    or    ███░░ 3/5
-    + [Evidence: passing tests, verified claims]
-    - [Gaps: unverified claims, hedging language]
-    → [Improvement if score < 5]
-
-  Completeness      █████ 5/5
-    + [What's covered: all requirements + edge cases]
-    - [What's missing: explicitly acknowledge gaps]
-    → [Improvement if score < 5]
-
-  Clarity           █████ 5/5
-    + [Structure: headings, code blocks, bullet points]
-    - [Issues: undefined terms, wall of text, no summary]
-    → [Improvement if score < 5]
-
-  Actionability     █████ 5/5
-    + [User can: merge PR, run command, review file]
-    - [Blockers: missing steps, vague suggestions]
-    → [Improvement if score < 5]
-
-  Conciseness       █████ 5/5
-    + [Tight: no repetition, high information density]
-    - [Bloat: filler, meta-commentary, repeated points]
-    → [Improvement if score < 5]
-
-  OVERALL           X.X/5
-
-CRITICAL ISSUES (axes ≤ 2):
-  [Axis] Score N/5 — specific fix needed
-  (or "None" if no axis ≤ 2)
-
-Self-check: Would the user agree with this assessment? [Yes/No + brief justification]
-
-TOP IMPROVEMENTS:
-  1. [Highest impact fix]
-  2. [Second highest]
-  (Only list axes scoring < 4, ranked by user impact)
-
-VERDICT: [Deliver as-is / Fix N issues then deliver / Redo from scratch]
-```
-
-## Quick Reference: Scoring Triggers
-
-| If you see this... | Accuracy | Completeness | Clarity | Actionability | Conciseness |
-|---|---|---|---|---|---|
-| "should work" / "probably fine" | ≤4 | — | — | — | — |
-| "I think" / "I believe" | ≤4 | — | — | — | — |
-| No test output cited | ≤4 | — | — | — | — |
-| "TODO" / "FIXME" left behind | ≤3 | ≤3 | — | ≤3 | — |
-| Missing error handling | — | ≤3 | — | — | — |
-| Only happy path covered | — | ≤3 | — | — | — |
-| Wall-of-text paragraph (>200 words) | — | — | ≤3 | — | — |
-| No headings or structure | — | — | ≤3 | — | — |
-| "You should..." without specifics | — | — | — | ≤3 | — |
-| No PR or file created | — | — | — | ≤3 | — |
-| User needs to figure out next step | — | — | — | ≤2 | — |
-| Repeated points (3+ times) | — | — | — | — | ≤3 |
-| "Let me explain..." / "To summarize..." x3+ | — | — | — | — | ≤3 |
-| Output >15x longer than task | — | — | — | — | ≤3 |
-
-## When to Skip
-
-Skip the evaluation if:
-- Task was a single tool call (e.g., "read this file" — nothing to evaluate)
-- User explicitly says "don't evaluate" or "just do it"
-- Task is purely conversational (greeting, small talk)
-- You're mid-workflow and the user will judge the final output, not intermediate steps
-
-## Post-Evaluation Actions
-
-| Overall Score | What to do |
-|---|---|
-| ≥4.5 | Deliver as-is. No changes needed. |
-| 3.5–4.4 | Flag top improvement but deliver. Fix if <30 seconds. |
-| 2.5–3.4 | State what you'd change. Ask user: "Should I redo [axis] or deliver as-is?" |
-| <2.5 | Don't deliver. Say: "This scored [score] because [evidence]. Let me redo this with [specific fix]." Then redo. |