@slowdini/slow-powers-opencode 0.3.0 → 0.4.0

package/skills/evaluating-skills/schema/stray-writes.schema.json

@@ -1,80 +0,0 @@
-{
-  "$schema": "http://json-schema.org/draft-07/schema#",
-  "$id": "https://slow-powers.dev/schemas/stray-writes.schema.json",
-  "title": "Stray-Write Report",
-  "description": "Output of evals:detect-stray-writes. Flags subagent file writes / mutating commands that landed outside a run's outputs dir. Lives at <workspace>/iteration-N/stray-writes.json.",
-  "type": "object",
-  "required": ["generated", "iteration", "totals", "runs"],
-  "additionalProperties": false,
-  "properties": {
-    "generated": { "type": "string", "description": "ISO timestamp" },
-    "iteration": { "type": "integer" },
-    "totals": {
-      "type": "object",
-      "required": ["violations", "warnings", "live_source_reads"],
-      "additionalProperties": false,
-      "properties": {
-        "violations": { "type": "integer" },
-        "warnings": { "type": "integer" },
-        "live_source_reads": { "type": "integer" }
-      }
-    },
-    "runs": {
-      "type": "array",
-      "description": "One entry per (eval, condition) run that had at least one finding.",
-      "items": {
-        "type": "object",
-        "required": [
-          "eval_id",
-          "condition",
-          "violations",
-          "warnings",
-          "live_source_reads"
-        ],
-        "additionalProperties": false,
-        "properties": {
-          "eval_id": { "type": "string" },
-          "condition": { "type": "string" },
-          "violations": {
-            "type": "array",
-            "description": "High-confidence: a write tool targeted a path outside the run's outputs dir.",
-            "items": { "$ref": "#/definitions/finding" }
-          },
-          "warnings": {
-            "type": "array",
-            "description": "Heuristic: a Bash command matched a mutating pattern (install, git, sed -i, redirection) without referencing the outputs dir.",
-            "items": { "$ref": "#/definitions/finding" }
-          },
-          "live_source_reads": {
-            "type": "array",
-            "description": "A read tool or Bash command accessed the live skill-under-test directory instead of the staged copy — the arm may be contaminated (staged-slug resolution race).",
-            "items": { "$ref": "#/definitions/finding" }
-          }
-        }
-      }
-    }
-  },
-  "definitions": {
-    "finding": {
-      "type": "object",
-      "required": ["tool", "ordinal", "reason"],
-      "additionalProperties": false,
-      "properties": {
-        "tool": { "type": "string" },
-        "path": {
-          "type": "string",
-          "description": "Target path for write-tool violations."
-        },
-        "command": {
-          "type": "string",
-          "description": "Command text for Bash warnings."
-        },
-        "ordinal": {
-          "type": "integer",
-          "description": "Position of the invocation in the run's tool_invocations."
-        },
-        "reason": { "type": "string" }
-      }
-    }
-  }
-}

package/skills/evaluating-skills/templates/eval-task-prompt.md

@@ -1,69 +0,0 @@
-# Eval task dispatch template
-
-Use this template when dispatching a fresh general-purpose subagent to execute a single eval test case.
-
-**The subagent MUST start with clean context.** State from previous runs invalidates the comparison.
-
-## Variables to fill
-
-| Variable | Source |
-|---|---|
-| `{{eval_id}}` | The eval's `id` from `evals.json` |
-| `{{condition}}` | `with_skill`, `without_skill`, `old_skill`, or `new_skill` |
-| `{{prompt}}` | The eval's `prompt`, verbatim |
-| `{{files}}` | Fixture paths the subagent can read (or "none") |
-| `{{output_dir}}` | The workspace directory the subagent writes to |
-| `{{skill_path}}` | Path to SKILL.md to load — omit entirely for `without_skill` |
-| `{{staged_skill_slug}}` | Unique slug the runner staged the skill-under-test under, if the harness supports project-local skill discovery (e.g. Claude Code) |
-| `{{bootstrap_content}}` | Plugin bootstrap / session-start text, injected to mirror what a real user sees when their session starts (optional; runners that don't have an equivalent leave this empty) |
-
-## Template
-
-```
-{{#if bootstrap_content}}
-<session-start-context>
-The following guidelines were loaded at session start by the plugin under evaluation
-(equivalent to the harness's session-start hook firing in a real user's environment):
-
-{{bootstrap_content}}
-</session-start-context>
-{{/if}}
-You are executing a single test case for a skill evaluation framework.
-Treat this as a real user request — do NOT optimize your behavior for the eval.
-
-{{#if staged_skill_slug}}
-The `{{skill_name}}` skill is registered under the identifier
-"{{staged_skill_slug}}" and is discoverable via the Skill tool. If you invoke it,
-use that identifier.
-{{else if skill_path}}
-The following skill is loaded into your operating guidelines. Apply it where relevant.
-<skill name="{{skill_name}}">
-{{skill_content}}
-</skill>
-{{else if bootstrap_content}}
-{{else}}
-No skill is loaded. Respond as you naturally would.
-{{/if}}
-
-Available fixture files: {{files}}
-Output directory: {{output_dir}}
-
-Instructions:
-- Write any files you produce into the output directory.
-- After completing the task, write your final user-facing response to {{output_dir}}/final-message.md.
-- Do not write anything outside the output directory.
-
-User request:
-{{prompt}}
-```
-
-`{{staged_skill_slug}}` and `{{bootstrap_content}}` are optional — they describe a *realistic-environment* dispatch where the runner has reproduced what a fresh plugin install would look like (siblings staged, bootstrap text prepended). A simpler runner can leave them empty and the conditional blocks degrade gracefully to the legacy inline / no-skill paths.
-
-## After the subagent completes
-
-Two records must exist per run: `{{output_dir}}/../run.json` (matching `schema/run-record.schema.json`) and `{{output_dir}}/../timing.json`.
-
-- **Harnesses with persisted transcripts (Claude Code):** `record-runs` assembles both from disk after all dispatches — carry-over fields from `dispatch.json`, `final_message` from `{{output_dir}}/final-message.md`, `tool_invocations`/tokens/duration from the transcript. The operator captures nothing per-task. Optionally, completion-event timing written to `timing.json` at dispatch time (with `"source": "completion-event"`) takes precedence — `record-runs` only backfills, never overwrites.
-- **Transcript-less harnesses:** the operator (or the runner) captures manually, as before:
-  1. The full transcript / tool invocations → convert via the harness adapter into `{{output_dir}}/../run.json`.
-  2. `total_tokens` and `duration_ms` from the harness's task completion event → `{{output_dir}}/../timing.json`. **These values may not be persisted anywhere else — save them immediately.**

package/skills/evaluating-skills/templates/evals.json.example

@@ -1,17 +0,0 @@
-{
-  "skill_name": "example-skill",
-  "evals": [
-    {
-      "id": "realistic-prompt",
-      "prompt": "A realistic user message — the kind of thing a real user would actually type. Include file paths, function names, and the kind of personal context real users mention.",
-      "expected_output": "Human-readable description of what a successful response looks like. Don't over-specify — leave room for valid variation.",
-      "files": ["fixtures/example.txt"]
-    },
-    {
-      "id": "edge-case-prompt",
-      "prompt": "A boundary condition, malformed input, or ambiguous instruction that tests where the skill's rules apply.",
-      "expected_output": "What the skill should produce given the awkward input."
-    }
-  ],
-  "_notes": "After iteration 1, add an `assertions` array to each eval. Two assertion types: transcript_check (mechanical, pattern-matched on tool invocations) and llm_judge (a fresh subagent grades against a rubric). Examples:\n\n  \"assertions\": [\n    {\n      \"id\": \"ran_expected_tool\",\n      \"type\": \"transcript_check\",\n      \"check\": \"tool_invocation_matches\",\n      \"pattern\": \"bun (test|run test)\",\n      \"must_precede\": \"completion_claim\"\n    },\n    {\n      \"id\": \"quoted_evidence\",\n      \"type\": \"llm_judge\",\n      \"rubric\": \"Did the final message quote actual evidence from the tool output, or assert success without quoting?\"\n    }\n  ]\n"
-}

package/skills/evaluating-skills/templates/judge-prompt.md

@@ -1,56 +0,0 @@
-# Judge prompt template
-
-Use this template when dispatching a fresh general-purpose subagent to grade `llm_judge` assertions for one run.
-
-**The judge subagent MUST start with clean context.** Bias from prior runs corrupts grading.
-
-## Variables to fill
-
-| Variable | Source |
-|---|---|
-| `{{run_record}}` | Contents of `run.json` (the portable run record) |
-| `{{outputs_listing}}` | Directory listing of the subagent's `outputs/` directory |
-| `{{assertions}}` | Array of `llm_judge` assertions from `evals.json` for this eval |
-
-## Template
-
-```
-You are grading a skill evaluation run. Be strict but fair.
-
-# Run record
-{{run_record}}
-
-# Outputs directory contents
-{{outputs_listing}}
-
-# Assertions to grade
-{{assertions}}
-
-# Instructions
-
-For each assertion, produce a result object with these fields:
-- `id`: the assertion's id (verbatim from the input)
-- `passed`: true or false
-- `evidence`: a direct quote or specific reference from the run record or outputs that justifies the verdict. Vague summaries are not evidence.
-- `confidence`: 0.0 to 1.0 — how confident you are in this verdict. Low confidence flags the result for human review.
-
-# Grading principles
-
-- PASS requires concrete evidence. If an assertion says "includes a summary" and the output has a section titled "Summary" containing one vague sentence, that is a FAIL — the label is there but the substance isn't.
-- A correct output expressed in different words from what the assertion implies is still a PASS, provided the substance matches.
-- If an assertion is unverifiable from the material you have (e.g. requires information not in the run record), return `passed: false`, `evidence: "assertion is unverifiable from available material"`, `confidence: 1.0`. The operator will fix the assertion.
-- Do not infer behavior not present in the record. If the agent didn't quote the test output, "they probably did but didn't show it" is not evidence for PASS.
-
-# Output format
-
-Emit a single JSON object matching `schema/grading.schema.json`:
-
-```json
-{
-  "assertion_results": [ ... ],
-  "summary": { "passed": N, "failed": N, "total": N, "pass_rate": N }
-}
-```
-
-Do not include any text outside the JSON object.
-```

package/skills/evaluating-skills/templates/revise-skill-prompt.md

@@ -1,56 +0,0 @@
-# Skill revision prompt template
-
-Use this template at the end of iteration N to feed eval signals into the next revision of SKILL.md.
-
-## Variables to fill
-
-| Variable | Source |
-|---|---|
-| `{{current_skill}}` | Current SKILL.md contents |
-| `{{failed_assertions}}` | List of `(eval_id, assertion_id, evidence)` for assertions that failed in the `with_skill` / `new_skill` condition |
-| `{{reviewer_feedback}}` | Per-eval notes from `feedback.json` (only the non-empty ones) |
-| `{{notable_transcripts}}` | Brief excerpts from the most informative run records (focus on transcripts that revealed *why* an assertion failed) |
-| `{{benchmark_summary}}` | Pass-rate delta and any anomalies (high stddev, time/token outliers) from `benchmark.json` |
-
-## Template
-
-```
-You are improving a skill based on signals from a recent eval iteration.
-
-# Current SKILL.md
-{{current_skill}}
-
-# Failed assertions
-{{failed_assertions}}
-
-# Reviewer feedback
-{{reviewer_feedback}}
-
-# Notable execution transcripts
-{{notable_transcripts}}
-
-# Benchmark summary
-{{benchmark_summary}}
-
-# Your task
-
-Propose changes to the skill. Guidelines:
-
-1. **Generalize from feedback.** The skill is used across many prompts, not just these test cases. Fixes should address underlying issues broadly, not patch specific failing examples.
-
-2. **Keep the skill lean.** Fewer, better instructions outperform exhaustive rules. If transcripts show wasted work — unnecessary validation, unneeded intermediate outputs — remove those instructions. If pass rates plateau despite adding rules, try removing instructions and see if results hold or improve.
-
-3. **Explain the why.** Reasoning-based instructions ("Do X because Y tends to cause Z") work better than rigid directives ("ALWAYS do X, NEVER do Y"). Models follow instructions more reliably when they understand the purpose.
-
-4. **Bundle repeated work.** If multiple runs independently wrote a similar helper script (chart builder, data parser, lookup table), bundle it into the skill's `scripts/` directory and reference it from the skill.
-
-5. **Do not just patch failing examples.** A change that fixes only the failing assertions is a regression risk if it doesn't address the underlying gap. Ask: "what is the smallest, most general rule that would have made these failures impossible?"
-
-# Output
-
-Either:
-- A unified diff of proposed SKILL.md changes, OR
-- A revised SKILL.md in full
-
-Plus a short rationale (≤ 200 words) explaining the structural choices and which signals each change addresses.
-```