@evo-dev/evodev 0.0.1-alpha → 0.0.1-alpha.10

package/dist/plugins/evodev/skills/knowledge-distillation/references/knowledge-distillation-methods.md

@@ -0,0 +1,126 @@
+# Knowledge Distillation Method Notes
+
+This reference explains why the skill uses an evidence curation pipeline instead of a generic summarization workflow.
+
+## Method Comparison
+
+| Method | What it contributes | EvoDev adaptation |
+|---|---|---|
+| Postmortem / SRE review | Blameless failure analysis, root cause, impact, action items | Use for failure-derived `warning`, `anti-pattern`, `checklist`, and `workflow-improvement` candidates |
+| After Action Review | Expected vs actual, what went well, what failed, what changes next time | Use for evos cases and team/process improvement candidates |
+| Lessons learned systems | Reviewed lessons from projects/programs, not raw incident notes | Keep active OKF knowledge separate from raw evidence and transient plans |
+| ADR / decision logs | One durable decision with context, alternatives, rationale, consequences | Use `decision` candidates for architecture/workflow choices |
+| SKOS-style knowledge organization | Concepts, labels, narrower/broader/related terms, scope notes | Keep role tags and domain tags controlled enough for retrieval |
+| Agent memory research | Semantic facts, episodic cases, procedural improvements | Split knowledge, evos cases, and repo asset suggestions instead of one memory bucket |
+| Agent skills | Small triggerable procedures with references/assets for larger material | Keep `SKILL.md` operational and place long examples/taxonomies in references |
+| Evaluation-driven development | Behavior changes need regression cases and objective assertions | Pair skill, role, team, workflow, routing, tool-use, and subagent proposals with evo eval sets |
+
+## Evidence Quality
+
+Prefer evidence in this order:
+
+1. Verified local outcomes: tests, typecheck, lint, build, release dry-run, smoke checks, or accepted fixes.
+2. Reviewed findings: code review, security review, user correction, or postmortem conclusion.
+3. Official or upstream references: standards, vendor docs, framework docs, release notes, or source repositories.
+4. Existing repository contracts: `AGENTS.md`, design docs, task contracts, tests, schemas, and explicit policy files.
+5. Model reflection: useful for candidate generation only; never enough for active OKF knowledge.
+
+## Candidate Types
+
+- `rule`: normative guidance with a clear trigger and expected behavior.
+- `decision`: accepted choice plus context, alternatives, and consequences.
+- `pattern`: repeatable implementation or workflow shape.
+- `anti-pattern`: repeatable bad shape to avoid.
+- `warning`: risk that should influence planning or review.
+- `checklist`: short verification list for a role or workflow.
+- `concept`: term, domain note, taxonomy entry, or glossary item.
+- `workflow-improvement`: process change with trigger and expected future behavior.
+- `task-split-improvement`: improved decomposition, dependencies, owner roles, evidence, or stop conditions.
+- `tool-use-improvement`: improved tool selection, risk classification, command gating, or verification behavior.
+- `skill-improvement`: proposed change to a skill trigger, input requirements, output schema, references, or behavior.
+- `repo-asset-suggestion`: proposed project rule, skill, agent, subagent, or config file.
+- `role-agent-suggestion`: proposed user/project role definition or role capability.
+- `team-suggestion`: proposed EvoHub team composition and trigger.
+- `eval-set`: proposed regression cases for a skill, role, team, workflow, routing, tool-use, or subagent behavior.
+- `open-question`: unresolved issue that should not become accepted memory yet.
+
+## Scoring Rubric
+
+Use 1-5 integers. Keep the score approximate; it is a review aid, not a scientific metric.
+
+| Score | High value means | Low value means |
+|---|---|---|
+| `evidenceStrength` | Directly supported by verified/reviewed evidence | Mostly inferred or speculative |
+| `reuseValue` | Likely useful across future similar tasks | One-off or too narrow |
+| `actionability` | Clear trigger and next action | Vague or only descriptive |
+| `stability` | Likely to remain true for the repo/workflow | Depends on volatile context |
+| `privacyRisk` | Higher is worse: needs sensitive context | Redacted summary is enough |
+| `duplicationRisk` | Higher is worse: likely duplicate | Distinct from existing entries |
+
+Suggested write queue rule:
+
+- Propose active OKF knowledge only when `evidenceStrength >= 4`, `reuseValue >= 3`, `actionability >= 3`, `privacyRisk <= 2`, and `duplicationRisk <= 3`.
+- Propose evos cases when the item is mainly about a trigger/intervention/outcome history.
+- Propose repo assets only when repeated future execution would benefit from a rule, skill, role, or team and explicit project opt-in can be obtained.
+- Propose evo eval sets whenever a skill, role-agent, team, workflow, routing, tool-use, or subagent behavior change is proposed. If no eval is useful, state why in the proposal.
+
+## Executable Contract Notes
+
+The runtime accepts only JSON outputs with `kind: "knowledge-distillation-output"` and `schemaVersion: 1`. Candidate decisions are `auto-accept`, `create`, `update`, `needs-human`, `skip`, and canonical `no_write`. Active writes must be metadata-only, evidence-backed, review-state gated, verified, and targeted at safe relative `.md` OKF paths. Behavior-changing active writes must include `evalSetRefs` that point to provided `evoEvalSets`.
+
+Contract validation runs before OKF writes. Validation failures are saved as redacted, non-resumable failed-plan artifacts; organizer failures preserve a resumable plan. Failed artifacts are organizer inputs only and must not influence runtime retrieval.
+
+## Evolution Event Analysis
+
+Use reviewed event evidence to explain why the future behavior should change.
+
+| Signal | Good analysis asks |
+|---|---|
+| Task slices | Was the work decomposed into verifiable units with owner roles, dependencies, and stop conditions? |
+| Tool calls | Was the selected tool appropriate, gated by risk, and followed by verification or recovery? |
+| Skill invocations | Did the skill trigger correctly, receive the needed evidence, and produce a complete schema with privacy checks? |
+| Subagent lifecycle | Was the role necessary, bounded, mergeable, and verified without storing transcript content? |
+| Verification | Did required checks run at the right time and produce evidence for the claim? |
+
+Common skill defects:
+
+- `trigger-mismatch`: the skill was invoked when it should not have been, or missed when it should have run.
+- `input-gap`: required evidence class was missing.
+- `output-omission`: a required proposal, field, path, role tag, privacy check, or eval set was absent.
+- `inaccurate-claim`: output contradicted verified evidence.
+- `schema-drift`: output did not match the declared contract.
+- `privacy-defect`: output depended on raw private content.
+
+## Evo Eval Set Types
+
+| Type | Checks |
+|---|---|
+| `schema` | Required fields, enum values, ids, and status are present. |
+| `omission-detection` | Missing recommendations, evidence refs, privacy checks, or eval sets are caught. |
+| `privacy` | Raw prompt/source/command output/secrets are not stored. |
+| `routing` | A skill, workflow, role, or team should or should not trigger. |
+| `role-attention` | Role-tagged knowledge is selected only when scope, consent, and role policy match. |
+| `tool-use` | Tool choice, risk classification, and verification behavior are correct. |
+| `subagent-routing` | Role selection, scope, output contract, and merge requirements are correct. |
+
+## Storage Mapping
+
+| Candidate target | Canonical layer | Notes |
+|---|---|---|
+| `okf` | Active knowledge | OKF Markdown concepts with frontmatter, provenance, links, directory indexes, and logs |
+| `evo-eval-set` | Evolution eval sets | Regression cases tied to proposed or accepted behavior changes |
+| `repo-asset-proposal` | Proposal queue first | Never write project files by default |
+| `none` | Report only | Useful but not durable |
+
+Indexes, vector search, graph exports, and runtime caches are derived views. They must be rebuildable from active OKF concepts and must not store raw private content. Transient knowledge plans are organizer inputs, not accepted knowledge.
+
+## Source Notes
+
+- Google SRE postmortem practice: https://sre.google/sre-book/postmortem-culture/
+- US Army AAR guidance: https://www.first.army.mil/Portals/102/FM%207-0%20Appendix%20K.pdf
+- NASA Lessons Learned: https://www.nasa.gov/nasa-lessons-learned/
+- ADR references and templates: https://adr.github.io/
+- W3C SKOS Reference: https://www.w3.org/TR/skos-reference/
+- W3C PROV-O provenance ontology: https://www.w3.org/TR/prov-o/
+- OpenAI Agents SDK sessions/memory reference: https://openai.github.io/openai-agents-python/sessions/
+- Claude Code skills documentation: https://docs.anthropic.com/en/docs/claude-code/skills