@hassangameryt/freeflow 0.6.0 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +3 -2
- package/.codex-plugin/plugin.json +5 -6
- package/.cursor-plugin/plugin.json +25 -0
- package/README.md +66 -56
- package/capabilities/cognitive-routing/SKILL.md +288 -130
- package/com.github.copilot/hooks/hooks.json +11 -0
- package/command-surface.json +0 -24
- package/gemini-extension.json +5 -0
- package/hooks/adapters/claude-session-start.mjs +16 -0
- package/hooks/adapters/codex-session-start.mjs +16 -0
- package/hooks/adapters/copilot-session-start.mjs +13 -0
- package/hooks/adapters/cursor-session-start.mjs +13 -0
- package/hooks/adapters/gemini-session-start.mjs +15 -0
- package/hooks/claude/hooks.json +18 -0
- package/hooks/codex/hooks.json +18 -0
- package/hooks/cursor/hooks.json +9 -0
- package/hooks/freeflow-runtime-context.mjs +3 -650
- package/hooks/hooks.json +4 -30
- package/hooks/shared/runtime-context.mjs +210 -0
- package/hooks/tests/check-host-adapters.test.mjs +82 -0
- package/hooks/tests/check-runtime-context-hook.sh +140 -320
- package/package.json +20 -4
- package/pi-extension/dist/cognitive-routing/commands.js +3 -3
- package/pi-extension/dist/cognitive-routing/config.js +16 -20
- package/pi-extension/dist/cognitive-routing/controller.js +478 -48
- package/pi-extension/dist/cognitive-routing/history.js +12 -0
- package/pi-extension/dist/cognitive-routing/host.js +74 -12
- package/pi-extension/dist/cognitive-routing/pi-routing-session.js +414 -0
- package/pi-extension/dist/cognitive-routing/pi-session-control.js +320 -0
- package/pi-extension/dist/cognitive-routing/tool.js +1 -0
- package/pi-extension/dist/cognitive-routing/types.js +1 -1
- package/pi-extension/dist/index.js +236 -219
- package/pi-extension/dist/runtime/runtime-context.js +126 -286
- package/pi-extension/dist/settings/settings-ui.js +52 -336
- package/pi-extension/src/cognitive-routing/commands.ts +15 -5
- package/pi-extension/src/cognitive-routing/config.ts +16 -20
- package/pi-extension/src/cognitive-routing/controller.ts +561 -51
- package/pi-extension/src/cognitive-routing/history.ts +12 -0
- package/pi-extension/src/cognitive-routing/host.ts +110 -16
- package/pi-extension/src/cognitive-routing/pi-routing-session.ts +538 -0
- package/pi-extension/src/cognitive-routing/pi-session-control.ts +444 -0
- package/pi-extension/src/cognitive-routing/tool.ts +1 -0
- package/pi-extension/src/cognitive-routing/types.ts +1 -1
- package/pi-extension/src/index.ts +249 -239
- package/pi-extension/src/runtime/runtime-context.ts +129 -305
- package/pi-extension/src/settings/settings-ui.ts +53 -371
- package/pi-extension/tests/cognitive-routing/commands.test.js +1 -1
- package/pi-extension/tests/cognitive-routing/controller.test.js +69 -40
- package/pi-extension/tests/cognitive-routing/extension-lifecycle.test.js +128 -57
- package/pi-extension/tests/cognitive-routing/pi-normal-lifecycle.test.js +434 -0
- package/pi-extension/tests/cognitive-routing/pi-session-control.test.js +174 -0
- package/pi-extension/tests/cognitive-routing/pi-session-history.test.js +91 -0
- package/pi-extension/tests/cognitive-routing/profile-resolution.test.js +47 -4
- package/pi-extension/tests/integration/mode-free-core.test.js +218 -0
- package/pi-extension/tests/integration/pi-extension.test.js +430 -2478
- package/pi-extension/tests/integration/prompt-architecture.test.js +124 -79
- package/pi-extension/tests/integration/unaffected-regressions.test.js +411 -0
- package/plugin.json +33 -0
- package/runtime/prompts/cognitive-routing.md +5 -4
- package/runtime/prompts/conversation-history.md +1 -1
- package/runtime/prompts/core.md +77 -6
- package/runtime/prompts/interaction-contract.md +6 -2
- package/skills/action-selection/SKILL.md +93 -44
- package/skills/action-selection/references/trajectory-stalls.md +71 -0
- package/skills/bypass/SKILL.md +109 -39
- package/skills/commit-work/SKILL.md +94 -55
- package/skills/commit-work/references/git-state-edges.md +94 -0
- package/skills/commit-work/references/simple-push.md +92 -0
- package/skills/commit-work/references/staging-decisions.md +79 -46
- package/skills/decision-gate/SKILL.md +80 -22
- package/skills/design-for-depth/SKILL.md +208 -43
- package/skills/design-for-depth/references/design-pressure-signals.md +108 -240
- package/skills/design-for-depth/references/interface-design-loop.md +77 -97
- package/skills/design-for-depth/references/module-and-dependency-design.md +147 -0
- package/skills/design-for-depth/references/software-design-philosophy.md +127 -173
- package/skills/design-for-depth/references/state-and-failure-boundaries.md +136 -0
- package/skills/diagnose-failure/SKILL.md +198 -52
- package/skills/diagnose-failure/references/diagnostic-loop-catalog.md +19 -5
- package/skills/diagnose-failure/references/flaky-and-performance-diagnosis.md +13 -5
- package/skills/discuss/SKILL.md +118 -76
- package/skills/discuss/references/discussion-continuity.md +84 -32
- package/skills/evaluate-skill/SKILL.md +166 -52
- package/skills/evaluate-skill/references/evaluation-definition-schema.md +2 -2
- package/skills/evaluate-skill/references/evaluation-design.md +56 -30
- package/skills/evaluate-skill/references/execution-and-evidence.md +1 -1
- package/skills/evaluate-skill/references/review-and-revision.md +62 -21
- package/skills/execute-work/SKILL.md +98 -59
- package/skills/execute-work/references/code-practices.md +4 -4
- package/skills/execute-work/references/execute-work-edges.md +42 -22
- package/skills/review-artifact/SKILL.md +132 -71
- package/skills/review-artifact/references/adjudicate-artifact-review.md +101 -0
- package/skills/review-artifact/references/independent-artifact-reviewer-contract.md +52 -41
- package/skills/review-work/SKILL.md +115 -72
- package/skills/review-work/references/adjudicate-work-review.md +97 -0
- package/skills/review-work/references/independent-work-reviewer-contract.md +40 -35
- package/skills/review-work/references/security-risk-lens.md +17 -15
- package/skills/setup-freeflow/SKILL.md +14 -17
- package/skills/setup-freeflow/references/activation-contract.md +24 -30
- package/skills/setup-freeflow/references/host-setup.md +6 -7
- package/skills/simplify-code/SKILL.md +97 -39
- package/skills/simplify-code/references/simplification-patterns.md +1 -1
- package/skills/tdd/SKILL.md +108 -41
- package/skills/tdd/references/test-design.md +1 -1
- package/skills/track-work/SKILL.md +133 -300
- package/skills/track-work/references/working-record-format.md +515 -0
- package/skills/track-work/scripts/lib/document.mjs +713 -0
- package/skills/track-work/scripts/lib/format.mjs +166 -0
- package/skills/track-work/scripts/lib/store.mjs +233 -0
- package/skills/track-work/scripts/lib/transitions.mjs +790 -0
- package/skills/track-work/scripts/lib/views.mjs +27 -430
- package/skills/track-work/scripts/working-record.mjs +292 -98
- package/skills/verify-work/SKILL.md +46 -20
- package/skills/verify-work/references/browser-runtime-evidence.md +1 -1
- package/skills/verify-work/references/integration-evidence.md +1 -7
- package/skills/verify-work/references/performance-evidence.md +1 -1
- package/skills/workflow/SKILL.md +110 -80
- package/skills/workflow/references/domain-skill-composition.md +24 -18
- package/skills/write-skill/SKILL.md +218 -62
- package/skills/write-skill/references/activation-boundaries.md +63 -21
- package/skills/write-skill/references/agent-first-instructions.md +61 -27
- package/skills/write-skill/references/progressive-disclosure.md +91 -41
- package/skills/write-skill/references/skill-dependency-graphs.md +190 -0
- package/runtime/prompts/skills.md +0 -65
- package/skills/mode-contract/SKILL.md +0 -104
- package/skills/track-work/scripts/lib/codec.mjs +0 -570
- package/skills/track-work/scripts/lib/commands.mjs +0 -568
- package/skills/track-work/scripts/lib/contract.mjs +0 -303
- package/skills/track-work/scripts/lib/model.mjs +0 -506
- package/skills/track-work/scripts/lib/operations.mjs +0 -965
- package/skills/track-work/scripts/lib/record-store.mjs +0 -264
- package/skills/track-work/scripts/lib/result.mjs +0 -37
- package/skills/track-work/scripts/lib/rewrites.mjs +0 -740
- package/skills/track-work/scripts/lib/schema.mjs +0 -277
- package/skills/track-work/scripts/lib/source-inventory.mjs +0 -162
- package/skills/track-work/scripts/lib/workspace.mjs +0 -134
|
@@ -5,71 +5,126 @@ description: Use when evaluating or comparing agent-skill behavior.
|
|
|
5
5
|
|
|
6
6
|
# Evaluate Skill
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
Determine whether an exact skill package changes agent behavior as intended under declared pressure. Evidence outranks confident prose.
|
|
9
9
|
|
|
10
10
|
A **group** asks one behavioral question through exactly two variants:
|
|
11
11
|
|
|
12
|
-
- `baseline`: no target skill
|
|
13
|
-
- `candidate`:
|
|
12
|
+
- `baseline`: no target skill for a new skill, or one exact immutable previous package for a revision;
|
|
13
|
+
- `candidate`: one exact new or updated package.
|
|
14
14
|
|
|
15
|
-
The **subject** performs the declared task under one variant.
|
|
15
|
+
The **subject** performs the declared task under one variant. The **evaluator** isolates subjects and preserves canonical evidence. The **deterministic grader** derives fixed mechanical facts after run persistence. A **suite** runs an ordered set of independent groups.
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
Evaluate Skill does not author or revise skills, decide semantic quality, claim readiness, or choose publication. [Write Skill](../write-skill/SKILL.md) owns package preservation and revision. The active agent or user judges unresolved meaning. The user decides whether to revise, use, publish, or reject the skill.
|
|
18
18
|
|
|
19
|
-
|
|
19
|
+
Inline examples teach the subject and are part of the candidate intervention. They are not independent evaluation evidence. Keep deterministic criteria and review questions outside natural subject prompts.
|
|
20
20
|
|
|
21
|
-
|
|
22
|
-
- Reuse an existing group unchanged when it preserves the question, pressure, and fixed criteria.
|
|
23
|
-
- For a draft-only request, use [Write Skill](../write-skill/SKILL.md); do not fake a run.
|
|
24
|
-
- Preserve partial infrastructure evidence without treating unavailable behavior as pass or fail.
|
|
25
|
-
- Do not mutate canonical evidence or erase an unfavorable result.
|
|
21
|
+
## Route From Existing Evidence First
|
|
26
22
|
|
|
27
|
-
Before a
|
|
23
|
+
Before proposing a run or revision:
|
|
28
24
|
|
|
29
|
-
|
|
25
|
+
- inspect adequate saved evidence;
|
|
26
|
+
- reuse an existing group unchanged when it still preserves the question, pressure, variants, and fixed criteria;
|
|
27
|
+
- use saved views and raw artifact reads when a complete result already answers the question;
|
|
28
|
+
- preserve partial infrastructure evidence without converting unavailable behavior into pass or fail;
|
|
29
|
+
- do not mutate canonical evidence or erase an unfavorable result.
|
|
30
30
|
|
|
31
|
-
|
|
31
|
+
For a draft-only request, use Write Skill. Do not fabricate behavioral evidence.
|
|
32
32
|
|
|
33
|
-
|
|
34
|
-
|
|
33
|
+
A revision comparison requires an exact immutable baseline identity. If none exists, return to Write Skill to preserve the complete current package before revision. Evaluate Skill consumes baseline and candidate identities; it does not create a baseline by editing the package it is evaluating.
|
|
34
|
+
|
|
35
|
+
## Name One Behavioral Question
|
|
36
|
+
|
|
37
|
+
Name the exact claim and observing mechanism before choosing the group shape.
|
|
38
|
+
|
|
39
|
+
Evidence classes include:
|
|
40
|
+
|
|
41
|
+
- **Description activation:** did a natural prompt cause the exact target to be read, and when?
|
|
42
|
+
- **First-read body behavior:** did explicit body delivery guide the subject using only guaranteed context?
|
|
35
43
|
- **Nearby behavior:** did a close case avoid activation or load without hijacking the task?
|
|
36
44
|
- **Dependency composition:** did exact ordered skills, resources, and context work together?
|
|
37
|
-
- **Retained use:** did guidance remain useful on later declared turns?
|
|
38
|
-
- **Artifact outcome:** did files, state, events, and responses match
|
|
39
|
-
- **Cross-host behavior:** did the same question hold on every named host?
|
|
45
|
+
- **Retained use:** did guidance remain useful on later declared turns without redelivery?
|
|
46
|
+
- **Artifact outcome:** did files, structured state, events, and responses match fixed criteria?
|
|
47
|
+
- **Cross-host behavior:** did the same behavioral question hold on every named host?
|
|
48
|
+
|
|
49
|
+
One evidence class does not prove another.
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
successful read
|
|
53
|
+
≠ behavioral compliance
|
|
54
|
+
|
|
55
|
+
explicit body delivery
|
|
56
|
+
≠ natural activation
|
|
57
|
+
|
|
58
|
+
one turn
|
|
59
|
+
≠ retained use
|
|
60
|
+
|
|
61
|
+
ambient installation
|
|
62
|
+
≠ declared composition
|
|
63
|
+
|
|
64
|
+
correct behavior
|
|
65
|
+
≠ proof that the target caused it
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Read [Evaluation Design](references/evaluation-design.md) before choosing or materially changing the question, evidence class, group shape, pressure, or baseline/candidate boundary.
|
|
69
|
+
|
|
70
|
+
## Design The Smallest Valid Comparison
|
|
71
|
+
|
|
72
|
+
Before subject output exists:
|
|
73
|
+
|
|
74
|
+
1. fix one behavioral question;
|
|
75
|
+
2. fix deterministic expectations and review questions;
|
|
76
|
+
3. preserve the earliest natural prompt or strongest pressure capable of distinguishing behavior;
|
|
77
|
+
4. declare exact baseline and candidate identities;
|
|
78
|
+
5. keep prompt or turns, fixture, tools, model, thinking, other skills, context, runtime, and criteria fixed except for the target difference;
|
|
79
|
+
6. run one group or an ordered suite serially;
|
|
80
|
+
7. inspect canonical evidence and deterministic facts before judging meaning;
|
|
81
|
+
8. classify the failed boundary before proposing revision;
|
|
82
|
+
9. return one measured revision target to Write Skill;
|
|
83
|
+
10. rerun the complete fixed group with both variants after a behavioral change.
|
|
84
|
+
|
|
85
|
+
For a new skill:
|
|
86
|
+
|
|
87
|
+
```text
|
|
88
|
+
baseline: no target skill
|
|
89
|
+
candidate: exact candidate package
|
|
90
|
+
```
|
|
40
91
|
|
|
41
|
-
|
|
92
|
+
For a revision:
|
|
42
93
|
|
|
43
|
-
|
|
94
|
+
```text
|
|
95
|
+
baseline: exact previous package
|
|
96
|
+
candidate: exact revised package
|
|
97
|
+
```
|
|
44
98
|
|
|
45
|
-
|
|
99
|
+
For a description-only revision, keep body, references, scripts, and other resources byte-identical.
|
|
46
100
|
|
|
47
|
-
|
|
101
|
+
If both variants pass, the pressure may be weak or the baseline already sufficient. If both fail, distinguish skill, fixture, dependency, environment, host, isolation, and criterion failures before revising instructions.
|
|
48
102
|
|
|
49
|
-
|
|
50
|
-
- If explicit body delivery succeeds but a natural prompt never reads the skill, body behavior is supported while description activation still fails.
|
|
103
|
+
## Declare The Complete Environment
|
|
51
104
|
|
|
52
|
-
|
|
105
|
+
Use the exact [Evaluation Definition Schema](references/evaluation-definition-schema.md) when authoring or checking group and suite JSON.
|
|
53
106
|
|
|
54
|
-
|
|
55
|
-
2. Preserve the earliest natural prompt or strongest pressure that should distinguish behavior.
|
|
56
|
-
3. For a new skill, use no target versus candidate. For a revision, use exact previous versus updated snapshots. For a description-only revision, keep body and resources byte-identical.
|
|
57
|
-
4. Keep prompts or turns, fixture, tools, model, thinking, other skills, context, and criteria fixed except for the target difference.
|
|
58
|
-
5. Run one group or an ordered suite serially.
|
|
59
|
-
6. Inspect deterministic facts and canonical evidence before judging meaning.
|
|
60
|
-
7. Let the active agent or user review unresolved behavior; never launch an automatic semantic grader.
|
|
61
|
-
8. Classify the failed boundary before editing and revise one measured pressure point.
|
|
62
|
-
9. Rerun the complete fixed group with both variants after a measured change.
|
|
107
|
+
Declare:
|
|
63
108
|
|
|
64
|
-
|
|
109
|
+
- prompt or ordered turns;
|
|
110
|
+
- fixture;
|
|
111
|
+
- exact ordered skill packages and target index;
|
|
112
|
+
- working-tree or Git source identity;
|
|
113
|
+
- context resources;
|
|
114
|
+
- tools;
|
|
115
|
+
- model and thinking;
|
|
116
|
+
- runtime host and extension bundles when needed;
|
|
117
|
+
- literal and inherited environment sources;
|
|
118
|
+
- deterministic expectations and comparison IDs;
|
|
119
|
+
- review questions.
|
|
65
120
|
|
|
66
|
-
|
|
121
|
+
Snapshot every declared subject resource. Ambient installation is not declared composition.
|
|
67
122
|
|
|
68
|
-
|
|
123
|
+
Criteria and review questions remain outside the subject prompt.
|
|
69
124
|
|
|
70
|
-
|
|
125
|
+
## Run And Preserve Canonical Evidence
|
|
71
126
|
|
|
72
|
-
|
|
127
|
+
Before operating the evaluator, resolving paths, interpreting states, or reasoning about host execution, read [Execution and Evidence](references/execution-and-evidence.md).
|
|
73
128
|
|
|
74
129
|
Use the canonical [skill evaluator](scripts/skill-eval.mjs):
|
|
75
130
|
|
|
@@ -78,28 +133,87 @@ node <evaluate-skill-directory>/scripts/skill-eval.mjs run <suite-or-group-path>
|
|
|
78
133
|
node <evaluate-skill-directory>/scripts/skill-eval.mjs view <result-id-or-directory> [--group <id-or-position>] [--variant baseline|candidate]
|
|
79
134
|
```
|
|
80
135
|
|
|
81
|
-
Run from the definition root.
|
|
136
|
+
Run from the definition root.
|
|
137
|
+
|
|
138
|
+
With no selectors, run or view every suite group and both variants. Selectors narrow execution or viewing scope; they do not modify definitions.
|
|
139
|
+
|
|
140
|
+
Canonical results live under:
|
|
141
|
+
|
|
142
|
+
```text
|
|
143
|
+
<definition-root>/.skill-eval/runs/<result-id>
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
A view is a projection of saved evidence, not its canonical source. Use ordinary file tools when exact run, event, transcript, response, stderr, workspace, definition, grade, context-observation, or group artifacts matter.
|
|
147
|
+
|
|
148
|
+
Do not invoke archived evaluators.
|
|
149
|
+
|
|
150
|
+
## Derive Facts Without Inventing Meaning
|
|
82
151
|
|
|
83
|
-
|
|
152
|
+
Append deterministic grades only after canonical run evidence exists.
|
|
84
153
|
|
|
85
|
-
|
|
154
|
+
Deterministic checks may establish exact facts such as:
|
|
86
155
|
|
|
87
|
-
|
|
156
|
+
- skill or resource reads;
|
|
157
|
+
- file and path state;
|
|
158
|
+
- changed paths;
|
|
159
|
+
- file or response text;
|
|
160
|
+
- JSON validity or values;
|
|
161
|
+
- tool-call outcomes and argument predicates;
|
|
162
|
+
- context text at a selected surface, request, and turn;
|
|
163
|
+
- factual baseline-to-candidate transitions.
|
|
88
164
|
|
|
89
|
-
|
|
165
|
+
A failed check remains ordinary behavioral evidence. It does not turn a valid subject run into infrastructure failure.
|
|
90
166
|
|
|
91
|
-
|
|
167
|
+
Missing evidence remains unavailable.
|
|
92
168
|
|
|
93
|
-
|
|
169
|
+
Malformed expectations or grading failures produce separate `grade-error` evidence and do not rewrite canonical run evidence.
|
|
94
170
|
|
|
95
|
-
|
|
171
|
+
A transition such as `fail-to-pass` is a factual comparison. It does not mean bad-to-good, not-ready-to-ready, or rejected-to-promotable.
|
|
96
172
|
|
|
97
|
-
|
|
173
|
+
## Review Meaning After Evidence Exists
|
|
174
|
+
|
|
175
|
+
After canonical evidence and deterministic grades exist, read [Review and Revision](references/review-and-revision.md) when:
|
|
176
|
+
|
|
177
|
+
- evidence conflicts;
|
|
178
|
+
- semantic meaning remains unresolved;
|
|
179
|
+
- the failed boundary must be classified;
|
|
180
|
+
- a revision target must be selected;
|
|
181
|
+
- rerun scope is uncertain.
|
|
182
|
+
|
|
183
|
+
The evaluator does not launch an automatic semantic grader.
|
|
184
|
+
|
|
185
|
+
The active agent or user may judge reasoning quality, authority preservation, architectural fitness, recommendation quality, or semantic completeness. Keep that judgment, cited evidence, uncertainty, and limits outside canonical run and grade state.
|
|
186
|
+
|
|
187
|
+
## Return Revision To The Author
|
|
188
|
+
|
|
189
|
+
When evidence supports a candidate change:
|
|
190
|
+
|
|
191
|
+
1. identify the exact failed boundary;
|
|
192
|
+
2. preserve the canonical result path and relevant evidence;
|
|
193
|
+
3. state one measured revision target;
|
|
194
|
+
4. keep unrelated instructions and criteria stable;
|
|
195
|
+
5. return revision to Write Skill;
|
|
196
|
+
6. rerun the complete fixed group with both variants after the change.
|
|
197
|
+
|
|
198
|
+
Do not edit the skill from Evaluate Skill.
|
|
199
|
+
|
|
200
|
+
If the evaluation definition or criterion is wrong, preserve the old result and state that it no longer answers the revised question. Do not reinterpret it as evidence for the new question.
|
|
98
201
|
|
|
99
202
|
## Report And Stop
|
|
100
203
|
|
|
101
|
-
Report
|
|
204
|
+
Report:
|
|
205
|
+
|
|
206
|
+
- what ran;
|
|
207
|
+
- exact baseline and candidate identities;
|
|
208
|
+
- selected groups and variants;
|
|
209
|
+
- deterministic results;
|
|
210
|
+
- infrastructure failures, cancellation, or unavailable evidence;
|
|
211
|
+
- viewed scope;
|
|
212
|
+
- canonical result paths;
|
|
213
|
+
- what the evidence proves;
|
|
214
|
+
- what it does not prove;
|
|
215
|
+
- any supported failed boundary or revision target.
|
|
102
216
|
|
|
103
|
-
Stop rather than overclaim when required activation, first-read, composition,
|
|
217
|
+
Stop rather than overclaim when required activation, first-read, composition, retained-use, isolation, artifact, or host evidence is unavailable.
|
|
104
218
|
|
|
105
|
-
|
|
219
|
+
The active agent or user judges unresolved behavior. The user decides whether to revise, use, publish, or reject the skill.
|
|
@@ -58,7 +58,7 @@ Rules:
|
|
|
58
58
|
- `schema_version` is `1`.
|
|
59
59
|
- `kind` is `group`.
|
|
60
60
|
- `id` starts with a lowercase letter, contains lowercase letters, numbers, or single hyphens, and is at most 64 characters.
|
|
61
|
-
- `type` is `description
|
|
61
|
+
- `type` is `description` or `body`.
|
|
62
62
|
- `input` contains exactly one of:
|
|
63
63
|
- `prompt`: one non-empty string;
|
|
64
64
|
- `turns`: a non-empty ordered array of non-empty strings. Repeated natural turns are allowed.
|
|
@@ -252,7 +252,7 @@ Each environment has exactly:
|
|
|
252
252
|
"source": { "kind": "git", "ref": "v1.2.3" },
|
|
253
253
|
"skills": ["skills/target", "skills/support"],
|
|
254
254
|
"target": 0,
|
|
255
|
-
"context": ["
|
|
255
|
+
"context": ["context/base-guidance.md"]
|
|
256
256
|
}
|
|
257
257
|
```
|
|
258
258
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Evaluation Design
|
|
2
2
|
|
|
3
|
-
Read this
|
|
3
|
+
Read this before choosing or materially changing the behavioral question, evidence class, group shape, pressure, or baseline/candidate boundary.
|
|
4
4
|
|
|
5
5
|
## Keep Roles Distinct
|
|
6
6
|
|
|
@@ -8,10 +8,10 @@ Read this when choosing the behavioral question, evidence class, group shape, or
|
|
|
8
8
|
- **Evaluator:** isolates subjects and preserves canonical evidence.
|
|
9
9
|
- **Deterministic grader:** derives fixed mechanical facts after run persistence.
|
|
10
10
|
- **Reviewer:** the active agent or user judges unresolved meaning.
|
|
11
|
-
- **Author:**
|
|
11
|
+
- **Author:** uses Write Skill to revise one measured pressure point.
|
|
12
12
|
- **User:** decides whether to revise, use, publish, or reject the skill.
|
|
13
13
|
|
|
14
|
-
The evaluator does not launch an automatic semantic grader or own readiness, promotion, or production status.
|
|
14
|
+
The evaluator does not revise the skill, launch an automatic semantic grader, or own readiness, promotion, or production status.
|
|
15
15
|
|
|
16
16
|
## Map Claims To Evidence
|
|
17
17
|
|
|
@@ -23,64 +23,90 @@ The evaluator does not launch an automatic semantic grader or own readiness, pro
|
|
|
23
23
|
| Declared dependencies compose | Exact ordered composition |
|
|
24
24
|
| Guidance remains useful later | Multi-turn evidence |
|
|
25
25
|
| Files or structured state match | Artifact outcome |
|
|
26
|
-
| Behavior holds
|
|
26
|
+
| Behavior holds across named hosts | The same group semantics on every named host |
|
|
27
27
|
|
|
28
|
-
Record
|
|
28
|
+
Record separately:
|
|
29
|
+
|
|
30
|
+
- declaration;
|
|
31
|
+
- resource materialization;
|
|
32
|
+
- delivery;
|
|
33
|
+
- observed reads;
|
|
34
|
+
- behavior;
|
|
35
|
+
- artifacts;
|
|
36
|
+
- derived grades;
|
|
37
|
+
- later semantic judgment.
|
|
38
|
+
|
|
39
|
+
A read does not prove compliance. Correct behavior does not prove which skill caused it.
|
|
29
40
|
|
|
30
41
|
## Choose The Smallest Group Shape
|
|
31
42
|
|
|
32
43
|
- **Description prompt:** natural activation and exact read timing.
|
|
33
|
-
- **Explicit body task:** first-read
|
|
44
|
+
- **Explicit body task:** first-read body behavior without activation ambiguity.
|
|
34
45
|
- **Fixture task:** files or repository state matter.
|
|
35
|
-
- **Stateful turns:** ordered conversation and
|
|
36
|
-
- **Saved-result review:**
|
|
37
|
-
- **Suite:** independent questions
|
|
46
|
+
- **Stateful turns:** ordered conversation and retained use matter.
|
|
47
|
+
- **Saved-result review:** canonical evidence already answers the question.
|
|
48
|
+
- **Suite:** several independent questions require one ordered serial invocation.
|
|
49
|
+
|
|
50
|
+
Keep activation and body behavior separate unless their integrated interaction is the exact question and the current runner supports that observing boundary.
|
|
51
|
+
|
|
52
|
+
## Fix Exact Variants
|
|
53
|
+
|
|
54
|
+
Every group contains exactly:
|
|
38
55
|
|
|
39
|
-
|
|
56
|
+
- `baseline`;
|
|
57
|
+
- `candidate`.
|
|
40
58
|
|
|
41
|
-
|
|
59
|
+
For a new skill, baseline has no target.
|
|
42
60
|
|
|
43
|
-
|
|
61
|
+
For a revision, baseline names an exact immutable previous package. If no exact baseline exists, return to Write Skill before revision.
|
|
44
62
|
|
|
45
|
-
|
|
46
|
-
- `candidate`: the new or updated snapshot.
|
|
63
|
+
For a description-only revision, body and resources remain byte-identical.
|
|
47
64
|
|
|
48
|
-
|
|
65
|
+
Freeze prompts or turns, fixture, tools, model, thinking, other skills, context, runtime, and criteria across variants.
|
|
49
66
|
|
|
50
67
|
## Design Common Questions
|
|
51
68
|
|
|
52
|
-
### Description
|
|
69
|
+
### Description activation
|
|
53
70
|
|
|
54
|
-
Use the earliest natural prompt where the
|
|
71
|
+
Use the earliest natural prompt where the target should become useful.
|
|
72
|
+
|
|
73
|
+
For a nearby prompt, predeclare whether success means:
|
|
74
|
+
|
|
75
|
+
- non-trigger; or
|
|
76
|
+
- safe behavior after activation.
|
|
55
77
|
|
|
56
78
|
### First-read body
|
|
57
79
|
|
|
58
|
-
Explicitly deliver the exact body with only guaranteed context. Ambient package context cannot repair a
|
|
80
|
+
Explicitly deliver the exact target body with only guaranteed context. Ambient package context cannot repair a hidden dependency.
|
|
59
81
|
|
|
60
|
-
###
|
|
82
|
+
### Dependency composition
|
|
61
83
|
|
|
62
|
-
Materialize exact ordered skills and context. Vary only the target snapshot.
|
|
84
|
+
Materialize exact ordered skills and context. Vary only the target snapshot. Do not provide a hidden base stack.
|
|
63
85
|
|
|
64
86
|
### Retained use
|
|
65
87
|
|
|
66
|
-
Use
|
|
88
|
+
Use ordered turns in one persistent subject process. Repeated body delivery does not prove retained use.
|
|
67
89
|
|
|
68
90
|
### Artifact outcome
|
|
69
91
|
|
|
70
|
-
Grade
|
|
92
|
+
Grade files, changed paths, structured state, and events before relying on the final response. Preserve turn-scoped workspace evidence when later turns may change earlier state.
|
|
71
93
|
|
|
72
|
-
##
|
|
94
|
+
## Apply Real Pressure
|
|
73
95
|
|
|
74
|
-
|
|
96
|
+
A useful group:
|
|
75
97
|
|
|
76
|
-
|
|
98
|
+
- creates a natural temptation to violate the intended behavior;
|
|
99
|
+
- keeps criteria and review questions outside the prompt;
|
|
100
|
+
- exposes deterministic evidence where possible;
|
|
101
|
+
- distinguishes baseline from candidate behavior;
|
|
102
|
+
- remains realistic enough that passing behavior is useful.
|
|
77
103
|
|
|
78
|
-
|
|
104
|
+
If both variants pass, the pressure may be weak or the baseline sufficient.
|
|
79
105
|
|
|
80
|
-
|
|
106
|
+
If both fail, classify skill, fixture, dependency, environment, host, isolation, or criterion before editing.
|
|
81
107
|
|
|
82
|
-
##
|
|
108
|
+
## Reuse Existing Evidence
|
|
83
109
|
|
|
84
|
-
|
|
110
|
+
When saved canonical evidence already answers the fixed question, inspect it through views and raw artifacts. Do not rerun merely to demonstrate process.
|
|
85
111
|
|
|
86
|
-
|
|
112
|
+
An explicitly delivered body may support first-read behavior while a natural prompt still fails activation. Preserve both conclusions; one does not repair the other.
|
|
@@ -33,7 +33,7 @@ With no selectors, `run` or `view` selects every suite group and both variants.
|
|
|
33
33
|
|
|
34
34
|
## Current Execution Boundary
|
|
35
35
|
|
|
36
|
-
`run` executes description and body groups.
|
|
36
|
+
`run` executes description and body groups. Unsupported group types are rejected before subject execution. Description groups allow no tools or `read`; body groups allow no tools or path-guarded `read`, `write`, and `edit`, plus non-native custom tools supplied by a declared runtime extension bundle. Native `bash`, `powershell`, `grep`, `find`, and `ls` tools and definition-supplied commands are unsupported.
|
|
37
37
|
|
|
38
38
|
A runtime profile selects the installed `pi` or `piflow` host, whether the variant uses an isolated persistent session directory, zero or more ordered extension bundles, and declarative environment sources. Literal values are non-secret configuration; only explicitly inherited parent variables are passed to the child without being persisted, alongside a small host-runtime baseline. Missing inherited variables invalidate the variant. Launch-control and loader keys are rejected from inherited values, and literal and inherited names must not overlap. For body runs, delivery remains `unavailable` until target setup succeeds; `natural-prompt` or `explicit-skill-command` records a prompt attempt. When a group declares context assertions, the evaluator launches the base guard first, declared extensions second, and a final non-mutating observer last. The observer records both the effective `before_agent_start` system prompt and each provider-neutral `context` message projection; it does not replace or sanitize extension injections after the base guard has established evaluator-owned isolation.
|
|
39
39
|
|
|
@@ -1,51 +1,92 @@
|
|
|
1
1
|
# Review And Revision
|
|
2
2
|
|
|
3
|
-
Read this after canonical evidence and deterministic grades exist, when evidence conflicts, meaning
|
|
3
|
+
Read this after canonical evidence and deterministic grades exist, when evidence conflicts, meaning remains unresolved, the failed boundary must be classified, or revision and rerun scope must be chosen.
|
|
4
4
|
|
|
5
5
|
## Use The Strongest Evidence
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
2. Structured events, tool activity, usage, process results, logs, and saved reports.
|
|
9
|
-
3. Final response.
|
|
10
|
-
4. Full transcript when earlier surfaces cannot answer the question.
|
|
11
|
-
5. Subject self-assessment.
|
|
7
|
+
Prefer, in order:
|
|
12
8
|
|
|
13
|
-
|
|
9
|
+
1. filesystem state, typed workspace manifests, diffs, and changed paths;
|
|
10
|
+
2. structured events, tool activity, usage, process results, logs, and saved reports;
|
|
11
|
+
3. final response;
|
|
12
|
+
4. full transcript when stronger surfaces cannot answer the question;
|
|
13
|
+
5. subject self-assessment.
|
|
14
|
+
|
|
15
|
+
Lower-priority evidence cannot repair a contradiction above it.
|
|
16
|
+
|
|
17
|
+
If a response claims success but a required file is absent, preserve both facts and judge from the stronger artifact evidence.
|
|
14
18
|
|
|
15
19
|
## Keep Deterministic Facts Separate
|
|
16
20
|
|
|
17
|
-
Built-in checks
|
|
21
|
+
Built-in checks establish exact observable facts. Missing evidence remains unavailable. A failed check remains behavioral evidence and does not relabel a complete run as infrastructure failure.
|
|
22
|
+
|
|
23
|
+
The active agent or user may judge:
|
|
18
24
|
|
|
19
|
-
|
|
25
|
+
- reasoning quality;
|
|
26
|
+
- authority preservation;
|
|
27
|
+
- architectural fitness;
|
|
28
|
+
- recommendation quality;
|
|
29
|
+
- semantic completeness.
|
|
20
30
|
|
|
21
|
-
|
|
31
|
+
Keep semantic judgment, cited evidence, uncertainty, and limits outside canonical run and deterministic grade state.
|
|
22
32
|
|
|
23
|
-
|
|
33
|
+
Do not turn semantic judgment into evaluator-owned readiness or promotion status.
|
|
34
|
+
|
|
35
|
+
## Diagnose The Failed Boundary
|
|
36
|
+
|
|
37
|
+
Classify before revising:
|
|
24
38
|
|
|
25
39
|
- activation timing;
|
|
26
40
|
- description language unavailable before body read;
|
|
27
41
|
- first-read body comprehension;
|
|
28
42
|
- undeclared, undelivered, unread, or ambient-only dependency;
|
|
29
|
-
- normal guidance hidden behind
|
|
43
|
+
- normal guidance hidden behind the wrong resource boundary;
|
|
30
44
|
- nearby behavior that hijacks the task;
|
|
31
45
|
- retained-use or multi-turn failure;
|
|
32
|
-
- wording, placement, example, or stop-condition pressure;
|
|
33
|
-
- fixture, environment, isolation, host, or deterministic-check failure.
|
|
46
|
+
- wording, placement, example, repetition, or stop-condition pressure;
|
|
47
|
+
- fixture, environment, isolation, host, runner, or deterministic-check failure.
|
|
48
|
+
|
|
49
|
+
Do not add broad prose before identifying which boundary failed.
|
|
34
50
|
|
|
35
|
-
|
|
51
|
+
## Return Revision To Write Skill
|
|
36
52
|
|
|
37
|
-
|
|
53
|
+
When evidence supports a skill change:
|
|
38
54
|
|
|
39
|
-
|
|
55
|
+
1. name the exact failed boundary;
|
|
56
|
+
2. cite the canonical evidence;
|
|
57
|
+
3. identify one measured pressure point;
|
|
58
|
+
4. preserve unrelated instructions and fixed criteria;
|
|
59
|
+
5. return the revision target to [Write Skill](../../write-skill/SKILL.md).
|
|
40
60
|
|
|
41
|
-
|
|
61
|
+
Write Skill owns package revision.
|
|
62
|
+
|
|
63
|
+
Evaluate Skill does not edit the skill or replace its baseline while evaluating it.
|
|
64
|
+
|
|
65
|
+
When evidence proves the group, fixture, or criterion is wrong, revise the evaluation instead. Preserve the old result and state that it no longer answers the revised question.
|
|
42
66
|
|
|
43
67
|
## Rerun The Complete Group
|
|
44
68
|
|
|
45
|
-
After a measured skill
|
|
69
|
+
After a measured skill revision or resolved infrastructure cause:
|
|
70
|
+
|
|
71
|
+
- rerun the complete fixed group;
|
|
72
|
+
- run both baseline and candidate;
|
|
73
|
+
- keep every result visible;
|
|
74
|
+
- do not reuse an earlier baseline run as a fresh counterpart;
|
|
75
|
+
- do not rerun only the preferred variant;
|
|
76
|
+
- do not use partial evidence as a comparable full result;
|
|
77
|
+
- do not rerun merely to erase failure.
|
|
46
78
|
|
|
47
|
-
For variance, use independent complete invocations
|
|
79
|
+
For variance, use independent complete invocations rather than repeated preferred variants.
|
|
48
80
|
|
|
49
81
|
## Report
|
|
50
82
|
|
|
51
|
-
Report
|
|
83
|
+
Report:
|
|
84
|
+
|
|
85
|
+
- failed boundary and evidence;
|
|
86
|
+
- exact skill revision or evaluation correction;
|
|
87
|
+
- criteria that remained fixed;
|
|
88
|
+
- old and new result paths;
|
|
89
|
+
- deterministic transitions;
|
|
90
|
+
- remaining unsupported claims.
|
|
91
|
+
|
|
92
|
+
The user decides whether to revise further, use, publish, or reject the skill.
|