superlab 0.1.12 → 0.1.13

package/package-assets/shared/skills/lab/stages/auto.md

@@ -16,9 +16,11 @@
 - `.lab/context/decisions.md`
 - `.lab/context/data-decisions.md`
 - `.lab/context/evidence-index.md`
+- `.lab/context/eval-protocol.md`
 - `.lab/context/terminology-lock.md`
 - `.lab/context/auto-mode.md`
 - `.lab/context/auto-status.md`
+- `.lab/context/auto-outcome.md`
 
 ## Context Write Set
 
@@ -29,16 +31,34 @@
 - `.lab/context/summary.md`
 - `.lab/context/session-brief.md`
 - `.lab/context/auto-status.md`
+- `.lab/context/auto-outcome.md`
 
 ## Boundary Rules
 
 - Treat `/lab:auto` as an orchestration layer, not a replacement for existing `/lab:*` stages.
+- Treat `.lab/context/eval-protocol.md` as the source of truth for paper-facing metrics, metric glossary, table plan, gates, and structured experiment ladders.
+- Treat the evaluation protocol as source-backed, not imagination-backed: metric definitions, baseline behavior, comparison implementations, and deviations must come from recorded sources before they are used in gates or promotions.
+- The contract must declare `Autonomy level` and `Approval status`, and execution starts only when approval is explicitly set to `approved`.
+- The contract must also declare a concrete terminal goal:
+  - `rounds`
+  - `metric-threshold`
+  - `task-completion`
+- The contract must provide both `Terminal goal target` and `Required terminal artifact`.
+- Recommended level meanings:
+  - `L1`: safe run validation over `run`, `review`, and `report`
+  - `L2`: bounded iteration over `run`, `iterate`, `review`, and `report`
+  - `L3`: aggressive campaign that may also include `write`
 - Default allowed stages are `run`, `iterate`, `review`, and `report`. Only include `write` when framing is already approved and manuscript drafting is within scope.
 - Do not automatically change the research mission, paper-facing framing, or core claims.
 - You may add exploratory datasets, benchmarks, and comparison methods inside the exploration envelope.
 - You may promote exploratory additions to the primary package only when the contract's promotion policy is satisfied and the promotion is written back into `data-decisions.md`, `decisions.md`, `state.md`, and `session-brief.md`.
 - Poll long-running commands until they finish, hit a timeout, or hit a stop condition.
 - Keep a poll-based waiting loop instead of sleeping blindly.
+- Always write a canonical `.lab/context/auto-outcome.md` when the run completes, stops, or fails.
+- When the evaluation protocol declares structured ladder rungs, execute them as a foreground rung state machine:
+  - each rung must declare `Stage`, `Goal`, `Command`, `Watch`, `Gate`, `On pass`, `On fail`, and `On stop`
+  - keep the session alive while the current rung is running
+  - write the current rung, watch target, and next rung to `.lab/context/auto-status.md`
 - Reuse the existing `/lab:run`, `/lab:iterate`, `/lab:review`, `/lab:report`, and optional `/lab:write` contracts instead of inventing a parallel workflow.
 - Enforce stage contracts, not just exit codes:
   - `run` and `iterate` must change persistent outputs under `results_root`
@@ -46,20 +66,24 @@
   - `report` must produce `<deliverables_root>/report.md`
   - `write` must produce LaTeX output under `<deliverables_root>/paper/`
 - Treat promotion as incomplete unless it writes back to `data-decisions.md`, `decisions.md`, `state.md`, and `session-brief.md`.
+- Do not stop or promote on the basis of a metric or comparison claim whose source-backed definition is missing from the approved evaluation protocol.
 
 ## Minimum Procedure
 
 1. Validate the auto-mode contract
-2. Set or refresh auto-status
-3. Choose the next allowed `/lab` stage
-4. Launch the bounded action
-5. Poll for process completion, checkpoint movement, or summary generation
-6. Evaluate stop, success, and promotion checks at the correct boundary
-7. Record results and decide continue, promote, stop, or escalate
+2. Confirm the approved autonomy level matches the requested stage envelope
+3. Set or refresh auto-status
+4. Choose the next allowed `/lab` stage or structured ladder rung
+5. Launch the bounded action
+6. Poll for process completion, checkpoint movement, or summary generation while keeping the session alive
+7. Evaluate the declared rung gate and transition to the next rung when structured ladder mode is active
+8. Evaluate the declared terminal goal semantics at the correct boundary
+9. Evaluate stop, success, and promotion checks at the correct boundary
+10. Write auto-outcome and decide continue, promote, stop, or escalate
 
 ## Interaction Contract
 
 - Start with a concise summary of the objective, the frozen core, and the next automatic stage.
 - If the contract is incomplete, ask one clarifying question at a time.
 - If multiple next actions are credible, present 2-3 bounded options with trade-offs before arming a long run.
-- Only ask for approval when the next step would leave the approved exploration envelope or materially change the frozen core.
+- Only ask for approval when the next step would leave the approved exploration envelope, exceed the chosen autonomy level, or materially change the frozen core.

package/package-assets/shared/skills/lab/stages/iterate.md

@@ -8,6 +8,7 @@ Declare and keep fixed:
 - baseline
 - primary metric
 - success threshold
+- evaluation ladder and benchmark expansion gates
 - verification commands
 - completion_promise
 - maximum iteration count
@@ -19,6 +20,7 @@ Declare and keep fixed:
 - `.lab/context/decisions.md`
 - `.lab/context/evidence-index.md`
 - `.lab/context/data-decisions.md`
+- `.lab/context/eval-protocol.md`
 - `.lab/config/workflow.json`
 
 ## Context Write Set
@@ -58,6 +60,8 @@ If the loop stops without success, record:
 - Keep durable run outputs, logs, and checkpoints under `results_root`.
 - Keep figures or plots under `figures_root`.
 - Do not accumulate long-lived results under `.lab/changes/<change-id>/runs`.
+- Do not change metric definitions, baseline semantics, or comparison implementations unless the approved evaluation protocol records both their sources and any deviations.
+- When you change ladders, sample sizes, or promotion gates, keep the resulting logic anchored to the source-backed evaluation protocol instead of ad-hoc chat reasoning.
 
 ## Interaction Contract
 

package/package-assets/shared/skills/lab/stages/report.md

@@ -17,6 +17,7 @@
 - `.lab/context/decisions.md`
 - `.lab/context/evidence-index.md`
 - `.lab/context/data-decisions.md`
+- `.lab/context/eval-protocol.md`
 - `.lab/context/terminology-lock.md`
 
 ## Context Write Set
@@ -28,6 +29,9 @@
 
 - Do not hide failed iterations.
 - Tie every major claim to recorded summaries or iteration artifacts.
+- Structure tables, gates, and main claims against the approved evaluation protocol.
+- Do not restate metric definitions, baseline behavior, or comparison implementations from memory; use the approved evaluation protocol and its recorded sources.
+- If the report depends on a deviation from an original metric or implementation, state that deviation explicitly instead of smoothing it over.
 - Prefer conservative interpretation over marketing language.
 - Leave a clear handoff path into `/lab:write` with evidence links that section drafts can cite.
 

package/package-assets/shared/skills/lab/stages/run.md

@@ -12,6 +12,7 @@
 - `.lab/context/mission.md`
 - `.lab/context/state.md`
 - `.lab/context/data-decisions.md`
+- `.lab/context/eval-protocol.md`
 - `.lab/config/workflow.json`
 
 ## Context Write Set
@@ -23,6 +24,8 @@
 
 - Prefer the smallest experiment that exercises the full pipeline.
 - Fail fast on data, environment, or metric wiring problems.
+- Tie the run to the approved evaluation protocol, not just an ad-hoc chat goal.
+- Do not invent metric definitions, baseline behavior, or comparison implementations from memory; anchor them to the approved evaluation protocol and its recorded sources.
 - Record the exact launch command and output location.
 - Write durable run outputs, logs, and checkpoints under `results_root`.
 - Write figures or plots under `figures_root`.
@@ -34,7 +37,7 @@
 2. Register the run
 3. Execute the smallest useful experiment
 4. Normalize raw metrics
-5. Validate the normalized summary
+5. Validate the normalized summary against the active evaluation protocol
 
 ## Interaction Contract
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "superlab",
-  "version": "0.1.12",
+  "version": "0.1.13",
   "description": "Strict /lab research workflow installer for Codex and Claude",
   "keywords": [
     "codex",