@researai/deepscientist 1.5.17 → 1.6.0

package/src/skills/baseline/references/artifact-flow-examples.md

@@ -0,0 +1,106 @@
+# Baseline Artifact Flow Examples
+
+Use this reference when the baseline route is clear but the exact artifact sequence is not.
+These are examples, not the only legal paths.
+The hard rule is still: attach/import/publish does not open the baseline gate by itself.
+
+## 1. Reuse an existing registry baseline
+
+Use when a trustworthy reusable baseline already exists in the registry and can support the current comparison contract.
+
+Typical sequence:
+
+1. `artifact.attach_baseline(...)`
+2. inspect the attached package, outputs, provenance, and metric contract
+3. run only the minimum extra verification needed for current trust
+4. write or reuse `<baseline_root>/json/metric_contract.json`
+5. `artifact.confirm_baseline(...)`
+
+Good:
+
+- the registry entry is materialized into the quest
+- the attached outputs are traceable
+- caveats are carried into confirmation if needed
+
+Bad:
+
+- treating `artifact.attach_baseline(...)` as if it already opened the gate
+- attaching one package and then comparing against a different local path without recording it
+
+## 2. Import a local package or bundle
+
+Use when the user already provided a prepared baseline package, bundle, or snapshot.
+
+Typical sequence:
+
+1. materialize the imported package under the quest baseline roots
+2. keep `attachment.yaml` or equivalent provenance durable
+3. inspect outputs / metrics / provenance
+4. write or reuse `<baseline_root>/json/metric_contract.json`
+5. `artifact.confirm_baseline(...)`
+
+Good:
+
+- imported package is readable and traceable
+- imported metrics are tied to real files, not only copied prose
+
+Bad:
+
+- importing a bundle and calling it accepted before checking what task / split / metric it actually used
+
+## 3. Verify a local-existing comparator
+
+Use when a local code path or local service already exists and can be evaluated cheaply.
+
+Typical sequence:
+
+1. identify the concrete path or endpoint
+2. identify the real evaluation command or endpoint
+3. verify outputs or metrics under the intended contract
+4. write `<baseline_root>/json/metric_contract.json`
+5. `artifact.confirm_baseline(...)`
+
+Good:
+
+- the local comparator is cheaper and more faithful than a full clean-room reproduction
+
+Bad:
+
+- assuming the local implementation matches the paper without checking split / metric / protocol
+
+## 4. Publish a quest-local baseline for reuse
+
+Use when the current quest already contains a verified reusable baseline and you want to publish it.
+
+Typical sequence:
+
+1. finish verification first
+2. make sure provenance, caveats, and metrics are trustworthy
+3. `artifact.confirm_baseline(...)` if the current quest gate is still unresolved
+4. `artifact.publish_baseline(...)`
+
+Good:
+
+- publish only after the local quest already trusts the baseline
+
+Bad:
+
+- publishing a half-verified line just because the files look complete
+
+## 5. Waive the baseline gate
+
+Use only when the quest must continue without a baseline and the reason is real, explicit, and durable.
+
+Typical sequence:
+
+1. record why the baseline cannot be cleared now
+2. record what was tried and what remains missing
+3. `artifact.waive_baseline(...)`
+
+Good:
+
+- the waiver reason is clear enough that a later turn can revisit it
+
+Bad:
+
+- waiving because reproduction is annoying rather than because the route is genuinely blocked or out of scope

package/src/skills/baseline/references/artifact-payload-examples.md

@@ -1,6 +1,6 @@
 # Artifact Payload Examples
 
-Use this reference when the `baseline` stage needs a stable payload shape without re-expanding the main skill body.
+Use this reference when `baseline` needs a stable artifact shape without re-expanding the whole skill body.
 
 ## Route or blocked decision
 

package/src/skills/baseline/references/baseline-checklist-template.md

@@ -1,46 +1,38 @@
-# Baseline Checklist Template
+# Baseline Gate Checklist Template
 
-Use this as a living checklist.
-Keep it short by default. For a fast path, complete the core checklist first and expand only if the route becomes complex or unstable.
+Use this as a compact acceptance-boundary checklist when it helps.
+It is optional; the hard requirement is that the accepted, blocked, waived, or route-changed state is durable and unambiguous.
 
 ## Identity
 
 - baseline id:
 - route:
-- owner stage:
+- acceptance target:
+- primary comparator:
 
-## Core
+## Current Frontier
 
-- [ ] baseline object and route are explicit
-- [ ] dataset / split and metric contract are explicit enough to judge comparability
-- [ ] `PLAN.md` captures the command path, expected outputs, acceptance condition, and fallback
-- [ ] smoke decision is explicit:
-  - skipped for a justified reason, or run once with outputs checked
-- [ ] real validation/run decision is explicit:
-  - skipped for a justified reason, or launched/read with durable evidence
-- [ ] expected result files and required metrics are checked
-- [ ] baseline is accepted, blocked, or waived with a durable note
+- [ ] next execution, verification, acceptance, blocker, or route-switch step is explicit
+- [ ] active uncertainty is written as a concrete question
+- [ ] next stage is known if this gate clears
+
+## Core Gate
+
+- [ ] comparator identity and provenance are explicit
+- [ ] dataset, split, evaluation path, required metrics, and metric directions are explicit enough to judge comparability
+- [ ] trusted outputs or metrics are traceable to concrete files, logs, service responses, source artifacts, or accepted package records
+- [ ] smoke was used, skipped, or replaced by direct verification for an explicit reason when that choice matters
+- [ ] expected result files or trusted-output pointers have been checked
+- [ ] `<baseline_root>/json/metric_contract.json` exists or will be produced before acceptance
+- [ ] baseline is accepted, blocked, waived, or route-changed with a durable note
+
+## Blocked Boundary
+
+- [ ] if blocked, the failure class is explicit
+- [ ] if blocked, tried steps and evidence paths are recorded
+- [ ] if blocked, next best move is attach, import, retry, repair, reset, waive, or ask the user
 
 ## Closeout
 
-- [ ] concise `1-2` sentence baseline summary written
-- [ ] next stage named explicitly
-
-## Optional Expansion
-
-Fill this only when the route becomes full-audit, repair-heavy, or publication-oriented.
-
-- [ ] paper source identified
-- [ ] repo source identified
-- [ ] paper read enough to restate the core method faithfully
-- [ ] repo read enough to identify the real entrypoints
-- [ ] main files to inspect or modify listed
-- [ ] working directory confirmed
-- [ ] environment route chosen
-- [ ] key dependencies checked
-- [ ] model / data download path confirmed
-- [ ] fallback source recorded for critical downloads
-- [ ] monitoring cadence started
-- [ ] health signals confirmed
-- [ ] any execution deviation reflected back into `PLAN.md`
-- [ ] verification note written
+- [ ] concise baseline summary written
+- [ ] next anchor named explicitly

package/src/skills/baseline/references/baseline-plan-template.md

@@ -1,89 +1,50 @@
-# Baseline Plan Template
+# Baseline Route Record Template
 
-Use this when the `baseline` stage becomes concrete enough to act.
-Keep it short when the route is simple. For fast-path attach/import/prebound validation, a one-screen plan is enough if it preserves the route, command path, outputs, acceptance condition, and fallback.
-Expand the optional sections only when the route is ambiguous, code-touching, broken, multi-variant, or intended for reuse beyond the current quest.
+Use this only when a durable route record would reduce ambiguity.
+It is a reference template, not a required path or required filename.
+Keep it one-screen for fast-path attach/import/local verification.
+Expand only when the route is ambiguous, code-touching, expensive, broken, long-running, or intended for reuse beyond the current quest.
 
-## 1. Core Contract
+## 1. Objective
 
-- quest goal:
-- user's core requirements:
-- non-negotiable user constraints:
-- chosen baseline route:
-  - attach / import / reproduce / repair
+- node objective:
+- acceptance target:
+- success condition:
+- user constraints that change comparability:
+
+## 2. Baseline Contract
+
+- chosen route:
+  - attach / import / verify-local-existing / reproduce / repair
 - baseline id:
-- variant id:
-- source paper:
-- source repo:
+- comparator identity:
+- source paper / repo / package / service:
 - source commit / version / tag:
 - task:
 - dataset / split:
-- metric contract:
-- expected command path:
-- expected outputs:
-- acceptance condition:
-- cheapest fallback:
-
-## 2. Execution Path
-
-- working directory:
-- environment plan:
-- required downloads:
-- hardware assumptions:
-- smoke test needed:
-  - yes / no
-- smoke command:
-- main validation or run command:
-- expected runtime / budget:
-- durable log path:
-- verification targets:
-- fastest failure signal:
-
-## 3. Risks And Revision
-
-- main risks:
-- when to escalate from fast path to full audit:
-- revision note:
+- required metric ids and directions:
+- known deviations:
 
-## 4. Optional Expansion
+## 3. Execution Choice
 
-Fill this only when the route is no longer simple.
-
-- fallback repo or mirror:
-- checkpoints / models:
-- likely external blockers:
-- safe efficiency levers to try first:
-- health signals that justify continued monitoring rather than intervention:
-- conditions that require plan revision or kill-and-relaunch:
-- paper summary in `1-3` bullets:
-- repo summary in `1-3` bullets:
-- what the baseline actually does:
-- what the likely bottlenecks or brittle points are:
-- what still needs verification:
-
-## 5. Optional Code Touchpoints
-
-List the main files or modules only when you expect real inspection or edits.
-
-| Path | Role | Why it matters now | Expected action | Notes |
-|---|---|---|---|---|
-| | | | inspect / modify / leave alone | |
-
-## 6. Optional Verification Plan
-
-- required result files:
-- required metric keys:
-- comparability checks:
-- acceptance condition:
-- downgrade / blocked condition:
+- working location:
+- environment route:
+- command / endpoint / trusted-output path:
+- expected outputs:
+- expected runtime / cost:
+- smoke or direct-run decision:
+- why this path is sufficient:
 
-## 7. Checklist Link
+## 4. Acceptance Boundary
 
-- checklist path:
-- which item should move next:
+- evidence required to accept:
+- fastest failure signal:
+- blocker condition:
+- fallback or route switch:
 
-## 8. Revision Log
+## 5. Current Frontier
 
-| Time | What changed | Why it changed | Impact on execution |
-|---|---|---|---|
-| | | | |
+- next action:
+- active blocker:
+- latest evidence:
+- next stage when accepted / blocked:

package/src/skills/baseline/references/boundary-cases.md

@@ -0,0 +1,86 @@
+# Baseline Boundary Cases
+
+Use this reference when the baseline route is not blocked but the success boundary still feels fuzzy.
+
+## 1. Comparison-ready but not paper-repro-ready
+
+This is acceptable when:
+
+- the comparator is trustworthy enough for downstream comparison
+- the core metric contract is durable
+- later experiment work does not need to guess task / split / metric / comparator identity
+
+This is not yet paper-repro-ready when:
+
+- exact paper-side setup details are still partially unknown
+- broader variant tables or extra subtasks are still missing
+- the package is not reusable enough to publish
+
+## 2. Trusted with caveats
+
+This is acceptable when:
+
+- the main comparator is still honest and usable
+- deviations are explicit
+- the caveat does not silently change the comparison meaning
+
+Examples:
+
+- local service path verified, but one optional auxiliary metric is unavailable
+- source repo uses a repo-native docker path rather than `uv`, and that change is more faithful, not less
+
+Not acceptable:
+
+- caveat hides a different dataset split or evaluation script
+
+## 3. Imported metrics but weak provenance
+
+Do not accept when:
+
+- the package contains a number but you cannot tie it to a real output or evaluator path
+- the user pasted a paper table without local or package-side evidence and the target requires verification
+
+Possible next routes:
+
+- verify local existing
+- reproduce from source
+- block and record weak provenance
+
+## 4. Local path exists but exact comparator is unclear
+
+Do not default into full reproduction immediately.
+First ask:
+
+- is there a real evaluation entrypoint?
+- is there a concrete output location?
+- can the split and metric contract be identified?
+
+Only escalate to reproduction if those answers stay too ambiguous for honest comparison.
+
+## 5. Route feels cleaner but not more trustworthy
+
+Do not replace a working comparison-ready comparator with a heavier route merely because:
+
+- the source reproduction feels aesthetically cleaner
+- the repo-native path is more complicated than you expected
+- the existing comparator is already good enough for the next scientific step
+
+The heavier route becomes justified only when it removes a named unresolved comparison risk.
+
+## 6. Repeated failure with no new evidence
+
+Stop looping when:
+
+- same command class fails again
+- no code changed
+- no environment changed
+- no route changed
+- no new evidence reduced uncertainty
+
+At that point:
+
+- record the blocker
+- switch route
+- repair
+- waive
+- or route through `decision`

package/src/skills/baseline/references/codebase-audit-checklist.md

@@ -1,6 +1,7 @@
 # Codebase Audit Checklist
 
-Use this during baseline analysis before deciding whether to attach, import, reproduce, or repair.
+Use this only when attach/import/verify-local-existing is not enough and you truly need a fuller source audit.
+Typical triggers are reproduce-from-source, repair with code changes, or an unclear evaluation entrypoint.
 
 ## Minimum audit coverage
 
@@ -8,7 +9,6 @@ Record:
 
 - repository or package identity
 - main entrypoints
-- training path
 - evaluation path
 - data loading and preprocessing path
 - configuration path
@@ -21,8 +21,6 @@ Identify:
 
 - key classes and functions
 - scripts that are likely to be run directly
-- modules with high coupling
-- modules where later changes are most likely to occur
 
 ## Practical constraints
 
@@ -30,7 +28,6 @@ Check:
 
 - external services or downloads
 - hardware assumptions
-- heavy memory or compute hotspots
 - brittle or undocumented environment requirements
 
 ## Baseline understanding goal
@@ -41,4 +38,3 @@ A later stage should be able to answer all of the following from your audit with
 - how it is run
 - how it is evaluated
 - where the main risks or bottlenecks are
-- which files will matter most for future modifications

package/src/skills/baseline/references/comparability-contract.md

@@ -2,31 +2,26 @@
 
 Use this reference when deciding whether a baseline is truly usable downstream.
 
-## Minimum contract
+## Minimum or core contract
 
 Make these fields explicit:
 
 - task identity
 - dataset identity
-- dataset version
 - split contract
-- preprocessing boundary
 - evaluation script or path
 - required metric keys
 - metric directions
-- seed policy if relevant
 - source commit or package identity
 - known deviations
 
+A core contract is enough for a `comparison_ready` baseline.
+Expand to a fuller contract only when later paper claims, variant-heavy comparison, or publication really need it.
+
 ## Verdict logic
 
-- `verified`:
-  - comparison contract is stable enough for downstream work
-- `partially_verified`:
-  - mostly usable, but caveats must travel with it
-- `operational_but_incomparable`:
-  - the system runs, but later stages should not treat it as a clean comparison
-- `failed`:
-  - not suitable as a baseline
+- usable now
+- usable with caveats
+- blocked
 
 If later `experiment` work would have to keep guessing the comparison contract, the baseline is not ready.

package/src/skills/baseline/references/operational-guidance.md

@@ -0,0 +1,56 @@
+# Operational Guidance
+
+Use this reference when the baseline route needs the longer operational notes rather than the main control surface.
+
+## Durable route records
+
+Durable records are required in substance, not in fixed filenames.
+The agent may choose the shortest durable form that lets a later turn resume without guessing.
+
+For non-trivial, code-touching, expensive, unstable, or long-running baseline work, leave a route record that states:
+
+- chosen route and acceptance target
+- comparator identity and source identity
+- command or evaluation path if one exists
+- expected outputs or trusted-output pointers
+- acceptance condition
+- current blocker or fallback
+- verification verdict
+
+`PLAN.md`, `CHECKLIST.md`, `setup.md`, `execution.md`, `verification.md`, `analysis_plan.md`, and `REPRO_CHECKLIST.md` are allowed compatibility surfaces, not mandatory success paths.
+Use `references/baseline-plan-template.md` and `references/baseline-checklist-template.md` when they help, but do not expand them as paperwork.
+
+`attachment.yaml` or equivalent provenance is required for attached or imported baselines.
+`<baseline_root>/json/metric_contract.json` as the canonical accepted comparison contract is required for accepted baselines.
+
+## Execution tactics
+
+Use whatever route is most faithful, observable, and efficient while preserving the hard gates.
+
+- If source reproduction or repair is actually the active route, read the source paper and source repo before substantial setup.
+- For attach, import, or verify-local-existing, inspect only the minimum evidence needed to trust the provided or local comparator.
+- A bounded smoke test is usually helpful only when command path, environment viability, evaluator wiring, or output schema is still unclear.
+- If the path is already concrete, go straight to real verification or the real run.
+- Treat smoke or pilot work as a `0-2` default budget, but the real rule is not to repeat an unchanged check without new evidence, a code/environment change, or a route change.
+- If runtime is uncertain or likely long, prefer `bash_exec(mode='detach', ...)` plus managed monitoring instead of pretending a short foreground timeout is enough.
+- If a run is clearly invalid, wedged, or superseded, stop it cleanly and relaunch with the new route rather than stacking more retries.
+
+## Environment tactics
+
+For Python baselines, prefer a reproducible isolated environment, but choose the route that is most faithful to the source package and most likely to produce comparable evidence.
+
+`uv` is a useful default tactic when the repo does not require a stronger native route.
+Examples include `uv sync`, `uv venv`, `uv pip install ...`, and `uv run ...`.
+Switch to repo-native conda, docker, poetry, shell scripts, service startup, or another local environment route when that is clearly more trustworthy, required by the source, or necessary to match the paper or package behavior.
+
+Record only environment facts that affect trust or comparability.
+Do not force a global `uv` route when it would make the reproduced baseline less faithful.
+
+## Reuse and memory
+
+Reuse or publish a baseline only after verification is complete and the current quest no longer depends on guesswork about provenance or comparability.
+Do not publish a baseline for reuse if verification is incomplete, metrics are untrusted, or provenance is still weak.
+
+Use memory only to avoid repeating known failures or to preserve reusable baseline lessons, not as a required step before every validation pass.
+Write quest memory for route rationale, setup failures, paper-to-code mismatch notes, and accepted caveats that later stages must carry forward.
+Promote to global memory only when another quest is likely to benefit from the lesson.

package/src/skills/baseline/references/route-selection.md

@@ -2,16 +2,10 @@
 
 Use this reference when the `baseline` stage needs a clearer decision among attach, import, reproduce, and repair.
 
-## Decision order
+## Decision rule
 
-Prefer:
-
-1. attach
-2. import
-3. reproduce
-4. repair
-
-But only when the chosen route still produces a trustworthy downstream reference.
+Choose the route that gives the best trust per unit time and compute.
+Do not follow a fixed ritual if another route reaches a cleaner comparison contract faster.
 
 ## Route meanings
 
@@ -19,23 +13,9 @@ But only when the chosen route still produces a trustworthy downstream reference
   - reuse an already published and trustworthy baseline
 - `import`:
   - bring a reusable baseline package or bundle into the current quest
+- `verify-local-existing`:
+  - validate a local code path or service that already exists
 - `reproduce`:
   - establish a baseline from source paper, repo, and evaluation path
 - `repair`:
   - fix a bounded failure in an existing baseline line
-
-## Questions to answer before choosing
-
-- what exact baseline object are we using?
-- what proof of trustworthiness already exists?
-- what evidence is still missing?
-- what is the cheapest credible next step?
-
-## Route rejection logic
-
-Reject a route if:
-
-- provenance is weak
-- metric or split compatibility is poor
-- the cost is much higher than an equally trustworthy alternative
-- the result would still be unusable downstream