@researai/deepscientist 1.5.16 → 1.6.0

package/src/skills/references/tool-usage-by-stage.md

@@ -0,0 +1,438 @@
+# Tool Usage By Stage
+
+Use this shared reference when a stage skill needs precise guidance for when to use `artifact`, `bash_exec`, and `memory`.
+
+The goal is not a generic tool summary.
+The goal is accurate stage control for long-running autonomous quests where durable management, validation, and continuation matter.
+
+## Core roles
+
+- `artifact`
+  - canonical state surface
+  - canonical verification surface
+  - canonical route and handoff surface
+- `bash_exec`
+  - execution surface
+  - first-hand evidence surface
+  - local inspection and monitoring surface
+- `memory`
+  - reusable lesson surface
+  - anti-repeat surface
+  - authoritative resume-hint surface
+
+## Global rules
+
+### Default stage-start order
+
+For most stage passes:
+
+1. recover durable state with `artifact`
+2. retrieve reusable lessons with `memory`
+3. execute or inspect with `bash_exec`
+
+### Default stage-end order
+
+For most stage passes:
+
+1. finish the concrete execution or inspection with `bash_exec`
+2. convert the result into durable truth with `artifact`
+3. write `memory` only if the lesson should change future default behavior
+
+### When `artifact` is mandatory
+
+Use `artifact` before exit whenever:
+
+- a stage result became durable truth
+- the active route changed
+- the next stage became clear
+- a blocker became authoritative
+- a user-visible milestone needs to survive resume or handoff
+
+### When `memory` is mandatory
+
+Use `memory` when:
+
+- the next default action should change because of this pass
+- repeated failure is likely unless the lesson is preserved
+- a reusable success pattern appeared
+- the authoritative resume point changed materially
+
+Do not use `memory` as the primary record of baselines, experiments, analyses, or paper state.
+
+## Baseline
+
+### Use `artifact` when
+
+- recovering the active baseline gate state
+- checking whether `requested_baseline_ref` or `confirmed_baseline_ref` already exists
+- attaching, confirming, waiving, publishing, or blocking a baseline
+- recording the accepted comparator and metric contract
+
+### Use `memory` when
+
+- resuming a previously blocked or ambiguous baseline route
+- reopening an old command path or environment fix
+- repeated failures or environment incidents risk repeating
+- a paper-to-code mismatch or accepted caveat should affect later stages
+
+### Use `bash_exec` when
+
+- verifying a local existing comparator
+- running reproduction or repair
+- checking the evaluator, service, dataset path, or environment
+- collecting first-hand evidence from files, logs, or outputs
+
+### Do not
+
+- do not keep running `bash_exec` retries after the same failure class repeats with no real change
+- do not stop at attach/import/publish alone; use `artifact.confirm_baseline(...)` or `artifact.waive_baseline(...)`
+
+## Scout
+
+### Use `artifact` when
+
+- reconstructing the current frame from durable state
+- reading quest docs before broad search
+- recording the evaluation contract, baseline direction, or next anchor
+- recording an explicit blocked scout state
+
+### Use `memory` when
+
+- resuming after a pause
+- broad literature or repo search is about to start
+- you suspect the survey or shortlist was already partially built
+- repeated search would likely revisit the same cluster
+
+### Use `bash_exec` when
+
+- inspecting local repo docs, benchmark scripts, evaluator code, or dataset paths
+- local code or file inspection materially changes the frame
+
+### Use external search or DeepXiv when
+
+- the task frame, metric contract, or baseline direction is still unclear
+- the current neighborhood is too thin or too recent-only
+- novelty, field map, or historical lineage is still ambiguous
+
+If DeepXiv is available, prefer it for broad paper-centric discovery, citation expansion, and shortlist triage.
+If DeepXiv is unavailable, use search engines directly plus backward and forward citation chaining.
+
+### Do not
+
+- do not continue searching once the next anchor is already clear
+- do not treat recent papers as a substitute for field history
+
+## Idea
+
+### Use `artifact` when
+
+- recovering current board state and research branches
+- writing the objective contract and current board packet
+- recording selected, deferred, and rejected candidates
+- submitting a selected idea as a durable route
+
+### Use `memory` when
+
+- resuming an old ideation thread
+- checking whether a route is stale and should not be reopened
+- broad literature search is about to begin
+- a contradiction, failure pattern, or rejected-route reason should influence later ideation
+
+### Use `bash_exec` when
+
+- local code or evaluator inspection changes feasibility or implementation-surface judgment
+- local branch or file state matters for choosing a foundation
+
+### Use DeepXiv or search when
+
+- broad, history-aware literature search is needed
+- you need seminal papers, turning-point papers, SOTA papers, or citation chaining
+- the strongest prior work or field lineage is still unclear
+
+If DeepXiv is available, prefer it for broad paper-centric discovery.
+If not, use search engines plus citation chaining directly.
+
+### Multiple strong and feasible ideas
+
+If several candidates are truly strong:
+
+- if they are same-family micro-variants:
+  - do not keep them as separate serious routes
+  - merge them into one family-level candidate
+- if they are meaningfully different families:
+  - keep `1` selected route
+  - keep `1-2` deferred but serious alternatives durably
+- if the next regime is algorithm-first and validation is cheap:
+  - keep a small serious frontier, usually `2-3`
+  - hand that frontier into `optimize` as candidate briefs
+- if validation is expensive:
+  - keep one lead plus at most one hedge route
+
+Do not promote every strong-looking idea.
+Do preserve the serious non-winners if they are structurally different and still valuable.
+
+### Do not
+
+- do not start from “swap A for B” before naming the important contradiction or bottleneck
+- do not let only excitement decide promotion
+
+## Optimize
+
+### Use `artifact` when
+
+- recovering the optimization frontier
+- creating candidate briefs
+- promoting durable lines
+- recording implementation-level optimization candidates
+- recording route changes or main measured results
+
+### Use `memory` when
+
+- broad new search is about to start
+- the same durable line has recent sibling attempts
+- repeated failure, plateau, fusion, or non-retry rules should influence the next move
+
+### Use `bash_exec` when
+
+- running smoke checks
+- running quick validations
+- running full evaluations
+- debugging or validating fusion candidates
+
+### Candidate and frontier rules
+
+- keep candidate briefs small and differentiated
+- default serious brief count: `2-3`
+- default line promotion count: usually `1`, sometimes `2`, rarely `3`
+- do not let one familiar mechanism family fill the whole promoted slate
+- keep implementation-level live pools small, usually `2-3`
+
+### Do not
+
+- do not create a durable line for every plausible brief
+- do not treat an implementation candidate as a new durable line
+
+## Experiment
+
+### Use `artifact` when
+
+- recovering runtime refs, baseline refs, and quest state before the run
+- recording the main experiment with `artifact.record_main_experiment(...)`
+- recording the route decision after the measured result
+
+### Use `memory` when
+
+- reopening an old command path
+- retrying after a known failure pattern
+- environment fixes or comparability caveats may repeat
+
+### Use `bash_exec` when
+
+- running smoke or pilot checks
+- running the real experiment
+- monitoring long-running jobs
+- validating outputs and logs
+
+### Practical trigger rules
+
+- if the command path or evaluator wiring is still unclear, start with a bounded smoke or pilot
+- if the path is already concrete, go straight to the real run
+- if the result is suspicious or confounded, do not blindly rerun; route through diagnosis, decision, or analysis
+
+### Do not
+
+- do not treat a run as complete before `artifact.record_main_experiment(...)` succeeds
+- do not let logs alone become the accepted system state
+
+## Analysis Campaign
+
+### Use `artifact` when
+
+- durable lineage matters
+- multiple slices exist
+- paper/rebuttal/review traceability matters
+- campaign launch, slice recording, synthesis, or closeout becomes durable truth
+
+### Use `memory` when
+
+- resuming an old campaign
+- repeated slice failures or comparability caveats may recur
+- a cross-slice lesson should change later defaults
+
+### Use `bash_exec` when
+
+- running slices
+- monitoring slices
+- validating outputs
+- diagnosing slice failures
+
+### Route rules
+
+- if one bounded question is enough and lineage does not matter, a lighter report route can be enough
+- if multiple slices, traceability, or replay matter, use an artifact-backed campaign
+- if multiple slices are valuable, run claim-critical ones first and queue the rest
+
+### Do not
+
+- do not leave slice truth only in chat
+- do not start every plausible slice before the first decisive ones return evidence
+
+## Decision
+
+### Use `artifact` when
+
+- making any consequential route judgment
+- recording verdict, action, reason, evidence paths, and next direction
+
+### Use `memory` when
+
+- the authoritative resume point changed
+- a do-not-reopen rule or reusable route rationale should persist
+
+### Use `bash_exec` when
+
+- local evidence is still missing and a quick concrete check would materially change the route judgment
+
+### Candidate-choice rule
+
+If several options are plausible:
+
+- still choose one winner when local evidence is enough
+- record the main rejected alternative and why it lost
+- only ask the user when the choice is genuinely preference, scope, or cost sensitive and cannot be derived locally
+
+## Finalize
+
+### Use `artifact` when
+
+- checking global quest status
+- checking paper contract health
+- reading the scoreboard
+- completing the quest after explicit approval
+
+### Use `memory` when
+
+- a reusable closure lesson or final authoritative checkpoint should persist
+
+### Use `bash_exec` when
+
+- building, exporting, compiling, or locally verifying final deliverables
+
+### Do not
+
+- do not finalize through a still-blocked paper or evidence contract
+- do not call `artifact.complete_quest(...)` before explicit approval
+
+## Intake Audit
+
+### Use `artifact` when
+
+- recovering mixed state
+- reading quest docs, artifacts, and current runtime state
+- recording the current board packet, trust ranking, or next anchor recommendation
+
+### Use `memory` when
+
+- an old checkpoint, stale route, or resume point may still contaminate current judgment
+
+### Use `bash_exec` when
+
+- local file tree, git state, or output inspection is needed to resolve ambiguity
+
+### Do not
+
+- do not let a mixed-state quest proceed on conversational guesses about the active mainline
+
+## Review
+
+### Use `artifact` when
+
+- reading paper contract health
+- reading quest docs and evidence state
+- recording review findings, revision routes, and experiment requests
+
+### Use `memory` when
+
+- recurring review weaknesses or novelty caveats should persist
+
+### Use `bash_exec` when
+
+- building, compiling, or checking manuscript artifacts locally
+
+### Routing rule
+
+- if the problem is missing evidence, route to `analysis-campaign`
+- if the problem is wording, positioning, or claim scope, route to `write`
+- if the problem is novelty ambiguity, route to `scout`
+
+## Rebuttal
+
+### Use `artifact` when
+
+- reading reviewer packet state
+- recording response matrix, action plan, manuscript delta route, and evidence route
+
+### Use `memory` when
+
+- a recurring rebuttal pattern or claim downgrade should influence later responses
+
+### Use `bash_exec` when
+
+- building, compiling, or verifying local manuscript artifacts
+
+### Do not
+
+- do not hide an evidence gap under rhetorical rewriting
+
+## Paper Plot
+
+### Use `artifact` when
+
+- the figure requirement, output path, and downstream paper role need to be explicit
+
+### Use `bash_exec` when
+
+- rendering figures
+- rerendering after data or spec change
+- validating output existence and export
+
+### Use `memory` when
+
+- a plotting failure or rendering lesson is likely to recur
+
+### Do not
+
+- do not use `figure-polish` when the figure is still missing; first create the structured first-pass figure here
+
+## Figure Polish
+
+### Use `artifact` when
+
+- the figure already exists and the remaining blocker is presentation quality, not missing evidence
+
+### Use `bash_exec` when
+
+- rerendering polished versions
+- checking exports
+
+### Use `memory` when
+
+- a reusable polish or style lesson will affect later figures
+
+## Mentor
+
+### Use `artifact` when
+
+- reading the current board, route, blocker, or paper state before giving calibration advice
+
+### Use `memory` when
+
+- a reusable calibration lesson, founder preference, or do-not-repeat route mistake should persist
+
+### Use `bash_exec` when
+
+- local evidence inspection is needed before making a real calibration judgment
+
+### Do not
+
+- do not let mentor advice float free from the current durable state

package/src/skills/review/SKILL.md

@@ -7,6 +7,7 @@ skill_role: companion
 # Review
 
 Use this skill when the quest already has a substantial draft, paper, or paper-like report and now needs an independent, skeptical, evidence-grounded audit.
+The goal is to reduce revision ambiguity, not to sound harsh for its own sake.
 
 This is not the same as ordinary `write`.
 It is also not the same as `rebuttal`.
@@ -22,6 +23,12 @@ It is also not the same as `rebuttal`.
 - When the review report, revision plan, or follow-up experiment TODO list becomes durable, send a richer `artifact.interact(kind='milestone', reply_mode='threaded', ...)` update that says what the main risks are, what should be fixed next, and whether the next route is writing, experiment, or claim downgrade.
 - Hard execution rule: if this stage needs terminal work such as document builds, scripted checks, Git inspection, or file inspection, every such command must go through `bash_exec`.
 
+## Three-layer todo contract
+
+- treat quest-root `plan.md` as the top-level research map whose next anchor must become explicit after the review
+- if the review pass is multi-step, use workspace `PLAN.md` as the current review-node contract and `CHECKLIST.md` as the execution frontier
+- treat `paper/review/experiment_todo.md` as a review-facing frontier subset, not as a replacement for the quest-level map
+
 ## Purpose
 
 `review` is an auxiliary audit skill for paper-like deliverables.
@@ -37,6 +44,8 @@ It should convert “the draft feels almost done” into a durable, skeptical, t
 
 Default review stance: independent audit before celebration.
 Do not treat “looks polished” as “is defensible”.
+Do not accept structural green lights as paper-quality proof. Coverage, compile success, and file counts must still survive a skeptical claim/evidence/reviewer-risk audit.
+Apply the publishability stop-loss rule during review: when novelty, evidence sufficiency, or reader value has collapsed beyond reasonable claim narrowing, the correct recommendation is `decision` with `stop` or `branch`, with any narrowed non-paper objective recorded as the next direction, not another cosmetic revision pass. If recommending `stop` because paper quality is too low, ask the user to confirm before ending the paper objective. Consider user publication, scope, cost, or non-paper preferences, and ask for a user decision when those preferences are unclear and would change the recommendation.
 
 ## Use when
 
@@ -63,6 +72,9 @@ Do not treat “looks polished” as “is defensible”.
 - Keep every serious criticism evidence-grounded.
 - Do not recommend more experiments when the real problem is wording, positioning, or claim scope.
 - Do not recommend rhetoric when the real problem is missing evidence.
+- Do not recommend continued paper revision when the review finds a fatal publishability or value collapse; make the stop/branch route explicit.
+- Do not execute a stop route for a low-quality paper judgment without user confirmation; present the evidence, the stop recommendation, and any branch or narrowed non-paper alternative first.
+- Do not silently override a user's stated paper, report, or non-paper preference; consider it in the route recommendation, or ask when it is unclear.
 - If novelty or positioning is uncertain, treat that as a literature-audit question first, not an automatic experiment request.
 - If a claim is too broad for the evidence, prefer narrowing or downgrading the claim over defending it with style.
 - If `startup_contract.review_followup_policy` is present, honor it:
@@ -76,10 +88,50 @@ Do not treat “looks polished” as “is defensible”.
 - If LaTeX source is unavailable while `latex_required` is requested, do not pretend the manuscript was edited; produce LaTeX-ready replacement text and an explicit blocker note instead.
 - Accept manuscript and review inputs from URLs, local file paths, local directories, or current-turn attachments; do not assume the draft is already perfectly normalized.
 
+## Evidence Authenticity & Manuscript Coverage Gate
+
+Before judging a manuscript as strong, final, or submission-ready, perform an evidence audit.
+
+Build an experiment inventory from logs, matrices, result files, summaries, recent memory, and manuscript claims.
+For each main experiment and analysis experiment, record:
+
+- expected experiment id / purpose
+- status from durable artifacts, not only checklist labels
+- executable or log/result artifact paths
+- metrics actually present in result files or official summaries
+- whether the result is current, stale, duplicate, failed, negative, or superseded
+- whether the result appears in the manuscript, table, figure, caption, or appendix
+- which exact claim it supports
+
+Do not trust ready counts when duplicate rows, stale outline refs, pending rows, or missing metrics are present.
+Recompute the real paper-facing count manually.
+
+Classify every important result as one of:
+
+- completed and written
+- completed but not written
+- written but not evidenced
+- appendix-only
+- failed or negative
+- stale / legacy / duplicate
+- contradicted by another artifact
+
+Explicitly assess unsupported-claim or fabrication risk using these labels:
+
+- no issue found
+- provenance gap
+- manuscript overclaim
+- written result unsupported by durable evidence
+- contradiction between manuscript and artifacts
+- likely false or fabricated claim
+
+A paper cannot be called submission-ready unless compile/PDF, evidence provenance, manuscript coverage, citation sufficiency, language hygiene, and experiment-matrix consistency all pass.
+
 ## Primary inputs
 
 Use, in roughly this order:
 
+- `artifact.validate_manuscript_coverage(detail='full')` for paper bundles, memos, or alleged near-final manuscripts
 - the current paper or report draft
 - the selected outline if one exists
 - the claim-evidence map if one exists
@@ -88,7 +140,7 @@ Use, in roughly this order:
 - figures, tables, and captions
 - current-turn attachments and user-provided local paths / directories / URLs for the manuscript bundle or review packet
 - prior self-review or reviewer-first notes as low-trust auxiliary input
-- nearby papers when novelty or comparison is unclear
+- nearby papers and high-level accepted papers found through current literature search, not only remembered examples
 
 If the draft/result state is still unclear, open `intake-audit` first before continuing the review workflow.
 Before proposing extra experiments, read those structured `evaluation_summary` blocks first so you do not request work that the recorded evidence already resolved.
@@ -121,7 +173,11 @@ Audit at least these dimensions:
 - experimental validity and baseline comparability
 - claim scope and over-claiming risk
 - writing defensibility and logical flow
+- manuscript language hygiene and provenance leakage
 - figure / table usefulness
+- citation sufficiency: count verified references and compare coverage against nearby strong papers
+- full-paper style, pacing, and section-level argument quality relative to strong accepted papers
+- experiment package completeness relative to nearby high-level papers
 - submission readiness
 
 ## Workflow
@@ -132,10 +188,15 @@ Before writing the review itself, make the audit explicit.
 
 Identify:
 
+- whether the package is only `draft_checkpoint`, a reviewable manuscript, or truly submission-ready
 - 1 to 3 core claims such as `C1`, `C2`, `C3`
 - the strongest current evidence
 - the weakest current evidence
 - the top 3 likely rejection reasons
+- the ready experiment/analysis group count versus the current target, including any user-specified target such as 4-8 analyses
+- what the closest high-quality paper comparators are and which parts of their writing / logic / experiment package should be used as review standards
+- whether manuscript text leaks user requirements, agent actions, route/control state, branch/worktree names, tool recommendations, TODOs, or raw implementation shorthand
+- whether `artifact.validate_academic_outline(detail='full')` and `artifact.validate_manuscript_language(detail='full')` pass when a selected outline or draft exists
 - whether the likely next route is:
   - text revision
   - literature / novelty audit
@@ -143,15 +204,33 @@ Identify:
   - supplementary experiment
   - claim downgrade
 
-### 2. Check novelty and positioning only when needed
+### 2. Run a paper-quality literature benchmark
 
-If novelty, related-work coverage, or field positioning is unclear:
+Before calling a substantial manuscript strong, run a current literature/style benchmark unless the review scope is explicitly local-only.
 
-1. open `scout`
-2. run a focused literature / comparison audit
-3. record what is genuinely overlapping, what remains novel, and what is merely better positioned writing
+Search for nearby high-quality papers using the best available current source:
 
-Do not request new experiments just to answer a literature-positioning question.
+- Prefer DeepXiv or OpenAlex when configured.
+- Use web search when needed for venue status, accepted-paper pages, PDFs, proceedings, journal pages, or recent related work.
+- Include high-level exemplars when relevant: ICLR, ICML, NeurIPS, CVPR, ACL, EMNLP, KDD, SIGIR, AAAI, IJCAI, Nature/Science family, or Q1 journal papers in the field.
+- Prefer accepted papers, proceedings pages, journal pages, arXiv PDFs with accepted-version metadata, and official conference / journal records over blog posts or low-trust summaries.
+
+Build a compact comparison set, usually 3-8 papers:
+
+- 1-3 closest technical neighbors
+- 1-3 strong writing / story exemplars from top venues
+- 1-2 experiment-package exemplars when the manuscript's evaluation design is uncertain
+
+For each comparator, record title, venue/source, why it is relevant, and what it teaches about:
+
+- abstract and introduction framing
+- problem -> gap -> method -> evidence logic
+- method explanation and reader onboarding
+- experiment design, ablations, robustness, baselines, and limitations
+- figure/table roles and result narrative
+- related-work positioning and claim boundaries
+
+Do not request new experiments just to answer a literature-positioning question. First decide whether the fix is writing, positioning, claim narrowing, or genuinely missing evidence.
 
 ### 3. Write a reliable review report
 
@@ -165,6 +244,8 @@ The review should be:
 - reader-aware
 - evidence-grounded
 
+Never write "no weaknesses", "no key issues", or equivalent closure language unless the review has first listed the main rejection routes and shown why each is fixed, downgraded, or out of scope.
+
 At minimum, the review report should cover:
 
 - summary
@@ -175,7 +256,9 @@ At minimum, the review report should cover:
 - storyline / outline advice
 - priority revision plan
 - experiment inventory and research experiment plan
+- analysis-count sufficiency and whether the count is real rather than inflated by duplicate/stale rows
 - novelty verification and related-work matrix
+- high-level paper comparison matrix covering writing, logic, full-paper organization, style, figures/tables, and experiment package gaps
 - references
 
 If helpful, include an internal conservative overall judgment or score, but do not pretend numerical precision when evidence is still unstable.
@@ -198,6 +281,16 @@ For each serious issue, record:
 
 Only if more evidence is truly needed, write `paper/review/experiment_todo.md` using `references/experiment-todo-template.md`.
 
+Before creating TODOs, separate three blocker types:
+
+- analysis blockers: missing, failed, unmapped, or legacy-method evidence needed for a claim
+- manuscript blockers: missing sections, missing figures/tables, weak narrative, or stale claim-evidence mapping
+- language/provenance blockers: user/operator/agent wording, restart or route-management phrasing, tool-promotion captions, TODOs, or raw endpoint/batch shorthand in paper-facing text
+- submission blockers: missing PDF, checklist, artifact availability, camera-ready cleanup, or user approval
+
+Do not turn manuscript/submission blockers into fake experiments or mark a `draft_checkpoint_ready` package final.
+Do not accept a draft as reviewable while it still reads like a run log, control memo, or user-request transcript.
+
 When the paper still lacks experimental support, also create or revise:
 
 - `paper/paper_experiment_matrix.md`
@@ -237,6 +330,7 @@ When building or revising that matrix, explicitly consider:
 Do not assume the paper only needs “analysis experiments”.
 Do not assume case studies belong in the required set.
 If efficiency or cost could become a reviewer-facing strength or concern, put that into the matrix explicitly.
+If local serving, batch size, ports, cache reuse, or command shape are only reproducibility details, route them to appendix/protocol rows rather than main-text claims.
 
 For the matrix, each row should usually record:
 
@@ -266,6 +360,8 @@ Before treating the experiments section as stable, require that every currently
 - excluded with a real reason
 - or blocked with a real reason
 
+Before treating the manuscript prose as stable, scan titles, abstract, captions, method setup, conclusion, and section openings for process language. Rewrite or block if they mention user requests, agent decisions, worktrees, branches, artifact ids, "paper restart", "paper should", "remaining work", tool recommendations, or shorthand such as `64 + 64`.
+
 When extra evidence is truly needed, use the shared supplementary-experiment protocol:
 
 - recover ids / refs first if needed
@@ -406,3 +502,5 @@ Useful tags include:
 
 The goal is not to sound severe.
 The goal is to make the next revision step technically clear and evidence-bound.
+
+A good review pass leaves fewer plausible next moves, not more.