@chllming/wave-orchestration 0.5.3 → 0.6.0

package/docs/concepts/what-is-a-wave.md

@@ -0,0 +1,183 @@
+# What Is A Wave?
+
+A wave is the main planning and execution unit in Wave Orchestration.
+
+It is not just a prompt file. A wave is a bounded slice of repository work with:
+
+- explicit scope
+- named owners
+- runtime and context requirements
+- proof and closure rules
+- durable coordination state
+- replayable execution artifacts
+
+## Core Terms
+
+- Lane
+  An ordered sequence of waves. The default lane in this repo is `main`.
+- Wave
+  One numbered work package inside a lane, usually stored as `docs/plans/waves/wave-<n>.md`.
+- Agent
+  One role inside the wave, such as implementation, `cont-EVAL`, security review, integration, documentation, cont-QA, infra, or deploy.
+- Attempt
+  One execution pass of a wave. A wave can have multiple attempts due to retries or fallback.
+- Closure
+  The final proof pass that decides whether the wave is actually done, not just partially implemented.
+
+## Why Waves Exist
+
+Waves force a higher planning bar than ad hoc prompts. A good wave answers:
+
+- What is changing now, and why now?
+- Which components or docs are in scope?
+- Which agent owns each slice?
+- What evidence closes the wave?
+- Which dependencies, helper requests, or escalations can still block completion?
+
+## Wave Anatomy
+
+Wave markdown is the authored execution surface today. A typical wave can include:
+
+- title and commit message
+- project profile details such as oversight mode and lane
+- sequencing note
+- reference rule
+- deploy environments
+- component promotions
+- eval targets
+- Context7 defaults
+- one `## Agent ...` block per role
+
+Inside each agent block, the important sections are:
+
+- `### Role prompts`
+  Standing role identity imported from `docs/agents/*.md`.
+- `### Executor`
+  Runtime selection, profile, model, fallbacks, and budgets.
+- `## Eval targets`
+  Optional wave-level contract for `cont-EVAL`, including benchmark family or pinned benchmarks, objective, and stop condition.
+  See [docs/evals/README.md](../evals/README.md) for guidance on delegated versus pinned targets and the coordination benchmark families.
+- `### Proof artifacts`
+  Optional machine-visible local evidence required for proof-centric waves, especially `pilot-live` and above.
+- `### Context7`
+  External library truth to prefetch and inject.
+- `### Skills`
+  Reusable repo-owned environment or workflow guidance resolved after runtime selection.
+- `### Components`
+  The components that agent is responsible for proving or promoting.
+- `### Capabilities`
+  Optional routing hints for follow-up work.
+- `### Deliverables`
+  Exact repo-relative outputs that must exist before closure can pass.
+- `### Prompt`
+  The specific task, file ownership, requirements, and validation instructions.
+- `### Exit contract`
+  The completion, durability, proof, and documentation expectations that gate closure.
+
+## Standard Roles
+
+The starter runtime expects three standard closure roles plus up to two optional review specialists:
+
+- `A8`
+  Integration steward
+- `A9`
+  Documentation steward
+- `A0`
+  cont-QA
+- `E0`
+  Optional `cont-EVAL` for iterative benchmark or output tuning; report-only by default, implementation-owning only when explicitly assigned non-report files
+- `A7`
+  Optional security reviewer; report-only by default and used to publish a threat-model-first security review before integration closure
+
+Implementation or specialist agents own the actual work slices. Closure roles do not replace implementation ownership; they decide whether the combined result is closure-ready. `cont-EVAL` is the one hybrid role: most waves keep it report-only, but human-authored waves may assign explicit tuning files to `E0`, in which case it must satisfy both implementation proof and eval proof.
+
+## Lifecycle Of A Wave
+
+1. Author or draft the wave.
+2. Run `wave launch --dry-run --no-dashboard`.
+3. The launcher validates the wave, resolves executors and skills, builds prompts, and materializes operator surfaces.
+4. A live run launches implementation agents first when implementation work remains.
+5. Agents write structured coordination events instead of relying on ad hoc terminal output.
+6. The launcher checks implementation contracts, promoted-component proof, helper assignments, dependencies, and clarification state.
+7. If implementation is ready, closure runs in order: optional `cont-EVAL`, optional security review, integration, documentation, then cont-QA.
+8. The attempt is captured in per-wave traces, ledgers, inboxes, summaries, and copied artifacts.
+
+## Runtime And Operating Posture
+
+Wave is runtime agnostic at the orchestration layer.
+
+Planning, ownership, closure, durable state, and traces do not depend on whether an agent runs on Codex, Claude Code, OpenCode, or the local smoke executor. Runtime-specific behavior is isolated to executor adapters and overlays.
+
+That means a wave should usually be authored in runtime-neutral terms:
+
+- ownership and deliverables
+- proof and validation
+- closure order
+- dependencies and helper flow
+- promoted component expectations
+
+The runtime choice resolves later, from the agent executor block, profile defaults, lane defaults, CLI overrides, and fallback policy.
+
+Wave also has an execution posture:
+
+- `oversight`
+  Human review or intervention is expected for risky or ambiguous work.
+- `dark-factory`
+  The wave is authored for routine execution without normal human intervention.
+
+Today these postures are planning vocabulary and saved project defaults, not two separate execution engines. Human feedback is still an escalation mechanism inside the orchestration loop, not the definition of the operating mode itself.
+
+If you need the narrower supporting pages, see [runtime-agnostic-orchestration.md](./runtime-agnostic-orchestration.md) and [operating-modes.md](./operating-modes.md).
+
+Current live waves are strict about closure artifacts:
+
+- `cont-EVAL` must emit a structured `[wave-eval]` marker whose `target_ids` matches the declared eval targets and whose `benchmark_ids` enumerates the executed benchmark set.
+- Security reviewers must leave a security review report and emit a final `[wave-security]` marker with `state=<clear|concerns|blocked>`, finding count, and approval count.
+- `cont-QA` must emit both a final `Verdict:` line and a final `[wave-gate]` marker.
+- Replay keeps read-only compatibility with older traces and older evaluator-era artifacts, but live waves do not pass on verdict-only or underspecified closure markers.
+
+## What Makes A Wave "Done"
+
+A wave is not done because an agent said so. It is done only when the runtime surfaces agree:
+
+- implementation exit contracts pass
+- required deliverables exist and stay within ownership boundaries
+- required proof artifacts exist when the wave declares proof-first live evidence
+- required component proof and promotions pass
+- helper assignments are resolved
+- required dependency tickets are resolved
+- clarification follow-ups or escalations are resolved
+- if present, `cont-EVAL` satisfies its declared eval targets
+- if present, the security reviewer publishes a report plus a final `[wave-security]` marker; `blocked` stops closure while `concerns` stays advisory
+- integration recommends closure
+- documentation and cont-QA closure pass
+
+For proof-first live-wave examples, see [docs/reference/live-proof-waves.md](../reference/live-proof-waves.md).
+
+## Where The State Lives
+
+The wave file is only part of the story. The runtime writes durable state under `.tmp/<lane>-wave-launcher/`, including:
+
+- prompts and logs
+- status summaries
+- coordination logs
+- rendered message boards
+- compiled inboxes
+- ledger and docs queue
+- security summaries
+- integration summaries
+- dependency snapshots
+- executor overlays
+- trace bundles
+
+That is why a wave is better understood as a bounded execution record, not just a markdown file.
+
+## Planner Specs vs Markdown
+
+The planner foundation adds a JSON draft spec at `docs/plans/waves/specs/wave-<n>.json`.
+
+- The JSON spec is the canonical planner artifact.
+- The rendered markdown stays compatible with the launcher and parser.
+- The launcher still executes the markdown wave file today.
+
+This split keeps authoring structured while preserving the established execution surface.

package/docs/evals/README.md

@@ -0,0 +1,166 @@
+---
+title: "Benchmark Catalog Guide"
+summary: "How to use delegated benchmark families, pinned benchmarks, and coordination-oriented eval targets in Wave."
+---
+
+# Benchmark Catalog Guide
+
+Wave's benchmark catalog lives in `docs/evals/benchmark-catalog.json`.
+
+It has two jobs:
+
+- give `cont-EVAL` a repo-governed menu of allowed benchmark families and benchmark ids
+- document what each benchmark is trying to catch, including coordination failure modes and static paper baselines
+
+The catalog is reference metadata, not a run-history database. It tells the wave author and `cont-EVAL` what kinds of checks are allowed and what external benchmark or paper baseline those checks map to.
+
+For a full authored wave example that uses these patterns, see [docs/reference/sample-waves.md](../reference/sample-waves.md).
+
+## Migrating From Legacy Evaluator Waves
+
+If your `0.5.4`-era repo still talks about a single `evaluator` role, split that surface before adopting `0.6.0`:
+
+- keep `A0` as `cont-QA` for the final closure verdict and `[wave-gate]`
+- add `E0` only when the wave needs benchmark-driven tuning or service-output evaluation
+- treat `cont-EVAL` as report-only unless the wave explicitly gives `E0` owned implementation files
+- keep `## Eval targets` at the wave level so `cont-EVAL` has an exact contract to satisfy
+
+`cont-EVAL` is not a rename of `cont-QA`. In `0.6.0`, `E0` proves the eval contract before integration, while `A0` still owns the final release verdict after documentation closure.
+
+## When To Use Delegated Vs Pinned Targets
+
+Use `selection: delegated` when the wave should authorize a benchmark family and let `cont-EVAL` choose the exact benchmark set inside that family.
+
+Use `selection: pinned` when the wave must require specific benchmark ids and does not want `cont-EVAL` to choose alternates.
+
+In practice:
+
+- `delegated` is better when you want flexibility inside a stable area such as `hidden-profile-pooling` or `latency`
+- `pinned` is better when you need an exact smoke gate such as `cold-start-smoke` or `private-evidence-integration`
+
+## Example Eval Targets
+
+Delegated family target:
+
+```md
+## Eval targets
+
+- id: coordination-pooling | selection: delegated | benchmark-family: hidden-profile-pooling | objective: Pool distributed private evidence before closure | threshold: Critical decision-changing facts appear in the final integrated answer before PASS
+```
+
+Pinned benchmark target:
+
+```md
+## Eval targets
+
+- id: contradiction-recovery-guard | selection: pinned | benchmarks: claim-conflict-detection,evidence-based-repair | objective: Detect and repair conflicting claims before closure | threshold: Material contradictions become explicit follow-up work and resolve before final pass
+```
+
+Mixed target set:
+
+```md
+## Eval targets
+
+- id: coordination-pooling | selection: delegated | benchmark-family: hidden-profile-pooling | objective: Pool distributed private evidence before closure | threshold: Critical decision-changing facts appear in the final integrated answer before PASS
+- id: summary-integrity | selection: pinned | benchmarks: shared-summary-fact-retention | objective: Preserve decision-changing facts through summary compression | threshold: Shared summaries retain the facts needed for the final recommendation
+```
+
+## Coordination Families
+
+The coordination-oriented families currently included in the catalog are:
+
+- `hidden-profile-pooling`
+  Use when the main risk is that agents fail to surface or integrate distributed private evidence. This maps most directly to HiddenBench.
+- `silo-escape`
+  Use when the risk is that agents communicate but still fail to reconstruct the correct global state. This maps most directly to Silo-Bench.
+- `simultaneous-coordination`
+  Use when the risk is contention, lockstep failure, or convergent reasoning under concurrent decisions. This maps most directly to DPBench.
+- `expertise-leverage`
+  Use when the risk is expert underuse, bad routing, or low-quality compromise across mixed-skill agents. This maps most directly to `Multi-Agent Teams Hold Experts Back`.
+- `blackboard-fidelity`
+  Use when the risk is information loss or distortion between the raw coordination log and derived artifacts like shared summaries, inboxes, ledger state, or integration summaries.
+- `contradiction-recovery`
+  Use when the risk is false consensus, unresolved conflicting claims, or clarification chains that appear resolved without real repair.
+
+## How To Choose The Right Family
+
+Choose the family based on the failure you are most worried about, not just on the surface area being changed.
+
+Use:
+
+- `hidden-profile-pooling` when the hard part is discovering missing facts
+- `silo-escape` when the hard part is integrating already-shared facts into one correct state
+- `simultaneous-coordination` when multiple owners or resources must move together
+- `expertise-leverage` when the right answer depends on preserving the best expert signal
+- `blackboard-fidelity` when summaries, inboxes, or integration artifacts may be dropping important evidence
+- `contradiction-recovery` when you expect conflicting claims and need the framework to turn them into bounded repair work
+
+## How `cont-EVAL` Should Use The Catalog
+
+When a wave delegates benchmark selection:
+
+1. Read the wave's `## Eval targets`.
+2. Resolve the allowed benchmark family from the catalog.
+3. Choose the smallest benchmark set that genuinely tests the target's failure mode.
+4. Record the exact selected benchmark ids in the `cont-EVAL` report.
+5. Emit the final `[wave-eval]` marker with the exact executed `benchmark_ids`.
+
+When a wave pins benchmarks:
+
+1. Run the named benchmark ids directly.
+2. Do not silently swap to nearby checks.
+3. Treat missing or unrun pinned benchmarks as an unsatisfied target.
+
+## How To Read The Static Baselines
+
+Some coordination families now include static paper baselines such as HiddenBench, Silo-Bench, DPBench, and `Multi-Agent Teams Hold Experts Back`.
+
+These baselines are:
+
+- reference points from papers
+- useful for framing whether Wave is still far from the broader state of the art
+- not the same thing as local run history
+
+They should answer:
+
+- what failure mode this benchmark family is grounded in
+- what the paper reported
+- what metric the paper used
+
+They should not be treated as:
+
+- a promise that Wave already matches the paper's best system
+- a local regression history
+- a substitute for actually running evals
+
+## Authoring Guidance
+
+Prefer one eval target per distinct risk.
+
+Good:
+
+- one target for distributed-information pooling
+- one target for contradiction recovery
+- one target for latency guardrails
+
+Avoid:
+
+- one overloaded target that mixes every coordination risk into a single vague threshold
+
+Prefer delegated targets early when the family is stable but the exact check should remain flexible.
+
+Prefer pinned targets when:
+
+- the wave is release-sensitive
+- the benchmark is small and repeatable
+- you need a precise regression gate
+
+## Current Limits
+
+The benchmark catalog does not yet store:
+
+- local benchmark run history
+- local-vs-paper delta computation
+- automated benchmark execution plans
+
+For now it is the schema and policy layer that keeps eval authoring, `cont-EVAL`, and coordination benchmarking aligned.