@jterrats/open-orchestra 0.1.0

package/docs/multi-agent-orchestrator-backlog.md

@@ -0,0 +1,445 @@
+# Multi-Agent Orchestrator Backlog
+
+## Product Goal
+
+Build a provider-agnostic middleware that lets a parent agent orchestrate specialized subagents through explicit roles, task graphs, handoffs, evidence, review gates, and shared artifacts even when the underlying LLM platform does not support native agent-to-agent collaboration.
+
+## Epic 1: Central Orchestration
+
+### Story ORCH-001: Create Structured Task Brief
+As a user, I want the parent agent to convert my request into a structured brief so that goals, scope, risks, assumptions, and Definition of Done are clear before work starts.
+
+Acceptance criteria:
+- Captures goal, backlog item, user value, scope, non-goals, assumptions, risks, constraints, and Definition of Done.
+- Identifies missing readiness information.
+- Blocks implementation when critical readiness fields are missing.
+
+### Story ORCH-002: Select Required Roles
+As a parent agent, I want to select only the roles required for the task so that work is complete without activating unnecessary agents.
+
+Acceptance criteria:
+- Maps task impact areas to roles.
+- Records selected roles and skipped roles with rationale.
+- Supports role combinations for cross-cutting work.
+
+### Story ORCH-003: Build Task Graph
+As a parent agent, I want to create a task graph with dependencies so that sequential and parallel work can be coordinated safely.
+
+Acceptance criteria:
+- Creates tasks with ID, owner role, status, dependencies, inputs, outputs, and exit criteria.
+- Detects blocked tasks.
+- Prevents downstream tasks from starting before required upstream outputs exist.
+
+### Story ORCH-004: Require User Alignment Gate
+As a user, I want to approve the proposed solution and architecture before implementation so that scope and design stay under control.
+
+Acceptance criteria:
+- Presents proposed approach, modules, data flow, trade-offs, risks, test strategy, and evidence plan.
+- Records user approval or requested changes.
+- Blocks non-trivial implementation until approval is recorded.
+
+### Story ORCH-005: Resume Interrupted Work
+As a parent agent, I want to rebuild workflow state from saved artifacts and events so that interrupted work can continue without starting over.
+
+Acceptance criteria:
+- Loads tasks, events, locks, decisions, handoffs, and evidence.
+- Reconstructs current status and next actions.
+- Flags inconsistent or missing workflow files.
+
+## Epic 2: Role Registry and Capabilities
+
+### Story ROLE-001: Define Role Catalog
+As an administrator, I want to define roles and responsibilities so that each agent knows its ownership, authority, and handoff expectations.
+
+Acceptance criteria:
+- Supports roles such as Product Owner, Architect, Developer, QA, Security, DevOps, SRE, DBA, UX/UI Designer, Release Manager, Compliance/Privacy, Technical Writer, and Game Designer.
+- Stores role purpose, responsibilities, required inputs, outputs, and blockers.
+- Allows project-specific role overrides.
+
+### Story ROLE-002: Map Capabilities to Roles
+As a parent agent, I want to query role capabilities so that tasks are assigned to the right agent.
+
+Acceptance criteria:
+- Supports capabilities such as coding, test planning, Playwright automation, threat review, infra review, docs, and release review.
+- Resolves the best role for each task impact area.
+- Reports when no configured role can own a needed capability.
+
+### Story ROLE-003: Enforce Role Authority
+As a reviewer, I want each role to have explicit approval and blocking authority so that release decisions are auditable.
+
+Acceptance criteria:
+- Supports approve, block, request changes, and risk accepted states.
+- Records role, rationale, severity, and affected artifact.
+- Prevents release when required role approval is missing.
+
+## Epic 3: Multi-Model Provider Layer
+
+### Story MODEL-001: Define Provider Interface
+As a developer, I want a common provider interface so that the orchestrator can call different LLM providers without coupling to their SDKs.
+
+Acceptance criteria:
+- Defines request, response, streaming, tool support, vision support, JSON mode, context length, timeout, and error contract.
+- Hides provider-specific SDK details behind adapters.
+- Includes test doubles for provider behavior.
+
+### Story MODEL-002: Configure Models by Role
+As an administrator, I want to assign provider and model preferences per role so that each role can use the right cost, quality, and capability profile.
+
+Acceptance criteria:
+- Supports provider, model, fallbacks, max tokens, max cost, timeout, and retry settings per role.
+- Allows project-level defaults.
+- Validates unknown providers or unsupported capabilities.
+
+### Story MODEL-003: Record Model Provenance
+As an auditor, I want every agent output to record provider, model, version, and cost metadata so that decisions are traceable.
+
+Acceptance criteria:
+- Logs provider, model, request ID, token usage, latency, estimated cost, and timestamp.
+- Links model metadata to generated artifacts and decisions.
+- Redacts sensitive content according to retention policy.
+
+### Story MODEL-004: Support Provider Fallback
+As a parent agent, I want fallback providers for non-critical tasks so that workflow can continue when one provider fails.
+
+Acceptance criteria:
+- Uses configured fallback order.
+- Records fallback reason.
+- Blocks fallback for tasks marked reproducibility-sensitive unless user approval is recorded.
+
+### Story MODEL-005: Compare Model Outputs
+As a reviewer, I want to run multiple models on critical analysis tasks so that high-risk decisions can be cross-checked.
+
+Acceptance criteria:
+- Runs configured comparison models.
+- Produces a structured agreement/disagreement summary.
+- Requires human or parent-agent resolution for material disagreement.
+
+## Epic 4: Task Graph and State Management
+
+### Story TASK-001: Persist Task State
+As a parent agent, I want task state persisted to disk so that workflow status survives process restarts.
+
+Acceptance criteria:
+- Stores tasks in a structured file.
+- Supports statuses: pending, ready, in_progress, blocked, review, approved, rejected, done, canceled.
+- Validates task references and dependencies.
+
+### Story TASK-002: Show Workflow Status
+As a user, I want to see all tasks and blockers so that I understand progress.
+
+Acceptance criteria:
+- Shows tasks grouped by status and role.
+- Highlights blockers, missing evidence, failed gates, and active locks.
+- Provides next recommended action.
+
+### Story TASK-003: Detect Dependency Violations
+As a parent agent, I want to prevent tasks from running before dependencies are complete so that agents do not work from stale or missing inputs.
+
+Acceptance criteria:
+- Blocks task start when required dependencies are incomplete.
+- Explains which dependency is missing.
+- Allows explicit override only with recorded rationale.
+
+## Epic 5: Handoffs Between Agents
+
+### Story HAND-001: Generate Developer-to-QA Handoff
+As a Developer, I want a structured handoff to QA so that QA can test with full context.
+
+Acceptance criteria:
+- Includes changed files/components, behavior changed, unit tests, commands run, known gaps, risks, and recommended Playwright coverage.
+- Validates required fields before handoff is accepted.
+- Links evidence artifacts.
+
+### Story HAND-002: Generate Review Handoff
+As a parent agent, I want role-specific handoff templates so that Security, SRE, DBA, UX, and Release Manager receive relevant context.
+
+Acceptance criteria:
+- Selects template by target role.
+- Includes role-specific required fields.
+- Links decisions, task context, and evidence.
+
+### Story HAND-003: Reject Incomplete Handoffs
+As a reviewer, I want incomplete handoffs to be rejected automatically so that reviews do not start with missing context.
+
+Acceptance criteria:
+- Validates required fields.
+- Reports missing fields and owner.
+- Keeps target task blocked until corrected.
+
+## Epic 6: Shared Memory and Event Log
+
+### Story MEM-001: Append Workflow Events
+As a system, I want all workflow events stored in an append-only log so that collaboration is auditable.
+
+Acceptance criteria:
+- Records event type, task ID, actor, timestamp, summary, artifact links, and metadata.
+- Supports events such as TASK_ASSIGNED, HANDOFF_READY, REVIEW_BLOCKED, QA_FAILED, APPROVED, RISK_ACCEPTED, and RELEASE_READY.
+- Does not overwrite historical events.
+
+### Story MEM-002: Store Decision Log
+As an Architect, I want decisions stored separately from chat so that important trade-offs remain visible.
+
+Acceptance criteria:
+- Stores decision title, context, decision, alternatives, consequences, owner, and status.
+- Links decisions to tasks and reviews.
+- Supports superseded decisions.
+
+### Story MEM-003: Query Shared Context
+As an agent, I want to query current assumptions, decisions, risks, and evidence so that I do not work in isolation.
+
+Acceptance criteria:
+- Returns relevant task context by task ID.
+- Includes upstream handoffs, active decisions, blockers, and evidence.
+- Flags stale or superseded context.
+
+## Epic 7: Artifact Store and Evidence
+
+### Story ART-001: Store Test Evidence
+As QA, I want to store test plans, results, screenshots, logs, videos, and traces so that release decisions are evidence-based.
+
+Acceptance criteria:
+- Stores artifacts under predictable paths.
+- Links artifacts to task, role, and event.
+- Supports pass, fail, blocked, and deferred outcomes.
+
+### Story ART-002: Store Command Evidence
+As a Developer, I want commands and results captured so that reviewers know exactly what was verified.
+
+Acceptance criteria:
+- Records command, working directory, exit code, timestamp, and summarized output.
+- Marks failed commands as blockers unless explicitly deferred.
+- Links command evidence to handoff and final summary.
+
+### Story ART-003: Generate Final Evidence Summary
+As a user, I want a final evidence summary so that I can understand what was changed, tested, and left risky.
+
+Acceptance criteria:
+- Summarizes tasks, tests, reviews, approvals, blockers, risk acceptances, and artifacts.
+- Highlights unresolved risks and follow-up items.
+- Can be used as PR body input.
+
+## Epic 8: File Ownership and Locking
+
+### Story LOCK-001: Assign File Ownership
+As a parent agent, I want to assign files or modules to a task owner so that parallel agents do not overwrite each other.
+
+Acceptance criteria:
+- Records owner, task ID, path pattern, reason, and expiration.
+- Prevents another task from claiming overlapping paths without integration owner.
+- Supports module-level and file-level locks.
+
+### Story LOCK-002: Detect Lock Conflicts
+As an integrator, I want lock conflicts detected before edits so that merge conflicts are avoided.
+
+Acceptance criteria:
+- Detects overlapping path claims.
+- Reports owners and impacted tasks.
+- Requires integration owner assignment to continue.
+
+### Story LOCK-003: Release Locks
+As an agent, I want locks released when work completes or is canceled so that future tasks can proceed.
+
+Acceptance criteria:
+- Releases locks on task done or canceled.
+- Supports manual release with rationale.
+- Reports stale locks.
+
+## Epic 9: Quality Gates
+
+### Story GATE-001: Enforce Definition of Ready
+As a parent agent, I want readiness validated before implementation so that work does not start with hidden ambiguity.
+
+Acceptance criteria:
+- Checks backlog item, goal, scope, acceptance criteria, non-goals, risks, user alignment, roles, and test strategy.
+- Blocks implementation when required fields are missing.
+- Produces a readiness report.
+
+### Story GATE-002: Enforce Architecture Approval
+As an Architect, I want non-trivial implementation blocked until architecture is approved so that design risk is handled early.
+
+Acceptance criteria:
+- Detects non-trivial work.
+- Requires solution proposal and user approval.
+- Records approval event.
+
+### Story GATE-003: Enforce QA Gate
+As QA, I want release blocked without a test plan and execution evidence so that unverified changes do not ship.
+
+Acceptance criteria:
+- Requires QA plan, execution status, and evidence.
+- Supports deferred test cases only with rationale and follow-up owner.
+- Blocks release on unresolved critical QA findings.
+
+### Story GATE-004: Enforce Security/SRE/DBA Gates
+As a risk owner, I want required reviews enforced when a change touches my area so that material risk is not bypassed.
+
+Acceptance criteria:
+- Maps change impact to required reviewers.
+- Blocks release for missing or blocking reviews.
+- Supports explicit risk acceptance by authorized role.
+
+## Epic 10: CLI / Operator Interface
+
+### Story CLI-001: Initialize Workflow
+As a user, I want `orchestra init` to create workflow files so that a repo can opt into orchestrated agent work.
+
+Acceptance criteria:
+- Creates `.agent-workflow/` with tasks, roles, locks, events, decisions, handoffs, evidence, and reviews directories/files.
+- Does not overwrite existing files without confirmation.
+- Validates generated structure.
+
+### Story CLI-002: Show Status
+As a user, I want `orchestra status` to show workflow progress so that I can inspect current state quickly.
+
+Acceptance criteria:
+- Shows tasks, blockers, locks, pending reviews, missing evidence, and next action.
+- Returns non-zero when critical blockers exist if configured for CI.
+- Supports human-readable and JSON output.
+
+### Story CLI-003: Create Handoff
+As an agent, I want `orchestra handoff` to generate and validate a handoff so that downstream agents get complete context.
+
+Acceptance criteria:
+- Accepts from role, to role, task ID, and artifact links.
+- Generates role-specific template.
+- Validates required fields.
+
+### Story CLI-004: Record Review
+As a reviewer, I want `orchestra review` to approve, block, or request changes so that gate status is auditable.
+
+Acceptance criteria:
+- Records reviewer role, result, severity, findings, and artifact links.
+- Updates task and gate status.
+- Appends event log entry.
+
+### Story CLI-005: Add Evidence
+As an agent, I want `orchestra evidence add` to register evidence artifacts so that tests and reviews are traceable.
+
+Acceptance criteria:
+- Supports command, file, screenshot, trace, video, log, and report evidence types.
+- Links evidence to task and role.
+- Validates referenced files exist when applicable.
+
+## Epic 11: Final Integration
+
+### Story INT-001: Validate Ready for Final Response
+As an integrator, I want to verify all gates before final response so that the user receives a coherent and complete result.
+
+Acceptance criteria:
+- Checks tasks, blockers, locks, handoffs, reviews, evidence, and risks.
+- Reports anything incomplete.
+- Blocks final "done" status when required gates fail.
+
+### Story INT-002: Generate Final Summary
+As a user, I want a final summary with changes, tests, evidence, risks, and follow-ups so that I can make a release or review decision.
+
+Acceptance criteria:
+- Summarizes completed tasks by role.
+- Includes evidence links and commands run.
+- Lists unresolved risks and risk acceptances.
+
+### Story INT-003: Generate PR Body
+As a contributor, I want a PR body generated from workflow artifacts so that review context is complete.
+
+Acceptance criteria:
+- Produces summary, test plan, evidence, risks, screenshots, migrations, rollout, and rollback sections.
+- Links backlog item and relevant decisions.
+- Flags missing required PR sections.
+
+## Epic 12: QA Automation
+
+### Story QA-001: Identify Automation Candidates
+As QA, I want manual checks classified for automation so that critical flows become repeatable.
+
+Acceptance criteria:
+- Identifies smoke, regression, high-risk, and low-value manual checks.
+- Recommends Playwright for browser flows.
+- Records automation owner and follow-up item.
+
+### Story QA-002: Generate Playwright Test Plan
+As QA, I want Playwright automation planned from acceptance criteria so that E2E coverage targets user value.
+
+Acceptance criteria:
+- Defines scenarios, fixtures, page objects, selectors, assertions, and evidence settings.
+- Covers happy path and critical failure paths.
+- Avoids brittle selectors.
+
+### Story QA-003: Attach Playwright Evidence
+As QA, I want screenshots, videos, and traces linked to E2E runs so that failures are diagnosable.
+
+Acceptance criteria:
+- Stores Playwright artifacts by task and test run.
+- Links failed tests to traces/screenshots/videos.
+- Includes artifact links in QA handoff.
+
+## Epic 13: Configuration and Extensibility
+
+### Story CFG-001: Configure Project Workflow
+As an administrator, I want project-specific workflow configuration so that different repos can customize roles, gates, and templates.
+
+Acceptance criteria:
+- Supports config for roles, gates, templates, provider routing, evidence requirements, and severity thresholds.
+- Validates config schema.
+- Provides sensible defaults.
+
+### Story CFG-002: Add Custom Templates
+As a team, I want custom templates for handoffs, reviews, and evidence so that workflow matches local process.
+
+Acceptance criteria:
+- Loads templates from project config.
+- Falls back to default templates.
+- Validates required placeholders.
+
+### Story CFG-003: Support Provider-Agnostic Tools
+As a developer, I want tool availability configured separately from models so that agents can use shell, git, tests, Playwright, CI, and cloud tools safely.
+
+Acceptance criteria:
+- Defines tool name, permission level, allowed roles, and risk level.
+- Blocks tool use when role is unauthorized.
+- Logs tool use events.
+
+### Story CFG-004: Support Hybrid Tool Runtimes
+As a developer, I want Node, Playwright, and optional Python tool runtimes configured through explicit contracts so that the orchestrator can prioritize web automation while still supporting Python-specialized work.
+
+Acceptance criteria:
+- TypeScript/Node is the default runtime for the CLI and workflow engine.
+- Playwright is a first-class tool runtime for browser automation and evidence.
+- Python workers are opt-in and invoked only through configured command contracts.
+- Tool runtime config defines command, args, working directory, allowed roles, timeout, evidence behavior, and risk level.
+- Runtime configuration is validated before any tool is executed.
+
+### Story CFG-005: Enforce Static Analysis Git Hooks
+As a maintainer, I want version-controlled git hooks to run static analysis before commits so that low-quality or unsafe changes are blocked early.
+
+Acceptance criteria:
+- Pre-commit hook runs formatting check, lint, typecheck, secret scan, and staged-file validation.
+- Hook fails closed and blocks commit on failure.
+- `--no-verify` bypass is disallowed unless explicitly approved and recorded as a follow-up risk.
+- CI runs the same checks plus deeper project-wide SAST, dependency, container, IaC, and E2E checks when applicable.
+- Hook commands are configured through tool runtime contracts.
+
+## Epic 14: Governance, Privacy, and Cost
+
+### Story GOV-001: Enforce Retention Policy
+As Compliance/Privacy, I want prompt, response, and artifact retention controlled so that sensitive information is not stored unnecessarily.
+
+Acceptance criteria:
+- Supports retention periods by artifact type.
+- Supports redaction rules for secrets and PII.
+- Records deletion events.
+
+### Story GOV-002: Track Cost and Usage
+As an administrator, I want model and tool usage tracked so that agent workflows stay within budget.
+
+Acceptance criteria:
+- Tracks tokens, cost, provider, role, task, and duration.
+- Enforces max cost per task or workflow when configured.
+- Reports cost by role and provider.
+
+### Story GOV-003: Audit Risk Acceptance
+As an auditor, I want risk acceptances recorded with owner and rationale so that exceptions are traceable.
+
+Acceptance criteria:
+- Records risk, severity, owner, expiration, compensating controls, and approval.
+- Links risk acceptance to release decision.
+- Flags expired risk acceptances.