@jterrats/open-orchestra 1.0.2 → 1.0.3

package/docs/reports/ac-evidence-backfill-20260517.md

@@ -0,0 +1,256 @@
+# Reconciled AC Evidence Backfill
+
+Date: 2026-05-17
+Task: AC-EVIDENCE-BACKFILL-20260517
+
+Scope: first evidence-generation batch for reconciled current acceptance criteria only.
+Superseded and duplicate historical criteria from
+`docs/reports/ac-evolution-reconciliation-20260517.md` are intentionally excluded.
+
+## Evidence Standard
+
+Each evidence item must include:
+
+- acceptance criterion or domain covered
+- command or artifact used
+- observed result
+- happy path coverage
+- edge case coverage
+- known failures or unresolved gaps
+
+## Batch 1 Result
+
+| Area | Command | Result | Evidence value |
+| --- | --- | --- | --- |
+| Docs adoption | `node --test test/docs-adoption.test.js` | Fail: 2 pass, 1 fail | Confirms current docs/site generated-content drift. |
+| Tracker sync | `node --test test/github.test.js` | Pass: 18 pass | Covers GitHub, MCP fallback, unsupported tracker, conflicts, close guard. |
+| Release readiness | `node --test test/release-check.test.js test/release-candidate.test.js` | Pass: 20 pass | Covers ready, blocked, accepted risk, rollback-required, dirty tree, version ahead/behind, private package state. |
+| QA coverage and CLI evidence | `node --test test/cli-output-evidence.test.js test/qa-coverage.test.js` | Pass: 7 pass | Covers stdout/stderr/exit code mapping, skipped criteria, missing assertions, unresolved QA gaps. |
+| Workflow governance | `node --test test/autonomous-workflow-cli.test.js test/autonomous-workflow-sizing.test.js test/autonomous-workflow-phase-status.test.js` | Pass: 54 pass | Covers gates, skips, invalid inputs, timeout, block/changes return, provider failure, budget abort, QA e2e generation. |
+| Runtime, skills, memory | `node --test test/skills-memory-domain.test.js test/delegation-decision.test.js test/runtime-adapters.test.js` | Pass: 47 pass | Covers role-only non-activation, runtime brief without API execution, provider setup docs, delegation guardrails, memory compaction. |
+| Diagram and MCP visual validation | `node --test test/visual-validation.test.js test/diagram-mcp.test.js` | Pass: 15 pass | Covers Mermaid lint success/failure, missing tool guidance, visual defect detection, TLS/no-secret OAuth behavior. |
+| Core CLI/API contracts | `node --test test/orchestra.test.js` | Pass: 153 pass | Covers init, target dir, task metadata, QA gates, risk gates, evidence, Playwright plan/evidence, policy, budget, validation, docs audit. |
+| Browser E2E | `npm run test:e2e` | Fail: 11 pass, 4 fail | Produces screenshots and traces for current web/site failures. |
+
+## Batch 2 Result
+
+| Area | Command | Result | Evidence value |
+| --- | --- | --- | --- |
+| Generated guidance and refresh | `node --test test/instruction-updates.test.js test/runtime-bootstrap.test.js test/health-checks.test.js` | Pass: 29 pass | Covers unchanged, drifted, blocked, forced, missing, unsupported targets, managed block preservation, dry-run, force, and target filtering. |
+| Estimation and benchmark | `node --test test/benchmark-*.test.js test/performance-benchmark.test.js` | Pass: 40 pass | Covers automatic/manual estimates, missing history, calibration, burndown, dashboard output, persistent bias lessons, and performance budgets. |
+| Phase playbooks, project detection, release matrix | `node --test test/phase-playbooks.test.js test/project-detection.test.js test/release-test-matrix.test.js` | Pass: 21 pass | Covers QA/docs/release/UX playbooks, deployment risk challenge, redo/sign-off observation protocol, config overrides, scaffolding, project profile detection, and release matrix commands. |
+| Web APIs, QA readiness, automation evidence | `node --test test/web-*.test.js test/automation-evidence.test.js test/qa-readiness.test.js` | Pass: 22 pass | Covers observable assertion requirements, deferred external validation, QA plan ready gate, web write authorization, artifact canvas, runtime sessions, provider profiles, recovery repair, and workflow progress summaries. |
+
+## Edge Cases Covered
+
+### Tracker Sync
+
+- CLI transport available.
+- CLI missing with MCP-skill fallback.
+- Unsupported tracker.
+- Failed fallback.
+- Existing issue/task conflict.
+- Issue close blocked without readiness or accepted risk.
+- Issue close allowed when release readiness passes.
+
+### Release Readiness
+
+- Ready release path.
+- Accepted-risk path.
+- Rollback-required path.
+- Dirty working tree.
+- Version ahead of latest tag.
+- Version behind latest tag.
+- Private package state in npm pack output.
+- Noisy command output.
+- Nested web server permission failure guidance.
+
+### QA Evidence
+
+- CLI stdout, stderr, and exit code are captured.
+- Evidence must map to acceptance criteria.
+- Not-applicable criteria are skipped explicitly.
+- Missing observable assertions are surfaced.
+- Release readiness exposes unresolved automation gaps.
+
+### Workflow Governance
+
+- `gates=none`, `gates=phase`, and `gates=all`.
+- Skipped phases.
+- Invalid skipped phases.
+- Gate approve idempotency.
+- Gate block/changes returns to responsible phase.
+- Timeout marks active phase failed.
+- Provider failure diagnostics.
+- Budget exceeded before provider execution.
+- Placeholder provider output is rejected.
+- QA phase generates browser/shell E2E artifacts or skipped evidence.
+
+### Runtime, Skills, Memory
+
+- Runtime delegation falls back to brief-only when subagents are unsupported.
+- Runtime packet renders without provider API execution.
+- Role eligibility alone does not activate skills.
+- Runtime task signals activate QA evidence and collection standards.
+- Memory hooks compact long lessons and expose trimming rationale.
+- Runtime handoff rejects non-workflow-local artifacts.
+
+### Diagram, Visual, MCP
+
+- Mermaid lint success and failure.
+- Missing diagram tool remediation.
+- Visual validator detects common diagram defects.
+- Visual validation blocks missing validation artifacts.
+- MCP OAuth requires TLS and opt-in secure storage.
+- OAuth refresh does not expose secrets.
+- Bearer injection is limited to HTTPS MCP calls.
+
+## Current Failures Captured
+
+### Docs Adoption Drift
+
+Command:
+
+```sh
+node --test test/docs-adoption.test.js
+```
+
+Failure:
+
+- `test/docs-adoption.test.js:51`
+- Expected `/First visible value/i` in `site/src/App.jsx`.
+- Actual site copy is now generated from `site/src/generated-site-content.js` and
+  `docs/site-manifest.json`.
+
+Classification:
+
+- test drift caused by dynamic site content, or missing assertion against generated content.
+
+### Project Site Architecture Viewer
+
+Command:
+
+```sh
+npm run test:e2e
+```
+
+Failures:
+
+- `e2e/project-site.spec.js:61`
+- `e2e/project-site.spec.js:187`
+
+Observed:
+
+- Timeout waiting for `[data-architecture-diagram] svg`.
+
+Artifacts:
+
+- `test-results/project-site-project-site--5d3bd-nd-interactive-on-localhost-chromium/test-failed-1.png`
+- `test-results/project-site-project-site--5d3bd-nd-interactive-on-localhost-chromium/trace.zip`
+- `test-results/project-site-project-site--69905-ontent-fits-mobile-viewport-chromium/test-failed-1.png`
+- `test-results/project-site-project-site--69905-ontent-fits-mobile-viewport-chromium/trace.zip`
+
+Classification:
+
+- current browser-visible failure. The architecture diagram container exists, but the SVG never becomes visible.
+
+### Web Ops Task Detail
+
+Command:
+
+```sh
+npm run test:e2e
+```
+
+Failure:
+
+- `e2e/web-console.spec.js:487`
+
+Observed:
+
+- Timeout waiting for `Release readiness passed`.
+
+Artifacts:
+
+- `test-results/web-console-WEB-OPS-accept-d24db-k-detail-and-recovery-state-chromium/test-failed-1.png`
+- `test-results/web-console-WEB-OPS-accept-d24db-k-detail-and-recovery-state-chromium/trace.zip`
+
+Classification:
+
+- UI copy/state drift or regression in task detail release readiness rendering.
+
+### Web Lifecycle Gate Controls
+
+Command:
+
+```sh
+npm run test:e2e
+```
+
+Failure:
+
+- `e2e/web-console.spec.js:636`
+
+Observed:
+
+- Timeout waiting for button `Block Gate`.
+
+Artifacts:
+
+- `test-results/web-console-WEB-LIFECYCLE--fdcf8-s-workflow-from-the-browser-chromium/test-failed-1.png`
+- `test-results/web-console-WEB-LIFECYCLE--fdcf8-s-workflow-from-the-browser-chromium/trace.zip`
+
+Classification:
+
+- current browser-visible failure in gate decision controls or expected button label/state.
+
+## Evidence Backfill Status
+
+This batch generated usable evidence for these reconciled domains:
+
+- tracker sync
+- release readiness
+- QA coverage and CLI output evidence
+- workflow governance
+- runtime/skills/memory
+- diagram/visual/MCP controls
+- core CLI/API contracts
+- generated guidance refresh
+- estimation and benchmark calibration
+- phase playbooks and redo/sign-off protocol
+- project detection and release test matrix
+- web API security, recovery, provider profiles, and workflow progress
+
+This batch produced failing evidence for:
+
+- docs adoption generated-site drift
+- site architecture diagram render
+- web ops release readiness copy/state
+- web lifecycle gate block control
+
+## Final Batch Disposition
+
+The reconciled evidence candidates are now split into three buckets:
+
+| Bucket | Count | Disposition |
+| --- | ---: | --- |
+| Demonstrated by passing command evidence | 48 | Covered by the batch 1 and batch 2 commands above. |
+| Demonstrated by failing evidence | 4 | Evidence exists, but proves current non-compliance. Bugfix stories should be created. |
+| Requires manual/product acceptance or doc review linkage | 4 | Evidence is mostly present as docs/contract tests, but should be explicitly accepted or linked before release. |
+
+The four demonstrated failures are:
+
+1. `GH-296` / docs adoption: generated site content moved visible-value copy out of `site/src/App.jsx`, but `test/docs-adoption.test.js` still asserts literal content in that file.
+2. `ARCH-SALESFORCE-INTEGRATION-DIAGRAM` / project site: architecture viewer does not render `[data-architecture-diagram] svg` in desktop or mobile Playwright runs.
+3. `GH-298` / web ops: task detail no longer renders `Release readiness passed` where the E2E expects it.
+4. `WEB-LIFECYCLE` / workflow UI: `Block Gate` control is unavailable after the request-changes path.
+
+The four manual/product acceptance candidates are:
+
+1. `DEV-STANDARDS-GENERALIZE`: confirm stack-agnostic standards examples remain current and intentionally contain no cross-project references.
+2. `SETUP-IMPORT-ROLE-ALIASES`: confirm PM/PO alias behavior is still a product-supported import compatibility contract.
+3. `RUN-VS-WORKFLOW`: confirm public copy around `run` versus `workflow run` remains accepted after site/docs restructuring.
+4. `ARCH-SALESFORCE-INTEGRATION-DIAGRAM`: after the render bug is fixed, BA/Architect should visually accept systems, connectors, boundaries, and directionality.
+
+The next remediation batch should create bugfix stories for the four current failures,
+then rerun `npm test` and `npm run test:e2e`.