@thebackstoryis/engineering-with-ai 0.2.9

package/Docs/quality/manual-qa-and-acceptance.md

@@ -0,0 +1,162 @@
+# Manual QA and acceptance guide
+
+Use this guide to turn a technically completed delivery into evidence that an authorised person can inspect and accept.
+
+## Manual QA has a distinct job
+
+Automated tests, standards checks, and external review answer important questions about implementation. Manual QA asks whether the delivered outcome works for people and operations in the intended context.
+
+It remains a separate human gate after the Delivery phase. A green pipeline cannot approve it automatically.
+
+## Define the acceptance decision
+
+Before writing steps, state:
+
+- the outcome being accepted;
+- the user or operator journey;
+- the environment and version;
+- included and excluded scope;
+- known limitations and residual risks;
+- who may accept the result;
+- what happens after pass, failure, or partial acceptance.
+
+Do not ask a tester to “check everything”.
+
+## Prepare a reproducible walkthrough
+
+Include:
+
+1. prerequisites and access;
+2. safe test data and reset conditions;
+3. exact starting state;
+4. numbered actions in user language;
+5. expected observable outcomes;
+6. negative, boundary, and recovery cases;
+7. accessibility and supported-device checks where relevant;
+8. operational observation, logs, or alerts where relevant;
+9. cleanup and data-retention instructions;
+10. a place to record actual results and evidence.
+
+Separate setup commands from the behaviour the tester is judging.
+
+## Choose scenarios from impact
+
+Cover at least:
+
+- the primary happy path;
+- one important alternate or error path;
+- permissions and organisation boundaries where applicable;
+- user-visible consequences identified through [Blast Radius and Impact Routing](../blast-radius-and-impact-routing-guide.md);
+- data creation, change, and deletion semantics;
+- recovery from a realistic interruption;
+- any accepted technical-debt or migration edge.
+
+When a reviewed persona scenario pack exists, use [Persona-driven test scenarios](persona-driven-test-scenarios.md) to select every accepted `manual-qa` or hybrid route assigned to the walkthrough. Preserve its `PTS-###` ID in the result table. A persona may have exposed the scenario, but only the named tester can record the actual result and approve the Manual QA gate.
+
+For a low-risk documentation change, the walkthrough may be a role-based read-through. For a sensitive public capability, it should include deeper security, privacy, accessibility, resilience, and operational evidence.
+
+## Record evidence
+
+A project-owned QA evidence file should contain:
+
+```markdown
+# Manual QA evidence
+
+- Intent: <slug>
+- Build or commit: <revision>
+- Environment: <environment>
+- Tester: <name and role>
+- Date: <time>
+
+## Results
+
+| Scenario | Expected | Actual | Result | Evidence |
+| --- | --- | --- | --- | --- |
+
+## Limitations and residual risks
+
+## Decision
+
+Pass, fail, or returned for correction—with rationale.
+```
+
+Use links, screenshots, logs, or recordings only when permitted by data and retention policy. Do not place secrets or unnecessary personal data in SPECS.
+
+## Fail, fix, and rerun
+
+When a scenario fails:
+
+1. preserve the observation and evidence;
+2. classify severity and affected scope;
+3. decide whether the issue blocks acceptance;
+4. return material scope or design changes to the appropriate delivery phase;
+5. implement an authorised correction;
+6. rerun affected automated checks, standards review, and Manual QA;
+7. retain the original failure and the successful retest.
+
+Do not edit the evidence to make the first run appear successful.
+
+## Approve the human gate
+
+After the authorised tester accepts the evidence:
+
+```bash
+ewai delivery approve-manual-qa <intent-slug> \
+  --project . \
+  --yes \
+  --approved-by "Tester or accountable owner" \
+  --evidence SPECS/6.Build/<intent-slug>/qa-evidence.md \
+  --notes "Accepted result and limitations"
+```
+
+The evidence file must already exist inside the project. Approval records its hash so later changes are detectable.
+
+## Acceptance is not deployment
+
+Manual QA approval confirms the defined acceptance decision. It does not automatically:
+
+- deploy to production;
+- approve a destructive migration;
+- authorise a release outside the project's change process;
+- certify regulatory compliance;
+- accept a later changed build;
+- close the retrospective.
+
+Follow the project's release and operational governance after acceptance.
+
+## Reader-based QA for documentation
+
+For guide or policy changes, ask representative readers to complete tasks:
+
+- Can they find the correct starting page?
+- Can they distinguish requirements from recommendations?
+- Do they know what they own and where to stop?
+- Can they follow every link and command example?
+- Can they identify project truth, runtime state, and approval evidence?
+- Do they know how to recover from likely mistakes?
+
+## Acceptance checklist
+
+- [ ] Decision, scope, environment, revision, and tester are recorded.
+- [ ] Steps are reproducible and outcomes observable.
+- [ ] Important negative, boundary, and recovery cases are covered.
+- [ ] Actual results and failures are retained honestly.
+- [ ] Residual risks and untested areas are visible.
+- [ ] The approver has authority for this outcome.
+- [ ] Evidence is project-owned and free of inappropriate sensitive data.
+- [ ] Deployment and Retro remain separate follow-on decisions.
+
+## Related guides
+
+- [Developer delivery guide](../developer-delivery-guide.md)
+- [Blast Radius and Impact Routing](../blast-radius-and-impact-routing-guide.md)
+- [Persona-driven test scenarios](persona-driven-test-scenarios.md)
+- [Human approval and assurance](../human-approval-and-assurance-guide.md)
+- [Governance team guide](../governance/governance-team-guide.md)
+
+## Current contract sources
+
+- `src/delivery.mjs`
+- `src/delivery-gates.mjs`
+- `config/delivery-artifacts.yaml`
+- `tests/delivery.test.mjs`

package/Docs/quality/persona-driven-test-scenarios.md

@@ -0,0 +1,172 @@
+# Persona-driven test scenarios
+
+Use persona-driven scenarios when a technically plausible test plan may still miss how a user, operator, maintainer, or specialist experiences the outcome. The process is especially useful for permissions, accessibility, privacy, recovery, operational failure, misuse, adversarial behaviour, and role-specific edge cases.
+
+The process does not let personas invent requirements. Evidence establishes the expected behaviour; personas challenge its coverage; a named person accepts or rejects the resulting scenario; and the selected proof route establishes what happened.
+
+Ask EWAI: “Review the test coverage for this intent from the users' and operators' perspectives. Show me what we might be missing.” The `ewai-test-scenarios` skill prepares the brief and uses relevant installed personas. You review the proposed scenarios, check their expected results against accepted requirements and decide which to record. The commands below are optional controls for engineers who want to inspect or repeat the process.
+
+## Understand the authority chain
+
+Keep five layers separate:
+
+1. **Authoritative sources** define accepted journeys, criteria, constraints, standards, planning claims, and test obligations.
+2. **Contextual impact evidence** identifies observed or inferred areas worth investigating but does not create an expected outcome by itself.
+3. **Persona contributions** expose concerns, gaps, or hypotheses.
+4. **Human decisions** accept, block, supersede, or retain a proposal as a hypothesis.
+5. **Test evidence** proves the accepted observable result through automation, Manual QA, specialist assurance, or representative-user validation.
+
+If a proposed expected result has no authoritative source, keep it as a blocked or unresolved hypothesis. Return to the Product Owner or relevant accountable person rather than quietly promoting it into the test plan.
+
+## Prepare the scenario brief
+
+EWAI prepares the current concern during the conversation. You can also run preparation directly:
+
+```bash
+ewai test-scenarios prepare <intent-slug> \
+  --focus "permissions, recovery, and accessibility" \
+  --project . \
+  --json
+```
+
+Preparation reads project-owned intent and delivery evidence. It returns:
+
+- stable source references and a source digest;
+- contextual impact evidence when recorded;
+- up to four active installed personas;
+- each persona's name, tier, matched concerns, and engagement reason;
+- honest availability for core, premium, personal, and project tiers;
+- the allowed scenario, status, automation, and evidence-route values.
+
+Preparation is read-only. It does not call an LLM, download premium content, mutate an approval, or create a scenario pack.
+
+## Work with the active personas
+
+EWAI shows which personas are active and the concern each is examining before they contribute challenges. Check that these perspectives cover the problem you want reviewed; ask for a different focus if they don't.
+
+When you change the focus, EWAI prepares a new relevant set rather than accumulating personas from earlier topics. If you're using the CLI, run preparation again for the new focus. A project or premium persona is prioritised when its metadata matches, but its tier never grants authority.
+
+Premium personas participate only when already installed. If premium is unavailable, you can continue with relevant project, personal and core personas. This workflow doesn't install or synchronise premium content; use the separate [premium setup](../operations/premium-personas-setup.md) workflow if you want to add it.
+
+For each proposed scenario, capture:
+
+- stable `PTS-###` ID and plain-language title;
+- scenario type;
+- cited authoritative source references;
+- persona ID and the concern it contributed;
+- preconditions and user- or operator-centred actions;
+- observable expected results;
+- evidence route and automation classification;
+- planned test file and test name where automation applies;
+- owner and status.
+
+Use the [complete scenario input example](../examples/test-scenario-input.md) and [v1 candidate contract](../../skills-src/ewai-test-scenarios/references/scenario-contract.md#candidate-schema). The `ewai-test-scenarios` skill normally prepares the input with you; these references are available without first finding an installed skill directory.
+
+## Review before recording
+
+A named reviewer should:
+
+1. remove duplicate scenarios without losing distinct concerns;
+2. compare every expected result with its cited source;
+3. preserve disagreements and missing authority as gaps;
+4. decide whether the scenario is accepted, blocked, superseded, or a hypothesis;
+5. select the credible proof route;
+6. name the owner and planned automated test where applicable;
+7. confirm that human evidence has not been simulated.
+
+Then record the project-local candidate:
+
+```bash
+ewai test-scenarios record <intent-slug> \
+  --input <project-relative-candidate.json> \
+  --reviewed-by "Product Owner" \
+  --project . \
+  --json
+```
+
+EWAI writes paired `test-scenarios.json` and `test-scenarios.md` files under the intent's Build directory. JSON is authoritative. The Markdown copy is readable evidence whose digest is checked; do not edit it as a shortcut.
+
+## Read status and recover safely
+
+```bash
+ewai test-scenarios status <intent-slug> --project . --json
+```
+
+| Status | Meaning | Safe response |
+| --- | --- | --- |
+| `missing` | No reviewed pack exists | Prepare, challenge, review, and record. |
+| `recorded` | The JSON, Markdown, digest, and current sources match | Use accepted scenarios for planning and assigned tests. |
+| `stale` | An authoritative source changed after review | Prepare again and obtain a fresh named review. Preserve the old record for comparison. |
+| `invalid` | The pair, schema, slug, or digest cannot be trusted | Investigate the files and recovery evidence. Do not infer truth from Markdown. |
+
+Open the work item's **Test scenarios** tab to see which behaviours have reviewed scenarios, where gaps remain and how each scenario will be checked. The panel labelled **Coverage loom** shows this coverage; it doesn't run tests. You can inspect the supporting evidence and active personas, then filter the view without changing the recorded work. Preparing, recording and running scenarios happens through the EWAI conversation or CLI, not this read-only panel.
+
+## Build automated tests
+
+During Build, select only scenarios that are:
+
+- `accepted`;
+- classified as `automated` or `hybrid`;
+- assigned to the current task through a planned test inside its write set.
+
+For each scenario:
+
+1. read the accepted source and expected result (the test's **oracle**: what would demonstrate correct behaviour);
+2. write and capture the failing test first;
+3. preserve the accepted oracle;
+4. make the smallest implementation change;
+5. run the focused green command;
+6. run refactor and standards checks;
+7. report the scenario ID, test name, result, and remaining human route.
+
+Never weaken the expected result to match current code. If implementation reveals that the oracle is wrong, return to the accountable reviewer and source evidence.
+
+## Choose the right evidence route
+
+| Route | Use it for | Authority boundary |
+| --- | --- | --- |
+| `automated` | Deterministic behaviour that code can observe reliably | A passing test proves its stated assertion, not the whole user outcome. |
+| `manual-qa` | User-visible flow, layout, assistive technology, real interaction, or environment-specific behaviour | A named tester records actual evidence and approves the Manual QA gate separately. |
+| `specialist-assurance` | Security, privacy, accessibility, legal, clinical, financial, or other accountable specialist judgement | A persona may improve questions but cannot supply the assurance decision. |
+| `representative-user` | Language, usability, workflow fit, adoption, or lived experience | Real representative participants are required. |
+
+A hybrid scenario may include automation and a human route. Automation does not mark the human portion passed.
+
+## Product Owner checklist
+
+- [ ] Each accepted oracle cites authoritative source evidence.
+- [ ] Persona contributions are labelled as concerns rather than stakeholder facts.
+- [ ] Active persona names, tiers, matched concerns, and reasons are visible.
+- [ ] Missing relevant perspectives are acknowledged and owned.
+- [ ] Conflicts and gaps remain visible.
+- [ ] Every accepted scenario has a credible proof route and owner.
+- [ ] Premium availability is reported honestly with no implicit sync.
+- [ ] Manual QA, specialist assurance, and representative-user work remain human.
+
+## Developer checklist
+
+- [ ] The recorded JSON is current rather than stale or invalid.
+- [ ] Selected scenario IDs belong to the leased task and write set.
+- [ ] The failing test was written before production implementation.
+- [ ] The accepted oracle was not weakened.
+- [ ] Focused green, refactor, standards, and integration checks were recorded.
+- [ ] Results trace back to scenario IDs and leave remaining human routes open.
+
+## Related guides
+
+- [Product Owner guide](../product-owner-guide.md)
+- [Developer delivery guide](../developer-delivery-guide.md)
+- [Persona engagement UI](../personas/persona-engagement-ui.md)
+- [Blast Radius and Impact Routing](../blast-radius-and-impact-routing-guide.md)
+- [Manual QA and acceptance](manual-qa-and-acceptance.md)
+- [Human approval and assurance](../human-approval-and-assurance-guide.md)
+
+## Current contract sources
+
+- `src/test-scenarios.mjs`
+- `skills-src/ewai-test-scenarios/SKILL.md`
+- `skills-src/ewai-test-scenarios/references/scenario-contract.md`
+- `src/runtime/dashboard-server.mjs`
+- `public/app.js`
+- `tests/test-scenarios.test.mjs`
+- `tests/runtime.test.mjs`

package/Docs/quality/reproducible-archaeology-depth-review-checklist.md

@@ -0,0 +1,89 @@
+# Review an Archaeology and Discovery depth run
+
+Use this checklist when preparing and reviewing an investigation in your project. It complements the general
+[Manual QA and acceptance guide](manual-qa-and-acceptance.md); it does not replace
+the canonical EWAI approval operation.
+
+## Roles
+
+Name the people performing each role before starting:
+
+| Role | Responsibility |
+| --- | --- |
+| Operator or facilitator | Prepares the workspace, preserves evidence limitations, and guides the review. |
+| Product or business owner | Confirms purpose, intended outcomes, users, and proportionate product depth. |
+| Technical owner | Challenges repository, architecture, delivery, security, and operational interpretation. |
+| Accountable reviewer | Selects depth, records rationale and assigns a reviewed outcome to every gap. Manual QA is a separate decision where applicable. |
+
+One person may perform more than one role when that reflects the project, but the
+record must not disguise which decisions lacked independent or specialist review.
+Personas advise these people; they do not occupy these roles.
+
+## Before preparing a run
+
+- [ ] Confirm `.ewai-pipeline/project.json` points to the intended project and SPECS root.
+- [ ] Record the repository revision and the configured repository topology.
+- [ ] Capture a human project briefing: purpose, users, desired outcomes, business context, constraints, and prohibited assumptions.
+- [ ] Refresh the Source Map.
+- [ ] Inspect `analysis_failed`, `inventory_only`, `skipped_sensitive`, `skipped_oversized`, and excluded surfaces.
+- [ ] Confirm applicable project, stack, technology, and Organisation Blueprint Packs.
+- [ ] Confirm the provider capability being used.
+- [ ] Record which persona tiers are available; do not install or update premium personas as a side effect of this review.
+- [ ] Keep secrets, absolute paths, source bodies, prompts, managed persona content, and free-text owner answers out of bounded review inputs.
+
+Stop if the Source Map is missing or stale. Refresh it rather than accepting a
+recommendation against an unknown repository state.
+
+## Prepare and inspect
+
+- [ ] Prepare evidence depth in Guided Setup, with the CLI, or through the MCP tool.
+- [ ] Confirm architecture, data, security, product, delivery, governance, and operations appear exactly once.
+- [ ] For each dimension, inspect coverage, recommendation, drivers, evidence references, and active personas.
+- [ ] Confirm active personas change with the concern rather than accumulating across the review.
+- [ ] Confirm each persona displays its identity, tier, and reason for engagement.
+- [ ] Confirm core and project personas provide a complete path when premium personas are absent.
+- [ ] Review stable gaps separately from generated questions and proposed grouping.
+- [ ] Preserve contradictory owner and repository evidence rather than choosing the convenient version silently.
+- [ ] Ask only the material questions needed to resolve or explicitly retain a gap.
+
+## Record the named review
+
+- [ ] Name the accountable reviewer.
+- [ ] Select `bounded`, `standard`, or `deep` independently for all seven dimensions.
+- [ ] Add substantive rationale whenever selecting less depth than recommended.
+- [ ] Assign every stable gap to one group exactly once.
+- [ ] Give every gap an explicit disposition: owned, shared, deferred, or excluded.
+- [ ] Confirm grouping describes work organisation and does not alter stable gap identity.
+- [ ] Record the review against the current preparation digest.
+- [ ] Retain the immutable run ID and content digest without copying sensitive source material into the QA record.
+
+If another preparation supersedes the reviewed digest, repeat the named review.
+Do not force an older decision onto new evidence.
+
+## Assess content quality
+
+Structural reproducibility alone is not sufficient. Ask the named product and
+technical owners to rate:
+
+- [ ] whether any material concern is missing;
+- [ ] whether questions are duplicated, irrelevant, or disproportionate;
+- [ ] whether every conclusion can be traced to safe evidence references;
+- [ ] whether the selected depth fits the intended change and risk;
+- [ ] whether failed, excluded, contradictory, or unknown evidence remains visible;
+- [ ] whether the next action for each gap is understandable and owned;
+- [ ] whether the process performs at least as well as the previous unconstrained route on engineering coverage.
+
+Record the ratings and rationale. A faster or lower-token run fails this review if
+it weakens material engineering understanding.
+
+## Retain the investigation decision
+
+Save the reviewed run ID, selected depths, rationale for reductions, gap owners, unresolved limitations and named review. Don't describe uninvestigated areas as complete. If preparation changes, obtain a fresh review against the new digest.
+
+An ordinary investigation doesn't require three clean sessions or interface testing at 390 px. Those checks belong to [accepting changes to EWAI itself](../maintainers/evidence-depth-acceptance.md). They remain required when that feature acceptance is the task.
+
+## Related guidance
+
+- [Reproducible Archaeology and Discovery Depth](../reproducible-archaeology-and-discovery-depth.md)
+- [Worked example](../examples/reproducible-archaeology-depth-example.md)
+- [Manual QA and acceptance](manual-qa-and-acceptance.md)