ado-sync 0.1.46 → 0.1.48

package/.next-steps.md

@@ -0,0 +1,179 @@
+What I'd build next (priority order)
+
+1. create_issue_on_failure — GitHub Issues + ADO Bugs (dual provider) ✅ DONE
+
+When publish-test-results finds failures, automatically file issues so agents can pick them up
+and propose fix PRs. Supports both GitHub Issues and ADO Bugs as providers under one config key.
+
+--- Why GitHub Issues is the better default ---
+
+GitHub Issues is preferred over ADO Bugs for most teams because:
+  - 5,000 req/hour API limit vs ADO's stricter rate limits
+  - The healer agent lives in GitHub Actions — Issue → PR → auto-close on merge is native
+  - Only requires repo `issues: write` permission, not ADO Work Items (Write)
+  - Dedup by label search is simpler than WiQL queries
+  - Issue body can embed the ADO TC URL, so full traceability is preserved
+
+ADO Bugs remain available for teams fully inside ADO (no GitHub, or policy requires bugs in ADO).
+
+--- The bulk-failure problem ---
+
+A full environment outage (app down, auth broken) can fail 10,000 tests and would create 10,000
+issues, hitting API limits and flooding the tracker. Guards are applied in order before any issue
+is filed:
+
+  1. Failure rate threshold (default 20%)
+     If >20% of tests fail → skip per-test issues entirely.
+     Create ONE environment-failure issue with a summary of all failures.
+
+  2. Error signature clustering
+     Group failures by hashed error message.
+     If 500 tests all fail with ERR_CONNECTION_REFUSED → ONE cluster issue listing affected tests
+     (up to 20 names, then "+ N more").
+
+  3. Hard cap (default 50)
+     Ceiling regardless of strategy. After the cap, one "overflow" summary issue is created:
+     "50 issues filed; X additional failures suppressed — possible environment issue."
+
+  4. Dedup against open issues
+     Skip creation if an open issue already exists for that TC (matched by label tc:ID or title).
+
+Decision tree at runtime:
+
+  failures > threshold% of total?
+    └─ YES → 1 environment issue, stop
+    └─ NO
+         └─ cluster by error signature
+              └─ cluster size > 10?
+                   └─ YES → 1 issue per cluster (lists affected TCs)
+                   └─ NO  → 1 issue per TC (up to maxIssues cap)
+                                 └─ cap hit? → 1 overflow summary issue
+
+--- CLI ---
+
+  ado-sync publish-test-results \
+    --testResult results/ctrf.json \
+    --create-issues-on-failure \
+    --github-repo owner/repo \
+    --github-token $GITHUB_TOKEN
+
+  # ADO Bugs instead:
+  ado-sync publish-test-results \
+    --testResult results/ctrf.json \
+    --create-issues-on-failure \
+    --issue-provider ado
+
+--- Config (ado-sync.json) ---
+
+  {
+    "publishTestResults": {
+      "createIssuesOnFailure": {
+        "provider": "github",          // "github" | "ado"
+        "repo": "owner/repo",          // GitHub only
+        "labels": ["test-failure", "automated"],
+        "threshold": 20,               // % of failures that triggers env-failure mode
+        "maxIssues": 50,               // hard cap per run
+        "clusterByError": true,        // group same-error failures into one issue
+        "dedupByTestCase": true,       // skip if open issue for TC already exists
+        "areaPath": "MyProject\\QA",   // ADO only
+        "assignTo": "$AZURE_DEVOPS_USER"
+      }
+    }
+  }
+
+--- MCP tools ---
+
+  create_issue({ title, failureDetails, testCaseId, buildId, provider })
+    → files one issue (GitHub or ADO) and returns the issue/bug URL
+
+  create_issues_from_run({ runId, provider, ...guardOptions })
+    → applies full guard logic against a completed test run, returns summary
+
+--- GitHub Issue body template ---
+
+  ## Test failure: {testName}
+
+  | Field        | Value |
+  |---|---|
+  | ADO Test Case | [#{testCaseId}]({adoTcUrl}) |
+  | Build         | #{buildId} |
+  | Run           | [View in Azure DevOps]({runUrl}) |
+  | File          | {filePath}:{line} |
+
+  ### Error
+  ```
+  {errorMessage}
+  ```
+
+  ### Stack trace
+  ```
+  {stackTrace}
+  ```
+
+  /cc @healer-agent
+
+This unblocks Use Case 4 — the GitHub Agentic Workflow where the Playwright Healer Agent reads
+the issue, understands the failure from errorMessage + stackTrace + ADO TC context, then proposes
+a fix PR. The issue is the handoff between CI and the healer.
+
+2. CTRF format support in publish-test-results ✅ DONE
+The scope doc and materia article both use CTRF. It's becoming the standard cross-framework report format. Teams using the materia pattern will arrive with CTRF output expecting ado-sync to accept it.
+Auto-detected from results.tests array structure. TC IDs from tags[] or test name. Attachments,
+stdout/stderr uploaded automatically. Documented in docs/publish-test-results.md.
+
+3. get_story_context MCP tool (Planner agent feed) ✅ DONE
+Returns AC items (bullet list), inferred tags (@smoke, @auth…), extracted actors, and linked TC IDs.
+CLI: ado-sync story-context --story-id 1234
+MCP: get_story_context({ storyId: 1234 })
+Implemented in src/azure/work-items.ts — getStoryContext() with tag inference + actor extraction.
+
+4. ai-workflow-manifest.json generator ✅ DONE
+CLI: ado-sync generate --manifest --story-ids 1234
+MCP: generate_manifest({ storyIds: [1234], outputFolder: "e2e/bdd", format: "gherkin" })
+Writes .ai-workflow-manifest-{id}.json with 8-step workflow, AC items, required docs checklist,
+validation steps, and output paths. Implemented in src/sync/manifest.ts.
+
+5. GitHub Actions reusable workflow
+The scope doc explicitly asks for this:
+
+"There is need to write GitHub Actions templates in TR Org Workflows repo"
+
+A reusable workflow file:
+
+
+# .github/workflows/ado-sync.yml (reusable)
+on:
+  workflow_call:
+    inputs:
+      test-result-path: ...
+      create-issues-on-failure: ...
+Teams subscribe to it — one-line adoption in their own pipelines.
+
+6. Flaky test detection in publish-test-results
+Materia's 4-method flaky detection:
+
+Summary-level flaky count
+Retry pattern analysis
+Explicit marking
+Passed-after-retry
+Add a --detect-flaky flag that outputs a flaky test report alongside the ADO publish. Tag flaky TCs in ADO with ado-sync:flaky so you can track trends across runs.
+
+The full picture
+
+User Story (ADO)
+    ↓  get_story_context (MCP)
+Planner Agent → Markdown spec
+    ↓  generate --manifest
+.ai-workflow-manifest.json + spec file
+    ↓  generate (existing)
+Playwright test skeleton
+    ↓  push (existing)
+ADO Test Case with @tc:ID writeback
+    ↓  CI runs tests
+CTRF / Playwright JSON results
+    ↓  publish-test-results (existing + flaky detection)
+ADO Test Run results
+    ↓  failures → create_issue_on_failure (new, GitHub Issues or ADO Bugs)
+GitHub Issue / ADO Bug → Healer Agent → fix PR
+ado-sync becomes the connective tissue across all 4 use cases in that scope document. Right now you cover the right half. Items 1–3 above close the loop on Use Cases 3 and 4.
+