ado-sync 0.1.49 → 0.1.51

package/.next-steps.md

@@ -1,179 +0,0 @@
-What I'd build next (priority order)
-
-1. create_issue_on_failure — GitHub Issues + ADO Bugs (dual provider) ✅ DONE
-
-When publish-test-results finds failures, automatically file issues so agents can pick them up
-and propose fix PRs. Supports both GitHub Issues and ADO Bugs as providers under one config key.
-
---- Why GitHub Issues is the better default ---
-
-GitHub Issues is preferred over ADO Bugs for most teams because:
-  - 5,000 req/hour API limit vs ADO's stricter rate limits
-  - The healer agent lives in GitHub Actions — Issue → PR → auto-close on merge is native
-  - Only requires repo `issues: write` permission, not ADO Work Items (Write)
-  - Dedup by label search is simpler than WiQL queries
-  - Issue body can embed the ADO TC URL, so full traceability is preserved
-
-ADO Bugs remain available for teams fully inside ADO (no GitHub, or policy requires bugs in ADO).
-
---- The bulk-failure problem ---
-
-A full environment outage (app down, auth broken) can fail 10,000 tests and would create 10,000
-issues, hitting API limits and flooding the tracker. Guards are applied in order before any issue
-is filed:
-
-  1. Failure rate threshold (default 20%)
-     If >20% of tests fail → skip per-test issues entirely.
-     Create ONE environment-failure issue with a summary of all failures.
-
-  2. Error signature clustering
-     Group failures by hashed error message.
-     If 500 tests all fail with ERR_CONNECTION_REFUSED → ONE cluster issue listing affected tests
-     (up to 20 names, then "+ N more").
-
-  3. Hard cap (default 50)
-     Ceiling regardless of strategy. After the cap, one "overflow" summary issue is created:
-     "50 issues filed; X additional failures suppressed — possible environment issue."
-
-  4. Dedup against open issues
-     Skip creation if an open issue already exists for that TC (matched by label tc:ID or title).
-
-Decision tree at runtime:
-
-  failures > threshold% of total?
-    └─ YES → 1 environment issue, stop
-    └─ NO
-         └─ cluster by error signature
-              └─ cluster size > 10?
-                   └─ YES → 1 issue per cluster (lists affected TCs)
-                   └─ NO  → 1 issue per TC (up to maxIssues cap)
-                                 └─ cap hit? → 1 overflow summary issue
-
---- CLI ---
-
-  ado-sync publish-test-results \
-    --testResult results/ctrf.json \
-    --create-issues-on-failure \
-    --github-repo owner/repo \
-    --github-token $GITHUB_TOKEN
-
-  # ADO Bugs instead:
-  ado-sync publish-test-results \
-    --testResult results/ctrf.json \
-    --create-issues-on-failure \
-    --issue-provider ado
-
---- Config (ado-sync.json) ---
-
-  {
-    "publishTestResults": {
-      "createIssuesOnFailure": {
-        "provider": "github",          // "github" | "ado"
-        "repo": "owner/repo",          // GitHub only
-        "labels": ["test-failure", "automated"],
-        "threshold": 20,               // % of failures that triggers env-failure mode
-        "maxIssues": 50,               // hard cap per run
-        "clusterByError": true,        // group same-error failures into one issue
-        "dedupByTestCase": true,       // skip if open issue for TC already exists
-        "areaPath": "MyProject\\QA",   // ADO only
-        "assignTo": "$AZURE_DEVOPS_USER"
-      }
-    }
-  }
-
---- MCP tools ---
-
-  create_issue({ title, failureDetails, testCaseId, buildId, provider })
-    → files one issue (GitHub or ADO) and returns the issue/bug URL
-
-  create_issues_from_run({ runId, provider, ...guardOptions })
-    → applies full guard logic against a completed test run, returns summary
-
---- GitHub Issue body template ---
-
-  ## Test failure: {testName}
-
-  | Field        | Value |
-  |---|---|
-  | ADO Test Case | [#{testCaseId}]({adoTcUrl}) |
-  | Build         | #{buildId} |
-  | Run           | [View in Azure DevOps]({runUrl}) |
-  | File          | {filePath}:{line} |
-
-  ### Error
-  ```
-  {errorMessage}
-  ```
-
-  ### Stack trace
-  ```
-  {stackTrace}
-  ```
-
-  /cc @healer-agent
-
-This unblocks Use Case 4 — the GitHub Agentic Workflow where the Playwright Healer Agent reads
-the issue, understands the failure from errorMessage + stackTrace + ADO TC context, then proposes
-a fix PR. The issue is the handoff between CI and the healer.
-
-2. CTRF format support in publish-test-results ✅ DONE
-The scope doc and materia article both use CTRF. It's becoming the standard cross-framework report format. Teams using the materia pattern will arrive with CTRF output expecting ado-sync to accept it.
-Auto-detected from results.tests array structure. TC IDs from tags[] or test name. Attachments,
-stdout/stderr uploaded automatically. Documented in docs/publish-test-results.md.
-
-3. get_story_context MCP tool (Planner agent feed) ✅ DONE
-Returns AC items (bullet list), inferred tags (@smoke, @auth…), extracted actors, and linked TC IDs.
-CLI: ado-sync story-context --story-id 1234
-MCP: get_story_context({ storyId: 1234 })
-Implemented in src/azure/work-items.ts — getStoryContext() with tag inference + actor extraction.
-
-4. ai-workflow-manifest.json generator ✅ DONE
-CLI: ado-sync generate --manifest --story-ids 1234
-MCP: generate_manifest({ storyIds: [1234], outputFolder: "e2e/bdd", format: "gherkin" })
-Writes .ai-workflow-manifest-{id}.json with 8-step workflow, AC items, required docs checklist,
-validation steps, and output paths. Implemented in src/sync/manifest.ts.
-
-5. GitHub Actions reusable workflow
-The scope doc explicitly asks for this:
-
-"There is need to write GitHub Actions templates in TR Org Workflows repo"
-
-A reusable workflow file:
-
-
-# .github/workflows/ado-sync.yml (reusable)
-on:
-  workflow_call:
-    inputs:
-      test-result-path: ...
-      create-issues-on-failure: ...
-Teams subscribe to it — one-line adoption in their own pipelines.
-
-6. Flaky test detection in publish-test-results
-Materia's 4-method flaky detection:
-
-Summary-level flaky count
-Retry pattern analysis
-Explicit marking
-Passed-after-retry
-Add a --detect-flaky flag that outputs a flaky test report alongside the ADO publish. Tag flaky TCs in ADO with ado-sync:flaky so you can track trends across runs.
-
-The full picture
-
-User Story (ADO)
-    ↓  get_story_context (MCP)
-Planner Agent → Markdown spec
-    ↓  generate --manifest
-.ai-workflow-manifest.json + spec file
-    ↓  generate (existing)
-Playwright test skeleton
-    ↓  push (existing)
-ADO Test Case with @tc:ID writeback
-    ↓  CI runs tests
-CTRF / Playwright JSON results
-    ↓  publish-test-results (existing + flaky detection)
-ADO Test Run results
-    ↓  failures → create_issue_on_failure (new, GitHub Issues or ADO Bugs)
-GitHub Issue / ADO Bug → Healer Agent → fix PR
-ado-sync becomes the connective tissue across all 4 use cases in that scope document. Right now you cover the right half. Items 1–3 above close the loop on Use Cases 3 and 4.
-