ado-sync 0.1.47 → 0.1.48

package/docs/mcp-server.md

@@ -9,54 +9,94 @@ Using the MCP server instead of spawning the CLI gives agents:
 
 ---
 
-## Start the server
+## Installation
 
+No separate install needed. `ado-sync-mcp` ships inside the `ado-sync` npm package.
+
+**Option A — Global install (recommended, fastest startup)**
 ```bash
-# stdio transport (standard for IDE integrations)
-ado-sync-mcp
+npm install -g ado-sync
 ```
 
-The server reads your `ado-sync.json` (or the path in `ADO_SYNC_CONFIG` env var) automatically.
+**Option B — npx (no install, always latest)**
+
+Use `npx --yes --package=ado-sync ado-sync-mcp` as the command in all configs below.
+npx downloads and runs the package automatically on first use.
+
+---
+
+## Claude Code — one-liner registration
 
 ```bash
-# Point at a specific config
-ADO_SYNC_CONFIG=/path/to/ado-sync.json ado-sync-mcp
+# With global install
+claude mcp add ado-sync \
+  --env AZURE_DEVOPS_TOKEN="$AZURE_DEVOPS_TOKEN" \
+  --env ADO_SYNC_CONFIG="$(pwd)/ado-sync.json" \
+  -- ado-sync-mcp
+
+# With npx (no global install required)
+claude mcp add ado-sync \
+  --env AZURE_DEVOPS_TOKEN="$AZURE_DEVOPS_TOKEN" \
+  --env ADO_SYNC_CONFIG="$(pwd)/ado-sync.json" \
+  -- npx --yes --package=ado-sync ado-sync-mcp
+```
+
+Verify it registered: run `/mcp` in Claude Code — `ado-sync` should appear in the list.
+
+**Manual config** — `~/.claude/claude_desktop_config.json`:
+```json
+{
+  "mcpServers": {
+    "ado-sync": {
+      "command": "npx",
+      "args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
+      "env": {
+        "AZURE_DEVOPS_TOKEN": "<your-pat>",
+        "ADO_SYNC_CONFIG": "/absolute/path/to/ado-sync.json"
+      }
+    }
+  }
+}
 ```
 
 ---
 
-## Register in Claude Code
+## Claude Desktop
 
-Add to `~/.claude/claude_desktop_config.json` (or the path shown in Claude Code → Settings → MCP):
+Config file location:
+- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
+- Windows: `%APPDATA%\Claude\claude_desktop_config.json`
 
 ```json
 {
   "mcpServers": {
     "ado-sync": {
-      "command": "ado-sync-mcp",
+      "command": "npx",
+      "args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
       "env": {
-        "AZURE_DEVOPS_TOKEN": "<your-token>",
-        "ADO_SYNC_CONFIG": "/path/to/ado-sync.json"
+        "AZURE_DEVOPS_TOKEN": "<your-pat>",
+        "ADO_SYNC_CONFIG": "/absolute/path/to/ado-sync.json"
       }
     }
   }
 }
 ```
 
-Restart Claude Code. The tools appear automatically under the `ado-sync` server namespace.
+Restart Claude Desktop after saving.
 
 ---
 
-## Register in VS Code (GitHub Copilot)
+## VS Code (GitHub Copilot agent mode)
 
-Add to `.vscode/mcp.json` in your workspace (or user settings):
+Create `.vscode/mcp.json` in your workspace root:
 
 ```json
 {
   "servers": {
     "ado-sync": {
       "type": "stdio",
-      "command": "ado-sync-mcp",
+      "command": "npx",
+      "args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
       "env": {
         "AZURE_DEVOPS_TOKEN": "${env:AZURE_DEVOPS_TOKEN}",
         "ADO_SYNC_CONFIG": "${workspaceFolder}/ado-sync.json"
@@ -66,20 +106,23 @@ Add to `.vscode/mcp.json` in your workspace (or user settings):
 }
 ```
 
+`${env:AZURE_DEVOPS_TOKEN}` reads the variable from your shell environment — no hardcoded secrets.
+
 ---
 
-## Register in Cursor
+## Cursor
 
-Add to `~/.cursor/mcp.json`:
+`~/.cursor/mcp.json`:
 
 ```json
 {
   "mcpServers": {
     "ado-sync": {
-      "command": "ado-sync-mcp",
+      "command": "npx",
+      "args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
       "env": {
-        "AZURE_DEVOPS_TOKEN": "<your-token>",
-        "ADO_SYNC_CONFIG": "/path/to/ado-sync.json"
+        "AZURE_DEVOPS_TOKEN": "<your-pat>",
+        "ADO_SYNC_CONFIG": "/absolute/path/to/ado-sync.json"
       }
     }
   }
@@ -88,6 +131,18 @@ Add to `~/.cursor/mcp.json`:
 
 ---
 
+## Verify it works
+
+After registration, ask your AI assistant:
+
+```
+"Call ado-sync validate_config and tell me what it returns"
+```
+
+Expected response includes config validity, Azure connection status, and test plan confirmation.
+
+---
+
 ## Available tools
 
 | Tool | Description |
@@ -101,7 +156,10 @@ Add to `~/.cursor/mcp.json`:
 | `diff` | Show field-level diff (richer than status) |
 | `generate_specs` | Generate local spec files from ADO User Stories |
 | `get_work_items` | Fetch ADO User Stories / work items |
-| `publish_test_results` | Publish TRX / JUnit / Playwright JSON results |
+| `publish_test_results` | Publish TRX / JUnit / Playwright JSON / CTRF results; optionally file issues for failures |
+| `create_issue` | File a single GitHub Issue or ADO Bug for a test failure (for healer agents) |
+| `get_story_context` | Planner-agent feed: AC items, suggested tags, actors, linked TC IDs |
+| `generate_manifest` | Write `.ai-workflow-manifest-{id}.json` for the full Planner→CI cycle |
 
 ---
 
@@ -111,7 +169,7 @@ All tools accept these optional base parameters:
 
 | Parameter | Type | Description |
 |-----------|------|-------------|
-| `configPath` | string | Path to `ado-sync.json` (overrides `ADO_SYNC_CONFIG`) |
+| `configPath` | string | Absolute path to `ado-sync.json` (overrides `ADO_SYNC_CONFIG`) |
 | `configOverrides` | string[] | Runtime overrides in `key=value` format (same as `--config-override`) |
 
 ### `push_specs`
@@ -141,10 +199,50 @@ All tools accept these optional base parameters:
 ```json
 {
   "testResult": "results/playwright.json",
-  "attachmentsFolder": "test-results/"
+  "attachmentsFolder": "test-results/",
+  "createIssuesOnFailure": true,
+  "issueProvider": "github",
+  "githubRepo": "myorg/myrepo",
+  "githubToken": "$GITHUB_TOKEN",
+  "bugThreshold": 20,
+  "maxIssues": 50
+}
+```
+
+### `create_issue`
+
+```json
+{
+  "title": "[FAILED] Login with valid credentials",
+  "body": "Error: Expected 200 but got 401\n\nStack: ...",
+  "provider": "github",
+  "githubRepo": "myorg/myrepo",
+  "githubToken": "$GITHUB_TOKEN",
+  "testCaseId": 1234
 }
 ```
 
+### `get_story_context`
+
+```json
+{ "storyId": 1234 }
+```
+
+Returns: AC items as a bullet list, inferred tags (`@smoke`, `@auth`, …), extracted actors, and IDs of any Test Cases already linked via TestedBy relation.
+
+### `generate_manifest`
+
+```json
+{
+  "storyIds": [1234, 5678],
+  "outputFolder": "e2e/bdd",
+  "format": "gherkin",
+  "dryRun": false
+}
+```
+
+Writes `.ai-workflow-manifest-1234.json` in `outputFolder`. The manifest contains the ordered 8-step workflow, AC items, required documents checklist, and validation steps.
+
 ---
 
 ## Example agent workflow
@@ -152,15 +250,17 @@ All tools accept these optional base parameters:
 Once registered, an AI agent can orchestrate the full test lifecycle without any human CLI invocation:
 
 ```
-Agent: "Push my Playwright specs to Azure DevOps"
-→ calls push_specs({ dryRun: true })        # preview
-→ calls push_specs({})                       # apply
+Agent: "Set up ado-sync for this repo"
+→ calls validate_config to check connectivity
+→ calls push_specs({ dryRun: true }) to preview
+→ calls push_specs({}) to create Test Cases
 
 Agent: "Generate spec files for User Story 1234"
 → calls generate_specs({ storyIds: [1234], format: "gherkin" })
 
-Agent: "Publish the latest test results"
-→ calls publish_test_results({ testResult: "results/playwright.json" })
+Agent: "Publish the latest test results and file issues for failures"
+→ calls publish_test_results({ testResult: "results/playwright.json", createIssuesOnFailure: true, githubRepo: "myorg/myrepo", githubToken: "$GITHUB_TOKEN" })
+→ returns issue URLs for any failures → healer agent opens fix PRs
 ```
 
 ---
@@ -169,5 +269,7 @@ Agent: "Publish the latest test results"
 
 | Variable | Description |
 |----------|-------------|
-| `ADO_SYNC_CONFIG` | Path to config file (default: `ado-sync.json` in cwd) |
+| `ADO_SYNC_CONFIG` | Absolute path to config file (default: `ado-sync.json` in cwd) |
 | `AZURE_DEVOPS_TOKEN` | PAT or access token (referenced by `auth.token` in config) |
+
+> **Important:** `ADO_SYNC_CONFIG` must be an **absolute path**. The MCP server process does not inherit the shell's working directory the same way as a CLI call.

package/docs/publish-test-results.md

@@ -1,6 +1,6 @@
 # publish-test-results
 
-Parses test result files (TRX, NUnit XML, JUnit, Cucumber JSON, Playwright JSON) and publishes them to an Azure DevOps Test Run, linking results back to Test Cases either directly by TC ID (when available in the result file) or by `AutomatedTestName` matching.
+Parses test result files (TRX, NUnit XML, JUnit, Cucumber JSON, Playwright JSON, CTRF JSON) and publishes them to an Azure DevOps Test Run, linking results back to Test Cases either directly by TC ID (when available in the result file) or by `AutomatedTestName` matching.
 
 ---
 
@@ -31,11 +31,17 @@ ado-sync publish-test-results \
 | Option | Description |
 |--------|-------------|
 | `--testResult <path>` | Path to a result file. Repeatable. |
-| `--testResultFormat <format>` | `trx` · `nunitXml` · `junit` · `cucumberJson` · `playwrightJson`. Auto-detected when omitted. |
+| `--testResultFormat <format>` | `trx` · `nunitXml` · `junit` · `cucumberJson` · `playwrightJson` · `ctrfJson`. Auto-detected when omitted. |
 | `--attachmentsFolder <path>` | Folder to scan for screenshots/videos/logs to attach to test results. |
 | `--runName <name>` | Name for the Test Run in Azure DevOps. Defaults to `ado-sync <ISO timestamp>`. |
 | `--buildId <id>` | Build ID to associate with the Test Run. |
 | `--dry-run` | Parse results and print summary without creating a run in Azure. |
+| `--create-issues-on-failure` | File GitHub Issues or ADO Bugs for each failed test after publishing. |
+| `--issue-provider <github\|ado>` | Issue provider. Default: `github`. |
+| `--github-repo <owner/repo>` | GitHub repository to file issues in. |
+| `--github-token <token>` | GitHub token. Supports `$ENV_VAR` references. |
+| `--bug-threshold <percent>` | If more than this % of tests fail, one environment-failure issue is filed instead of per-test issues. Default: `20`. |
+| `--max-issues <n>` | Hard cap on issues filed per run. Default: `50`. |
 | `--config-override` | Override config values (repeatable, same as other commands). |
 
 ---
@@ -50,6 +56,7 @@ ado-sync publish-test-results \
 | Cucumber JSON | `.json` | Yes (JSON array, Cucumber format) | Yes — via `@tc:ID` tag on scenario | `step.embeddings[]` (base64 screenshots/video) |
 | Playwright JSON | `.json` | Yes (JSON object with `suites` key) | Yes — via `test.annotations[{ type: 'tc', description: 'ID' }]` (preferred) or `@tc:ID` in test title | `test.results[].attachments[]` (screenshots, videos, traces) |
 | Robot Framework XML | `output.xml` | Yes (`<robot>` root element) | Yes — via `<tags><tag>tc:ID</tag></tags>` | — |
+| CTRF JSON | `.json` | Yes (`results.tests` array) | Yes — via `tags: ["@tc:ID"]` or `@tc:ID` in test name | `attachments[].path` files, `stdout`/`stderr` arrays |
 
 > **NUnit via TRX**: when NUnit tests are run through the VSTest adapter (`--logger trx`), `[Property]` values are **not** included in the TRX output. Use `--logger "nunit3;LogFileName=results.xml"` to get the native NUnit XML format, which does include property values.
 
@@ -441,6 +448,38 @@ Recommended config:
 
 ---
 
+### CTRF (Common Test Report Format)
+
+[CTRF](https://ctrf.io) is a framework-agnostic JSON report format supported by reporters for Playwright, Cypress, Jest, k6, and many others. ado-sync auto-detects CTRF from the `results.tests` array structure.
+
+```bash
+# Example: Playwright with CTRF reporter
+npm install --save-dev playwright-ctrf-json-reporter
+
+# playwright.config.ts:
+# reporter: [['playwright-ctrf-json-reporter', { outputFile: 'results/ctrf.json' }]]
+
+npx playwright test
+ado-sync publish-test-results --testResult results/ctrf.json
+```
+
+```bash
+# Example: Jest with CTRF reporter
+npm install --save-dev jest-ctrf-json-reporter
+
+# jest.config.ts:
+# reporters: [['jest-ctrf-json-reporter', { outputFile: 'results/ctrf.json' }]]
+
+npx jest
+ado-sync publish-test-results --testResult results/ctrf.json
+```
+
+TC IDs are extracted from the `tags` array (e.g. `["@tc:1234", "@smoke"]`) or, as a fallback, from `@tc:ID` in the test name. `stdout`/`stderr` arrays and `attachments[].path` files are uploaded automatically.
+
+> **Status mapping**: CTRF `passed` → `Passed`, `failed` → `Failed`, `skipped`/`pending`/`other` → `NotExecuted`.
+
+---
+
 ### Flutter
 
 Flutter can produce JUnit XML via the `flutter_test_junit` package or by piping `--reporter junit`:
@@ -483,6 +522,7 @@ TC linking uses `AutomatedTestName` matching — set `sync.markAutomated: true`
 | Espresso | JUnit XML | ❌ AutomatedTestName matching only | `<system-out>`, `<system-err>` | |
 | Flutter | JUnit XML | ❌ AutomatedTestName matching only | `<system-out>`, `<system-err>` | |
 | Robot Framework | Robot XML (`output.xml`) | ✅ `tc:N` in `<tags>` | — | |
+| CTRF (any framework) | CTRF JSON | ✅ `tags: ["@tc:ID"]` or `@tc:ID` in name | `attachments[].path` files + `stdout`/`stderr` | |
 
 ---
 
@@ -499,6 +539,7 @@ ado-sync uploads screenshots, videos, and logs from test results to the correspo
 | JUnit XML | `<system-out>` → log; `<system-err>` → log; `[[ATTACHMENT\|path]]` → Playwright files |
 | Cucumber JSON | `step.embeddings[]` → base64-encoded screenshots/video |
 | Playwright JSON | `results[].attachments[].path` → files on disk (screenshots, videos, traces) |
+| CTRF JSON | `tests[].attachments[].path` → files on disk; `tests[].stdout[]` / `tests[].stderr[]` → console logs |
 
 > **Note**: All file paths are resolved relative to the result file's directory, not the process working directory. This matches how test runners (Playwright, MSTest, NUnit) write relative paths in their output.
 
@@ -670,6 +711,99 @@ Results can also be configured in the config file under `publishTestResults`:
 
 ---
 
+## Creating issues on failure
+
+`--create-issues-on-failure` automatically files a GitHub Issue or ADO Bug for each failed test
+after the run is published. Multiple guards prevent flooding your tracker when the environment is
+the problem rather than individual tests.
+
+### Guard logic (applied in order)
+
+```
+failures > threshold% of total?
+  └─ YES → 1 environment-failure issue, stop
+  └─ NO
+       └─ cluster by error signature
+            └─ cluster size > 1?
+                 └─ YES → 1 issue per cluster (lists affected test names)
+                 └─ NO  → 1 issue per TC (up to maxIssues cap)
+                               └─ cap hit? → 1 overflow summary issue
+```
+
+| Guard | Default | Description |
+|---|---|---|
+| Failure-rate threshold | 20% | Above this, one env-failure issue is filed instead of per-test |
+| Error clustering | enabled | Tests with the same error message are grouped into one issue |
+| Hard cap | 50 | No more than this many issues per run; one overflow summary when exceeded |
+| Dedup | enabled | Skip if an open issue already exists for the same TC (GitHub: by `tc:ID` label; ADO: by title) |
+
+### GitHub Issues (recommended)
+
+```bash
+ado-sync publish-test-results \
+  --testResult results/ctrf.json \
+  --create-issues-on-failure \
+  --github-repo myorg/myrepo \
+  --github-token $GITHUB_TOKEN
+```
+
+Each issue is labelled `test-failure` and `tc:{ID}` (when a TC ID is available). The issue body
+contains the error message, stack trace, ADO TC link, and run URL — everything a healer agent
+needs to propose a fix PR.
+
+### ADO Bugs
+
+```bash
+ado-sync publish-test-results \
+  --testResult results/junit.xml \
+  --create-issues-on-failure \
+  --issue-provider ado
+```
+
+ADO Bugs are created as Bug work items in the same project. The `Repro Steps` field is populated
+with the error details. When a TC ID is known, a `TestedBy` relation is added linking the Bug to
+the Test Case.
+
+### Config-based setup
+
+```json
+{
+  "publishTestResults": {
+    "createIssuesOnFailure": {
+      "provider": "github",
+      "repo": "myorg/myrepo",
+      "token": "$GITHUB_TOKEN",
+      "labels": ["test-failure", "automated"],
+      "threshold": 20,
+      "maxIssues": 50,
+      "clusterByError": true,
+      "dedupByTestCase": true
+    }
+  }
+}
+```
+
+CLI flags override the config values when both are present.
+
+### MCP tool: `create_issue`
+
+The `create_issue` MCP tool lets healer agents file a single issue directly:
+
+```
+create_issue({
+  title:       "[FAILED] Login with valid credentials",
+  body:        "Error: Expected 200 but got 401\n\nStack: ...",
+  provider:    "github",
+  githubRepo:  "myorg/myrepo",
+  githubToken: "$GITHUB_TOKEN",
+  testCaseId:  1234
+})
+```
+
+Returns the issue URL immediately, which the agent can embed in its fix PR.
+
+---
+
 ## Output
 
 ```

package/llms.txt

@@ -9,7 +9,7 @@ npm package: `ado-sync` | bin: `ado-sync` and `ado-sync-mcp`
 - `pull` — reads Test Cases from Azure DevOps and updates matching local files
 - `status` — dry-run diff: shows what would change without touching anything
 - `diff` — field-level diff showing exactly which fields (title, steps, description) differ
-- `publish-test-results` — publishes TRX / JUnit XML / NUnit XML / Cucumber JSON / Playwright JSON results to an Azure DevOps Test Run
+- `publish-test-results` — publishes TRX / JUnit XML / NUnit XML / Cucumber JSON / Playwright JSON / CTRF JSON results to an Azure DevOps Test Run
 - `generate` — creates local spec files (.feature or .md) from Azure DevOps User Stories
 - `validate` — checks config validity and Azure connectivity
 - `init` — interactive wizard that generates ado-sync.json
@@ -133,9 +133,10 @@ A `.env` file in the working directory is loaded automatically.
 
 ## MCP server
 
-`ado-sync-mcp` starts an MCP stdio server exposing 10 tools:
+`ado-sync-mcp` starts an MCP stdio server exposing 13 tools:
 `validate_config`, `get_test_cases`, `get_test_case`, `push_specs`, `pull_specs`,
-`status`, `diff`, `generate_specs`, `get_work_items`, `publish_test_results`
+`status`, `diff`, `generate_specs`, `get_work_items`, `publish_test_results`,
+`create_issue`, `get_story_context`, `generate_manifest`
 
 Register in Claude Code (`~/.claude/claude_desktop_config.json`):
 ```json

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ado-sync",
-  "version": "0.1.47",
+  "version": "0.1.48",
   "description": "Bidirectional sync between local test specs (Cucumber/Markdown) and Azure DevOps Test Cases",
   "bin": {
     "ado-sync": "./dist/cli.js",