ado-sync 0.1.46 → 0.1.48

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (53) hide show
  1. package/.next-steps.md +179 -0
  2. package/README.md +22 -945
  3. package/dist/azure/test-cases.js +39 -6
  4. package/dist/azure/test-cases.js.map +1 -1
  5. package/dist/azure/work-items.d.ts +60 -0
  6. package/dist/azure/work-items.js +194 -0
  7. package/dist/azure/work-items.js.map +1 -0
  8. package/dist/cli.js +396 -28
  9. package/dist/cli.js.map +1 -1
  10. package/dist/config.js +50 -0
  11. package/dist/config.js.map +1 -1
  12. package/dist/issues/ado-bugs.d.ts +23 -0
  13. package/dist/issues/ado-bugs.js +59 -0
  14. package/dist/issues/ado-bugs.js.map +1 -0
  15. package/dist/issues/create-issues.d.ts +32 -0
  16. package/dist/issues/create-issues.js +236 -0
  17. package/dist/issues/create-issues.js.map +1 -0
  18. package/dist/issues/github.d.ts +22 -0
  19. package/dist/issues/github.js +95 -0
  20. package/dist/issues/github.js.map +1 -0
  21. package/dist/mcp-server.d.ts +35 -0
  22. package/dist/mcp-server.js +507 -0
  23. package/dist/mcp-server.js.map +1 -0
  24. package/dist/sync/cache.d.ts +7 -1
  25. package/dist/sync/cache.js.map +1 -1
  26. package/dist/sync/engine.js +105 -51
  27. package/dist/sync/engine.js.map +1 -1
  28. package/dist/sync/generate.d.ts +27 -0
  29. package/dist/sync/generate.js +184 -0
  30. package/dist/sync/generate.js.map +1 -0
  31. package/dist/sync/manifest.d.ts +69 -0
  32. package/dist/sync/manifest.js +197 -0
  33. package/dist/sync/manifest.js.map +1 -0
  34. package/dist/sync/publish-results.d.ts +8 -1
  35. package/dist/sync/publish-results.js +139 -2
  36. package/dist/sync/publish-results.js.map +1 -1
  37. package/dist/types.d.ts +55 -0
  38. package/docs/agent-setup.md +204 -0
  39. package/docs/cli.md +241 -0
  40. package/docs/mcp-server.md +275 -0
  41. package/docs/publish-test-results.md +136 -2
  42. package/docs/troubleshooting.md +101 -0
  43. package/docs/work-item-links.md +115 -0
  44. package/docs/workflows.md +381 -0
  45. package/llms.txt +164 -0
  46. package/package.json +4 -2
  47. package/.cucumber/.ado-sync-state.json +0 -282
  48. package/.cucumber/ado-sync.yaml +0 -21
  49. package/.cucumber/features/cart.feature +0 -62
  50. package/.cucumber/features/checkout.feature +0 -100
  51. package/.cucumber/features/homepage.feature +0 -7
  52. package/.cucumber/features/inventory.feature +0 -59
  53. package/.cucumber/features/login.feature +0 -74
package/docs/cli.md ADDED
@@ -0,0 +1,241 @@
1
+ # CLI Reference
2
+
3
+ ```
4
+ ado-sync [options] [command]
5
+
6
+ Options:
7
+ -c, --config <path> Path to config file (default: ado-sync.json)
8
+ --output <format> Output format: text (default) or json
9
+ -V, --version Print version
10
+ -h, --help Show help
11
+
12
+ Commands:
13
+ init Generate a starter config file (interactive wizard)
14
+ validate Check config and Azure DevOps connectivity
15
+ push Push local specs to Azure DevOps
16
+ pull Pull updates from Azure DevOps into local files
17
+ status Show diff without making changes
18
+ diff Show field-level diff between local and Azure
19
+ generate Generate local spec files from ADO User Stories
20
+ publish-test-results Publish TRX / JUnit / Cucumber JSON results
21
+ help [command] Help for a specific command
22
+ ```
23
+
24
+ ---
25
+
26
+ ## `init`
27
+
28
+ Generate a starter config file. Runs an **interactive wizard** when stdin is a TTY.
29
+
30
+ ```bash
31
+ ado-sync init # creates ado-sync.json (wizard)
32
+ ado-sync init ado-sync.yml # YAML format
33
+ ado-sync init --no-interactive # dump template without prompting
34
+ ```
35
+
36
+ The wizard asks for org URL, project, auth type, token env var, test plan ID, local spec type, and include glob — then writes a minimal valid config.
37
+
38
+ ---
39
+
40
+ ## `validate`
41
+
42
+ Check that the config is valid and Azure DevOps is reachable. Run this before your first `push`.
43
+
44
+ ```bash
45
+ ado-sync validate
46
+ ado-sync validate -c path/to/ado-sync.json
47
+ ```
48
+
49
+ Output:
50
+ ```
51
+ ✓ Config loaded — /project/ado-sync.json
52
+ ✓ Azure connection — https://dev.azure.com/myorg
53
+ ✓ Project "MyProject" found
54
+ ✓ Test Plan #42 "Regression Suite" found
55
+
56
+ All checks passed.
57
+ ```
58
+
59
+ ---
60
+
61
+ ## `push`
62
+
63
+ Push local spec files to Azure DevOps — creates new Test Cases or updates existing ones.
64
+
65
+ ```bash
66
+ ado-sync push
67
+ ado-sync push --dry-run
68
+ ado-sync push --tags "@smoke and not @wip"
69
+ ado-sync push --config-override testPlan.id=9999
70
+
71
+ # AI-generated test steps for code files
72
+ ado-sync push --ai-provider heuristic # fast regex, no model needed
73
+ ado-sync push --ai-provider local --ai-model ~/.cache/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
74
+ ado-sync push --ai-provider ollama --ai-model qwen2.5-coder:7b
75
+ ado-sync push --ai-provider openai --ai-key $OPENAI_API_KEY
76
+ ado-sync push --ai-provider anthropic --ai-key $ANTHROPIC_API_KEY
77
+ ado-sync push --ai-provider none # disable AI entirely
78
+ ado-sync push --ai-context ./docs/ai-context.md # inject domain context
79
+ ```
80
+
81
+ | Scenario | Action |
82
+ |----------|--------|
83
+ | No ID tag | Creates a new Test Case, writes ID back |
84
+ | ID tag, no changes | Skipped |
85
+ | ID tag, content changed | Updates the existing Test Case |
86
+ | Deleted locally, still in Azure | Tagged `ado-sync:removed` in Azure |
87
+
88
+ ### AI auto-summary
89
+
90
+ For code-based types (`java`, `csharp`, `python`, `javascript`, `playwright`, `cypress`, `testcafe`, `detox`, `espresso`, `xcuitest`, `flutter`), ado-sync reads test function bodies and generates a TC **title**, **description**, and **steps** automatically.
91
+
92
+ > No setup required — `push` always works, falling back to fast heuristic analysis.
93
+
94
+ | Provider | Quality | Setup |
95
+ |---|---|---|
96
+ | `local` *(default)* | Good–Excellent | Download a GGUF model (see below) |
97
+ | `heuristic` | Basic | None — offline, zero dependencies |
98
+ | `ollama` | Good–Excellent | `ollama pull qwen2.5-coder:7b` |
99
+ | `openai` | Excellent | `--ai-key $OPENAI_API_KEY` |
100
+ | `anthropic` | Excellent | `--ai-key $ANTHROPIC_API_KEY` |
101
+ | `openai` + `--ai-url` | Excellent | Any OpenAI-compatible proxy (LiteLLM, Azure OpenAI, vLLM) |
102
+
103
+ #### Local LLM — model download
104
+
105
+ ```bash
106
+ # macOS / Linux
107
+ mkdir -p ~/.cache/ado-sync/models
108
+ curl -L -o ~/.cache/ado-sync/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf \
109
+ "https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF/resolve/main/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf"
110
+
111
+ ado-sync push --ai-model ~/.cache/ado-sync/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
112
+ ```
113
+
114
+ | Model | RAM | Quality |
115
+ |-------|-----|---------|
116
+ | 1.5B Q4_K_M | ~1.1 GB | Good |
117
+ | 7B Q4_K_M | ~4.5 GB | Better |
118
+ | 14B Q4_K_M | ~8.5 GB | Excellent |
119
+
120
+ #### Inject domain context
121
+
122
+ ```bash
123
+ ado-sync push --ai-context ./docs/ai-context.md
124
+ ```
125
+
126
+ Or in config:
127
+ ```json
128
+ { "sync": { "ai": { "provider": "anthropic", "contextFile": "./docs/ai-context.md" } } }
129
+ ```
130
+
131
+ #### Freeze steps in source files
132
+
133
+ Enable `writebackDocComment: true` to write AI-generated steps as JSDoc comments above each `test()` call. On subsequent pushes the parser reads the JSDoc, so AI is not re-invoked.
134
+
135
+ ```json
136
+ { "sync": { "ai": { "writebackDocComment": true } } }
137
+ ```
138
+
139
+ ---
140
+
141
+ ## `pull`
142
+
143
+ Pull Azure DevOps Test Case changes into local spec files.
144
+
145
+ ```bash
146
+ ado-sync pull
147
+ ado-sync pull --dry-run
148
+ ado-sync pull --tags "@smoke"
149
+ ```
150
+
151
+ ---
152
+
153
+ ## `status`
154
+
155
+ Show what would change on the next push without making any modifications.
156
+
157
+ ```bash
158
+ ado-sync status
159
+ ado-sync status --tags "@smoke"
160
+ ado-sync status --output json # machine-readable
161
+ ```
162
+
163
+ ---
164
+
165
+ ## `diff`
166
+
167
+ Show a field-level diff between local specs and Azure DevOps — richer than `status`.
168
+
169
+ ```bash
170
+ ado-sync diff
171
+ ado-sync diff --tags "@smoke"
172
+ ```
173
+
174
+ Output example:
175
+ ```
176
+ ~ specs/login.feature:12 · Login with valid credentials [#1234]
177
+ changed fields: title, steps
178
+ ```
179
+
180
+ ---
181
+
182
+ ## `generate`
183
+
184
+ Generate local spec files (`.feature` or `.md`) from Azure DevOps User Stories, pulling title, description, and acceptance criteria to scaffold each file.
185
+
186
+ ```bash
187
+ # By explicit work item IDs
188
+ ado-sync generate --story-ids 1234,5678
189
+
190
+ # By area path (fetches all User Stories under it)
191
+ ado-sync generate --area-path "MyProject\\\\Teams\\\\QA"
192
+
193
+ # By WIQL query
194
+ ado-sync generate --query "SELECT [System.Id] FROM WorkItems WHERE [System.WorkItemType]='User Story' AND [System.State]='Active'"
195
+
196
+ # Options
197
+ ado-sync generate --story-ids 1234 --format gherkin # output .feature files
198
+ ado-sync generate --story-ids 1234 --format markdown # output .md files (default)
199
+ ado-sync generate --story-ids 1234 --output-folder specs/generated
200
+ ado-sync generate --story-ids 1234 --force # overwrite existing files
201
+ ado-sync generate --story-ids 1234 --dry-run # preview without writing
202
+ ```
203
+
204
+ ---
205
+
206
+ ## `publish-test-results`
207
+
208
+ Publish test results from TRX, JUnit XML, NUnit XML, Cucumber JSON, or Playwright JSON files to Azure DevOps Test Runs.
209
+
210
+ ```bash
211
+ ado-sync publish-test-results --testResult results/test.trx
212
+ ado-sync publish-test-results --testResult results/test.xml --testResultFormat junit --dry-run
213
+ ado-sync publish-test-results --testResult results/playwright.json --attachmentsFolder test-results/
214
+ ```
215
+
216
+ See [publish-test-results.md](publish-test-results.md) for full reference.
217
+
218
+ ---
219
+
220
+ ## `--config-override`
221
+
222
+ All commands accept `--config-override path=value` (repeatable) to set config values at runtime:
223
+
224
+ ```bash
225
+ ado-sync push --config-override testPlan.id=9999
226
+ ado-sync push --config-override sync.disableLocalChanges=true
227
+ ado-sync push --config-override sync.tagPrefix=test --config-override testPlan.id=42
228
+ ```
229
+
230
+ ---
231
+
232
+ ## `--output json`
233
+
234
+ All sync commands support `--output json` for machine-readable output:
235
+
236
+ ```bash
237
+ ado-sync status --output json | jq '.[] | select(.action=="updated")'
238
+ ado-sync push --output json > results.json
239
+ ```
240
+
241
+ Each result object: `{ action, filePath, title, azureId?, detail?, changedFields? }`
@@ -0,0 +1,275 @@
1
+ # MCP Server
2
+
3
+ `ado-sync` ships a built-in MCP (Model Context Protocol) server that exposes its sync operations as **structured tools** for AI agents — Claude Code, GitHub Copilot, Cursor, and any other MCP-compatible host.
4
+
5
+ Using the MCP server instead of spawning the CLI gives agents:
6
+ - Typed JSON responses (no output parsing)
7
+ - Structured error objects with status codes
8
+ - Fine-grained tool calls (get one test case, push a subset, etc.)
9
+
10
+ ---
11
+
12
+ ## Installation
13
+
14
+ No separate install needed. `ado-sync-mcp` ships inside the `ado-sync` npm package.
15
+
16
+ **Option A — Global install (recommended, fastest startup)**
17
+ ```bash
18
+ npm install -g ado-sync
19
+ ```
20
+
21
+ **Option B — npx (no install, always latest)**
22
+
23
+ Use `npx --yes --package=ado-sync ado-sync-mcp` as the command in all configs below.
24
+ npx downloads and runs the package automatically on first use.
25
+
26
+ ---
27
+
28
+ ## Claude Code — one-liner registration
29
+
30
+ ```bash
31
+ # With global install
32
+ claude mcp add ado-sync \
33
+ --env AZURE_DEVOPS_TOKEN="$AZURE_DEVOPS_TOKEN" \
34
+ --env ADO_SYNC_CONFIG="$(pwd)/ado-sync.json" \
35
+ -- ado-sync-mcp
36
+
37
+ # With npx (no global install required)
38
+ claude mcp add ado-sync \
39
+ --env AZURE_DEVOPS_TOKEN="$AZURE_DEVOPS_TOKEN" \
40
+ --env ADO_SYNC_CONFIG="$(pwd)/ado-sync.json" \
41
+ -- npx --yes --package=ado-sync ado-sync-mcp
42
+ ```
43
+
44
+ Verify it registered: run `/mcp` in Claude Code — `ado-sync` should appear in the list.
45
+
46
+ **Manual config** — `~/.claude/claude_desktop_config.json`:
47
+ ```json
48
+ {
49
+ "mcpServers": {
50
+ "ado-sync": {
51
+ "command": "npx",
52
+ "args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
53
+ "env": {
54
+ "AZURE_DEVOPS_TOKEN": "<your-pat>",
55
+ "ADO_SYNC_CONFIG": "/absolute/path/to/ado-sync.json"
56
+ }
57
+ }
58
+ }
59
+ }
60
+ ```
61
+
62
+ ---
63
+
64
+ ## Claude Desktop
65
+
66
+ Config file location:
67
+ - macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
68
+ - Windows: `%APPDATA%\Claude\claude_desktop_config.json`
69
+
70
+ ```json
71
+ {
72
+ "mcpServers": {
73
+ "ado-sync": {
74
+ "command": "npx",
75
+ "args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
76
+ "env": {
77
+ "AZURE_DEVOPS_TOKEN": "<your-pat>",
78
+ "ADO_SYNC_CONFIG": "/absolute/path/to/ado-sync.json"
79
+ }
80
+ }
81
+ }
82
+ }
83
+ ```
84
+
85
+ Restart Claude Desktop after saving.
86
+
87
+ ---
88
+
89
+ ## VS Code (GitHub Copilot agent mode)
90
+
91
+ Create `.vscode/mcp.json` in your workspace root:
92
+
93
+ ```json
94
+ {
95
+ "servers": {
96
+ "ado-sync": {
97
+ "type": "stdio",
98
+ "command": "npx",
99
+ "args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
100
+ "env": {
101
+ "AZURE_DEVOPS_TOKEN": "${env:AZURE_DEVOPS_TOKEN}",
102
+ "ADO_SYNC_CONFIG": "${workspaceFolder}/ado-sync.json"
103
+ }
104
+ }
105
+ }
106
+ }
107
+ ```
108
+
109
+ `${env:AZURE_DEVOPS_TOKEN}` reads the variable from your shell environment — no hardcoded secrets.
110
+
111
+ ---
112
+
113
+ ## Cursor
114
+
115
+ `~/.cursor/mcp.json`:
116
+
117
+ ```json
118
+ {
119
+ "mcpServers": {
120
+ "ado-sync": {
121
+ "command": "npx",
122
+ "args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
123
+ "env": {
124
+ "AZURE_DEVOPS_TOKEN": "<your-pat>",
125
+ "ADO_SYNC_CONFIG": "/absolute/path/to/ado-sync.json"
126
+ }
127
+ }
128
+ }
129
+ }
130
+ ```
131
+
132
+ ---
133
+
134
+ ## Verify it works
135
+
136
+ After registration, ask your AI assistant:
137
+
138
+ ```
139
+ "Call ado-sync validate_config and tell me what it returns"
140
+ ```
141
+
142
+ Expected response includes config validity, Azure connection status, and test plan confirmation.
143
+
144
+ ---
145
+
146
+ ## Available tools
147
+
148
+ | Tool | Description |
149
+ |------|-------------|
150
+ | `validate_config` | Check config validity and Azure DevOps connectivity |
151
+ | `get_test_cases` | List all Test Cases in a suite |
152
+ | `get_test_case` | Get a single Test Case by ID |
153
+ | `push_specs` | Push local spec files to Azure DevOps |
154
+ | `pull_specs` | Pull Azure DevOps changes into local files |
155
+ | `status` | Show diff between local and Azure without making changes |
156
+ | `diff` | Show field-level diff (richer than status) |
157
+ | `generate_specs` | Generate local spec files from ADO User Stories |
158
+ | `get_work_items` | Fetch ADO User Stories / work items |
159
+ | `publish_test_results` | Publish TRX / JUnit / Playwright JSON / CTRF results; optionally file issues for failures |
160
+ | `create_issue` | File a single GitHub Issue or ADO Bug for a test failure (for healer agents) |
161
+ | `get_story_context` | Planner-agent feed: AC items, suggested tags, actors, linked TC IDs |
162
+ | `generate_manifest` | Write `.ai-workflow-manifest-{id}.json` for the full Planner→CI cycle |
163
+
164
+ ---
165
+
166
+ ## Tool parameters
167
+
168
+ All tools accept these optional base parameters:
169
+
170
+ | Parameter | Type | Description |
171
+ |-----------|------|-------------|
172
+ | `configPath` | string | Absolute path to `ado-sync.json` (overrides `ADO_SYNC_CONFIG`) |
173
+ | `configOverrides` | string[] | Runtime overrides in `key=value` format (same as `--config-override`) |
174
+
175
+ ### `push_specs`
176
+
177
+ ```json
178
+ {
179
+ "dryRun": false,
180
+ "tags": "@smoke and not @wip",
181
+ "aiProvider": "heuristic"
182
+ }
183
+ ```
184
+
185
+ ### `generate_specs`
186
+
187
+ ```json
188
+ {
189
+ "storyIds": [1234, 5678],
190
+ "format": "gherkin",
191
+ "outputFolder": "specs/generated",
192
+ "dryRun": false,
193
+ "force": false
194
+ }
195
+ ```
196
+
197
+ ### `publish_test_results`
198
+
199
+ ```json
200
+ {
201
+ "testResult": "results/playwright.json",
202
+ "attachmentsFolder": "test-results/",
203
+ "createIssuesOnFailure": true,
204
+ "issueProvider": "github",
205
+ "githubRepo": "myorg/myrepo",
206
+ "githubToken": "$GITHUB_TOKEN",
207
+ "bugThreshold": 20,
208
+ "maxIssues": 50
209
+ }
210
+ ```
211
+
212
+ ### `create_issue`
213
+
214
+ ```json
215
+ {
216
+ "title": "[FAILED] Login with valid credentials",
217
+ "body": "Error: Expected 200 but got 401\n\nStack: ...",
218
+ "provider": "github",
219
+ "githubRepo": "myorg/myrepo",
220
+ "githubToken": "$GITHUB_TOKEN",
221
+ "testCaseId": 1234
222
+ }
223
+ ```
224
+
225
+ ### `get_story_context`
226
+
227
+ ```json
228
+ { "storyId": 1234 }
229
+ ```
230
+
231
+ Returns: AC items as a bullet list, inferred tags (`@smoke`, `@auth`, …), extracted actors, and IDs of any Test Cases already linked via TestedBy relation.
232
+
233
+ ### `generate_manifest`
234
+
235
+ ```json
236
+ {
237
+ "storyIds": [1234, 5678],
238
+ "outputFolder": "e2e/bdd",
239
+ "format": "gherkin",
240
+ "dryRun": false
241
+ }
242
+ ```
243
+
244
+ Writes `.ai-workflow-manifest-1234.json` in `outputFolder`. The manifest contains the ordered 8-step workflow, AC items, required documents checklist, and validation steps.
245
+
246
+ ---
247
+
248
+ ## Example agent workflow
249
+
250
+ Once registered, an AI agent can orchestrate the full test lifecycle without any human CLI invocation:
251
+
252
+ ```
253
+ Agent: "Set up ado-sync for this repo"
254
+ → calls validate_config to check connectivity
255
+ → calls push_specs({ dryRun: true }) to preview
256
+ → calls push_specs({}) to create Test Cases
257
+
258
+ Agent: "Generate spec files for User Story 1234"
259
+ → calls generate_specs({ storyIds: [1234], format: "gherkin" })
260
+
261
+ Agent: "Publish the latest test results and file issues for failures"
262
+ → calls publish_test_results({ testResult: "results/playwright.json", createIssuesOnFailure: true, githubRepo: "myorg/myrepo", githubToken: "$GITHUB_TOKEN" })
263
+ → returns issue URLs for any failures → healer agent opens fix PRs
264
+ ```
265
+
266
+ ---
267
+
268
+ ## Environment variables
269
+
270
+ | Variable | Description |
271
+ |----------|-------------|
272
+ | `ADO_SYNC_CONFIG` | Absolute path to config file (default: `ado-sync.json` in cwd) |
273
+ | `AZURE_DEVOPS_TOKEN` | PAT or access token (referenced by `auth.token` in config) |
274
+
275
+ > **Important:** `ADO_SYNC_CONFIG` must be an **absolute path**. The MCP server process does not inherit the shell's working directory the same way as a CLI call.
@@ -1,6 +1,6 @@
1
1
  # publish-test-results
2
2
 
3
- Parses test result files (TRX, NUnit XML, JUnit, Cucumber JSON, Playwright JSON) and publishes them to an Azure DevOps Test Run, linking results back to Test Cases either directly by TC ID (when available in the result file) or by `AutomatedTestName` matching.
3
+ Parses test result files (TRX, NUnit XML, JUnit, Cucumber JSON, Playwright JSON, CTRF JSON) and publishes them to an Azure DevOps Test Run, linking results back to Test Cases either directly by TC ID (when available in the result file) or by `AutomatedTestName` matching.
4
4
 
5
5
  ---
6
6
 
@@ -31,11 +31,17 @@ ado-sync publish-test-results \
31
31
  | Option | Description |
32
32
  |--------|-------------|
33
33
  | `--testResult <path>` | Path to a result file. Repeatable. |
34
- | `--testResultFormat <format>` | `trx` · `nunitXml` · `junit` · `cucumberJson` · `playwrightJson`. Auto-detected when omitted. |
34
+ | `--testResultFormat <format>` | `trx` · `nunitXml` · `junit` · `cucumberJson` · `playwrightJson` · `ctrfJson`. Auto-detected when omitted. |
35
35
  | `--attachmentsFolder <path>` | Folder to scan for screenshots/videos/logs to attach to test results. |
36
36
  | `--runName <name>` | Name for the Test Run in Azure DevOps. Defaults to `ado-sync <ISO timestamp>`. |
37
37
  | `--buildId <id>` | Build ID to associate with the Test Run. |
38
38
  | `--dry-run` | Parse results and print summary without creating a run in Azure. |
39
+ | `--create-issues-on-failure` | File GitHub Issues or ADO Bugs for each failed test after publishing. |
40
+ | `--issue-provider <github\|ado>` | Issue provider. Default: `github`. |
41
+ | `--github-repo <owner/repo>` | GitHub repository to file issues in. |
42
+ | `--github-token <token>` | GitHub token. Supports `$ENV_VAR` references. |
43
+ | `--bug-threshold <percent>` | If more than this % of tests fail, one environment-failure issue is filed instead of per-test issues. Default: `20`. |
44
+ | `--max-issues <n>` | Hard cap on issues filed per run. Default: `50`. |
39
45
  | `--config-override` | Override config values (repeatable, same as other commands). |
40
46
 
41
47
  ---
@@ -50,6 +56,7 @@ ado-sync publish-test-results \
50
56
  | Cucumber JSON | `.json` | Yes (JSON array, Cucumber format) | Yes — via `@tc:ID` tag on scenario | `step.embeddings[]` (base64 screenshots/video) |
51
57
  | Playwright JSON | `.json` | Yes (JSON object with `suites` key) | Yes — via `test.annotations[{ type: 'tc', description: 'ID' }]` (preferred) or `@tc:ID` in test title | `test.results[].attachments[]` (screenshots, videos, traces) |
52
58
  | Robot Framework XML | `output.xml` | Yes (`<robot>` root element) | Yes — via `<tags><tag>tc:ID</tag></tags>` | — |
59
+ | CTRF JSON | `.json` | Yes (`results.tests` array) | Yes — via `tags: ["@tc:ID"]` or `@tc:ID` in test name | `attachments[].path` files, `stdout`/`stderr` arrays |
53
60
 
54
61
  > **NUnit via TRX**: when NUnit tests are run through the VSTest adapter (`--logger trx`), `[Property]` values are **not** included in the TRX output. Use `--logger "nunit3;LogFileName=results.xml"` to get the native NUnit XML format, which does include property values.
55
62
 
@@ -441,6 +448,38 @@ Recommended config:
441
448
 
442
449
  ---
443
450
 
451
+ ### CTRF (Common Test Report Format)
452
+
453
+ [CTRF](https://ctrf.io) is a framework-agnostic JSON report format supported by reporters for Playwright, Cypress, Jest, k6, and many others. ado-sync auto-detects CTRF from the `results.tests` array structure.
454
+
455
+ ```bash
456
+ # Example: Playwright with CTRF reporter
457
+ npm install --save-dev playwright-ctrf-json-reporter
458
+
459
+ # playwright.config.ts:
460
+ # reporter: [['playwright-ctrf-json-reporter', { outputFile: 'results/ctrf.json' }]]
461
+
462
+ npx playwright test
463
+ ado-sync publish-test-results --testResult results/ctrf.json
464
+ ```
465
+
466
+ ```bash
467
+ # Example: Jest with CTRF reporter
468
+ npm install --save-dev jest-ctrf-json-reporter
469
+
470
+ # jest.config.ts:
471
+ # reporters: [['jest-ctrf-json-reporter', { outputFile: 'results/ctrf.json' }]]
472
+
473
+ npx jest
474
+ ado-sync publish-test-results --testResult results/ctrf.json
475
+ ```
476
+
477
+ TC IDs are extracted from the `tags` array (e.g. `["@tc:1234", "@smoke"]`) or, as a fallback, from `@tc:ID` in the test name. `stdout`/`stderr` arrays and `attachments[].path` files are uploaded automatically.
478
+
479
+ > **Status mapping**: CTRF `passed` → `Passed`, `failed` → `Failed`, `skipped`/`pending`/`other` → `NotExecuted`.
480
+
481
+ ---
482
+
444
483
  ### Flutter
445
484
 
446
485
  Flutter can produce JUnit XML via the `flutter_test_junit` package or by piping `--reporter junit`:
@@ -483,6 +522,7 @@ TC linking uses `AutomatedTestName` matching — set `sync.markAutomated: true`
483
522
  | Espresso | JUnit XML | ❌ AutomatedTestName matching only | `<system-out>`, `<system-err>` | |
484
523
  | Flutter | JUnit XML | ❌ AutomatedTestName matching only | `<system-out>`, `<system-err>` | |
485
524
  | Robot Framework | Robot XML (`output.xml`) | ✅ `tc:N` in `<tags>` | — | |
525
+ | CTRF (any framework) | CTRF JSON | ✅ `tags: ["@tc:ID"]` or `@tc:ID` in name | `attachments[].path` files + `stdout`/`stderr` | |
486
526
 
487
527
  ---
488
528
 
@@ -499,6 +539,7 @@ ado-sync uploads screenshots, videos, and logs from test results to the correspo
499
539
  | JUnit XML | `<system-out>` → log; `<system-err>` → log; `[[ATTACHMENT\|path]]` → Playwright files |
500
540
  | Cucumber JSON | `step.embeddings[]` → base64-encoded screenshots/video |
501
541
  | Playwright JSON | `results[].attachments[].path` → files on disk (screenshots, videos, traces) |
542
+ | CTRF JSON | `tests[].attachments[].path` → files on disk; `tests[].stdout[]` / `tests[].stderr[]` → console logs |
502
543
 
503
544
  > **Note**: All file paths are resolved relative to the result file's directory, not the process working directory. This matches how test runners (Playwright, MSTest, NUnit) write relative paths in their output.
504
545
 
@@ -670,6 +711,99 @@ Results can also be configured in the config file under `publishTestResults`:
670
711
 
671
712
  ---
672
713
 
714
+ ## Creating issues on failure
715
+
716
+ `--create-issues-on-failure` automatically files a GitHub Issue or ADO Bug for each failed test
717
+ after the run is published. Multiple guards prevent flooding your tracker when the environment is
718
+ the problem rather than individual tests.
719
+
720
+ ### Guard logic (applied in order)
721
+
722
+ ```
723
+ failures > threshold% of total?
724
+ └─ YES → 1 environment-failure issue, stop
725
+ └─ NO
726
+ └─ cluster by error signature
727
+ └─ cluster size > 1?
728
+ └─ YES → 1 issue per cluster (lists affected test names)
729
+ └─ NO → 1 issue per TC (up to maxIssues cap)
730
+ └─ cap hit? → 1 overflow summary issue
731
+ ```
732
+
733
+ | Guard | Default | Description |
734
+ |---|---|---|
735
+ | Failure-rate threshold | 20% | Above this, one env-failure issue is filed instead of per-test |
736
+ | Error clustering | enabled | Tests with the same error message are grouped into one issue |
737
+ | Hard cap | 50 | No more than this many issues per run; one overflow summary when exceeded |
738
+ | Dedup | enabled | Skip if an open issue already exists for the same TC (GitHub: by `tc:ID` label; ADO: by title) |
739
+
740
+ ### GitHub Issues (recommended)
741
+
742
+ ```bash
743
+ ado-sync publish-test-results \
744
+ --testResult results/ctrf.json \
745
+ --create-issues-on-failure \
746
+ --github-repo myorg/myrepo \
747
+ --github-token $GITHUB_TOKEN
748
+ ```
749
+
750
+ Each issue is labelled `test-failure` and `tc:{ID}` (when a TC ID is available). The issue body
751
+ contains the error message, stack trace, ADO TC link, and run URL — everything a healer agent
752
+ needs to propose a fix PR.
753
+
754
+ ### ADO Bugs
755
+
756
+ ```bash
757
+ ado-sync publish-test-results \
758
+ --testResult results/junit.xml \
759
+ --create-issues-on-failure \
760
+ --issue-provider ado
761
+ ```
762
+
763
+ ADO Bugs are created as Bug work items in the same project. The `Repro Steps` field is populated
764
+ with the error details. When a TC ID is known, a `TestedBy` relation is added linking the Bug to
765
+ the Test Case.
766
+
767
+ ### Config-based setup
768
+
769
+ ```json
770
+ {
771
+ "publishTestResults": {
772
+ "createIssuesOnFailure": {
773
+ "provider": "github",
774
+ "repo": "myorg/myrepo",
775
+ "token": "$GITHUB_TOKEN",
776
+ "labels": ["test-failure", "automated"],
777
+ "threshold": 20,
778
+ "maxIssues": 50,
779
+ "clusterByError": true,
780
+ "dedupByTestCase": true
781
+ }
782
+ }
783
+ }
784
+ ```
785
+
786
+ CLI flags override the config values when both are present.
787
+
788
+ ### MCP tool: `create_issue`
789
+
790
+ The `create_issue` MCP tool lets healer agents file a single issue directly:
791
+
792
+ ```
793
+ create_issue({
794
+ title: "[FAILED] Login with valid credentials",
795
+ body: "Error: Expected 200 but got 401\n\nStack: ...",
796
+ provider: "github",
797
+ githubRepo: "myorg/myrepo",
798
+ githubToken: "$GITHUB_TOKEN",
799
+ testCaseId: 1234
800
+ })
801
+ ```
802
+
803
+ Returns the issue URL immediately, which the agent can embed in its fix PR.
804
+
805
+ ---
806
+
673
807
  ## Output
674
808
 
675
809
  ```