ado-sync 0.1.46 → 0.1.48
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.next-steps.md +179 -0
- package/README.md +22 -945
- package/dist/azure/test-cases.js +39 -6
- package/dist/azure/test-cases.js.map +1 -1
- package/dist/azure/work-items.d.ts +60 -0
- package/dist/azure/work-items.js +194 -0
- package/dist/azure/work-items.js.map +1 -0
- package/dist/cli.js +396 -28
- package/dist/cli.js.map +1 -1
- package/dist/config.js +50 -0
- package/dist/config.js.map +1 -1
- package/dist/issues/ado-bugs.d.ts +23 -0
- package/dist/issues/ado-bugs.js +59 -0
- package/dist/issues/ado-bugs.js.map +1 -0
- package/dist/issues/create-issues.d.ts +32 -0
- package/dist/issues/create-issues.js +236 -0
- package/dist/issues/create-issues.js.map +1 -0
- package/dist/issues/github.d.ts +22 -0
- package/dist/issues/github.js +95 -0
- package/dist/issues/github.js.map +1 -0
- package/dist/mcp-server.d.ts +35 -0
- package/dist/mcp-server.js +507 -0
- package/dist/mcp-server.js.map +1 -0
- package/dist/sync/cache.d.ts +7 -1
- package/dist/sync/cache.js.map +1 -1
- package/dist/sync/engine.js +105 -51
- package/dist/sync/engine.js.map +1 -1
- package/dist/sync/generate.d.ts +27 -0
- package/dist/sync/generate.js +184 -0
- package/dist/sync/generate.js.map +1 -0
- package/dist/sync/manifest.d.ts +69 -0
- package/dist/sync/manifest.js +197 -0
- package/dist/sync/manifest.js.map +1 -0
- package/dist/sync/publish-results.d.ts +8 -1
- package/dist/sync/publish-results.js +139 -2
- package/dist/sync/publish-results.js.map +1 -1
- package/dist/types.d.ts +55 -0
- package/docs/agent-setup.md +204 -0
- package/docs/cli.md +241 -0
- package/docs/mcp-server.md +275 -0
- package/docs/publish-test-results.md +136 -2
- package/docs/troubleshooting.md +101 -0
- package/docs/work-item-links.md +115 -0
- package/docs/workflows.md +381 -0
- package/llms.txt +164 -0
- package/package.json +4 -2
- package/.cucumber/.ado-sync-state.json +0 -282
- package/.cucumber/ado-sync.yaml +0 -21
- package/.cucumber/features/cart.feature +0 -62
- package/.cucumber/features/checkout.feature +0 -100
- package/.cucumber/features/homepage.feature +0 -7
- package/.cucumber/features/inventory.feature +0 -59
- package/.cucumber/features/login.feature +0 -74
package/docs/cli.md
ADDED
|
@@ -0,0 +1,241 @@
|
|
|
1
|
+
# CLI Reference
|
|
2
|
+
|
|
3
|
+
```
|
|
4
|
+
ado-sync [options] [command]
|
|
5
|
+
|
|
6
|
+
Options:
|
|
7
|
+
-c, --config <path> Path to config file (default: ado-sync.json)
|
|
8
|
+
--output <format> Output format: text (default) or json
|
|
9
|
+
-V, --version Print version
|
|
10
|
+
-h, --help Show help
|
|
11
|
+
|
|
12
|
+
Commands:
|
|
13
|
+
init Generate a starter config file (interactive wizard)
|
|
14
|
+
validate Check config and Azure DevOps connectivity
|
|
15
|
+
push Push local specs to Azure DevOps
|
|
16
|
+
pull Pull updates from Azure DevOps into local files
|
|
17
|
+
status Show diff without making changes
|
|
18
|
+
diff Show field-level diff between local and Azure
|
|
19
|
+
generate Generate local spec files from ADO User Stories
|
|
20
|
+
publish-test-results Publish TRX / JUnit / Cucumber JSON results
|
|
21
|
+
help [command] Help for a specific command
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## `init`
|
|
27
|
+
|
|
28
|
+
Generate a starter config file. Runs an **interactive wizard** when stdin is a TTY.
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
ado-sync init # creates ado-sync.json (wizard)
|
|
32
|
+
ado-sync init ado-sync.yml # YAML format
|
|
33
|
+
ado-sync init --no-interactive # dump template without prompting
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
The wizard asks for org URL, project, auth type, token env var, test plan ID, local spec type, and include glob — then writes a minimal valid config.
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
## `validate`
|
|
41
|
+
|
|
42
|
+
Check that the config is valid and Azure DevOps is reachable. Run this before your first `push`.
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
ado-sync validate
|
|
46
|
+
ado-sync validate -c path/to/ado-sync.json
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Output:
|
|
50
|
+
```
|
|
51
|
+
✓ Config loaded — /project/ado-sync.json
|
|
52
|
+
✓ Azure connection — https://dev.azure.com/myorg
|
|
53
|
+
✓ Project "MyProject" found
|
|
54
|
+
✓ Test Plan #42 "Regression Suite" found
|
|
55
|
+
|
|
56
|
+
All checks passed.
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## `push`
|
|
62
|
+
|
|
63
|
+
Push local spec files to Azure DevOps — creates new Test Cases or updates existing ones.
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
ado-sync push
|
|
67
|
+
ado-sync push --dry-run
|
|
68
|
+
ado-sync push --tags "@smoke and not @wip"
|
|
69
|
+
ado-sync push --config-override testPlan.id=9999
|
|
70
|
+
|
|
71
|
+
# AI-generated test steps for code files
|
|
72
|
+
ado-sync push --ai-provider heuristic # fast regex, no model needed
|
|
73
|
+
ado-sync push --ai-provider local --ai-model ~/.cache/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
|
|
74
|
+
ado-sync push --ai-provider ollama --ai-model qwen2.5-coder:7b
|
|
75
|
+
ado-sync push --ai-provider openai --ai-key $OPENAI_API_KEY
|
|
76
|
+
ado-sync push --ai-provider anthropic --ai-key $ANTHROPIC_API_KEY
|
|
77
|
+
ado-sync push --ai-provider none # disable AI entirely
|
|
78
|
+
ado-sync push --ai-context ./docs/ai-context.md # inject domain context
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
| Scenario | Action |
|
|
82
|
+
|----------|--------|
|
|
83
|
+
| No ID tag | Creates a new Test Case, writes ID back |
|
|
84
|
+
| ID tag, no changes | Skipped |
|
|
85
|
+
| ID tag, content changed | Updates the existing Test Case |
|
|
86
|
+
| Deleted locally, still in Azure | Tagged `ado-sync:removed` in Azure |
|
|
87
|
+
|
|
88
|
+
### AI auto-summary
|
|
89
|
+
|
|
90
|
+
For code-based types (`java`, `csharp`, `python`, `javascript`, `playwright`, `cypress`, `testcafe`, `detox`, `espresso`, `xcuitest`, `flutter`), ado-sync reads test function bodies and generates a TC **title**, **description**, and **steps** automatically.
|
|
91
|
+
|
|
92
|
+
> No setup required — `push` always works, falling back to fast heuristic analysis.
|
|
93
|
+
|
|
94
|
+
| Provider | Quality | Setup |
|
|
95
|
+
|---|---|---|
|
|
96
|
+
| `local` *(default)* | Good–Excellent | Download a GGUF model (see below) |
|
|
97
|
+
| `heuristic` | Basic | None — offline, zero dependencies |
|
|
98
|
+
| `ollama` | Good–Excellent | `ollama pull qwen2.5-coder:7b` |
|
|
99
|
+
| `openai` | Excellent | `--ai-key $OPENAI_API_KEY` |
|
|
100
|
+
| `anthropic` | Excellent | `--ai-key $ANTHROPIC_API_KEY` |
|
|
101
|
+
| `openai` + `--ai-url` | Excellent | Any OpenAI-compatible proxy (LiteLLM, Azure OpenAI, vLLM) |
|
|
102
|
+
|
|
103
|
+
#### Local LLM — model download
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
# macOS / Linux
|
|
107
|
+
mkdir -p ~/.cache/ado-sync/models
|
|
108
|
+
curl -L -o ~/.cache/ado-sync/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf \
|
|
109
|
+
"https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF/resolve/main/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf"
|
|
110
|
+
|
|
111
|
+
ado-sync push --ai-model ~/.cache/ado-sync/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
| Model | RAM | Quality |
|
|
115
|
+
|-------|-----|---------|
|
|
116
|
+
| 1.5B Q4_K_M | ~1.1 GB | Good |
|
|
117
|
+
| 7B Q4_K_M | ~4.5 GB | Better |
|
|
118
|
+
| 14B Q4_K_M | ~8.5 GB | Excellent |
|
|
119
|
+
|
|
120
|
+
#### Inject domain context
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
ado-sync push --ai-context ./docs/ai-context.md
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Or in config:
|
|
127
|
+
```json
|
|
128
|
+
{ "sync": { "ai": { "provider": "anthropic", "contextFile": "./docs/ai-context.md" } } }
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
#### Freeze steps in source files
|
|
132
|
+
|
|
133
|
+
Enable `writebackDocComment: true` to write AI-generated steps as JSDoc comments above each `test()` call. On subsequent pushes the parser reads the JSDoc, so AI is not re-invoked.
|
|
134
|
+
|
|
135
|
+
```json
|
|
136
|
+
{ "sync": { "ai": { "writebackDocComment": true } } }
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
---
|
|
140
|
+
|
|
141
|
+
## `pull`
|
|
142
|
+
|
|
143
|
+
Pull Azure DevOps Test Case changes into local spec files.
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
ado-sync pull
|
|
147
|
+
ado-sync pull --dry-run
|
|
148
|
+
ado-sync pull --tags "@smoke"
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
---
|
|
152
|
+
|
|
153
|
+
## `status`
|
|
154
|
+
|
|
155
|
+
Show what would change on the next push without making any modifications.
|
|
156
|
+
|
|
157
|
+
```bash
|
|
158
|
+
ado-sync status
|
|
159
|
+
ado-sync status --tags "@smoke"
|
|
160
|
+
ado-sync status --output json # machine-readable
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
|
|
165
|
+
## `diff`
|
|
166
|
+
|
|
167
|
+
Show a field-level diff between local specs and Azure DevOps — richer than `status`.
|
|
168
|
+
|
|
169
|
+
```bash
|
|
170
|
+
ado-sync diff
|
|
171
|
+
ado-sync diff --tags "@smoke"
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
Output example:
|
|
175
|
+
```
|
|
176
|
+
~ specs/login.feature:12 · Login with valid credentials [#1234]
|
|
177
|
+
changed fields: title, steps
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
---
|
|
181
|
+
|
|
182
|
+
## `generate`
|
|
183
|
+
|
|
184
|
+
Generate local spec files (`.feature` or `.md`) from Azure DevOps User Stories, pulling title, description, and acceptance criteria to scaffold each file.
|
|
185
|
+
|
|
186
|
+
```bash
|
|
187
|
+
# By explicit work item IDs
|
|
188
|
+
ado-sync generate --story-ids 1234,5678
|
|
189
|
+
|
|
190
|
+
# By area path (fetches all User Stories under it)
|
|
191
|
+
ado-sync generate --area-path "MyProject\\\\Teams\\\\QA"
|
|
192
|
+
|
|
193
|
+
# By WIQL query
|
|
194
|
+
ado-sync generate --query "SELECT [System.Id] FROM WorkItems WHERE [System.WorkItemType]='User Story' AND [System.State]='Active'"
|
|
195
|
+
|
|
196
|
+
# Options
|
|
197
|
+
ado-sync generate --story-ids 1234 --format gherkin # output .feature files
|
|
198
|
+
ado-sync generate --story-ids 1234 --format markdown # output .md files (default)
|
|
199
|
+
ado-sync generate --story-ids 1234 --output-folder specs/generated
|
|
200
|
+
ado-sync generate --story-ids 1234 --force # overwrite existing files
|
|
201
|
+
ado-sync generate --story-ids 1234 --dry-run # preview without writing
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
---
|
|
205
|
+
|
|
206
|
+
## `publish-test-results`
|
|
207
|
+
|
|
208
|
+
Publish test results from TRX, JUnit XML, NUnit XML, Cucumber JSON, or Playwright JSON files to Azure DevOps Test Runs.
|
|
209
|
+
|
|
210
|
+
```bash
|
|
211
|
+
ado-sync publish-test-results --testResult results/test.trx
|
|
212
|
+
ado-sync publish-test-results --testResult results/test.xml --testResultFormat junit --dry-run
|
|
213
|
+
ado-sync publish-test-results --testResult results/playwright.json --attachmentsFolder test-results/
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
See [publish-test-results.md](publish-test-results.md) for full reference.
|
|
217
|
+
|
|
218
|
+
---
|
|
219
|
+
|
|
220
|
+
## `--config-override`
|
|
221
|
+
|
|
222
|
+
All commands accept `--config-override path=value` (repeatable) to set config values at runtime:
|
|
223
|
+
|
|
224
|
+
```bash
|
|
225
|
+
ado-sync push --config-override testPlan.id=9999
|
|
226
|
+
ado-sync push --config-override sync.disableLocalChanges=true
|
|
227
|
+
ado-sync push --config-override sync.tagPrefix=test --config-override testPlan.id=42
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
---
|
|
231
|
+
|
|
232
|
+
## `--output json`
|
|
233
|
+
|
|
234
|
+
All sync commands support `--output json` for machine-readable output:
|
|
235
|
+
|
|
236
|
+
```bash
|
|
237
|
+
ado-sync status --output json | jq '.[] | select(.action=="updated")'
|
|
238
|
+
ado-sync push --output json > results.json
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
Each result object: `{ action, filePath, title, azureId?, detail?, changedFields? }`
|
|
@@ -0,0 +1,275 @@
|
|
|
1
|
+
# MCP Server
|
|
2
|
+
|
|
3
|
+
`ado-sync` ships a built-in MCP (Model Context Protocol) server that exposes its sync operations as **structured tools** for AI agents — Claude Code, GitHub Copilot, Cursor, and any other MCP-compatible host.
|
|
4
|
+
|
|
5
|
+
Using the MCP server instead of spawning the CLI gives agents:
|
|
6
|
+
- Typed JSON responses (no output parsing)
|
|
7
|
+
- Structured error objects with status codes
|
|
8
|
+
- Fine-grained tool calls (get one test case, push a subset, etc.)
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## Installation
|
|
13
|
+
|
|
14
|
+
No separate install needed. `ado-sync-mcp` ships inside the `ado-sync` npm package.
|
|
15
|
+
|
|
16
|
+
**Option A — Global install (recommended, fastest startup)**
|
|
17
|
+
```bash
|
|
18
|
+
npm install -g ado-sync
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
**Option B — npx (no install, always latest)**
|
|
22
|
+
|
|
23
|
+
Use `npx --yes --package=ado-sync ado-sync-mcp` as the command in all configs below.
|
|
24
|
+
npx downloads and runs the package automatically on first use.
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## Claude Code — one-liner registration
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
# With global install
|
|
32
|
+
claude mcp add ado-sync \
|
|
33
|
+
--env AZURE_DEVOPS_TOKEN="$AZURE_DEVOPS_TOKEN" \
|
|
34
|
+
--env ADO_SYNC_CONFIG="$(pwd)/ado-sync.json" \
|
|
35
|
+
-- ado-sync-mcp
|
|
36
|
+
|
|
37
|
+
# With npx (no global install required)
|
|
38
|
+
claude mcp add ado-sync \
|
|
39
|
+
--env AZURE_DEVOPS_TOKEN="$AZURE_DEVOPS_TOKEN" \
|
|
40
|
+
--env ADO_SYNC_CONFIG="$(pwd)/ado-sync.json" \
|
|
41
|
+
-- npx --yes --package=ado-sync ado-sync-mcp
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Verify it registered: run `/mcp` in Claude Code — `ado-sync` should appear in the list.
|
|
45
|
+
|
|
46
|
+
**Manual config** — `~/.claude/claude_desktop_config.json`:
|
|
47
|
+
```json
|
|
48
|
+
{
|
|
49
|
+
"mcpServers": {
|
|
50
|
+
"ado-sync": {
|
|
51
|
+
"command": "npx",
|
|
52
|
+
"args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
|
|
53
|
+
"env": {
|
|
54
|
+
"AZURE_DEVOPS_TOKEN": "<your-pat>",
|
|
55
|
+
"ADO_SYNC_CONFIG": "/absolute/path/to/ado-sync.json"
|
|
56
|
+
}
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## Claude Desktop
|
|
65
|
+
|
|
66
|
+
Config file location:
|
|
67
|
+
- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
|
|
68
|
+
- Windows: `%APPDATA%\Claude\claude_desktop_config.json`
|
|
69
|
+
|
|
70
|
+
```json
|
|
71
|
+
{
|
|
72
|
+
"mcpServers": {
|
|
73
|
+
"ado-sync": {
|
|
74
|
+
"command": "npx",
|
|
75
|
+
"args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
|
|
76
|
+
"env": {
|
|
77
|
+
"AZURE_DEVOPS_TOKEN": "<your-pat>",
|
|
78
|
+
"ADO_SYNC_CONFIG": "/absolute/path/to/ado-sync.json"
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
}
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Restart Claude Desktop after saving.
|
|
86
|
+
|
|
87
|
+
---
|
|
88
|
+
|
|
89
|
+
## VS Code (GitHub Copilot agent mode)
|
|
90
|
+
|
|
91
|
+
Create `.vscode/mcp.json` in your workspace root:
|
|
92
|
+
|
|
93
|
+
```json
|
|
94
|
+
{
|
|
95
|
+
"servers": {
|
|
96
|
+
"ado-sync": {
|
|
97
|
+
"type": "stdio",
|
|
98
|
+
"command": "npx",
|
|
99
|
+
"args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
|
|
100
|
+
"env": {
|
|
101
|
+
"AZURE_DEVOPS_TOKEN": "${env:AZURE_DEVOPS_TOKEN}",
|
|
102
|
+
"ADO_SYNC_CONFIG": "${workspaceFolder}/ado-sync.json"
|
|
103
|
+
}
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
`${env:AZURE_DEVOPS_TOKEN}` reads the variable from your shell environment — no hardcoded secrets.
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## Cursor
|
|
114
|
+
|
|
115
|
+
`~/.cursor/mcp.json`:
|
|
116
|
+
|
|
117
|
+
```json
|
|
118
|
+
{
|
|
119
|
+
"mcpServers": {
|
|
120
|
+
"ado-sync": {
|
|
121
|
+
"command": "npx",
|
|
122
|
+
"args": ["--yes", "--package=ado-sync", "ado-sync-mcp"],
|
|
123
|
+
"env": {
|
|
124
|
+
"AZURE_DEVOPS_TOKEN": "<your-pat>",
|
|
125
|
+
"ADO_SYNC_CONFIG": "/absolute/path/to/ado-sync.json"
|
|
126
|
+
}
|
|
127
|
+
}
|
|
128
|
+
}
|
|
129
|
+
}
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
## Verify it works
|
|
135
|
+
|
|
136
|
+
After registration, ask your AI assistant:
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
"Call ado-sync validate_config and tell me what it returns"
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
Expected response includes config validity, Azure connection status, and test plan confirmation.
|
|
143
|
+
|
|
144
|
+
---
|
|
145
|
+
|
|
146
|
+
## Available tools
|
|
147
|
+
|
|
148
|
+
| Tool | Description |
|
|
149
|
+
|------|-------------|
|
|
150
|
+
| `validate_config` | Check config validity and Azure DevOps connectivity |
|
|
151
|
+
| `get_test_cases` | List all Test Cases in a suite |
|
|
152
|
+
| `get_test_case` | Get a single Test Case by ID |
|
|
153
|
+
| `push_specs` | Push local spec files to Azure DevOps |
|
|
154
|
+
| `pull_specs` | Pull Azure DevOps changes into local files |
|
|
155
|
+
| `status` | Show diff between local and Azure without making changes |
|
|
156
|
+
| `diff` | Show field-level diff (richer than status) |
|
|
157
|
+
| `generate_specs` | Generate local spec files from ADO User Stories |
|
|
158
|
+
| `get_work_items` | Fetch ADO User Stories / work items |
|
|
159
|
+
| `publish_test_results` | Publish TRX / JUnit / Playwright JSON / CTRF results; optionally file issues for failures |
|
|
160
|
+
| `create_issue` | File a single GitHub Issue or ADO Bug for a test failure (for healer agents) |
|
|
161
|
+
| `get_story_context` | Planner-agent feed: AC items, suggested tags, actors, linked TC IDs |
|
|
162
|
+
| `generate_manifest` | Write `.ai-workflow-manifest-{id}.json` for the full Planner→CI cycle |
|
|
163
|
+
|
|
164
|
+
---
|
|
165
|
+
|
|
166
|
+
## Tool parameters
|
|
167
|
+
|
|
168
|
+
All tools accept these optional base parameters:
|
|
169
|
+
|
|
170
|
+
| Parameter | Type | Description |
|
|
171
|
+
|-----------|------|-------------|
|
|
172
|
+
| `configPath` | string | Absolute path to `ado-sync.json` (overrides `ADO_SYNC_CONFIG`) |
|
|
173
|
+
| `configOverrides` | string[] | Runtime overrides in `key=value` format (same as `--config-override`) |
|
|
174
|
+
|
|
175
|
+
### `push_specs`
|
|
176
|
+
|
|
177
|
+
```json
|
|
178
|
+
{
|
|
179
|
+
"dryRun": false,
|
|
180
|
+
"tags": "@smoke and not @wip",
|
|
181
|
+
"aiProvider": "heuristic"
|
|
182
|
+
}
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
### `generate_specs`
|
|
186
|
+
|
|
187
|
+
```json
|
|
188
|
+
{
|
|
189
|
+
"storyIds": [1234, 5678],
|
|
190
|
+
"format": "gherkin",
|
|
191
|
+
"outputFolder": "specs/generated",
|
|
192
|
+
"dryRun": false,
|
|
193
|
+
"force": false
|
|
194
|
+
}
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
### `publish_test_results`
|
|
198
|
+
|
|
199
|
+
```json
|
|
200
|
+
{
|
|
201
|
+
"testResult": "results/playwright.json",
|
|
202
|
+
"attachmentsFolder": "test-results/",
|
|
203
|
+
"createIssuesOnFailure": true,
|
|
204
|
+
"issueProvider": "github",
|
|
205
|
+
"githubRepo": "myorg/myrepo",
|
|
206
|
+
"githubToken": "$GITHUB_TOKEN",
|
|
207
|
+
"bugThreshold": 20,
|
|
208
|
+
"maxIssues": 50
|
|
209
|
+
}
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
### `create_issue`
|
|
213
|
+
|
|
214
|
+
```json
|
|
215
|
+
{
|
|
216
|
+
"title": "[FAILED] Login with valid credentials",
|
|
217
|
+
"body": "Error: Expected 200 but got 401\n\nStack: ...",
|
|
218
|
+
"provider": "github",
|
|
219
|
+
"githubRepo": "myorg/myrepo",
|
|
220
|
+
"githubToken": "$GITHUB_TOKEN",
|
|
221
|
+
"testCaseId": 1234
|
|
222
|
+
}
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
### `get_story_context`
|
|
226
|
+
|
|
227
|
+
```json
|
|
228
|
+
{ "storyId": 1234 }
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
Returns: AC items as a bullet list, inferred tags (`@smoke`, `@auth`, …), extracted actors, and IDs of any Test Cases already linked via TestedBy relation.
|
|
232
|
+
|
|
233
|
+
### `generate_manifest`
|
|
234
|
+
|
|
235
|
+
```json
|
|
236
|
+
{
|
|
237
|
+
"storyIds": [1234, 5678],
|
|
238
|
+
"outputFolder": "e2e/bdd",
|
|
239
|
+
"format": "gherkin",
|
|
240
|
+
"dryRun": false
|
|
241
|
+
}
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Writes `.ai-workflow-manifest-1234.json` in `outputFolder`. The manifest contains the ordered 8-step workflow, AC items, required documents checklist, and validation steps.
|
|
245
|
+
|
|
246
|
+
---
|
|
247
|
+
|
|
248
|
+
## Example agent workflow
|
|
249
|
+
|
|
250
|
+
Once registered, an AI agent can orchestrate the full test lifecycle without any human CLI invocation:
|
|
251
|
+
|
|
252
|
+
```
|
|
253
|
+
Agent: "Set up ado-sync for this repo"
|
|
254
|
+
→ calls validate_config to check connectivity
|
|
255
|
+
→ calls push_specs({ dryRun: true }) to preview
|
|
256
|
+
→ calls push_specs({}) to create Test Cases
|
|
257
|
+
|
|
258
|
+
Agent: "Generate spec files for User Story 1234"
|
|
259
|
+
→ calls generate_specs({ storyIds: [1234], format: "gherkin" })
|
|
260
|
+
|
|
261
|
+
Agent: "Publish the latest test results and file issues for failures"
|
|
262
|
+
→ calls publish_test_results({ testResult: "results/playwright.json", createIssuesOnFailure: true, githubRepo: "myorg/myrepo", githubToken: "$GITHUB_TOKEN" })
|
|
263
|
+
→ returns issue URLs for any failures → healer agent opens fix PRs
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
---
|
|
267
|
+
|
|
268
|
+
## Environment variables
|
|
269
|
+
|
|
270
|
+
| Variable | Description |
|
|
271
|
+
|----------|-------------|
|
|
272
|
+
| `ADO_SYNC_CONFIG` | Absolute path to config file (default: `ado-sync.json` in cwd) |
|
|
273
|
+
| `AZURE_DEVOPS_TOKEN` | PAT or access token (referenced by `auth.token` in config) |
|
|
274
|
+
|
|
275
|
+
> **Important:** `ADO_SYNC_CONFIG` must be an **absolute path**. The MCP server process does not inherit the shell's working directory the same way as a CLI call.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# publish-test-results
|
|
2
2
|
|
|
3
|
-
Parses test result files (TRX, NUnit XML, JUnit, Cucumber JSON, Playwright JSON) and publishes them to an Azure DevOps Test Run, linking results back to Test Cases either directly by TC ID (when available in the result file) or by `AutomatedTestName` matching.
|
|
3
|
+
Parses test result files (TRX, NUnit XML, JUnit, Cucumber JSON, Playwright JSON, CTRF JSON) and publishes them to an Azure DevOps Test Run, linking results back to Test Cases either directly by TC ID (when available in the result file) or by `AutomatedTestName` matching.
|
|
4
4
|
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -31,11 +31,17 @@ ado-sync publish-test-results \
|
|
|
31
31
|
| Option | Description |
|
|
32
32
|
|--------|-------------|
|
|
33
33
|
| `--testResult <path>` | Path to a result file. Repeatable. |
|
|
34
|
-
| `--testResultFormat <format>` | `trx` · `nunitXml` · `junit` · `cucumberJson` · `playwrightJson`. Auto-detected when omitted. |
|
|
34
|
+
| `--testResultFormat <format>` | `trx` · `nunitXml` · `junit` · `cucumberJson` · `playwrightJson` · `ctrfJson`. Auto-detected when omitted. |
|
|
35
35
|
| `--attachmentsFolder <path>` | Folder to scan for screenshots/videos/logs to attach to test results. |
|
|
36
36
|
| `--runName <name>` | Name for the Test Run in Azure DevOps. Defaults to `ado-sync <ISO timestamp>`. |
|
|
37
37
|
| `--buildId <id>` | Build ID to associate with the Test Run. |
|
|
38
38
|
| `--dry-run` | Parse results and print summary without creating a run in Azure. |
|
|
39
|
+
| `--create-issues-on-failure` | File GitHub Issues or ADO Bugs for each failed test after publishing. |
|
|
40
|
+
| `--issue-provider <github\|ado>` | Issue provider. Default: `github`. |
|
|
41
|
+
| `--github-repo <owner/repo>` | GitHub repository to file issues in. |
|
|
42
|
+
| `--github-token <token>` | GitHub token. Supports `$ENV_VAR` references. |
|
|
43
|
+
| `--bug-threshold <percent>` | If more than this % of tests fail, one environment-failure issue is filed instead of per-test issues. Default: `20`. |
|
|
44
|
+
| `--max-issues <n>` | Hard cap on issues filed per run. Default: `50`. |
|
|
39
45
|
| `--config-override` | Override config values (repeatable, same as other commands). |
|
|
40
46
|
|
|
41
47
|
---
|
|
@@ -50,6 +56,7 @@ ado-sync publish-test-results \
|
|
|
50
56
|
| Cucumber JSON | `.json` | Yes (JSON array, Cucumber format) | Yes — via `@tc:ID` tag on scenario | `step.embeddings[]` (base64 screenshots/video) |
|
|
51
57
|
| Playwright JSON | `.json` | Yes (JSON object with `suites` key) | Yes — via `test.annotations[{ type: 'tc', description: 'ID' }]` (preferred) or `@tc:ID` in test title | `test.results[].attachments[]` (screenshots, videos, traces) |
|
|
52
58
|
| Robot Framework XML | `output.xml` | Yes (`<robot>` root element) | Yes — via `<tags><tag>tc:ID</tag></tags>` | — |
|
|
59
|
+
| CTRF JSON | `.json` | Yes (`results.tests` array) | Yes — via `tags: ["@tc:ID"]` or `@tc:ID` in test name | `attachments[].path` files, `stdout`/`stderr` arrays |
|
|
53
60
|
|
|
54
61
|
> **NUnit via TRX**: when NUnit tests are run through the VSTest adapter (`--logger trx`), `[Property]` values are **not** included in the TRX output. Use `--logger "nunit3;LogFileName=results.xml"` to get the native NUnit XML format, which does include property values.
|
|
55
62
|
|
|
@@ -441,6 +448,38 @@ Recommended config:
|
|
|
441
448
|
|
|
442
449
|
---
|
|
443
450
|
|
|
451
|
+
### CTRF (Common Test Report Format)
|
|
452
|
+
|
|
453
|
+
[CTRF](https://ctrf.io) is a framework-agnostic JSON report format supported by reporters for Playwright, Cypress, Jest, k6, and many others. ado-sync auto-detects CTRF from the `results.tests` array structure.
|
|
454
|
+
|
|
455
|
+
```bash
|
|
456
|
+
# Example: Playwright with CTRF reporter
|
|
457
|
+
npm install --save-dev playwright-ctrf-json-reporter
|
|
458
|
+
|
|
459
|
+
# playwright.config.ts:
|
|
460
|
+
# reporter: [['playwright-ctrf-json-reporter', { outputFile: 'results/ctrf.json' }]]
|
|
461
|
+
|
|
462
|
+
npx playwright test
|
|
463
|
+
ado-sync publish-test-results --testResult results/ctrf.json
|
|
464
|
+
```
|
|
465
|
+
|
|
466
|
+
```bash
|
|
467
|
+
# Example: Jest with CTRF reporter
|
|
468
|
+
npm install --save-dev jest-ctrf-json-reporter
|
|
469
|
+
|
|
470
|
+
# jest.config.ts:
|
|
471
|
+
# reporters: [['jest-ctrf-json-reporter', { outputFile: 'results/ctrf.json' }]]
|
|
472
|
+
|
|
473
|
+
npx jest
|
|
474
|
+
ado-sync publish-test-results --testResult results/ctrf.json
|
|
475
|
+
```
|
|
476
|
+
|
|
477
|
+
TC IDs are extracted from the `tags` array (e.g. `["@tc:1234", "@smoke"]`) or, as a fallback, from `@tc:ID` in the test name. `stdout`/`stderr` arrays and `attachments[].path` files are uploaded automatically.
|
|
478
|
+
|
|
479
|
+
> **Status mapping**: CTRF `passed` → `Passed`, `failed` → `Failed`, `skipped`/`pending`/`other` → `NotExecuted`.
|
|
480
|
+
|
|
481
|
+
---
|
|
482
|
+
|
|
444
483
|
### Flutter
|
|
445
484
|
|
|
446
485
|
Flutter can produce JUnit XML via the `flutter_test_junit` package or by piping `--reporter junit`:
|
|
@@ -483,6 +522,7 @@ TC linking uses `AutomatedTestName` matching — set `sync.markAutomated: true`
|
|
|
483
522
|
| Espresso | JUnit XML | ❌ AutomatedTestName matching only | `<system-out>`, `<system-err>` | |
|
|
484
523
|
| Flutter | JUnit XML | ❌ AutomatedTestName matching only | `<system-out>`, `<system-err>` | |
|
|
485
524
|
| Robot Framework | Robot XML (`output.xml`) | ✅ `tc:N` in `<tags>` | — | |
|
|
525
|
+
| CTRF (any framework) | CTRF JSON | ✅ `tags: ["@tc:ID"]` or `@tc:ID` in name | `attachments[].path` files + `stdout`/`stderr` | |
|
|
486
526
|
|
|
487
527
|
---
|
|
488
528
|
|
|
@@ -499,6 +539,7 @@ ado-sync uploads screenshots, videos, and logs from test results to the correspo
|
|
|
499
539
|
| JUnit XML | `<system-out>` → log; `<system-err>` → log; `[[ATTACHMENT\|path]]` → Playwright files |
|
|
500
540
|
| Cucumber JSON | `step.embeddings[]` → base64-encoded screenshots/video |
|
|
501
541
|
| Playwright JSON | `results[].attachments[].path` → files on disk (screenshots, videos, traces) |
|
|
542
|
+
| CTRF JSON | `tests[].attachments[].path` → files on disk; `tests[].stdout[]` / `tests[].stderr[]` → console logs |
|
|
502
543
|
|
|
503
544
|
> **Note**: All file paths are resolved relative to the result file's directory, not the process working directory. This matches how test runners (Playwright, MSTest, NUnit) write relative paths in their output.
|
|
504
545
|
|
|
@@ -670,6 +711,99 @@ Results can also be configured in the config file under `publishTestResults`:
|
|
|
670
711
|
|
|
671
712
|
---
|
|
672
713
|
|
|
714
|
+
## Creating issues on failure
|
|
715
|
+
|
|
716
|
+
`--create-issues-on-failure` automatically files a GitHub Issue or ADO Bug for each failed test
|
|
717
|
+
after the run is published. Multiple guards prevent flooding your tracker when the environment is
|
|
718
|
+
the problem rather than individual tests.
|
|
719
|
+
|
|
720
|
+
### Guard logic (applied in order)
|
|
721
|
+
|
|
722
|
+
```
|
|
723
|
+
failures > threshold% of total?
|
|
724
|
+
└─ YES → 1 environment-failure issue, stop
|
|
725
|
+
└─ NO
|
|
726
|
+
└─ cluster by error signature
|
|
727
|
+
└─ cluster size > 1?
|
|
728
|
+
└─ YES → 1 issue per cluster (lists affected test names)
|
|
729
|
+
└─ NO → 1 issue per TC (up to maxIssues cap)
|
|
730
|
+
└─ cap hit? → 1 overflow summary issue
|
|
731
|
+
```
|
|
732
|
+
|
|
733
|
+
| Guard | Default | Description |
|
|
734
|
+
|---|---|---|
|
|
735
|
+
| Failure-rate threshold | 20% | Above this, one env-failure issue is filed instead of per-test |
|
|
736
|
+
| Error clustering | enabled | Tests with the same error message are grouped into one issue |
|
|
737
|
+
| Hard cap | 50 | No more than this many issues per run; one overflow summary when exceeded |
|
|
738
|
+
| Dedup | enabled | Skip if an open issue already exists for the same TC (GitHub: by `tc:ID` label; ADO: by title) |
|
|
739
|
+
|
|
740
|
+
### GitHub Issues (recommended)
|
|
741
|
+
|
|
742
|
+
```bash
|
|
743
|
+
ado-sync publish-test-results \
|
|
744
|
+
--testResult results/ctrf.json \
|
|
745
|
+
--create-issues-on-failure \
|
|
746
|
+
--github-repo myorg/myrepo \
|
|
747
|
+
--github-token $GITHUB_TOKEN
|
|
748
|
+
```
|
|
749
|
+
|
|
750
|
+
Each issue is labelled `test-failure` and `tc:{ID}` (when a TC ID is available). The issue body
|
|
751
|
+
contains the error message, stack trace, ADO TC link, and run URL — everything a healer agent
|
|
752
|
+
needs to propose a fix PR.
|
|
753
|
+
|
|
754
|
+
### ADO Bugs
|
|
755
|
+
|
|
756
|
+
```bash
|
|
757
|
+
ado-sync publish-test-results \
|
|
758
|
+
--testResult results/junit.xml \
|
|
759
|
+
--create-issues-on-failure \
|
|
760
|
+
--issue-provider ado
|
|
761
|
+
```
|
|
762
|
+
|
|
763
|
+
ADO Bugs are created as Bug work items in the same project. The `Repro Steps` field is populated
|
|
764
|
+
with the error details. When a TC ID is known, a `TestedBy` relation is added linking the Bug to
|
|
765
|
+
the Test Case.
|
|
766
|
+
|
|
767
|
+
### Config-based setup
|
|
768
|
+
|
|
769
|
+
```json
|
|
770
|
+
{
|
|
771
|
+
"publishTestResults": {
|
|
772
|
+
"createIssuesOnFailure": {
|
|
773
|
+
"provider": "github",
|
|
774
|
+
"repo": "myorg/myrepo",
|
|
775
|
+
"token": "$GITHUB_TOKEN",
|
|
776
|
+
"labels": ["test-failure", "automated"],
|
|
777
|
+
"threshold": 20,
|
|
778
|
+
"maxIssues": 50,
|
|
779
|
+
"clusterByError": true,
|
|
780
|
+
"dedupByTestCase": true
|
|
781
|
+
}
|
|
782
|
+
}
|
|
783
|
+
}
|
|
784
|
+
```
|
|
785
|
+
|
|
786
|
+
CLI flags override the config values when both are present.
|
|
787
|
+
|
|
788
|
+
### MCP tool: `create_issue`
|
|
789
|
+
|
|
790
|
+
The `create_issue` MCP tool lets healer agents file a single issue directly:
|
|
791
|
+
|
|
792
|
+
```
|
|
793
|
+
create_issue({
|
|
794
|
+
title: "[FAILED] Login with valid credentials",
|
|
795
|
+
body: "Error: Expected 200 but got 401\n\nStack: ...",
|
|
796
|
+
provider: "github",
|
|
797
|
+
githubRepo: "myorg/myrepo",
|
|
798
|
+
githubToken: "$GITHUB_TOKEN",
|
|
799
|
+
testCaseId: 1234
|
|
800
|
+
})
|
|
801
|
+
```
|
|
802
|
+
|
|
803
|
+
Returns the issue URL immediately, which the agent can embed in its fix PR.
|
|
804
|
+
|
|
805
|
+
---
|
|
806
|
+
|
|
673
807
|
## Output
|
|
674
808
|
|
|
675
809
|
```
|