ado-sync 0.1.46 → 0.1.47
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +22 -945
- package/dist/azure/test-cases.js +39 -6
- package/dist/azure/test-cases.js.map +1 -1
- package/dist/azure/work-items.d.ts +32 -0
- package/dist/azure/work-items.js +98 -0
- package/dist/azure/work-items.js.map +1 -0
- package/dist/cli.js +277 -26
- package/dist/cli.js.map +1 -1
- package/dist/config.js +50 -0
- package/dist/config.js.map +1 -1
- package/dist/mcp-server.d.ts +32 -0
- package/dist/mcp-server.js +372 -0
- package/dist/mcp-server.js.map +1 -0
- package/dist/sync/cache.d.ts +7 -1
- package/dist/sync/cache.js.map +1 -1
- package/dist/sync/engine.js +105 -51
- package/dist/sync/engine.js.map +1 -1
- package/dist/sync/generate.d.ts +27 -0
- package/dist/sync/generate.js +184 -0
- package/dist/sync/generate.js.map +1 -0
- package/dist/types.d.ts +9 -0
- package/docs/agent-setup.md +204 -0
- package/docs/cli.md +241 -0
- package/docs/mcp-server.md +173 -0
- package/docs/troubleshooting.md +101 -0
- package/docs/work-item-links.md +115 -0
- package/docs/workflows.md +381 -0
- package/llms.txt +163 -0
- package/package.json +4 -2
package/README.md
CHANGED
|
@@ -41,13 +41,9 @@ Local files ado-sync Azure DevOps
|
|
|
41
41
|
────────────── ───────────────── ────────────────
|
|
42
42
|
.feature files ── push ──► create / update ──► Test Cases
|
|
43
43
|
.md spec files ◄── pull ── apply changes ◄── (Work Items)
|
|
44
|
-
.cs
|
|
45
|
-
.
|
|
46
|
-
.
|
|
47
|
-
.js / .ts files ── push ──► (push-only) ──► + Associated Automation
|
|
48
|
-
.robot files ── push ──► (push-only) ──► + Associated Automation
|
|
49
|
-
.csv files ── push ──► (push-only)
|
|
50
|
-
.xlsx files ── push ──► (push-only)
|
|
44
|
+
.cs / .java / ── push ──► (push-only) ──► + Associated Automation
|
|
45
|
+
.py / .js / .ts
|
|
46
|
+
.robot / .csv ── push ──► (push-only)
|
|
51
47
|
write ID back
|
|
52
48
|
@tc:12345 / [TestProperty] / @Tag / @pytest.mark / // @tc:
|
|
53
49
|
TRX / JUnit / ── publish-test-results ──► Test Run results (linked to TCs)
|
|
@@ -56,34 +52,11 @@ Cucumber JSON
|
|
|
56
52
|
|
|
57
53
|
On the **first push**, a new Test Case is created in Azure DevOps and its ID is written back into the local file. Every subsequent push uses that ID to update the existing Test Case.
|
|
58
54
|
|
|
59
|
-
**ID writeback format per framework:**
|
|
60
|
-
|
|
61
|
-
| Framework | ID written as |
|
|
62
|
-
|---|---|
|
|
63
|
-
| Gherkin / Markdown | `@tc:12345` tag / comment |
|
|
64
|
-
| C# MSTest | `[TestProperty("tc", "12345")]` |
|
|
65
|
-
| C# NUnit | `[Property("tc", "12345")]` |
|
|
66
|
-
| Java JUnit 4 / TestNG | `// @tc:12345` comment above `@Test` |
|
|
67
|
-
| Java JUnit 5 | `@Tag("tc:12345")` above `@Test` |
|
|
68
|
-
| Python pytest | `@pytest.mark.tc(12345)` above `def test_*` |
|
|
69
|
-
| JavaScript/TS (Jest/Jasmine/WebdriverIO) | `// @tc:12345` comment above `it()`/`test()` |
|
|
70
|
-
| Playwright | `annotation: { type: 'tc', description: '12345' }` in test options *(preferred)*; or `// @tc:12345` comment fallback |
|
|
71
|
-
| Puppeteer | `// @tc:12345` comment above `it()`/`test()` |
|
|
72
|
-
| Cypress | `// @tc:12345` comment above `it()`/`specify()` |
|
|
73
|
-
| TestCafe | `test.meta('tc', '12345')('title', fn)` *(preferred)*; or `// @tc:12345` comment fallback |
|
|
74
|
-
| Detox | `// @tc:12345` comment above `it()`/`test()` |
|
|
75
|
-
| Espresso (Java/Kotlin) | `// @tc:12345` comment above `@Test` |
|
|
76
|
-
| XCUITest (Swift) | `// @tc:12345` comment above `func test*()` |
|
|
77
|
-
| Flutter (Dart) | `// @tc:12345` comment above `testWidgets()`/`test()` |
|
|
78
|
-
| Robot Framework | `tc:12345` value in `[Tags]` row of the test case body |
|
|
79
|
-
| CSV | Numeric ID in column A |
|
|
80
|
-
| Excel | Numeric ID in cell A |
|
|
81
|
-
|
|
82
55
|
---
|
|
83
56
|
|
|
84
57
|
## Quick start
|
|
85
58
|
|
|
86
|
-
**
|
|
59
|
+
**1 — Install**
|
|
87
60
|
|
|
88
61
|
```bash
|
|
89
62
|
npm install -g ado-sync
|
|
@@ -91,16 +64,14 @@ npm install -g ado-sync
|
|
|
91
64
|
npx ado-sync --help
|
|
92
65
|
```
|
|
93
66
|
|
|
94
|
-
**
|
|
67
|
+
**2 — Generate a config file**
|
|
95
68
|
|
|
96
69
|
```bash
|
|
97
|
-
ado-sync init # creates ado-sync.json
|
|
98
|
-
ado-sync init ado-sync.yml # YAML format
|
|
70
|
+
ado-sync init # interactive wizard → creates ado-sync.json
|
|
71
|
+
ado-sync init ado-sync.yml # YAML format
|
|
99
72
|
```
|
|
100
73
|
|
|
101
|
-
**
|
|
102
|
-
|
|
103
|
-
Open `ado-sync.json` and replace the placeholders:
|
|
74
|
+
**3 — Fill in your details**
|
|
104
75
|
|
|
105
76
|
```json
|
|
106
77
|
{
|
|
@@ -119,306 +90,29 @@ Open `ado-sync.json` and replace the placeholders:
|
|
|
119
90
|
| `testPlan.id` | Test Plans → click your plan → the number in the URL |
|
|
120
91
|
| `AZURE_DEVOPS_TOKEN` | Azure DevOps → User Settings → Personal Access Tokens → New Token (scope: Test Management read/write) |
|
|
121
92
|
|
|
122
|
-
**
|
|
93
|
+
**4 — Set your token**
|
|
123
94
|
|
|
124
95
|
```bash
|
|
125
96
|
export AZURE_DEVOPS_TOKEN=your_personal_access_token
|
|
126
97
|
# or add it to a .env file in this directory
|
|
127
98
|
```
|
|
128
99
|
|
|
129
|
-
**
|
|
100
|
+
**5 — Preview then push**
|
|
130
101
|
|
|
131
102
|
```bash
|
|
132
103
|
ado-sync push --dry-run # preview — no changes made
|
|
133
104
|
ado-sync push # create / update Test Cases in Azure DevOps
|
|
134
105
|
```
|
|
135
106
|
|
|
136
|
-
On the first push, new Test Cases are created and their IDs are written back into your local files (`@tc:12345`). Every subsequent push updates them.
|
|
137
|
-
|
|
138
107
|
---
|
|
139
108
|
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
```
|
|
143
|
-
ado-sync [options] [command]
|
|
144
|
-
|
|
145
|
-
Options:
|
|
146
|
-
-c, --config <path> Path to config file (default: ado-sync.json)
|
|
147
|
-
-V, --version Print version
|
|
148
|
-
-h, --help Show help
|
|
149
|
-
|
|
150
|
-
Commands:
|
|
151
|
-
init [output] Generate a starter config file
|
|
152
|
-
push [options] Push local specs to Azure DevOps
|
|
153
|
-
pull [options] Pull updates from Azure DevOps into local files
|
|
154
|
-
status [options] Show diff without making changes
|
|
155
|
-
publish-test-results [opts] Publish TRX / JUnit / Cucumber JSON results to Azure DevOps
|
|
156
|
-
help [command] Help for a specific command
|
|
157
|
-
```
|
|
158
|
-
|
|
159
|
-
### `init`
|
|
109
|
+
**6 — Publish test results**
|
|
160
110
|
|
|
161
111
|
```bash
|
|
162
|
-
ado-sync
|
|
163
|
-
ado-sync init ado-sync.yml # YAML format
|
|
164
|
-
```
|
|
165
|
-
|
|
166
|
-
### `push`
|
|
167
|
-
|
|
168
|
-
```bash
|
|
169
|
-
ado-sync push
|
|
170
|
-
ado-sync push --dry-run
|
|
171
|
-
ado-sync push --tags "@smoke and not @wip"
|
|
172
|
-
ado-sync push --config-override testPlan.id=9999
|
|
173
|
-
|
|
174
|
-
# AI-generated test steps for code files (Java, C#, Python, JS/TS, Playwright)
|
|
175
|
-
ado-sync push --ai-provider heuristic # fast regex-based (no model needed)
|
|
176
|
-
ado-sync push --ai-provider local --ai-model ~/.cache/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
|
|
177
|
-
ado-sync push --ai-provider ollama --ai-model qwen2.5-coder:7b
|
|
178
|
-
ado-sync push --ai-provider openai --ai-key $OPENAI_API_KEY
|
|
179
|
-
ado-sync push --ai-provider none # disable AI summary entirely
|
|
180
|
-
ado-sync push --ai-context ./docs/ai-context.md # inject domain context into AI prompt
|
|
181
|
-
```
|
|
182
|
-
|
|
183
|
-
| Scenario state | Action |
|
|
184
|
-
|----------------|--------|
|
|
185
|
-
| No ID tag | Creates a new Test Case, writes ID back |
|
|
186
|
-
| ID tag, no changes | Skipped |
|
|
187
|
-
| ID tag, content changed | Updates the existing Test Case |
|
|
188
|
-
| Deleted locally, still in Azure suite | Tagged `ado-sync:removed` in Azure |
|
|
189
|
-
|
|
190
|
-
### `pull`
|
|
191
|
-
|
|
192
|
-
```bash
|
|
193
|
-
ado-sync pull
|
|
194
|
-
ado-sync pull --dry-run
|
|
195
|
-
ado-sync pull --tags "@smoke"
|
|
196
|
-
```
|
|
197
|
-
|
|
198
|
-
### `status`
|
|
199
|
-
|
|
200
|
-
```bash
|
|
201
|
-
ado-sync status
|
|
202
|
-
ado-sync status --tags "@smoke"
|
|
203
|
-
ado-sync status --ai-provider heuristic
|
|
204
|
-
```
|
|
205
|
-
|
|
206
|
-
Compares local specs against Azure DevOps and prints a diff — no changes made.
|
|
207
|
-
|
|
208
|
-
### AI auto-summary
|
|
209
|
-
|
|
210
|
-
For code-based test types (`java`, `csharp`, `python`, `javascript`, `playwright`, `cypress`, `testcafe`, `detox`, `espresso`, `xcuitest`, `flutter`), ado-sync reads your test function bodies and automatically generates a TC **title**, **description**, and **steps** — so you don't need doc comments on every test.
|
|
211
|
-
|
|
212
|
-
> **No setup required to try it.** `ado-sync push` always works, even without a model — it falls back to fast regex-based analysis automatically.
|
|
213
|
-
|
|
214
|
-
#### Choose a provider
|
|
215
|
-
|
|
216
|
-
| Provider | Quality | Setup |
|
|
217
|
-
|---|---|---|
|
|
218
|
-
| `local` *(default)* | Good–Excellent | Download a GGUF model file (see below) |
|
|
219
|
-
| `heuristic` | Basic | None — works offline, zero dependencies |
|
|
220
|
-
| `ollama` | Good–Excellent | Install [Ollama](https://ollama.com) + `ollama pull qwen2.5-coder:7b` |
|
|
221
|
-
| `openai` | Excellent | `--ai-key $OPENAI_API_KEY` |
|
|
222
|
-
| `anthropic` | Excellent | `--ai-key $ANTHROPIC_API_KEY` — also works with **Claude Code** credentials |
|
|
223
|
-
| `openai` + `--ai-url` | Excellent | Any OpenAI-compatible proxy: LiteLLM, Azure OpenAI, vLLM, LM Studio — also works for **GitHub Copilot Enterprise** (Azure OpenAI) |
|
|
224
|
-
| `openai` + `--ai-url` (HF) | Good–Excellent | [Hugging Face Inference API](https://huggingface.co/settings/tokens) — free, open-source models |
|
|
225
|
-
|
|
226
|
-
> **GitHub Copilot or Claude Code user?** See [docs/advanced.md — Using GitHub Copilot or Claude Code](docs/advanced.md#using-github-copilot-or-claude-code) for a step-by-step guide on which provider to use and how to invoke ado-sync from within your IDE assistant.
|
|
227
|
-
|
|
228
|
-
#### Option A — No setup (heuristic, instant)
|
|
229
|
-
|
|
230
|
-
```bash
|
|
231
|
-
ado-sync push --ai-provider heuristic
|
|
232
|
-
```
|
|
233
|
-
|
|
234
|
-
Uses regex pattern matching. No model download, no internet required. Good for CI pipelines.
|
|
235
|
-
|
|
236
|
-
#### Option B — Local LLM (best privacy, no API cost)
|
|
237
|
-
|
|
238
|
-
`node-llama-cpp` is bundled — **no extra install needed**. You only need to download a model file once.
|
|
239
|
-
|
|
240
|
-
**1. Pick a model size**
|
|
241
|
-
|
|
242
|
-
| Model | RAM needed | Quality |
|
|
243
|
-
|-------|-----------|---------|
|
|
244
|
-
| 1.5B Q4_K_M *(start here)* | ~1.1 GB | Good |
|
|
245
|
-
| 7B Q4_K_M | ~4.5 GB | Better |
|
|
246
|
-
| 14B Q4_K_M | ~8.5 GB | Excellent |
|
|
247
|
-
|
|
248
|
-
**2. Download the model**
|
|
249
|
-
|
|
250
|
-
macOS / Linux:
|
|
251
|
-
```bash
|
|
252
|
-
mkdir -p ~/.cache/ado-sync/models
|
|
253
|
-
curl -L -o ~/.cache/ado-sync/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf \
|
|
254
|
-
"https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF/resolve/main/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf"
|
|
255
|
-
```
|
|
256
|
-
|
|
257
|
-
Windows (PowerShell):
|
|
258
|
-
```powershell
|
|
259
|
-
New-Item -ItemType Directory -Force "$env:LOCALAPPDATA\ado-sync\models"
|
|
260
|
-
Invoke-WebRequest `
|
|
261
|
-
-Uri "https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF/resolve/main/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf" `
|
|
262
|
-
-OutFile "$env:LOCALAPPDATA\ado-sync\models\qwen2.5-coder-1.5b-instruct-q4_k_m.gguf"
|
|
263
|
-
```
|
|
264
|
-
|
|
265
|
-
**3. Run push with the model**
|
|
266
|
-
|
|
267
|
-
```bash
|
|
268
|
-
# macOS / Linux
|
|
269
|
-
ado-sync push --ai-model ~/.cache/ado-sync/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
|
|
270
|
-
|
|
271
|
-
# Windows
|
|
272
|
-
ado-sync push --ai-model "$env:LOCALAPPDATA\ado-sync\models\qwen2.5-coder-1.5b-instruct-q4_k_m.gguf"
|
|
273
|
-
```
|
|
274
|
-
|
|
275
|
-
#### Option C — Ollama (model management UI, easy upgrades)
|
|
276
|
-
|
|
277
|
-
```bash
|
|
278
|
-
# 1. Install Ollama from https://ollama.com, then:
|
|
279
|
-
ollama pull qwen2.5-coder:7b
|
|
280
|
-
|
|
281
|
-
# 2. Push using Ollama
|
|
282
|
-
ado-sync push --ai-provider ollama --ai-model qwen2.5-coder:7b
|
|
283
|
-
```
|
|
284
|
-
|
|
285
|
-
#### Option D — Cloud AI (OpenAI / Anthropic)
|
|
286
|
-
|
|
287
|
-
```bash
|
|
288
|
-
ado-sync push --ai-provider openai --ai-key $OPENAI_API_KEY
|
|
289
|
-
ado-sync push --ai-provider anthropic --ai-key $ANTHROPIC_API_KEY
|
|
290
|
-
```
|
|
291
|
-
|
|
292
|
-
#### Option E — LiteLLM or any OpenAI-compatible proxy
|
|
293
|
-
|
|
294
|
-
[LiteLLM](https://github.com/BerriAI/litellm) is a proxy that routes to 100+ providers (Azure OpenAI, Bedrock, Gemini, Mistral, vLLM, and more) via a single OpenAI-compatible API. Point the `openai` provider at it with `--ai-url`:
|
|
295
|
-
|
|
296
|
-
```bash
|
|
297
|
-
ado-sync push \
|
|
298
|
-
--ai-provider openai \
|
|
299
|
-
--ai-url http://localhost:4000 \
|
|
300
|
-
--ai-key $LITELLM_API_KEY \
|
|
301
|
-
--ai-model gpt-4o-mini
|
|
302
|
-
```
|
|
303
|
-
|
|
304
|
-
> **Hosted LiteLLM + Anthropic models:** Use `--ai-url https://<your-host>/v1` and prefix model names with `anthropic/` (e.g. `anthropic/claude-opus-4-6`). Check your instance's `/v1/models` for registered names.
|
|
305
|
-
|
|
306
|
-
The same pattern works for Azure OpenAI, vLLM, LM Studio, and LocalAI — just change `--ai-url` to the endpoint's base URL. See [docs/advanced.md](docs/advanced.md) for a full compatibility table.
|
|
307
|
-
|
|
308
|
-
#### Option F — Hugging Face Inference API (free, open-source models)
|
|
309
|
-
|
|
310
|
-
[Hugging Face](https://huggingface.co) provides free serverless inference for open-source models via an OpenAI-compatible API:
|
|
311
|
-
|
|
312
|
-
```bash
|
|
313
|
-
ado-sync push \
|
|
314
|
-
--ai-provider openai \
|
|
315
|
-
--ai-url https://router.huggingface.co/v1 \
|
|
316
|
-
--ai-key $HF_TOKEN \
|
|
317
|
-
--ai-model Qwen/Qwen2.5-Coder-7B-Instruct
|
|
318
|
-
```
|
|
319
|
-
|
|
320
|
-
Get a token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) (requires **Inference** permission). Recommended models: `Qwen/Qwen2.5-Coder-7B-Instruct`, `meta-llama/Llama-3.1-8B-Instruct`, `mistralai/Mistral-7B-Instruct-v0.3`.
|
|
321
|
-
|
|
322
|
-
#### Option G — Inject domain context into the AI prompt
|
|
323
|
-
|
|
324
|
-
Provide a markdown file with domain-specific instructions (glossary, naming conventions, step style guidelines). Its content is injected into every AI prompt before the test code:
|
|
325
|
-
|
|
326
|
-
```bash
|
|
327
|
-
ado-sync push --ai-provider anthropic --ai-key $ANTHROPIC_API_KEY \
|
|
328
|
-
--ai-context ./docs/ai-context.md
|
|
329
|
-
```
|
|
330
|
-
|
|
331
|
-
Or set it once in config (relative path resolved from the config file directory):
|
|
332
|
-
|
|
333
|
-
```json
|
|
334
|
-
{
|
|
335
|
-
"sync": {
|
|
336
|
-
"ai": {
|
|
337
|
-
"provider": "anthropic",
|
|
338
|
-
"model": "claude-sonnet-4-6",
|
|
339
|
-
"apiKey": "$ANTHROPIC_API_KEY",
|
|
340
|
-
"contextFile": "./docs/ai-context.md"
|
|
341
|
-
}
|
|
342
|
-
}
|
|
343
|
-
}
|
|
344
|
-
```
|
|
345
|
-
|
|
346
|
-
Example `ai-context.md`:
|
|
347
|
-
|
|
348
|
-
```markdown
|
|
349
|
-
## Glossary
|
|
350
|
-
- "Checkout" means the 3-step payment flow (cart → shipping → payment)
|
|
351
|
-
- "PDP" means Product Detail Page
|
|
352
|
-
|
|
353
|
-
## Step style
|
|
354
|
-
- Start every action step with a verb: Click, Enter, Select, Navigate
|
|
355
|
-
- Use the customer-facing label for buttons, not the CSS selector
|
|
356
|
-
- Preconditions go first; verification steps ("Check: ...") go last
|
|
357
|
-
```
|
|
358
|
-
|
|
359
|
-
Context files work with all LLM providers (`local`, `ollama`, `openai`, `anthropic`). The `heuristic` provider ignores the file (no prompt involved).
|
|
360
|
-
|
|
361
|
-
#### Option H — Freeze steps in source files (stable across runs)
|
|
362
|
-
|
|
363
|
-
By default, AI-generated steps exist only in Azure DevOps. Enable `writebackDocComment` to also write the generated steps as a JSDoc comment directly above each `test()` call in the source file:
|
|
364
|
-
|
|
365
|
-
```json
|
|
366
|
-
{
|
|
367
|
-
"sync": {
|
|
368
|
-
"ai": {
|
|
369
|
-
"provider": "anthropic",
|
|
370
|
-
"model": "claude-sonnet-4-6",
|
|
371
|
-
"apiKey": "$ANTHROPIC_API_KEY",
|
|
372
|
-
"writebackDocComment": true
|
|
373
|
-
}
|
|
374
|
-
}
|
|
375
|
-
}
|
|
376
|
-
```
|
|
377
|
-
|
|
378
|
-
On the **first push**, AI generates the steps and writes them to the source file:
|
|
379
|
-
|
|
380
|
-
```typescript
|
|
381
|
-
/**
|
|
382
|
-
* User can log in with valid credentials
|
|
383
|
-
* Description: Verifies the login form accepts a correct email/password pair
|
|
384
|
-
* 1. Navigate to the login page
|
|
385
|
-
* 2. Enter a valid email address
|
|
386
|
-
* 3. Enter the matching password
|
|
387
|
-
* 4. Click the Sign In button
|
|
388
|
-
* 5. Check: The dashboard is displayed
|
|
389
|
-
*/
|
|
390
|
-
test('should log in with valid credentials', async ({ page }) => { ... });
|
|
391
|
-
```
|
|
392
|
-
|
|
393
|
-
On **every subsequent push**, the parser reads the JSDoc back — AI is not re-invoked. Steps remain stable even if the test body changes.
|
|
394
|
-
|
|
395
|
-
> Only applies to JS/TS frameworks (`javascript`, `playwright`, `puppeteer`, `cypress`, `detox`). Has no effect when `sync.disableLocalChanges` is `true`.
|
|
396
|
-
|
|
397
|
-
#### Disable AI entirely
|
|
398
|
-
|
|
399
|
-
```bash
|
|
400
|
-
ado-sync push --ai-provider none
|
|
401
|
-
```
|
|
402
|
-
|
|
403
|
-
> Tests with existing doc comments (JSDoc / Javadoc / C# XML doc / Python docstring) that already contain steps are **never overwritten**. When `writebackDocComment` is `true`, local JS/TS source files are updated on the first push to freeze the generated steps; subsequent pushes leave the files unchanged.
|
|
404
|
-
|
|
405
|
-
### `publish-test-results`
|
|
406
|
-
|
|
407
|
-
```bash
|
|
408
|
-
ado-sync publish-test-results --testResult results/test.trx
|
|
409
|
-
ado-sync publish-test-results --testResult results/test.xml --testResultFormat junit --dry-run
|
|
112
|
+
ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
|
|
410
113
|
```
|
|
411
114
|
|
|
412
|
-
See [docs/publish-test-results.md](docs/publish-test-results.md) for
|
|
413
|
-
|
|
414
|
-
### `--config-override`
|
|
415
|
-
|
|
416
|
-
All commands accept `--config-override path=value` to set config values without editing the file:
|
|
417
|
-
|
|
418
|
-
```bash
|
|
419
|
-
ado-sync push --config-override testPlan.id=9999
|
|
420
|
-
ado-sync push --config-override sync.disableLocalChanges=true
|
|
421
|
-
```
|
|
115
|
+
See [docs/publish-test-results.md](docs/publish-test-results.md) for TRX, NUnit, Playwright JSON, Cucumber JSON, and attachment options.
|
|
422
116
|
|
|
423
117
|
---
|
|
424
118
|
|
|
@@ -440,632 +134,15 @@ ado-sync push --config-override sync.disableLocalChanges=true
|
|
|
440
134
|
|
|
441
135
|
| Topic | Link |
|
|
442
136
|
|-------|------|
|
|
137
|
+
| **AI agent setup guide** (framework detection → config → validate → push) | [docs/agent-setup.md](docs/agent-setup.md) |
|
|
138
|
+
| CLI reference (all commands and flags) | [docs/cli.md](docs/cli.md) |
|
|
443
139
|
| Full configuration reference | [docs/configuration.md](docs/configuration.md) |
|
|
444
|
-
| Spec file formats (Gherkin, Markdown, C
|
|
445
|
-
|
|
|
446
|
-
|
|
|
140
|
+
| Spec file formats (Gherkin, Markdown, C#, CSV, Excel, JS/TS) | [docs/spec-formats.md](docs/spec-formats.md) |
|
|
141
|
+
| Framework workflow examples (C#, Java, Python, Playwright, Robot, CI) | [docs/workflows.md](docs/workflows.md) |
|
|
142
|
+
| Work Item Links (User Story, Bug, etc.) | [docs/work-item-links.md](docs/work-item-links.md) |
|
|
447
143
|
| Publishing test results | [docs/publish-test-results.md](docs/publish-test-results.md) |
|
|
144
|
+
| Advanced features (format, state, fieldUpdates, attachments, AI, CI) | [docs/advanced.md](docs/advanced.md) |
|
|
145
|
+
| MCP Server (use ado-sync from AI agents) | [docs/mcp-server.md](docs/mcp-server.md) |
|
|
146
|
+
| Troubleshooting | [docs/troubleshooting.md](docs/troubleshooting.md) |
|
|
448
147
|
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
## Workflow examples
|
|
452
|
-
|
|
453
|
-
### Day-to-day: local changes first
|
|
454
|
-
|
|
455
|
-
```bash
|
|
456
|
-
# Edit your .feature or .md files, then push
|
|
457
|
-
ado-sync push
|
|
458
|
-
```
|
|
459
|
-
|
|
460
|
-
### Day-to-day: Azure changes first
|
|
461
|
-
|
|
462
|
-
```bash
|
|
463
|
-
# Someone edited a Test Case in the Azure DevOps UI
|
|
464
|
-
ado-sync pull
|
|
465
|
-
```
|
|
466
|
-
|
|
467
|
-
### C# MSTest / NUnit / SpecFlow: create TCs, run tests, publish results
|
|
468
|
-
|
|
469
|
-
```bash
|
|
470
|
-
# 1. Create TCs and write IDs back into .cs / .feature files
|
|
471
|
-
ado-sync push --dry-run # preview
|
|
472
|
-
ado-sync push # writes [TestProperty("tc","ID")] / [Property("tc","ID")] / @tc:ID
|
|
473
|
-
|
|
474
|
-
# 2a. MSTest — TRX contains [TestProperty] values; TC IDs extracted automatically
|
|
475
|
-
dotnet test --logger "trx;LogFileName=results.trx"
|
|
476
|
-
ado-sync publish-test-results --testResult results/results.trx
|
|
477
|
-
|
|
478
|
-
# 2b. NUnit — use native XML logger so [Property("tc","ID")] values are included
|
|
479
|
-
dotnet test --logger "nunit3;LogFileName=results.xml"
|
|
480
|
-
ado-sync publish-test-results --testResult results/results.xml
|
|
481
|
-
|
|
482
|
-
# 2c. SpecFlow — TRX format (SpecFlow writes @tc:ID into TestProperty automatically)
|
|
483
|
-
dotnet test --logger "trx;LogFileName=results.trx"
|
|
484
|
-
ado-sync publish-test-results --testResult results/results.trx
|
|
485
|
-
```
|
|
486
|
-
|
|
487
|
-
Recommended `ado-sync.json` for C# MSTest:
|
|
488
|
-
|
|
489
|
-
```json
|
|
490
|
-
{
|
|
491
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
492
|
-
"project": "MyProject",
|
|
493
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
494
|
-
"testPlan": { "id": 1234 },
|
|
495
|
-
"local": {
|
|
496
|
-
"type": "csharp",
|
|
497
|
-
"include": ["**/RegressionTests/**/*.cs"],
|
|
498
|
-
"exclude": ["**/*BaseTest.cs", "**/*Helper.cs"]
|
|
499
|
-
},
|
|
500
|
-
"sync": {
|
|
501
|
-
"markAutomated": true,
|
|
502
|
-
"format": { "useExpectedResult": true }
|
|
503
|
-
}
|
|
504
|
-
}
|
|
505
|
-
```
|
|
506
|
-
|
|
507
|
-
Recommended `ado-sync.json` for C# SpecFlow:
|
|
508
|
-
|
|
509
|
-
```json
|
|
510
|
-
{
|
|
511
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
512
|
-
"project": "MyProject",
|
|
513
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
514
|
-
"testPlan": { "id": 1234 },
|
|
515
|
-
"local": {
|
|
516
|
-
"type": "gherkin",
|
|
517
|
-
"include": ["Features/**/*.feature"]
|
|
518
|
-
},
|
|
519
|
-
"sync": {
|
|
520
|
-
"tagPrefix": "tc",
|
|
521
|
-
"markAutomated": true
|
|
522
|
-
}
|
|
523
|
-
}
|
|
524
|
-
```
|
|
525
|
-
|
|
526
|
-
### Java JUnit / TestNG: create TCs and publish results
|
|
527
|
-
|
|
528
|
-
```bash
|
|
529
|
-
# 1. Create TCs and write IDs back into .java files
|
|
530
|
-
ado-sync push --dry-run # preview
|
|
531
|
-
ado-sync push # writes // @tc:ID (JUnit 4/TestNG) or @Tag("tc:ID") (JUnit 5)
|
|
532
|
-
|
|
533
|
-
# 2. Run tests and generate JUnit XML
|
|
534
|
-
mvn test # Surefire writes target/surefire-reports/*.xml by default
|
|
535
|
-
# or with Gradle:
|
|
536
|
-
./gradlew test # writes build/test-results/test/*.xml
|
|
537
|
-
|
|
538
|
-
# 3. Publish results
|
|
539
|
-
ado-sync publish-test-results --testResult target/surefire-reports/TEST-*.xml --testResultFormat junit
|
|
540
|
-
```
|
|
541
|
-
|
|
542
|
-
Recommended `ado-sync.json` for Java:
|
|
543
|
-
|
|
544
|
-
```json
|
|
545
|
-
{
|
|
546
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
547
|
-
"project": "MyProject",
|
|
548
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
549
|
-
"testPlan": { "id": 1234 },
|
|
550
|
-
"local": {
|
|
551
|
-
"type": "java",
|
|
552
|
-
"include": ["**/src/test/**/*.java"],
|
|
553
|
-
"exclude": ["**/*BaseTest.java", "**/*Helper.java"]
|
|
554
|
-
},
|
|
555
|
-
"sync": {
|
|
556
|
-
"markAutomated": true
|
|
557
|
-
}
|
|
558
|
-
}
|
|
559
|
-
```
|
|
560
|
-
|
|
561
|
-
### Python pytest: create TCs and publish results
|
|
562
|
-
|
|
563
|
-
```bash
|
|
564
|
-
# 1. Create TCs and write IDs back into .py files
|
|
565
|
-
ado-sync push --dry-run # preview
|
|
566
|
-
ado-sync push # writes @pytest.mark.tc(ID) above each test function
|
|
567
|
-
|
|
568
|
-
# 2. Run tests and generate JUnit XML
|
|
569
|
-
pytest --junitxml=results/junit.xml
|
|
570
|
-
|
|
571
|
-
# 3. Publish results
|
|
572
|
-
ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
|
|
573
|
-
```
|
|
574
|
-
|
|
575
|
-
By default, TC linking uses `AutomatedTestName` matching (requires `sync.markAutomated: true`).
|
|
576
|
-
|
|
577
|
-
**Optional — embed TC IDs into JUnit XML** for direct linking (more reliable, works even if the class/method is renamed):
|
|
578
|
-
|
|
579
|
-
Add this to `conftest.py`:
|
|
580
|
-
|
|
581
|
-
```python
|
|
582
|
-
# conftest.py
|
|
583
|
-
def pytest_runtest_makereport(item, call):
|
|
584
|
-
"""Write @pytest.mark.tc(N) as a JUnit XML property for ado-sync to pick up."""
|
|
585
|
-
for marker in item.iter_markers("tc"):
|
|
586
|
-
if marker.args:
|
|
587
|
-
item.user_properties.append(("tc", str(marker.args[0])))
|
|
588
|
-
```
|
|
589
|
-
|
|
590
|
-
pytest will then write each TC ID into the JUnit XML as `<property name="tc" value="N"/>`, and ado-sync will link the result directly to that Test Case — no AutomatedTestName matching needed.
|
|
591
|
-
|
|
592
|
-
Recommended `ado-sync.json` for Python:
|
|
593
|
-
|
|
594
|
-
```json
|
|
595
|
-
{
|
|
596
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
597
|
-
"project": "MyProject",
|
|
598
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
599
|
-
"testPlan": { "id": 1234 },
|
|
600
|
-
"local": {
|
|
601
|
-
"type": "python",
|
|
602
|
-
"include": ["tests/**/*.py"],
|
|
603
|
-
"exclude": ["tests/conftest.py", "tests/**/helpers.py"]
|
|
604
|
-
},
|
|
605
|
-
"sync": {
|
|
606
|
-
"markAutomated": true
|
|
607
|
-
}
|
|
608
|
-
}
|
|
609
|
-
```
|
|
610
|
-
|
|
611
|
-
### JavaScript / TypeScript (Jest, Jasmine, WebdriverIO): create TCs
|
|
612
|
-
|
|
613
|
-
```bash
|
|
614
|
-
# 1. Create TCs and write IDs back into .js / .ts files
|
|
615
|
-
ado-sync push --dry-run # preview
|
|
616
|
-
ado-sync push # writes // @tc:ID above each it() / test()
|
|
617
|
-
|
|
618
|
-
# 2. Run tests and generate JUnit XML (Jest example)
|
|
619
|
-
npx jest --reporters=default --reporters=jest-junit
|
|
620
|
-
# JEST_JUNIT_OUTPUT_DIR=results JEST_JUNIT_OUTPUT_NAME=junit.xml
|
|
621
|
-
|
|
622
|
-
# 3. Publish results
|
|
623
|
-
ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
|
|
624
|
-
```
|
|
625
|
-
|
|
626
|
-
### Playwright: create TCs and publish results with screenshots
|
|
627
|
-
|
|
628
|
-
```bash
|
|
629
|
-
# 1. Create TCs and write IDs back into .ts spec files
|
|
630
|
-
ado-sync push --dry-run # preview
|
|
631
|
-
ado-sync push # writes annotation: { type: 'tc', description: 'ID' } into test options
|
|
632
|
-
|
|
633
|
-
# 2. Run Playwright with JSON reporter
|
|
634
|
-
# playwright.config.ts: reporter: [['json', { outputFile: 'results/playwright.json' }]]
|
|
635
|
-
npx playwright test
|
|
636
|
-
|
|
637
|
-
# 3. Publish — TC IDs from native test.annotations; screenshots/videos from attachments[]
|
|
638
|
-
ado-sync publish-test-results --testResult results/playwright.json
|
|
639
|
-
```
|
|
640
|
-
|
|
641
|
-
Recommended `ado-sync.json` for Playwright:
|
|
642
|
-
|
|
643
|
-
```json
|
|
644
|
-
{
|
|
645
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
646
|
-
"project": "MyProject",
|
|
647
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
648
|
-
"testPlan": { "id": 1234 },
|
|
649
|
-
"local": {
|
|
650
|
-
"type": "playwright",
|
|
651
|
-
"include": ["tests/**/*.spec.ts"],
|
|
652
|
-
"exclude": ["**/*.helper.ts"]
|
|
653
|
-
},
|
|
654
|
-
"sync": {
|
|
655
|
-
"markAutomated": true
|
|
656
|
-
}
|
|
657
|
-
}
|
|
658
|
-
```
|
|
659
|
-
|
|
660
|
-
Recommended `ado-sync.json` for Jest/Jasmine/WebdriverIO:
|
|
661
|
-
|
|
662
|
-
```json
|
|
663
|
-
{
|
|
664
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
665
|
-
"project": "MyProject",
|
|
666
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
667
|
-
"testPlan": { "id": 1234 },
|
|
668
|
-
"local": {
|
|
669
|
-
"type": "javascript",
|
|
670
|
-
"include": ["src/**/*.spec.ts", "tests/**/*.test.js"],
|
|
671
|
-
"exclude": ["**/*.helper.ts"]
|
|
672
|
-
},
|
|
673
|
-
"sync": {
|
|
674
|
-
"markAutomated": true
|
|
675
|
-
}
|
|
676
|
-
}
|
|
677
|
-
```
|
|
678
|
-
|
|
679
|
-
### Detox (React Native): create TCs and push
|
|
680
|
-
|
|
681
|
-
```bash
|
|
682
|
-
# 1. Create TCs and write IDs back into .ts files
|
|
683
|
-
ado-sync push --dry-run # preview
|
|
684
|
-
ado-sync push # writes // @tc:ID above each it() / test()
|
|
685
|
-
|
|
686
|
-
# 2. Run Detox tests (Jest runner)
|
|
687
|
-
npx detox test --configuration ios.sim.release
|
|
688
|
-
|
|
689
|
-
# 3. Publish results (Jest JUnit reporter)
|
|
690
|
-
ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
|
|
691
|
-
```
|
|
692
|
-
|
|
693
|
-
Recommended `ado-sync.json` for Detox:
|
|
694
|
-
|
|
695
|
-
```json
|
|
696
|
-
{
|
|
697
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
698
|
-
"project": "MyProject",
|
|
699
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
700
|
-
"testPlan": { "id": 1234 },
|
|
701
|
-
"local": {
|
|
702
|
-
"type": "detox",
|
|
703
|
-
"include": ["e2e/**/*.test.ts"]
|
|
704
|
-
},
|
|
705
|
-
"sync": { "markAutomated": true }
|
|
706
|
-
}
|
|
707
|
-
```
|
|
708
|
-
|
|
709
|
-
### Espresso (Android): create TCs and push
|
|
710
|
-
|
|
711
|
-
```bash
|
|
712
|
-
# 1. Create TCs and write IDs back into .java / .kt files
|
|
713
|
-
ado-sync push --dry-run # preview
|
|
714
|
-
ado-sync push # writes // @tc:ID above @Test
|
|
715
|
-
|
|
716
|
-
# 2. Run instrumented tests and generate JUnit XML
|
|
717
|
-
./gradlew connectedAndroidTest
|
|
718
|
-
# XML output: app/build/outputs/androidTest-results/connected/TEST-*.xml
|
|
719
|
-
|
|
720
|
-
# 3. Publish results
|
|
721
|
-
ado-sync publish-test-results \
|
|
722
|
-
--testResult "app/build/outputs/androidTest-results/connected/TEST-*.xml" \
|
|
723
|
-
--testResultFormat junit
|
|
724
|
-
```
|
|
725
|
-
|
|
726
|
-
Recommended `ado-sync.json` for Espresso:
|
|
727
|
-
|
|
728
|
-
```json
|
|
729
|
-
{
|
|
730
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
731
|
-
"project": "MyProject",
|
|
732
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
733
|
-
"testPlan": { "id": 1234 },
|
|
734
|
-
"local": {
|
|
735
|
-
"type": "espresso",
|
|
736
|
-
"include": ["app/src/androidTest/**/*.java", "app/src/androidTest/**/*.kt"],
|
|
737
|
-
"exclude": ["**/*BaseTest.java"]
|
|
738
|
-
},
|
|
739
|
-
"sync": { "markAutomated": true }
|
|
740
|
-
}
|
|
741
|
-
```
|
|
742
|
-
|
|
743
|
-
### XCUITest (iOS): create TCs and push
|
|
744
|
-
|
|
745
|
-
```bash
|
|
746
|
-
# 1. Create TCs and write IDs back into .swift files
|
|
747
|
-
ado-sync push --dry-run # preview
|
|
748
|
-
ado-sync push # writes // @tc:ID above func test*()
|
|
749
|
-
|
|
750
|
-
# 2. Run XCUITest and export JUnit XML
|
|
751
|
-
xcodebuild test \
|
|
752
|
-
-project MyApp.xcodeproj \
|
|
753
|
-
-scheme MyApp \
|
|
754
|
-
-destination 'platform=iOS Simulator,name=iPhone 15' \
|
|
755
|
-
-resultBundlePath TestResults.xcresult
|
|
756
|
-
xcrun xcresulttool get --path TestResults.xcresult --format junit > results/junit.xml
|
|
757
|
-
|
|
758
|
-
# 3. Publish results
|
|
759
|
-
ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
|
|
760
|
-
```
|
|
761
|
-
|
|
762
|
-
Recommended `ado-sync.json` for XCUITest:
|
|
763
|
-
|
|
764
|
-
```json
|
|
765
|
-
{
|
|
766
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
767
|
-
"project": "MyProject",
|
|
768
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
769
|
-
"testPlan": { "id": 1234 },
|
|
770
|
-
"local": {
|
|
771
|
-
"type": "xcuitest",
|
|
772
|
-
"include": ["UITests/**/*.swift"],
|
|
773
|
-
"exclude": ["UITests/**/*Helper.swift", "UITests/**/*Base.swift"]
|
|
774
|
-
},
|
|
775
|
-
"sync": { "markAutomated": true }
|
|
776
|
-
}
|
|
777
|
-
```
|
|
778
|
-
|
|
779
|
-
### Flutter: create TCs and push
|
|
780
|
-
|
|
781
|
-
```bash
|
|
782
|
-
# 1. Create TCs and write IDs back into _test.dart files
|
|
783
|
-
ado-sync push --dry-run # preview
|
|
784
|
-
ado-sync push # writes // @tc:ID above testWidgets() / test()
|
|
785
|
-
|
|
786
|
-
# 2. Run Flutter tests with machine-readable output
|
|
787
|
-
flutter test --machine > results/flutter_test.jsonl
|
|
788
|
-
# Or generate JUnit XML with the flutter_test_junit package:
|
|
789
|
-
flutter test --reporter junit > results/junit.xml
|
|
790
|
-
|
|
791
|
-
# 3. Publish results
|
|
792
|
-
ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
|
|
793
|
-
```
|
|
794
|
-
|
|
795
|
-
Recommended `ado-sync.json` for Flutter:
|
|
796
|
-
|
|
797
|
-
```json
|
|
798
|
-
{
|
|
799
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
800
|
-
"project": "MyProject",
|
|
801
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
802
|
-
"testPlan": { "id": 1234 },
|
|
803
|
-
"local": {
|
|
804
|
-
"type": "flutter",
|
|
805
|
-
"include": ["test/**/*_test.dart", "integration_test/**/*_test.dart"]
|
|
806
|
-
},
|
|
807
|
-
"sync": { "markAutomated": true }
|
|
808
|
-
}
|
|
809
|
-
```
|
|
810
|
-
|
|
811
|
-
### Robot Framework: create TCs and publish results
|
|
812
|
-
|
|
813
|
-
```bash
|
|
814
|
-
# 1. Create TCs and write IDs back into .robot files
|
|
815
|
-
ado-sync push --dry-run # preview
|
|
816
|
-
ado-sync push # inserts/updates tc:ID in [Tags] of each test case
|
|
817
|
-
|
|
818
|
-
# 2. Run Robot Framework tests (generates output.xml)
|
|
819
|
-
robot --outputdir results tests/
|
|
820
|
-
|
|
821
|
-
# 3. Publish results
|
|
822
|
-
ado-sync publish-test-results --testResult results/output.xml
|
|
823
|
-
```
|
|
824
|
-
|
|
825
|
-
Recommended `ado-sync.json` for Robot Framework:
|
|
826
|
-
|
|
827
|
-
```json
|
|
828
|
-
{
|
|
829
|
-
"orgUrl": "https://dev.azure.com/my-org",
|
|
830
|
-
"project": "MyProject",
|
|
831
|
-
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
832
|
-
"testPlan": { "id": 1234 },
|
|
833
|
-
"local": {
|
|
834
|
-
"type": "robot",
|
|
835
|
-
"include": ["tests/**/*.robot"],
|
|
836
|
-
"exclude": ["tests/resources/**"]
|
|
837
|
-
},
|
|
838
|
-
"sync": {
|
|
839
|
-
"markAutomated": true
|
|
840
|
-
}
|
|
841
|
-
}
|
|
842
|
-
```
|
|
843
|
-
|
|
844
|
-
TC IDs are read directly from the `tc:N` tag in `<tags>` in `output.xml` — no extra config needed.
|
|
845
|
-
|
|
846
|
-
---
|
|
847
|
-
|
|
848
|
-
### CI pipeline
|
|
849
|
-
|
|
850
|
-
```yaml
|
|
851
|
-
# GitHub Actions
|
|
852
|
-
- name: Sync test cases to Azure DevOps
|
|
853
|
-
run: ado-sync push --config-override sync.disableLocalChanges=true
|
|
854
|
-
env:
|
|
855
|
-
AZURE_DEVOPS_TOKEN: ${{ secrets.AZURE_DEVOPS_TOKEN }}
|
|
856
|
-
|
|
857
|
-
- name: Sync test cases to Azure DevOps (with AI summary)
|
|
858
|
-
run: ado-sync push --ai-provider heuristic --config-override sync.disableLocalChanges=true
|
|
859
|
-
env:
|
|
860
|
-
AZURE_DEVOPS_TOKEN: ${{ secrets.AZURE_DEVOPS_TOKEN }}
|
|
861
|
-
|
|
862
|
-
- name: Publish test results
|
|
863
|
-
run: ado-sync publish-test-results --testResult results/test.trx
|
|
864
|
-
env:
|
|
865
|
-
AZURE_DEVOPS_TOKEN: ${{ secrets.AZURE_DEVOPS_TOKEN }}
|
|
866
|
-
```
|
|
867
|
-
|
|
868
|
-
### Check for drift before a PR
|
|
869
|
-
|
|
870
|
-
```bash
|
|
871
|
-
ado-sync status
|
|
872
|
-
```
|
|
873
|
-
|
|
874
|
-
---
|
|
875
|
-
|
|
876
|
-
## Work Item Links
|
|
877
|
-
|
|
878
|
-
Link each Test Case to related Azure DevOps work items (User Stories, Bugs, etc.) automatically on every push.
|
|
879
|
-
|
|
880
|
-
### Configure `sync.links`
|
|
881
|
-
|
|
882
|
-
```json
|
|
883
|
-
{
|
|
884
|
-
"sync": {
|
|
885
|
-
"links": [
|
|
886
|
-
{
|
|
887
|
-
"prefix": "story",
|
|
888
|
-
"relationship": "Microsoft.VSTS.Common.TestedBy-Reverse",
|
|
889
|
-
"workItemType": "User Story"
|
|
890
|
-
},
|
|
891
|
-
{
|
|
892
|
-
"prefix": "bug",
|
|
893
|
-
"relationship": "System.LinkTypes.Related",
|
|
894
|
-
"workItemType": "Bug"
|
|
895
|
-
}
|
|
896
|
-
]
|
|
897
|
-
}
|
|
898
|
-
}
|
|
899
|
-
```
|
|
900
|
-
|
|
901
|
-
| Field | Description |
|
|
902
|
-
|-------|-------------|
|
|
903
|
-
| `prefix` | The tag prefix used in your spec files (e.g. `story` → `@story:555`) |
|
|
904
|
-
| `relationship` | ADO relation type (see common values below) |
|
|
905
|
-
| `workItemType` | Optional — used in log output only |
|
|
906
|
-
|
|
907
|
-
**Common relationship values:**
|
|
908
|
-
|
|
909
|
-
| Relationship | Meaning |
|
|
910
|
-
|---|---|
|
|
911
|
-
| `Microsoft.VSTS.Common.TestedBy-Reverse` | Test Case "Tested By" ↔ User Story |
|
|
912
|
-
| `System.LinkTypes.Related` | Simple "Related" link |
|
|
913
|
-
| `System.LinkTypes.Dependency-Forward` | "Successor" (this item depends on) |
|
|
914
|
-
| `System.LinkTypes.Hierarchy-Reverse` | "Parent" link |
|
|
915
|
-
|
|
916
|
-
### Tag your tests
|
|
917
|
-
|
|
918
|
-
**Gherkin (`.feature`):**
|
|
919
|
-
```gherkin
|
|
920
|
-
# @story:555 @bug:789
|
|
921
|
-
Scenario: User can log in
|
|
922
|
-
Given I am on the login page
|
|
923
|
-
```
|
|
924
|
-
|
|
925
|
-
**JavaScript / TypeScript (Jest, Playwright, Cypress, TestCafe, Puppeteer):**
|
|
926
|
-
```typescript
|
|
927
|
-
// @story:555
|
|
928
|
-
// @bug:789
|
|
929
|
-
test('user can log in', async ({ page }) => { ... });
|
|
930
|
-
```
|
|
931
|
-
|
|
932
|
-
**Markdown (`.md`):**
|
|
933
|
-
```markdown
|
|
934
|
-
### User can log in @story:555 @bug:789
|
|
935
|
-
|
|
936
|
-
1. Navigate to the login page
|
|
937
|
-
2. Check: Login form is visible
|
|
938
|
-
```
|
|
939
|
-
|
|
940
|
-
**Python (pytest):**
|
|
941
|
-
```python
|
|
942
|
-
# @story:555 @bug:789
|
|
943
|
-
def test_user_can_log_in():
|
|
944
|
-
...
|
|
945
|
-
```
|
|
946
|
-
|
|
947
|
-
**C# / Java / Espresso:** Add `// @story:555` in the comment block immediately above the `[TestMethod]` / `@Test` line.
|
|
948
|
-
|
|
949
|
-
**Robot Framework (`.robot`):**
|
|
950
|
-
```robot
|
|
951
|
-
*** Test Cases ***
|
|
952
|
-
My Login Test
|
|
953
|
-
[Tags] tc:12345 story:555 bug:789
|
|
954
|
-
Open Browser ${URL}
|
|
955
|
-
Login As user pass
|
|
956
|
-
```
|
|
957
|
-
|
|
958
|
-
**Swift (XCUITest):**
|
|
959
|
-
```swift
|
|
960
|
-
// @story:555
|
|
961
|
-
// @bug:789
|
|
962
|
-
func testUserCanLogin() { ... }
|
|
963
|
-
```
|
|
964
|
-
|
|
965
|
-
**Dart (Flutter):**
|
|
966
|
-
```dart
|
|
967
|
-
// @story:555
|
|
968
|
-
// @bug:789
|
|
969
|
-
testWidgets('user can log in', (WidgetTester tester) async { ... });
|
|
970
|
-
```
|
|
971
|
-
|
|
972
|
-
**Detox / React Native:**
|
|
973
|
-
```typescript
|
|
974
|
-
// @story:555
|
|
975
|
-
// @bug:789
|
|
976
|
-
it('user can log in', async () => { ... });
|
|
977
|
-
```
|
|
978
|
-
|
|
979
|
-
### How it works
|
|
980
|
-
|
|
981
|
-
- On each `push`, ado-sync reads the `@story:N` / `@bug:N` tags from the spec file.
|
|
982
|
-
- **New links** found in the file are added to the Test Case in Azure DevOps.
|
|
983
|
-
- **Stale links** (present in Azure but no longer tagged locally) are removed automatically.
|
|
984
|
-
- The sync is non-destructive for links not covered by a configured prefix — only the prefixes listed in `sync.links` are managed.
|
|
985
|
-
|
|
986
|
-
---
|
|
987
|
-
|
|
988
|
-
## Environment variables
|
|
989
|
-
|
|
990
|
-
| Variable | Description |
|
|
991
|
-
|----------|-------------|
|
|
992
|
-
| `AZURE_DEVOPS_TOKEN` | PAT or access token. Reference in config with `"$AZURE_DEVOPS_TOKEN"`. |
|
|
993
|
-
| Any name | Any env var — set `auth.token` to `"$MY_VAR_NAME"`. |
|
|
994
|
-
|
|
995
|
-
A `.env` file in the working directory is loaded automatically.
|
|
996
|
-
|
|
997
|
-
---
|
|
998
|
-
|
|
999
|
-
## Troubleshooting
|
|
1000
|
-
|
|
1001
|
-
**`No config file found`**
|
|
1002
|
-
Run `ado-sync init` or pass `-c path/to/config.json`.
|
|
1003
|
-
|
|
1004
|
-
**`Environment variable 'X' is not set`**
|
|
1005
|
-
Your config references `$X` in `auth.token` but the variable is not exported. Run `export X=...` or add it to a `.env` file.
|
|
1006
|
-
|
|
1007
|
-
**Test Case created but ID not written back**
|
|
1008
|
-
Check that the file is writable, or that `sync.disableLocalChanges` is not `true`.
|
|
1009
|
-
|
|
1010
|
-
**`Test case #N not found in Azure DevOps`**
|
|
1011
|
-
The test case was deleted in Azure. Remove the ID tag from the local file to recreate it, or restore the test case in Azure.
|
|
1012
|
-
|
|
1013
|
-
**`Failed to parse <file>`**
|
|
1014
|
-
Gherkin syntax error. Run `npx cucumber-js --dry-run` to identify the problem line.
|
|
1015
|
-
|
|
1016
|
-
**Changes not detected on push**
|
|
1017
|
-
The comparison uses title + steps + description. Touch any step to force an update, or reset the cache by deleting `.ado-sync-state.json`.
|
|
1018
|
-
|
|
1019
|
-
**Conflict detected unexpectedly**
|
|
1020
|
-
Delete `.ado-sync-state.json` to reset the cache. The next push re-populates it from Azure.
|
|
1021
|
-
|
|
1022
|
-
**CSV/Excel IDs not written back**
|
|
1023
|
-
Ensure the file is not open in another application and that `sync.disableLocalChanges` is not `true`. If a TC was deleted from Azure and re-created on push, the old ID in column A is now replaced with the new ID automatically.
|
|
1024
|
-
|
|
1025
|
-
**Excel file not parsed / `No worksheet found`**
|
|
1026
|
-
ado-sync searches for the first worksheet by reading `xl/_rels/workbook.xml.rels` from the xlsx ZIP, falling back to common names (`sheet.xml`, `sheet1.xml`). Non-standard sheet names and multi-sheet workbooks are handled automatically. If parsing still fails, re-export from Azure DevOps.
|
|
1027
|
-
|
|
1028
|
-
**Pull has no effect on CSV files**
|
|
1029
|
-
CSV pull is now supported — `ado-sync pull` updates the Title and step rows in CSV files to match the current Azure DevOps Test Case. Run `ado-sync pull --dry-run` first to preview changes.
|
|
1030
|
-
|
|
1031
|
-
**Pull has no effect on Excel files**
|
|
1032
|
-
Excel (xlsx) pull is not yet supported — only push. Use CSV export instead if bidirectional sync is needed, or pull the changes manually and re-export.
|
|
1033
|
-
|
|
1034
|
-
**C# categories show as constant names instead of values**
|
|
1035
|
-
ado-sync resolves `const string` declarations in the same file. Constants defined in a base class are not resolved — use string literals in `[TestCategory("...")]` for reliable tagging.
|
|
1036
|
-
|
|
1037
|
-
**C# test methods not detected**
|
|
1038
|
-
Ensure the method has `[TestMethod]` on its own line. Nested classes or abstract base methods are not parsed. Add base class files to `local.exclude`.
|
|
1039
|
-
|
|
1040
|
-
**TRX results not linked to Test Cases**
|
|
1041
|
-
For MSTest, TC IDs are read directly from `[TestProperty("tc","ID")]` embedded in the TRX — no further config needed. For NUnit, use `--logger "nunit3;LogFileName=results.xml"` (native XML format) instead of TRX so `[Property("tc","ID")]` values are included. If neither is available, set `sync.markAutomated: true` and rely on `AutomatedTestName` FQMN matching.
|
|
1042
|
-
|
|
1043
|
-
**Java test methods not detected**
|
|
1044
|
-
Ensure each test method has a `@Test` annotation. Abstract base methods and methods with only `@Before`/`@After` are not parsed. Add base class files to `local.exclude`.
|
|
1045
|
-
|
|
1046
|
-
**Java ID not written back (JUnit 5)**
|
|
1047
|
-
ado-sync writes `@Tag("tc:ID")` above the `@Test` annotation. Ensure the file is writable. The `@Tag` import (`org.junit.jupiter.api.Tag`) must already be present or will be added automatically.
|
|
1048
|
-
|
|
1049
|
-
**Python test functions not detected**
|
|
1050
|
-
ado-sync detects functions starting with `test_` at module level and inside classes. Ensure functions follow the `def test_*()` convention. Abstract base test methods should be excluded from `local.include`.
|
|
1051
|
-
|
|
1052
|
-
**Python ID not written back**
|
|
1053
|
-
ado-sync writes `@pytest.mark.tc(ID)` directly above the `def test_*` line. Ensure `pytest` is in your test environment. The `pytest` import is not required in the file itself — the mark is a decorator, not a function call.
|
|
1054
|
-
|
|
1055
|
-
**JavaScript/TypeScript tests not detected**
|
|
1056
|
-
ado-sync detects `it()`, `test()`, `xit()`, `xtest()`, and `.only`/`.skip`/`.concurrent` variants. Tests with dynamic titles (template literals or computed values) are skipped — use string literals for the test title.
|
|
1057
|
-
|
|
1058
|
-
**JavaScript ID not written back**
|
|
1059
|
-
ado-sync inserts `// @tc:ID` immediately above the `it()`/`test()` line. There must be no blank line between the comment and the test function call.
|
|
1060
|
-
|
|
1061
|
-
**`publish-test-results` — "TestPointId, testCaseId must be specified for planned test results"**
|
|
1062
|
-
This error means the Test Run was created as a "planned" run (tied to a test plan), which requires test point IDs for each result. ado-sync creates standalone automated runs — do not pass `plan.id` in `runModel`. This is handled automatically; if you see this error, ensure you are on the latest version.
|
|
1063
|
-
|
|
1064
|
-
**TRX screenshots / `<ResultFiles>` not attached**
|
|
1065
|
-
In TRX format, `<ResultFiles>` is a child of `<Output>`, not a direct child of `<UnitTestResult>`. Make sure `TestContext.AddResultFile("path/to/screenshot.png")` is called in your test code. ado-sync reads from the correct nested path automatically.
|
|
1066
|
-
|
|
1067
|
-
**Attachment paths resolve incorrectly**
|
|
1068
|
-
Attachment paths embedded in result files (TRX `<ResultFiles>`, NUnit `<filePath>`, JUnit `[[ATTACHMENT|path]]`, Playwright `attachments[].path`) are resolved **relative to the result file's directory**, not the working directory. Keep result files and screenshots in the same output folder hierarchy as your test runner produces them.
|
|
1069
|
-
|
|
1070
|
-
**"Invalid AttachmentType specified" from Azure DevOps API**
|
|
1071
|
-
Azure DevOps only accepts `GeneralAttachment` and `ConsoleLog` as attachment types. Screenshot and video files are uploaded as `GeneralAttachment` automatically — no special type is needed.
|
|
148
|
+
> **AI models / LLM crawlers:** [`llms.txt`](llms.txt) contains a flat single-file summary of this entire project — framework type mappings, config schema, CLI flags, ID writeback formats, and the full doc index. Read that first.
|