ado-sync 0.1.24 → 0.1.27
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +240 -678
- package/dist/azure/client.d.ts +3 -0
- package/dist/azure/client.js +6 -0
- package/dist/azure/client.js.map +1 -1
- package/dist/azure/test-cases.d.ts +8 -3
- package/dist/azure/test-cases.js +406 -25
- package/dist/azure/test-cases.js.map +1 -1
- package/dist/cli.js +51 -4
- package/dist/cli.js.map +1 -1
- package/dist/config.js +111 -5
- package/dist/config.js.map +1 -1
- package/dist/parsers/csharp.d.ts +30 -0
- package/dist/parsers/csharp.js +257 -0
- package/dist/parsers/csharp.js.map +1 -0
- package/dist/parsers/gherkin.d.ts +4 -1
- package/dist/parsers/gherkin.js +19 -4
- package/dist/parsers/gherkin.js.map +1 -1
- package/dist/parsers/java.d.ts +40 -0
- package/dist/parsers/java.js +329 -0
- package/dist/parsers/java.js.map +1 -0
- package/dist/parsers/javascript.d.ts +33 -0
- package/dist/parsers/javascript.js +261 -0
- package/dist/parsers/javascript.js.map +1 -0
- package/dist/parsers/markdown.d.ts +4 -1
- package/dist/parsers/markdown.js +5 -3
- package/dist/parsers/markdown.js.map +1 -1
- package/dist/parsers/python.d.ts +34 -0
- package/dist/parsers/python.js +305 -0
- package/dist/parsers/python.js.map +1 -0
- package/dist/parsers/shared.d.ts +18 -0
- package/dist/parsers/shared.js +40 -0
- package/dist/parsers/shared.js.map +1 -1
- package/dist/sync/engine.js +114 -5
- package/dist/sync/engine.js.map +1 -1
- package/dist/sync/publish-results.d.ts +49 -0
- package/dist/sync/publish-results.js +476 -0
- package/dist/sync/publish-results.js.map +1 -0
- package/dist/sync/writeback.d.ts +57 -1
- package/dist/sync/writeback.js +243 -0
- package/dist/sync/writeback.js.map +1 -1
- package/dist/types.d.ts +159 -2
- package/docs/advanced.md +350 -0
- package/docs/configuration.md +293 -0
- package/docs/publish-test-results.md +317 -0
- package/docs/spec-formats.md +595 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -2,7 +2,19 @@
|
|
|
2
2
|
|
|
3
3
|
Bidirectional sync between local test specs and Azure DevOps Test Cases.
|
|
4
4
|
|
|
5
|
-
Supports
|
|
5
|
+
Supports a wide range of test file formats and frameworks:
|
|
6
|
+
|
|
7
|
+
| `local.type` | Framework / Format | Files |
|
|
8
|
+
|---|---|---|
|
|
9
|
+
| `gherkin` | Cucumber / Gherkin | `.feature` |
|
|
10
|
+
| `markdown` | Prose specs, Playwright test plans | `.md` |
|
|
11
|
+
| `csharp` | MSTest, NUnit | `.cs` |
|
|
12
|
+
| `java` | JUnit 4, JUnit 5, TestNG + Selenium | `.java` |
|
|
13
|
+
| `python` | pytest + Selenium | `.py` |
|
|
14
|
+
| `javascript` | Jest, Jasmine, WebdriverIO | `.js` / `.ts` |
|
|
15
|
+
| `csv` | Azure DevOps tabular export | `.csv` |
|
|
16
|
+
| `excel` | Azure DevOps tabular export | `.xlsx` |
|
|
17
|
+
|
|
6
18
|
Inspired by [SpecSync](https://docs.specsolutions.eu/specsync/).
|
|
7
19
|
|
|
8
20
|
---
|
|
@@ -10,17 +22,37 @@ Inspired by [SpecSync](https://docs.specsolutions.eu/specsync/).
|
|
|
10
22
|
## How it works
|
|
11
23
|
|
|
12
24
|
```
|
|
13
|
-
Local files
|
|
14
|
-
──────────────
|
|
15
|
-
.feature files
|
|
16
|
-
.md spec files
|
|
17
|
-
.
|
|
18
|
-
.
|
|
19
|
-
|
|
20
|
-
|
|
25
|
+
Local files ado-sync Azure DevOps
|
|
26
|
+
────────────── ───────────────── ────────────────
|
|
27
|
+
.feature files ── push ──► create / update ──► Test Cases
|
|
28
|
+
.md spec files ◄── pull ── apply changes ◄── (Work Items)
|
|
29
|
+
.cs files ── push ──► (push-only) ──► + Associated Automation
|
|
30
|
+
.java files ── push ──► (push-only) ──► + Associated Automation
|
|
31
|
+
.py files ── push ──► (push-only) ──► + Associated Automation
|
|
32
|
+
.js / .ts files ── push ──► (push-only) ──► + Associated Automation
|
|
33
|
+
.csv files ── push ──► (push-only)
|
|
34
|
+
.xlsx files ── push ──► (push-only)
|
|
35
|
+
write ID back
|
|
36
|
+
@tc:12345 / [TestProperty] / @Tag / @pytest.mark / // @tc:
|
|
37
|
+
TRX / JUnit / ── publish-test-results ──► Test Run results (linked to TCs)
|
|
38
|
+
Cucumber JSON
|
|
21
39
|
```
|
|
22
40
|
|
|
23
|
-
On the **first push
|
|
41
|
+
On the **first push**, a new Test Case is created in Azure DevOps and its ID is written back into the local file. Every subsequent push uses that ID to update the existing Test Case.
|
|
42
|
+
|
|
43
|
+
**ID writeback format per framework:**
|
|
44
|
+
|
|
45
|
+
| Framework | ID written as |
|
|
46
|
+
|---|---|
|
|
47
|
+
| Gherkin / Markdown | `@tc:12345` tag / comment |
|
|
48
|
+
| C# MSTest | `[TestProperty("tc", "12345")]` |
|
|
49
|
+
| C# NUnit | `[Property("tc", "12345")]` |
|
|
50
|
+
| Java JUnit 4 / TestNG | `// @tc:12345` comment above `@Test` |
|
|
51
|
+
| Java JUnit 5 | `@Tag("tc:12345")` above `@Test` |
|
|
52
|
+
| Python pytest | `@pytest.mark.tc(12345)` above `def test_*` |
|
|
53
|
+
| JavaScript/TS (Jest/Jasmine/WebdriverIO) | `// @tc:12345` comment above `it()`/`test()` |
|
|
54
|
+
| CSV | Numeric ID in column A |
|
|
55
|
+
| Excel | Numeric ID in cell A |
|
|
24
56
|
|
|
25
57
|
---
|
|
26
58
|
|
|
@@ -28,7 +60,7 @@ On the **first push** of a scenario, a new Test Case is created in Azure DevOps
|
|
|
28
60
|
|
|
29
61
|
```bash
|
|
30
62
|
npm install -g ado-sync
|
|
31
|
-
# or
|
|
63
|
+
# or run without installing
|
|
32
64
|
npx ado-sync --help
|
|
33
65
|
```
|
|
34
66
|
|
|
@@ -36,51 +68,33 @@ npx ado-sync --help
|
|
|
36
68
|
|
|
37
69
|
## Quick start
|
|
38
70
|
|
|
39
|
-
### 1. Generate a config file
|
|
40
|
-
|
|
41
71
|
```bash
|
|
72
|
+
# 1. Generate a config file
|
|
42
73
|
ado-sync init # creates ado-sync.json
|
|
43
|
-
ado-sync init ado-sync.yml #
|
|
74
|
+
ado-sync init ado-sync.yml # YAML format
|
|
75
|
+
|
|
76
|
+
# 2. Edit the config with your org, project, plan ID, and token
|
|
77
|
+
export AZURE_DEVOPS_TOKEN=your_personal_access_token
|
|
78
|
+
|
|
79
|
+
# 3. Preview what will be created
|
|
80
|
+
ado-sync push --dry-run
|
|
81
|
+
|
|
82
|
+
# 4. Push to Azure DevOps
|
|
83
|
+
ado-sync push
|
|
44
84
|
```
|
|
45
85
|
|
|
46
|
-
|
|
86
|
+
Minimal config:
|
|
47
87
|
|
|
48
88
|
```json
|
|
49
89
|
{
|
|
50
90
|
"orgUrl": "https://dev.azure.com/my-org",
|
|
51
91
|
"project": "MyProject",
|
|
52
|
-
"auth": {
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
},
|
|
56
|
-
"testPlan": {
|
|
57
|
-
"id": 1234,
|
|
58
|
-
"suiteId": 5678
|
|
59
|
-
},
|
|
60
|
-
"local": {
|
|
61
|
-
"type": "gherkin",
|
|
62
|
-
"include": "specs/**/*.feature"
|
|
63
|
-
},
|
|
64
|
-
"sync": {
|
|
65
|
-
"tagPrefix": "tc"
|
|
66
|
-
}
|
|
92
|
+
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
93
|
+
"testPlan": { "id": 1234 },
|
|
94
|
+
"local": { "type": "gherkin", "include": "specs/**/*.feature" }
|
|
67
95
|
}
|
|
68
96
|
```
|
|
69
97
|
|
|
70
|
-
### 3. Set your token
|
|
71
|
-
|
|
72
|
-
```bash
|
|
73
|
-
export AZURE_DEVOPS_TOKEN=your_personal_access_token
|
|
74
|
-
```
|
|
75
|
-
|
|
76
|
-
### 4. Push
|
|
77
|
-
|
|
78
|
-
```bash
|
|
79
|
-
ado-sync push
|
|
80
|
-
```
|
|
81
|
-
|
|
82
|
-
New scenarios are created in Azure DevOps. Their IDs are written back into your local files automatically.
|
|
83
|
-
|
|
84
98
|
---
|
|
85
99
|
|
|
86
100
|
## CLI reference
|
|
@@ -94,49 +108,36 @@ Options:
|
|
|
94
108
|
-h, --help Show help
|
|
95
109
|
|
|
96
110
|
Commands:
|
|
97
|
-
init [output]
|
|
98
|
-
push [options]
|
|
99
|
-
pull [options]
|
|
100
|
-
status [options]
|
|
101
|
-
|
|
111
|
+
init [output] Generate a starter config file
|
|
112
|
+
push [options] Push local specs to Azure DevOps
|
|
113
|
+
pull [options] Pull updates from Azure DevOps into local files
|
|
114
|
+
status [options] Show diff without making changes
|
|
115
|
+
publish-test-results [opts] Publish TRX / JUnit / Cucumber JSON results to Azure DevOps
|
|
116
|
+
help [command] Help for a specific command
|
|
102
117
|
```
|
|
103
118
|
|
|
104
119
|
### `init`
|
|
105
120
|
|
|
106
121
|
```bash
|
|
107
122
|
ado-sync init # creates ado-sync.json
|
|
108
|
-
ado-sync init ado-sync.yml #
|
|
109
|
-
ado-sync init path/to/config.json
|
|
123
|
+
ado-sync init ado-sync.yml # YAML format
|
|
110
124
|
```
|
|
111
125
|
|
|
112
|
-
Creates a starter config file in JSON or YAML format depending on the file extension. Safe — will not overwrite an existing file.
|
|
113
|
-
|
|
114
|
-
---
|
|
115
|
-
|
|
116
126
|
### `push`
|
|
117
127
|
|
|
118
128
|
```bash
|
|
119
129
|
ado-sync push
|
|
120
130
|
ado-sync push --dry-run
|
|
121
|
-
ado-sync push --tags "@smoke"
|
|
122
131
|
ado-sync push --tags "@smoke and not @wip"
|
|
123
132
|
ado-sync push --config-override testPlan.id=9999
|
|
124
|
-
ado-sync -c other-config.json push
|
|
125
133
|
```
|
|
126
134
|
|
|
127
|
-
For every scenario / heading in your local spec files:
|
|
128
|
-
|
|
129
135
|
| Scenario state | Action |
|
|
130
136
|
|----------------|--------|
|
|
131
|
-
| No ID tag
|
|
132
|
-
|
|
|
133
|
-
|
|
|
137
|
+
| No ID tag | Creates a new Test Case, writes ID back |
|
|
138
|
+
| ID tag, no changes | Skipped |
|
|
139
|
+
| ID tag, content changed | Updates the existing Test Case |
|
|
134
140
|
| Deleted locally, still in Azure suite | Tagged `ado-sync:removed` in Azure |
|
|
135
|
-
| Has ID tag but Test Case deleted in Azure | Reported as error |
|
|
136
|
-
|
|
137
|
-
`--dry-run` prints what would happen without modifying anything.
|
|
138
|
-
|
|
139
|
-
---
|
|
140
141
|
|
|
141
142
|
### `pull`
|
|
142
143
|
|
|
@@ -144,622 +145,235 @@ For every scenario / heading in your local spec files:
|
|
|
144
145
|
ado-sync pull
|
|
145
146
|
ado-sync pull --dry-run
|
|
146
147
|
ado-sync pull --tags "@smoke"
|
|
147
|
-
ado-sync pull --config-override sync.conflictAction=skip
|
|
148
148
|
```
|
|
149
149
|
|
|
150
|
-
For every locally-linked scenario (has an ID tag):
|
|
151
|
-
|
|
152
|
-
| Remote state | Action |
|
|
153
|
-
|--------------|--------|
|
|
154
|
-
| Title, steps, or description changed in Azure | Updates the local file |
|
|
155
|
-
| No changes | Skipped |
|
|
156
|
-
| Test Case not found | Reported as error |
|
|
157
|
-
|
|
158
|
-
`--dry-run` prints what would change without modifying local files.
|
|
159
|
-
|
|
160
|
-
---
|
|
161
|
-
|
|
162
150
|
### `status`
|
|
163
151
|
|
|
164
152
|
```bash
|
|
165
153
|
ado-sync status
|
|
166
154
|
ado-sync status --tags "@smoke"
|
|
167
|
-
ado-sync status --config-override testPlan.id=9999
|
|
168
155
|
```
|
|
169
156
|
|
|
170
157
|
Compares local specs against Azure DevOps and prints a diff — no changes made.
|
|
171
158
|
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
### `--config-override`
|
|
175
|
-
|
|
176
|
-
All commands accept one or more `--config-override path=value` options to set config values from the command line without editing the file. Useful in CI pipelines.
|
|
159
|
+
### `publish-test-results`
|
|
177
160
|
|
|
178
161
|
```bash
|
|
179
|
-
|
|
180
|
-
ado-sync
|
|
181
|
-
|
|
182
|
-
# Override tag prefix
|
|
183
|
-
ado-sync push --config-override sync.tagPrefix=azure
|
|
184
|
-
|
|
185
|
-
# Multiple overrides (repeat the flag)
|
|
186
|
-
ado-sync push \
|
|
187
|
-
--config-override testPlan.id=9999 \
|
|
188
|
-
--config-override sync.disableLocalChanges=true
|
|
162
|
+
ado-sync publish-test-results --testResult results/test.trx
|
|
163
|
+
ado-sync publish-test-results --testResult results/test.xml --testResultFormat junit --dry-run
|
|
189
164
|
```
|
|
190
165
|
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
---
|
|
194
|
-
|
|
195
|
-
## Config file reference
|
|
196
|
-
|
|
197
|
-
Config files can be JSON (`.json`) or YAML (`.yml` / `.yaml`). Run `ado-sync init` to generate a template.
|
|
198
|
-
|
|
199
|
-
### Full example (JSON)
|
|
200
|
-
|
|
201
|
-
```json
|
|
202
|
-
{
|
|
203
|
-
"orgUrl": "https://dev.azure.com/YOUR_ORG",
|
|
204
|
-
"project": "YOUR_PROJECT",
|
|
205
|
-
|
|
206
|
-
"auth": {
|
|
207
|
-
"type": "pat",
|
|
208
|
-
"token": "$AZURE_DEVOPS_TOKEN"
|
|
209
|
-
},
|
|
210
|
-
|
|
211
|
-
"testPlan": {
|
|
212
|
-
"id": 1234,
|
|
213
|
-
"suiteId": 5678,
|
|
214
|
-
"suiteMapping": "flat"
|
|
215
|
-
},
|
|
216
|
-
|
|
217
|
-
"local": {
|
|
218
|
-
"type": "gherkin",
|
|
219
|
-
"include": "specs/**/*.feature",
|
|
220
|
-
"exclude": []
|
|
221
|
-
},
|
|
222
|
-
|
|
223
|
-
"sync": {
|
|
224
|
-
"tagPrefix": "tc",
|
|
225
|
-
"titleField": "System.Title",
|
|
226
|
-
"areaPath": "MyProject\\Team A",
|
|
227
|
-
"iterationPath": "MyProject\\Sprint 1",
|
|
228
|
-
"disableLocalChanges": false,
|
|
229
|
-
"conflictAction": "overwrite",
|
|
230
|
-
"links": [
|
|
231
|
-
{ "prefix": "story", "relationship": "System.LinkTypes.Related" },
|
|
232
|
-
{ "prefix": "bug", "relationship": "System.LinkTypes.Related" }
|
|
233
|
-
]
|
|
234
|
-
}
|
|
235
|
-
}
|
|
236
|
-
```
|
|
237
|
-
|
|
238
|
-
---
|
|
239
|
-
|
|
240
|
-
### `orgUrl`
|
|
241
|
-
|
|
242
|
-
Azure DevOps organisation URL. Format: `https://dev.azure.com/YOUR_ORG`.
|
|
243
|
-
|
|
244
|
-
---
|
|
245
|
-
|
|
246
|
-
### `project`
|
|
247
|
-
|
|
248
|
-
Azure DevOps project name.
|
|
249
|
-
|
|
250
|
-
---
|
|
251
|
-
|
|
252
|
-
### `auth`
|
|
253
|
-
|
|
254
|
-
| Field | Description |
|
|
255
|
-
|-------|-------------|
|
|
256
|
-
| `type` | `"pat"` · `"accessToken"` · `"managedIdentity"` |
|
|
257
|
-
| `token` | PAT or access token value. Prefix with `$` to read from an environment variable (e.g. `"$AZURE_DEVOPS_TOKEN"`). |
|
|
258
|
-
| `applicationIdURI` | Required only when `type` is `"managedIdentity"`. |
|
|
259
|
-
|
|
260
|
-
**PAT permissions required:** Test Management (Read & Write), Work Items (Read & Write).
|
|
261
|
-
|
|
262
|
-
---
|
|
263
|
-
|
|
264
|
-
### `testPlan`
|
|
166
|
+
See [docs/publish-test-results.md](docs/publish-test-results.md) for full reference.
|
|
265
167
|
|
|
266
|
-
|
|
267
|
-
|-------|---------|-------------|
|
|
268
|
-
| `id` | *(required)* | ID of the Azure DevOps Test Plan. |
|
|
269
|
-
| `suiteId` | plan root | ID of the Test Suite within the plan. Defaults to the plan's root suite. |
|
|
270
|
-
| `suiteMapping` | `"flat"` | `"flat"` — all Test Cases go into one suite. `"byFolder"` — folder structure is mirrored as nested child suites (e.g. `specs/login/` → suite named `login`). |
|
|
271
|
-
|
|
272
|
-
---
|
|
273
|
-
|
|
274
|
-
### `testPlans` (multi-plan)
|
|
168
|
+
### `--config-override`
|
|
275
169
|
|
|
276
|
-
|
|
170
|
+
All commands accept `--config-override path=value` to set config values without editing the file:
|
|
277
171
|
|
|
278
|
-
```
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
{
|
|
282
|
-
"id": 1001,
|
|
283
|
-
"suiteId": 2001,
|
|
284
|
-
"include": "specs/smoke/**/*.feature"
|
|
285
|
-
},
|
|
286
|
-
{
|
|
287
|
-
"id": 1002,
|
|
288
|
-
"suiteId": 2002,
|
|
289
|
-
"include": "specs/regression/**/*.feature",
|
|
290
|
-
"suiteMapping": "byFolder"
|
|
291
|
-
}
|
|
292
|
-
]
|
|
293
|
-
}
|
|
172
|
+
```bash
|
|
173
|
+
ado-sync push --config-override testPlan.id=9999
|
|
174
|
+
ado-sync push --config-override sync.disableLocalChanges=true
|
|
294
175
|
```
|
|
295
176
|
|
|
296
|
-
| Field | Description |
|
|
297
|
-
|-------|-------------|
|
|
298
|
-
| `id` | Test Plan ID |
|
|
299
|
-
| `suiteId` | *(Optional)* Target suite ID |
|
|
300
|
-
| `suiteMapping` | *(Optional)* `"flat"` or `"byFolder"` |
|
|
301
|
-
| `include` | *(Optional)* Override `local.include` for this plan |
|
|
302
|
-
| `exclude` | *(Optional)* Override `local.exclude` for this plan |
|
|
303
|
-
|
|
304
177
|
---
|
|
305
178
|
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
| Field | Description |
|
|
309
|
-
|-------|-------------|
|
|
310
|
-
| `type` | `"gherkin"` · `"markdown"` · `"csv"` · `"excel"` — see [Spec file formats](#spec-file-formats) |
|
|
311
|
-
| `include` | Glob pattern(s) relative to the config file. String or array. |
|
|
312
|
-
| `exclude` | *(Optional)* Glob pattern(s) to exclude. String or array. |
|
|
313
|
-
|
|
314
|
-
---
|
|
315
|
-
|
|
316
|
-
### `sync`
|
|
317
|
-
|
|
318
|
-
| Field | Default | Description |
|
|
319
|
-
|-------|---------|-------------|
|
|
320
|
-
| `tagPrefix` | `"tc"` | Prefix used in ID tags. See [ID tags](#id-tags). |
|
|
321
|
-
| `titleField` | `"System.Title"` | Azure DevOps field used as the test case title. |
|
|
322
|
-
| `areaPath` | *(none)* | Area path for newly created Test Cases. |
|
|
323
|
-
| `iterationPath` | *(none)* | Iteration path for newly created Test Cases. |
|
|
324
|
-
| `disableLocalChanges` | `false` | When `true`, no local files are modified — no ID writeback, no pull apply. Use in CI pipelines that should not commit file changes. |
|
|
325
|
-
| `conflictAction` | `"overwrite"` | How to handle conflicts (remote changed since last sync AND local also differs). `"overwrite"` — push local version. `"skip"` — emit a conflict result, leave both sides unchanged. `"fail"` — throw an error listing all conflicts. |
|
|
326
|
-
| `links` | `[]` | Work item link configurations. See [Work item linking](#work-item-linking). |
|
|
327
|
-
|
|
328
|
-
---
|
|
329
|
-
|
|
330
|
-
## ID tags
|
|
331
|
-
|
|
332
|
-
After a scenario is pushed for the first time, ado-sync writes the Azure Test Case ID back into the local file. The format depends on the spec type.
|
|
333
|
-
|
|
334
|
-
| Format | Writeback location |
|
|
335
|
-
|--------|--------------------|
|
|
336
|
-
| Gherkin | `@tc:12345` tag above the `Scenario:` line |
|
|
337
|
-
| Markdown | `@tc:12345` tag after the `### heading` |
|
|
338
|
-
| CSV | Numeric ID in column A of the matching title row |
|
|
339
|
-
| Excel | Numeric ID in cell A of the matching title row |
|
|
340
|
-
|
|
341
|
-
### Gherkin
|
|
342
|
-
|
|
343
|
-
The ID tag is inserted on its own line immediately above the `Scenario:` or `Scenario Outline:` keyword:
|
|
344
|
-
|
|
345
|
-
```gherkin
|
|
346
|
-
Feature: Login
|
|
179
|
+
## Output symbols
|
|
347
180
|
|
|
348
|
-
@tc:1042
|
|
349
|
-
Scenario: Successful login with valid credentials
|
|
350
|
-
Given I am on the login page
|
|
351
|
-
When I enter username "standard_user" and password "secret_sauce"
|
|
352
|
-
Then I am redirected to the inventory page
|
|
353
181
|
```
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
Then I can see "<dashboard>"
|
|
362
|
-
|
|
363
|
-
Examples:
|
|
364
|
-
| role | dashboard |
|
|
365
|
-
| admin | Admin view |
|
|
366
|
-
| tester | Test view |
|
|
182
|
+
+ created — new Test Case created in Azure DevOps
|
|
183
|
+
~ updated — existing Test Case updated
|
|
184
|
+
↓ pulled — local file updated from Azure DevOps
|
|
185
|
+
= skipped — no changes detected
|
|
186
|
+
! conflict — both sides changed (see conflictAction)
|
|
187
|
+
− removed — local scenario deleted; Test Case tagged ado-sync:removed in Azure
|
|
188
|
+
✗ error — something went wrong (see detail message)
|
|
367
189
|
```
|
|
368
190
|
|
|
369
|
-
### Markdown
|
|
370
|
-
|
|
371
|
-
The ID tag is inserted on the line immediately after the `### heading`:
|
|
372
|
-
|
|
373
|
-
```markdown
|
|
374
|
-
### 1. Login (happy path)
|
|
375
|
-
@tc:1042
|
|
376
|
-
|
|
377
|
-
Assumption: Fresh browser session.
|
|
378
|
-
|
|
379
|
-
Steps:
|
|
380
|
-
1. Enter username `standard_user` in the Username field.
|
|
381
|
-
2. Enter password `secret_sauce` in the Password field.
|
|
382
|
-
3. Click `Login`.
|
|
383
|
-
|
|
384
|
-
Expected results:
|
|
385
|
-
- User is redirected to `/inventory.html`.
|
|
386
|
-
- Product list is visible.
|
|
387
|
-
|
|
388
191
|
---
|
|
389
|
-
```
|
|
390
|
-
|
|
391
|
-
### Custom prefix
|
|
392
|
-
|
|
393
|
-
Change the `sync.tagPrefix` in your config to use a different prefix:
|
|
394
|
-
|
|
395
|
-
```json
|
|
396
|
-
{ "sync": { "tagPrefix": "azure" } }
|
|
397
|
-
```
|
|
398
192
|
|
|
399
|
-
|
|
400
|
-
- Gherkin: `@azure:1042`
|
|
401
|
-
- Markdown: `@azure:1042`
|
|
193
|
+
## Documentation
|
|
402
194
|
|
|
403
|
-
|
|
195
|
+
| Topic | Link |
|
|
196
|
+
|-------|------|
|
|
197
|
+
| Full configuration reference | [docs/configuration.md](docs/configuration.md) |
|
|
198
|
+
| Spec file formats (Gherkin, Markdown, C# MSTest, CSV, Excel) | [docs/spec-formats.md](docs/spec-formats.md) |
|
|
199
|
+
| Advanced features (format, state, fieldUpdates, customizations, attachments, CI mode) | [docs/advanced.md](docs/advanced.md) |
|
|
200
|
+
| Publishing test results | [docs/publish-test-results.md](docs/publish-test-results.md) |
|
|
404
201
|
|
|
405
202
|
---
|
|
406
203
|
|
|
407
|
-
##
|
|
204
|
+
## Workflow examples
|
|
408
205
|
|
|
409
|
-
|
|
206
|
+
### Day-to-day: local changes first
|
|
410
207
|
|
|
411
208
|
```bash
|
|
412
|
-
#
|
|
413
|
-
ado-sync push
|
|
414
|
-
|
|
415
|
-
# Sync everything except @wip
|
|
416
|
-
ado-sync push --tags "not @wip"
|
|
417
|
-
|
|
418
|
-
# Combine tags with and / or
|
|
419
|
-
ado-sync push --tags "@smoke and not @slow"
|
|
420
|
-
ado-sync pull --tags "@regression or @critical"
|
|
421
|
-
```
|
|
422
|
-
|
|
423
|
-
Tags are evaluated against all tags on a scenario, including:
|
|
424
|
-
|
|
425
|
-
- Tags inherited from the Feature block (Gherkin)
|
|
426
|
-
- Tags on the Scenario / Scenario Outline block
|
|
427
|
-
- Tags on individual Examples tables
|
|
428
|
-
- Inline `<!-- tags: -->` comments in Markdown (see below)
|
|
429
|
-
- **Path-based auto-tags** — directory segments prefixed with `@` are automatically applied as tags to all scenarios in that directory:
|
|
430
|
-
|
|
431
|
-
```
|
|
432
|
-
specs/
|
|
433
|
-
@smoke/
|
|
434
|
-
login.feature ← all scenarios get tag 'smoke'
|
|
435
|
-
@regression/
|
|
436
|
-
@slow/
|
|
437
|
-
checkout.feature ← all scenarios get tags 'regression' and 'slow'
|
|
438
|
-
```
|
|
439
|
-
|
|
440
|
-
```bash
|
|
441
|
-
ado-sync push --tags "@smoke" # only push specs/@smoke/** scenarios
|
|
442
|
-
```
|
|
443
|
-
|
|
444
|
-
### Tags in Markdown
|
|
445
|
-
|
|
446
|
-
Add a `<!-- tags: -->` HTML comment anywhere inside a scenario block to assign tags:
|
|
447
|
-
|
|
448
|
-
```markdown
|
|
449
|
-
### Login with valid credentials
|
|
450
|
-
@tc:1042
|
|
451
|
-
<!-- tags: @smoke, @regression -->
|
|
452
|
-
|
|
453
|
-
Steps:
|
|
454
|
-
1. Navigate to the login page
|
|
455
|
-
2. Enter valid credentials
|
|
456
|
-
3. Click Login
|
|
457
|
-
|
|
458
|
-
Expected results:
|
|
459
|
-
- Dashboard is shown
|
|
460
|
-
|
|
461
|
-
---
|
|
462
|
-
```
|
|
463
|
-
|
|
464
|
-
Path-based auto-tagging also works for Markdown files — placing them inside an `@smoke/` folder automatically adds the `smoke` tag.
|
|
465
|
-
|
|
466
|
-
---
|
|
467
|
-
|
|
468
|
-
## Spec file formats
|
|
469
|
-
|
|
470
|
-
### Gherkin `.feature`
|
|
471
|
-
|
|
472
|
-
Standard Gherkin syntax is supported, including:
|
|
473
|
-
- `Feature` / `Scenario` / `Scenario Outline`
|
|
474
|
-
- `Given` / `When` / `Then` / `And` / `But`
|
|
475
|
-
- Feature-level and scenario-level tags
|
|
476
|
-
- `Scenario Outline` with `Examples` tables — creates a **single** parametrized Test Case in Azure (data table included), not one TC per row
|
|
477
|
-
|
|
478
|
-
```gherkin
|
|
479
|
-
Feature: Checkout
|
|
480
|
-
|
|
481
|
-
@smoke
|
|
482
|
-
Scenario: Add item and complete checkout
|
|
483
|
-
Given I am logged in as "standard_user"
|
|
484
|
-
When I add "Sauce Labs Backpack" to the cart
|
|
485
|
-
And I proceed through checkout with name "Test User" and zip "12345"
|
|
486
|
-
Then I see the order confirmation page
|
|
487
|
-
|
|
488
|
-
Scenario Outline: Checkout with different users
|
|
489
|
-
Given I am logged in as "<user>"
|
|
490
|
-
When I complete a checkout
|
|
491
|
-
Then the result is "<result>"
|
|
492
|
-
|
|
493
|
-
Examples:
|
|
494
|
-
| user | result |
|
|
495
|
-
| standard_user | success |
|
|
496
|
-
| performance_glitch_user | success |
|
|
209
|
+
# Edit your .feature or .md files, then push
|
|
210
|
+
ado-sync push
|
|
497
211
|
```
|
|
498
212
|
|
|
499
|
-
###
|
|
500
|
-
|
|
501
|
-
Each `### heading` is treated as one test case. The file can contain any number of scenarios separated by `---`.
|
|
502
|
-
|
|
503
|
-
```markdown
|
|
504
|
-
# My Feature Test Plan
|
|
505
|
-
|
|
506
|
-
## Test scenarios
|
|
507
|
-
|
|
508
|
-
### Login with valid credentials
|
|
509
|
-
<!-- tags: @smoke -->
|
|
510
|
-
|
|
511
|
-
Assumption: Fresh browser session.
|
|
512
|
-
|
|
513
|
-
Steps:
|
|
514
|
-
1. Navigate to https://example.com/login
|
|
515
|
-
2. Enter username "admin" and password "secret"
|
|
516
|
-
3. Click the Login button
|
|
517
|
-
|
|
518
|
-
Expected results:
|
|
519
|
-
- The dashboard page is shown
|
|
520
|
-
- The username appears in the top navigation
|
|
521
|
-
|
|
522
|
-
---
|
|
523
|
-
|
|
524
|
-
### Login with invalid credentials
|
|
525
|
-
|
|
526
|
-
Steps:
|
|
527
|
-
1. Navigate to https://example.com/login
|
|
528
|
-
2. Enter username "wrong" and password "wrong"
|
|
529
|
-
3. Click the Login button
|
|
530
|
-
|
|
531
|
-
Expected results:
|
|
532
|
-
- An error message "Invalid credentials" is displayed
|
|
533
|
-
- The user remains on the login page
|
|
213
|
+
### Day-to-day: Azure changes first
|
|
534
214
|
|
|
535
|
-
|
|
215
|
+
```bash
|
|
216
|
+
# Someone edited a Test Case in the Azure DevOps UI
|
|
217
|
+
ado-sync pull
|
|
536
218
|
```
|
|
537
219
|
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
#### Playwright test-plan markdown
|
|
541
|
-
|
|
542
|
-
Markdown `.md` files generated by the [Playwright MCP agent](https://playwright.dev/docs/test-agents) are fully supported as a local spec source. Set `local.type: "markdown"` and point `local.include` at the generated files to sync them into Azure DevOps as Test Cases.
|
|
220
|
+
### C# MSTest / NUnit: create TCs, run tests, publish results
|
|
543
221
|
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
Set `local.type: "csv"` to parse Azure DevOps / SpecSync tabular CSV exports.
|
|
549
|
-
|
|
550
|
-
**Expected column layout (9 columns):**
|
|
551
|
-
|
|
552
|
-
| Col | Field | Description |
|
|
553
|
-
|-----|-------|-------------|
|
|
554
|
-
| A (0) | ID | Azure Test Case ID — empty for new cases, filled in after first push |
|
|
555
|
-
| B (1) | Work Item Type | Always `Test Case` (ignored) |
|
|
556
|
-
| C (2) | Title | `Scenario: My test` or just `My test` — non-empty on the header row, empty on step rows |
|
|
557
|
-
| D (3) | Test Step | Step number (1, 2, …) — empty on the header row |
|
|
558
|
-
| E (4) | Step Action | Step text, optionally prefixed with a Gherkin keyword (`Given`, `When`, `Then`, …) |
|
|
559
|
-
| F (5) | Step Expected | Expected result text (optional) |
|
|
560
|
-
| G–I (6–8) | Area Path, Assigned To, State | Preserved but not used during parse |
|
|
222
|
+
```bash
|
|
223
|
+
# 1. Create TCs and write IDs back into .cs files
|
|
224
|
+
ado-sync push --dry-run # preview
|
|
225
|
+
ado-sync push # writes [TestProperty("tc","ID")] / [Property("tc","ID")]
|
|
561
226
|
|
|
562
|
-
|
|
227
|
+
# 2a. MSTest — TRX contains [TestProperty] values; TC IDs extracted automatically
|
|
228
|
+
dotnet test --logger "trx;LogFileName=results.trx"
|
|
229
|
+
ado-sync publish-test-results --testResult results/results.trx
|
|
563
230
|
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
,,,,,"1","Given I open the app","",,...
|
|
568
|
-
,,,,,"2","When I enter valid credentials","Then I see the dashboard",...
|
|
569
|
-
"","Test Case","Scenario: Login with bad password",,,,...
|
|
570
|
-
,,,,,"1","Given I enter wrong credentials","Then I see an error",...
|
|
231
|
+
# 2b. NUnit — use native XML logger so [Property("tc","ID")] values are included
|
|
232
|
+
dotnet test --logger "nunit3;LogFileName=results.xml"
|
|
233
|
+
ado-sync publish-test-results --testResult results/results.xml
|
|
571
234
|
```
|
|
572
235
|
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
**ID writeback:** After the first push, the numeric TC ID is written into **column A** of the matching title row.
|
|
576
|
-
|
|
577
|
-
> **Note:** Pull (`ado-sync pull`) is **not supported** for CSV files — these files are typically generated by external tools (e.g. the Playwright planner agent) and are not hand-authored. Only push is supported.
|
|
578
|
-
|
|
579
|
-
**Config example:**
|
|
236
|
+
Recommended `ado-sync.json` for C# MSTest:
|
|
580
237
|
|
|
581
238
|
```json
|
|
582
239
|
{
|
|
240
|
+
"orgUrl": "https://dev.azure.com/my-org",
|
|
241
|
+
"project": "MyProject",
|
|
242
|
+
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
243
|
+
"testPlan": { "id": 1234 },
|
|
583
244
|
"local": {
|
|
584
|
-
"type": "
|
|
585
|
-
"include": "
|
|
245
|
+
"type": "csharp",
|
|
246
|
+
"include": ["**/RegressionTests/**/*.cs"],
|
|
247
|
+
"exclude": ["**/*BaseTest.cs", "**/*Helper.cs"]
|
|
248
|
+
},
|
|
249
|
+
"sync": {
|
|
250
|
+
"markAutomated": true,
|
|
251
|
+
"format": { "useExpectedResult": true }
|
|
586
252
|
}
|
|
587
253
|
}
|
|
588
254
|
```
|
|
589
255
|
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
### Excel `.xlsx`
|
|
256
|
+
### Java JUnit / TestNG: create TCs and publish results
|
|
593
257
|
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
258
|
+
```bash
|
|
259
|
+
# 1. Create TCs and write IDs back into .java files
|
|
260
|
+
ado-sync push --dry-run # preview
|
|
261
|
+
ado-sync push # writes // @tc:ID (JUnit 4/TestNG) or @Tag("tc:ID") (JUnit 5)
|
|
597
262
|
|
|
598
|
-
|
|
263
|
+
# 2. Run tests and generate JUnit XML
|
|
264
|
+
mvn test # Surefire writes target/surefire-reports/*.xml by default
|
|
265
|
+
# or with Gradle:
|
|
266
|
+
./gradlew test # writes build/test-results/test/*.xml
|
|
599
267
|
|
|
600
|
-
|
|
268
|
+
# 3. Publish results
|
|
269
|
+
ado-sync publish-test-results --testResult target/surefire-reports/TEST-*.xml --testResultFormat junit
|
|
270
|
+
```
|
|
601
271
|
|
|
602
|
-
|
|
272
|
+
Recommended `ado-sync.json` for Java:
|
|
603
273
|
|
|
604
274
|
```json
|
|
605
275
|
{
|
|
276
|
+
"orgUrl": "https://dev.azure.com/my-org",
|
|
277
|
+
"project": "MyProject",
|
|
278
|
+
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
279
|
+
"testPlan": { "id": 1234 },
|
|
606
280
|
"local": {
|
|
607
|
-
"type": "
|
|
608
|
-
"include": "
|
|
281
|
+
"type": "java",
|
|
282
|
+
"include": ["**/src/test/**/*.java"],
|
|
283
|
+
"exclude": ["**/*BaseTest.java", "**/*Helper.java"]
|
|
284
|
+
},
|
|
285
|
+
"sync": {
|
|
286
|
+
"markAutomated": true
|
|
609
287
|
}
|
|
610
288
|
}
|
|
611
289
|
```
|
|
612
290
|
|
|
613
|
-
|
|
291
|
+
### Python pytest: create TCs and publish results
|
|
292
|
+
|
|
293
|
+
```bash
|
|
294
|
+
# 1. Create TCs and write IDs back into .py files
|
|
295
|
+
ado-sync push --dry-run # preview
|
|
296
|
+
ado-sync push # writes @pytest.mark.tc(ID) above each test function
|
|
614
297
|
|
|
615
|
-
|
|
298
|
+
# 2. Run tests and generate JUnit XML
|
|
299
|
+
pytest --junitxml=results/junit.xml
|
|
616
300
|
|
|
617
|
-
|
|
301
|
+
# 3. Publish results
|
|
302
|
+
ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
|
|
303
|
+
```
|
|
618
304
|
|
|
619
|
-
|
|
305
|
+
Recommended `ado-sync.json` for Python:
|
|
620
306
|
|
|
621
307
|
```json
|
|
622
308
|
{
|
|
309
|
+
"orgUrl": "https://dev.azure.com/my-org",
|
|
310
|
+
"project": "MyProject",
|
|
311
|
+
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
312
|
+
"testPlan": { "id": 1234 },
|
|
313
|
+
"local": {
|
|
314
|
+
"type": "python",
|
|
315
|
+
"include": ["tests/**/*.py"],
|
|
316
|
+
"exclude": ["tests/conftest.py", "tests/**/helpers.py"]
|
|
317
|
+
},
|
|
623
318
|
"sync": {
|
|
624
|
-
"
|
|
625
|
-
{ "prefix": "story", "relationship": "System.LinkTypes.Related" },
|
|
626
|
-
{ "prefix": "bug", "relationship": "System.LinkTypes.Related" },
|
|
627
|
-
{ "prefix": "req", "relationship": "Microsoft.VSTS.Common.TestedBy-Reverse" }
|
|
628
|
-
]
|
|
319
|
+
"markAutomated": true
|
|
629
320
|
}
|
|
630
321
|
}
|
|
631
322
|
```
|
|
632
323
|
|
|
633
|
-
###
|
|
324
|
+
### JavaScript / TypeScript (Jest, Jasmine, WebdriverIO): create TCs
|
|
634
325
|
|
|
635
|
-
```
|
|
636
|
-
|
|
637
|
-
|
|
638
|
-
|
|
639
|
-
```
|
|
326
|
+
```bash
|
|
327
|
+
# 1. Create TCs and write IDs back into .js / .ts files
|
|
328
|
+
ado-sync push --dry-run # preview
|
|
329
|
+
ado-sync push # writes // @tc:ID above each it() / test()
|
|
640
330
|
|
|
641
|
-
|
|
331
|
+
# 2. Run tests and generate JUnit XML (Jest example)
|
|
332
|
+
npx jest --reporters=default --reporters=jest-junit
|
|
333
|
+
# JEST_JUNIT_OUTPUT_DIR=results JEST_JUNIT_OUTPUT_NAME=junit.xml
|
|
642
334
|
|
|
643
|
-
|
|
644
|
-
|
|
645
|
-
@tc:1042
|
|
646
|
-
<!-- tags: @story:555, @bug:789 -->
|
|
335
|
+
# 3. Publish results
|
|
336
|
+
ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
|
|
647
337
|
```
|
|
648
338
|
|
|
649
|
-
|
|
650
|
-
|
|
651
|
-
---
|
|
652
|
-
|
|
653
|
-
## Suite hierarchy
|
|
654
|
-
|
|
655
|
-
By default, all Test Cases are created in a single flat suite (`suiteMapping: "flat"`). Setting `suiteMapping: "byFolder"` mirrors the folder structure of your spec files as nested child suites in Azure.
|
|
339
|
+
Recommended `ado-sync.json` for Jest/Jasmine/WebdriverIO:
|
|
656
340
|
|
|
657
341
|
```json
|
|
658
342
|
{
|
|
659
|
-
"
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
343
|
+
"orgUrl": "https://dev.azure.com/my-org",
|
|
344
|
+
"project": "MyProject",
|
|
345
|
+
"auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
|
|
346
|
+
"testPlan": { "id": 1234 },
|
|
347
|
+
"local": {
|
|
348
|
+
"type": "javascript",
|
|
349
|
+
"include": ["src/**/*.spec.ts", "tests/**/*.test.js"],
|
|
350
|
+
"exclude": ["**/*.helper.ts"]
|
|
351
|
+
},
|
|
352
|
+
"sync": {
|
|
353
|
+
"markAutomated": true
|
|
663
354
|
}
|
|
664
355
|
}
|
|
665
356
|
```
|
|
666
357
|
|
|
667
|
-
|
|
668
|
-
specs/
|
|
669
|
-
login/
|
|
670
|
-
basic.feature → suite "login" → TC "Successful login"
|
|
671
|
-
checkout/
|
|
672
|
-
happy.feature → suite "checkout" → TC "Add item and checkout"
|
|
673
|
-
```
|
|
674
|
-
|
|
675
|
-
Child suites are created automatically if they do not exist. The suite hierarchy is re-used across runs.
|
|
676
|
-
|
|
677
|
-
---
|
|
358
|
+
### CI pipeline
|
|
678
359
|
|
|
679
|
-
|
|
680
|
-
|
|
681
|
-
|
|
682
|
-
|
|
683
|
-
|
|
684
|
-
|
|
685
|
-
| Value | Behaviour |
|
|
686
|
-
|-------|-----------|
|
|
687
|
-
| `"overwrite"` *(default)* | Push the local version to Azure, overwriting the remote change. |
|
|
688
|
-
| `"skip"` | Emit a `!` conflict result for the scenario and leave both sides unchanged. |
|
|
689
|
-
| `"fail"` | Throw an error listing all conflicting scenarios and abort the sync. |
|
|
690
|
-
|
|
691
|
-
```json
|
|
692
|
-
{ "sync": { "conflictAction": "skip" } }
|
|
693
|
-
```
|
|
694
|
-
|
|
695
|
-
---
|
|
696
|
-
|
|
697
|
-
## Local state cache
|
|
698
|
-
|
|
699
|
-
ado-sync writes a `.ado-sync-state.json` file in the same directory as your config file. This cache stores SHA-256 hashes of the last-synced title, steps, and description for each Test Case, plus the Azure `changedDate` timestamp.
|
|
700
|
-
|
|
701
|
-
**Commit this file to version control** so all team members and CI share the same last-synced state, enabling accurate conflict detection across machines.
|
|
702
|
-
|
|
703
|
-
The cache also speeds up `push` — if neither the local file nor the Azure `changedDate` has changed since the last sync, the Azure API call is skipped entirely and the scenario is marked `skipped`.
|
|
704
|
-
|
|
705
|
-
---
|
|
706
|
-
|
|
707
|
-
## Build server / CI mode
|
|
708
|
-
|
|
709
|
-
Set `sync.disableLocalChanges: true` to prevent ado-sync from writing back to local files. In this mode:
|
|
710
|
-
|
|
711
|
-
- `push` — creates and updates Test Cases in Azure, but does **not** write ID tags back to local files.
|
|
712
|
-
- `pull` — computes what would change but does **not** modify local files (behaves like `--dry-run`).
|
|
360
|
+
```yaml
|
|
361
|
+
# GitHub Actions
|
|
362
|
+
- name: Sync test cases to Azure DevOps
|
|
363
|
+
run: ado-sync push --config-override sync.disableLocalChanges=true
|
|
364
|
+
env:
|
|
365
|
+
AZURE_DEVOPS_TOKEN: ${{ secrets.AZURE_DEVOPS_TOKEN }}
|
|
713
366
|
|
|
714
|
-
|
|
715
|
-
|
|
367
|
+
- name: Publish test results
|
|
368
|
+
run: ado-sync publish-test-results --testResult results/test.trx
|
|
369
|
+
env:
|
|
370
|
+
AZURE_DEVOPS_TOKEN: ${{ secrets.AZURE_DEVOPS_TOKEN }}
|
|
716
371
|
```
|
|
717
372
|
|
|
718
|
-
|
|
373
|
+
### Check for drift before a PR
|
|
719
374
|
|
|
720
375
|
```bash
|
|
721
|
-
ado-sync
|
|
722
|
-
```
|
|
723
|
-
|
|
724
|
-
---
|
|
725
|
-
|
|
726
|
-
## Removed scenario detection
|
|
727
|
-
|
|
728
|
-
When a scenario is deleted from a local spec file but its Test Case still exists in the Azure suite, ado-sync detects this on the next `push` and appends the tag `ado-sync:removed` to the Azure Test Case (without deleting it). A `−` removed line is printed in the output.
|
|
729
|
-
|
|
730
|
-
To completely remove the Test Case from Azure, delete it manually in Test Plans after reviewing.
|
|
731
|
-
|
|
732
|
-
---
|
|
733
|
-
|
|
734
|
-
## YAML config example
|
|
735
|
-
|
|
736
|
-
```yaml
|
|
737
|
-
orgUrl: https://dev.azure.com/my-org
|
|
738
|
-
project: MyProject
|
|
739
|
-
|
|
740
|
-
auth:
|
|
741
|
-
type: pat
|
|
742
|
-
token: $AZURE_DEVOPS_TOKEN
|
|
743
|
-
|
|
744
|
-
testPlan:
|
|
745
|
-
id: 1234
|
|
746
|
-
suiteId: 5678
|
|
747
|
-
suiteMapping: byFolder
|
|
748
|
-
|
|
749
|
-
local:
|
|
750
|
-
type: gherkin
|
|
751
|
-
include: specs/**/*.feature
|
|
752
|
-
exclude:
|
|
753
|
-
- specs/archive/**
|
|
754
|
-
|
|
755
|
-
sync:
|
|
756
|
-
tagPrefix: tc
|
|
757
|
-
areaPath: "MyProject\\QA Team"
|
|
758
|
-
conflictAction: skip
|
|
759
|
-
disableLocalChanges: false
|
|
760
|
-
links:
|
|
761
|
-
- prefix: story
|
|
762
|
-
relationship: System.LinkTypes.Related
|
|
376
|
+
ado-sync status
|
|
763
377
|
```
|
|
764
378
|
|
|
765
379
|
---
|
|
@@ -768,86 +382,10 @@ sync:
|
|
|
768
382
|
|
|
769
383
|
| Variable | Description |
|
|
770
384
|
|----------|-------------|
|
|
771
|
-
| `AZURE_DEVOPS_TOKEN` | PAT or access token. Reference
|
|
772
|
-
| Any name | Any env var
|
|
773
|
-
|
|
774
|
-
You can also use a `.env` file in the working directory. It is loaded automatically.
|
|
775
|
-
|
|
776
|
-
---
|
|
777
|
-
|
|
778
|
-
## Output symbols
|
|
779
|
-
|
|
780
|
-
```
|
|
781
|
-
+ created — new Test Case created in Azure DevOps
|
|
782
|
-
~ updated — existing Test Case updated
|
|
783
|
-
↓ pulled — local file updated from Azure DevOps
|
|
784
|
-
= skipped — no changes detected
|
|
785
|
-
! conflict — both sides changed (see conflictAction)
|
|
786
|
-
− removed — local scenario deleted; Test Case tagged ado-sync:removed in Azure
|
|
787
|
-
✗ error — something went wrong (see detail message)
|
|
788
|
-
```
|
|
789
|
-
|
|
790
|
-
---
|
|
791
|
-
|
|
792
|
-
## Workflow examples
|
|
793
|
-
|
|
794
|
-
### First-time setup
|
|
795
|
-
|
|
796
|
-
```bash
|
|
797
|
-
# Generate config
|
|
798
|
-
ado-sync init
|
|
799
|
-
|
|
800
|
-
# Edit ado-sync.json with your org, project, plan ID, and token
|
|
801
|
-
export AZURE_DEVOPS_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
|
802
|
-
|
|
803
|
-
# Preview what will be created
|
|
804
|
-
ado-sync push --dry-run
|
|
805
|
-
|
|
806
|
-
# Create test cases in Azure DevOps
|
|
807
|
-
ado-sync push
|
|
808
|
-
```
|
|
809
|
-
|
|
810
|
-
### Day-to-day: local changes first
|
|
385
|
+
| `AZURE_DEVOPS_TOKEN` | PAT or access token. Reference in config with `"$AZURE_DEVOPS_TOKEN"`. |
|
|
386
|
+
| Any name | Any env var — set `auth.token` to `"$MY_VAR_NAME"`. |
|
|
811
387
|
|
|
812
|
-
|
|
813
|
-
# Edit your .feature or .md files locally
|
|
814
|
-
# Then push changes to Azure DevOps
|
|
815
|
-
ado-sync push
|
|
816
|
-
```
|
|
817
|
-
|
|
818
|
-
### Day-to-day: Azure changes first
|
|
819
|
-
|
|
820
|
-
```bash
|
|
821
|
-
# Someone edited a test case in Azure DevOps Test Plans UI
|
|
822
|
-
# Pull the changes into your local files
|
|
823
|
-
ado-sync pull
|
|
824
|
-
```
|
|
825
|
-
|
|
826
|
-
### Check for drift before a PR
|
|
827
|
-
|
|
828
|
-
```bash
|
|
829
|
-
ado-sync status
|
|
830
|
-
```
|
|
831
|
-
|
|
832
|
-
### CI pipeline (no file writeback)
|
|
833
|
-
|
|
834
|
-
```yaml
|
|
835
|
-
# GitHub Actions example
|
|
836
|
-
- name: Sync test cases to Azure DevOps
|
|
837
|
-
run: ado-sync push --config-override sync.disableLocalChanges=true
|
|
838
|
-
env:
|
|
839
|
-
AZURE_DEVOPS_TOKEN: ${{ secrets.AZURE_DEVOPS_TOKEN }}
|
|
840
|
-
```
|
|
841
|
-
|
|
842
|
-
### Override plan per environment
|
|
843
|
-
|
|
844
|
-
```bash
|
|
845
|
-
# Push to staging plan
|
|
846
|
-
ado-sync push --config-override testPlan.id=2001
|
|
847
|
-
|
|
848
|
-
# Push to production plan
|
|
849
|
-
ado-sync push --config-override testPlan.id=3001
|
|
850
|
-
```
|
|
388
|
+
A `.env` file in the working directory is loaded automatically.
|
|
851
389
|
|
|
852
390
|
---
|
|
853
391
|
|
|
@@ -860,28 +398,52 @@ Run `ado-sync init` or pass `-c path/to/config.json`.
|
|
|
860
398
|
Your config references `$X` in `auth.token` but the variable is not exported. Run `export X=...` or add it to a `.env` file.
|
|
861
399
|
|
|
862
400
|
**Test Case created but ID not written back**
|
|
863
|
-
Check that the
|
|
401
|
+
Check that the file is writable, or that `sync.disableLocalChanges` is not `true`.
|
|
864
402
|
|
|
865
403
|
**`Test case #N not found in Azure DevOps`**
|
|
866
404
|
The test case was deleted in Azure. Remove the ID tag from the local file to recreate it, or restore the test case in Azure.
|
|
867
405
|
|
|
868
406
|
**`Failed to parse <file>`**
|
|
869
|
-
Gherkin syntax error
|
|
407
|
+
Gherkin syntax error. Run `npx cucumber-js --dry-run` to identify the problem line.
|
|
870
408
|
|
|
871
409
|
**Changes not detected on push**
|
|
872
|
-
The comparison uses title + steps + description.
|
|
410
|
+
The comparison uses title + steps + description. Touch any step to force an update, or reset the cache by deleting `.ado-sync-state.json`.
|
|
873
411
|
|
|
874
412
|
**Conflict detected unexpectedly**
|
|
875
|
-
Delete `.ado-sync-state.json` to reset the cache. The next push
|
|
876
|
-
|
|
877
|
-
**`init` generated JSON instead of YAML**
|
|
878
|
-
Pass the filename as a positional argument: `ado-sync init ado-sync.yml` (not `--output`).
|
|
413
|
+
Delete `.ado-sync-state.json` to reset the cache. The next push re-populates it from Azure.
|
|
879
414
|
|
|
880
415
|
**CSV/Excel IDs not written back**
|
|
881
|
-
Ensure the file is not open in
|
|
416
|
+
Ensure the file is not open in another application. Check that `sync.disableLocalChanges` is not `true`.
|
|
882
417
|
|
|
883
418
|
**Excel file not parsed / `No worksheet found`**
|
|
884
|
-
ado-sync looks for `xl/worksheets/sheet.xml` or `xl/worksheets/sheet1.xml` inside the xlsx ZIP.
|
|
419
|
+
ado-sync looks for `xl/worksheets/sheet.xml` or `xl/worksheets/sheet1.xml` inside the xlsx ZIP. Re-export from Azure DevOps to get a compatible file.
|
|
885
420
|
|
|
886
421
|
**Pull has no effect on CSV/Excel files**
|
|
887
|
-
Pull is not supported for CSV and Excel
|
|
422
|
+
Pull is not supported for CSV and Excel — only push. These formats are managed by external tools.
|
|
423
|
+
|
|
424
|
+
**C# categories show as constant names instead of values**
|
|
425
|
+
ado-sync resolves `const string` declarations in the same file. Constants defined in a base class are not resolved — use string literals in `[TestCategory("...")]` for reliable tagging.
|
|
426
|
+
|
|
427
|
+
**C# test methods not detected**
|
|
428
|
+
Ensure the method has `[TestMethod]` on its own line. Nested classes or abstract base methods are not parsed. Add base class files to `local.exclude`.
|
|
429
|
+
|
|
430
|
+
**TRX results not linked to Test Cases**
|
|
431
|
+
For MSTest, TC IDs are read directly from `[TestProperty("tc","ID")]` embedded in the TRX — no further config needed. For NUnit, use `--logger "nunit3;LogFileName=results.xml"` (native XML format) instead of TRX so `[Property("tc","ID")]` values are included. If neither is available, set `sync.markAutomated: true` and rely on `AutomatedTestName` FQMN matching.
|
|
432
|
+
|
|
433
|
+
**Java test methods not detected**
|
|
434
|
+
Ensure each test method has a `@Test` annotation. Abstract base methods and methods with only `@Before`/`@After` are not parsed. Add base class files to `local.exclude`.
|
|
435
|
+
|
|
436
|
+
**Java ID not written back (JUnit 5)**
|
|
437
|
+
ado-sync writes `@Tag("tc:ID")` above the `@Test` annotation. Ensure the file is writable. The `@Tag` import (`org.junit.jupiter.api.Tag`) must already be present or will be added automatically.
|
|
438
|
+
|
|
439
|
+
**Python test functions not detected**
|
|
440
|
+
ado-sync detects functions starting with `test_` at module level and inside classes. Ensure functions follow the `def test_*()` convention. Abstract base test methods should be excluded from `local.include`.
|
|
441
|
+
|
|
442
|
+
**Python ID not written back**
|
|
443
|
+
ado-sync writes `@pytest.mark.tc(ID)` directly above the `def test_*` line. Ensure `pytest` is in your test environment. The `pytest` import is not required in the file itself — the mark is a decorator, not a function call.
|
|
444
|
+
|
|
445
|
+
**JavaScript/TypeScript tests not detected**
|
|
446
|
+
ado-sync detects `it()`, `test()`, `xit()`, `xtest()`, and `.only`/`.skip`/`.concurrent` variants. Tests with dynamic titles (template literals or computed values) are skipped — use string literals for the test title.
|
|
447
|
+
|
|
448
|
+
**JavaScript ID not written back**
|
|
449
|
+
ado-sync inserts `// @tc:ID` immediately above the `it()`/`test()` line. There must be no blank line between the comment and the test function call.
|