ado-sync 0.1.64 → 0.1.67

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (63) hide show
  1. package/README.md +20 -15
  2. package/dist/__tests__/regressions.test.js +1011 -1
  3. package/dist/__tests__/regressions.test.js.map +1 -1
  4. package/dist/ai/generate-spec.d.ts +1 -1
  5. package/dist/ai/generate-spec.js +23 -0
  6. package/dist/ai/generate-spec.js.map +1 -1
  7. package/dist/ai/summarizer.d.ts +3 -2
  8. package/dist/ai/summarizer.js +50 -1
  9. package/dist/ai/summarizer.js.map +1 -1
  10. package/dist/azure/test-cases.d.ts +11 -1
  11. package/dist/azure/test-cases.js +286 -43
  12. package/dist/azure/test-cases.js.map +1 -1
  13. package/dist/cli.js +91 -14
  14. package/dist/cli.js.map +1 -1
  15. package/dist/config.js +74 -1
  16. package/dist/config.js.map +1 -1
  17. package/dist/id-markers.d.ts +1 -0
  18. package/dist/id-markers.js +13 -0
  19. package/dist/id-markers.js.map +1 -1
  20. package/dist/mcp-server.js +1 -1
  21. package/dist/mcp-server.js.map +1 -1
  22. package/dist/sync/cache.d.ts +2 -0
  23. package/dist/sync/cache.js.map +1 -1
  24. package/dist/sync/engine.d.ts +12 -1
  25. package/dist/sync/engine.js +210 -41
  26. package/dist/sync/engine.js.map +1 -1
  27. package/dist/types.d.ts +56 -4
  28. package/llms.txt +12 -11
  29. package/package.json +8 -1
  30. package/docs/advanced.md +0 -988
  31. package/docs/agent-setup.md +0 -204
  32. package/docs/capability-roadmap.md +0 -280
  33. package/docs/cli.md +0 -609
  34. package/docs/configuration.md +0 -322
  35. package/docs/examples/csharp-mstest-local-llm.yaml +0 -35
  36. package/docs/examples/csharp-mstest.yaml +0 -21
  37. package/docs/examples/csharp-nunit.yaml +0 -21
  38. package/docs/examples/csharp-specflow.yaml +0 -16
  39. package/docs/examples/cypress.yaml +0 -21
  40. package/docs/examples/detox-react-native.yaml +0 -21
  41. package/docs/examples/espresso-android.yaml +0 -21
  42. package/docs/examples/flutter-dart.yaml +0 -21
  43. package/docs/examples/java-junit.yaml +0 -21
  44. package/docs/examples/java-testng.yaml +0 -21
  45. package/docs/examples/js-jasmine-wdio.yaml +0 -21
  46. package/docs/examples/js-jest.yaml +0 -21
  47. package/docs/examples/playwright-js.yaml +0 -21
  48. package/docs/examples/playwright-ts.yaml +0 -21
  49. package/docs/examples/puppeteer.yaml +0 -21
  50. package/docs/examples/python-pytest.yaml +0 -21
  51. package/docs/examples/robot-framework.yaml +0 -19
  52. package/docs/examples/testcafe.yaml +0 -21
  53. package/docs/examples/xcuitest-ios.yaml +0 -21
  54. package/docs/mcp-server.md +0 -312
  55. package/docs/publish-test-results.md +0 -939
  56. package/docs/spec-formats.md +0 -1357
  57. package/docs/troubleshooting.md +0 -101
  58. package/docs/vscode-extension.md +0 -139
  59. package/docs/work-item-links.md +0 -115
  60. package/docs/workflows.md +0 -457
  61. package/mkdocs.yml +0 -40
  62. package/requirements-docs.txt +0 -4
  63. package/scripts/build_site.sh +0 -6
@@ -1,1357 +0,0 @@
1
- # Spec file formats
2
-
3
- ---
4
-
5
- ## Feature support matrix
6
-
7
- | `local.type` | Push | Pull | AI summary | Outline expansion | Attachments | Path auto-tags |
8
- |---|:---:|:---:|:---:|:---:|:---:|:---:|
9
- | `gherkin` | ✓ | ✓ | — | ✓ | ✓ | ✓ |
10
- | `reqnroll` | ✓ | ✓ | — | ✓ | ✓ | ✓ |
11
- | `markdown` | ✓ | ✓ | — | — | ✓ | ✓ |
12
- | `csv` | ✓ | — | — | — | — | — |
13
- | `excel` | ✓ | — | — | — | — | — |
14
- | `csharp` | ✓ | — | ✓ | — | — | ✓ |
15
- | `java` | ✓ | — | ✓ | — | — | ✓ |
16
- | `python` | ✓ | — | ✓ | — | — | ✓ |
17
- | `javascript` | ✓ | — | ✓ | — | — | ✓ |
18
- | `playwright` | ✓ | — | ✓ | — | — | ✓ |
19
- | `puppeteer` | ✓ | — | ✓ | — | — | ✓ |
20
- | `cypress` | ✓ | — | ✓ | — | — | ✓ |
21
- | `testcafe` | ✓ | — | ✓ | — | — | ✓ |
22
- | `detox` | ✓ | — | ✓ | — | — | ✓ |
23
- | `espresso` | ✓ | — | ✓ | — | — | ✓ |
24
- | `xcuitest` | ✓ | — | ✓ | — | — | ✓ |
25
- | `flutter` | ✓ | — | ✓ | — | — | ✓ |
26
- | `robot` | ✓ | — | — | — | — | ✓ |
27
-
28
- **Column notes:**
29
- - **Pull** — `ado-sync pull` can create/update local files from Azure Test Cases.
30
- - **AI summary** — `sync.ai` auto-generates title, description and steps from the test function body. Requires a non-`heuristic` provider for best results; `heuristic` regex matching works for all code-based types.
31
- - **Outline expansion** — Scenario Outline / `Examples` tables create a single parametrized TC (not one TC per row).
32
- - **Attachments** — `sync.attachments` file-attachment sync via `@attachment:` tags.
33
- - **Path auto-tags** — directories starting with `@` (e.g. `specs/@smoke/`) are automatically applied as tags.
34
-
35
- ---
36
-
37
- ## Gherkin `.feature`
38
-
39
- Set `local.type: "gherkin"`. Standard Gherkin syntax is supported:
40
-
41
- - `Feature` / `Scenario` / `Scenario Outline` / `Background`
42
- - `Given` / `When` / `Then` / `And` / `But`
43
- - Feature-level and scenario-level tags
44
- - `Scenario Outline` with `Examples` tables — creates a **single** parametrized Test Case in Azure, not one TC per row
45
- - Inline data tables (synced as sub-steps or plain text — see [Format configuration](advanced.md#format-configuration))
46
-
47
- ```gherkin
48
- Feature: Checkout
49
-
50
- Background:
51
- Given I am on the storefront
52
-
53
- @smoke
54
- @tc:1041
55
- Scenario: Add item and complete checkout
56
- Given I am logged in as "standard_user"
57
- When I add "Sauce Labs Backpack" to the cart
58
- And I proceed through checkout with name "Test User" and zip "12345"
59
- Then I see the order confirmation page
60
-
61
- @tc:1042
62
- Scenario Outline: Checkout with different users
63
- Given I am logged in as "<user>"
64
- When I complete a checkout
65
- Then the result is "<result>"
66
-
67
- Examples:
68
- | user | result |
69
- | standard_user | success |
70
- | performance_glitch_user | success |
71
- ```
72
-
73
- ---
74
-
75
- ## Markdown `.md`
76
-
77
- Set `local.type: "markdown"`. Each `### heading` is one test case. Scenarios are separated by `---`.
78
-
79
- ```markdown
80
- # My Feature Test Plan
81
-
82
- ## Test scenarios
83
-
84
- ### Login with valid credentials
85
- @tc:1042 @smoke
86
-
87
- Assumption: Fresh browser session.
88
-
89
- Steps:
90
- 1. Navigate to https://example.com/login
91
- 2. Enter username "admin" and password "secret"
92
- 3. Click the Login button
93
-
94
- Expected results:
95
- - The dashboard page is shown
96
- - The username appears in the top navigation
97
-
98
- ---
99
-
100
- ### Login with invalid credentials
101
-
102
- Steps:
103
- 1. Navigate to https://example.com/login
104
- 2. Enter username "wrong" and password "wrong"
105
- 3. Click the Login button
106
-
107
- Expected results:
108
- - An error message "Invalid credentials" is displayed
109
- - The user remains on the login page
110
-
111
- ---
112
- ```
113
-
114
- Sections recognised (case-insensitive): `Steps:`, `Expected results:`. All other prose is captured as the test case description. Heading number prefixes (`### 1. Title` or `### Title`) are both supported.
115
-
116
- ### Playwright test-plan markdown
117
-
118
- Markdown files generated by the Playwright MCP agent are fully supported. Set `local.type: "markdown"` and point `local.include` at the generated files.
119
-
120
- ---
121
-
122
- ## C# MSTest / NUnit `.cs`
123
-
124
- Set `local.type: "csharp"`. Both **MSTest** and **NUnit** frameworks are supported. Each test method becomes one Test Case.
125
-
126
- ### Framework attribute mapping
127
-
128
- | Concern | MSTest | NUnit |
129
- |---------|--------|-------|
130
- | Test marker | `[TestMethod]` | `[Test]` |
131
- | Category / tag | `[TestCategory("name")]` | `[Category("name")]` |
132
- | Custom property | `[TestProperty("key","val")]` | `[Property("key","val")]` |
133
- | TC ID writeback | `[TestProperty("tc","ID")]` | `[Property("tc","ID")]` |
134
-
135
- The framework is auto-detected per method — mixed files (e.g. a shared helper class) are handled correctly.
136
-
137
- ### Source mapping (both frameworks)
138
-
139
- | C# source | Azure TC field |
140
- |-----------|---------------|
141
- | XML doc `<summary>` first line | TC **Title** |
142
- | Numbered lines `N. text` in summary | TC **Steps** (action) |
143
- | Numbered lines `N. Check: text` | TC **Steps** (expected result, when `useExpectedResult: true`) |
144
- | `[TestCategory("…")]` / `[Category("…")]` | TC **Tags** (string literals and `const string` constants resolved) |
145
- | `[TestProperty("tc","ID")]` / `[Property("tc","ID")]` | TC ID (written back after first push) |
146
- | `Namespace.Class.MethodName` | `AutomatedTestName` (for TRX result linking) |
147
-
148
- ### MSTest example
149
-
150
- ```csharp
151
- /// <summary>
152
- /// Verify CoCounsel dialog opens on Edge
153
- /// Test case: 1883058
154
- /// 1. Sign in to Westlaw Edge
155
- /// 2. Click CoCounsel link from page header
156
- /// 3. Check: Verify CoCounsel dialog opens with correct title
157
- /// 4. Check: Verify Close button is displayed
158
- /// </summary>
159
- [TestMethod]
160
- [TestProperty("tc", "1883058")] // written back by ado-sync after first push
161
- [TestCategory("EdgeAalp")]
162
- [TestCategory("EdgeAalpSmoke")]
163
- public void EdgeAalpAccessTest()
164
- {
165
- // ...
166
- }
167
- ```
168
-
169
- ### NUnit example
170
-
171
- ```csharp
172
- /// <summary>
173
- /// Verify login with valid credentials
174
- /// 1. Navigate to the login page
175
- /// 2. Enter username and password
176
- /// 3. Click the Login button
177
- /// 4. Check: Dashboard page is shown
178
- /// 5. Check: Username appears in the top navigation
179
- /// </summary>
180
- [Test]
181
- [Property("tc", "2001")] // written back by ado-sync after first push
182
- [Category("Smoke")]
183
- [Category("Authentication")]
184
- public void LoginWithValidCredentials()
185
- {
186
- // ...
187
- }
188
- ```
189
-
190
- ### Recommended config
191
-
192
- ```json
193
- {
194
- "local": {
195
- "type": "csharp",
196
- "include": ["**/RegressionTests/**/*.cs"],
197
- "exclude": ["**/*BaseTest.cs", "**/*BaseFixture.cs", "**/*Helper.cs"]
198
- },
199
- "sync": {
200
- "markAutomated": true,
201
- "format": { "useExpectedResult": true }
202
- }
203
- }
204
- ```
205
-
206
- With `useExpectedResult: true`, `Check:` lines go into the Expected Result column of each TC step. With `markAutomated: true`, the TC's Associated Automation is set to the fully-qualified method name (`Namespace.Class.Method`), enabling `publish-test-results` to link TRX run outcomes back to TCs.
207
-
208
- ### Notes
209
-
210
- - **Constants** — ado-sync resolves `const string` declarations within the same file. Constants defined in a base class are not resolved; use string literals for reliable tagging.
211
- - **`pull` is not supported** for C# files. Only `push` and `publish-test-results` apply.
212
- - **Base classes / fixtures** — exclude them from `local.include` to avoid treating non-test methods as test cases.
213
- - **NUnit `[TestCase]`** — parameterised data rows (`[TestCase(1, "foo")]`) are not expanded into separate TCs. Only the `[Test]` marker is treated as the test definition.
214
-
215
- ---
216
-
217
- ## Java JUnit / TestNG `.java`
218
-
219
- Set `local.type: "java"`. Supports **JUnit 4**, **JUnit 5**, and **TestNG** (including Selenium-based tests). Each `@Test`-annotated method becomes one Test Case.
220
-
221
- ### Framework attribute mapping
222
-
223
- | Concern | JUnit 4 | JUnit 5 | TestNG |
224
- |---------|---------|---------|--------|
225
- | Test marker | `@Test` (`org.junit.Test`) | `@Test` (`org.junit.jupiter.api.Test`) | `@Test` (`org.testng.annotations.Test`) |
226
- | Tag / category | `@Category(Smoke.class)` | `@Tag("smoke")` | `groups = {"smoke"}` in `@Test` |
227
- | TC ID writeback | `// @tc:ID` above `@Test` | `@Tag("tc:ID")` above `@Test` | `// @tc:ID` above `@Test` |
228
-
229
- The framework is auto-detected per file from import statements — no config required.
230
-
231
- ### Source mapping (all frameworks)
232
-
233
- | Java source | Azure TC field |
234
- |-------------|---------------|
235
- | Javadoc `/** ... */` first non-numbered line | TC **Title** |
236
- | Numbered lines `N. text` in Javadoc | TC **Steps** (action) |
237
- | Numbered lines `N. Check: text` in Javadoc | TC **Steps** (expected result, when `useExpectedResult: true`) |
238
- | `@Category`, `@Tag`, `groups` in `@Test` | TC **Tags** |
239
- | `// @tc:ID` or `@Tag("tc:ID")` | TC ID (written back after first push) |
240
- | `com.example.MyClass.myMethod` | `AutomatedTestName` (for JUnit XML result linking) |
241
-
242
- ### JUnit 5 example
243
-
244
- ```java
245
- import org.junit.jupiter.api.Tag;
246
- import org.junit.jupiter.api.Test;
247
-
248
- class CheckoutTests {
249
-
250
- /**
251
- * Add item and complete checkout
252
- * 1. Sign in as standard_user
253
- * 2. Add "Sauce Labs Backpack" to cart
254
- * 3. Proceed through checkout
255
- * 4. Check: Order confirmation page is shown
256
- */
257
- @Tag("tc:1041") // written back by ado-sync after first push
258
- @Tag("smoke")
259
- @Test
260
- void addItemAndCompleteCheckout() {
261
- // ...
262
- }
263
- }
264
- ```
265
-
266
- ### JUnit 4 example
267
-
268
- ```java
269
- import org.junit.Test;
270
- import org.junit.experimental.categories.Category;
271
-
272
- public class LoginTests {
273
-
274
- /**
275
- * Login with valid credentials
276
- * 1. Navigate to the login page
277
- * 2. Enter username and password
278
- * 3. Check: Dashboard page is shown
279
- */
280
- // @tc:1042 // written back by ado-sync after first push
281
- @Test
282
- @Category(Smoke.class)
283
- public void loginWithValidCredentials() {
284
- // ...
285
- }
286
- }
287
- ```
288
-
289
- ### TestNG example
290
-
291
- ```java
292
- import org.testng.annotations.Test;
293
-
294
- public class SearchTests {
295
-
296
- /**
297
- * Search returns relevant results
298
- * 1. Open the search page
299
- * 2. Enter "junit 5" in the search box
300
- * 3. Check: Results list contains at least one entry
301
- */
302
- // @tc:1043 // written back by ado-sync after first push
303
- @Test(groups = {"regression", "search"})
304
- public void searchReturnsRelevantResults() {
305
- // ...
306
- }
307
- }
308
- ```
309
-
310
- ### Recommended config
311
-
312
- ```json
313
- {
314
- "local": {
315
- "type": "java",
316
- "include": ["**/src/test/**/*.java"],
317
- "exclude": ["**/*BaseTest.java", "**/*Helper.java", "**/*Utils.java"]
318
- },
319
- "sync": {
320
- "markAutomated": true
321
- }
322
- }
323
- ```
324
-
325
- ### Notes
326
-
327
- - **`pull` is not supported** for Java files. Only `push` and `publish-test-results` apply.
328
- - **Abstract base test classes** — exclude them from `local.include` to avoid treating base methods as test cases.
329
- - **JUnit 5 `@Tag`** — the `org.junit.jupiter.api.Tag` import must be present or the file must already use Jupiter annotations; ado-sync adds the tag annotation but not the import if it's missing.
330
- - **TestNG `groups`** — `groups = {"smoke", "regression"}` inside `@Test(...)` are extracted as TC tags.
331
- - **`@ParameterizedTest` / `@DataProvider`** — parameterised data rows are not expanded into separate TCs; only the method definition is treated as the test case.
332
-
333
- ---
334
-
335
- ## Python pytest `.py`
336
-
337
- Set `local.type: "python"`. Each `def test_*` function — at module level or inside a class — becomes one Test Case. No extra dependencies required beyond `pytest` itself.
338
-
339
- ### Source mapping
340
-
341
- | Python source | Azure TC field |
342
- |---------------|---------------|
343
- | Docstring first non-numbered line | TC **Title** |
344
- | Numbered lines `N. text` in docstring | TC **Steps** (action) |
345
- | Numbered lines `N. Check: text` in docstring | TC **Steps** (expected result, when `useExpectedResult: true`) |
346
- | `@pytest.mark.<tag>` decorators | TC **Tags** (built-in pytest marks excluded) |
347
- | `@pytest.mark.tc(ID)` | TC ID (written back after first push) |
348
- | `module.path.ClassName.test_method` | `AutomatedTestName` (for JUnit XML result linking) |
349
-
350
- ### Example
351
-
352
- ```python
353
- import pytest
354
-
355
- class TestCheckout:
356
-
357
- @pytest.mark.tc(1041) # written back by ado-sync after first push
358
- @pytest.mark.smoke
359
- def test_add_item_and_complete_checkout(self):
360
- """
361
- Add item and complete checkout
362
- 1. Sign in as standard_user
363
- 2. Add Sauce Labs Backpack to cart
364
- 3. Proceed through checkout
365
- 4. Check: Order confirmation page is shown
366
- """
367
- # ...
368
-
369
- @pytest.mark.tc(1042)
370
- @pytest.mark.regression
371
- def test_checkout_with_invalid_card(self):
372
- """
373
- Checkout fails with invalid card
374
- 1. Add an item to the cart
375
- 2. Enter an invalid credit card number
376
- 3. Click Place Order
377
- 4. Check: Error message is displayed
378
- """
379
- # ...
380
- ```
381
-
382
- ### Recommended config
383
-
384
- ```json
385
- {
386
- "local": {
387
- "type": "python",
388
- "include": ["tests/**/*.py"],
389
- "exclude": ["tests/conftest.py", "tests/**/fixtures.py"]
390
- },
391
- "sync": {
392
- "markAutomated": true
393
- }
394
- }
395
- ```
396
-
397
- ### Notes
398
-
399
- - **`pull` is not supported** for Python files. Only `push` and `publish-test-results` apply.
400
- - **Built-in pytest marks** (`parametrize`, `skip`, `skipif`, `xfail`, `usefixtures`, `filterwarnings`) are not pushed as TC tags.
401
- - **`@pytest.mark.parametrize`** — parameterised data rows are not expanded into separate TCs; only the function definition is treated as the test case.
402
- - **Class hierarchy** — only the immediately enclosing class is used for the `automatedTestName`. Nested classes are supported.
403
-
404
- ---
405
-
406
- ## JavaScript / TypeScript (Jest, Jasmine, WebdriverIO) `.js` / `.ts`
407
-
408
- Set `local.type: "javascript"`. Supports **Jest**, **Jasmine**, and **WebdriverIO** (which uses Jest or Jasmine as its runner). All three share the same `describe()`/`it()`/`test()` API, so a single parser handles all of them.
409
-
410
- > **Playwright Test** (`@playwright/test`) has its own type: use `local.type: "playwright"` instead — see below.
411
- > **Cucumber `.feature` files** are handled by `local.type: "gherkin"` — not this type.
412
-
413
- ### Detected test functions
414
-
415
- | Function | Description |
416
- |----------|-------------|
417
- | `it(title, fn)` | Standard test |
418
- | `test(title, fn)` | Alias for `it` |
419
- | `it.only` / `test.only` | Focused test |
420
- | `it.skip` / `test.skip` | Skipped test (still synced) |
421
- | `xit` / `xtest` | Jasmine-style skip |
422
- | `it.concurrent` / `test.concurrent` | Concurrent test |
423
-
424
- ### Source mapping
425
-
426
- | JavaScript/TS source | Azure TC field |
427
- |----------------------|---------------|
428
- | JSDoc `/** ... */` first non-numbered line | TC **Title** |
429
- | Numbered lines `N. text` in JSDoc | TC **Steps** (action) |
430
- | Numbered lines `N. Check: text` in JSDoc | TC **Steps** (expected result, when `useExpectedResult: true`) |
431
- | `// @tags: smoke, regression` above `it()` | TC **Tags** (comma-separated list) |
432
- | `// @smoke` above `it()` (single-word) | TC **Tag** |
433
- | `// @tc:ID` above `it()` | TC ID (written back after first push) |
434
- | `{basename} > {describe} > {it title}` | `AutomatedTestName` (Jest report format) |
435
-
436
- ### Example
437
-
438
- ```typescript
439
- describe('Checkout', () => {
440
-
441
- /**
442
- * Add item and complete checkout
443
- * 1. Sign in as standard_user
444
- * 2. Add Sauce Labs Backpack to cart
445
- * 3. Proceed through checkout
446
- * 4. Check: Order confirmation page is shown
447
- */
448
- // @tc:1041 // written back by ado-sync after first push
449
- // @tags: smoke, regression
450
- it('adds item and completes checkout', async () => {
451
- // ...
452
- });
453
-
454
- // @tc:1042
455
- // @smoke
456
- it('shows error on invalid card', async () => {
457
- // ...
458
- });
459
-
460
- });
461
- ```
462
-
463
- ### Recommended config
464
-
465
- ```json
466
- {
467
- "local": {
468
- "type": "javascript",
469
- "include": ["src/**/*.spec.ts", "tests/**/*.test.js"],
470
- "exclude": ["**/*.helper.ts", "**/*.fixture.ts"]
471
- },
472
- "sync": {
473
- "markAutomated": true
474
- }
475
- }
476
- ```
477
-
478
- ### Notes
479
-
480
- - **`pull` is not supported** for JavaScript/TypeScript files. Only `push` applies.
481
- - **Dynamic titles** — `it` / `test` calls with template literals or computed values are skipped. Use string literals for reliable syncing.
482
- - **`describe` nesting** — arbitrarily deep `describe()` nesting is supported. All enclosing describe titles are included in the `automatedTestName`.
483
- - **Path-based auto-tagging** — directory segments starting with `@` (e.g. `tests/@smoke/`) are automatically applied as tags on all tests inside.
484
-
485
- ---
486
-
487
- ## Playwright Test `.js` / `.ts`
488
-
489
- Set `local.type: "playwright"`. Supports **Playwright Test** (`@playwright/test`) for both JavaScript and TypeScript. Each `test()` call becomes one Test Case.
490
-
491
- > **Jest/Jasmine/WebdriverIO** tests use `local.type: "javascript"` — not this type.
492
-
493
- ### Detected test functions
494
-
495
- | Function | Description |
496
- |----------|-------------|
497
- | `test(title, fn)` | Standard test |
498
- | `test.only` | Focused test |
499
- | `test.skip` | Skipped test (still synced) |
500
- | `test.fixme` | Test marked as broken (still synced, tagged `wip`) |
501
- | `test.fail` | Test expected to fail (still synced) |
502
- | `test.describe` | Nested describe block |
503
- | `test.describe.parallel` | Parallel describe block |
504
- | `test.describe.serial` | Serial describe block |
505
-
506
- ### ID tagging — native annotation (recommended)
507
-
508
- Playwright Test has a built-in annotation API. Use it to attach the TC ID directly in the test definition — no comments needed:
509
-
510
- ```typescript
511
- // Single annotation (most common)
512
- test('completes checkout', {
513
- annotation: { type: 'tc', description: '1041' },
514
- tag: '@smoke',
515
- }, async ({ page }) => { ... });
516
-
517
- // Tag-style ID (alternative)
518
- test('cart badge shows count', {
519
- tag: ['@tc:1043', '@smoke'],
520
- }, async ({ page }) => { ... });
521
-
522
- // Array form — combine TC ID with other annotations
523
- test.fixme('promo code applies discount', {
524
- annotation: [
525
- { type: 'tc', description: '1042' },
526
- { type: 'issue', description: 'promo-code not yet implemented' },
527
- ],
528
- tag: '@wip',
529
- }, async ({ page }) => { ... });
530
- ```
531
-
532
- **Writeback** — on the first `push`, ado-sync injects `{ annotation: { type: 'tc', description: 'N' } }` into the test options object. Subsequent pushes update the description in place.
533
-
534
- **Comment fallback** — `// @tc:ID` above `test()` is still recognised and written for edge cases where the options object cannot be parsed (e.g., multi-line title spanning several lines).
535
-
536
- ### Source mapping
537
-
538
- | Playwright source | Azure TC field |
539
- |-------------------|---------------|
540
- | JSDoc `/** ... */` first non-numbered line | TC **Title** |
541
- | Numbered lines `N. text` in JSDoc | TC **Steps** (action) |
542
- | Numbered lines `N. Check: text` in JSDoc | TC **Steps** (expected result, when `useExpectedResult: true`) |
543
- | `annotation: { type: 'tc', description: 'N' }` | TC ID (**preferred**) |
544
- | `annotation: [{ type: 'tc', description: 'N' }, ...]` | TC ID (array form) |
545
- | `tag: '@tc:N'` or `tag: ['@tc:N', ...]` | TC ID (tag form) |
546
- | `tag: '@smoke'` etc. | TC **Tags** |
547
- | `// @tags: smoke, regression` above `test()` | TC **Tags** (comment form) |
548
- | `// @tc:ID` above `test()` | TC ID (comment fallback) |
549
- | `{basename} > {describe} > {test title}` | `AutomatedTestName` |
550
-
551
- **Priority**: native `annotation` > native `tag` > `// @tc:N` comment.
552
-
553
- ### TypeScript example
554
-
555
- ```typescript
556
- import { test, expect } from '@playwright/test';
557
-
558
- test.describe('SauceDemo Checkout', () => {
559
-
560
- /**
561
- * Complete checkout with valid details
562
- * 1. Log in as standard_user
563
- * 2. Add Sauce Labs Backpack to cart
564
- * 3. Proceed through checkout and enter name and zip
565
- * 4. Check: Order confirmation page is displayed
566
- */
567
- test('completes checkout with valid details', {
568
- annotation: { type: 'tc', description: '1041' },
569
- tag: '@smoke',
570
- }, async ({ page }) => {
571
- // ...
572
- });
573
-
574
- test.fixme('promo code applies discount', {
575
- annotation: [
576
- { type: 'tc', description: '1042' },
577
- { type: 'issue', description: 'promo-code not yet implemented' },
578
- ],
579
- tag: '@wip',
580
- }, async ({ page }) => {
581
- // known issue — still synced to Azure
582
- });
583
- });
584
-
585
- test.describe.parallel('Cart assertions', () => {
586
-
587
- test('cart badge shows correct count', {
588
- annotation: { type: 'tc', description: '1043' },
589
- }, async ({ page }) => {
590
- // ...
591
- });
592
- });
593
- ```
594
-
595
- ### JavaScript example (comment fallback style)
596
-
597
- ```javascript
598
- const { test, expect } = require('@playwright/test');
599
-
600
- test.describe('SauceDemo Login', () => {
601
-
602
- /**
603
- * Valid credentials redirect to inventory page
604
- * 1. Navigate to https://www.saucedemo.com
605
- * 2. Enter username "standard_user" and password "secret_sauce"
606
- * 3. Click the login button
607
- * 4. Check: URL contains "inventory.html"
608
- */
609
- // @tc:1044 ← comment style also works; ado-sync upgrades to annotation on next push
610
- test('valid credentials redirect to inventory', async ({ page }) => {
611
- // ...
612
- });
613
- });
614
- ```
615
-
616
- ### Recommended config
617
-
618
- ```json
619
- {
620
- "local": {
621
- "type": "playwright",
622
- "include": ["tests/**/*.spec.ts", "tests/**/*.spec.js"],
623
- "exclude": ["**/*.helper.ts", "**/*.fixture.ts"]
624
- },
625
- "sync": {
626
- "markAutomated": true
627
- }
628
- }
629
- ```
630
-
631
- ### Notes
632
-
633
- - **`pull` is not supported** for Playwright files. Only `push` applies.
634
- - **`test.describe` nesting** — arbitrarily deep nesting is supported. All enclosing describe titles are included in the `automatedTestName`.
635
- - **`test.fixme` / `test.fail`** — both are parsed and synced as normal test cases. Use `tag: '@wip'` on `test.fixme` to mark them in Azure.
636
- - **Publishing results** — set `testResultFormat: playwrightJson` when using `publish-test-results`.
637
- - **Native annotation priority** — `annotation: { type: 'tc', … }` is read before `tag:` which is read before `// @tc:N` comments. The tool writes back using native annotation on every sync.
638
-
639
- ---
640
-
641
- ## Puppeteer `.js` / `.ts`
642
-
643
- Set `local.type: "puppeteer"`. Supports **Puppeteer** tests written with Jest or Mocha as the test runner. Puppeteer tests use the same `describe()` / `it()` / `test()` API as Jest, so this type is an alias for `javascript` — choose it for clarity when your project uses Puppeteer.
644
-
645
- ```typescript
646
- import puppeteer from 'puppeteer';
647
-
648
- describe('SauceDemo Login', () => {
649
-
650
- /**
651
- * Valid credentials redirect to inventory page
652
- * 1. Navigate to https://www.saucedemo.com
653
- * 2. Enter "standard_user" into the username field
654
- * 3. Enter "secret_sauce" into the password field
655
- * 4. Click the login button
656
- * 5. Check: URL contains "inventory"
657
- */
658
- // @tc:1060
659
- it('redirects to inventory on valid login', async () => {
660
- // ...
661
- });
662
- });
663
- ```
664
-
665
- ### Recommended config
666
-
667
- ```json
668
- {
669
- "local": {
670
- "type": "puppeteer",
671
- "include": ["tests/**/*.test.ts"],
672
- "exclude": ["**/*.helper.ts"]
673
- },
674
- "sync": { "markAutomated": true }
675
- }
676
- ```
677
-
678
- ### Notes
679
-
680
- - ID writeback format: `// @tc:12345` immediately above the `it()`/`test()` line.
681
- - `pull` is not supported for Puppeteer files — only `push` applies.
682
- - `describe()` nesting is fully supported.
683
-
684
- ---
685
-
686
- ## Cypress `.cy.js` / `.cy.ts`
687
-
688
- Set `local.type: "cypress"`. Supports **Cypress** tests. Cypress uses `describe()` / `context()` / `it()` / `specify()` — all four are detected.
689
-
690
- ```typescript
691
- describe('SauceDemo Login', () => {
692
-
693
- context('valid credentials', () => {
694
-
695
- /**
696
- * Successful login redirects to inventory page
697
- * 1. Visit https://www.saucedemo.com
698
- * 2. Enter "standard_user" into the username field
699
- * 3. Enter "secret_sauce" into the password field
700
- * 4. Click the login button
701
- * 5. Check: URL contains "inventory"
702
- */
703
- // @tc:1070
704
- // @tags: smoke
705
- it('redirects to inventory', () => {
706
- // ...
707
- });
708
-
709
- // @tc:1071
710
- specify('locked out user sees error', () => {
711
- // ...
712
- });
713
- });
714
- });
715
- ```
716
-
717
- ### Recommended config
718
-
719
- ```json
720
- {
721
- "local": {
722
- "type": "cypress",
723
- "include": ["cypress/e2e/**/*.cy.ts", "cypress/e2e/**/*.cy.js"],
724
- "exclude": ["cypress/support/**"]
725
- },
726
- "sync": { "markAutomated": true }
727
- }
728
- ```
729
-
730
- ### Notes
731
-
732
- - `context()` is treated as a `describe()` alias (Cypress convention).
733
- - `specify()` is treated as an `it()` alias (Cypress convention).
734
- - ID writeback format: `// @tc:12345` immediately above the `it()`/`specify()` line.
735
- - `pull` is not supported for Cypress files — only `push` applies.
736
-
737
- ---
738
-
739
- ## TestCafe `.js` / `.ts`
740
-
741
- Set `local.type: "testcafe"`. Supports **TestCafe** tests using the `fixture` / `test` API.
742
-
743
- ### ID tagging — native `test.meta()` (recommended)
744
-
745
- TestCafe has a built-in `.meta()` method that attaches key-value metadata to a test. Use it to store the TC ID natively — no comments needed:
746
-
747
- ```javascript
748
- // Key-value form (recommended — clean and simple)
749
- test.meta('tc', '1080')('redirects to inventory on valid login', async t => { ... });
750
-
751
- // Object form (use when adding multiple metadata fields)
752
- test.meta({ tc: '1081', priority: 'high' })('locked out user sees error', async t => { ... });
753
-
754
- // Combined with test.skip
755
- test.skip.meta('tc', '1082')('skipped test', async t => { ... });
756
- ```
757
-
758
- **Writeback** — on the first `push`, ado-sync injects `.meta('tc', 'N')` between the test function and its title call. For example:
759
- ```
760
- test('title', fn) → test.meta('tc', '12345')('title', fn)
761
- ```
762
- Subsequent pushes update the existing `.meta()` value in place.
763
-
764
- **Comment fallback** — `// @tc:N` above `test()` is still recognised.
765
-
766
- ### Source mapping
767
-
768
- | TestCafe source | Azure TC field |
769
- |-----------------|---------------|
770
- | JSDoc `/** ... */` first non-numbered line | TC **Title** |
771
- | Numbered lines `N. text` in JSDoc | TC **Steps** (action) |
772
- | Numbered lines `N. Check: text` in JSDoc | TC **Steps** (expected result) |
773
- | `test.meta('tc', 'N')` | TC ID (**preferred**) |
774
- | `test.meta({ tc: 'N' })` | TC ID (object form) |
775
- | `// @tags: smoke, regression` above `test()` | TC **Tags** |
776
- | `// @tc:N` above `test()` | TC ID (comment fallback) |
777
- | `{basename} > {fixture} > {test title}` | `AutomatedTestName` |
778
-
779
- ### Example
780
-
781
- ```javascript
782
- fixture('SauceDemo Login')
783
- .page('https://www.saucedemo.com');
784
-
785
- /**
786
- * Valid credentials redirect to inventory page
787
- * 1. Type "standard_user" into the username field
788
- * 2. Type "secret_sauce" into the password field
789
- * 3. Click the login button
790
- * 4. Check: URL contains "inventory"
791
- */
792
- // @smoke
793
- test.meta('tc', '1080')('redirects to inventory on valid login', async t => {
794
- // ...
795
- });
796
-
797
- test.meta({ tc: '1081', priority: 'high' })('locked out user sees error', async t => {
798
- // ...
799
- });
800
-
801
- // Comment style still works (written by ado-sync before meta support was added)
802
- // @tc:1082
803
- test.skip('skipped test — still synced', async t => { ... });
804
- ```
805
-
806
- ### Recommended config
807
-
808
- ```json
809
- {
810
- "local": {
811
- "type": "testcafe",
812
- "include": ["tests/**/*.js", "tests/**/*.ts"],
813
- "exclude": ["tests/helpers/**"]
814
- },
815
- "sync": { "markAutomated": true }
816
- }
817
- ```
818
-
819
- ### Notes
820
-
821
- - The `fixture()` title is used as the group / suite name and included in `automatedTestName`.
822
- - `test.skip()` and `test.only()` are both parsed and synced.
823
- - `pull` is not supported for TestCafe files — only `push` applies.
824
- - JSDoc `/** ... */` above a `test.meta(...)` call is parsed for TC title and numbered steps.
825
-
826
- ---
827
-
828
- ## Appium
829
-
830
- Appium tests are written in your language of choice. Use the `local.type` that matches the language:
831
-
832
- | Appium language binding | Use `local.type` | Notes |
833
- |-------------------------|-----------------|-------|
834
- | JavaScript / TypeScript (WebdriverIO, Mocha, Jest) | `javascript` | Detects `describe()`/`it()`/`test()` |
835
- | Java (JUnit 4, JUnit 5, TestNG) | `java` | Detects `@Test` annotations |
836
- | Python (pytest) | `python` | Detects `def test_*()` functions |
837
- | C# (NUnit, MSTest) | `csharp` | Detects `[TestMethod]`/`[Test]` attributes |
838
-
839
- **JavaScript/TypeScript Appium example:**
840
-
841
- ```typescript
842
- // @tc:12345
843
- // @smoke
844
- describe('Login', () => {
845
- /**
846
- * User can log in with valid credentials
847
- *
848
- * 1. Navigate to the login screen
849
- * 2. Enter email into the email field
850
- * 3. Enter password into the password field
851
- * 4. Tap the login button
852
- * 5. Check: Home screen is displayed
853
- */
854
- it('can log in with valid credentials', async () => {
855
- await driver.execute('mobile: launchApp', { bundleId: 'com.example.app' });
856
- await $('~emailField').setValue('user@example.com');
857
- await $('~passwordField').setValue('secret');
858
- await $('~loginButton').click();
859
- await expect($('~homeScreen')).toBeDisplayed();
860
- });
861
- });
862
- ```
863
-
864
- Recommended `ado-sync.json`:
865
-
866
- ```json
867
- {
868
- "local": {
869
- "type": "javascript",
870
- "include": ["test/**/*.spec.ts"]
871
- },
872
- "sync": { "markAutomated": true }
873
- }
874
- ```
875
-
876
- ---
877
-
878
- ## Detox `.js` / `.ts`
879
-
880
- Set `local.type: "detox"` for React Native end-to-end tests using [Detox](https://wix.github.io/Detox/).
881
-
882
- Detox uses the same Jest `describe()`/`it()` API as Jest — the existing JavaScript parser handles it.
883
-
884
- ```typescript
885
- // @tc:12345
886
- // @smoke
887
- describe('Login flow', () => {
888
- /**
889
- * User can log in with valid credentials
890
- *
891
- * 1. Launch the app
892
- * 2. Type email into the email field
893
- * 3. Tap the login button
894
- * 4. Check: Welcome screen is visible
895
- */
896
- it('logs in with valid credentials', async () => {
897
- await device.launchApp();
898
- await element(by.id('email')).typeText('user@example.com');
899
- await element(by.id('password')).typeText('secret');
900
- await element(by.id('login-btn')).tap();
901
- await expect(element(by.id('welcome-screen'))).toBeVisible();
902
- });
903
- });
904
- ```
905
-
906
- Recommended `ado-sync.json`:
907
-
908
- ```json
909
- {
910
- "local": {
911
- "type": "detox",
912
- "include": ["e2e/**/*.test.ts"],
913
- "exclude": ["e2e/setup/**"]
914
- },
915
- "sync": { "markAutomated": true }
916
- }
917
- ```
918
-
919
- **Notes:**
920
- - `describe()` blocks define the test group (same as Jest).
921
- - `// @tc:12345` is inserted immediately above the `it()`/`test()` line on writeback.
922
- - `pull` is not supported for Detox files — only `push`.
923
-
924
- ---
925
-
926
- ## Espresso `.java` / `.kt`
927
-
928
- Set `local.type: "espresso"` for Android UI tests using [Espresso](https://developer.android.com/training/testing/espresso).
929
-
930
- Espresso tests use JUnit 4 `@Test` annotations — the same parser as `java` handles them.
931
-
932
- ```java
933
- @RunWith(AndroidJUnit4.class)
934
- public class LoginInstrumentedTest {
935
-
936
- // @tc:12345
937
- // @smoke
938
- /**
939
- * User can log in with valid credentials
940
- *
941
- * 1. Type email into the email field
942
- * 2. Type password into the password field
943
- * 3. Click the login button
944
- * 4. Check: Welcome screen is displayed
945
- */
946
- @Test
947
- public void userCanLoginWithValidCredentials() {
948
- onView(withId(R.id.email)).perform(typeText("user@example.com"), closeSoftKeyboard());
949
- onView(withId(R.id.password)).perform(typeText("secret"), closeSoftKeyboard());
950
- onView(withId(R.id.login_button)).perform(click());
951
- onView(withId(R.id.welcome_screen)).check(matches(isDisplayed()));
952
- }
953
- }
954
- ```
955
-
956
- **Kotlin** (same `@Test` detection):
957
-
958
- ```kotlin
959
- @RunWith(AndroidJUnit4::class)
960
- class LoginInstrumentedTest {
961
-
962
- // @tc:12345
963
- @Test
964
- fun `user can login with valid credentials`() {
965
- onView(withId(R.id.email)).perform(typeText("user@example.com"), closeSoftKeyboard())
966
- onView(withId(R.id.login_button)).perform(click())
967
- onView(withId(R.id.welcome_screen)).check(matches(isDisplayed()))
968
- }
969
- }
970
- ```
971
-
972
- Recommended `ado-sync.json`:
973
-
974
- ```json
975
- {
976
- "local": {
977
- "type": "espresso",
978
- "include": ["app/src/androidTest/**/*.java", "app/src/androidTest/**/*.kt"],
979
- "exclude": ["**/*BaseTest.java", "**/*Helper.java"]
980
- },
981
- "sync": { "markAutomated": true }
982
- }
983
- ```
984
-
985
- **Notes:**
986
- - ID writeback format: `// @tc:12345` immediately above the `@Test` annotation line.
987
- - Kotlin backtick method names (`` fun `my test name`() ``) are supported — the method name is used as the fallback title.
988
- - `pull` is not supported — only `push`.
989
-
990
- ---
991
-
992
- ## XCUITest `.swift`
993
-
994
- Set `local.type: "xcuitest"` for iOS and macOS UI automation tests using Apple's [XCTest](https://developer.apple.com/documentation/xctest) framework.
995
-
996
- Tests are `func test*()` methods inside classes that extend `XCTestCase`.
997
-
998
- ```swift
999
- import XCTest
1000
-
1001
- class LoginTests: XCTestCase {
1002
-
1003
- // @tc:12345
1004
- // @smoke
1005
- /// User can log in with valid credentials
1006
- ///
1007
- /// 1. Launch the app
1008
- /// 2. Enter email into the email field
1009
- /// 3. Enter password into the password field
1010
- /// 4. Tap the login button
1011
- /// 5. Check: Welcome screen is visible
1012
- func testUserCanLoginWithValidCredentials() {
1013
- let app = XCUIApplication()
1014
- app.launch()
1015
- app.textFields["Email"].typeText("user@example.com")
1016
- app.secureTextFields["Password"].typeText("secret")
1017
- app.buttons["Log In"].tap()
1018
- XCTAssertTrue(app.staticTexts["Welcome"].exists)
1019
- }
1020
-
1021
- func testLoginFailsWithInvalidPassword() {
1022
- let app = XCUIApplication()
1023
- app.launch()
1024
- app.textFields["Email"].typeText("user@example.com")
1025
- app.secureTextFields["Password"].typeText("wrong")
1026
- app.buttons["Log In"].tap()
1027
- XCTAssertTrue(app.staticTexts["Invalid credentials"].exists)
1028
- }
1029
- }
1030
- ```
1031
-
1032
- Recommended `ado-sync.json`:
1033
-
1034
- ```json
1035
- {
1036
- "local": {
1037
- "type": "xcuitest",
1038
- "include": ["UITests/**/*.swift"],
1039
- "exclude": ["UITests/**/*Helper.swift", "UITests/**/*Base.swift"]
1040
- },
1041
- "sync": { "markAutomated": true }
1042
- }
1043
- ```
1044
-
1045
- **Notes:**
1046
- - The enclosing class name is used as the test group (`automatedTestName = FileName > ClassName > methodName`).
1047
- - Both `///` triple-slash (Swift idiomatic) and `/** ... */` block doc comments are parsed for TC title and numbered steps.
1048
- - Method name → title fallback: `testUserCanLogin` → "User can login", `test_submit_form` → "Submit form".
1049
- - ID writeback format: `// @tc:12345` immediately above the `func test*()` line.
1050
- - `pull` is not supported — only `push`.
1051
-
1052
- ---
1053
-
1054
- ## Flutter `_test.dart`
1055
-
1056
- Set `local.type: "flutter"` for Flutter widget tests and integration tests using the [`flutter_test`](https://api.flutter.dev/flutter/flutter_test/flutter_test-library.html) package.
1057
-
1058
- ```dart
1059
- import 'package:flutter_test/flutter_test.dart';
1060
-
1061
- void main() {
1062
- group('Login', () {
1063
- // @tc:12345
1064
- // @smoke
1065
- /// User can log in with valid credentials
1066
- ///
1067
- /// 1. Tap the email field
1068
- /// 2. Enter the email address
1069
- /// 3. Tap the password field
1070
- /// 4. Enter the password
1071
- /// 5. Tap the Login button
1072
- /// 6. Check: Welcome screen is visible
1073
- testWidgets('can log in with valid credentials', (WidgetTester tester) async {
1074
- await tester.pumpWidget(MyApp());
1075
- await tester.enterText(find.byKey(Key('email')), 'user@example.com');
1076
- await tester.enterText(find.byKey(Key('password')), 'secret');
1077
- await tester.tap(find.byKey(Key('login-btn')));
1078
- await tester.pumpAndSettle();
1079
- expect(find.text('Welcome'), findsOneWidget);
1080
- });
1081
-
1082
- // @tc:12346
1083
- testWidgets('shows error for invalid credentials', (WidgetTester tester) async {
1084
- await tester.pumpWidget(MyApp());
1085
- await tester.enterText(find.byKey(Key('email')), 'bad@example.com');
1086
- await tester.enterText(find.byKey(Key('password')), 'wrong');
1087
- await tester.tap(find.byKey(Key('login-btn')));
1088
- await tester.pumpAndSettle();
1089
- expect(find.text('Invalid credentials'), findsOneWidget);
1090
- });
1091
- });
1092
- }
1093
- ```
1094
-
1095
- Recommended `ado-sync.json`:
1096
-
1097
- ```json
1098
- {
1099
- "local": {
1100
- "type": "flutter",
1101
- "include": ["test/**/*_test.dart", "integration_test/**/*_test.dart"]
1102
- },
1103
- "sync": { "markAutomated": true }
1104
- }
1105
- ```
1106
-
1107
- **Notes:**
1108
- - `group('title', () { ... })` is the describe equivalent. Nested groups are supported.
1109
- - `testWidgets()`, `test()`, and `testUI()` (integration_test alias) are all detected.
1110
- - The file base name is stripped of `_test.dart` / `.dart` suffixes for `automatedTestName`.
1111
- - Both `///` triple-slash and `/** ... */` block doc comments are parsed.
1112
- - ID writeback format: `// @tc:12345` immediately above the `testWidgets()`/`test()` line.
1113
- - `pull` is not supported — only `push`.
1114
-
1115
- ---
1116
-
1117
- ## Robot Framework `.robot`
1118
-
1119
- Set `local.type: "robot"`. Supports **Robot Framework** test suites. Each test case in the `*** Test Cases ***` or `*** Tasks ***` section becomes one Azure Test Case.
1120
-
1121
- ### Source mapping
1122
-
1123
- | Robot Framework source | Azure TC field |
1124
- |------------------------|---------------|
1125
- | Test case name (non-indented line) | TC **Title** |
1126
- | `[Documentation]` row | TC **Description** |
1127
- | Numbered lines `N. text` in `[Documentation]` | TC **Steps** (action) |
1128
- | Numbered lines `N. Check: text` in `[Documentation]` | TC **Steps** (expected result, when `useExpectedResult: true`) |
1129
- | `[Tags]` values (excluding `tc:N`) | TC **Tags** |
1130
- | `[Tags] tc:N` | TC ID (**preferred**; written back after first push) |
1131
- | `# @tc:N` comment immediately above the test name | TC ID (comment fallback) |
1132
- | Test case name | `AutomatedTestName` (for result linking) |
1133
-
1134
- ### Example
1135
-
1136
- ```robot
1137
- *** Settings ***
1138
- Library SeleniumLibrary
1139
-
1140
- *** Test Cases ***
1141
- # @tc:1041
1142
- Add Item And Complete Checkout
1143
- [Documentation] Add item and complete checkout
1144
- ... 1. Sign in as standard_user
1145
- ... 2. Add Sauce Labs Backpack to cart
1146
- ... 3. Proceed through checkout
1147
- ... 4. Check: Order confirmation page is shown
1148
- [Tags] tc:1041 smoke
1149
- Open Browser ${URL} chrome
1150
- Input Text id:user-name standard_user
1151
- Input Password id:password secret_sauce
1152
- Click Button id:login-button
1153
-
1154
- User Login Fails With Invalid Credentials
1155
- [Documentation] Login fails with invalid credentials
1156
- ... 1. Navigate to the login page
1157
- ... 2. Enter an invalid username and password
1158
- ... 3. Click Login
1159
- ... 4. Check: Error message is displayed
1160
- [Tags] regression
1161
- Open Browser ${URL} chrome
1162
- Input Text id:user-name bad_user
1163
- Input Password id:password wrong
1164
- Click Button id:login-button
1165
- ```
1166
-
1167
- On the first `push`, ado-sync inserts `tc:N` into the `[Tags]` row. If no `[Tags]` row exists, one is created immediately after the test name line.
1168
-
1169
- ### ID writeback
1170
-
1171
- | Location | Format |
1172
- |----------|--------|
1173
- | `[Tags]` row (preferred) | `tc:12345` value added/updated in the existing `[Tags]` row |
1174
- | No `[Tags]` row | ` [Tags] tc:12345` inserted after the test name line |
1175
-
1176
- The `# @tc:N` comment form above the test name is also recognised on read but is not used for writeback — `[Tags]` is always the authoritative location after a push.
1177
-
1178
- ### Path-based auto-tagging
1179
-
1180
- Directory segments starting with `@` are automatically applied as tags on all test cases inside:
1181
-
1182
- ```
1183
- tests/
1184
- @smoke/
1185
- login.robot ← all test cases get tag 'smoke'
1186
- @regression/
1187
- checkout.robot ← all test cases get tag 'regression'
1188
- ```
1189
-
1190
- ### Recommended config
1191
-
1192
- ```json
1193
- {
1194
- "local": {
1195
- "type": "robot",
1196
- "include": ["tests/**/*.robot"],
1197
- "exclude": ["tests/resources/**", "tests/keywords/**"]
1198
- },
1199
- "sync": {
1200
- "markAutomated": true
1201
- }
1202
- }
1203
- ```
1204
-
1205
- ### Notes
1206
-
1207
- - **`pull` is not supported** for Robot Framework files. Only `push` and `publish-test-results` apply.
1208
- - **`*** Tasks ***` sections** are treated the same as `*** Test Cases ***`.
1209
- - **Keyword lines** — all indented non-settings rows are treated as TC steps if no numbered steps are found in `[Documentation]`.
1210
- - **`[Setup]` / `[Teardown]`** — these settings rows are skipped; only keyword steps and `[Documentation]` / `[Tags]` are used.
1211
-
1212
- ---
1213
-
1214
- ## CSV `.csv`
1215
-
1216
- Set `local.type: "csv"` to parse Azure DevOps / SpecSync tabular CSV exports.
1217
-
1218
- **Expected column layout (9 columns):**
1219
-
1220
- | Col | Field | Description |
1221
- |-----|-------|-------------|
1222
- | A (0) | ID | Azure Test Case ID — empty for new, filled after first push |
1223
- | B (1) | Work Item Type | Always `Test Case` (ignored) |
1224
- | C (2) | Title | `Scenario: My test` — non-empty on header row, empty on step rows |
1225
- | D (3) | Test Step | Step number — empty on header row |
1226
- | E (4) | Step Action | Step text, optionally prefixed with a Gherkin keyword |
1227
- | F (5) | Step Expected | Expected result text (optional) |
1228
- | G–I (6–8) | Area Path, Assigned To, State | Preserved but not used |
1229
-
1230
- The first row (column headers) is always skipped automatically.
1231
-
1232
- > **Note:** `ado-sync pull` is **not supported** for CSV files. Only push is supported.
1233
-
1234
- ---
1235
-
1236
- ## Excel `.xlsx`
1237
-
1238
- Set `local.type: "excel"`. Uses the same 9-column layout as CSV. ado-sync reads the raw worksheet XML (no external xlsx library) and handles both `t="str"` and `t="inlineStr"` cells from Azure DevOps exports.
1239
-
1240
- > **Note:** `ado-sync pull` is **not supported** for Excel files. Only push is supported.
1241
-
1242
- ---
1243
-
1244
- ## ID tags
1245
-
1246
- After a first push, ado-sync writes the Azure TC ID back into the local file.
1247
-
1248
- | Format | Location |
1249
- |--------|----------|
1250
- | Gherkin | `@tc:12345` tag on its own line above the `Scenario:` line |
1251
- | Markdown | `@tc:12345` tag on the line immediately after the `### heading` |
1252
- | C# MSTest | `[TestProperty("tc", "12345")]` attribute on the test method |
1253
- | C# NUnit | `[Property("tc", "12345")]` attribute on the test method |
1254
- | Java JUnit 5 | `@Tag("tc:12345")` annotation above `@Test` |
1255
- | Java JUnit 4 / TestNG | `// @tc:12345` comment on the line above `@Test` |
1256
- | Python pytest | `@pytest.mark.tc(12345)` decorator above `def test_*` |
1257
- | JavaScript/TS | `// @tc:12345` comment on the line above `it()`/`test()` |
1258
- | Playwright | `// @tc:12345` comment on the line above `test()` |
1259
- | Robot Framework | `tc:12345` value in the `[Tags]` row of the test case body |
1260
- | CSV | Numeric ID in column A of the matching title row |
1261
- | Excel | Numeric ID in cell A of the matching title row |
1262
-
1263
- ### Custom prefix
1264
-
1265
- ```json
1266
- { "sync": { "tagPrefix": "azure" } }
1267
- ```
1268
-
1269
- Result: `@azure:12345` in both Gherkin and Markdown files.
1270
-
1271
- > **Warning:** Changing the prefix on an existing project means existing ID tags are no longer recognised. Do a project-wide find-and-replace before changing.
1272
-
1273
- ---
1274
-
1275
- ## Tag filtering
1276
-
1277
- All commands accept `--tags` to limit which scenarios are processed. Uses the standard Cucumber tag expression language.
1278
-
1279
- ```bash
1280
- ado-sync push --tags "@smoke"
1281
- ado-sync push --tags "@smoke and not @wip"
1282
- ado-sync pull --tags "@regression or @critical"
1283
- ```
1284
-
1285
- Tags are evaluated against all tags on a scenario, including:
1286
-
1287
- - Tags on the `Feature` block (Gherkin)
1288
- - Tags on the `Scenario` / `Scenario Outline` block
1289
- - Tags on individual `Examples` tables
1290
- - Tags on the Markdown tag line (same `### heading` line or line below it)
1291
- - **Path-based auto-tags** — directory segments starting with `@` are automatically applied as tags:
1292
-
1293
- ```
1294
- specs/
1295
- @smoke/
1296
- login.feature ← all scenarios get tag 'smoke'
1297
- @regression/
1298
- @slow/
1299
- checkout.feature ← all scenarios get tags 'regression' and 'slow'
1300
- ```
1301
-
1302
- ### `local.condition` — permanent filter
1303
-
1304
- Use `local.condition` in config to permanently exclude scenarios without needing `--tags` on every command:
1305
-
1306
- ```json
1307
- { "local": { "condition": "@done and not (@ignored or @planned)" } }
1308
- ```
1309
-
1310
- This filter is applied before `--tags`, so it acts as a baseline inclusion gate.
1311
-
1312
- ---
1313
-
1314
- ## Work item linking
1315
-
1316
- Tags matching a configured `links` prefix are synced as Azure DevOps work item relations on the Test Case.
1317
-
1318
- ### Config
1319
-
1320
- ```json
1321
- {
1322
- "sync": {
1323
- "links": [
1324
- { "prefix": "story", "relationship": "System.LinkTypes.Related" },
1325
- { "prefix": "bug", "relationship": "System.LinkTypes.Related" },
1326
- { "prefix": "req", "relationship": "Microsoft.VSTS.Common.TestedBy-Reverse" }
1327
- ]
1328
- }
1329
- }
1330
- ```
1331
-
1332
- ### Usage in Gherkin
1333
-
1334
- ```gherkin
1335
- @tc:1042 @story:555 @bug:789
1336
- Scenario: User can add items to cart
1337
- ...
1338
- ```
1339
-
1340
- ### Usage in Markdown
1341
-
1342
- ```markdown
1343
- ### User can add items to cart
1344
- @tc:1042 @story:555 @bug:789
1345
-
1346
- Steps:
1347
- 1. Add an item to the cart
1348
- ...
1349
- ```
1350
-
1351
- On each `push`, relations are synced: new links are added, stale links (whose tag was removed) are deleted. The `relationship` value is the ADO relation type — `"System.LinkTypes.Related"` is the default if omitted.
1352
-
1353
- ---
1354
-
1355
- ## Attachments
1356
-
1357
- Files can be attached to Azure Test Cases via tags. See [Attachments](advanced.md#attachments).