ado-sync 0.1.64 → 0.1.67
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +20 -15
- package/dist/__tests__/regressions.test.js +1011 -1
- package/dist/__tests__/regressions.test.js.map +1 -1
- package/dist/ai/generate-spec.d.ts +1 -1
- package/dist/ai/generate-spec.js +23 -0
- package/dist/ai/generate-spec.js.map +1 -1
- package/dist/ai/summarizer.d.ts +3 -2
- package/dist/ai/summarizer.js +50 -1
- package/dist/ai/summarizer.js.map +1 -1
- package/dist/azure/test-cases.d.ts +11 -1
- package/dist/azure/test-cases.js +286 -43
- package/dist/azure/test-cases.js.map +1 -1
- package/dist/cli.js +91 -14
- package/dist/cli.js.map +1 -1
- package/dist/config.js +74 -1
- package/dist/config.js.map +1 -1
- package/dist/id-markers.d.ts +1 -0
- package/dist/id-markers.js +13 -0
- package/dist/id-markers.js.map +1 -1
- package/dist/mcp-server.js +1 -1
- package/dist/mcp-server.js.map +1 -1
- package/dist/sync/cache.d.ts +2 -0
- package/dist/sync/cache.js.map +1 -1
- package/dist/sync/engine.d.ts +12 -1
- package/dist/sync/engine.js +210 -41
- package/dist/sync/engine.js.map +1 -1
- package/dist/types.d.ts +56 -4
- package/llms.txt +12 -11
- package/package.json +8 -1
- package/docs/advanced.md +0 -988
- package/docs/agent-setup.md +0 -204
- package/docs/capability-roadmap.md +0 -280
- package/docs/cli.md +0 -609
- package/docs/configuration.md +0 -322
- package/docs/examples/csharp-mstest-local-llm.yaml +0 -35
- package/docs/examples/csharp-mstest.yaml +0 -21
- package/docs/examples/csharp-nunit.yaml +0 -21
- package/docs/examples/csharp-specflow.yaml +0 -16
- package/docs/examples/cypress.yaml +0 -21
- package/docs/examples/detox-react-native.yaml +0 -21
- package/docs/examples/espresso-android.yaml +0 -21
- package/docs/examples/flutter-dart.yaml +0 -21
- package/docs/examples/java-junit.yaml +0 -21
- package/docs/examples/java-testng.yaml +0 -21
- package/docs/examples/js-jasmine-wdio.yaml +0 -21
- package/docs/examples/js-jest.yaml +0 -21
- package/docs/examples/playwright-js.yaml +0 -21
- package/docs/examples/playwright-ts.yaml +0 -21
- package/docs/examples/puppeteer.yaml +0 -21
- package/docs/examples/python-pytest.yaml +0 -21
- package/docs/examples/robot-framework.yaml +0 -19
- package/docs/examples/testcafe.yaml +0 -21
- package/docs/examples/xcuitest-ios.yaml +0 -21
- package/docs/mcp-server.md +0 -312
- package/docs/publish-test-results.md +0 -939
- package/docs/spec-formats.md +0 -1357
- package/docs/troubleshooting.md +0 -101
- package/docs/vscode-extension.md +0 -139
- package/docs/work-item-links.md +0 -115
- package/docs/workflows.md +0 -457
- package/mkdocs.yml +0 -40
- package/requirements-docs.txt +0 -4
- package/scripts/build_site.sh +0 -6
package/docs/spec-formats.md
DELETED
|
@@ -1,1357 +0,0 @@
|
|
|
1
|
-
# Spec file formats
|
|
2
|
-
|
|
3
|
-
---
|
|
4
|
-
|
|
5
|
-
## Feature support matrix
|
|
6
|
-
|
|
7
|
-
| `local.type` | Push | Pull | AI summary | Outline expansion | Attachments | Path auto-tags |
|
|
8
|
-
|---|:---:|:---:|:---:|:---:|:---:|:---:|
|
|
9
|
-
| `gherkin` | ✓ | ✓ | — | ✓ | ✓ | ✓ |
|
|
10
|
-
| `reqnroll` | ✓ | ✓ | — | ✓ | ✓ | ✓ |
|
|
11
|
-
| `markdown` | ✓ | ✓ | — | — | ✓ | ✓ |
|
|
12
|
-
| `csv` | ✓ | — | — | — | — | — |
|
|
13
|
-
| `excel` | ✓ | — | — | — | — | — |
|
|
14
|
-
| `csharp` | ✓ | — | ✓ | — | — | ✓ |
|
|
15
|
-
| `java` | ✓ | — | ✓ | — | — | ✓ |
|
|
16
|
-
| `python` | ✓ | — | ✓ | — | — | ✓ |
|
|
17
|
-
| `javascript` | ✓ | — | ✓ | — | — | ✓ |
|
|
18
|
-
| `playwright` | ✓ | — | ✓ | — | — | ✓ |
|
|
19
|
-
| `puppeteer` | ✓ | — | ✓ | — | — | ✓ |
|
|
20
|
-
| `cypress` | ✓ | — | ✓ | — | — | ✓ |
|
|
21
|
-
| `testcafe` | ✓ | — | ✓ | — | — | ✓ |
|
|
22
|
-
| `detox` | ✓ | — | ✓ | — | — | ✓ |
|
|
23
|
-
| `espresso` | ✓ | — | ✓ | — | — | ✓ |
|
|
24
|
-
| `xcuitest` | ✓ | — | ✓ | — | — | ✓ |
|
|
25
|
-
| `flutter` | ✓ | — | ✓ | — | — | ✓ |
|
|
26
|
-
| `robot` | ✓ | — | — | — | — | ✓ |
|
|
27
|
-
|
|
28
|
-
**Column notes:**
|
|
29
|
-
- **Pull** — `ado-sync pull` can create/update local files from Azure Test Cases.
|
|
30
|
-
- **AI summary** — `sync.ai` auto-generates title, description and steps from the test function body. Requires a non-`heuristic` provider for best results; `heuristic` regex matching works for all code-based types.
|
|
31
|
-
- **Outline expansion** — Scenario Outline / `Examples` tables create a single parametrized TC (not one TC per row).
|
|
32
|
-
- **Attachments** — `sync.attachments` file-attachment sync via `@attachment:` tags.
|
|
33
|
-
- **Path auto-tags** — directories starting with `@` (e.g. `specs/@smoke/`) are automatically applied as tags.
|
|
34
|
-
|
|
35
|
-
---
|
|
36
|
-
|
|
37
|
-
## Gherkin `.feature`
|
|
38
|
-
|
|
39
|
-
Set `local.type: "gherkin"`. Standard Gherkin syntax is supported:
|
|
40
|
-
|
|
41
|
-
- `Feature` / `Scenario` / `Scenario Outline` / `Background`
|
|
42
|
-
- `Given` / `When` / `Then` / `And` / `But`
|
|
43
|
-
- Feature-level and scenario-level tags
|
|
44
|
-
- `Scenario Outline` with `Examples` tables — creates a **single** parametrized Test Case in Azure, not one TC per row
|
|
45
|
-
- Inline data tables (synced as sub-steps or plain text — see [Format configuration](advanced.md#format-configuration))
|
|
46
|
-
|
|
47
|
-
```gherkin
|
|
48
|
-
Feature: Checkout
|
|
49
|
-
|
|
50
|
-
Background:
|
|
51
|
-
Given I am on the storefront
|
|
52
|
-
|
|
53
|
-
@smoke
|
|
54
|
-
@tc:1041
|
|
55
|
-
Scenario: Add item and complete checkout
|
|
56
|
-
Given I am logged in as "standard_user"
|
|
57
|
-
When I add "Sauce Labs Backpack" to the cart
|
|
58
|
-
And I proceed through checkout with name "Test User" and zip "12345"
|
|
59
|
-
Then I see the order confirmation page
|
|
60
|
-
|
|
61
|
-
@tc:1042
|
|
62
|
-
Scenario Outline: Checkout with different users
|
|
63
|
-
Given I am logged in as "<user>"
|
|
64
|
-
When I complete a checkout
|
|
65
|
-
Then the result is "<result>"
|
|
66
|
-
|
|
67
|
-
Examples:
|
|
68
|
-
| user | result |
|
|
69
|
-
| standard_user | success |
|
|
70
|
-
| performance_glitch_user | success |
|
|
71
|
-
```
|
|
72
|
-
|
|
73
|
-
---
|
|
74
|
-
|
|
75
|
-
## Markdown `.md`
|
|
76
|
-
|
|
77
|
-
Set `local.type: "markdown"`. Each `### heading` is one test case. Scenarios are separated by `---`.
|
|
78
|
-
|
|
79
|
-
```markdown
|
|
80
|
-
# My Feature Test Plan
|
|
81
|
-
|
|
82
|
-
## Test scenarios
|
|
83
|
-
|
|
84
|
-
### Login with valid credentials
|
|
85
|
-
@tc:1042 @smoke
|
|
86
|
-
|
|
87
|
-
Assumption: Fresh browser session.
|
|
88
|
-
|
|
89
|
-
Steps:
|
|
90
|
-
1. Navigate to https://example.com/login
|
|
91
|
-
2. Enter username "admin" and password "secret"
|
|
92
|
-
3. Click the Login button
|
|
93
|
-
|
|
94
|
-
Expected results:
|
|
95
|
-
- The dashboard page is shown
|
|
96
|
-
- The username appears in the top navigation
|
|
97
|
-
|
|
98
|
-
---
|
|
99
|
-
|
|
100
|
-
### Login with invalid credentials
|
|
101
|
-
|
|
102
|
-
Steps:
|
|
103
|
-
1. Navigate to https://example.com/login
|
|
104
|
-
2. Enter username "wrong" and password "wrong"
|
|
105
|
-
3. Click the Login button
|
|
106
|
-
|
|
107
|
-
Expected results:
|
|
108
|
-
- An error message "Invalid credentials" is displayed
|
|
109
|
-
- The user remains on the login page
|
|
110
|
-
|
|
111
|
-
---
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
Sections recognised (case-insensitive): `Steps:`, `Expected results:`. All other prose is captured as the test case description. Heading number prefixes (`### 1. Title` or `### Title`) are both supported.
|
|
115
|
-
|
|
116
|
-
### Playwright test-plan markdown
|
|
117
|
-
|
|
118
|
-
Markdown files generated by the Playwright MCP agent are fully supported. Set `local.type: "markdown"` and point `local.include` at the generated files.
|
|
119
|
-
|
|
120
|
-
---
|
|
121
|
-
|
|
122
|
-
## C# MSTest / NUnit `.cs`
|
|
123
|
-
|
|
124
|
-
Set `local.type: "csharp"`. Both **MSTest** and **NUnit** frameworks are supported. Each test method becomes one Test Case.
|
|
125
|
-
|
|
126
|
-
### Framework attribute mapping
|
|
127
|
-
|
|
128
|
-
| Concern | MSTest | NUnit |
|
|
129
|
-
|---------|--------|-------|
|
|
130
|
-
| Test marker | `[TestMethod]` | `[Test]` |
|
|
131
|
-
| Category / tag | `[TestCategory("name")]` | `[Category("name")]` |
|
|
132
|
-
| Custom property | `[TestProperty("key","val")]` | `[Property("key","val")]` |
|
|
133
|
-
| TC ID writeback | `[TestProperty("tc","ID")]` | `[Property("tc","ID")]` |
|
|
134
|
-
|
|
135
|
-
The framework is auto-detected per method — mixed files (e.g. a shared helper class) are handled correctly.
|
|
136
|
-
|
|
137
|
-
### Source mapping (both frameworks)
|
|
138
|
-
|
|
139
|
-
| C# source | Azure TC field |
|
|
140
|
-
|-----------|---------------|
|
|
141
|
-
| XML doc `<summary>` first line | TC **Title** |
|
|
142
|
-
| Numbered lines `N. text` in summary | TC **Steps** (action) |
|
|
143
|
-
| Numbered lines `N. Check: text` | TC **Steps** (expected result, when `useExpectedResult: true`) |
|
|
144
|
-
| `[TestCategory("…")]` / `[Category("…")]` | TC **Tags** (string literals and `const string` constants resolved) |
|
|
145
|
-
| `[TestProperty("tc","ID")]` / `[Property("tc","ID")]` | TC ID (written back after first push) |
|
|
146
|
-
| `Namespace.Class.MethodName` | `AutomatedTestName` (for TRX result linking) |
|
|
147
|
-
|
|
148
|
-
### MSTest example
|
|
149
|
-
|
|
150
|
-
```csharp
|
|
151
|
-
/// <summary>
|
|
152
|
-
/// Verify CoCounsel dialog opens on Edge
|
|
153
|
-
/// Test case: 1883058
|
|
154
|
-
/// 1. Sign in to Westlaw Edge
|
|
155
|
-
/// 2. Click CoCounsel link from page header
|
|
156
|
-
/// 3. Check: Verify CoCounsel dialog opens with correct title
|
|
157
|
-
/// 4. Check: Verify Close button is displayed
|
|
158
|
-
/// </summary>
|
|
159
|
-
[TestMethod]
|
|
160
|
-
[TestProperty("tc", "1883058")] // written back by ado-sync after first push
|
|
161
|
-
[TestCategory("EdgeAalp")]
|
|
162
|
-
[TestCategory("EdgeAalpSmoke")]
|
|
163
|
-
public void EdgeAalpAccessTest()
|
|
164
|
-
{
|
|
165
|
-
// ...
|
|
166
|
-
}
|
|
167
|
-
```
|
|
168
|
-
|
|
169
|
-
### NUnit example
|
|
170
|
-
|
|
171
|
-
```csharp
|
|
172
|
-
/// <summary>
|
|
173
|
-
/// Verify login with valid credentials
|
|
174
|
-
/// 1. Navigate to the login page
|
|
175
|
-
/// 2. Enter username and password
|
|
176
|
-
/// 3. Click the Login button
|
|
177
|
-
/// 4. Check: Dashboard page is shown
|
|
178
|
-
/// 5. Check: Username appears in the top navigation
|
|
179
|
-
/// </summary>
|
|
180
|
-
[Test]
|
|
181
|
-
[Property("tc", "2001")] // written back by ado-sync after first push
|
|
182
|
-
[Category("Smoke")]
|
|
183
|
-
[Category("Authentication")]
|
|
184
|
-
public void LoginWithValidCredentials()
|
|
185
|
-
{
|
|
186
|
-
// ...
|
|
187
|
-
}
|
|
188
|
-
```
|
|
189
|
-
|
|
190
|
-
### Recommended config
|
|
191
|
-
|
|
192
|
-
```json
|
|
193
|
-
{
|
|
194
|
-
"local": {
|
|
195
|
-
"type": "csharp",
|
|
196
|
-
"include": ["**/RegressionTests/**/*.cs"],
|
|
197
|
-
"exclude": ["**/*BaseTest.cs", "**/*BaseFixture.cs", "**/*Helper.cs"]
|
|
198
|
-
},
|
|
199
|
-
"sync": {
|
|
200
|
-
"markAutomated": true,
|
|
201
|
-
"format": { "useExpectedResult": true }
|
|
202
|
-
}
|
|
203
|
-
}
|
|
204
|
-
```
|
|
205
|
-
|
|
206
|
-
With `useExpectedResult: true`, `Check:` lines go into the Expected Result column of each TC step. With `markAutomated: true`, the TC's Associated Automation is set to the fully-qualified method name (`Namespace.Class.Method`), enabling `publish-test-results` to link TRX run outcomes back to TCs.
|
|
207
|
-
|
|
208
|
-
### Notes
|
|
209
|
-
|
|
210
|
-
- **Constants** — ado-sync resolves `const string` declarations within the same file. Constants defined in a base class are not resolved; use string literals for reliable tagging.
|
|
211
|
-
- **`pull` is not supported** for C# files. Only `push` and `publish-test-results` apply.
|
|
212
|
-
- **Base classes / fixtures** — exclude them from `local.include` to avoid treating non-test methods as test cases.
|
|
213
|
-
- **NUnit `[TestCase]`** — parameterised data rows (`[TestCase(1, "foo")]`) are not expanded into separate TCs. Only the `[Test]` marker is treated as the test definition.
|
|
214
|
-
|
|
215
|
-
---
|
|
216
|
-
|
|
217
|
-
## Java JUnit / TestNG `.java`
|
|
218
|
-
|
|
219
|
-
Set `local.type: "java"`. Supports **JUnit 4**, **JUnit 5**, and **TestNG** (including Selenium-based tests). Each `@Test`-annotated method becomes one Test Case.
|
|
220
|
-
|
|
221
|
-
### Framework attribute mapping
|
|
222
|
-
|
|
223
|
-
| Concern | JUnit 4 | JUnit 5 | TestNG |
|
|
224
|
-
|---------|---------|---------|--------|
|
|
225
|
-
| Test marker | `@Test` (`org.junit.Test`) | `@Test` (`org.junit.jupiter.api.Test`) | `@Test` (`org.testng.annotations.Test`) |
|
|
226
|
-
| Tag / category | `@Category(Smoke.class)` | `@Tag("smoke")` | `groups = {"smoke"}` in `@Test` |
|
|
227
|
-
| TC ID writeback | `// @tc:ID` above `@Test` | `@Tag("tc:ID")` above `@Test` | `// @tc:ID` above `@Test` |
|
|
228
|
-
|
|
229
|
-
The framework is auto-detected per file from import statements — no config required.
|
|
230
|
-
|
|
231
|
-
### Source mapping (all frameworks)
|
|
232
|
-
|
|
233
|
-
| Java source | Azure TC field |
|
|
234
|
-
|-------------|---------------|
|
|
235
|
-
| Javadoc `/** ... */` first non-numbered line | TC **Title** |
|
|
236
|
-
| Numbered lines `N. text` in Javadoc | TC **Steps** (action) |
|
|
237
|
-
| Numbered lines `N. Check: text` in Javadoc | TC **Steps** (expected result, when `useExpectedResult: true`) |
|
|
238
|
-
| `@Category`, `@Tag`, `groups` in `@Test` | TC **Tags** |
|
|
239
|
-
| `// @tc:ID` or `@Tag("tc:ID")` | TC ID (written back after first push) |
|
|
240
|
-
| `com.example.MyClass.myMethod` | `AutomatedTestName` (for JUnit XML result linking) |
|
|
241
|
-
|
|
242
|
-
### JUnit 5 example
|
|
243
|
-
|
|
244
|
-
```java
|
|
245
|
-
import org.junit.jupiter.api.Tag;
|
|
246
|
-
import org.junit.jupiter.api.Test;
|
|
247
|
-
|
|
248
|
-
class CheckoutTests {
|
|
249
|
-
|
|
250
|
-
/**
|
|
251
|
-
* Add item and complete checkout
|
|
252
|
-
* 1. Sign in as standard_user
|
|
253
|
-
* 2. Add "Sauce Labs Backpack" to cart
|
|
254
|
-
* 3. Proceed through checkout
|
|
255
|
-
* 4. Check: Order confirmation page is shown
|
|
256
|
-
*/
|
|
257
|
-
@Tag("tc:1041") // written back by ado-sync after first push
|
|
258
|
-
@Tag("smoke")
|
|
259
|
-
@Test
|
|
260
|
-
void addItemAndCompleteCheckout() {
|
|
261
|
-
// ...
|
|
262
|
-
}
|
|
263
|
-
}
|
|
264
|
-
```
|
|
265
|
-
|
|
266
|
-
### JUnit 4 example
|
|
267
|
-
|
|
268
|
-
```java
|
|
269
|
-
import org.junit.Test;
|
|
270
|
-
import org.junit.experimental.categories.Category;
|
|
271
|
-
|
|
272
|
-
public class LoginTests {
|
|
273
|
-
|
|
274
|
-
/**
|
|
275
|
-
* Login with valid credentials
|
|
276
|
-
* 1. Navigate to the login page
|
|
277
|
-
* 2. Enter username and password
|
|
278
|
-
* 3. Check: Dashboard page is shown
|
|
279
|
-
*/
|
|
280
|
-
// @tc:1042 // written back by ado-sync after first push
|
|
281
|
-
@Test
|
|
282
|
-
@Category(Smoke.class)
|
|
283
|
-
public void loginWithValidCredentials() {
|
|
284
|
-
// ...
|
|
285
|
-
}
|
|
286
|
-
}
|
|
287
|
-
```
|
|
288
|
-
|
|
289
|
-
### TestNG example
|
|
290
|
-
|
|
291
|
-
```java
|
|
292
|
-
import org.testng.annotations.Test;
|
|
293
|
-
|
|
294
|
-
public class SearchTests {
|
|
295
|
-
|
|
296
|
-
/**
|
|
297
|
-
* Search returns relevant results
|
|
298
|
-
* 1. Open the search page
|
|
299
|
-
* 2. Enter "junit 5" in the search box
|
|
300
|
-
* 3. Check: Results list contains at least one entry
|
|
301
|
-
*/
|
|
302
|
-
// @tc:1043 // written back by ado-sync after first push
|
|
303
|
-
@Test(groups = {"regression", "search"})
|
|
304
|
-
public void searchReturnsRelevantResults() {
|
|
305
|
-
// ...
|
|
306
|
-
}
|
|
307
|
-
}
|
|
308
|
-
```
|
|
309
|
-
|
|
310
|
-
### Recommended config
|
|
311
|
-
|
|
312
|
-
```json
|
|
313
|
-
{
|
|
314
|
-
"local": {
|
|
315
|
-
"type": "java",
|
|
316
|
-
"include": ["**/src/test/**/*.java"],
|
|
317
|
-
"exclude": ["**/*BaseTest.java", "**/*Helper.java", "**/*Utils.java"]
|
|
318
|
-
},
|
|
319
|
-
"sync": {
|
|
320
|
-
"markAutomated": true
|
|
321
|
-
}
|
|
322
|
-
}
|
|
323
|
-
```
|
|
324
|
-
|
|
325
|
-
### Notes
|
|
326
|
-
|
|
327
|
-
- **`pull` is not supported** for Java files. Only `push` and `publish-test-results` apply.
|
|
328
|
-
- **Abstract base test classes** — exclude them from `local.include` to avoid treating base methods as test cases.
|
|
329
|
-
- **JUnit 5 `@Tag`** — the `org.junit.jupiter.api.Tag` import must be present or the file must already use Jupiter annotations; ado-sync adds the tag annotation but not the import if it's missing.
|
|
330
|
-
- **TestNG `groups`** — `groups = {"smoke", "regression"}` inside `@Test(...)` are extracted as TC tags.
|
|
331
|
-
- **`@ParameterizedTest` / `@DataProvider`** — parameterised data rows are not expanded into separate TCs; only the method definition is treated as the test case.
|
|
332
|
-
|
|
333
|
-
---
|
|
334
|
-
|
|
335
|
-
## Python pytest `.py`
|
|
336
|
-
|
|
337
|
-
Set `local.type: "python"`. Each `def test_*` function — at module level or inside a class — becomes one Test Case. No extra dependencies required beyond `pytest` itself.
|
|
338
|
-
|
|
339
|
-
### Source mapping
|
|
340
|
-
|
|
341
|
-
| Python source | Azure TC field |
|
|
342
|
-
|---------------|---------------|
|
|
343
|
-
| Docstring first non-numbered line | TC **Title** |
|
|
344
|
-
| Numbered lines `N. text` in docstring | TC **Steps** (action) |
|
|
345
|
-
| Numbered lines `N. Check: text` in docstring | TC **Steps** (expected result, when `useExpectedResult: true`) |
|
|
346
|
-
| `@pytest.mark.<tag>` decorators | TC **Tags** (built-in pytest marks excluded) |
|
|
347
|
-
| `@pytest.mark.tc(ID)` | TC ID (written back after first push) |
|
|
348
|
-
| `module.path.ClassName.test_method` | `AutomatedTestName` (for JUnit XML result linking) |
|
|
349
|
-
|
|
350
|
-
### Example
|
|
351
|
-
|
|
352
|
-
```python
|
|
353
|
-
import pytest
|
|
354
|
-
|
|
355
|
-
class TestCheckout:
|
|
356
|
-
|
|
357
|
-
@pytest.mark.tc(1041) # written back by ado-sync after first push
|
|
358
|
-
@pytest.mark.smoke
|
|
359
|
-
def test_add_item_and_complete_checkout(self):
|
|
360
|
-
"""
|
|
361
|
-
Add item and complete checkout
|
|
362
|
-
1. Sign in as standard_user
|
|
363
|
-
2. Add Sauce Labs Backpack to cart
|
|
364
|
-
3. Proceed through checkout
|
|
365
|
-
4. Check: Order confirmation page is shown
|
|
366
|
-
"""
|
|
367
|
-
# ...
|
|
368
|
-
|
|
369
|
-
@pytest.mark.tc(1042)
|
|
370
|
-
@pytest.mark.regression
|
|
371
|
-
def test_checkout_with_invalid_card(self):
|
|
372
|
-
"""
|
|
373
|
-
Checkout fails with invalid card
|
|
374
|
-
1. Add an item to the cart
|
|
375
|
-
2. Enter an invalid credit card number
|
|
376
|
-
3. Click Place Order
|
|
377
|
-
4. Check: Error message is displayed
|
|
378
|
-
"""
|
|
379
|
-
# ...
|
|
380
|
-
```
|
|
381
|
-
|
|
382
|
-
### Recommended config
|
|
383
|
-
|
|
384
|
-
```json
|
|
385
|
-
{
|
|
386
|
-
"local": {
|
|
387
|
-
"type": "python",
|
|
388
|
-
"include": ["tests/**/*.py"],
|
|
389
|
-
"exclude": ["tests/conftest.py", "tests/**/fixtures.py"]
|
|
390
|
-
},
|
|
391
|
-
"sync": {
|
|
392
|
-
"markAutomated": true
|
|
393
|
-
}
|
|
394
|
-
}
|
|
395
|
-
```
|
|
396
|
-
|
|
397
|
-
### Notes
|
|
398
|
-
|
|
399
|
-
- **`pull` is not supported** for Python files. Only `push` and `publish-test-results` apply.
|
|
400
|
-
- **Built-in pytest marks** (`parametrize`, `skip`, `skipif`, `xfail`, `usefixtures`, `filterwarnings`) are not pushed as TC tags.
|
|
401
|
-
- **`@pytest.mark.parametrize`** — parameterised data rows are not expanded into separate TCs; only the function definition is treated as the test case.
|
|
402
|
-
- **Class hierarchy** — only the immediately enclosing class is used for the `automatedTestName`. Nested classes are supported.
|
|
403
|
-
|
|
404
|
-
---
|
|
405
|
-
|
|
406
|
-
## JavaScript / TypeScript (Jest, Jasmine, WebdriverIO) `.js` / `.ts`
|
|
407
|
-
|
|
408
|
-
Set `local.type: "javascript"`. Supports **Jest**, **Jasmine**, and **WebdriverIO** (which uses Jest or Jasmine as its runner). All three share the same `describe()`/`it()`/`test()` API, so a single parser handles all of them.
|
|
409
|
-
|
|
410
|
-
> **Playwright Test** (`@playwright/test`) has its own type: use `local.type: "playwright"` instead — see below.
|
|
411
|
-
> **Cucumber `.feature` files** are handled by `local.type: "gherkin"` — not this type.
|
|
412
|
-
|
|
413
|
-
### Detected test functions
|
|
414
|
-
|
|
415
|
-
| Function | Description |
|
|
416
|
-
|----------|-------------|
|
|
417
|
-
| `it(title, fn)` | Standard test |
|
|
418
|
-
| `test(title, fn)` | Alias for `it` |
|
|
419
|
-
| `it.only` / `test.only` | Focused test |
|
|
420
|
-
| `it.skip` / `test.skip` | Skipped test (still synced) |
|
|
421
|
-
| `xit` / `xtest` | Jasmine-style skip |
|
|
422
|
-
| `it.concurrent` / `test.concurrent` | Concurrent test |
|
|
423
|
-
|
|
424
|
-
### Source mapping
|
|
425
|
-
|
|
426
|
-
| JavaScript/TS source | Azure TC field |
|
|
427
|
-
|----------------------|---------------|
|
|
428
|
-
| JSDoc `/** ... */` first non-numbered line | TC **Title** |
|
|
429
|
-
| Numbered lines `N. text` in JSDoc | TC **Steps** (action) |
|
|
430
|
-
| Numbered lines `N. Check: text` in JSDoc | TC **Steps** (expected result, when `useExpectedResult: true`) |
|
|
431
|
-
| `// @tags: smoke, regression` above `it()` | TC **Tags** (comma-separated list) |
|
|
432
|
-
| `// @smoke` above `it()` (single-word) | TC **Tag** |
|
|
433
|
-
| `// @tc:ID` above `it()` | TC ID (written back after first push) |
|
|
434
|
-
| `{basename} > {describe} > {it title}` | `AutomatedTestName` (Jest report format) |
|
|
435
|
-
|
|
436
|
-
### Example
|
|
437
|
-
|
|
438
|
-
```typescript
|
|
439
|
-
describe('Checkout', () => {
|
|
440
|
-
|
|
441
|
-
/**
|
|
442
|
-
* Add item and complete checkout
|
|
443
|
-
* 1. Sign in as standard_user
|
|
444
|
-
* 2. Add Sauce Labs Backpack to cart
|
|
445
|
-
* 3. Proceed through checkout
|
|
446
|
-
* 4. Check: Order confirmation page is shown
|
|
447
|
-
*/
|
|
448
|
-
// @tc:1041 // written back by ado-sync after first push
|
|
449
|
-
// @tags: smoke, regression
|
|
450
|
-
it('adds item and completes checkout', async () => {
|
|
451
|
-
// ...
|
|
452
|
-
});
|
|
453
|
-
|
|
454
|
-
// @tc:1042
|
|
455
|
-
// @smoke
|
|
456
|
-
it('shows error on invalid card', async () => {
|
|
457
|
-
// ...
|
|
458
|
-
});
|
|
459
|
-
|
|
460
|
-
});
|
|
461
|
-
```
|
|
462
|
-
|
|
463
|
-
### Recommended config
|
|
464
|
-
|
|
465
|
-
```json
|
|
466
|
-
{
|
|
467
|
-
"local": {
|
|
468
|
-
"type": "javascript",
|
|
469
|
-
"include": ["src/**/*.spec.ts", "tests/**/*.test.js"],
|
|
470
|
-
"exclude": ["**/*.helper.ts", "**/*.fixture.ts"]
|
|
471
|
-
},
|
|
472
|
-
"sync": {
|
|
473
|
-
"markAutomated": true
|
|
474
|
-
}
|
|
475
|
-
}
|
|
476
|
-
```
|
|
477
|
-
|
|
478
|
-
### Notes
|
|
479
|
-
|
|
480
|
-
- **`pull` is not supported** for JavaScript/TypeScript files. Only `push` applies.
|
|
481
|
-
- **Dynamic titles** — `it` / `test` calls with template literals or computed values are skipped. Use string literals for reliable syncing.
|
|
482
|
-
- **`describe` nesting** — arbitrarily deep `describe()` nesting is supported. All enclosing describe titles are included in the `automatedTestName`.
|
|
483
|
-
- **Path-based auto-tagging** — directory segments starting with `@` (e.g. `tests/@smoke/`) are automatically applied as tags on all tests inside.
|
|
484
|
-
|
|
485
|
-
---
|
|
486
|
-
|
|
487
|
-
## Playwright Test `.js` / `.ts`
|
|
488
|
-
|
|
489
|
-
Set `local.type: "playwright"`. Supports **Playwright Test** (`@playwright/test`) for both JavaScript and TypeScript. Each `test()` call becomes one Test Case.
|
|
490
|
-
|
|
491
|
-
> **Jest/Jasmine/WebdriverIO** tests use `local.type: "javascript"` — not this type.
|
|
492
|
-
|
|
493
|
-
### Detected test functions
|
|
494
|
-
|
|
495
|
-
| Function | Description |
|
|
496
|
-
|----------|-------------|
|
|
497
|
-
| `test(title, fn)` | Standard test |
|
|
498
|
-
| `test.only` | Focused test |
|
|
499
|
-
| `test.skip` | Skipped test (still synced) |
|
|
500
|
-
| `test.fixme` | Test marked as broken (still synced, tagged `wip`) |
|
|
501
|
-
| `test.fail` | Test expected to fail (still synced) |
|
|
502
|
-
| `test.describe` | Nested describe block |
|
|
503
|
-
| `test.describe.parallel` | Parallel describe block |
|
|
504
|
-
| `test.describe.serial` | Serial describe block |
|
|
505
|
-
|
|
506
|
-
### ID tagging — native annotation (recommended)
|
|
507
|
-
|
|
508
|
-
Playwright Test has a built-in annotation API. Use it to attach the TC ID directly in the test definition — no comments needed:
|
|
509
|
-
|
|
510
|
-
```typescript
|
|
511
|
-
// Single annotation (most common)
|
|
512
|
-
test('completes checkout', {
|
|
513
|
-
annotation: { type: 'tc', description: '1041' },
|
|
514
|
-
tag: '@smoke',
|
|
515
|
-
}, async ({ page }) => { ... });
|
|
516
|
-
|
|
517
|
-
// Tag-style ID (alternative)
|
|
518
|
-
test('cart badge shows count', {
|
|
519
|
-
tag: ['@tc:1043', '@smoke'],
|
|
520
|
-
}, async ({ page }) => { ... });
|
|
521
|
-
|
|
522
|
-
// Array form — combine TC ID with other annotations
|
|
523
|
-
test.fixme('promo code applies discount', {
|
|
524
|
-
annotation: [
|
|
525
|
-
{ type: 'tc', description: '1042' },
|
|
526
|
-
{ type: 'issue', description: 'promo-code not yet implemented' },
|
|
527
|
-
],
|
|
528
|
-
tag: '@wip',
|
|
529
|
-
}, async ({ page }) => { ... });
|
|
530
|
-
```
|
|
531
|
-
|
|
532
|
-
**Writeback** — on the first `push`, ado-sync injects `{ annotation: { type: 'tc', description: 'N' } }` into the test options object. Subsequent pushes update the description in place.
|
|
533
|
-
|
|
534
|
-
**Comment fallback** — `// @tc:ID` above `test()` is still recognised and written for edge cases where the options object cannot be parsed (e.g., multi-line title spanning several lines).
|
|
535
|
-
|
|
536
|
-
### Source mapping
|
|
537
|
-
|
|
538
|
-
| Playwright source | Azure TC field |
|
|
539
|
-
|-------------------|---------------|
|
|
540
|
-
| JSDoc `/** ... */` first non-numbered line | TC **Title** |
|
|
541
|
-
| Numbered lines `N. text` in JSDoc | TC **Steps** (action) |
|
|
542
|
-
| Numbered lines `N. Check: text` in JSDoc | TC **Steps** (expected result, when `useExpectedResult: true`) |
|
|
543
|
-
| `annotation: { type: 'tc', description: 'N' }` | TC ID (**preferred**) |
|
|
544
|
-
| `annotation: [{ type: 'tc', description: 'N' }, ...]` | TC ID (array form) |
|
|
545
|
-
| `tag: '@tc:N'` or `tag: ['@tc:N', ...]` | TC ID (tag form) |
|
|
546
|
-
| `tag: '@smoke'` etc. | TC **Tags** |
|
|
547
|
-
| `// @tags: smoke, regression` above `test()` | TC **Tags** (comment form) |
|
|
548
|
-
| `// @tc:ID` above `test()` | TC ID (comment fallback) |
|
|
549
|
-
| `{basename} > {describe} > {test title}` | `AutomatedTestName` |
|
|
550
|
-
|
|
551
|
-
**Priority**: native `annotation` > native `tag` > `// @tc:N` comment.
|
|
552
|
-
|
|
553
|
-
### TypeScript example
|
|
554
|
-
|
|
555
|
-
```typescript
|
|
556
|
-
import { test, expect } from '@playwright/test';
|
|
557
|
-
|
|
558
|
-
test.describe('SauceDemo Checkout', () => {
|
|
559
|
-
|
|
560
|
-
/**
|
|
561
|
-
* Complete checkout with valid details
|
|
562
|
-
* 1. Log in as standard_user
|
|
563
|
-
* 2. Add Sauce Labs Backpack to cart
|
|
564
|
-
* 3. Proceed through checkout and enter name and zip
|
|
565
|
-
* 4. Check: Order confirmation page is displayed
|
|
566
|
-
*/
|
|
567
|
-
test('completes checkout with valid details', {
|
|
568
|
-
annotation: { type: 'tc', description: '1041' },
|
|
569
|
-
tag: '@smoke',
|
|
570
|
-
}, async ({ page }) => {
|
|
571
|
-
// ...
|
|
572
|
-
});
|
|
573
|
-
|
|
574
|
-
test.fixme('promo code applies discount', {
|
|
575
|
-
annotation: [
|
|
576
|
-
{ type: 'tc', description: '1042' },
|
|
577
|
-
{ type: 'issue', description: 'promo-code not yet implemented' },
|
|
578
|
-
],
|
|
579
|
-
tag: '@wip',
|
|
580
|
-
}, async ({ page }) => {
|
|
581
|
-
// known issue — still synced to Azure
|
|
582
|
-
});
|
|
583
|
-
});
|
|
584
|
-
|
|
585
|
-
test.describe.parallel('Cart assertions', () => {
|
|
586
|
-
|
|
587
|
-
test('cart badge shows correct count', {
|
|
588
|
-
annotation: { type: 'tc', description: '1043' },
|
|
589
|
-
}, async ({ page }) => {
|
|
590
|
-
// ...
|
|
591
|
-
});
|
|
592
|
-
});
|
|
593
|
-
```
|
|
594
|
-
|
|
595
|
-
### JavaScript example (comment fallback style)
|
|
596
|
-
|
|
597
|
-
```javascript
|
|
598
|
-
const { test, expect } = require('@playwright/test');
|
|
599
|
-
|
|
600
|
-
test.describe('SauceDemo Login', () => {
|
|
601
|
-
|
|
602
|
-
/**
|
|
603
|
-
* Valid credentials redirect to inventory page
|
|
604
|
-
* 1. Navigate to https://www.saucedemo.com
|
|
605
|
-
* 2. Enter username "standard_user" and password "secret_sauce"
|
|
606
|
-
* 3. Click the login button
|
|
607
|
-
* 4. Check: URL contains "inventory.html"
|
|
608
|
-
*/
|
|
609
|
-
// @tc:1044 ← comment style also works; ado-sync upgrades to annotation on next push
|
|
610
|
-
test('valid credentials redirect to inventory', async ({ page }) => {
|
|
611
|
-
// ...
|
|
612
|
-
});
|
|
613
|
-
});
|
|
614
|
-
```
|
|
615
|
-
|
|
616
|
-
### Recommended config
|
|
617
|
-
|
|
618
|
-
```json
|
|
619
|
-
{
|
|
620
|
-
"local": {
|
|
621
|
-
"type": "playwright",
|
|
622
|
-
"include": ["tests/**/*.spec.ts", "tests/**/*.spec.js"],
|
|
623
|
-
"exclude": ["**/*.helper.ts", "**/*.fixture.ts"]
|
|
624
|
-
},
|
|
625
|
-
"sync": {
|
|
626
|
-
"markAutomated": true
|
|
627
|
-
}
|
|
628
|
-
}
|
|
629
|
-
```
|
|
630
|
-
|
|
631
|
-
### Notes
|
|
632
|
-
|
|
633
|
-
- **`pull` is not supported** for Playwright files. Only `push` applies.
|
|
634
|
-
- **`test.describe` nesting** — arbitrarily deep nesting is supported. All enclosing describe titles are included in the `automatedTestName`.
|
|
635
|
-
- **`test.fixme` / `test.fail`** — both are parsed and synced as normal test cases. Use `tag: '@wip'` on `test.fixme` to mark them in Azure.
|
|
636
|
-
- **Publishing results** — set `testResultFormat: playwrightJson` when using `publish-test-results`.
|
|
637
|
-
- **Native annotation priority** — `annotation: { type: 'tc', … }` is read before `tag:` which is read before `// @tc:N` comments. The tool writes back using native annotation on every sync.
|
|
638
|
-
|
|
639
|
-
---
|
|
640
|
-
|
|
641
|
-
## Puppeteer `.js` / `.ts`
|
|
642
|
-
|
|
643
|
-
Set `local.type: "puppeteer"`. Supports **Puppeteer** tests written with Jest or Mocha as the test runner. Puppeteer tests use the same `describe()` / `it()` / `test()` API as Jest, so this type is an alias for `javascript` — choose it for clarity when your project uses Puppeteer.
|
|
644
|
-
|
|
645
|
-
```typescript
|
|
646
|
-
import puppeteer from 'puppeteer';
|
|
647
|
-
|
|
648
|
-
describe('SauceDemo Login', () => {
|
|
649
|
-
|
|
650
|
-
/**
|
|
651
|
-
* Valid credentials redirect to inventory page
|
|
652
|
-
* 1. Navigate to https://www.saucedemo.com
|
|
653
|
-
* 2. Enter "standard_user" into the username field
|
|
654
|
-
* 3. Enter "secret_sauce" into the password field
|
|
655
|
-
* 4. Click the login button
|
|
656
|
-
* 5. Check: URL contains "inventory"
|
|
657
|
-
*/
|
|
658
|
-
// @tc:1060
|
|
659
|
-
it('redirects to inventory on valid login', async () => {
|
|
660
|
-
// ...
|
|
661
|
-
});
|
|
662
|
-
});
|
|
663
|
-
```
|
|
664
|
-
|
|
665
|
-
### Recommended config
|
|
666
|
-
|
|
667
|
-
```json
|
|
668
|
-
{
|
|
669
|
-
"local": {
|
|
670
|
-
"type": "puppeteer",
|
|
671
|
-
"include": ["tests/**/*.test.ts"],
|
|
672
|
-
"exclude": ["**/*.helper.ts"]
|
|
673
|
-
},
|
|
674
|
-
"sync": { "markAutomated": true }
|
|
675
|
-
}
|
|
676
|
-
```
|
|
677
|
-
|
|
678
|
-
### Notes
|
|
679
|
-
|
|
680
|
-
- ID writeback format: `// @tc:12345` immediately above the `it()`/`test()` line.
|
|
681
|
-
- `pull` is not supported for Puppeteer files — only `push` applies.
|
|
682
|
-
- `describe()` nesting is fully supported.
|
|
683
|
-
|
|
684
|
-
---
|
|
685
|
-
|
|
686
|
-
## Cypress `.cy.js` / `.cy.ts`
|
|
687
|
-
|
|
688
|
-
Set `local.type: "cypress"`. Supports **Cypress** tests. Cypress uses `describe()` / `context()` / `it()` / `specify()` — all four are detected.
|
|
689
|
-
|
|
690
|
-
```typescript
|
|
691
|
-
describe('SauceDemo Login', () => {
|
|
692
|
-
|
|
693
|
-
context('valid credentials', () => {
|
|
694
|
-
|
|
695
|
-
/**
|
|
696
|
-
* Successful login redirects to inventory page
|
|
697
|
-
* 1. Visit https://www.saucedemo.com
|
|
698
|
-
* 2. Enter "standard_user" into the username field
|
|
699
|
-
* 3. Enter "secret_sauce" into the password field
|
|
700
|
-
* 4. Click the login button
|
|
701
|
-
* 5. Check: URL contains "inventory"
|
|
702
|
-
*/
|
|
703
|
-
// @tc:1070
|
|
704
|
-
// @tags: smoke
|
|
705
|
-
it('redirects to inventory', () => {
|
|
706
|
-
// ...
|
|
707
|
-
});
|
|
708
|
-
|
|
709
|
-
// @tc:1071
|
|
710
|
-
specify('locked out user sees error', () => {
|
|
711
|
-
// ...
|
|
712
|
-
});
|
|
713
|
-
});
|
|
714
|
-
});
|
|
715
|
-
```
|
|
716
|
-
|
|
717
|
-
### Recommended config
|
|
718
|
-
|
|
719
|
-
```json
|
|
720
|
-
{
|
|
721
|
-
"local": {
|
|
722
|
-
"type": "cypress",
|
|
723
|
-
"include": ["cypress/e2e/**/*.cy.ts", "cypress/e2e/**/*.cy.js"],
|
|
724
|
-
"exclude": ["cypress/support/**"]
|
|
725
|
-
},
|
|
726
|
-
"sync": { "markAutomated": true }
|
|
727
|
-
}
|
|
728
|
-
```
|
|
729
|
-
|
|
730
|
-
### Notes
|
|
731
|
-
|
|
732
|
-
- `context()` is treated as a `describe()` alias (Cypress convention).
|
|
733
|
-
- `specify()` is treated as an `it()` alias (Cypress convention).
|
|
734
|
-
- ID writeback format: `// @tc:12345` immediately above the `it()`/`specify()` line.
|
|
735
|
-
- `pull` is not supported for Cypress files — only `push` applies.
|
|
736
|
-
|
|
737
|
-
---
|
|
738
|
-
|
|
739
|
-
## TestCafe `.js` / `.ts`
|
|
740
|
-
|
|
741
|
-
Set `local.type: "testcafe"`. Supports **TestCafe** tests using the `fixture` / `test` API.
|
|
742
|
-
|
|
743
|
-
### ID tagging — native `test.meta()` (recommended)
|
|
744
|
-
|
|
745
|
-
TestCafe has a built-in `.meta()` method that attaches key-value metadata to a test. Use it to store the TC ID natively — no comments needed:
|
|
746
|
-
|
|
747
|
-
```javascript
|
|
748
|
-
// Key-value form (recommended — clean and simple)
|
|
749
|
-
test.meta('tc', '1080')('redirects to inventory on valid login', async t => { ... });
|
|
750
|
-
|
|
751
|
-
// Object form (use when adding multiple metadata fields)
|
|
752
|
-
test.meta({ tc: '1081', priority: 'high' })('locked out user sees error', async t => { ... });
|
|
753
|
-
|
|
754
|
-
// Combined with test.skip
|
|
755
|
-
test.skip.meta('tc', '1082')('skipped test', async t => { ... });
|
|
756
|
-
```
|
|
757
|
-
|
|
758
|
-
**Writeback** — on the first `push`, ado-sync injects `.meta('tc', 'N')` between the test function and its title call. For example:
|
|
759
|
-
```
|
|
760
|
-
test('title', fn) → test.meta('tc', '12345')('title', fn)
|
|
761
|
-
```
|
|
762
|
-
Subsequent pushes update the existing `.meta()` value in place.
|
|
763
|
-
|
|
764
|
-
**Comment fallback** — `// @tc:N` above `test()` is still recognised.
|
|
765
|
-
|
|
766
|
-
### Source mapping
|
|
767
|
-
|
|
768
|
-
| TestCafe source | Azure TC field |
|
|
769
|
-
|-----------------|---------------|
|
|
770
|
-
| JSDoc `/** ... */` first non-numbered line | TC **Title** |
|
|
771
|
-
| Numbered lines `N. text` in JSDoc | TC **Steps** (action) |
|
|
772
|
-
| Numbered lines `N. Check: text` in JSDoc | TC **Steps** (expected result) |
|
|
773
|
-
| `test.meta('tc', 'N')` | TC ID (**preferred**) |
|
|
774
|
-
| `test.meta({ tc: 'N' })` | TC ID (object form) |
|
|
775
|
-
| `// @tags: smoke, regression` above `test()` | TC **Tags** |
|
|
776
|
-
| `// @tc:N` above `test()` | TC ID (comment fallback) |
|
|
777
|
-
| `{basename} > {fixture} > {test title}` | `AutomatedTestName` |
|
|
778
|
-
|
|
779
|
-
### Example
|
|
780
|
-
|
|
781
|
-
```javascript
|
|
782
|
-
fixture('SauceDemo Login')
|
|
783
|
-
.page('https://www.saucedemo.com');
|
|
784
|
-
|
|
785
|
-
/**
|
|
786
|
-
* Valid credentials redirect to inventory page
|
|
787
|
-
* 1. Type "standard_user" into the username field
|
|
788
|
-
* 2. Type "secret_sauce" into the password field
|
|
789
|
-
* 3. Click the login button
|
|
790
|
-
* 4. Check: URL contains "inventory"
|
|
791
|
-
*/
|
|
792
|
-
// @smoke
|
|
793
|
-
test.meta('tc', '1080')('redirects to inventory on valid login', async t => {
|
|
794
|
-
// ...
|
|
795
|
-
});
|
|
796
|
-
|
|
797
|
-
test.meta({ tc: '1081', priority: 'high' })('locked out user sees error', async t => {
|
|
798
|
-
// ...
|
|
799
|
-
});
|
|
800
|
-
|
|
801
|
-
// Comment style still works (written by ado-sync before meta support was added)
|
|
802
|
-
// @tc:1082
|
|
803
|
-
test.skip('skipped test — still synced', async t => { ... });
|
|
804
|
-
```
|
|
805
|
-
|
|
806
|
-
### Recommended config
|
|
807
|
-
|
|
808
|
-
```json
|
|
809
|
-
{
|
|
810
|
-
"local": {
|
|
811
|
-
"type": "testcafe",
|
|
812
|
-
"include": ["tests/**/*.js", "tests/**/*.ts"],
|
|
813
|
-
"exclude": ["tests/helpers/**"]
|
|
814
|
-
},
|
|
815
|
-
"sync": { "markAutomated": true }
|
|
816
|
-
}
|
|
817
|
-
```
|
|
818
|
-
|
|
819
|
-
### Notes
|
|
820
|
-
|
|
821
|
-
- The `fixture()` title is used as the group / suite name and included in `automatedTestName`.
|
|
822
|
-
- `test.skip()` and `test.only()` are both parsed and synced.
|
|
823
|
-
- `pull` is not supported for TestCafe files — only `push` applies.
|
|
824
|
-
- JSDoc `/** ... */` above a `test.meta(...)` call is parsed for TC title and numbered steps.
|
|
825
|
-
|
|
826
|
-
---
|
|
827
|
-
|
|
828
|
-
## Appium
|
|
829
|
-
|
|
830
|
-
Appium tests are written in your language of choice. Use the `local.type` that matches the language:
|
|
831
|
-
|
|
832
|
-
| Appium language binding | Use `local.type` | Notes |
|
|
833
|
-
|-------------------------|-----------------|-------|
|
|
834
|
-
| JavaScript / TypeScript (WebdriverIO, Mocha, Jest) | `javascript` | Detects `describe()`/`it()`/`test()` |
|
|
835
|
-
| Java (JUnit 4, JUnit 5, TestNG) | `java` | Detects `@Test` annotations |
|
|
836
|
-
| Python (pytest) | `python` | Detects `def test_*()` functions |
|
|
837
|
-
| C# (NUnit, MSTest) | `csharp` | Detects `[TestMethod]`/`[Test]` attributes |
|
|
838
|
-
|
|
839
|
-
**JavaScript/TypeScript Appium example:**
|
|
840
|
-
|
|
841
|
-
```typescript
|
|
842
|
-
// @tc:12345
|
|
843
|
-
// @smoke
|
|
844
|
-
describe('Login', () => {
|
|
845
|
-
/**
|
|
846
|
-
* User can log in with valid credentials
|
|
847
|
-
*
|
|
848
|
-
* 1. Navigate to the login screen
|
|
849
|
-
* 2. Enter email into the email field
|
|
850
|
-
* 3. Enter password into the password field
|
|
851
|
-
* 4. Tap the login button
|
|
852
|
-
* 5. Check: Home screen is displayed
|
|
853
|
-
*/
|
|
854
|
-
it('can log in with valid credentials', async () => {
|
|
855
|
-
await driver.execute('mobile: launchApp', { bundleId: 'com.example.app' });
|
|
856
|
-
await $('~emailField').setValue('user@example.com');
|
|
857
|
-
await $('~passwordField').setValue('secret');
|
|
858
|
-
await $('~loginButton').click();
|
|
859
|
-
await expect($('~homeScreen')).toBeDisplayed();
|
|
860
|
-
});
|
|
861
|
-
});
|
|
862
|
-
```
|
|
863
|
-
|
|
864
|
-
Recommended `ado-sync.json`:
|
|
865
|
-
|
|
866
|
-
```json
|
|
867
|
-
{
|
|
868
|
-
"local": {
|
|
869
|
-
"type": "javascript",
|
|
870
|
-
"include": ["test/**/*.spec.ts"]
|
|
871
|
-
},
|
|
872
|
-
"sync": { "markAutomated": true }
|
|
873
|
-
}
|
|
874
|
-
```
|
|
875
|
-
|
|
876
|
-
---
|
|
877
|
-
|
|
878
|
-
## Detox `.js` / `.ts`
|
|
879
|
-
|
|
880
|
-
Set `local.type: "detox"` for React Native end-to-end tests using [Detox](https://wix.github.io/Detox/).
|
|
881
|
-
|
|
882
|
-
Detox uses the same Jest `describe()`/`it()` API as Jest — the existing JavaScript parser handles it.
|
|
883
|
-
|
|
884
|
-
```typescript
|
|
885
|
-
// @tc:12345
|
|
886
|
-
// @smoke
|
|
887
|
-
describe('Login flow', () => {
|
|
888
|
-
/**
|
|
889
|
-
* User can log in with valid credentials
|
|
890
|
-
*
|
|
891
|
-
* 1. Launch the app
|
|
892
|
-
* 2. Type email into the email field
|
|
893
|
-
* 3. Tap the login button
|
|
894
|
-
* 4. Check: Welcome screen is visible
|
|
895
|
-
*/
|
|
896
|
-
it('logs in with valid credentials', async () => {
|
|
897
|
-
await device.launchApp();
|
|
898
|
-
await element(by.id('email')).typeText('user@example.com');
|
|
899
|
-
await element(by.id('password')).typeText('secret');
|
|
900
|
-
await element(by.id('login-btn')).tap();
|
|
901
|
-
await expect(element(by.id('welcome-screen'))).toBeVisible();
|
|
902
|
-
});
|
|
903
|
-
});
|
|
904
|
-
```
|
|
905
|
-
|
|
906
|
-
Recommended `ado-sync.json`:
|
|
907
|
-
|
|
908
|
-
```json
|
|
909
|
-
{
|
|
910
|
-
"local": {
|
|
911
|
-
"type": "detox",
|
|
912
|
-
"include": ["e2e/**/*.test.ts"],
|
|
913
|
-
"exclude": ["e2e/setup/**"]
|
|
914
|
-
},
|
|
915
|
-
"sync": { "markAutomated": true }
|
|
916
|
-
}
|
|
917
|
-
```
|
|
918
|
-
|
|
919
|
-
**Notes:**
|
|
920
|
-
- `describe()` blocks define the test group (same as Jest).
|
|
921
|
-
- `// @tc:12345` is inserted immediately above the `it()`/`test()` line on writeback.
|
|
922
|
-
- `pull` is not supported for Detox files — only `push`.
|
|
923
|
-
|
|
924
|
-
---
|
|
925
|
-
|
|
926
|
-
## Espresso `.java` / `.kt`
|
|
927
|
-
|
|
928
|
-
Set `local.type: "espresso"` for Android UI tests using [Espresso](https://developer.android.com/training/testing/espresso).
|
|
929
|
-
|
|
930
|
-
Espresso tests use JUnit 4 `@Test` annotations — the same parser as `java` handles them.
|
|
931
|
-
|
|
932
|
-
```java
|
|
933
|
-
@RunWith(AndroidJUnit4.class)
|
|
934
|
-
public class LoginInstrumentedTest {
|
|
935
|
-
|
|
936
|
-
// @tc:12345
|
|
937
|
-
// @smoke
|
|
938
|
-
/**
|
|
939
|
-
* User can log in with valid credentials
|
|
940
|
-
*
|
|
941
|
-
* 1. Type email into the email field
|
|
942
|
-
* 2. Type password into the password field
|
|
943
|
-
* 3. Click the login button
|
|
944
|
-
* 4. Check: Welcome screen is displayed
|
|
945
|
-
*/
|
|
946
|
-
@Test
|
|
947
|
-
public void userCanLoginWithValidCredentials() {
|
|
948
|
-
onView(withId(R.id.email)).perform(typeText("user@example.com"), closeSoftKeyboard());
|
|
949
|
-
onView(withId(R.id.password)).perform(typeText("secret"), closeSoftKeyboard());
|
|
950
|
-
onView(withId(R.id.login_button)).perform(click());
|
|
951
|
-
onView(withId(R.id.welcome_screen)).check(matches(isDisplayed()));
|
|
952
|
-
}
|
|
953
|
-
}
|
|
954
|
-
```
|
|
955
|
-
|
|
956
|
-
**Kotlin** (same `@Test` detection):
|
|
957
|
-
|
|
958
|
-
```kotlin
|
|
959
|
-
@RunWith(AndroidJUnit4::class)
|
|
960
|
-
class LoginInstrumentedTest {
|
|
961
|
-
|
|
962
|
-
// @tc:12345
|
|
963
|
-
@Test
|
|
964
|
-
fun `user can login with valid credentials`() {
|
|
965
|
-
onView(withId(R.id.email)).perform(typeText("user@example.com"), closeSoftKeyboard())
|
|
966
|
-
onView(withId(R.id.login_button)).perform(click())
|
|
967
|
-
onView(withId(R.id.welcome_screen)).check(matches(isDisplayed()))
|
|
968
|
-
}
|
|
969
|
-
}
|
|
970
|
-
```
|
|
971
|
-
|
|
972
|
-
Recommended `ado-sync.json`:
|
|
973
|
-
|
|
974
|
-
```json
|
|
975
|
-
{
|
|
976
|
-
"local": {
|
|
977
|
-
"type": "espresso",
|
|
978
|
-
"include": ["app/src/androidTest/**/*.java", "app/src/androidTest/**/*.kt"],
|
|
979
|
-
"exclude": ["**/*BaseTest.java", "**/*Helper.java"]
|
|
980
|
-
},
|
|
981
|
-
"sync": { "markAutomated": true }
|
|
982
|
-
}
|
|
983
|
-
```
|
|
984
|
-
|
|
985
|
-
**Notes:**
|
|
986
|
-
- ID writeback format: `// @tc:12345` immediately above the `@Test` annotation line.
|
|
987
|
-
- Kotlin backtick method names (`` fun `my test name`() ``) are supported — the method name is used as the fallback title.
|
|
988
|
-
- `pull` is not supported — only `push`.
|
|
989
|
-
|
|
990
|
-
---
|
|
991
|
-
|
|
992
|
-
## XCUITest `.swift`
|
|
993
|
-
|
|
994
|
-
Set `local.type: "xcuitest"` for iOS and macOS UI automation tests using Apple's [XCTest](https://developer.apple.com/documentation/xctest) framework.
|
|
995
|
-
|
|
996
|
-
Tests are `func test*()` methods inside classes that extend `XCTestCase`.
|
|
997
|
-
|
|
998
|
-
```swift
|
|
999
|
-
import XCTest
|
|
1000
|
-
|
|
1001
|
-
class LoginTests: XCTestCase {
|
|
1002
|
-
|
|
1003
|
-
// @tc:12345
|
|
1004
|
-
// @smoke
|
|
1005
|
-
/// User can log in with valid credentials
|
|
1006
|
-
///
|
|
1007
|
-
/// 1. Launch the app
|
|
1008
|
-
/// 2. Enter email into the email field
|
|
1009
|
-
/// 3. Enter password into the password field
|
|
1010
|
-
/// 4. Tap the login button
|
|
1011
|
-
/// 5. Check: Welcome screen is visible
|
|
1012
|
-
func testUserCanLoginWithValidCredentials() {
|
|
1013
|
-
let app = XCUIApplication()
|
|
1014
|
-
app.launch()
|
|
1015
|
-
app.textFields["Email"].typeText("user@example.com")
|
|
1016
|
-
app.secureTextFields["Password"].typeText("secret")
|
|
1017
|
-
app.buttons["Log In"].tap()
|
|
1018
|
-
XCTAssertTrue(app.staticTexts["Welcome"].exists)
|
|
1019
|
-
}
|
|
1020
|
-
|
|
1021
|
-
func testLoginFailsWithInvalidPassword() {
|
|
1022
|
-
let app = XCUIApplication()
|
|
1023
|
-
app.launch()
|
|
1024
|
-
app.textFields["Email"].typeText("user@example.com")
|
|
1025
|
-
app.secureTextFields["Password"].typeText("wrong")
|
|
1026
|
-
app.buttons["Log In"].tap()
|
|
1027
|
-
XCTAssertTrue(app.staticTexts["Invalid credentials"].exists)
|
|
1028
|
-
}
|
|
1029
|
-
}
|
|
1030
|
-
```
|
|
1031
|
-
|
|
1032
|
-
Recommended `ado-sync.json`:
|
|
1033
|
-
|
|
1034
|
-
```json
|
|
1035
|
-
{
|
|
1036
|
-
"local": {
|
|
1037
|
-
"type": "xcuitest",
|
|
1038
|
-
"include": ["UITests/**/*.swift"],
|
|
1039
|
-
"exclude": ["UITests/**/*Helper.swift", "UITests/**/*Base.swift"]
|
|
1040
|
-
},
|
|
1041
|
-
"sync": { "markAutomated": true }
|
|
1042
|
-
}
|
|
1043
|
-
```
|
|
1044
|
-
|
|
1045
|
-
**Notes:**
|
|
1046
|
-
- The enclosing class name is used as the test group (`automatedTestName = FileName > ClassName > methodName`).
|
|
1047
|
-
- Both `///` triple-slash (Swift idiomatic) and `/** ... */` block doc comments are parsed for TC title and numbered steps.
|
|
1048
|
-
- Method name → title fallback: `testUserCanLogin` → "User can login", `test_submit_form` → "Submit form".
|
|
1049
|
-
- ID writeback format: `// @tc:12345` immediately above the `func test*()` line.
|
|
1050
|
-
- `pull` is not supported — only `push`.
|
|
1051
|
-
|
|
1052
|
-
---
|
|
1053
|
-
|
|
1054
|
-
## Flutter `_test.dart`
|
|
1055
|
-
|
|
1056
|
-
Set `local.type: "flutter"` for Flutter widget tests and integration tests using the [`flutter_test`](https://api.flutter.dev/flutter/flutter_test/flutter_test-library.html) package.
|
|
1057
|
-
|
|
1058
|
-
```dart
|
|
1059
|
-
import 'package:flutter_test/flutter_test.dart';
|
|
1060
|
-
|
|
1061
|
-
void main() {
|
|
1062
|
-
group('Login', () {
|
|
1063
|
-
// @tc:12345
|
|
1064
|
-
// @smoke
|
|
1065
|
-
/// User can log in with valid credentials
|
|
1066
|
-
///
|
|
1067
|
-
/// 1. Tap the email field
|
|
1068
|
-
/// 2. Enter the email address
|
|
1069
|
-
/// 3. Tap the password field
|
|
1070
|
-
/// 4. Enter the password
|
|
1071
|
-
/// 5. Tap the Login button
|
|
1072
|
-
/// 6. Check: Welcome screen is visible
|
|
1073
|
-
testWidgets('can log in with valid credentials', (WidgetTester tester) async {
|
|
1074
|
-
await tester.pumpWidget(MyApp());
|
|
1075
|
-
await tester.enterText(find.byKey(Key('email')), 'user@example.com');
|
|
1076
|
-
await tester.enterText(find.byKey(Key('password')), 'secret');
|
|
1077
|
-
await tester.tap(find.byKey(Key('login-btn')));
|
|
1078
|
-
await tester.pumpAndSettle();
|
|
1079
|
-
expect(find.text('Welcome'), findsOneWidget);
|
|
1080
|
-
});
|
|
1081
|
-
|
|
1082
|
-
// @tc:12346
|
|
1083
|
-
testWidgets('shows error for invalid credentials', (WidgetTester tester) async {
|
|
1084
|
-
await tester.pumpWidget(MyApp());
|
|
1085
|
-
await tester.enterText(find.byKey(Key('email')), 'bad@example.com');
|
|
1086
|
-
await tester.enterText(find.byKey(Key('password')), 'wrong');
|
|
1087
|
-
await tester.tap(find.byKey(Key('login-btn')));
|
|
1088
|
-
await tester.pumpAndSettle();
|
|
1089
|
-
expect(find.text('Invalid credentials'), findsOneWidget);
|
|
1090
|
-
});
|
|
1091
|
-
});
|
|
1092
|
-
}
|
|
1093
|
-
```
|
|
1094
|
-
|
|
1095
|
-
Recommended `ado-sync.json`:
|
|
1096
|
-
|
|
1097
|
-
```json
|
|
1098
|
-
{
|
|
1099
|
-
"local": {
|
|
1100
|
-
"type": "flutter",
|
|
1101
|
-
"include": ["test/**/*_test.dart", "integration_test/**/*_test.dart"]
|
|
1102
|
-
},
|
|
1103
|
-
"sync": { "markAutomated": true }
|
|
1104
|
-
}
|
|
1105
|
-
```
|
|
1106
|
-
|
|
1107
|
-
**Notes:**
|
|
1108
|
-
- `group('title', () { ... })` is the describe equivalent. Nested groups are supported.
|
|
1109
|
-
- `testWidgets()`, `test()`, and `testUI()` (integration_test alias) are all detected.
|
|
1110
|
-
- The file base name is stripped of `_test.dart` / `.dart` suffixes for `automatedTestName`.
|
|
1111
|
-
- Both `///` triple-slash and `/** ... */` block doc comments are parsed.
|
|
1112
|
-
- ID writeback format: `// @tc:12345` immediately above the `testWidgets()`/`test()` line.
|
|
1113
|
-
- `pull` is not supported — only `push`.
|
|
1114
|
-
|
|
1115
|
-
---
|
|
1116
|
-
|
|
1117
|
-
## Robot Framework `.robot`
|
|
1118
|
-
|
|
1119
|
-
Set `local.type: "robot"`. Supports **Robot Framework** test suites. Each test case in the `*** Test Cases ***` or `*** Tasks ***` section becomes one Azure Test Case.
|
|
1120
|
-
|
|
1121
|
-
### Source mapping
|
|
1122
|
-
|
|
1123
|
-
| Robot Framework source | Azure TC field |
|
|
1124
|
-
|------------------------|---------------|
|
|
1125
|
-
| Test case name (non-indented line) | TC **Title** |
|
|
1126
|
-
| `[Documentation]` row | TC **Description** |
|
|
1127
|
-
| Numbered lines `N. text` in `[Documentation]` | TC **Steps** (action) |
|
|
1128
|
-
| Numbered lines `N. Check: text` in `[Documentation]` | TC **Steps** (expected result, when `useExpectedResult: true`) |
|
|
1129
|
-
| `[Tags]` values (excluding `tc:N`) | TC **Tags** |
|
|
1130
|
-
| `[Tags] tc:N` | TC ID (**preferred**; written back after first push) |
|
|
1131
|
-
| `# @tc:N` comment immediately above the test name | TC ID (comment fallback) |
|
|
1132
|
-
| Test case name | `AutomatedTestName` (for result linking) |
|
|
1133
|
-
|
|
1134
|
-
### Example
|
|
1135
|
-
|
|
1136
|
-
```robot
|
|
1137
|
-
*** Settings ***
|
|
1138
|
-
Library SeleniumLibrary
|
|
1139
|
-
|
|
1140
|
-
*** Test Cases ***
|
|
1141
|
-
# @tc:1041
|
|
1142
|
-
Add Item And Complete Checkout
|
|
1143
|
-
[Documentation] Add item and complete checkout
|
|
1144
|
-
... 1. Sign in as standard_user
|
|
1145
|
-
... 2. Add Sauce Labs Backpack to cart
|
|
1146
|
-
... 3. Proceed through checkout
|
|
1147
|
-
... 4. Check: Order confirmation page is shown
|
|
1148
|
-
[Tags] tc:1041 smoke
|
|
1149
|
-
Open Browser ${URL} chrome
|
|
1150
|
-
Input Text id:user-name standard_user
|
|
1151
|
-
Input Password id:password secret_sauce
|
|
1152
|
-
Click Button id:login-button
|
|
1153
|
-
|
|
1154
|
-
User Login Fails With Invalid Credentials
|
|
1155
|
-
[Documentation] Login fails with invalid credentials
|
|
1156
|
-
... 1. Navigate to the login page
|
|
1157
|
-
... 2. Enter an invalid username and password
|
|
1158
|
-
... 3. Click Login
|
|
1159
|
-
... 4. Check: Error message is displayed
|
|
1160
|
-
[Tags] regression
|
|
1161
|
-
Open Browser ${URL} chrome
|
|
1162
|
-
Input Text id:user-name bad_user
|
|
1163
|
-
Input Password id:password wrong
|
|
1164
|
-
Click Button id:login-button
|
|
1165
|
-
```
|
|
1166
|
-
|
|
1167
|
-
On the first `push`, ado-sync inserts `tc:N` into the `[Tags]` row. If no `[Tags]` row exists, one is created immediately after the test name line.
|
|
1168
|
-
|
|
1169
|
-
### ID writeback
|
|
1170
|
-
|
|
1171
|
-
| Location | Format |
|
|
1172
|
-
|----------|--------|
|
|
1173
|
-
| `[Tags]` row (preferred) | `tc:12345` value added/updated in the existing `[Tags]` row |
|
|
1174
|
-
| No `[Tags]` row | ` [Tags] tc:12345` inserted after the test name line |
|
|
1175
|
-
|
|
1176
|
-
The `# @tc:N` comment form above the test name is also recognised on read but is not used for writeback — `[Tags]` is always the authoritative location after a push.
|
|
1177
|
-
|
|
1178
|
-
### Path-based auto-tagging
|
|
1179
|
-
|
|
1180
|
-
Directory segments starting with `@` are automatically applied as tags on all test cases inside:
|
|
1181
|
-
|
|
1182
|
-
```
|
|
1183
|
-
tests/
|
|
1184
|
-
@smoke/
|
|
1185
|
-
login.robot ← all test cases get tag 'smoke'
|
|
1186
|
-
@regression/
|
|
1187
|
-
checkout.robot ← all test cases get tag 'regression'
|
|
1188
|
-
```
|
|
1189
|
-
|
|
1190
|
-
### Recommended config
|
|
1191
|
-
|
|
1192
|
-
```json
|
|
1193
|
-
{
|
|
1194
|
-
"local": {
|
|
1195
|
-
"type": "robot",
|
|
1196
|
-
"include": ["tests/**/*.robot"],
|
|
1197
|
-
"exclude": ["tests/resources/**", "tests/keywords/**"]
|
|
1198
|
-
},
|
|
1199
|
-
"sync": {
|
|
1200
|
-
"markAutomated": true
|
|
1201
|
-
}
|
|
1202
|
-
}
|
|
1203
|
-
```
|
|
1204
|
-
|
|
1205
|
-
### Notes
|
|
1206
|
-
|
|
1207
|
-
- **`pull` is not supported** for Robot Framework files. Only `push` and `publish-test-results` apply.
|
|
1208
|
-
- **`*** Tasks ***` sections** are treated the same as `*** Test Cases ***`.
|
|
1209
|
-
- **Keyword lines** — all indented non-settings rows are treated as TC steps if no numbered steps are found in `[Documentation]`.
|
|
1210
|
-
- **`[Setup]` / `[Teardown]`** — these settings rows are skipped; only keyword steps and `[Documentation]` / `[Tags]` are used.
|
|
1211
|
-
|
|
1212
|
-
---
|
|
1213
|
-
|
|
1214
|
-
## CSV `.csv`
|
|
1215
|
-
|
|
1216
|
-
Set `local.type: "csv"` to parse Azure DevOps / SpecSync tabular CSV exports.
|
|
1217
|
-
|
|
1218
|
-
**Expected column layout (9 columns):**
|
|
1219
|
-
|
|
1220
|
-
| Col | Field | Description |
|
|
1221
|
-
|-----|-------|-------------|
|
|
1222
|
-
| A (0) | ID | Azure Test Case ID — empty for new, filled after first push |
|
|
1223
|
-
| B (1) | Work Item Type | Always `Test Case` (ignored) |
|
|
1224
|
-
| C (2) | Title | `Scenario: My test` — non-empty on header row, empty on step rows |
|
|
1225
|
-
| D (3) | Test Step | Step number — empty on header row |
|
|
1226
|
-
| E (4) | Step Action | Step text, optionally prefixed with a Gherkin keyword |
|
|
1227
|
-
| F (5) | Step Expected | Expected result text (optional) |
|
|
1228
|
-
| G–I (6–8) | Area Path, Assigned To, State | Preserved but not used |
|
|
1229
|
-
|
|
1230
|
-
The first row (column headers) is always skipped automatically.
|
|
1231
|
-
|
|
1232
|
-
> **Note:** `ado-sync pull` is **not supported** for CSV files. Only push is supported.
|
|
1233
|
-
|
|
1234
|
-
---
|
|
1235
|
-
|
|
1236
|
-
## Excel `.xlsx`
|
|
1237
|
-
|
|
1238
|
-
Set `local.type: "excel"`. Uses the same 9-column layout as CSV. ado-sync reads the raw worksheet XML (no external xlsx library) and handles both `t="str"` and `t="inlineStr"` cells from Azure DevOps exports.
|
|
1239
|
-
|
|
1240
|
-
> **Note:** `ado-sync pull` is **not supported** for Excel files. Only push is supported.
|
|
1241
|
-
|
|
1242
|
-
---
|
|
1243
|
-
|
|
1244
|
-
## ID tags
|
|
1245
|
-
|
|
1246
|
-
After a first push, ado-sync writes the Azure TC ID back into the local file.
|
|
1247
|
-
|
|
1248
|
-
| Format | Location |
|
|
1249
|
-
|--------|----------|
|
|
1250
|
-
| Gherkin | `@tc:12345` tag on its own line above the `Scenario:` line |
|
|
1251
|
-
| Markdown | `@tc:12345` tag on the line immediately after the `### heading` |
|
|
1252
|
-
| C# MSTest | `[TestProperty("tc", "12345")]` attribute on the test method |
|
|
1253
|
-
| C# NUnit | `[Property("tc", "12345")]` attribute on the test method |
|
|
1254
|
-
| Java JUnit 5 | `@Tag("tc:12345")` annotation above `@Test` |
|
|
1255
|
-
| Java JUnit 4 / TestNG | `// @tc:12345` comment on the line above `@Test` |
|
|
1256
|
-
| Python pytest | `@pytest.mark.tc(12345)` decorator above `def test_*` |
|
|
1257
|
-
| JavaScript/TS | `// @tc:12345` comment on the line above `it()`/`test()` |
|
|
1258
|
-
| Playwright | `// @tc:12345` comment on the line above `test()` |
|
|
1259
|
-
| Robot Framework | `tc:12345` value in the `[Tags]` row of the test case body |
|
|
1260
|
-
| CSV | Numeric ID in column A of the matching title row |
|
|
1261
|
-
| Excel | Numeric ID in cell A of the matching title row |
|
|
1262
|
-
|
|
1263
|
-
### Custom prefix
|
|
1264
|
-
|
|
1265
|
-
```json
|
|
1266
|
-
{ "sync": { "tagPrefix": "azure" } }
|
|
1267
|
-
```
|
|
1268
|
-
|
|
1269
|
-
Result: `@azure:12345` in both Gherkin and Markdown files.
|
|
1270
|
-
|
|
1271
|
-
> **Warning:** Changing the prefix on an existing project means existing ID tags are no longer recognised. Do a project-wide find-and-replace before changing.
|
|
1272
|
-
|
|
1273
|
-
---
|
|
1274
|
-
|
|
1275
|
-
## Tag filtering
|
|
1276
|
-
|
|
1277
|
-
All commands accept `--tags` to limit which scenarios are processed. Uses the standard Cucumber tag expression language.
|
|
1278
|
-
|
|
1279
|
-
```bash
|
|
1280
|
-
ado-sync push --tags "@smoke"
|
|
1281
|
-
ado-sync push --tags "@smoke and not @wip"
|
|
1282
|
-
ado-sync pull --tags "@regression or @critical"
|
|
1283
|
-
```
|
|
1284
|
-
|
|
1285
|
-
Tags are evaluated against all tags on a scenario, including:
|
|
1286
|
-
|
|
1287
|
-
- Tags on the `Feature` block (Gherkin)
|
|
1288
|
-
- Tags on the `Scenario` / `Scenario Outline` block
|
|
1289
|
-
- Tags on individual `Examples` tables
|
|
1290
|
-
- Tags on the Markdown tag line (same `### heading` line or line below it)
|
|
1291
|
-
- **Path-based auto-tags** — directory segments starting with `@` are automatically applied as tags:
|
|
1292
|
-
|
|
1293
|
-
```
|
|
1294
|
-
specs/
|
|
1295
|
-
@smoke/
|
|
1296
|
-
login.feature ← all scenarios get tag 'smoke'
|
|
1297
|
-
@regression/
|
|
1298
|
-
@slow/
|
|
1299
|
-
checkout.feature ← all scenarios get tags 'regression' and 'slow'
|
|
1300
|
-
```
|
|
1301
|
-
|
|
1302
|
-
### `local.condition` — permanent filter
|
|
1303
|
-
|
|
1304
|
-
Use `local.condition` in config to permanently exclude scenarios without needing `--tags` on every command:
|
|
1305
|
-
|
|
1306
|
-
```json
|
|
1307
|
-
{ "local": { "condition": "@done and not (@ignored or @planned)" } }
|
|
1308
|
-
```
|
|
1309
|
-
|
|
1310
|
-
This filter is applied before `--tags`, so it acts as a baseline inclusion gate.
|
|
1311
|
-
|
|
1312
|
-
---
|
|
1313
|
-
|
|
1314
|
-
## Work item linking
|
|
1315
|
-
|
|
1316
|
-
Tags matching a configured `links` prefix are synced as Azure DevOps work item relations on the Test Case.
|
|
1317
|
-
|
|
1318
|
-
### Config
|
|
1319
|
-
|
|
1320
|
-
```json
|
|
1321
|
-
{
|
|
1322
|
-
"sync": {
|
|
1323
|
-
"links": [
|
|
1324
|
-
{ "prefix": "story", "relationship": "System.LinkTypes.Related" },
|
|
1325
|
-
{ "prefix": "bug", "relationship": "System.LinkTypes.Related" },
|
|
1326
|
-
{ "prefix": "req", "relationship": "Microsoft.VSTS.Common.TestedBy-Reverse" }
|
|
1327
|
-
]
|
|
1328
|
-
}
|
|
1329
|
-
}
|
|
1330
|
-
```
|
|
1331
|
-
|
|
1332
|
-
### Usage in Gherkin
|
|
1333
|
-
|
|
1334
|
-
```gherkin
|
|
1335
|
-
@tc:1042 @story:555 @bug:789
|
|
1336
|
-
Scenario: User can add items to cart
|
|
1337
|
-
...
|
|
1338
|
-
```
|
|
1339
|
-
|
|
1340
|
-
### Usage in Markdown
|
|
1341
|
-
|
|
1342
|
-
```markdown
|
|
1343
|
-
### User can add items to cart
|
|
1344
|
-
@tc:1042 @story:555 @bug:789
|
|
1345
|
-
|
|
1346
|
-
Steps:
|
|
1347
|
-
1. Add an item to the cart
|
|
1348
|
-
...
|
|
1349
|
-
```
|
|
1350
|
-
|
|
1351
|
-
On each `push`, relations are synced: new links are added, stale links (whose tag was removed) are deleted. The `relationship` value is the ADO relation type — `"System.LinkTypes.Related"` is the default if omitted.
|
|
1352
|
-
|
|
1353
|
-
---
|
|
1354
|
-
|
|
1355
|
-
## Attachments
|
|
1356
|
-
|
|
1357
|
-
Files can be attached to Azure Test Cases via tags. See [Attachments](advanced.md#attachments).
|