ado-sync 0.1.46 → 0.1.47

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -41,13 +41,9 @@ Local files ado-sync Azure DevOps
41
41
  ────────────── ───────────────── ────────────────
42
42
  .feature files ── push ──► create / update ──► Test Cases
43
43
  .md spec files ◄── pull ── apply changes ◄── (Work Items)
44
- .cs files ── push ──► (push-only) ──► + Associated Automation
45
- .java files ── push ──► (push-only) ──► + Associated Automation
46
- .py files ── push ──► (push-only) ──► + Associated Automation
47
- .js / .ts files ── push ──► (push-only) ──► + Associated Automation
48
- .robot files ── push ──► (push-only) ──► + Associated Automation
49
- .csv files ── push ──► (push-only)
50
- .xlsx files ── push ──► (push-only)
44
+ .cs / .java / ── push ──► (push-only) ──► + Associated Automation
45
+ .py / .js / .ts
46
+ .robot / .csv ── push ──► (push-only)
51
47
  write ID back
52
48
  @tc:12345 / [TestProperty] / @Tag / @pytest.mark / // @tc:
53
49
  TRX / JUnit / ── publish-test-results ──► Test Run results (linked to TCs)
@@ -56,34 +52,11 @@ Cucumber JSON
56
52
 
57
53
  On the **first push**, a new Test Case is created in Azure DevOps and its ID is written back into the local file. Every subsequent push uses that ID to update the existing Test Case.
58
54
 
59
- **ID writeback format per framework:**
60
-
61
- | Framework | ID written as |
62
- |---|---|
63
- | Gherkin / Markdown | `@tc:12345` tag / comment |
64
- | C# MSTest | `[TestProperty("tc", "12345")]` |
65
- | C# NUnit | `[Property("tc", "12345")]` |
66
- | Java JUnit 4 / TestNG | `// @tc:12345` comment above `@Test` |
67
- | Java JUnit 5 | `@Tag("tc:12345")` above `@Test` |
68
- | Python pytest | `@pytest.mark.tc(12345)` above `def test_*` |
69
- | JavaScript/TS (Jest/Jasmine/WebdriverIO) | `// @tc:12345` comment above `it()`/`test()` |
70
- | Playwright | `annotation: { type: 'tc', description: '12345' }` in test options *(preferred)*; or `// @tc:12345` comment fallback |
71
- | Puppeteer | `// @tc:12345` comment above `it()`/`test()` |
72
- | Cypress | `// @tc:12345` comment above `it()`/`specify()` |
73
- | TestCafe | `test.meta('tc', '12345')('title', fn)` *(preferred)*; or `// @tc:12345` comment fallback |
74
- | Detox | `// @tc:12345` comment above `it()`/`test()` |
75
- | Espresso (Java/Kotlin) | `// @tc:12345` comment above `@Test` |
76
- | XCUITest (Swift) | `// @tc:12345` comment above `func test*()` |
77
- | Flutter (Dart) | `// @tc:12345` comment above `testWidgets()`/`test()` |
78
- | Robot Framework | `tc:12345` value in `[Tags]` row of the test case body |
79
- | CSV | Numeric ID in column A |
80
- | Excel | Numeric ID in cell A |
81
-
82
55
  ---
83
56
 
84
57
  ## Quick start
85
58
 
86
- **Step 1 — Install**
59
+ **1 — Install**
87
60
 
88
61
  ```bash
89
62
  npm install -g ado-sync
@@ -91,16 +64,14 @@ npm install -g ado-sync
91
64
  npx ado-sync --help
92
65
  ```
93
66
 
94
- **Step 2 — Generate a config file**
67
+ **2 — Generate a config file**
95
68
 
96
69
  ```bash
97
- ado-sync init # creates ado-sync.json
98
- ado-sync init ado-sync.yml # YAML format if you prefer
70
+ ado-sync init # interactive wizard → creates ado-sync.json
71
+ ado-sync init ado-sync.yml # YAML format
99
72
  ```
100
73
 
101
- **Step 3 — Fill in your details**
102
-
103
- Open `ado-sync.json` and replace the placeholders:
74
+ **3 — Fill in your details**
104
75
 
105
76
  ```json
106
77
  {
@@ -119,306 +90,29 @@ Open `ado-sync.json` and replace the placeholders:
119
90
  | `testPlan.id` | Test Plans → click your plan → the number in the URL |
120
91
  | `AZURE_DEVOPS_TOKEN` | Azure DevOps → User Settings → Personal Access Tokens → New Token (scope: Test Management read/write) |
121
92
 
122
- **Step 4 — Set your token**
93
+ **4 — Set your token**
123
94
 
124
95
  ```bash
125
96
  export AZURE_DEVOPS_TOKEN=your_personal_access_token
126
97
  # or add it to a .env file in this directory
127
98
  ```
128
99
 
129
- **Step 5 — Preview then push**
100
+ **5 — Preview then push**
130
101
 
131
102
  ```bash
132
103
  ado-sync push --dry-run # preview — no changes made
133
104
  ado-sync push # create / update Test Cases in Azure DevOps
134
105
  ```
135
106
 
136
- On the first push, new Test Cases are created and their IDs are written back into your local files (`@tc:12345`). Every subsequent push updates them.
137
-
138
107
  ---
139
108
 
140
- ## CLI reference
141
-
142
- ```
143
- ado-sync [options] [command]
144
-
145
- Options:
146
- -c, --config <path> Path to config file (default: ado-sync.json)
147
- -V, --version Print version
148
- -h, --help Show help
149
-
150
- Commands:
151
- init [output] Generate a starter config file
152
- push [options] Push local specs to Azure DevOps
153
- pull [options] Pull updates from Azure DevOps into local files
154
- status [options] Show diff without making changes
155
- publish-test-results [opts] Publish TRX / JUnit / Cucumber JSON results to Azure DevOps
156
- help [command] Help for a specific command
157
- ```
158
-
159
- ### `init`
109
+ **6 Publish test results**
160
110
 
161
111
  ```bash
162
- ado-sync init # creates ado-sync.json
163
- ado-sync init ado-sync.yml # YAML format
164
- ```
165
-
166
- ### `push`
167
-
168
- ```bash
169
- ado-sync push
170
- ado-sync push --dry-run
171
- ado-sync push --tags "@smoke and not @wip"
172
- ado-sync push --config-override testPlan.id=9999
173
-
174
- # AI-generated test steps for code files (Java, C#, Python, JS/TS, Playwright)
175
- ado-sync push --ai-provider heuristic # fast regex-based (no model needed)
176
- ado-sync push --ai-provider local --ai-model ~/.cache/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
177
- ado-sync push --ai-provider ollama --ai-model qwen2.5-coder:7b
178
- ado-sync push --ai-provider openai --ai-key $OPENAI_API_KEY
179
- ado-sync push --ai-provider none # disable AI summary entirely
180
- ado-sync push --ai-context ./docs/ai-context.md # inject domain context into AI prompt
181
- ```
182
-
183
- | Scenario state | Action |
184
- |----------------|--------|
185
- | No ID tag | Creates a new Test Case, writes ID back |
186
- | ID tag, no changes | Skipped |
187
- | ID tag, content changed | Updates the existing Test Case |
188
- | Deleted locally, still in Azure suite | Tagged `ado-sync:removed` in Azure |
189
-
190
- ### `pull`
191
-
192
- ```bash
193
- ado-sync pull
194
- ado-sync pull --dry-run
195
- ado-sync pull --tags "@smoke"
196
- ```
197
-
198
- ### `status`
199
-
200
- ```bash
201
- ado-sync status
202
- ado-sync status --tags "@smoke"
203
- ado-sync status --ai-provider heuristic
204
- ```
205
-
206
- Compares local specs against Azure DevOps and prints a diff — no changes made.
207
-
208
- ### AI auto-summary
209
-
210
- For code-based test types (`java`, `csharp`, `python`, `javascript`, `playwright`, `cypress`, `testcafe`, `detox`, `espresso`, `xcuitest`, `flutter`), ado-sync reads your test function bodies and automatically generates a TC **title**, **description**, and **steps** — so you don't need doc comments on every test.
211
-
212
- > **No setup required to try it.** `ado-sync push` always works, even without a model — it falls back to fast regex-based analysis automatically.
213
-
214
- #### Choose a provider
215
-
216
- | Provider | Quality | Setup |
217
- |---|---|---|
218
- | `local` *(default)* | Good–Excellent | Download a GGUF model file (see below) |
219
- | `heuristic` | Basic | None — works offline, zero dependencies |
220
- | `ollama` | Good–Excellent | Install [Ollama](https://ollama.com) + `ollama pull qwen2.5-coder:7b` |
221
- | `openai` | Excellent | `--ai-key $OPENAI_API_KEY` |
222
- | `anthropic` | Excellent | `--ai-key $ANTHROPIC_API_KEY` — also works with **Claude Code** credentials |
223
- | `openai` + `--ai-url` | Excellent | Any OpenAI-compatible proxy: LiteLLM, Azure OpenAI, vLLM, LM Studio — also works for **GitHub Copilot Enterprise** (Azure OpenAI) |
224
- | `openai` + `--ai-url` (HF) | Good–Excellent | [Hugging Face Inference API](https://huggingface.co/settings/tokens) — free, open-source models |
225
-
226
- > **GitHub Copilot or Claude Code user?** See [docs/advanced.md — Using GitHub Copilot or Claude Code](docs/advanced.md#using-github-copilot-or-claude-code) for a step-by-step guide on which provider to use and how to invoke ado-sync from within your IDE assistant.
227
-
228
- #### Option A — No setup (heuristic, instant)
229
-
230
- ```bash
231
- ado-sync push --ai-provider heuristic
232
- ```
233
-
234
- Uses regex pattern matching. No model download, no internet required. Good for CI pipelines.
235
-
236
- #### Option B — Local LLM (best privacy, no API cost)
237
-
238
- `node-llama-cpp` is bundled — **no extra install needed**. You only need to download a model file once.
239
-
240
- **1. Pick a model size**
241
-
242
- | Model | RAM needed | Quality |
243
- |-------|-----------|---------|
244
- | 1.5B Q4_K_M *(start here)* | ~1.1 GB | Good |
245
- | 7B Q4_K_M | ~4.5 GB | Better |
246
- | 14B Q4_K_M | ~8.5 GB | Excellent |
247
-
248
- **2. Download the model**
249
-
250
- macOS / Linux:
251
- ```bash
252
- mkdir -p ~/.cache/ado-sync/models
253
- curl -L -o ~/.cache/ado-sync/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf \
254
- "https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF/resolve/main/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf"
255
- ```
256
-
257
- Windows (PowerShell):
258
- ```powershell
259
- New-Item -ItemType Directory -Force "$env:LOCALAPPDATA\ado-sync\models"
260
- Invoke-WebRequest `
261
- -Uri "https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF/resolve/main/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf" `
262
- -OutFile "$env:LOCALAPPDATA\ado-sync\models\qwen2.5-coder-1.5b-instruct-q4_k_m.gguf"
263
- ```
264
-
265
- **3. Run push with the model**
266
-
267
- ```bash
268
- # macOS / Linux
269
- ado-sync push --ai-model ~/.cache/ado-sync/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
270
-
271
- # Windows
272
- ado-sync push --ai-model "$env:LOCALAPPDATA\ado-sync\models\qwen2.5-coder-1.5b-instruct-q4_k_m.gguf"
273
- ```
274
-
275
- #### Option C — Ollama (model management UI, easy upgrades)
276
-
277
- ```bash
278
- # 1. Install Ollama from https://ollama.com, then:
279
- ollama pull qwen2.5-coder:7b
280
-
281
- # 2. Push using Ollama
282
- ado-sync push --ai-provider ollama --ai-model qwen2.5-coder:7b
283
- ```
284
-
285
- #### Option D — Cloud AI (OpenAI / Anthropic)
286
-
287
- ```bash
288
- ado-sync push --ai-provider openai --ai-key $OPENAI_API_KEY
289
- ado-sync push --ai-provider anthropic --ai-key $ANTHROPIC_API_KEY
290
- ```
291
-
292
- #### Option E — LiteLLM or any OpenAI-compatible proxy
293
-
294
- [LiteLLM](https://github.com/BerriAI/litellm) is a proxy that routes to 100+ providers (Azure OpenAI, Bedrock, Gemini, Mistral, vLLM, and more) via a single OpenAI-compatible API. Point the `openai` provider at it with `--ai-url`:
295
-
296
- ```bash
297
- ado-sync push \
298
- --ai-provider openai \
299
- --ai-url http://localhost:4000 \
300
- --ai-key $LITELLM_API_KEY \
301
- --ai-model gpt-4o-mini
302
- ```
303
-
304
- > **Hosted LiteLLM + Anthropic models:** Use `--ai-url https://<your-host>/v1` and prefix model names with `anthropic/` (e.g. `anthropic/claude-opus-4-6`). Check your instance's `/v1/models` for registered names.
305
-
306
- The same pattern works for Azure OpenAI, vLLM, LM Studio, and LocalAI — just change `--ai-url` to the endpoint's base URL. See [docs/advanced.md](docs/advanced.md) for a full compatibility table.
307
-
308
- #### Option F — Hugging Face Inference API (free, open-source models)
309
-
310
- [Hugging Face](https://huggingface.co) provides free serverless inference for open-source models via an OpenAI-compatible API:
311
-
312
- ```bash
313
- ado-sync push \
314
- --ai-provider openai \
315
- --ai-url https://router.huggingface.co/v1 \
316
- --ai-key $HF_TOKEN \
317
- --ai-model Qwen/Qwen2.5-Coder-7B-Instruct
318
- ```
319
-
320
- Get a token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) (requires **Inference** permission). Recommended models: `Qwen/Qwen2.5-Coder-7B-Instruct`, `meta-llama/Llama-3.1-8B-Instruct`, `mistralai/Mistral-7B-Instruct-v0.3`.
321
-
322
- #### Option G — Inject domain context into the AI prompt
323
-
324
- Provide a markdown file with domain-specific instructions (glossary, naming conventions, step style guidelines). Its content is injected into every AI prompt before the test code:
325
-
326
- ```bash
327
- ado-sync push --ai-provider anthropic --ai-key $ANTHROPIC_API_KEY \
328
- --ai-context ./docs/ai-context.md
329
- ```
330
-
331
- Or set it once in config (relative path resolved from the config file directory):
332
-
333
- ```json
334
- {
335
- "sync": {
336
- "ai": {
337
- "provider": "anthropic",
338
- "model": "claude-sonnet-4-6",
339
- "apiKey": "$ANTHROPIC_API_KEY",
340
- "contextFile": "./docs/ai-context.md"
341
- }
342
- }
343
- }
344
- ```
345
-
346
- Example `ai-context.md`:
347
-
348
- ```markdown
349
- ## Glossary
350
- - "Checkout" means the 3-step payment flow (cart → shipping → payment)
351
- - "PDP" means Product Detail Page
352
-
353
- ## Step style
354
- - Start every action step with a verb: Click, Enter, Select, Navigate
355
- - Use the customer-facing label for buttons, not the CSS selector
356
- - Preconditions go first; verification steps ("Check: ...") go last
357
- ```
358
-
359
- Context files work with all LLM providers (`local`, `ollama`, `openai`, `anthropic`). The `heuristic` provider ignores the file (no prompt involved).
360
-
361
- #### Option H — Freeze steps in source files (stable across runs)
362
-
363
- By default, AI-generated steps exist only in Azure DevOps. Enable `writebackDocComment` to also write the generated steps as a JSDoc comment directly above each `test()` call in the source file:
364
-
365
- ```json
366
- {
367
- "sync": {
368
- "ai": {
369
- "provider": "anthropic",
370
- "model": "claude-sonnet-4-6",
371
- "apiKey": "$ANTHROPIC_API_KEY",
372
- "writebackDocComment": true
373
- }
374
- }
375
- }
376
- ```
377
-
378
- On the **first push**, AI generates the steps and writes them to the source file:
379
-
380
- ```typescript
381
- /**
382
- * User can log in with valid credentials
383
- * Description: Verifies the login form accepts a correct email/password pair
384
- * 1. Navigate to the login page
385
- * 2. Enter a valid email address
386
- * 3. Enter the matching password
387
- * 4. Click the Sign In button
388
- * 5. Check: The dashboard is displayed
389
- */
390
- test('should log in with valid credentials', async ({ page }) => { ... });
391
- ```
392
-
393
- On **every subsequent push**, the parser reads the JSDoc back — AI is not re-invoked. Steps remain stable even if the test body changes.
394
-
395
- > Only applies to JS/TS frameworks (`javascript`, `playwright`, `puppeteer`, `cypress`, `detox`). Has no effect when `sync.disableLocalChanges` is `true`.
396
-
397
- #### Disable AI entirely
398
-
399
- ```bash
400
- ado-sync push --ai-provider none
401
- ```
402
-
403
- > Tests with existing doc comments (JSDoc / Javadoc / C# XML doc / Python docstring) that already contain steps are **never overwritten**. When `writebackDocComment` is `true`, local JS/TS source files are updated on the first push to freeze the generated steps; subsequent pushes leave the files unchanged.
404
-
405
- ### `publish-test-results`
406
-
407
- ```bash
408
- ado-sync publish-test-results --testResult results/test.trx
409
- ado-sync publish-test-results --testResult results/test.xml --testResultFormat junit --dry-run
112
+ ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
410
113
  ```
411
114
 
412
- See [docs/publish-test-results.md](docs/publish-test-results.md) for full reference.
413
-
414
- ### `--config-override`
415
-
416
- All commands accept `--config-override path=value` to set config values without editing the file:
417
-
418
- ```bash
419
- ado-sync push --config-override testPlan.id=9999
420
- ado-sync push --config-override sync.disableLocalChanges=true
421
- ```
115
+ See [docs/publish-test-results.md](docs/publish-test-results.md) for TRX, NUnit, Playwright JSON, Cucumber JSON, and attachment options.
422
116
 
423
117
  ---
424
118
 
@@ -440,632 +134,15 @@ ado-sync push --config-override sync.disableLocalChanges=true
440
134
 
441
135
  | Topic | Link |
442
136
  |-------|------|
137
+ | **AI agent setup guide** (framework detection → config → validate → push) | [docs/agent-setup.md](docs/agent-setup.md) |
138
+ | CLI reference (all commands and flags) | [docs/cli.md](docs/cli.md) |
443
139
  | Full configuration reference | [docs/configuration.md](docs/configuration.md) |
444
- | Spec file formats (Gherkin, Markdown, C# MSTest, CSV, Excel, JS/TS) | [docs/spec-formats.md](docs/spec-formats.md) |
445
- | Work Item Links (User Story, Bug, etc.) | [docs/spec-formats.md#work-item-links](docs/spec-formats.md#work-item-links) |
446
- | Advanced features (format, state, fieldUpdates, customizations, attachments, CI mode) | [docs/advanced.md](docs/advanced.md) |
140
+ | Spec file formats (Gherkin, Markdown, C#, CSV, Excel, JS/TS) | [docs/spec-formats.md](docs/spec-formats.md) |
141
+ | Framework workflow examples (C#, Java, Python, Playwright, Robot, CI) | [docs/workflows.md](docs/workflows.md) |
142
+ | Work Item Links (User Story, Bug, etc.) | [docs/work-item-links.md](docs/work-item-links.md) |
447
143
  | Publishing test results | [docs/publish-test-results.md](docs/publish-test-results.md) |
144
+ | Advanced features (format, state, fieldUpdates, attachments, AI, CI) | [docs/advanced.md](docs/advanced.md) |
145
+ | MCP Server (use ado-sync from AI agents) | [docs/mcp-server.md](docs/mcp-server.md) |
146
+ | Troubleshooting | [docs/troubleshooting.md](docs/troubleshooting.md) |
448
147
 
449
- ---
450
-
451
- ## Workflow examples
452
-
453
- ### Day-to-day: local changes first
454
-
455
- ```bash
456
- # Edit your .feature or .md files, then push
457
- ado-sync push
458
- ```
459
-
460
- ### Day-to-day: Azure changes first
461
-
462
- ```bash
463
- # Someone edited a Test Case in the Azure DevOps UI
464
- ado-sync pull
465
- ```
466
-
467
- ### C# MSTest / NUnit / SpecFlow: create TCs, run tests, publish results
468
-
469
- ```bash
470
- # 1. Create TCs and write IDs back into .cs / .feature files
471
- ado-sync push --dry-run # preview
472
- ado-sync push # writes [TestProperty("tc","ID")] / [Property("tc","ID")] / @tc:ID
473
-
474
- # 2a. MSTest — TRX contains [TestProperty] values; TC IDs extracted automatically
475
- dotnet test --logger "trx;LogFileName=results.trx"
476
- ado-sync publish-test-results --testResult results/results.trx
477
-
478
- # 2b. NUnit — use native XML logger so [Property("tc","ID")] values are included
479
- dotnet test --logger "nunit3;LogFileName=results.xml"
480
- ado-sync publish-test-results --testResult results/results.xml
481
-
482
- # 2c. SpecFlow — TRX format (SpecFlow writes @tc:ID into TestProperty automatically)
483
- dotnet test --logger "trx;LogFileName=results.trx"
484
- ado-sync publish-test-results --testResult results/results.trx
485
- ```
486
-
487
- Recommended `ado-sync.json` for C# MSTest:
488
-
489
- ```json
490
- {
491
- "orgUrl": "https://dev.azure.com/my-org",
492
- "project": "MyProject",
493
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
494
- "testPlan": { "id": 1234 },
495
- "local": {
496
- "type": "csharp",
497
- "include": ["**/RegressionTests/**/*.cs"],
498
- "exclude": ["**/*BaseTest.cs", "**/*Helper.cs"]
499
- },
500
- "sync": {
501
- "markAutomated": true,
502
- "format": { "useExpectedResult": true }
503
- }
504
- }
505
- ```
506
-
507
- Recommended `ado-sync.json` for C# SpecFlow:
508
-
509
- ```json
510
- {
511
- "orgUrl": "https://dev.azure.com/my-org",
512
- "project": "MyProject",
513
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
514
- "testPlan": { "id": 1234 },
515
- "local": {
516
- "type": "gherkin",
517
- "include": ["Features/**/*.feature"]
518
- },
519
- "sync": {
520
- "tagPrefix": "tc",
521
- "markAutomated": true
522
- }
523
- }
524
- ```
525
-
526
- ### Java JUnit / TestNG: create TCs and publish results
527
-
528
- ```bash
529
- # 1. Create TCs and write IDs back into .java files
530
- ado-sync push --dry-run # preview
531
- ado-sync push # writes // @tc:ID (JUnit 4/TestNG) or @Tag("tc:ID") (JUnit 5)
532
-
533
- # 2. Run tests and generate JUnit XML
534
- mvn test # Surefire writes target/surefire-reports/*.xml by default
535
- # or with Gradle:
536
- ./gradlew test # writes build/test-results/test/*.xml
537
-
538
- # 3. Publish results
539
- ado-sync publish-test-results --testResult target/surefire-reports/TEST-*.xml --testResultFormat junit
540
- ```
541
-
542
- Recommended `ado-sync.json` for Java:
543
-
544
- ```json
545
- {
546
- "orgUrl": "https://dev.azure.com/my-org",
547
- "project": "MyProject",
548
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
549
- "testPlan": { "id": 1234 },
550
- "local": {
551
- "type": "java",
552
- "include": ["**/src/test/**/*.java"],
553
- "exclude": ["**/*BaseTest.java", "**/*Helper.java"]
554
- },
555
- "sync": {
556
- "markAutomated": true
557
- }
558
- }
559
- ```
560
-
561
- ### Python pytest: create TCs and publish results
562
-
563
- ```bash
564
- # 1. Create TCs and write IDs back into .py files
565
- ado-sync push --dry-run # preview
566
- ado-sync push # writes @pytest.mark.tc(ID) above each test function
567
-
568
- # 2. Run tests and generate JUnit XML
569
- pytest --junitxml=results/junit.xml
570
-
571
- # 3. Publish results
572
- ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
573
- ```
574
-
575
- By default, TC linking uses `AutomatedTestName` matching (requires `sync.markAutomated: true`).
576
-
577
- **Optional — embed TC IDs into JUnit XML** for direct linking (more reliable, works even if the class/method is renamed):
578
-
579
- Add this to `conftest.py`:
580
-
581
- ```python
582
- # conftest.py
583
- def pytest_runtest_makereport(item, call):
584
- """Write @pytest.mark.tc(N) as a JUnit XML property for ado-sync to pick up."""
585
- for marker in item.iter_markers("tc"):
586
- if marker.args:
587
- item.user_properties.append(("tc", str(marker.args[0])))
588
- ```
589
-
590
- pytest will then write each TC ID into the JUnit XML as `<property name="tc" value="N"/>`, and ado-sync will link the result directly to that Test Case — no AutomatedTestName matching needed.
591
-
592
- Recommended `ado-sync.json` for Python:
593
-
594
- ```json
595
- {
596
- "orgUrl": "https://dev.azure.com/my-org",
597
- "project": "MyProject",
598
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
599
- "testPlan": { "id": 1234 },
600
- "local": {
601
- "type": "python",
602
- "include": ["tests/**/*.py"],
603
- "exclude": ["tests/conftest.py", "tests/**/helpers.py"]
604
- },
605
- "sync": {
606
- "markAutomated": true
607
- }
608
- }
609
- ```
610
-
611
- ### JavaScript / TypeScript (Jest, Jasmine, WebdriverIO): create TCs
612
-
613
- ```bash
614
- # 1. Create TCs and write IDs back into .js / .ts files
615
- ado-sync push --dry-run # preview
616
- ado-sync push # writes // @tc:ID above each it() / test()
617
-
618
- # 2. Run tests and generate JUnit XML (Jest example)
619
- npx jest --reporters=default --reporters=jest-junit
620
- # JEST_JUNIT_OUTPUT_DIR=results JEST_JUNIT_OUTPUT_NAME=junit.xml
621
-
622
- # 3. Publish results
623
- ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
624
- ```
625
-
626
- ### Playwright: create TCs and publish results with screenshots
627
-
628
- ```bash
629
- # 1. Create TCs and write IDs back into .ts spec files
630
- ado-sync push --dry-run # preview
631
- ado-sync push # writes annotation: { type: 'tc', description: 'ID' } into test options
632
-
633
- # 2. Run Playwright with JSON reporter
634
- # playwright.config.ts: reporter: [['json', { outputFile: 'results/playwright.json' }]]
635
- npx playwright test
636
-
637
- # 3. Publish — TC IDs from native test.annotations; screenshots/videos from attachments[]
638
- ado-sync publish-test-results --testResult results/playwright.json
639
- ```
640
-
641
- Recommended `ado-sync.json` for Playwright:
642
-
643
- ```json
644
- {
645
- "orgUrl": "https://dev.azure.com/my-org",
646
- "project": "MyProject",
647
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
648
- "testPlan": { "id": 1234 },
649
- "local": {
650
- "type": "playwright",
651
- "include": ["tests/**/*.spec.ts"],
652
- "exclude": ["**/*.helper.ts"]
653
- },
654
- "sync": {
655
- "markAutomated": true
656
- }
657
- }
658
- ```
659
-
660
- Recommended `ado-sync.json` for Jest/Jasmine/WebdriverIO:
661
-
662
- ```json
663
- {
664
- "orgUrl": "https://dev.azure.com/my-org",
665
- "project": "MyProject",
666
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
667
- "testPlan": { "id": 1234 },
668
- "local": {
669
- "type": "javascript",
670
- "include": ["src/**/*.spec.ts", "tests/**/*.test.js"],
671
- "exclude": ["**/*.helper.ts"]
672
- },
673
- "sync": {
674
- "markAutomated": true
675
- }
676
- }
677
- ```
678
-
679
- ### Detox (React Native): create TCs and push
680
-
681
- ```bash
682
- # 1. Create TCs and write IDs back into .ts files
683
- ado-sync push --dry-run # preview
684
- ado-sync push # writes // @tc:ID above each it() / test()
685
-
686
- # 2. Run Detox tests (Jest runner)
687
- npx detox test --configuration ios.sim.release
688
-
689
- # 3. Publish results (Jest JUnit reporter)
690
- ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
691
- ```
692
-
693
- Recommended `ado-sync.json` for Detox:
694
-
695
- ```json
696
- {
697
- "orgUrl": "https://dev.azure.com/my-org",
698
- "project": "MyProject",
699
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
700
- "testPlan": { "id": 1234 },
701
- "local": {
702
- "type": "detox",
703
- "include": ["e2e/**/*.test.ts"]
704
- },
705
- "sync": { "markAutomated": true }
706
- }
707
- ```
708
-
709
- ### Espresso (Android): create TCs and push
710
-
711
- ```bash
712
- # 1. Create TCs and write IDs back into .java / .kt files
713
- ado-sync push --dry-run # preview
714
- ado-sync push # writes // @tc:ID above @Test
715
-
716
- # 2. Run instrumented tests and generate JUnit XML
717
- ./gradlew connectedAndroidTest
718
- # XML output: app/build/outputs/androidTest-results/connected/TEST-*.xml
719
-
720
- # 3. Publish results
721
- ado-sync publish-test-results \
722
- --testResult "app/build/outputs/androidTest-results/connected/TEST-*.xml" \
723
- --testResultFormat junit
724
- ```
725
-
726
- Recommended `ado-sync.json` for Espresso:
727
-
728
- ```json
729
- {
730
- "orgUrl": "https://dev.azure.com/my-org",
731
- "project": "MyProject",
732
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
733
- "testPlan": { "id": 1234 },
734
- "local": {
735
- "type": "espresso",
736
- "include": ["app/src/androidTest/**/*.java", "app/src/androidTest/**/*.kt"],
737
- "exclude": ["**/*BaseTest.java"]
738
- },
739
- "sync": { "markAutomated": true }
740
- }
741
- ```
742
-
743
- ### XCUITest (iOS): create TCs and push
744
-
745
- ```bash
746
- # 1. Create TCs and write IDs back into .swift files
747
- ado-sync push --dry-run # preview
748
- ado-sync push # writes // @tc:ID above func test*()
749
-
750
- # 2. Run XCUITest and export JUnit XML
751
- xcodebuild test \
752
- -project MyApp.xcodeproj \
753
- -scheme MyApp \
754
- -destination 'platform=iOS Simulator,name=iPhone 15' \
755
- -resultBundlePath TestResults.xcresult
756
- xcrun xcresulttool get --path TestResults.xcresult --format junit > results/junit.xml
757
-
758
- # 3. Publish results
759
- ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
760
- ```
761
-
762
- Recommended `ado-sync.json` for XCUITest:
763
-
764
- ```json
765
- {
766
- "orgUrl": "https://dev.azure.com/my-org",
767
- "project": "MyProject",
768
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
769
- "testPlan": { "id": 1234 },
770
- "local": {
771
- "type": "xcuitest",
772
- "include": ["UITests/**/*.swift"],
773
- "exclude": ["UITests/**/*Helper.swift", "UITests/**/*Base.swift"]
774
- },
775
- "sync": { "markAutomated": true }
776
- }
777
- ```
778
-
779
- ### Flutter: create TCs and push
780
-
781
- ```bash
782
- # 1. Create TCs and write IDs back into _test.dart files
783
- ado-sync push --dry-run # preview
784
- ado-sync push # writes // @tc:ID above testWidgets() / test()
785
-
786
- # 2. Run Flutter tests with machine-readable output
787
- flutter test --machine > results/flutter_test.jsonl
788
- # Or generate JUnit XML with the flutter_test_junit package:
789
- flutter test --reporter junit > results/junit.xml
790
-
791
- # 3. Publish results
792
- ado-sync publish-test-results --testResult results/junit.xml --testResultFormat junit
793
- ```
794
-
795
- Recommended `ado-sync.json` for Flutter:
796
-
797
- ```json
798
- {
799
- "orgUrl": "https://dev.azure.com/my-org",
800
- "project": "MyProject",
801
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
802
- "testPlan": { "id": 1234 },
803
- "local": {
804
- "type": "flutter",
805
- "include": ["test/**/*_test.dart", "integration_test/**/*_test.dart"]
806
- },
807
- "sync": { "markAutomated": true }
808
- }
809
- ```
810
-
811
- ### Robot Framework: create TCs and publish results
812
-
813
- ```bash
814
- # 1. Create TCs and write IDs back into .robot files
815
- ado-sync push --dry-run # preview
816
- ado-sync push # inserts/updates tc:ID in [Tags] of each test case
817
-
818
- # 2. Run Robot Framework tests (generates output.xml)
819
- robot --outputdir results tests/
820
-
821
- # 3. Publish results
822
- ado-sync publish-test-results --testResult results/output.xml
823
- ```
824
-
825
- Recommended `ado-sync.json` for Robot Framework:
826
-
827
- ```json
828
- {
829
- "orgUrl": "https://dev.azure.com/my-org",
830
- "project": "MyProject",
831
- "auth": { "type": "pat", "token": "$AZURE_DEVOPS_TOKEN" },
832
- "testPlan": { "id": 1234 },
833
- "local": {
834
- "type": "robot",
835
- "include": ["tests/**/*.robot"],
836
- "exclude": ["tests/resources/**"]
837
- },
838
- "sync": {
839
- "markAutomated": true
840
- }
841
- }
842
- ```
843
-
844
- TC IDs are read directly from the `tc:N` tag in `<tags>` in `output.xml` — no extra config needed.
845
-
846
- ---
847
-
848
- ### CI pipeline
849
-
850
- ```yaml
851
- # GitHub Actions
852
- - name: Sync test cases to Azure DevOps
853
- run: ado-sync push --config-override sync.disableLocalChanges=true
854
- env:
855
- AZURE_DEVOPS_TOKEN: ${{ secrets.AZURE_DEVOPS_TOKEN }}
856
-
857
- - name: Sync test cases to Azure DevOps (with AI summary)
858
- run: ado-sync push --ai-provider heuristic --config-override sync.disableLocalChanges=true
859
- env:
860
- AZURE_DEVOPS_TOKEN: ${{ secrets.AZURE_DEVOPS_TOKEN }}
861
-
862
- - name: Publish test results
863
- run: ado-sync publish-test-results --testResult results/test.trx
864
- env:
865
- AZURE_DEVOPS_TOKEN: ${{ secrets.AZURE_DEVOPS_TOKEN }}
866
- ```
867
-
868
- ### Check for drift before a PR
869
-
870
- ```bash
871
- ado-sync status
872
- ```
873
-
874
- ---
875
-
876
- ## Work Item Links
877
-
878
- Link each Test Case to related Azure DevOps work items (User Stories, Bugs, etc.) automatically on every push.
879
-
880
- ### Configure `sync.links`
881
-
882
- ```json
883
- {
884
- "sync": {
885
- "links": [
886
- {
887
- "prefix": "story",
888
- "relationship": "Microsoft.VSTS.Common.TestedBy-Reverse",
889
- "workItemType": "User Story"
890
- },
891
- {
892
- "prefix": "bug",
893
- "relationship": "System.LinkTypes.Related",
894
- "workItemType": "Bug"
895
- }
896
- ]
897
- }
898
- }
899
- ```
900
-
901
- | Field | Description |
902
- |-------|-------------|
903
- | `prefix` | The tag prefix used in your spec files (e.g. `story` → `@story:555`) |
904
- | `relationship` | ADO relation type (see common values below) |
905
- | `workItemType` | Optional — used in log output only |
906
-
907
- **Common relationship values:**
908
-
909
- | Relationship | Meaning |
910
- |---|---|
911
- | `Microsoft.VSTS.Common.TestedBy-Reverse` | Test Case "Tested By" ↔ User Story |
912
- | `System.LinkTypes.Related` | Simple "Related" link |
913
- | `System.LinkTypes.Dependency-Forward` | "Successor" (this item depends on) |
914
- | `System.LinkTypes.Hierarchy-Reverse` | "Parent" link |
915
-
916
- ### Tag your tests
917
-
918
- **Gherkin (`.feature`):**
919
- ```gherkin
920
- # @story:555 @bug:789
921
- Scenario: User can log in
922
- Given I am on the login page
923
- ```
924
-
925
- **JavaScript / TypeScript (Jest, Playwright, Cypress, TestCafe, Puppeteer):**
926
- ```typescript
927
- // @story:555
928
- // @bug:789
929
- test('user can log in', async ({ page }) => { ... });
930
- ```
931
-
932
- **Markdown (`.md`):**
933
- ```markdown
934
- ### User can log in @story:555 @bug:789
935
-
936
- 1. Navigate to the login page
937
- 2. Check: Login form is visible
938
- ```
939
-
940
- **Python (pytest):**
941
- ```python
942
- # @story:555 @bug:789
943
- def test_user_can_log_in():
944
- ...
945
- ```
946
-
947
- **C# / Java / Espresso:** Add `// @story:555` in the comment block immediately above the `[TestMethod]` / `@Test` line.
948
-
949
- **Robot Framework (`.robot`):**
950
- ```robot
951
- *** Test Cases ***
952
- My Login Test
953
- [Tags] tc:12345 story:555 bug:789
954
- Open Browser ${URL}
955
- Login As user pass
956
- ```
957
-
958
- **Swift (XCUITest):**
959
- ```swift
960
- // @story:555
961
- // @bug:789
962
- func testUserCanLogin() { ... }
963
- ```
964
-
965
- **Dart (Flutter):**
966
- ```dart
967
- // @story:555
968
- // @bug:789
969
- testWidgets('user can log in', (WidgetTester tester) async { ... });
970
- ```
971
-
972
- **Detox / React Native:**
973
- ```typescript
974
- // @story:555
975
- // @bug:789
976
- it('user can log in', async () => { ... });
977
- ```
978
-
979
- ### How it works
980
-
981
- - On each `push`, ado-sync reads the `@story:N` / `@bug:N` tags from the spec file.
982
- - **New links** found in the file are added to the Test Case in Azure DevOps.
983
- - **Stale links** (present in Azure but no longer tagged locally) are removed automatically.
984
- - The sync is non-destructive for links not covered by a configured prefix — only the prefixes listed in `sync.links` are managed.
985
-
986
- ---
987
-
988
- ## Environment variables
989
-
990
- | Variable | Description |
991
- |----------|-------------|
992
- | `AZURE_DEVOPS_TOKEN` | PAT or access token. Reference in config with `"$AZURE_DEVOPS_TOKEN"`. |
993
- | Any name | Any env var — set `auth.token` to `"$MY_VAR_NAME"`. |
994
-
995
- A `.env` file in the working directory is loaded automatically.
996
-
997
- ---
998
-
999
- ## Troubleshooting
1000
-
1001
- **`No config file found`**
1002
- Run `ado-sync init` or pass `-c path/to/config.json`.
1003
-
1004
- **`Environment variable 'X' is not set`**
1005
- Your config references `$X` in `auth.token` but the variable is not exported. Run `export X=...` or add it to a `.env` file.
1006
-
1007
- **Test Case created but ID not written back**
1008
- Check that the file is writable, or that `sync.disableLocalChanges` is not `true`.
1009
-
1010
- **`Test case #N not found in Azure DevOps`**
1011
- The test case was deleted in Azure. Remove the ID tag from the local file to recreate it, or restore the test case in Azure.
1012
-
1013
- **`Failed to parse <file>`**
1014
- Gherkin syntax error. Run `npx cucumber-js --dry-run` to identify the problem line.
1015
-
1016
- **Changes not detected on push**
1017
- The comparison uses title + steps + description. Touch any step to force an update, or reset the cache by deleting `.ado-sync-state.json`.
1018
-
1019
- **Conflict detected unexpectedly**
1020
- Delete `.ado-sync-state.json` to reset the cache. The next push re-populates it from Azure.
1021
-
1022
- **CSV/Excel IDs not written back**
1023
- Ensure the file is not open in another application and that `sync.disableLocalChanges` is not `true`. If a TC was deleted from Azure and re-created on push, the old ID in column A is now replaced with the new ID automatically.
1024
-
1025
- **Excel file not parsed / `No worksheet found`**
1026
- ado-sync searches for the first worksheet by reading `xl/_rels/workbook.xml.rels` from the xlsx ZIP, falling back to common names (`sheet.xml`, `sheet1.xml`). Non-standard sheet names and multi-sheet workbooks are handled automatically. If parsing still fails, re-export from Azure DevOps.
1027
-
1028
- **Pull has no effect on CSV files**
1029
- CSV pull is now supported — `ado-sync pull` updates the Title and step rows in CSV files to match the current Azure DevOps Test Case. Run `ado-sync pull --dry-run` first to preview changes.
1030
-
1031
- **Pull has no effect on Excel files**
1032
- Excel (xlsx) pull is not yet supported — only push. Use CSV export instead if bidirectional sync is needed, or pull the changes manually and re-export.
1033
-
1034
- **C# categories show as constant names instead of values**
1035
- ado-sync resolves `const string` declarations in the same file. Constants defined in a base class are not resolved — use string literals in `[TestCategory("...")]` for reliable tagging.
1036
-
1037
- **C# test methods not detected**
1038
- Ensure the method has `[TestMethod]` on its own line. Nested classes or abstract base methods are not parsed. Add base class files to `local.exclude`.
1039
-
1040
- **TRX results not linked to Test Cases**
1041
- For MSTest, TC IDs are read directly from `[TestProperty("tc","ID")]` embedded in the TRX — no further config needed. For NUnit, use `--logger "nunit3;LogFileName=results.xml"` (native XML format) instead of TRX so `[Property("tc","ID")]` values are included. If neither is available, set `sync.markAutomated: true` and rely on `AutomatedTestName` FQMN matching.
1042
-
1043
- **Java test methods not detected**
1044
- Ensure each test method has a `@Test` annotation. Abstract base methods and methods with only `@Before`/`@After` are not parsed. Add base class files to `local.exclude`.
1045
-
1046
- **Java ID not written back (JUnit 5)**
1047
- ado-sync writes `@Tag("tc:ID")` above the `@Test` annotation. Ensure the file is writable. The `@Tag` import (`org.junit.jupiter.api.Tag`) must already be present or will be added automatically.
1048
-
1049
- **Python test functions not detected**
1050
- ado-sync detects functions starting with `test_` at module level and inside classes. Ensure functions follow the `def test_*()` convention. Abstract base test methods should be excluded from `local.include`.
1051
-
1052
- **Python ID not written back**
1053
- ado-sync writes `@pytest.mark.tc(ID)` directly above the `def test_*` line. Ensure `pytest` is in your test environment. The `pytest` import is not required in the file itself — the mark is a decorator, not a function call.
1054
-
1055
- **JavaScript/TypeScript tests not detected**
1056
- ado-sync detects `it()`, `test()`, `xit()`, `xtest()`, and `.only`/`.skip`/`.concurrent` variants. Tests with dynamic titles (template literals or computed values) are skipped — use string literals for the test title.
1057
-
1058
- **JavaScript ID not written back**
1059
- ado-sync inserts `// @tc:ID` immediately above the `it()`/`test()` line. There must be no blank line between the comment and the test function call.
1060
-
1061
- **`publish-test-results` — "TestPointId, testCaseId must be specified for planned test results"**
1062
- This error means the Test Run was created as a "planned" run (tied to a test plan), which requires test point IDs for each result. ado-sync creates standalone automated runs — do not pass `plan.id` in `runModel`. This is handled automatically; if you see this error, ensure you are on the latest version.
1063
-
1064
- **TRX screenshots / `<ResultFiles>` not attached**
1065
- In TRX format, `<ResultFiles>` is a child of `<Output>`, not a direct child of `<UnitTestResult>`. Make sure `TestContext.AddResultFile("path/to/screenshot.png")` is called in your test code. ado-sync reads from the correct nested path automatically.
1066
-
1067
- **Attachment paths resolve incorrectly**
1068
- Attachment paths embedded in result files (TRX `<ResultFiles>`, NUnit `<filePath>`, JUnit `[[ATTACHMENT|path]]`, Playwright `attachments[].path`) are resolved **relative to the result file's directory**, not the working directory. Keep result files and screenshots in the same output folder hierarchy as your test runner produces them.
1069
-
1070
- **"Invalid AttachmentType specified" from Azure DevOps API**
1071
- Azure DevOps only accepts `GeneralAttachment` and `ConsoleLog` as attachment types. Screenshot and video files are uploaded as `GeneralAttachment` automatically — no special type is needed.
148
+ > **AI models / LLM crawlers:** [`llms.txt`](llms.txt) contains a flat single-file summary of this entire project — framework type mappings, config schema, CLI flags, ID writeback formats, and the full doc index. Read that first.