ado-sync 0.1.59 → 0.1.60

package/docs/capability-roadmap.md

@@ -0,0 +1,280 @@
+# Capability Roadmap
+
+This roadmap turns the recent benchmark review into a product-owned plan for expanding ado-sync without copying another tool's terminology or structure.
+
+The goal is straightforward: make large-scale synchronization more predictable, make result publishing more trustworthy, and give teams a cleaner path from single-suite setups to portfolio-scale usage.
+
+---
+
+## Current strengths
+
+ado-sync already delivers a solid base:
+
+- bidirectional push and pull for local specs and Azure DevOps test cases
+- broad parser coverage across Gherkin, source-code, tabular, and result-file formats
+- Azure work item linking, state updates, attachments, and field updates
+- test result publishing with multiple input formats
+- higher-level workflows such as stale detection, coverage, trend analysis, AI-assisted generation, and editor integration
+
+The roadmap below focuses on gaps that limit scale, predictability, or user trust.
+
+---
+
+## Phase 1: Fix correctness gaps first
+
+### Why this comes first
+
+Some configuration surfaces are documented and typed, but not fully honored at runtime. That weakens confidence in the tool even when the rest of the workflow is sound.
+
+### Priority work
+
+1. Align `publish-test-results` docs, types, and runtime behavior.
+2. Either implement or remove unsupported settings from public docs and config types.
+3. Add focused regression tests around run creation and configuration lookup.
+
+### Main files
+
+- `src/sync/publish-results.ts`
+- `src/types.ts`
+- `docs/publish-test-results.md`
+- `src/__tests__/regressions.test.ts`
+
+### Concrete targets
+
+- honor `publishTestResults.testConfiguration.name` by resolving configuration names to IDs before run creation
+- support explicit test suite targeting for result publication when configured
+- implement deterministic flaky-outcome policies instead of exposing them as no-op configuration
+- document only behavior that is actually live
+
+### Exit criteria
+
+- every documented `publishTestResults` setting is either implemented end-to-end or removed
+- dry-run and live-run behavior produce matching routing decisions
+- regression coverage exists for name-based configuration selection and flaky handling
+
+---
+
+## Phase 2: Introduce scoped synchronization
+
+### Problem to solve
+
+Today the model is centered on a plan, optional suite mapping, and local filters. That works for smaller installations, but it does not give teams a strong way to define which remote inventory a configuration owns.
+
+### Product direction
+
+Add a first-class sync target model that tells ado-sync exactly which remote set belongs to a configuration. This should be original to ado-sync, not a copy of anyone else's schema.
+
+### Proposed design direction
+
+Introduce a new top-level concept such as `syncTarget` or `ownership` with selectable modes:
+
+- `tagged` for test cases carrying a generated ownership tag
+- `suite` for test cases managed through a designated suite root
+- `query` for test cases matched by an Azure DevOps query or rule set
+
+This model should work alongside `configurationKey`, not replace it.
+
+### Main files
+
+- `src/types.ts`
+- `src/config.ts`
+- `src/sync/engine.ts`
+- `src/cli.ts`
+- `docs/configuration.md`
+
+### Exit criteria
+
+- push, pull, stale detection, and status all use the same ownership boundary
+- removing a local scenario does not risk touching unrelated remote items
+- multiple configurations can target the same project without collisions
+
+---
+
+## Phase 3: Build a real hierarchy engine
+
+### Problem to solve
+
+Current suite support is useful but narrow. It covers basic mapping and conditional routing, but not richer hierarchy construction, cleanup rules, or multiple parallel views of the same inventory.
+
+### Product direction
+
+Create named hierarchy definitions that can materialize test suites from local structure, tags, or explicit routing rules.
+
+### Proposed capabilities
+
+- folder-based hierarchy generation
+- file-based hierarchy generation
+- tag-driven hierarchy generation
+- explicit level rules for multi-level suite trees
+- cleanup policies for stale nodes and stale memberships
+- optional multiple hierarchy outputs from the same local corpus
+
+### Main files
+
+- `src/azure/test-cases.ts`
+- `src/sync/engine.ts`
+- `src/types.ts`
+- `docs/advanced.md`
+- `docs/configuration.md`
+
+### Exit criteria
+
+- hierarchy definitions are declarative and testable
+- hierarchy sync can add, move, and prune memberships safely
+- teams can maintain more than one suite view without duplicating specs
+
+---
+
+## Phase 4: Expand the command surface for scale workflows
+
+### Problem to solve
+
+The current command line is clean, but it is missing a few control points that matter in large repositories and CI pipelines.
+
+### Priority work
+
+1. Add source-file filters for push and pull.
+2. Add operation modes such as create-only, link-only, and update-only where applicable.
+3. Add consistent diagnostic controls across commands.
+4. Allow temporary auth and routing overrides in a controlled way.
+
+### Main files
+
+- `src/cli.ts`
+- `src/config.ts`
+- `src/sync/engine.ts`
+- `docs/cli.md`
+
+### Exit criteria
+
+- operators can target a small slice of a large repo without changing config files
+- CI jobs can run low-risk creation-only or link-only flows
+- troubleshooting output is consistent across push, pull, status, and result publication
+
+---
+
+## Phase 5: Modernize configuration ergonomics
+
+### Problem to solve
+
+The configuration model is powerful enough for a single repository, but it becomes awkward when shared across many teams or environments.
+
+### Product direction
+
+Make configuration layering easier, safer, and more portable.
+
+### Priority work
+
+1. Support comment-tolerant JSON configuration parsing.
+2. Add optional parent-config discovery through ancestor folders.
+3. Add personal config overlays for credentials and machine-local overrides.
+4. Add reusable remote profiles so org-level connection settings do not need to be copied into every repository.
+
+### Main files
+
+- `src/config.ts`
+- `src/types.ts`
+- `docs/configuration.md`
+- `src/cli.ts`
+
+### Exit criteria
+
+- teams can share base config safely without checking credentials into the repo
+- repository config stays small even when many projects use the same Azure DevOps org
+- the override and inheritance rules are explicit and easy to debug
+
+---
+
+## Phase 6: Deepen automation and result publishing
+
+### Problem to solve
+
+The core publishing flow works, but automation metadata and run routing are still simpler than what enterprise teams usually need.
+
+### Product direction
+
+Expand the publishing model so it can express how tests were executed, where runs should land, and how automated metadata should be maintained.
+
+### Priority work
+
+1. Support automation classification rules instead of a single coarse toggle.
+2. Resolve named test configurations at runtime.
+3. Allow controlled publication to more than one destination suite when desired.
+4. Add stronger matching and fallback rules for automated result association.
+5. Expand attachment and comment policies for passing, flaky, and retried tests.
+
+### Main files
+
+- `src/sync/publish-results.ts`
+- `src/azure/test-runs.ts`
+- `src/azure/test-cases.ts`
+- `src/types.ts`
+- `docs/publish-test-results.md`
+
+### Exit criteria
+
+- automation metadata is predictable across push and publish flows
+- result publication can be routed by configuration rather than ad hoc flags alone
+- retry-heavy pipelines produce consistent final outcomes
+
+---
+
+## Phase 7: Add extension points instead of hard-coding every variant
+
+### Problem to solve
+
+ado-sync already supports many ecosystems, but every new integration currently tends to require first-party changes.
+
+### Product direction
+
+Introduce a narrow extension model for parser registration, result ingestion, custom routing, and metadata transforms.
+
+### Candidate extension seams
+
+- local spec discovery
+- parser registration
+- result-file ingestion
+- test-case field transformation
+- run-routing policies
+
+### Main files
+
+- `src/types.ts`
+- `src/config.ts`
+- parser modules under `src/parsers/`
+- `src/sync/engine.ts`
+- `src/sync/publish-results.ts`
+
+### Exit criteria
+
+- new ecosystems can be added without invasive changes to the core sync engine
+- experimental integrations can ship behind stable interfaces
+- core maintenance load stays bounded as format coverage expands
+
+---
+
+## Recommended implementation order
+
+1. Phase 1: correctness and documentation trust
+2. Phase 2: scoped synchronization model
+3. Phase 4: command-surface improvements needed to operate the new scope model
+4. Phase 3: hierarchy engine
+5. Phase 6: deeper automation and result-routing behavior
+6. Phase 5: configuration ergonomics and shared profiles
+7. Phase 7: extension model
+
+This order reduces user-facing confusion first, then hardens ownership boundaries before adding more advanced routing and hierarchy features.
+
+---
+
+## Near-term backlog candidates
+
+If the goal is to start implementation immediately, the highest-value first batch is:
+
+1. fix `publish-test-results` contract drift
+2. add named test configuration lookup
+3. design `syncTarget` ownership schema in `src/types.ts`
+4. wire ownership filtering into `status`, `push`, `pull`, and `stale`
+5. add `--source-file`, `--create-only`, and `--link-only` style command controls
+
+That batch improves trust, reduces operational risk, and lays the foundation for the larger roadmap.

package/docs/publish-test-results.md

@@ -24,6 +24,13 @@ ado-sync publish-test-results --testResult results/test.trx --dry-run
 ado-sync publish-test-results \
   --testResult results/test.trx \
   --buildId 12345
+
+# Publish to a specific planned suite/configuration
+ado-sync publish-test-results \
+  --testResult results/test.trx \
+  --testPlan "Smoke Plan" \
+  --testSuite "BDD" \
+  --testConfiguration "Windows 10"
 ```
 
 ### Options
@@ -35,6 +42,9 @@ ado-sync publish-test-results \
 | `--attachmentsFolder <path>` | Folder to scan for screenshots/videos/logs to attach to test results. |
 | `--runName <name>` | Name for the Test Run in Azure DevOps. Defaults to `ado-sync <ISO timestamp>`. |
 | `--buildId <id>` | Build ID to associate with the Test Run. |
+| `--testConfiguration <nameOrId>` | Azure test configuration name or numeric ID for the published run. |
+| `--testSuite <nameOrId>` | Azure test suite name or numeric ID for planned run publication. |
+| `--testPlan <nameOrId>` | Azure test plan name or numeric ID used with `--testSuite`. |
 | `--dry-run` | Parse results and print summary without creating a run in Azure. |
 | `--create-issues-on-failure` | File GitHub Issues or ADO Bugs for each failed test after publishing. |
 | `--issue-provider <github\|ado>` | Issue provider. Default: `github`. |
@@ -772,12 +782,17 @@ Results can also be configured in the config file under `publishTestResults`:
 | `flakyTestOutcome` | How to handle flaky tests: `"lastAttemptOutcome"` *(default)* · `"firstAttemptOutcome"` · `"worstOutcome"`. |
 | `testConfiguration.name` | Name of the Azure test configuration to associate. |
 | `testConfiguration.id` | ID of the Azure test configuration. |
+| `testSuite.name` | Name of the Azure test suite to publish against. Requires every result to resolve to a test case ID. |
+| `testSuite.id` | ID of the Azure test suite to publish against. |
+| `testSuite.testPlan` | Optional test plan name or ID used when resolving the target suite. Defaults to `testPlan.id` from the main config. |
 | `testRunSettings.name` | Name for the Test Run. |
 | `testRunSettings.comment` | Comment attached to the Test Run. |
 | `testRunSettings.runType` | `"Automated"` *(default)* · `"Manual"`. |
 | `testResultSettings.comment` | Comment applied to every test result. |
 | `publishAttachmentsForPassingTests` | `"none"` *(default)* · `"files"` · `"all"`. |
 
+When `testSuite` is configured, ado-sync creates a planned run and binds each published result to the matching test point in that suite. If a suite contains multiple configurations for the same test case, set `testConfiguration.id` or `testConfiguration.name` so the target point can be resolved unambiguously.
+
 ---
 
 ## Creating issues on failure

package/mkdocs.yml

@@ -15,6 +15,7 @@ nav:
   - Welcome: index.md
   - CLI: cli.md
   - Configuration: configuration.md
+  - Capability Roadmap: capability-roadmap.md
   - Agent Setup: agent-setup.md
   - MCP Server: mcp-server.md
   - Work Item Links: work-item-links.md

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ado-sync",
-  "version": "0.1.59",
+  "version": "0.1.60",
   "description": "Bidirectional sync between local test specs (Cucumber/Markdown) and Azure DevOps Test Cases",
   "bin": {
     "ado-sync": "./dist/cli.js",