trxa-tool 0.1.1 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (120) hide show
  1. package/MANUAL.md +376 -368
  2. package/README.md +75 -51
  3. package/SETUP.md +401 -0
  4. package/STUDIO.md +417 -309
  5. package/dist/fixture-schema.d.ts +45 -5
  6. package/dist/fixture-schema.js +97 -13
  7. package/dist/fixture-store.d.ts +37 -0
  8. package/dist/fixture-store.js +75 -0
  9. package/dist/index.js +76 -40
  10. package/dist/mapping.js +4 -1
  11. package/dist/mcp-bridge.d.ts +52 -0
  12. package/dist/mcp-bridge.js +290 -0
  13. package/dist/planrun-schema.d.ts +14 -6
  14. package/dist/planrun-schema.js +31 -7
  15. package/dist/planrun.d.ts +1 -1
  16. package/dist/planrun.js +2 -2
  17. package/dist/report-schema.d.ts +10 -1
  18. package/dist/report-schema.js +31 -5
  19. package/dist/report.d.ts +4 -1
  20. package/dist/report.js +9 -2
  21. package/dist/suite-runner.js +8 -2
  22. package/media/studio/fixtures-test-cases.png +0 -0
  23. package/media/studio/plans-editor.png +0 -0
  24. package/media/studio/recordings-create-test-case.png +0 -0
  25. package/media/studio/run-dialog.png +0 -0
  26. package/media/studio/settings-agent-runs.png +0 -0
  27. package/media/studio/settings-working-folder.png +0 -0
  28. package/media/studio/studio-recordings.png +0 -0
  29. package/media/studio/suites-canvas.png +0 -0
  30. package/media/studio/suites-inspector.png +0 -0
  31. package/media/studio/testcases-expectations.png +0 -0
  32. package/media/studio/testcases-outputs.png +0 -0
  33. package/media/studio/testrun-details.png +0 -0
  34. package/media/studio/testruns-list.png +0 -0
  35. package/package.json +6 -2
  36. package/studio/client-dist/assets/CreatePanel-C5bb5OIs.js +6 -0
  37. package/studio/client-dist/assets/FactTile-G1pURLys.js +1 -0
  38. package/studio/client-dist/assets/FieldLabel-DPmIEQ08.js +1 -0
  39. package/studio/client-dist/assets/FixtureNamePicker-137AAgwZ.js +1 -0
  40. package/studio/client-dist/assets/Fixtures-D0t-soph.js +1 -0
  41. package/studio/client-dist/assets/GeneralSection-CWkTeSX2.js +1 -0
  42. package/studio/client-dist/assets/ListPaneHeader-DXKqX36N.js +1 -0
  43. package/studio/client-dist/assets/Plans-C70evkGc.js +1 -0
  44. package/studio/client-dist/assets/Recordings-BT0dyVLs.js +1 -0
  45. package/studio/client-dist/assets/Settings-k-J3u5lH.js +1 -0
  46. package/studio/client-dist/assets/Suites-DLioOiRN.css +1 -0
  47. package/studio/client-dist/assets/Suites-DQhhMxjT.js +7 -0
  48. package/studio/client-dist/assets/TestCases-BdiopVGI.js +1 -0
  49. package/studio/client-dist/assets/TestRuns-CylAe-3t.js +9 -0
  50. package/studio/client-dist/assets/api-DJZ703_F.js +24 -0
  51. package/studio/client-dist/assets/bucket-18-viuwGcaj.js +1 -0
  52. package/studio/client-dist/assets/fixtureName-CTZT_jKu.js +1 -0
  53. package/studio/client-dist/assets/ibm-plex-mono-cyrillic-400-normal-BSMlKf0J.woff2 +0 -0
  54. package/studio/client-dist/assets/ibm-plex-mono-cyrillic-400-normal-CEL4l2ZJ.woff +0 -0
  55. package/studio/client-dist/assets/ibm-plex-mono-cyrillic-ext-400-normal-DMdlQ8Kv.woff +0 -0
  56. package/studio/client-dist/assets/ibm-plex-mono-cyrillic-ext-400-normal-xuaO2J-f.woff2 +0 -0
  57. package/studio/client-dist/assets/ibm-plex-mono-latin-400-normal-CvHOgSBP.woff +0 -0
  58. package/studio/client-dist/assets/ibm-plex-mono-latin-400-normal-DMJ8VG8y.woff2 +0 -0
  59. package/studio/client-dist/assets/ibm-plex-mono-latin-ext-400-normal-BmRBH3aV.woff2 +0 -0
  60. package/studio/client-dist/assets/ibm-plex-mono-latin-ext-400-normal-D3D2R8hC.woff +0 -0
  61. package/studio/client-dist/assets/ibm-plex-mono-vietnamese-400-normal-BulugwFq.woff2 +0 -0
  62. package/studio/client-dist/assets/ibm-plex-mono-vietnamese-400-normal-DDuiU_S-.woff +0 -0
  63. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-300-normal-LoaTnByX.woff +0 -0
  64. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-300-normal-RhT0Ayi3.woff2 +0 -0
  65. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-400-italic-Bf1n82Tb.woff +0 -0
  66. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-400-italic-D9J6q4_O.woff2 +0 -0
  67. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-400-normal-BTotfTJu.woff +0 -0
  68. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-400-normal-DZqxrq2p.woff2 +0 -0
  69. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-600-normal-71GNu3SW.woff2 +0 -0
  70. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-600-normal-BGq0mW3O.woff +0 -0
  71. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-300-normal-Bbd5v1xK.woff +0 -0
  72. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-300-normal-DqmgDkIN.woff2 +0 -0
  73. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-400-italic-BELOEI8R.woff2 +0 -0
  74. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-400-italic-CfSXsIbO.woff +0 -0
  75. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-400-normal-Dsrv2Tcn.woff +0 -0
  76. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-400-normal-g30qAdWV.woff2 +0 -0
  77. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-600-normal-Bz0x94Yp.woff +0 -0
  78. package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-600-normal-DUMzJB7m.woff2 +0 -0
  79. package/studio/client-dist/assets/ibm-plex-sans-greek-300-normal-J-CBqZoJ.woff2 +0 -0
  80. package/studio/client-dist/assets/ibm-plex-sans-greek-300-normal-STc8XiDr.woff +0 -0
  81. package/studio/client-dist/assets/ibm-plex-sans-greek-400-italic-BEZwmy7D.woff2 +0 -0
  82. package/studio/client-dist/assets/ibm-plex-sans-greek-400-italic-CTYLy0lX.woff +0 -0
  83. package/studio/client-dist/assets/ibm-plex-sans-greek-400-normal-D9ESIMu3.woff +0 -0
  84. package/studio/client-dist/assets/ibm-plex-sans-greek-400-normal-_efipK4i.woff2 +0 -0
  85. package/studio/client-dist/assets/ibm-plex-sans-greek-600-normal-D-CqTdkO.woff +0 -0
  86. package/studio/client-dist/assets/ibm-plex-sans-greek-600-normal-DzTrcv_p.woff2 +0 -0
  87. package/studio/client-dist/assets/ibm-plex-sans-latin-300-normal-C9Of02-5.woff +0 -0
  88. package/studio/client-dist/assets/ibm-plex-sans-latin-300-normal-b4i7YzLG.woff2 +0 -0
  89. package/studio/client-dist/assets/ibm-plex-sans-latin-400-italic-CZTNEAuW.woff2 +0 -0
  90. package/studio/client-dist/assets/ibm-plex-sans-latin-400-italic-CsGl1sm0.woff +0 -0
  91. package/studio/client-dist/assets/ibm-plex-sans-latin-400-normal-CDDApCn2.woff2 +0 -0
  92. package/studio/client-dist/assets/ibm-plex-sans-latin-400-normal-CYLoc0-x.woff +0 -0
  93. package/studio/client-dist/assets/ibm-plex-sans-latin-600-normal-Cu4Hd6ag.woff +0 -0
  94. package/studio/client-dist/assets/ibm-plex-sans-latin-600-normal-CuJfVYMP.woff2 +0 -0
  95. package/studio/client-dist/assets/ibm-plex-sans-latin-ext-300-normal-B8Zgwjh3.woff +0 -0
  96. package/studio/client-dist/assets/ibm-plex-sans-latin-ext-300-normal-BLFuR0P1.woff2 +0 -0
  97. package/studio/client-dist/assets/ibm-plex-sans-latin-ext-400-italic-Du9zzoL2.woff2 +0 -0
  98. package/studio/client-dist/assets/ibm-plex-sans-latin-ext-400-italic-V_9i1P_m.woff +0 -0
  99. package/studio/client-dist/assets/ibm-plex-sans-latin-ext-400-normal-C5H60-Va.woff2 +0 -0
  100. package/studio/client-dist/assets/ibm-plex-sans-latin-ext-400-normal-RBey6euL.woff +0 -0
  101. package/studio/client-dist/assets/ibm-plex-sans-latin-ext-600-normal-DIrixKbi.woff +0 -0
  102. package/studio/client-dist/assets/ibm-plex-sans-latin-ext-600-normal-DOrvGEcy.woff2 +0 -0
  103. package/studio/client-dist/assets/ibm-plex-sans-vietnamese-300-normal-CkYRZD2a.woff +0 -0
  104. package/studio/client-dist/assets/ibm-plex-sans-vietnamese-300-normal-SDfX6ktj.woff2 +0 -0
  105. package/studio/client-dist/assets/ibm-plex-sans-vietnamese-400-italic-78jqtfZH.woff +0 -0
  106. package/studio/client-dist/assets/ibm-plex-sans-vietnamese-400-italic-BbRnwouv.woff2 +0 -0
  107. package/studio/client-dist/assets/ibm-plex-sans-vietnamese-400-normal-DG4YqDda.woff2 +0 -0
  108. package/studio/client-dist/assets/ibm-plex-sans-vietnamese-400-normal-fK1oJ5dG.woff +0 -0
  109. package/studio/client-dist/assets/ibm-plex-sans-vietnamese-600-normal-DgdngZtN.woff +0 -0
  110. package/studio/client-dist/assets/ibm-plex-sans-vietnamese-600-normal-DpPYBSTl.woff2 +0 -0
  111. package/studio/client-dist/assets/index-cxBc2iRv.js +10 -0
  112. package/studio/client-dist/assets/index-wjWuVHXN.css +1 -0
  113. package/studio/client-dist/assets/status-DUGxC7G7.js +1 -0
  114. package/studio/client-dist/assets/testCaseTree-Cl1-bc5O.js +1 -0
  115. package/studio/client-dist/axtr-icon.svg +2 -5
  116. package/studio/client-dist/favicon.svg +2 -5
  117. package/studio/client-dist/index.html +5 -4
  118. package/studio/server/dist/index.js +1530 -292
  119. package/studio/client-dist/assets/index-CgWhGkEn.js +0 -24
  120. package/studio/client-dist/assets/index-CwAUMVCL.css +0 -1
package/MANUAL.md CHANGED
@@ -1,234 +1,214 @@
1
- # TRXA-TOOL setup & usage manual
2
-
3
- This is the step-by-step guide for setting up `trxa-tool` and using it to turn a library of
4
- D365 Finance & Operations Task Recorder (`.axtr`) files into automated, sequenced, data-driven
5
- tests that LLMs can execute against a real D365FO environment via `d365-erp-mcp`.
6
-
7
- This document is the "how do I actually run this?" guide. For the browser UI, see
8
- [STUDIO.md](STUDIO.md).
9
-
10
- ---
11
-
12
- ## 1. Mental model
13
-
14
- Six artifacts make up the whole pipeline, each one building on the last. A raw screen recording
15
- becomes a generic, mechanical test script; you turn that into a named test case with real inputs
16
- and pass/fail criteria; concrete test data and a run sequence tie multiple test cases together
17
- into a suite; related suites can be grouped into a plan for combined reporting; and running a
18
- suite produces a durable, trackable result instead of something that only ever existed in a chat
19
- transcript. Here's how those pieces connect:
20
-
21
- ```
22
- 📼 .axtr file
23
- This is the Task Recorder output, a recording file you get from D365FO's Task Recorder, or from an
24
- LCS Business Process Modeler library export.
25
- │
26
- │ trxa parse
27
- ▼
28
- 📋 Test Script (testscripts/*.json)
29
- This file contains the mechanics: ordered steps, each with a suggested d365-erp-mcp tool call. Fully
30
- generic, never hardcodes a control name, form name, or business value.
31
- │
32
- │ trxa suggest-inputs (optional, drafts a starting point)
33
- ▼
34
- 🧩 Test Case (testcases/*.json)
35
- This is where you come in: you edit this file and define which values are real test inputs, and what a passing run
36
- looks like. This is the atomic pass/fail unit.
37
- │
38
- │ you also edit one or more...
39
- ▼
40
- 🗃️ Fixture (testdata/*.json)
41
- These represent the concrete values for a test case's inputs (e.g. "basic" vs. "eu-company"), so the
42
- same test case can run against different data.
43
- │
44
- │ a suite ties a test case + a fixture together, and defines their sequence.
45
- ▼
46
- 🔗 Test Suite (suites/*.json)
47
- In here you define which test cases run, in what order, and where each one's
48
- inputs come from (a fixture, a literal, or an earlier test case's output).
49
- │
50
- │ optionally, group related suites for combined reporting:
51
- ▼
52
- 📚 Test Plan (plans/*.json)
53
- You use this to define a list of suites (e.g. every suite in "Accounts
54
- Receivable"), run and reported on together. No dependencies between suites;
55
- if two suites need to share data, they belong in one suite instead.
56
- │
57
- │ trxa init-run
58
- ▼
59
- ✅ Test Run (testruns/*.json)
60
- Tracks pass/fail per test case, captured outputs, and
61
- assertion results. Named `<suiteId>-<runId>.json` so every run is kept, not
62
- overwritten, and a plan's suites can share one `runId` for combined reporting.
1
+ # TRXA-TOOL manual
2
+
3
+ This manual explains how to use the `trxa` command line to turn Dynamics 365 Finance and Supply
4
+ Chain Management Task Recorder recordings (`.axtr` files) into automated, data-driven tests, and
5
+ how an AI agent runs those tests against an environment through the Dynamics 365 ERP MCP server.
6
+ It also documents every file format that TRXA-TOOL reads and writes.
7
+
8
+ If you prefer a visual workspace, [TRXA Studio](STUDIO.md) does everything in this manual without
9
+ editing files by hand. Studio and the command line use the same files, so you can switch between
10
+ them at any time.
11
+
12
+ ## How TRXA-TOOL works
13
+
14
+ TRXA-TOOL works with seven kinds of files. Each one builds on the one before it.
15
+
16
+ ```text
17
+ .axtr recording A Task Recorder file, recorded in Dynamics 365 or exported
18
+ | from a Business Process Modeler library in Lifecycle Services.
19
+ | trxa parse
20
+ v
21
+ Test script The recording's ordered steps, each with a suggested MCP tool call.
22
+ (testscripts/*.json) Generic: it never hard-codes a business value.
23
+ |
24
+ | trxa suggest-inputs (optional: drafts a starting point)
25
+ v
26
+ Test case One pass/fail scenario. You define its inputs, the values it
27
+ (testcases/*.json) captures (outputs), and what a passing run looks like (expectations).
28
+ |
29
+ v
30
+ Fixture A named set of test data, such as one company or one scenario,
31
+ (testdata/*.json) with input values for every test case it covers.
32
+ |
33
+ | a suite combines test cases with a fixture and defines their order
34
+ v
35
+ Test suite Which test cases run, in what order, and where each input
36
+ (suites/*.json) gets its value: a fixture, a literal, or an earlier test case's output.
37
+ |
38
+ | optional: group related suites for combined reporting
39
+ v
40
+ Test plan A list of suites that run and report together, such as every
41
+ (plans/*.json) suite for Accounts receivable. Suites in a plan don't share data.
42
+ |
43
+ | trxa init-run
44
+ v
45
+ Test run Pass or fail per test case, captured outputs, and the result of
46
+ (testruns/*.json) each check. Every run is kept, never overwritten.
63
47
  ```
64
48
 
65
- Claude or another LLM client (using the `replay-trxa-testplan` skill) is what actually drives
66
- the last mile: it asks `trxa resolve-next` what to run, replays that test case's Test
67
- Script using `d365-erp-mcp` with the resolved concrete inputs, checks its `expectations`, and
68
- calls `trxa record-result` to persist the outcome.
49
+ TRXA-TOOL handles everything that has a fixed answer: reading recordings, ordering test cases,
50
+ detecting dependency loops, resolving input values and rolling up plan results. An AI agent, using
51
+ the `replay-trxa-testplan` skill, does everything that needs a live Dynamics 365 session. It asks
52
+ `trxa resolve-next` which test case to run, replays that test case's test script through the
53
+ `d365-erp-mcp` server, checks its expectations, and calls `trxa record-result` to save the outcome.
69
54
 
70
- Everything that is deterministic in nature (parsing, DAG ordering, cycle detection, binding
71
- resolution, plan-level rollups) is driven by code.
55
+ ## Prerequisites
72
56
 
73
- Everything that requires an actual D365FO session (clicking, typing, querying, judging
74
- pass/fail) is the LLM's job.
57
+ Before you run a suite against an environment, make sure that you have:
75
58
 
76
- ## 2. Prerequisites
59
+ - **Node.js** 18.19 or later. To check, run `node --version`.
60
+ - **An AI agent that can use MCP tools**, such as GitHub Copilot CLI, Claude Code, Claude Cowork or
61
+ Codex.
62
+ - **The `d365-erp-mcp` MCP server**, connected to the environment that you want to test and
63
+ available to your agent. For setup instructions, including the `trxa mcp-bridge` connector, see
64
+ [Set up TRXA Studio for agent runs](SETUP.md).
65
+ - **One or more `.axtr` recordings.** Record them in Dynamics 365 with Task recorder, or export
66
+ them from a Business Process Modeler library in Lifecycle Services.
77
67
 
78
- Before you can run a full suite against a live environment, make sure you have each of the
79
- following in place:
68
+ ## Install TRXA-TOOL
80
69
 
81
- - **Node.js** ≥ 18.19 (`node --version`)
82
- - **[Claude Cowork](https://claude.com/claude-code)** (or another agent runtime capable of
83
- driving MCP tools). This is what actually executes a test suite
84
- - **The `d365-erp-mcp` MCP server**, connected to the D365 Finance & Operations environment you
85
- want to test against, available in the same Claude Cowork session/project.
86
- - One or more `.axtr` files. Get these either by recording them yourself in D365FO (System
87
- administration → the Task Recorder add-in), or from an existing LCS Business Process Modeler
88
- library.
70
+ 1. Install the package globally. This puts the `trxa` command on your `PATH`. Every command in
71
+ this manual uses it.
89
72
 
90
- ## 3. Installation
73
+ ```sh
74
+ npm install -g trxa-tool
75
+ ```
91
76
 
92
- Install the package globally. This puts the `trxa` command on your `PATH`, and every command in
93
- the rest of this manual is written that way:
77
+ 1. Confirm the installation. The command lists every `trxa` command.
94
78
 
95
- ```sh
96
- npm install -g trxa-tool
97
- trxa help # confirms the install and lists every command
98
- ```
79
+ ```sh
80
+ trxa help
81
+ ```
99
82
 
100
- Then, in your own project folder (where your recordings, test cases and suites will live), run
101
- once:
83
+ 1. In your project folder, where your recordings, test cases and suites will be, install the
84
+ replay skill:
102
85
 
103
- ```sh
104
- trxa init-skill
105
- ```
86
+ ```sh
87
+ trxa init-skill
88
+ ```
106
89
 
107
- Claude Code and Codex discover project skills from their own folders (`.claude/skills/` and
108
- `.agents/skills/`), not from inside an installed package. `init-skill` copies the
109
- `replay-trxa-testplan` skill (step 7.7 below) into both, so whichever agent you use finds it.
90
+ Agents find project skills in their own folders, not inside an installed package. `init-skill`
91
+ copies the `replay-trxa-testplan` skill into `.claude/skills` (used by Claude Code and GitHub
92
+ Copilot CLI) and `.agents/skills` (used by Codex). Run it once per project.
110
93
 
111
- Studio, the browser UI (see [STUDIO.md](STUDIO.md)), is included in the package. There's nothing
112
- extra to install; start it with `trxa studio`.
94
+ TRXA Studio is included in the package. To start it, run `trxa studio`.
113
95
 
114
- ## 4. Directory layout
96
+ ## Project folders
115
97
 
116
- Directory-by-directory contents are below; which of these get committed to git (and why some are
117
- deliberately ignored by default) is covered in [Data hygiene](#5-data-hygiene).
98
+ TRXA-TOOL expects these folders in your project. To decide which of them to commit to source
99
+ control, see [Data hygiene](#data-hygiene).
118
100
 
119
- | Directory | Contents |
101
+ | Folder | Contents |
120
102
  |---|---|
121
- | `recordings/` | Raw `.axtr` recordings |
122
- | `testscripts/` | Generated Test Script JSON (output of `parse`) |
123
- | `testcases/` | Test Case JSON, the reusable, shareable test library |
124
- | `testdata/` | Fixture JSON, concrete input values |
125
- | `suites/` | Test Suite JSON, sequencing/DAGs |
126
- | `plans/` | Test Plan JSON, groups of suites for combined reporting |
127
- | `testruns/` | Generated Test Run JSON per run |
128
-
129
- ## 5. Data hygiene
130
-
131
- Read this before uploading a real `.axtr` library. A `.axtr` recording captures whatever the
132
- user interacted with when the recording was made (real account numbers, journal entries,
133
- amounts, customer names, etc.). `testscripts/*.json` carries the same data forward, and
134
- `suggest-inputs`'s draft `example` values are copied verbatim from the recording too.
135
-
136
- If you keep your project in git, consider:
137
-
138
- - **Ignoring `recordings/` and `testdata/` by default** in `.gitignore`, and allow-listing only
139
- files you've reviewed and know to be safe to share (for example, recordings made against
140
- standard demo data).
141
- - **Ignoring `testscripts/` and `testruns/` entirely.** Test scripts are fully derived from a
142
- recording (regenerate them with `parse`), and test runs are generated output.
143
- - **Reviewing the `example` fields** of any `testcases/*.json` you author (by hand or via
144
- `suggest-inputs`) before committing. Replace real business data with a placeholder of your
145
- choice.
146
- - **Keeping `analysis/` local.** `analyze-library`'s `library-report.json` replaces every
147
- recorded value with an opaque token, so it's safe to share with a cloud LLM, but
148
- `scrubbed-values.csv` maps those tokens back to the real values. Don't open it anywhere an LLM
149
- can read it.
150
-
151
- ## 6. Full CLI reference
152
-
153
- Run everything as `trxa <command> ...`. Running a command with no further arguments prints its
154
- specific usage.
155
-
156
- | Command | Purpose |
103
+ | `recordings` | Task Recorder recordings (`.axtr`) |
104
+ | `testscripts` | Test scripts, created by `trxa parse` |
105
+ | `testcases` | Test cases: your reusable test library |
106
+ | `testdata` | Fixtures: test data values |
107
+ | `suites` | Test suites: run order and data flow |
108
+ | `plans` | Test plans: groups of suites for combined reporting |
109
+ | `runs` | Plan run records, created by `trxa init-plan-run` |
110
+ | `testruns` | Test run results, one file per suite run |
111
+
112
+ ## Data hygiene
113
+
114
+ Read this section before you add a library of real recordings. A recording captures whatever the
115
+ user worked with when it was made, such as real account numbers, journal entries, amounts and
116
+ customer names. Test scripts carry the same data forward, and the `example` values that
117
+ `suggest-inputs` drafts are copied directly from the recording.
118
+
119
+ If you keep your project in source control, consider these practices:
120
+
121
+ - **Ignore `recordings` and `testdata` by default** in `.gitignore`. Allow only files that you
122
+ reviewed and know are safe to share, such as recordings made with standard demo data.
123
+ - **Ignore `testscripts` and `testruns` entirely.** You can regenerate test scripts from the
124
+ recordings with `trxa parse`, and test runs are output.
125
+ - **Review the `example` fields** of every test case before you commit it, and replace real
126
+ business data with placeholders.
127
+ - **Keep the `analysis` folder local.** The `library-report.json` file that `analyze-library`
128
+ writes replaces every recorded value with a token, so it's safe to share with a cloud AI service.
129
+ The `scrubbed-values.csv` file maps those tokens back to the real values, so never open it where
130
+ an AI service can read it.
131
+
132
+ ## Command reference
133
+
134
+ Run each command as `trxa <command>`. To see a command's options, run it with no arguments.
135
+
136
+ | Command | Description |
157
137
  |---|---|
158
- | `parse <file.axtr> [-o out.testscript.json]` | Extract + normalize a recording into a Test Script |
159
- | `suggest-inputs <file.axtr> [-o draft.json]` | Draft candidate `inputs` for a Test Case from a recording |
160
- | `validate-testcase <testcase.json>` | Validate a Test Case file |
161
- | `validate-fixture <fixture.json>` | Validate a Fixture file |
162
- | `validate-suite <suite.json>` | Validate a Test Suite, including DAG cycle/unknown-id checks |
163
- | `plan-suite <suite.json>` | Print a suite's topological execution order |
164
- | `init-skill [--force]` | Copy the `replay-trxa-testplan` skill into this project's `.claude/skills/` and `.agents/skills/`, so Claude Code and Codex both pick it up automatically -- run this once per project; refuses to overwrite an existing copy at either location unless `--force` is passed |
165
- | `init-run <suite.json> -o testruns/<suiteId>-<runId>.json` | Create a fresh Test Run, one `"pending"` entry per suite entry |
166
- | `resolve-next <suite.json> <testrun.json> --testcases-dir <dir> [--fixtures-dir <dir>]` | Get the next runnable test case (inputs already resolved), or `blocked`/`waiting`/`complete` |
167
- | `measure-tokens --since <isoTimestamp> --until <isoTimestamp> [--transcript <path>]` | Sum real token usage from Claude Code's own session log for a time window (mostly for manual/debugging use -- `record-result` already calls this internally; only reliable when no other Claude Code session is active against this project, see "Token/cost reporting" below) |
168
- | `record-result <testrun.json> <testCaseId> --status passed\|failed\|blocked --outputs '{...}' [--assertions '[...]']` | Persist a test case's outcome. Every JSON flag has a `-file` form (`--outputs-file <path>`, `--assertions-file`, `--inputs-file`, `--messages-file`); see [Windows PowerShell](#windows-powershell-pass-json-through-files) below |
169
- | `validate-plan <plan.json>` | Validate a Test Plan, including that every referenced suite exists and is itself valid |
170
- | `init-plan-run <plan.json> -o runs/` | Create a fresh PlanRun record for a plan, auto-generating its id -- do this once per plan execution, before running any of its suites |
171
- | `plan-status <plan.json> <run.json> --testruns-dir <dir>` | Aggregate pass/fail across every suite in a plan for one PlanRun |
172
- | `analyze-library <recordingsDir> [--report ...] [--csv ...]` | Privacy-safe coverage and gap analysis across a whole `recordings/` library (see [Data hygiene](#5-data-hygiene) for its two output files) |
173
- | `studio` | Start the Studio browser UI in the background and open `http://localhost:5173` (see [STUDIO.md](STUDIO.md)) |
174
- | `studio stop` | Stop the backgrounded Studio server + client started by `studio` |
175
- | `export -o <backup.zip>` | Bundle `recordings/`/`testcases/`/`testdata/`/`testscripts/`/`suites/`/`plans/`/`testruns/` into one zip |
176
- | `import <backup.zip> [--force]` | Unpack a library backup zip; refuses on filename collisions unless `--force` |
177
-
178
- Every command exits non-zero and prints a specific, descriptive error on invalid input, nothing
179
- fails silently.
180
-
181
- ### Windows PowerShell: pass JSON through files
182
-
183
- Windows PowerShell 5.1 can silently corrupt JSON passed inline as a command-line argument before
184
- `trxa` ever receives it: it strips the double quotes from JSON without spaces, and splits JSON
185
- with spaces into several arguments. Either way `record-result` fails with a JSON syntax error even
138
+ | `parse <file.axtr> [-o out.testscript.json]` | Reads a recording and writes its test script. |
139
+ | `suggest-inputs <file.axtr> [-o draft.json]` | Drafts candidate test case inputs from a recording. |
140
+ | `validate-testcase <testcase.json>` | Checks a test case file. |
141
+ | `validate-fixture <fixture.json>` | Checks a fixture file. |
142
+ | `migrate-fixtures <testdata-dir> [--dry-run]` | Merges fixture files in the old one-test-case format into one file per fixture. See [Create a fixture](#step-4-create-a-fixture). |
143
+ | `validate-suite <suite.json>` | Checks a suite file, including dependency loops and unknown test case IDs. |
144
+ | `plan-suite <suite.json>` | Prints the order that a suite's test cases run in. |
145
+ | `validate-plan <plan.json>` | Checks a plan file, and that every suite it refers to exists and is valid. |
146
+ | `init-skill [--force]` | Copies the `replay-trxa-testplan` skill into the project's `.claude/skills` and `.agents/skills` folders. Doesn't overwrite an existing copy unless you add `--force`. |
147
+ | `mcp-bridge` | Connects an agent to an environment's Dynamics 365 ERP MCP server. The agent starts it; you don't run it yourself. See [Step 5](SETUP.md#step-5-connect-the-agent-to-dynamics-365) of the setup article. |
148
+ | `init-plan-run <plan.json> -o runs/ [--name "<run name>"]` | Creates a plan run record with a generated ID. Run it once per plan run, before you run any of its suites. `--name` labels the run, for example `"AP regression, PQU 10.0.48"`. |
149
+ | `init-run <suite.json> -o testruns/<suiteId>-<runId>.json [--name "<run name>"]` | Creates a test run with every test case set to `pending`. `--name` labels the run. |
150
+ | `resolve-next <suite.json> <testrun.json> --testcases-dir <dir> [--fixtures-dir <dir>]` | Returns the next test case to run, with its inputs resolved, or reports `blocked`, `waiting` or `complete`. |
151
+ | `record-result <testrun.json> <testCaseId> --status passed\|failed\|blocked --outputs '{...}' [--assertions '[...]']` | Saves a test case's outcome. Each JSON option has a file form: `--outputs-file`, `--assertions-file`, `--inputs-file` and `--messages-file`. See [Pass JSON through files in Windows PowerShell](#pass-json-through-files-in-windows-powershell). |
152
+ | `measure-tokens --since <isoTimestamp> --until <isoTimestamp> [--transcript <path>]` | Adds up the tokens that Claude Code used in a time window. `record-result` already does this. See [Token and cost reporting](#token-and-cost-reporting). |
153
+ | `plan-status <plan.json> <run.json> --testruns-dir <dir>` | Rolls up pass and fail across every suite in a plan for one plan run. |
154
+ | `analyze-library <recordingsDir> [--report ...] [--csv ...]` | Reports coverage and gaps across a whole recordings library without exposing recorded values. See [Data hygiene](#data-hygiene). |
155
+ | `studio` | Starts TRXA Studio in the background and opens `http://127.0.0.1:4321` in your browser. See [Use TRXA Studio](STUDIO.md). |
156
+ | `studio stop` | Stops TRXA Studio. |
157
+ | `export -o <backup.zip>` | Bundles the `recordings`, `testcases`, `testdata`, `testscripts`, `suites`, `plans` and `testruns` folders into one zip file. |
158
+ | `import <backup.zip> [--force]` | Restores a library backup. Stops if a file already exists, unless you add `--force`. |
159
+
160
+ If an input is invalid, every command prints a specific error and exits with a nonzero code.
161
+
162
+ ### Pass JSON through files in Windows PowerShell
163
+
164
+ Windows PowerShell 5.1 can corrupt JSON that you pass directly as a command-line argument, before
165
+ `trxa` receives it. It removes the double quotes from JSON without spaces, and splits JSON with
166
+ spaces into several arguments. Either way, `record-result` fails with a JSON syntax error even
186
167
  though the command looks correct.
187
168
 
188
- Use the `-file` form of each JSON flag instead: write the JSON to a file and pass its path.
169
+ Instead, use the file form of each JSON option. Write the JSON to a file, and then pass the file's
170
+ path:
189
171
 
190
172
  ```powershell
191
173
  [IO.File]::WriteAllText("$PWD\outputs.json", '{"PurchId":"PO-0001"}')
192
174
  trxa record-result testruns/my-run.json create-order --status passed --outputs-file outputs.json
193
175
  ```
194
176
 
195
- Write the file without a byte-order mark, as above. In PowerShell 5.1, `Set-Content -Encoding utf8`
196
- and `Out-File` add one, and the JSON then fails to parse.
177
+ Write the file without a byte order mark, as in this example. In PowerShell 5.1,
178
+ `Set-Content -Encoding utf8` and `Out-File` add one, and the JSON then fails to parse.
197
179
 
198
- This works the same in every shell, so it's a safe default anywhere. The replay skill already uses
199
- it.
180
+ The file form works the same in every shell, so it's a safe default anywhere. The replay skill
181
+ already uses it.
200
182
 
201
- ---
183
+ ## Tutorial: Build and run your first suite
202
184
 
203
- ## 7. Step-by-step: building your first test suite
185
+ This tutorial builds a suite of two test cases: one creates a sales order, and the next posts its
186
+ packing slip. The names are generic, so the steps apply to any scenario.
204
187
 
205
- This walks through a small two-test-case suite as a self-contained tutorial, using a generic
206
- `create-order`/`post-packing-slip` example so the steps apply to any scenario.
207
-
208
- ### 7.1 Parse your recording
188
+ ### Step 1: Parse your recording
209
189
 
210
190
  ```sh
211
191
  trxa parse recordings/my-recording.axtr -o testscripts/my-recording.testscript.json
212
192
  ```
213
193
 
214
- Open the output and skim `steps[]`. Each step has a `hint`, the suggested `d365-erp-mcp` tool
215
- call, with a `confidence` of `"high"`, `"medium"`, or `"low"`. Low-confidence steps are almost
216
- always "find and select a record" steps, because the recording only captured a screen position,
217
- not a durable business key; those are exactly the steps that need to become test **inputs**.
194
+ Open the output and review `steps`. Each step has a `hint`, the suggested `d365-erp-mcp` tool call,
195
+ with a `confidence` of `high`, `medium` or `low`. Low-confidence steps are almost always steps that
196
+ find and select a record. The recording captured only a position on the screen, not a business
197
+ key, so those steps need to become test case inputs.
218
198
 
219
- ### 7.2 Draft candidate inputs
199
+ ### Step 2: Draft candidate inputs
220
200
 
221
201
  ```sh
222
202
  trxa suggest-inputs recordings/my-recording.axtr -o testcases/my-recording.draft.json
223
203
  ```
224
204
 
225
- This scans the Test Script and proposes one candidate per recorded field value
226
- (`PropertyUserAction`) and one per low-confidence record-selection step. Each candidate has a
227
- `bindTo` telling you exactly which step/control it plugs into, plus (for field values) the
228
- recorded `example`; review and scrub that before it goes anywhere shared. This draft is *not*
229
- a valid Test Case yet; it's the `inputs` array to copy into one.
205
+ The command proposes one candidate input for each value typed in the recording
206
+ (`PropertyUserAction`) and one for each low-confidence record selection. Each candidate has a
207
+ `bindTo` that names the step and control it applies to and, for typed values, the recorded
208
+ `example`. Review the examples and remove any real business data before you share them. The draft
209
+ isn't a valid test case yet. It's the `inputs` array that you copy into one.
230
210
 
231
- ### 7.3 Author the Test Case
211
+ ### Step 3: Create the test case
232
212
 
233
213
  Create `testcases/<your-id>.json`:
234
214
 
@@ -254,75 +234,92 @@ Create `testcases/<your-id>.json`:
254
234
  "outputs": [
255
235
  {
256
236
  "name": "salesOrderId",
257
- "kind": "odata", // preferred when there's a solid business key to filter by
237
+ "kind": "odata", // preferred when a reliable business key exists
258
238
  "entitySet": "SalesOrderHeadersV2",
259
239
  "filter": "CustomerAccount eq '{{customerAccount}}'", // {{name}} = resolved input value
260
240
  "field": "SalesOrderNumber"
261
241
  }
262
- // alternative, e.g. for a value only visible on screen (an auto-assigned ID Task Recorder
263
- // never captured because nobody clicked it) -- free at replay time, no extra query:
242
+ // Alternative for a value that's only on screen, such as a number assigned by the system.
243
+ // No extra query is needed at replay time:
264
244
  // { "name": "...", "kind": "uiField", "afterStepSequence": N, "controlName": "..." }
265
- // if the control name isn't known yet, use "hint" instead (plain English); the replaying LLM
266
- // resolves it against live form state and writes the exact controlName back for future runs:
245
+ // If you don't know the control name yet, use "hint" (plain English). The agent finds the
246
+ // control during the run and writes its exact controlName back for later runs:
267
247
  // { "name": "...", "kind": "uiField", "afterStepSequence": N, "hint": "..." }
268
248
  ],
269
249
  "expectations": [
270
250
  {
271
251
  "description": "No error dialog appeared",
272
252
  "checkAfterStepSequence": 46,
273
- "kind": "uiMessage", // fallback when there's no entity to query
253
+ "kind": "uiMessage", // use when there's no entity to query
274
254
  "severity": "Error",
275
- "pattern": "", // plain substring match; empty = any message of this severity
255
+ "pattern": "", // plain text match; empty = any message of this severity
276
256
  "shouldExist": false
277
257
  }
278
- // preferred alternative:
258
+ // Preferred alternative:
279
259
  // { "description": "...", "checkAfterStepSequence": N, "kind": "odata", "entitySet": "...",
280
260
  // "filter": "...", "expectedField": "...", "expectedValue": "...", "comparator": "equals" }
281
261
  ]
282
262
  }
283
263
  ```
284
264
 
285
- Field reference:
265
+ The following table describes the fields.
286
266
 
287
- | Field | Type | Notes |
267
+ | Field | Type | Description |
288
268
  |---|---|---|
289
- | `id`, `name`, `description` | `string` | `id` must be unique within any suite that references it |
290
- | `testScriptPath` | `string` | Path to the Test Script JSON, relative to this file |
291
- | `inputs[].bindTo.kind` | `"control"` \| `"grid-row-selector"` | `"control"` needs `controlName`; `"grid-row-selector"` needs a `note` (why/what to search for) |
292
- | `outputs[].kind` | `"odata"` \| `"uiField"` | `"odata"` needs `entitySet`/`filter`/`field`; `"uiField"` needs `afterStepSequence` plus at least one of `controlName` (exact, free at replay time) or `hint` (plain English, for when the control name isn't known -- the replaying LLM resolves it against live form state and is expected to write the resolved `controlName` back) |
293
- | `expectations[].kind` | `"odata"` \| `"uiMessage"` | `"odata"` needs `entitySet`/`filter`/`expectedField`/`expectedValue`/`comparator` (`equals`\|`notEquals`\|`contains`\|`greaterThan`\|`lessThan`); `"uiMessage"` needs `severity` (`Info`\|`Warning`\|`Error`)/`pattern` (plain substring of the message text, not a regex -- empty matches any message of that severity)/`shouldExist` (should a matching message be present, not whether a record exists) |
269
+ | `id`, `name`, `description` | `string` | `id` must be unique within every suite that uses the test case. |
270
+ | `testScriptPath` | `string` | The path to the test script, relative to the test case file. |
271
+ | `inputs[].bindTo.kind` | `"control"` or `"grid-row-selector"` | `"control"` needs `controlName`. `"grid-row-selector"` needs a `note` that says what to search for. |
272
+ | `outputs[].kind` | `"odata"` or `"uiField"` | `"odata"` needs `entitySet`, `filter` and `field`. `"uiField"` needs `afterStepSequence` and at least one of `controlName` (exact, no query needed) or `hint` (plain English, for when you don't know the control name; the agent writes the control name back after it finds it). |
273
+ | `expectations[].kind` | `"odata"` or `"uiMessage"` | `"odata"` needs `entitySet`, `filter`, `expectedField`, `expectedValue` and `comparator` (`equals`, `notEquals`, `contains`, `greaterThan` or `lessThan`). `"uiMessage"` needs `severity` (`Info`, `Warning` or `Error`), `pattern` (plain text that the message contains, not a regular expression; empty matches any message of that severity) and `shouldExist` (whether a matching message must appear). |
294
274
 
295
- `filter` and `expectedValue` may contain `{{inputName}}` placeholders; Claude substitutes the
296
- resolved input value before querying, as plain string substitution (not a templating engine).
275
+ `filter` and `expectedValue` can contain `{{inputName}}` placeholders. The agent replaces each one
276
+ with the resolved input value before it runs the query. The replacement is plain text, not a
277
+ template language.
297
278
 
298
- Validate as you go:
279
+ Check the file as you work:
299
280
 
300
281
  ```sh
301
282
  trxa validate-testcase testcases/create-order.json
302
283
  ```
303
284
 
304
- ### 7.4 Author a Fixture (concrete test data)
285
+ ### Step 4: Create a fixture
305
286
 
306
- Create `testdata/<your-id>.<fixture-name>.json`:
287
+ A fixture is one set of test data, such as one company or one scenario, with values for every test
288
+ case it covers. Create `testdata/<fixture-name>.json`:
307
289
 
308
290
  ```json
309
291
  {
310
- "testCaseId": "create-order",
311
292
  "name": "basic",
312
- "values": { "customerAccount": "US-001" }
293
+ "description": "US company, standard customer",
294
+ "testCases": {
295
+ "create-order": { "customerAccount": "US-001" },
296
+ "post-packing-slip": { "warehouse": "11" }
297
+ }
313
298
  }
314
299
  ```
315
300
 
316
- You can have multiple fixtures per test case (e.g. `basic`, `eu-company`); the suite picks
317
- which one to use per run. The fixture is not part of the test case itself; it's chosen at the
318
- suite level (see [Author the Test Suite](#75-author-the-test-suite)), which is what lets the
319
- same Test Case run against different data.
301
+ Suites refer to the fixture by `name`, which must match the file name. A name can contain letters,
302
+ digits, spaces, periods, underscores and hyphens, and fixture names must be unique, ignoring case.
303
+ A test case can have values in several fixtures, such as `basic` and `eu-company`. The suite
304
+ chooses which fixture to use (see [Step 5](#step-5-create-the-suite)), so the same test case can
305
+ run with different data.
320
306
 
321
307
  ```sh
322
- trxa validate-fixture testdata/create-order.basic.json
308
+ trxa validate-fixture testdata/basic.json
323
309
  ```
324
310
 
325
- ### 7.5 Author the Test Suite
311
+ > [!NOTE]
312
+ > Earlier versions stored one test case's values per fixture file
313
+ > (`testdata/<test-case-id>.<fixture-name>.json`, with `testCaseId` and `values`). Those files
314
+ > still work, because `resolve-next` merges files with the same `name`. To convert them to the
315
+ > current format, run `trxa migrate-fixtures testdata --dry-run` to preview the changes, and then
316
+ > run it again without `--dry-run`. The command checks everything before it writes. If two files
317
+ > hold values for the same test case under the same fixture name, it changes nothing and names
318
+ > both files.
319
+
320
+ ### Step 5: Create the suite
321
+
322
+ Create `suites/<your-id>.json`:
326
323
 
327
324
  ```json
328
325
  {
@@ -347,202 +344,213 @@ trxa validate-fixture testdata/create-order.basic.json
347
344
  }
348
345
  ```
349
346
 
350
- `inputBindings` per param, one of:
347
+ Each input binding has one of these kinds:
351
348
 
352
- | `kind` | Fields | Meaning |
349
+ | `kind` | Fields | The value comes from |
353
350
  |---|---|---|
354
- | `"fixture"` | `fixtureName`, `param` | Pull a value from a named Fixture's `values` |
355
- | `"testCaseOutput"` | `testCaseId`, `outputName` | Pull a value from an **already-passed** upstream test case's captured output; this is what threads data through a sequence |
356
- | `"literal"` | `value` | A hardcoded value right in the suite file (fine for constants, not for real business data) |
357
- | `"autogenerated"` | *(none)* | No value here at all -- `resolve-next` reports this param in `autogeneratedParams` (with the input's own `description`/`example` as guidance) instead of resolving it, and the orchestrating LLM replaying the suite invents a value at that point, typically for a business key that just needs to be unique per run |
351
+ | `"fixture"` | `fixtureName`, `param` | The named fixture's values for this test case. |
352
+ | `"testCaseOutput"` | `testCaseId`, `outputName` | An output captured by an upstream test case that already passed. This is how data flows through a suite. |
353
+ | `"literal"` | `value` | A value written in the suite file. Use it for constants, not for real business data. |
354
+ | `"autogenerated"` | None | The agent. `resolve-next` lists the input in `autogeneratedParams`, with its `description` and `example` as guidance, and the agent invents a value during the run. Use it for values that only need to be unique, such as a new business key. |
358
355
 
359
- An entry can also declare `runIf` to skip it conditionally, e.g. added to `post-packing-slip`
360
- above:
356
+ An entry can also have a `runIf` condition that skips it. For example, add this to
357
+ `post-packing-slip`:
361
358
 
362
359
  ```json
363
360
  "runIf": { "testCaseId": "create-order", "outputName": "currency", "comparator": "equals", "value": "USD" }
364
361
  ```
365
362
 
366
- `testCaseId` must be one of this entry's own `dependsOn` -- by the time `runIf` is evaluated that
367
- dependency has already passed, so the condition compares its captured *output*, not its status
368
- (which would always be `"passed"` here, so checking it would be pointless). A false condition
369
- records the entry as `"skipped"` instead of running it -- not a pass, not a failure, and (like
370
- `"blocked"`) a downstream entry that depends on a skipped one is itself reported `"blocked"`, not
371
- left waiting forever.
363
+ `testCaseId` must be one of the entry's own `dependsOn` test cases. When `runIf` is evaluated, that
364
+ dependency has already passed, so the condition compares one of its outputs, not its status. If
365
+ the condition is false, the entry is recorded as `skipped` instead of running. A skipped entry is
366
+ neither a pass nor a failure. Entries that depend on a skipped entry are reported as `blocked`.
367
+
368
+ Check the suite, and print its run order:
372
369
 
373
370
  ```sh
374
371
  trxa validate-suite suites/packing-slip-flow.json
375
372
  trxa plan-suite suites/packing-slip-flow.json
376
373
  ```
377
374
 
378
- `validate-suite` also runs the topological sort, so it'll immediately tell you about a cycle or
379
- a `dependsOn` referencing a `testCaseId` that doesn't exist in the suite. (`plan-suite` here
380
- means "print this suite's execution plan", a pre-existing command name that predates, and is
381
- unrelated to, the Test Plan concept covered in
382
- [Grouping suites into a Test Plan](#8-grouping-suites-into-a-test-plan).)
375
+ `validate-suite` immediately reports a dependency loop, or a `dependsOn` that names a test case
376
+ that isn't in the suite.
377
+
378
+ > [!NOTE]
379
+ > `plan-suite` prints a suite's execution plan. The name predates test plans and isn't related to
380
+ > them. For test plans, see [Group suites into a test plan](#group-suites-into-a-test-plan).
383
381
 
384
- ### 7.6 Start a run
382
+ ### Step 6: Start a run
385
383
 
386
384
  ```sh
387
385
  trxa init-run suites/packing-slip-flow.json -o testruns/packing-slip-flow-$(date +%s).json
388
386
  ```
389
387
 
390
- ### 7.7 Run it: hand this to your LLM Client (Claude Cowork)
388
+ ### Step 7: Ask your agent to run the suite
391
389
 
392
- Open a Claude Cowork session with `d365-erp-mcp` connected, and ask it to run the suite (invoking
393
- the `replay-trxa-testplan` skill; it will pick this up automatically from context, or you can
394
- say so explicitly: *"use the replay-trxa-testplan skill to run suites/packing-slip-flow.json
395
- against testruns/packing-slip-flow-....json"*).
390
+ Open a session in your agent, with `d365-erp-mcp` connected, and ask it to run the suite. The agent
391
+ uses the `replay-trxa-testplan` skill. To be explicit, name the skill, for example: *"Use the
392
+ replay-trxa-testplan skill to run suites/packing-slip-flow.json against
393
+ testruns/packing-slip-flow-....json."*
396
394
 
397
- What the LLM actually does, per test case, in a loop (see the skill's "Running as part of a Test
398
- Suite" section for the authoritative version):
395
+ For each test case, the agent repeats these steps. The skill's "Running as part of a Test Suite"
396
+ section is the authoritative version.
399
397
 
400
- 1. `resolve-next` → get the next test case with inputs already resolved (or `blocked`/`waiting`/`complete`)
401
- 2. Replay its `testScriptPath` per the normal per-step procedure, substituting each `bindTo`'d
402
- input with its resolved value instead of the recorded one
403
- 3. Evaluate `expectations` (OData query preferred, UI message fallback)
404
- 4. Capture `outputs` the same way
405
- 5. `record-result` with status + outputs + assertion results
406
- 6. Repeat until `resolve-next` reports `"complete"`
398
+ 1. Runs `resolve-next` to get the next test case, with its inputs resolved, or a `blocked`,
399
+ `waiting` or `complete` status.
400
+ 1. Replays the test case's test script, using the resolved value for each input instead of the
401
+ recorded one.
402
+ 1. Checks the `expectations`. It prefers OData queries and falls back to messages on the form.
403
+ 1. Captures the `outputs`.
404
+ 1. Runs `record-result` with the status, the outputs and the result of each check.
405
+ 1. Stops when `resolve-next` reports `complete`.
407
406
 
408
- ### 7.8 Read the result
407
+ ### Step 8: Read the results
408
+
409
+ Open the test run file, for example:
409
410
 
410
411
  ```sh
411
412
  cat testruns/packing-slip-flow-....json
412
413
  ```
413
414
 
414
- `results[testCaseId].status` is `"passed"`, `"failed"`, `"blocked"` (an upstream dependency
415
- failed or was skipped), or `"skipped"` (its own `runIf` evaluated false -- see §7.5). `outputs`
416
- shows what was captured. `assertionResults` shows each expectation's actual observed value; this
417
- is your expected-vs-actual audit trail, not something living only in a
418
- chat transcript.
419
-
420
- ### Token/cost reporting
421
-
422
- Each result can optionally carry `tokensUsed` (a raw token count) and/or `costUsd` (a dollar
423
- figure). Neither is computed by this CLI itself -- it never talks to the model, so it has no
424
- ground truth of its own -- but they aren't guesses either:
425
-
426
- - `tokensUsed` is measured automatically -- `record-result` calls `measure-tokens` internally
427
- whenever it isn't passed `--tokens` explicitly and the test case has a known start time, no
428
- action needed from the orchestrating LLM or the person driving the CLI by hand. It reads the
429
- real `usage` data Claude Code writes to `~/.claude/projects/.../<session>.jsonl` on every turn
430
- and sums it for the window between this test case's `"running"` transition and the
431
- `record-result` call.
432
-
433
- **Treat this as a rough order-of-magnitude signal, not a trustworthy number.** Live-tested
434
- 2026-08-20 by running two suites headlessly in parallel: the reported total was inflated 24% to
435
- 98% (nearly double) versus each session's own authoritative usage total, because the underlying
436
- heuristic (it reads whichever session log was modified most recently) occasionally attributes another Claude Code session's turns to this one whenever more
437
- than one session is active against this project directory at the same time. That's not a rare
438
- edge case -- it's close to the default way this actually gets used (parallel suites, or a Desktop
439
- session left open while a background job runs). If you need a real number instead of a rough
440
- signal, a `claude -p --output-format json` invocation reports its own authoritative
441
- `total_cost_usd`/`usage` for exactly that invocation, with no cross-session risk at all -- that's
442
- a materially better source than this field, if you're already running headlessly.
443
- If no transcript can be found at all, `tokensUsed` is simply left unset for that result rather
444
- than guessed. This is a deliberate Claude-Code-only bonus, not a core requirement -- the whole
445
- `resolve-next`/`record-result` pipeline and the replay skill's instructions are meant to work
446
- with any sufficiently capable agentic LLM driving them, and no part of pass/fail or captured outputs
447
- depends on this field. Running under a different agent, or under Claude Cowork (which executes
448
- on Anthropic's own servers rather than the local machine), simply leaves it unset -- there's no
449
- local session transcript to read either way, which is expected, not a bug.
450
- - `costUsd` is left unset in the interactive-skill execution mode described in this manual --
451
- there's no dollar figure worth trusting without a real, current, cache-aware pricing table, and
452
- this CLI has no business maintaining one. The field exists for a future execution mode (e.g.
453
- headless `claude -p --output-format json`, which reports `total_cost_usd` itself) to populate
454
- directly once that path exists.
455
-
456
- Studio's Test Runs page shows both, per test case and aggregated across a whole run, whenever at
457
- least one result reports them -- see [STUDIO.md](STUDIO.md).
458
-
459
- ---
460
-
461
- ## 8. Grouping suites into a Test Plan
462
-
463
- Once you have more than one suite covering a related business area (e.g. "Accounts Receivable"
464
- spanning customer onboarding, order-to-cash, and collections), a Test Plan groups them for
465
- combined scheduling/reporting without adding a dependency graph between suites (if two suites
466
- truly need to share data, put those test cases in one suite instead).
467
-
468
- Create `plans/<your-id>.json`:
415
+ For each test case, `results[testCaseId].status` is one of these values:
469
416
 
470
- ```json
471
- {
472
- "id": "accounts-receivable",
473
- "name": "Accounts Receivable regression pack",
474
- "description": "...",
475
- "suites": [
476
- { "suiteId": "packing-slip-flow", "suitePath": "../suites/packing-slip-flow.json" }
477
- ]
478
- }
479
- ```
417
+ | Status | Meaning |
418
+ |---|---|
419
+ | `passed` | Every expectation held. |
420
+ | `failed` | At least one expectation didn't hold, or a step couldn't be performed. |
421
+ | `blocked` | An upstream test case failed or was skipped, so this one didn't run. |
422
+ | `skipped` | The test case's own `runIf` condition was false. See [Step 5](#step-5-create-the-suite). |
480
423
 
481
- `suitePath` is relative to the plan file, same convention as a Test Case's `testScriptPath`.
482
- Array order is just the suggested run order; there's no `dependsOn` at the plan level.
424
+ `outputs` shows the captured values. `assertionResults` shows the value that the agent observed for
425
+ each expectation, so you have a record of expected and actual values that doesn't depend on a chat
426
+ transcript.
483
427
 
484
- ```sh
485
- trxa validate-plan plans/accounts-receivable.json
486
- ```
428
+ To review results visually, open **Test Runs** in [TRXA Studio](STUDIO.md#review-test-runs).
429
+
430
+ ### Token and cost reporting
431
+
432
+ Each result can include `tokensUsed`, a token count, and `costUsd`, a cost in US dollars. The CLI
433
+ never talks to an AI model, so it doesn't calculate either value itself.
434
+
435
+ - **`tokensUsed`** is measured automatically. Unless you pass `--tokens`, `record-result` runs
436
+ `measure-tokens` for the time between the test case's start and the `record-result` call. It
437
+ reads the usage data that Claude Code writes to its session log in
438
+ `~/.claude/projects/.../<session>.jsonl`.
439
+
440
+ Treat this value as a rough estimate. In a test with two suites running in parallel, the reported
441
+ totals were 24% to 98% higher than each session's own usage totals. The measurement reads the
442
+ most recently changed session log, so when more than one Claude Code session works in the same
443
+ project folder at the same time, it can count another session's usage. That situation is common,
444
+ for example with suites running in parallel, or a desktop session left open during a background
445
+ run. For an exact number, run Claude Code with `claude -p --output-format json`, which reports
446
+ `total_cost_usd` and `usage` for that run only.
447
+
448
+ If no session log is found, `tokensUsed` is left empty. The value is available only with Claude
449
+ Code. Other agents, and Claude Cowork, which runs in the cloud, leave it empty. Pass or fail and
450
+ captured outputs never depend on it.
451
+
452
+ - **`costUsd`** is left empty when an agent runs tests interactively, because a reliable cost needs
453
+ a current pricing table that this CLI doesn't maintain. The field is for a future mode, such as
454
+ `claude -p --output-format json`, that reports its own cost.
487
455
 
488
- This resolves and validates every referenced suite too (catching a typo'd path or a broken
489
- suite immediately), not just the plan's own shape.
456
+ When at least one result has these values, TRXA Studio shows them for each test case and as totals
457
+ for the run.
490
458
 
491
- **Running a plan** starts with one `init-plan-run` call to get a PlanRun record (its `id` is what
492
- ties every suite's Test Run together), then the same per-suite loop from
493
- [Run it: hand this to your LLM Client](#77-run-it-hand-this-to-your-llm-client-claude-cowork),
494
- repeated once per suite in the plan:
459
+ ## Group suites into a test plan
460
+
461
+ When you have several suites for one business area, such as customer onboarding, order to cash and
462
+ collections for Accounts receivable, a test plan groups them so that you can run and report on them
463
+ together. A plan doesn't add dependencies between suites. If two suites need to share data, put
464
+ their test cases in one suite.
465
+
466
+ 1. Create `plans/<your-id>.json`:
467
+
468
+ ```json
469
+ {
470
+ "id": "accounts-receivable",
471
+ "name": "Accounts Receivable regression pack",
472
+ "description": "...",
473
+ "suites": [
474
+ { "suiteId": "packing-slip-flow", "suitePath": "../suites/packing-slip-flow.json" }
475
+ ]
476
+ }
477
+ ```
478
+
479
+ `suitePath` is relative to the plan file, the same as a test case's `testScriptPath`. The order
480
+ of `suites` is the suggested run order. Plans have no `dependsOn`.
481
+
482
+ 1. Check the plan. The command also checks every suite that the plan refers to, so it catches a
483
+ wrong path or a broken suite right away.
484
+
485
+ ```sh
486
+ trxa validate-plan plans/accounts-receivable.json
487
+ ```
488
+
489
+ ### Run a plan
490
+
491
+ A plan run starts with one `init-plan-run` call. Its result has an `id` that ties every suite's
492
+ test run together. Then run each suite as in
493
+ [Step 7](#step-7-ask-your-agent-to-run-the-suite), using that same ID:
495
494
 
496
495
  ```sh
497
496
  trxa init-plan-run plans/accounts-receivable.json -o runs/
498
- # prints the generated PlanRun, e.g. { "id": "2026-08-25T14-30-00.000Z", "planId": "accounts-receivable", ... }
497
+ # prints the new plan run, for example { "id": "2026-08-25T14-30-00.000Z", "planId": "accounts-receivable", ... }
499
498
  runId=2026-08-25T14-30-00.000Z
500
499
  trxa init-run suites/packing-slip-flow.json -o testruns/packing-slip-flow-$runId.json
501
- # ... run that suite to completion via the skill, same as step 7.7 ...
502
- # ... repeat init-run + the skill loop for each other suite in the plan, same $runId ...
500
+ # ... run that suite to completion with the skill, as in step 7 ...
501
+ # ... repeat init-run and the skill for each other suite in the plan, with the same $runId ...
503
502
  trxa plan-status plans/accounts-receivable.json runs/$runId.json --testruns-dir testruns
504
503
  ```
505
504
 
506
505
  `plan-status` reads `testruns/<suiteId>-<runId>.json` for every suite in the plan and reports a
507
- rollup: `overall` is `"all-passed"`, `"has-failures"`, `"incomplete"` (a suite hasn't finished
508
- running yet), or `"missing-runs"` (a suite hasn't been run at all for this PlanRun), plus a
509
- per-suite breakdown. It's a pure, read-only query; nothing is persisted beyond the individual
510
- suites' own Test Run files, so you can call it as often as you like while suites are still in
511
- progress.
506
+ per-suite breakdown and an `overall` status:
507
+
508
+ | `overall` | Meaning |
509
+ |---|---|
510
+ | `all-passed` | Every suite passed. |
511
+ | `has-failures` | At least one suite has a failed or blocked test case. |
512
+ | `incomplete` | At least one suite is still running. |
513
+ | `missing-runs` | At least one suite hasn't run yet in this plan run. |
512
514
 
513
- ---
515
+ `plan-status` only reads files, so you can run it as often as you like while suites are still
516
+ running.
514
517
 
515
- ## 9. Troubleshooting
518
+ ## Troubleshoot
516
519
 
517
- | Symptom | Cause | Fix |
520
+ | Symptom | Cause | Solution |
518
521
  |---|---|---|
519
- | `validate-suite` throws `has a dependency cycle involving: ...` | Two entries depend on each other (directly or transitively) | Fix `dependsOn` in the listed entries |
520
- | `validate-suite` throws `depends on unknown testCaseId "..."` | Typo in `dependsOn`, or referenced entry missing from `entries[]` | Add the entry or fix the id |
521
- | `validate-suite` throws `runIf references "..." which must be one of its own dependsOn` | A `runIf.testCaseId` isn't in that entry's own `dependsOn` | Add it to `dependsOn`, or point `runIf` at a test case that's already a direct dependency |
522
- | `resolveInputs` throws `no inputBindings entry for required param "..."` | A Test Case input has no matching entry in the suite's `inputBindings` | Add a binding for that param name in the suite file |
523
- | `resolveInputs` throws `has not passed yet` | A `testCaseOutput` binding points at a test case that hasn't run (or failed) | Check `dependsOn` includes that test case, and that it actually passed |
524
- | `resolve-next` reports `"blocked"` | An upstream test case failed, or was itself skipped | Fix the upstream failure, or accept that dependent is blocked for this run |
525
- | `resolve-next` reports `"skipped"` | An entry's `runIf` evaluated false against its dependency's captured output | Not an error -- this is the conditional-skip feature working as intended; nothing to fix |
526
- | `resolve-next` reports `"waiting"` | A test case is stuck in `"running"` status without a matching `record-result` call | Finish that test case's `record-result` call |
527
- | A step's hint has `confidence: "low"` | The recording captured a row position, not a business key | Expected. This is exactly what a Test Case input with `bindTo.kind: "grid-row-selector"` exists to solve |
528
- | `validate-plan` throws `entry declares suiteId "..." but ... has id "..."` | The `suiteId` in a plan entry doesn't match the `id` inside the suite file it points at | Fix whichever one is wrong; they must match |
529
- | `plan-status` reports a suite as `"missing"` | No `testruns/<suiteId>-<runId>.json` exists for that PlanRun's id | You haven't run that suite yet for this PlanRun; `init-run` + run it, using the same id as the rest of the plan |
530
- | `plan-status` throws `Plan run at ... belongs to plan "..."` | The `run.json` passed in was created via `init-plan-run` against a *different* plan | Pass the PlanRun file that was actually created for this plan |
531
-
532
- ## 10. Best practices
533
-
534
- - Never hand-guess business data during a replay; if a value should vary per run, it belongs in
535
- a Fixture, not typed in on the fly.
536
- - Prefer `kind: "odata"` for `expectations`/`outputs` whenever the outcome maps to a queryable
537
- entity: it's far more robust than parsing UI text, and it's what actually proved the concept
538
- (an inventory-shortage defect was caught this way during the initial dry run).
539
- - Run `validate-testcase`/`validate-fixture`/`validate-suite`/`validate-plan` before every test
540
- run; they're free and catch authoring mistakes deterministically instead of burning through
541
- tokens on a live D365FO session because of a typo.
542
- - Review `example` values and any literal test data before committing; see
543
- [Data hygiene](#5-data-hygiene).
544
- - One Test Case per distinct business scenario, reused across multiple suites/fixtures as
545
- needed, rather than one giant do-everything test case.
546
- - Reach for a Test Plan once you have several suites covering one business area, rather than
547
- running each suite ad hoc; it's free composition/reporting with no added authoring cost per
548
- suite.
522
+ | `validate-suite` reports `has a dependency cycle involving: ...` | Two entries depend on each other, directly or through other entries. | Fix `dependsOn` in the listed entries. |
523
+ | `validate-suite` reports `depends on unknown testCaseId "..."` | A `dependsOn` value is misspelled, or the entry it names isn't in `entries`. | Add the entry, or fix the ID. |
524
+ | `validate-suite` reports `runIf references "..." which must be one of its own dependsOn` | A `runIf.testCaseId` isn't in that entry's `dependsOn`. | Add it to `dependsOn`, or point `runIf` at a direct dependency. |
525
+ | `resolve-next` reports `no inputBindings entry for required param "..."` | A required test case input has no binding in the suite. | Add a binding for that input in the suite file. |
526
+ | `resolve-next` reports `has not passed yet` | A `testCaseOutput` binding points at a test case that hasn't run, or that failed. | Make sure `dependsOn` includes that test case, and that it passed. |
527
+ | `resolve-next` reports `blocked` | An upstream test case failed or was skipped. | Fix the upstream failure, or accept that the dependent test case is blocked for this run. |
528
+ | `resolve-next` reports `skipped` | An entry's `runIf` condition was false. | Nothing. The condition worked as intended. |
529
+ | `resolve-next` reports `waiting` | A test case is still `running`, with no `record-result` call. | Finish that test case's `record-result` call. |
530
+ | A step's hint has `confidence: "low"` | The recording captured a row position, not a business key. | This is expected. Add a test case input with `bindTo.kind: "grid-row-selector"` for the step. |
531
+ | `validate-plan` reports `entry declares suiteId "..." but ... has id "..."` | A plan entry's `suiteId` doesn't match the `id` in the suite file it points at. | Fix whichever value is wrong. They must match. |
532
+ | `plan-status` reports a suite as `missing` | No `testruns/<suiteId>-<runId>.json` exists for the plan run's ID. | Run `init-run` for that suite with the plan run's ID, and then run it. |
533
+ | `plan-status` reports `Plan run at ... belongs to plan "..."` | The plan run file was created for a different plan. | Pass the plan run file that was created for this plan. |
534
+ | `record-result` fails with a JSON syntax error in PowerShell | PowerShell changed the inline JSON. | Use the file options. See [Pass JSON through files in Windows PowerShell](#pass-json-through-files-in-windows-powershell). |
535
+
536
+ ## Best practices
537
+
538
+ - **Never let the agent guess business data.** If a value should change from run to run, put it in
539
+ a fixture.
540
+ - **Prefer OData outputs and expectations** whenever the result is a record you can query. They're
541
+ much more reliable than reading text on a form.
542
+ - **Validate before every run.** `validate-testcase`, `validate-fixture`, `validate-suite` and
543
+ `validate-plan` are free, and they catch mistakes before a live session spends time and tokens on
544
+ them.
545
+ - **Review `example` values and literal test data before you commit them.** See
546
+ [Data hygiene](#data-hygiene).
547
+ - **Keep one test case per business scenario**, and reuse it across suites and fixtures, instead of
548
+ building one large test case that does everything.
549
+ - **Use a test plan** once you have several suites for one business area. It costs nothing extra
550
+ per suite, and it gives you one combined result.
551
+
552
+ ## Related content
553
+
554
+ - [Set up TRXA Studio for agent runs](SETUP.md)
555
+ - [Use TRXA Studio](STUDIO.md)
556
+ - [Use Model Context Protocol for finance and operations apps](https://learn.microsoft.com/dynamics365/fin-ops-core/dev-itpro/copilot/copilot-mcp)