trxa-tool 0.1.1 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/MANUAL.md +376 -368
- package/README.md +75 -51
- package/SETUP.md +401 -0
- package/STUDIO.md +417 -309
- package/dist/fixture-schema.d.ts +45 -5
- package/dist/fixture-schema.js +97 -13
- package/dist/fixture-store.d.ts +37 -0
- package/dist/fixture-store.js +75 -0
- package/dist/index.js +76 -40
- package/dist/mapping.js +4 -1
- package/dist/mcp-bridge.d.ts +52 -0
- package/dist/mcp-bridge.js +290 -0
- package/dist/planrun-schema.d.ts +14 -6
- package/dist/planrun-schema.js +31 -7
- package/dist/planrun.d.ts +1 -1
- package/dist/planrun.js +2 -2
- package/dist/report-schema.d.ts +10 -1
- package/dist/report-schema.js +31 -5
- package/dist/report.d.ts +4 -1
- package/dist/report.js +9 -2
- package/dist/suite-runner.js +8 -2
- package/media/studio/fixtures-test-cases.png +0 -0
- package/media/studio/plans-editor.png +0 -0
- package/media/studio/recordings-create-test-case.png +0 -0
- package/media/studio/run-dialog.png +0 -0
- package/media/studio/settings-agent-runs.png +0 -0
- package/media/studio/settings-working-folder.png +0 -0
- package/media/studio/studio-recordings.png +0 -0
- package/media/studio/suites-canvas.png +0 -0
- package/media/studio/suites-inspector.png +0 -0
- package/media/studio/testcases-expectations.png +0 -0
- package/media/studio/testcases-outputs.png +0 -0
- package/media/studio/testrun-details.png +0 -0
- package/media/studio/testruns-list.png +0 -0
- package/package.json +6 -2
- package/studio/client-dist/assets/CreatePanel-C5bb5OIs.js +6 -0
- package/studio/client-dist/assets/FactTile-G1pURLys.js +1 -0
- package/studio/client-dist/assets/FieldLabel-DPmIEQ08.js +1 -0
- package/studio/client-dist/assets/FixtureNamePicker-137AAgwZ.js +1 -0
- package/studio/client-dist/assets/Fixtures-D0t-soph.js +1 -0
- package/studio/client-dist/assets/GeneralSection-CWkTeSX2.js +1 -0
- package/studio/client-dist/assets/ListPaneHeader-DXKqX36N.js +1 -0
- package/studio/client-dist/assets/Plans-C70evkGc.js +1 -0
- package/studio/client-dist/assets/Recordings-BT0dyVLs.js +1 -0
- package/studio/client-dist/assets/Settings-k-J3u5lH.js +1 -0
- package/studio/client-dist/assets/Suites-DLioOiRN.css +1 -0
- package/studio/client-dist/assets/Suites-DQhhMxjT.js +7 -0
- package/studio/client-dist/assets/TestCases-BdiopVGI.js +1 -0
- package/studio/client-dist/assets/TestRuns-CylAe-3t.js +9 -0
- package/studio/client-dist/assets/api-DJZ703_F.js +24 -0
- package/studio/client-dist/assets/bucket-18-viuwGcaj.js +1 -0
- package/studio/client-dist/assets/fixtureName-CTZT_jKu.js +1 -0
- package/studio/client-dist/assets/ibm-plex-mono-cyrillic-400-normal-BSMlKf0J.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-mono-cyrillic-400-normal-CEL4l2ZJ.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-mono-cyrillic-ext-400-normal-DMdlQ8Kv.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-mono-cyrillic-ext-400-normal-xuaO2J-f.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-mono-latin-400-normal-CvHOgSBP.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-mono-latin-400-normal-DMJ8VG8y.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-mono-latin-ext-400-normal-BmRBH3aV.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-mono-latin-ext-400-normal-D3D2R8hC.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-mono-vietnamese-400-normal-BulugwFq.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-mono-vietnamese-400-normal-DDuiU_S-.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-300-normal-LoaTnByX.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-300-normal-RhT0Ayi3.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-400-italic-Bf1n82Tb.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-400-italic-D9J6q4_O.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-400-normal-BTotfTJu.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-400-normal-DZqxrq2p.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-600-normal-71GNu3SW.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-600-normal-BGq0mW3O.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-300-normal-Bbd5v1xK.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-300-normal-DqmgDkIN.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-400-italic-BELOEI8R.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-400-italic-CfSXsIbO.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-400-normal-Dsrv2Tcn.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-400-normal-g30qAdWV.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-600-normal-Bz0x94Yp.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-cyrillic-ext-600-normal-DUMzJB7m.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-greek-300-normal-J-CBqZoJ.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-greek-300-normal-STc8XiDr.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-greek-400-italic-BEZwmy7D.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-greek-400-italic-CTYLy0lX.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-greek-400-normal-D9ESIMu3.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-greek-400-normal-_efipK4i.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-greek-600-normal-D-CqTdkO.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-greek-600-normal-DzTrcv_p.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-300-normal-C9Of02-5.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-300-normal-b4i7YzLG.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-400-italic-CZTNEAuW.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-400-italic-CsGl1sm0.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-400-normal-CDDApCn2.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-400-normal-CYLoc0-x.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-600-normal-Cu4Hd6ag.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-600-normal-CuJfVYMP.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-ext-300-normal-B8Zgwjh3.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-ext-300-normal-BLFuR0P1.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-ext-400-italic-Du9zzoL2.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-ext-400-italic-V_9i1P_m.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-ext-400-normal-C5H60-Va.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-ext-400-normal-RBey6euL.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-ext-600-normal-DIrixKbi.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-latin-ext-600-normal-DOrvGEcy.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-vietnamese-300-normal-CkYRZD2a.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-vietnamese-300-normal-SDfX6ktj.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-vietnamese-400-italic-78jqtfZH.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-vietnamese-400-italic-BbRnwouv.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-vietnamese-400-normal-DG4YqDda.woff2 +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-vietnamese-400-normal-fK1oJ5dG.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-vietnamese-600-normal-DgdngZtN.woff +0 -0
- package/studio/client-dist/assets/ibm-plex-sans-vietnamese-600-normal-DpPYBSTl.woff2 +0 -0
- package/studio/client-dist/assets/index-cxBc2iRv.js +10 -0
- package/studio/client-dist/assets/index-wjWuVHXN.css +1 -0
- package/studio/client-dist/assets/status-DUGxC7G7.js +1 -0
- package/studio/client-dist/assets/testCaseTree-Cl1-bc5O.js +1 -0
- package/studio/client-dist/axtr-icon.svg +2 -5
- package/studio/client-dist/favicon.svg +2 -5
- package/studio/client-dist/index.html +5 -4
- package/studio/server/dist/index.js +1530 -292
- package/studio/client-dist/assets/index-CgWhGkEn.js +0 -24
- package/studio/client-dist/assets/index-CwAUMVCL.css +0 -1
package/MANUAL.md
CHANGED
|
@@ -1,234 +1,214 @@
|
|
|
1
|
-
# TRXA-TOOL
|
|
2
|
-
|
|
3
|
-
This
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
[
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
##
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
In here you define which test cases run, in what order, and where each one's
|
|
48
|
-
inputs come from (a fixture, a literal, or an earlier test case's output).
|
|
49
|
-
│
|
|
50
|
-
│ optionally, group related suites for combined reporting:
|
|
51
|
-
▼
|
|
52
|
-
📚 Test Plan (plans/*.json)
|
|
53
|
-
You use this to define a list of suites (e.g. every suite in "Accounts
|
|
54
|
-
Receivable"), run and reported on together. No dependencies between suites;
|
|
55
|
-
if two suites need to share data, they belong in one suite instead.
|
|
56
|
-
│
|
|
57
|
-
│ trxa init-run
|
|
58
|
-
▼
|
|
59
|
-
✅ Test Run (testruns/*.json)
|
|
60
|
-
Tracks pass/fail per test case, captured outputs, and
|
|
61
|
-
assertion results. Named `<suiteId>-<runId>.json` so every run is kept, not
|
|
62
|
-
overwritten, and a plan's suites can share one `runId` for combined reporting.
|
|
1
|
+
# TRXA-TOOL manual
|
|
2
|
+
|
|
3
|
+
This manual explains how to use the `trxa` command line to turn Dynamics 365 Finance and Supply
|
|
4
|
+
Chain Management Task Recorder recordings (`.axtr` files) into automated, data-driven tests, and
|
|
5
|
+
how an AI agent runs those tests against an environment through the Dynamics 365 ERP MCP server.
|
|
6
|
+
It also documents every file format that TRXA-TOOL reads and writes.
|
|
7
|
+
|
|
8
|
+
If you prefer a visual workspace, [TRXA Studio](STUDIO.md) does everything in this manual without
|
|
9
|
+
editing files by hand. Studio and the command line use the same files, so you can switch between
|
|
10
|
+
them at any time.
|
|
11
|
+
|
|
12
|
+
## How TRXA-TOOL works
|
|
13
|
+
|
|
14
|
+
TRXA-TOOL works with seven kinds of files. Each one builds on the one before it.
|
|
15
|
+
|
|
16
|
+
```text
|
|
17
|
+
.axtr recording A Task Recorder file, recorded in Dynamics 365 or exported
|
|
18
|
+
| from a Business Process Modeler library in Lifecycle Services.
|
|
19
|
+
| trxa parse
|
|
20
|
+
v
|
|
21
|
+
Test script The recording's ordered steps, each with a suggested MCP tool call.
|
|
22
|
+
(testscripts/*.json) Generic: it never hard-codes a business value.
|
|
23
|
+
|
|
|
24
|
+
| trxa suggest-inputs (optional: drafts a starting point)
|
|
25
|
+
v
|
|
26
|
+
Test case One pass/fail scenario. You define its inputs, the values it
|
|
27
|
+
(testcases/*.json) captures (outputs), and what a passing run looks like (expectations).
|
|
28
|
+
|
|
|
29
|
+
v
|
|
30
|
+
Fixture A named set of test data, such as one company or one scenario,
|
|
31
|
+
(testdata/*.json) with input values for every test case it covers.
|
|
32
|
+
|
|
|
33
|
+
| a suite combines test cases with a fixture and defines their order
|
|
34
|
+
v
|
|
35
|
+
Test suite Which test cases run, in what order, and where each input
|
|
36
|
+
(suites/*.json) gets its value: a fixture, a literal, or an earlier test case's output.
|
|
37
|
+
|
|
|
38
|
+
| optional: group related suites for combined reporting
|
|
39
|
+
v
|
|
40
|
+
Test plan A list of suites that run and report together, such as every
|
|
41
|
+
(plans/*.json) suite for Accounts receivable. Suites in a plan don't share data.
|
|
42
|
+
|
|
|
43
|
+
| trxa init-run
|
|
44
|
+
v
|
|
45
|
+
Test run Pass or fail per test case, captured outputs, and the result of
|
|
46
|
+
(testruns/*.json) each check. Every run is kept, never overwritten.
|
|
63
47
|
```
|
|
64
48
|
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
49
|
+
TRXA-TOOL handles everything that has a fixed answer: reading recordings, ordering test cases,
|
|
50
|
+
detecting dependency loops, resolving input values and rolling up plan results. An AI agent, using
|
|
51
|
+
the `replay-trxa-testplan` skill, does everything that needs a live Dynamics 365 session. It asks
|
|
52
|
+
`trxa resolve-next` which test case to run, replays that test case's test script through the
|
|
53
|
+
`d365-erp-mcp` server, checks its expectations, and calls `trxa record-result` to save the outcome.
|
|
69
54
|
|
|
70
|
-
|
|
71
|
-
resolution, plan-level rollups) is driven by code.
|
|
55
|
+
## Prerequisites
|
|
72
56
|
|
|
73
|
-
|
|
74
|
-
pass/fail) is the LLM's job.
|
|
57
|
+
Before you run a suite against an environment, make sure that you have:
|
|
75
58
|
|
|
76
|
-
|
|
59
|
+
- **Node.js** 18.19 or later. To check, run `node --version`.
|
|
60
|
+
- **An AI agent that can use MCP tools**, such as GitHub Copilot CLI, Claude Code, Claude Cowork or
|
|
61
|
+
Codex.
|
|
62
|
+
- **The `d365-erp-mcp` MCP server**, connected to the environment that you want to test and
|
|
63
|
+
available to your agent. For setup instructions, including the `trxa mcp-bridge` connector, see
|
|
64
|
+
[Set up TRXA Studio for agent runs](SETUP.md).
|
|
65
|
+
- **One or more `.axtr` recordings.** Record them in Dynamics 365 with Task recorder, or export
|
|
66
|
+
them from a Business Process Modeler library in Lifecycle Services.
|
|
77
67
|
|
|
78
|
-
|
|
79
|
-
following in place:
|
|
68
|
+
## Install TRXA-TOOL
|
|
80
69
|
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
driving MCP tools). This is what actually executes a test suite
|
|
84
|
-
- **The `d365-erp-mcp` MCP server**, connected to the D365 Finance & Operations environment you
|
|
85
|
-
want to test against, available in the same Claude Cowork session/project.
|
|
86
|
-
- One or more `.axtr` files. Get these either by recording them yourself in D365FO (System
|
|
87
|
-
administration → the Task Recorder add-in), or from an existing LCS Business Process Modeler
|
|
88
|
-
library.
|
|
70
|
+
1. Install the package globally. This puts the `trxa` command on your `PATH`. Every command in
|
|
71
|
+
this manual uses it.
|
|
89
72
|
|
|
90
|
-
|
|
73
|
+
```sh
|
|
74
|
+
npm install -g trxa-tool
|
|
75
|
+
```
|
|
91
76
|
|
|
92
|
-
|
|
93
|
-
the rest of this manual is written that way:
|
|
77
|
+
1. Confirm the installation. The command lists every `trxa` command.
|
|
94
78
|
|
|
95
|
-
```sh
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
```
|
|
79
|
+
```sh
|
|
80
|
+
trxa help
|
|
81
|
+
```
|
|
99
82
|
|
|
100
|
-
|
|
101
|
-
|
|
83
|
+
1. In your project folder, where your recordings, test cases and suites will be, install the
|
|
84
|
+
replay skill:
|
|
102
85
|
|
|
103
|
-
```sh
|
|
104
|
-
trxa init-skill
|
|
105
|
-
```
|
|
86
|
+
```sh
|
|
87
|
+
trxa init-skill
|
|
88
|
+
```
|
|
106
89
|
|
|
107
|
-
|
|
108
|
-
`.
|
|
109
|
-
`
|
|
90
|
+
Agents find project skills in their own folders, not inside an installed package. `init-skill`
|
|
91
|
+
copies the `replay-trxa-testplan` skill into `.claude/skills` (used by Claude Code and GitHub
|
|
92
|
+
Copilot CLI) and `.agents/skills` (used by Codex). Run it once per project.
|
|
110
93
|
|
|
111
|
-
Studio
|
|
112
|
-
extra to install; start it with `trxa studio`.
|
|
94
|
+
TRXA Studio is included in the package. To start it, run `trxa studio`.
|
|
113
95
|
|
|
114
|
-
##
|
|
96
|
+
## Project folders
|
|
115
97
|
|
|
116
|
-
|
|
117
|
-
|
|
98
|
+
TRXA-TOOL expects these folders in your project. To decide which of them to commit to source
|
|
99
|
+
control, see [Data hygiene](#data-hygiene).
|
|
118
100
|
|
|
119
|
-
|
|
|
101
|
+
| Folder | Contents |
|
|
120
102
|
|---|---|
|
|
121
|
-
| `recordings
|
|
122
|
-
| `testscripts
|
|
123
|
-
| `testcases
|
|
124
|
-
| `testdata
|
|
125
|
-
| `suites
|
|
126
|
-
| `plans
|
|
127
|
-
| `
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
standard demo data
|
|
141
|
-
- **
|
|
142
|
-
|
|
143
|
-
- **
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
| Command | Purpose |
|
|
103
|
+
| `recordings` | Task Recorder recordings (`.axtr`) |
|
|
104
|
+
| `testscripts` | Test scripts, created by `trxa parse` |
|
|
105
|
+
| `testcases` | Test cases: your reusable test library |
|
|
106
|
+
| `testdata` | Fixtures: test data values |
|
|
107
|
+
| `suites` | Test suites: run order and data flow |
|
|
108
|
+
| `plans` | Test plans: groups of suites for combined reporting |
|
|
109
|
+
| `runs` | Plan run records, created by `trxa init-plan-run` |
|
|
110
|
+
| `testruns` | Test run results, one file per suite run |
|
|
111
|
+
|
|
112
|
+
## Data hygiene
|
|
113
|
+
|
|
114
|
+
Read this section before you add a library of real recordings. A recording captures whatever the
|
|
115
|
+
user worked with when it was made, such as real account numbers, journal entries, amounts and
|
|
116
|
+
customer names. Test scripts carry the same data forward, and the `example` values that
|
|
117
|
+
`suggest-inputs` drafts are copied directly from the recording.
|
|
118
|
+
|
|
119
|
+
If you keep your project in source control, consider these practices:
|
|
120
|
+
|
|
121
|
+
- **Ignore `recordings` and `testdata` by default** in `.gitignore`. Allow only files that you
|
|
122
|
+
reviewed and know are safe to share, such as recordings made with standard demo data.
|
|
123
|
+
- **Ignore `testscripts` and `testruns` entirely.** You can regenerate test scripts from the
|
|
124
|
+
recordings with `trxa parse`, and test runs are output.
|
|
125
|
+
- **Review the `example` fields** of every test case before you commit it, and replace real
|
|
126
|
+
business data with placeholders.
|
|
127
|
+
- **Keep the `analysis` folder local.** The `library-report.json` file that `analyze-library`
|
|
128
|
+
writes replaces every recorded value with a token, so it's safe to share with a cloud AI service.
|
|
129
|
+
The `scrubbed-values.csv` file maps those tokens back to the real values, so never open it where
|
|
130
|
+
an AI service can read it.
|
|
131
|
+
|
|
132
|
+
## Command reference
|
|
133
|
+
|
|
134
|
+
Run each command as `trxa <command>`. To see a command's options, run it with no arguments.
|
|
135
|
+
|
|
136
|
+
| Command | Description |
|
|
157
137
|
|---|---|
|
|
158
|
-
| `parse <file.axtr> [-o out.testscript.json]` |
|
|
159
|
-
| `suggest-inputs <file.axtr> [-o draft.json]` |
|
|
160
|
-
| `validate-testcase <testcase.json>` |
|
|
161
|
-
| `validate-fixture <fixture.json>` |
|
|
162
|
-
| `
|
|
163
|
-
| `
|
|
164
|
-
| `
|
|
165
|
-
| `
|
|
166
|
-
| `
|
|
167
|
-
| `
|
|
168
|
-
| `
|
|
169
|
-
| `
|
|
170
|
-
| `
|
|
171
|
-
| `
|
|
172
|
-
| `
|
|
173
|
-
| `
|
|
174
|
-
| `
|
|
175
|
-
| `
|
|
176
|
-
| `
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
138
|
+
| `parse <file.axtr> [-o out.testscript.json]` | Reads a recording and writes its test script. |
|
|
139
|
+
| `suggest-inputs <file.axtr> [-o draft.json]` | Drafts candidate test case inputs from a recording. |
|
|
140
|
+
| `validate-testcase <testcase.json>` | Checks a test case file. |
|
|
141
|
+
| `validate-fixture <fixture.json>` | Checks a fixture file. |
|
|
142
|
+
| `migrate-fixtures <testdata-dir> [--dry-run]` | Merges fixture files in the old one-test-case format into one file per fixture. See [Create a fixture](#step-4-create-a-fixture). |
|
|
143
|
+
| `validate-suite <suite.json>` | Checks a suite file, including dependency loops and unknown test case IDs. |
|
|
144
|
+
| `plan-suite <suite.json>` | Prints the order that a suite's test cases run in. |
|
|
145
|
+
| `validate-plan <plan.json>` | Checks a plan file, and that every suite it refers to exists and is valid. |
|
|
146
|
+
| `init-skill [--force]` | Copies the `replay-trxa-testplan` skill into the project's `.claude/skills` and `.agents/skills` folders. Doesn't overwrite an existing copy unless you add `--force`. |
|
|
147
|
+
| `mcp-bridge` | Connects an agent to an environment's Dynamics 365 ERP MCP server. The agent starts it; you don't run it yourself. See [Step 5](SETUP.md#step-5-connect-the-agent-to-dynamics-365) of the setup article. |
|
|
148
|
+
| `init-plan-run <plan.json> -o runs/ [--name "<run name>"]` | Creates a plan run record with a generated ID. Run it once per plan run, before you run any of its suites. `--name` labels the run, for example `"AP regression, PQU 10.0.48"`. |
|
|
149
|
+
| `init-run <suite.json> -o testruns/<suiteId>-<runId>.json [--name "<run name>"]` | Creates a test run with every test case set to `pending`. `--name` labels the run. |
|
|
150
|
+
| `resolve-next <suite.json> <testrun.json> --testcases-dir <dir> [--fixtures-dir <dir>]` | Returns the next test case to run, with its inputs resolved, or reports `blocked`, `waiting` or `complete`. |
|
|
151
|
+
| `record-result <testrun.json> <testCaseId> --status passed\|failed\|blocked --outputs '{...}' [--assertions '[...]']` | Saves a test case's outcome. Each JSON option has a file form: `--outputs-file`, `--assertions-file`, `--inputs-file` and `--messages-file`. See [Pass JSON through files in Windows PowerShell](#pass-json-through-files-in-windows-powershell). |
|
|
152
|
+
| `measure-tokens --since <isoTimestamp> --until <isoTimestamp> [--transcript <path>]` | Adds up the tokens that Claude Code used in a time window. `record-result` already does this. See [Token and cost reporting](#token-and-cost-reporting). |
|
|
153
|
+
| `plan-status <plan.json> <run.json> --testruns-dir <dir>` | Rolls up pass and fail across every suite in a plan for one plan run. |
|
|
154
|
+
| `analyze-library <recordingsDir> [--report ...] [--csv ...]` | Reports coverage and gaps across a whole recordings library without exposing recorded values. See [Data hygiene](#data-hygiene). |
|
|
155
|
+
| `studio` | Starts TRXA Studio in the background and opens `http://127.0.0.1:4321` in your browser. See [Use TRXA Studio](STUDIO.md). |
|
|
156
|
+
| `studio stop` | Stops TRXA Studio. |
|
|
157
|
+
| `export -o <backup.zip>` | Bundles the `recordings`, `testcases`, `testdata`, `testscripts`, `suites`, `plans` and `testruns` folders into one zip file. |
|
|
158
|
+
| `import <backup.zip> [--force]` | Restores a library backup. Stops if a file already exists, unless you add `--force`. |
|
|
159
|
+
|
|
160
|
+
If an input is invalid, every command prints a specific error and exits with a nonzero code.
|
|
161
|
+
|
|
162
|
+
### Pass JSON through files in Windows PowerShell
|
|
163
|
+
|
|
164
|
+
Windows PowerShell 5.1 can corrupt JSON that you pass directly as a command-line argument, before
|
|
165
|
+
`trxa` receives it. It removes the double quotes from JSON without spaces, and splits JSON with
|
|
166
|
+
spaces into several arguments. Either way, `record-result` fails with a JSON syntax error even
|
|
186
167
|
though the command looks correct.
|
|
187
168
|
|
|
188
|
-
|
|
169
|
+
Instead, use the file form of each JSON option. Write the JSON to a file, and then pass the file's
|
|
170
|
+
path:
|
|
189
171
|
|
|
190
172
|
```powershell
|
|
191
173
|
[IO.File]::WriteAllText("$PWD\outputs.json", '{"PurchId":"PO-0001"}')
|
|
192
174
|
trxa record-result testruns/my-run.json create-order --status passed --outputs-file outputs.json
|
|
193
175
|
```
|
|
194
176
|
|
|
195
|
-
Write the file without a byte
|
|
196
|
-
and `Out-File` add one, and the JSON then fails to parse.
|
|
177
|
+
Write the file without a byte order mark, as in this example. In PowerShell 5.1,
|
|
178
|
+
`Set-Content -Encoding utf8` and `Out-File` add one, and the JSON then fails to parse.
|
|
197
179
|
|
|
198
|
-
|
|
199
|
-
it.
|
|
180
|
+
The file form works the same in every shell, so it's a safe default anywhere. The replay skill
|
|
181
|
+
already uses it.
|
|
200
182
|
|
|
201
|
-
|
|
183
|
+
## Tutorial: Build and run your first suite
|
|
202
184
|
|
|
203
|
-
|
|
185
|
+
This tutorial builds a suite of two test cases: one creates a sales order, and the next posts its
|
|
186
|
+
packing slip. The names are generic, so the steps apply to any scenario.
|
|
204
187
|
|
|
205
|
-
|
|
206
|
-
`create-order`/`post-packing-slip` example so the steps apply to any scenario.
|
|
207
|
-
|
|
208
|
-
### 7.1 Parse your recording
|
|
188
|
+
### Step 1: Parse your recording
|
|
209
189
|
|
|
210
190
|
```sh
|
|
211
191
|
trxa parse recordings/my-recording.axtr -o testscripts/my-recording.testscript.json
|
|
212
192
|
```
|
|
213
193
|
|
|
214
|
-
Open the output and
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
194
|
+
Open the output and review `steps`. Each step has a `hint`, the suggested `d365-erp-mcp` tool call,
|
|
195
|
+
with a `confidence` of `high`, `medium` or `low`. Low-confidence steps are almost always steps that
|
|
196
|
+
find and select a record. The recording captured only a position on the screen, not a business
|
|
197
|
+
key, so those steps need to become test case inputs.
|
|
218
198
|
|
|
219
|
-
###
|
|
199
|
+
### Step 2: Draft candidate inputs
|
|
220
200
|
|
|
221
201
|
```sh
|
|
222
202
|
trxa suggest-inputs recordings/my-recording.axtr -o testcases/my-recording.draft.json
|
|
223
203
|
```
|
|
224
204
|
|
|
225
|
-
|
|
226
|
-
(`PropertyUserAction`) and one
|
|
227
|
-
`bindTo`
|
|
228
|
-
|
|
229
|
-
a valid
|
|
205
|
+
The command proposes one candidate input for each value typed in the recording
|
|
206
|
+
(`PropertyUserAction`) and one for each low-confidence record selection. Each candidate has a
|
|
207
|
+
`bindTo` that names the step and control it applies to and, for typed values, the recorded
|
|
208
|
+
`example`. Review the examples and remove any real business data before you share them. The draft
|
|
209
|
+
isn't a valid test case yet. It's the `inputs` array that you copy into one.
|
|
230
210
|
|
|
231
|
-
###
|
|
211
|
+
### Step 3: Create the test case
|
|
232
212
|
|
|
233
213
|
Create `testcases/<your-id>.json`:
|
|
234
214
|
|
|
@@ -254,75 +234,92 @@ Create `testcases/<your-id>.json`:
|
|
|
254
234
|
"outputs": [
|
|
255
235
|
{
|
|
256
236
|
"name": "salesOrderId",
|
|
257
|
-
"kind": "odata", // preferred when
|
|
237
|
+
"kind": "odata", // preferred when a reliable business key exists
|
|
258
238
|
"entitySet": "SalesOrderHeadersV2",
|
|
259
239
|
"filter": "CustomerAccount eq '{{customerAccount}}'", // {{name}} = resolved input value
|
|
260
240
|
"field": "SalesOrderNumber"
|
|
261
241
|
}
|
|
262
|
-
//
|
|
263
|
-
//
|
|
242
|
+
// Alternative for a value that's only on screen, such as a number assigned by the system.
|
|
243
|
+
// No extra query is needed at replay time:
|
|
264
244
|
// { "name": "...", "kind": "uiField", "afterStepSequence": N, "controlName": "..." }
|
|
265
|
-
//
|
|
266
|
-
//
|
|
245
|
+
// If you don't know the control name yet, use "hint" (plain English). The agent finds the
|
|
246
|
+
// control during the run and writes its exact controlName back for later runs:
|
|
267
247
|
// { "name": "...", "kind": "uiField", "afterStepSequence": N, "hint": "..." }
|
|
268
248
|
],
|
|
269
249
|
"expectations": [
|
|
270
250
|
{
|
|
271
251
|
"description": "No error dialog appeared",
|
|
272
252
|
"checkAfterStepSequence": 46,
|
|
273
|
-
"kind": "uiMessage", //
|
|
253
|
+
"kind": "uiMessage", // use when there's no entity to query
|
|
274
254
|
"severity": "Error",
|
|
275
|
-
"pattern": "", // plain
|
|
255
|
+
"pattern": "", // plain text match; empty = any message of this severity
|
|
276
256
|
"shouldExist": false
|
|
277
257
|
}
|
|
278
|
-
//
|
|
258
|
+
// Preferred alternative:
|
|
279
259
|
// { "description": "...", "checkAfterStepSequence": N, "kind": "odata", "entitySet": "...",
|
|
280
260
|
// "filter": "...", "expectedField": "...", "expectedValue": "...", "comparator": "equals" }
|
|
281
261
|
]
|
|
282
262
|
}
|
|
283
263
|
```
|
|
284
264
|
|
|
285
|
-
|
|
265
|
+
The following table describes the fields.
|
|
286
266
|
|
|
287
|
-
| Field | Type |
|
|
267
|
+
| Field | Type | Description |
|
|
288
268
|
|---|---|---|
|
|
289
|
-
| `id`, `name`, `description` | `string` | `id` must be unique within
|
|
290
|
-
| `testScriptPath` | `string` |
|
|
291
|
-
| `inputs[].bindTo.kind` | `"control"`
|
|
292
|
-
| `outputs[].kind` | `"odata"`
|
|
293
|
-
| `expectations[].kind` | `"odata"`
|
|
269
|
+
| `id`, `name`, `description` | `string` | `id` must be unique within every suite that uses the test case. |
|
|
270
|
+
| `testScriptPath` | `string` | The path to the test script, relative to the test case file. |
|
|
271
|
+
| `inputs[].bindTo.kind` | `"control"` or `"grid-row-selector"` | `"control"` needs `controlName`. `"grid-row-selector"` needs a `note` that says what to search for. |
|
|
272
|
+
| `outputs[].kind` | `"odata"` or `"uiField"` | `"odata"` needs `entitySet`, `filter` and `field`. `"uiField"` needs `afterStepSequence` and at least one of `controlName` (exact, no query needed) or `hint` (plain English, for when you don't know the control name; the agent writes the control name back after it finds it). |
|
|
273
|
+
| `expectations[].kind` | `"odata"` or `"uiMessage"` | `"odata"` needs `entitySet`, `filter`, `expectedField`, `expectedValue` and `comparator` (`equals`, `notEquals`, `contains`, `greaterThan` or `lessThan`). `"uiMessage"` needs `severity` (`Info`, `Warning` or `Error`), `pattern` (plain text that the message contains, not a regular expression; empty matches any message of that severity) and `shouldExist` (whether a matching message must appear). |
|
|
294
274
|
|
|
295
|
-
`filter` and `expectedValue`
|
|
296
|
-
resolved input value before
|
|
275
|
+
`filter` and `expectedValue` can contain `{{inputName}}` placeholders. The agent replaces each one
|
|
276
|
+
with the resolved input value before it runs the query. The replacement is plain text, not a
|
|
277
|
+
template language.
|
|
297
278
|
|
|
298
|
-
|
|
279
|
+
Check the file as you work:
|
|
299
280
|
|
|
300
281
|
```sh
|
|
301
282
|
trxa validate-testcase testcases/create-order.json
|
|
302
283
|
```
|
|
303
284
|
|
|
304
|
-
###
|
|
285
|
+
### Step 4: Create a fixture
|
|
305
286
|
|
|
306
|
-
|
|
287
|
+
A fixture is one set of test data, such as one company or one scenario, with values for every test
|
|
288
|
+
case it covers. Create `testdata/<fixture-name>.json`:
|
|
307
289
|
|
|
308
290
|
```json
|
|
309
291
|
{
|
|
310
|
-
"testCaseId": "create-order",
|
|
311
292
|
"name": "basic",
|
|
312
|
-
"
|
|
293
|
+
"description": "US company, standard customer",
|
|
294
|
+
"testCases": {
|
|
295
|
+
"create-order": { "customerAccount": "US-001" },
|
|
296
|
+
"post-packing-slip": { "warehouse": "11" }
|
|
297
|
+
}
|
|
313
298
|
}
|
|
314
299
|
```
|
|
315
300
|
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
301
|
+
Suites refer to the fixture by `name`, which must match the file name. A name can contain letters,
|
|
302
|
+
digits, spaces, periods, underscores and hyphens, and fixture names must be unique, ignoring case.
|
|
303
|
+
A test case can have values in several fixtures, such as `basic` and `eu-company`. The suite
|
|
304
|
+
chooses which fixture to use (see [Step 5](#step-5-create-the-suite)), so the same test case can
|
|
305
|
+
run with different data.
|
|
320
306
|
|
|
321
307
|
```sh
|
|
322
|
-
trxa validate-fixture testdata/
|
|
308
|
+
trxa validate-fixture testdata/basic.json
|
|
323
309
|
```
|
|
324
310
|
|
|
325
|
-
|
|
311
|
+
> [!NOTE]
|
|
312
|
+
> Earlier versions stored one test case's values per fixture file
|
|
313
|
+
> (`testdata/<test-case-id>.<fixture-name>.json`, with `testCaseId` and `values`). Those files
|
|
314
|
+
> still work, because `resolve-next` merges files with the same `name`. To convert them to the
|
|
315
|
+
> current format, run `trxa migrate-fixtures testdata --dry-run` to preview the changes, and then
|
|
316
|
+
> run it again without `--dry-run`. The command checks everything before it writes. If two files
|
|
317
|
+
> hold values for the same test case under the same fixture name, it changes nothing and names
|
|
318
|
+
> both files.
|
|
319
|
+
|
|
320
|
+
### Step 5: Create the suite
|
|
321
|
+
|
|
322
|
+
Create `suites/<your-id>.json`:
|
|
326
323
|
|
|
327
324
|
```json
|
|
328
325
|
{
|
|
@@ -347,202 +344,213 @@ trxa validate-fixture testdata/create-order.basic.json
|
|
|
347
344
|
}
|
|
348
345
|
```
|
|
349
346
|
|
|
350
|
-
|
|
347
|
+
Each input binding has one of these kinds:
|
|
351
348
|
|
|
352
|
-
| `kind` | Fields |
|
|
349
|
+
| `kind` | Fields | The value comes from |
|
|
353
350
|
|---|---|---|
|
|
354
|
-
| `"fixture"` | `fixtureName`, `param` |
|
|
355
|
-
| `"testCaseOutput"` | `testCaseId`, `outputName` |
|
|
356
|
-
| `"literal"` | `value` | A
|
|
357
|
-
| `"autogenerated"` |
|
|
351
|
+
| `"fixture"` | `fixtureName`, `param` | The named fixture's values for this test case. |
|
|
352
|
+
| `"testCaseOutput"` | `testCaseId`, `outputName` | An output captured by an upstream test case that already passed. This is how data flows through a suite. |
|
|
353
|
+
| `"literal"` | `value` | A value written in the suite file. Use it for constants, not for real business data. |
|
|
354
|
+
| `"autogenerated"` | None | The agent. `resolve-next` lists the input in `autogeneratedParams`, with its `description` and `example` as guidance, and the agent invents a value during the run. Use it for values that only need to be unique, such as a new business key. |
|
|
358
355
|
|
|
359
|
-
An entry can also
|
|
360
|
-
|
|
356
|
+
An entry can also have a `runIf` condition that skips it. For example, add this to
|
|
357
|
+
`post-packing-slip`:
|
|
361
358
|
|
|
362
359
|
```json
|
|
363
360
|
"runIf": { "testCaseId": "create-order", "outputName": "currency", "comparator": "equals", "value": "USD" }
|
|
364
361
|
```
|
|
365
362
|
|
|
366
|
-
`testCaseId` must be one of
|
|
367
|
-
dependency has already passed, so the condition compares its
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
363
|
+
`testCaseId` must be one of the entry's own `dependsOn` test cases. When `runIf` is evaluated, that
|
|
364
|
+
dependency has already passed, so the condition compares one of its outputs, not its status. If
|
|
365
|
+
the condition is false, the entry is recorded as `skipped` instead of running. A skipped entry is
|
|
366
|
+
neither a pass nor a failure. Entries that depend on a skipped entry are reported as `blocked`.
|
|
367
|
+
|
|
368
|
+
Check the suite, and print its run order:
|
|
372
369
|
|
|
373
370
|
```sh
|
|
374
371
|
trxa validate-suite suites/packing-slip-flow.json
|
|
375
372
|
trxa plan-suite suites/packing-slip-flow.json
|
|
376
373
|
```
|
|
377
374
|
|
|
378
|
-
`validate-suite`
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
375
|
+
`validate-suite` immediately reports a dependency loop, or a `dependsOn` that names a test case
|
|
376
|
+
that isn't in the suite.
|
|
377
|
+
|
|
378
|
+
> [!NOTE]
|
|
379
|
+
> `plan-suite` prints a suite's execution plan. The name predates test plans and isn't related to
|
|
380
|
+
> them. For test plans, see [Group suites into a test plan](#group-suites-into-a-test-plan).
|
|
383
381
|
|
|
384
|
-
###
|
|
382
|
+
### Step 6: Start a run
|
|
385
383
|
|
|
386
384
|
```sh
|
|
387
385
|
trxa init-run suites/packing-slip-flow.json -o testruns/packing-slip-flow-$(date +%s).json
|
|
388
386
|
```
|
|
389
387
|
|
|
390
|
-
### 7
|
|
388
|
+
### Step 7: Ask your agent to run the suite
|
|
391
389
|
|
|
392
|
-
Open a
|
|
393
|
-
the `replay-trxa-testplan` skill
|
|
394
|
-
|
|
395
|
-
|
|
390
|
+
Open a session in your agent, with `d365-erp-mcp` connected, and ask it to run the suite. The agent
|
|
391
|
+
uses the `replay-trxa-testplan` skill. To be explicit, name the skill, for example: *"Use the
|
|
392
|
+
replay-trxa-testplan skill to run suites/packing-slip-flow.json against
|
|
393
|
+
testruns/packing-slip-flow-....json."*
|
|
396
394
|
|
|
397
|
-
|
|
398
|
-
|
|
395
|
+
For each test case, the agent repeats these steps. The skill's "Running as part of a Test Suite"
|
|
396
|
+
section is the authoritative version.
|
|
399
397
|
|
|
400
|
-
1. `resolve-next`
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
398
|
+
1. Runs `resolve-next` to get the next test case, with its inputs resolved, or a `blocked`,
|
|
399
|
+
`waiting` or `complete` status.
|
|
400
|
+
1. Replays the test case's test script, using the resolved value for each input instead of the
|
|
401
|
+
recorded one.
|
|
402
|
+
1. Checks the `expectations`. It prefers OData queries and falls back to messages on the form.
|
|
403
|
+
1. Captures the `outputs`.
|
|
404
|
+
1. Runs `record-result` with the status, the outputs and the result of each check.
|
|
405
|
+
1. Stops when `resolve-next` reports `complete`.
|
|
407
406
|
|
|
408
|
-
###
|
|
407
|
+
### Step 8: Read the results
|
|
408
|
+
|
|
409
|
+
Open the test run file, for example:
|
|
409
410
|
|
|
410
411
|
```sh
|
|
411
412
|
cat testruns/packing-slip-flow-....json
|
|
412
413
|
```
|
|
413
414
|
|
|
414
|
-
`results[testCaseId].status` is
|
|
415
|
-
failed or was skipped), or `"skipped"` (its own `runIf` evaluated false -- see §7.5). `outputs`
|
|
416
|
-
shows what was captured. `assertionResults` shows each expectation's actual observed value; this
|
|
417
|
-
is your expected-vs-actual audit trail, not something living only in a
|
|
418
|
-
chat transcript.
|
|
419
|
-
|
|
420
|
-
### Token/cost reporting
|
|
421
|
-
|
|
422
|
-
Each result can optionally carry `tokensUsed` (a raw token count) and/or `costUsd` (a dollar
|
|
423
|
-
figure). Neither is computed by this CLI itself -- it never talks to the model, so it has no
|
|
424
|
-
ground truth of its own -- but they aren't guesses either:
|
|
425
|
-
|
|
426
|
-
- `tokensUsed` is measured automatically -- `record-result` calls `measure-tokens` internally
|
|
427
|
-
whenever it isn't passed `--tokens` explicitly and the test case has a known start time, no
|
|
428
|
-
action needed from the orchestrating LLM or the person driving the CLI by hand. It reads the
|
|
429
|
-
real `usage` data Claude Code writes to `~/.claude/projects/.../<session>.jsonl` on every turn
|
|
430
|
-
and sums it for the window between this test case's `"running"` transition and the
|
|
431
|
-
`record-result` call.
|
|
432
|
-
|
|
433
|
-
**Treat this as a rough order-of-magnitude signal, not a trustworthy number.** Live-tested
|
|
434
|
-
2026-08-20 by running two suites headlessly in parallel: the reported total was inflated 24% to
|
|
435
|
-
98% (nearly double) versus each session's own authoritative usage total, because the underlying
|
|
436
|
-
heuristic (it reads whichever session log was modified most recently) occasionally attributes another Claude Code session's turns to this one whenever more
|
|
437
|
-
than one session is active against this project directory at the same time. That's not a rare
|
|
438
|
-
edge case -- it's close to the default way this actually gets used (parallel suites, or a Desktop
|
|
439
|
-
session left open while a background job runs). If you need a real number instead of a rough
|
|
440
|
-
signal, a `claude -p --output-format json` invocation reports its own authoritative
|
|
441
|
-
`total_cost_usd`/`usage` for exactly that invocation, with no cross-session risk at all -- that's
|
|
442
|
-
a materially better source than this field, if you're already running headlessly.
|
|
443
|
-
If no transcript can be found at all, `tokensUsed` is simply left unset for that result rather
|
|
444
|
-
than guessed. This is a deliberate Claude-Code-only bonus, not a core requirement -- the whole
|
|
445
|
-
`resolve-next`/`record-result` pipeline and the replay skill's instructions are meant to work
|
|
446
|
-
with any sufficiently capable agentic LLM driving them, and no part of pass/fail or captured outputs
|
|
447
|
-
depends on this field. Running under a different agent, or under Claude Cowork (which executes
|
|
448
|
-
on Anthropic's own servers rather than the local machine), simply leaves it unset -- there's no
|
|
449
|
-
local session transcript to read either way, which is expected, not a bug.
|
|
450
|
-
- `costUsd` is left unset in the interactive-skill execution mode described in this manual --
|
|
451
|
-
there's no dollar figure worth trusting without a real, current, cache-aware pricing table, and
|
|
452
|
-
this CLI has no business maintaining one. The field exists for a future execution mode (e.g.
|
|
453
|
-
headless `claude -p --output-format json`, which reports `total_cost_usd` itself) to populate
|
|
454
|
-
directly once that path exists.
|
|
455
|
-
|
|
456
|
-
Studio's Test Runs page shows both, per test case and aggregated across a whole run, whenever at
|
|
457
|
-
least one result reports them -- see [STUDIO.md](STUDIO.md).
|
|
458
|
-
|
|
459
|
-
---
|
|
460
|
-
|
|
461
|
-
## 8. Grouping suites into a Test Plan
|
|
462
|
-
|
|
463
|
-
Once you have more than one suite covering a related business area (e.g. "Accounts Receivable"
|
|
464
|
-
spanning customer onboarding, order-to-cash, and collections), a Test Plan groups them for
|
|
465
|
-
combined scheduling/reporting without adding a dependency graph between suites (if two suites
|
|
466
|
-
truly need to share data, put those test cases in one suite instead).
|
|
467
|
-
|
|
468
|
-
Create `plans/<your-id>.json`:
|
|
415
|
+
For each test case, `results[testCaseId].status` is one of these values:
|
|
469
416
|
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
{ "suiteId": "packing-slip-flow", "suitePath": "../suites/packing-slip-flow.json" }
|
|
477
|
-
]
|
|
478
|
-
}
|
|
479
|
-
```
|
|
417
|
+
| Status | Meaning |
|
|
418
|
+
|---|---|
|
|
419
|
+
| `passed` | Every expectation held. |
|
|
420
|
+
| `failed` | At least one expectation didn't hold, or a step couldn't be performed. |
|
|
421
|
+
| `blocked` | An upstream test case failed or was skipped, so this one didn't run. |
|
|
422
|
+
| `skipped` | The test case's own `runIf` condition was false. See [Step 5](#step-5-create-the-suite). |
|
|
480
423
|
|
|
481
|
-
`
|
|
482
|
-
|
|
424
|
+
`outputs` shows the captured values. `assertionResults` shows the value that the agent observed for
|
|
425
|
+
each expectation, so you have a record of expected and actual values that doesn't depend on a chat
|
|
426
|
+
transcript.
|
|
483
427
|
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
428
|
+
To review results visually, open **Test Runs** in [TRXA Studio](STUDIO.md#review-test-runs).
|
|
429
|
+
|
|
430
|
+
### Token and cost reporting
|
|
431
|
+
|
|
432
|
+
Each result can include `tokensUsed`, a token count, and `costUsd`, a cost in US dollars. The CLI
|
|
433
|
+
never talks to an AI model, so it doesn't calculate either value itself.
|
|
434
|
+
|
|
435
|
+
- **`tokensUsed`** is measured automatically. Unless you pass `--tokens`, `record-result` runs
|
|
436
|
+
`measure-tokens` for the time between the test case's start and the `record-result` call. It
|
|
437
|
+
reads the usage data that Claude Code writes to its session log in
|
|
438
|
+
`~/.claude/projects/.../<session>.jsonl`.
|
|
439
|
+
|
|
440
|
+
Treat this value as a rough estimate. In a test with two suites running in parallel, the reported
|
|
441
|
+
totals were 24% to 98% higher than each session's own usage totals. The measurement reads the
|
|
442
|
+
most recently changed session log, so when more than one Claude Code session works in the same
|
|
443
|
+
project folder at the same time, it can count another session's usage. That situation is common,
|
|
444
|
+
for example with suites running in parallel, or a desktop session left open during a background
|
|
445
|
+
run. For an exact number, run Claude Code with `claude -p --output-format json`, which reports
|
|
446
|
+
`total_cost_usd` and `usage` for that run only.
|
|
447
|
+
|
|
448
|
+
If no session log is found, `tokensUsed` is left empty. The value is available only with Claude
|
|
449
|
+
Code. Other agents, and Claude Cowork, which runs in the cloud, leave it empty. Pass or fail and
|
|
450
|
+
captured outputs never depend on it.
|
|
451
|
+
|
|
452
|
+
- **`costUsd`** is left empty when an agent runs tests interactively, because a reliable cost needs
|
|
453
|
+
a current pricing table that this CLI doesn't maintain. The field is for a future mode, such as
|
|
454
|
+
`claude -p --output-format json`, that reports its own cost.
|
|
487
455
|
|
|
488
|
-
|
|
489
|
-
|
|
456
|
+
When at least one result has these values, TRXA Studio shows them for each test case and as totals
|
|
457
|
+
for the run.
|
|
490
458
|
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
|
|
459
|
+
## Group suites into a test plan
|
|
460
|
+
|
|
461
|
+
When you have several suites for one business area, such as customer onboarding, order to cash and
|
|
462
|
+
collections for Accounts receivable, a test plan groups them so that you can run and report on them
|
|
463
|
+
together. A plan doesn't add dependencies between suites. If two suites need to share data, put
|
|
464
|
+
their test cases in one suite.
|
|
465
|
+
|
|
466
|
+
1. Create `plans/<your-id>.json`:
|
|
467
|
+
|
|
468
|
+
```json
|
|
469
|
+
{
|
|
470
|
+
"id": "accounts-receivable",
|
|
471
|
+
"name": "Accounts Receivable regression pack",
|
|
472
|
+
"description": "...",
|
|
473
|
+
"suites": [
|
|
474
|
+
{ "suiteId": "packing-slip-flow", "suitePath": "../suites/packing-slip-flow.json" }
|
|
475
|
+
]
|
|
476
|
+
}
|
|
477
|
+
```
|
|
478
|
+
|
|
479
|
+
`suitePath` is relative to the plan file, the same as a test case's `testScriptPath`. The order
|
|
480
|
+
of `suites` is the suggested run order. Plans have no `dependsOn`.
|
|
481
|
+
|
|
482
|
+
1. Check the plan. The command also checks every suite that the plan refers to, so it catches a
|
|
483
|
+
wrong path or a broken suite right away.
|
|
484
|
+
|
|
485
|
+
```sh
|
|
486
|
+
trxa validate-plan plans/accounts-receivable.json
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
### Run a plan
|
|
490
|
+
|
|
491
|
+
A plan run starts with one `init-plan-run` call. Its result has an `id` that ties every suite's
|
|
492
|
+
test run together. Then run each suite as in
|
|
493
|
+
[Step 7](#step-7-ask-your-agent-to-run-the-suite), using that same ID:
|
|
495
494
|
|
|
496
495
|
```sh
|
|
497
496
|
trxa init-plan-run plans/accounts-receivable.json -o runs/
|
|
498
|
-
# prints the
|
|
497
|
+
# prints the new plan run, for example { "id": "2026-08-25T14-30-00.000Z", "planId": "accounts-receivable", ... }
|
|
499
498
|
runId=2026-08-25T14-30-00.000Z
|
|
500
499
|
trxa init-run suites/packing-slip-flow.json -o testruns/packing-slip-flow-$runId.json
|
|
501
|
-
# ... run that suite to completion
|
|
502
|
-
# ... repeat init-run
|
|
500
|
+
# ... run that suite to completion with the skill, as in step 7 ...
|
|
501
|
+
# ... repeat init-run and the skill for each other suite in the plan, with the same $runId ...
|
|
503
502
|
trxa plan-status plans/accounts-receivable.json runs/$runId.json --testruns-dir testruns
|
|
504
503
|
```
|
|
505
504
|
|
|
506
505
|
`plan-status` reads `testruns/<suiteId>-<runId>.json` for every suite in the plan and reports a
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
506
|
+
per-suite breakdown and an `overall` status:
|
|
507
|
+
|
|
508
|
+
| `overall` | Meaning |
|
|
509
|
+
|---|---|
|
|
510
|
+
| `all-passed` | Every suite passed. |
|
|
511
|
+
| `has-failures` | At least one suite has a failed or blocked test case. |
|
|
512
|
+
| `incomplete` | At least one suite is still running. |
|
|
513
|
+
| `missing-runs` | At least one suite hasn't run yet in this plan run. |
|
|
512
514
|
|
|
513
|
-
|
|
515
|
+
`plan-status` only reads files, so you can run it as often as you like while suites are still
|
|
516
|
+
running.
|
|
514
517
|
|
|
515
|
-
##
|
|
518
|
+
## Troubleshoot
|
|
516
519
|
|
|
517
|
-
| Symptom | Cause |
|
|
520
|
+
| Symptom | Cause | Solution |
|
|
518
521
|
|---|---|---|
|
|
519
|
-
| `validate-suite`
|
|
520
|
-
| `validate-suite`
|
|
521
|
-
| `validate-suite`
|
|
522
|
-
| `
|
|
523
|
-
| `
|
|
524
|
-
| `resolve-next` reports `
|
|
525
|
-
| `resolve-next` reports `
|
|
526
|
-
| `resolve-next` reports `
|
|
527
|
-
| A step's hint has `confidence: "low"` | The recording captured a row position, not a business key |
|
|
528
|
-
| `validate-plan`
|
|
529
|
-
| `plan-status` reports a suite as `
|
|
530
|
-
| `plan-status`
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
- Review `example` values and
|
|
543
|
-
[Data hygiene](#
|
|
544
|
-
-
|
|
545
|
-
|
|
546
|
-
-
|
|
547
|
-
|
|
548
|
-
|
|
522
|
+
| `validate-suite` reports `has a dependency cycle involving: ...` | Two entries depend on each other, directly or through other entries. | Fix `dependsOn` in the listed entries. |
|
|
523
|
+
| `validate-suite` reports `depends on unknown testCaseId "..."` | A `dependsOn` value is misspelled, or the entry it names isn't in `entries`. | Add the entry, or fix the ID. |
|
|
524
|
+
| `validate-suite` reports `runIf references "..." which must be one of its own dependsOn` | A `runIf.testCaseId` isn't in that entry's `dependsOn`. | Add it to `dependsOn`, or point `runIf` at a direct dependency. |
|
|
525
|
+
| `resolve-next` reports `no inputBindings entry for required param "..."` | A required test case input has no binding in the suite. | Add a binding for that input in the suite file. |
|
|
526
|
+
| `resolve-next` reports `has not passed yet` | A `testCaseOutput` binding points at a test case that hasn't run, or that failed. | Make sure `dependsOn` includes that test case, and that it passed. |
|
|
527
|
+
| `resolve-next` reports `blocked` | An upstream test case failed or was skipped. | Fix the upstream failure, or accept that the dependent test case is blocked for this run. |
|
|
528
|
+
| `resolve-next` reports `skipped` | An entry's `runIf` condition was false. | Nothing. The condition worked as intended. |
|
|
529
|
+
| `resolve-next` reports `waiting` | A test case is still `running`, with no `record-result` call. | Finish that test case's `record-result` call. |
|
|
530
|
+
| A step's hint has `confidence: "low"` | The recording captured a row position, not a business key. | This is expected. Add a test case input with `bindTo.kind: "grid-row-selector"` for the step. |
|
|
531
|
+
| `validate-plan` reports `entry declares suiteId "..." but ... has id "..."` | A plan entry's `suiteId` doesn't match the `id` in the suite file it points at. | Fix whichever value is wrong. They must match. |
|
|
532
|
+
| `plan-status` reports a suite as `missing` | No `testruns/<suiteId>-<runId>.json` exists for the plan run's ID. | Run `init-run` for that suite with the plan run's ID, and then run it. |
|
|
533
|
+
| `plan-status` reports `Plan run at ... belongs to plan "..."` | The plan run file was created for a different plan. | Pass the plan run file that was created for this plan. |
|
|
534
|
+
| `record-result` fails with a JSON syntax error in PowerShell | PowerShell changed the inline JSON. | Use the file options. See [Pass JSON through files in Windows PowerShell](#pass-json-through-files-in-windows-powershell). |
|
|
535
|
+
|
|
536
|
+
## Best practices
|
|
537
|
+
|
|
538
|
+
- **Never let the agent guess business data.** If a value should change from run to run, put it in
|
|
539
|
+
a fixture.
|
|
540
|
+
- **Prefer OData outputs and expectations** whenever the result is a record you can query. They're
|
|
541
|
+
much more reliable than reading text on a form.
|
|
542
|
+
- **Validate before every run.** `validate-testcase`, `validate-fixture`, `validate-suite` and
|
|
543
|
+
`validate-plan` are free, and they catch mistakes before a live session spends time and tokens on
|
|
544
|
+
them.
|
|
545
|
+
- **Review `example` values and literal test data before you commit them.** See
|
|
546
|
+
[Data hygiene](#data-hygiene).
|
|
547
|
+
- **Keep one test case per business scenario**, and reuse it across suites and fixtures, instead of
|
|
548
|
+
building one large test case that does everything.
|
|
549
|
+
- **Use a test plan** once you have several suites for one business area. It costs nothing extra
|
|
550
|
+
per suite, and it gives you one combined result.
|
|
551
|
+
|
|
552
|
+
## Related content
|
|
553
|
+
|
|
554
|
+
- [Set up TRXA Studio for agent runs](SETUP.md)
|
|
555
|
+
- [Use TRXA Studio](STUDIO.md)
|
|
556
|
+
- [Use Model Context Protocol for finance and operations apps](https://learn.microsoft.com/dynamics365/fin-ops-core/dev-itpro/copilot/copilot-mcp)
|