squally-mcp 0.8.4 → 0.9.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +42 -9
- package/dist/api.js +7 -0
- package/dist/instructions.js +17 -0
- package/dist/openapi/v1.json +1072 -210
- package/dist/openapi.d.ts +2 -1
- package/dist/openapi.js +39 -2
- package/dist/tools.js +30 -13
- package/dist/validate.js +5 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -29,7 +29,7 @@ agent can read:
|
|
|
29
29
|
- **per-test results** for a run, including which shard ran what;
|
|
30
30
|
- for a failing attempt: the **error message and stack**, the **code snippet**
|
|
31
31
|
— **verbatim source code from your test file** — and the **full-context
|
|
32
|
-
prompt** (the text
|
|
32
|
+
prompt** (the text Squally's Fix panel copies under Fix in your agent, Without MCP), which carries the same snippet plus an **ARIA snapshot of your
|
|
33
33
|
application at the moment of failure** (whatever was on screen, truncated to
|
|
34
34
|
3000 characters);
|
|
35
35
|
- **per-test metrics**: how often each test passed on the first try, passed
|
|
@@ -141,8 +141,8 @@ squally-mcp trusts your operating system's certificate store, like your browser
|
|
|
141
141
|
| `squally-find-run` | Which runs happened — latest, or by branch, commit SHA, status or source. Counters only. | cheap |
|
|
142
142
|
| `squally-get-run` | One run with its per-test rows, across all shards. Which test is red. | cheap |
|
|
143
143
|
| `squally-debug-failure` | Every attempt of one test in one run: error, stack, full-context prompt. | cheap |
|
|
144
|
-
| `squally-get-test-metrics` | One test's stability, flaky rate and failure rate over a period. | cheap — one test |
|
|
145
|
-
| `squally-list-tests` | Every test with a run in the period, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
|
|
144
|
+
| `squally-get-test-metrics` | One test's stability, flaky rate and failure rate over a period, across its Playwright projects. | cheap — one test |
|
|
145
|
+
| `squally-list-tests` | Every test with a run in the period, once across its Playwright projects, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
|
|
146
146
|
| `squally-list-errors` | Error signatures in a period: what keeps failing, grouped, and how its runs ended. | cheap |
|
|
147
147
|
|
|
148
148
|
For a single test use `squally-get-test-metrics`, not `squally-list-tests` —
|
|
@@ -167,6 +167,29 @@ timeout - and, from squally-reporter 0.11.0, which job of its CI run it is
|
|
|
167
167
|
(`ciRunUrl`; for GitHub Actions also for older reporters when the project
|
|
168
168
|
has one linked repository).
|
|
169
169
|
|
|
170
|
+
### Playwright projects
|
|
171
|
+
|
|
172
|
+
A run counts like Playwright: `squally-find-run`, `squally-get-run` and
|
|
173
|
+
`squally-debug-failure` have one result per test and Playwright project, named
|
|
174
|
+
with its project (`chromium > cart.spec.ts > adds`). What is collected across
|
|
175
|
+
runs — `squally-list-tests`, `squally-get-test-metrics` and
|
|
176
|
+
`squally-list-errors` — counts a test **once across its Playwright projects**,
|
|
177
|
+
by its name without the project (`cart.spec.ts > adds`), and names the
|
|
178
|
+
projects in `projects`: a run failed for the test when it failed in any
|
|
179
|
+
project, and passed only after a retry when it did so in some project and
|
|
180
|
+
failed in none. `variants` carries each project's own numbers. Since 0.9.0
|
|
181
|
+
(read API 1.0.0-beta.19); before, each project's variant was its own test.
|
|
182
|
+
|
|
183
|
+
`squally-list-tests` takes `project` as a list (`["chromium", "firefox"]`) to
|
|
184
|
+
count only those projects. `squally-get-test-metrics` takes the name without
|
|
185
|
+
the project for every project; `project`, or the stored name with its project,
|
|
186
|
+
gives one project's variant. `squally-debug-failure` takes the name
|
|
187
|
+
`squally-get-run` lists, project included.
|
|
188
|
+
|
|
189
|
+
Flaky is a count, not a run status: a run's status is `passed`, `failed`,
|
|
190
|
+
`cancelled`, `incomplete` or `running`, and its `counts.flaky` says how many
|
|
191
|
+
tests passed only after a retry.
|
|
192
|
+
|
|
170
193
|
### What the numbers mean
|
|
171
194
|
|
|
172
195
|
The two test tools count **finished runs, CI and local,** in the period (7, 14, 30 or 90
|
|
@@ -200,17 +223,23 @@ The three least stable tests of the last 30 days (abridged):
|
|
|
200
223
|
"population": "ci_groups_finished_not_mass_failure",
|
|
201
224
|
"days": 30,
|
|
202
225
|
"sort": "stability",
|
|
203
|
-
"total":
|
|
226
|
+
"total": 205,
|
|
204
227
|
"items": [
|
|
205
228
|
{
|
|
206
|
-
"testName": "
|
|
229
|
+
"testName": "checkout.spec.ts > Checkout > applies a coupon code",
|
|
207
230
|
"filePath": "tests/checkout.spec.ts",
|
|
231
|
+
"projects": ["chromium", "firefox"],
|
|
208
232
|
"runs": 8,
|
|
209
233
|
"stability": 0.5,
|
|
210
234
|
"flakyRate": 0.5,
|
|
211
235
|
"failureRate": 0,
|
|
212
236
|
"timeLostMs": 7361,
|
|
213
|
-
"
|
|
237
|
+
"medianDurationMs": 1840,
|
|
238
|
+
"topBranch": { "branch": "main", "kind": "retries", "count": 3 },
|
|
239
|
+
"variants": [
|
|
240
|
+
{ "testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code", "project": "chromium", "runs": 8, "flakyRuns": 4 },
|
|
241
|
+
{ "testName": "firefox > checkout.spec.ts > Checkout > applies a coupon code", "project": "firefox", "runs": 8, "flakyRuns": 0 }
|
|
242
|
+
]
|
|
214
243
|
}
|
|
215
244
|
]
|
|
216
245
|
}
|
|
@@ -221,16 +250,19 @@ The three least stable tests of the last 30 days (abridged):
|
|
|
221
250
|
```json
|
|
222
251
|
{
|
|
223
252
|
"projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
|
|
224
|
-
"testName": "
|
|
253
|
+
"testName": "checkout.spec.ts > Checkout > applies a coupon code"
|
|
225
254
|
}
|
|
226
255
|
```
|
|
227
256
|
|
|
228
|
-
The same numbers as that test's row, for one test
|
|
257
|
+
The same numbers as that test's row, for one test across its projects
|
|
258
|
+
(abridged); add `"project": "firefox"` for firefox's variant alone:
|
|
229
259
|
|
|
230
260
|
```json
|
|
231
261
|
{
|
|
232
262
|
"days": 30,
|
|
233
263
|
"branch": null,
|
|
264
|
+
"projectFilter": null,
|
|
265
|
+
"projects": ["chromium", "firefox"],
|
|
234
266
|
"runs": 8,
|
|
235
267
|
"stableRuns": 4,
|
|
236
268
|
"flakyRuns": 4,
|
|
@@ -242,7 +274,8 @@ The same numbers as that test's row, for one test (abridged):
|
|
|
242
274
|
}
|
|
243
275
|
```
|
|
244
276
|
|
|
245
|
-
Pass the name exactly as Squally shows it
|
|
277
|
+
Pass the name exactly as Squally shows it — without the project for the test
|
|
278
|
+
across projects, with it for one project. When it exists in several spec
|
|
246
279
|
files, the tool answers with the `filePath` values to retry with; a test with
|
|
247
280
|
results but no counted run in the period answers `runs: 0` and `null`
|
|
248
281
|
rates — an answer, not an error.
|
package/dist/api.js
CHANGED
|
@@ -52,6 +52,13 @@ export function buildUrl(apiBase, operation, args) {
|
|
|
52
52
|
// empty string IS the value (EMPTY_IS_A_VALUE).
|
|
53
53
|
if (value === undefined || value === null)
|
|
54
54
|
continue;
|
|
55
|
+
// A list is the parameter repeated (?project=a&project=b, API
|
|
56
|
+
// 1.0.0-beta.19); an empty list is "not specified", like an empty string.
|
|
57
|
+
if (Array.isArray(value)) {
|
|
58
|
+
for (const item of value)
|
|
59
|
+
query.append(parameter.name, String(item));
|
|
60
|
+
continue;
|
|
61
|
+
}
|
|
55
62
|
if (value === "" && !EMPTY_IS_A_VALUE.has(parameter.name))
|
|
56
63
|
continue;
|
|
57
64
|
query.set(parameter.name, String(value));
|
package/dist/instructions.js
CHANGED
|
@@ -54,11 +54,28 @@
|
|
|
54
54
|
// row. Said here because "which tests were flaky in this run" is the question
|
|
55
55
|
// the run page's Fix prompt sends an agent with.
|
|
56
56
|
//
|
|
57
|
+
// THE COUNTING RULE SINCE 0.9.0 (API 1.0.0-beta.19): a run counts like
|
|
58
|
+
// Playwright, one result per test and Playwright project; what is collected
|
|
59
|
+
// across runs counts a test once across its projects. Said here because the
|
|
60
|
+
// two tool families now name a test differently - with and without its
|
|
61
|
+
// project - and a model that mixes them asks squally-debug-failure for a name
|
|
62
|
+
// it cannot resolve, or adds up per-project rows as if they were tests. "Flaky
|
|
63
|
+
// is a count, not a run status" because a run's counts carry `flaky` beside
|
|
64
|
+
// four statuses, and a model reads a fifth status into it.
|
|
65
|
+
//
|
|
57
66
|
// Exported as `squally-mcp/instructions` (package.json), so it stays a plain
|
|
58
67
|
// constant - test/exports.test.js imports it through the package name.
|
|
59
68
|
export const INSTRUCTIONS = [
|
|
60
69
|
"Start with squally-list-projects; every other tool needs a projectId from it.",
|
|
61
70
|
"For one test, use squally-get-test-metrics, not squally-list-tests.",
|
|
71
|
+
"A run counts like Playwright: squally-find-run, squally-get-run and squally-debug-failure",
|
|
72
|
+
"have one result per test and Playwright project, its name with the project",
|
|
73
|
+
'("chromium > cart.spec.ts > adds"). What is collected across runs - squally-list-tests,',
|
|
74
|
+
"squally-get-test-metrics, squally-list-errors - counts a test once across its Playwright",
|
|
75
|
+
"projects, by its name without the project, and names the projects: a run failed for the",
|
|
76
|
+
"test when it failed in any project, and passed only after a retry when it did so in some",
|
|
77
|
+
"project and failed in none. Flaky is a count, not a run status: a run's counts say how many",
|
|
78
|
+
"tests passed only after a retry, and its status never says flaky.",
|
|
62
79
|
"Test metrics count the finished runs of the period, CI and local - a sharded run counts once -",
|
|
63
80
|
"without runs that were a mass failure as a whole; pass source=ci or source=local",
|
|
64
81
|
"to count only one of them. A run still in progress counts once all its shards have",
|