squally-mcp 0.8.4 → 0.9.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -29,7 +29,7 @@ agent can read:
29
29
  - **per-test results** for a run, including which shard ran what;
30
30
  - for a failing attempt: the **error message and stack**, the **code snippet**
31
31
  — **verbatim source code from your test file** — and the **full-context
32
- prompt** (the text behind Copy the full context in Squally), which carries the same snippet plus an **ARIA snapshot of your
32
+ prompt** (the text Squally's Fix panel copies under Fix in your agent, Without MCP), which carries the same snippet plus an **ARIA snapshot of your
33
33
  application at the moment of failure** (whatever was on screen, truncated to
34
34
  3000 characters);
35
35
  - **per-test metrics**: how often each test passed on the first try, passed
@@ -141,8 +141,8 @@ squally-mcp trusts your operating system's certificate store, like your browser
141
141
  | `squally-find-run` | Which runs happened — latest, or by branch, commit SHA, status or source. Counters only. | cheap |
142
142
  | `squally-get-run` | One run with its per-test rows, across all shards. Which test is red. | cheap |
143
143
  | `squally-debug-failure` | Every attempt of one test in one run: error, stack, full-context prompt. | cheap |
144
- | `squally-get-test-metrics` | One test's stability, flaky rate and failure rate over a period. | cheap — one test |
145
- | `squally-list-tests` | Every test with a run in the period, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
144
+ | `squally-get-test-metrics` | One test's stability, flaky rate and failure rate over a period, across its Playwright projects. | cheap — one test |
145
+ | `squally-list-tests` | Every test with a run in the period, once across its Playwright projects, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
146
146
  | `squally-list-errors` | Error signatures in a period: what keeps failing, grouped, and how its runs ended. | cheap |
147
147
 
148
148
  For a single test use `squally-get-test-metrics`, not `squally-list-tests` —
@@ -167,6 +167,29 @@ timeout - and, from squally-reporter 0.11.0, which job of its CI run it is
167
167
  (`ciRunUrl`; for GitHub Actions also for older reporters when the project
168
168
  has one linked repository).
169
169
 
170
+ ### Playwright projects
171
+
172
+ A run counts like Playwright: `squally-find-run`, `squally-get-run` and
173
+ `squally-debug-failure` have one result per test and Playwright project, named
174
+ with its project (`chromium > cart.spec.ts > adds`). What is collected across
175
+ runs — `squally-list-tests`, `squally-get-test-metrics` and
176
+ `squally-list-errors` — counts a test **once across its Playwright projects**,
177
+ by its name without the project (`cart.spec.ts > adds`), and names the
178
+ projects in `projects`: a run failed for the test when it failed in any
179
+ project, and passed only after a retry when it did so in some project and
180
+ failed in none. `variants` carries each project's own numbers. Since 0.9.0
181
+ (read API 1.0.0-beta.19); before, each project's variant was its own test.
182
+
183
+ `squally-list-tests` takes `project` as a list (`["chromium", "firefox"]`) to
184
+ count only those projects. `squally-get-test-metrics` takes the name without
185
+ the project for every project; `project`, or the stored name with its project,
186
+ gives one project's variant. `squally-debug-failure` takes the name
187
+ `squally-get-run` lists, project included.
188
+
189
+ Flaky is a count, not a run status: a run's status is `passed`, `failed`,
190
+ `cancelled`, `incomplete` or `running`, and its `counts.flaky` says how many
191
+ tests passed only after a retry.
192
+
170
193
  ### What the numbers mean
171
194
 
172
195
  The two test tools count **finished runs, CI and local,** in the period (7, 14, 30 or 90
@@ -200,17 +223,23 @@ The three least stable tests of the last 30 days (abridged):
200
223
  "population": "ci_groups_finished_not_mass_failure",
201
224
  "days": 30,
202
225
  "sort": "stability",
203
- "total": 409,
226
+ "total": 205,
204
227
  "items": [
205
228
  {
206
- "testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code",
229
+ "testName": "checkout.spec.ts > Checkout > applies a coupon code",
207
230
  "filePath": "tests/checkout.spec.ts",
231
+ "projects": ["chromium", "firefox"],
208
232
  "runs": 8,
209
233
  "stability": 0.5,
210
234
  "flakyRate": 0.5,
211
235
  "failureRate": 0,
212
236
  "timeLostMs": 7361,
213
- "topBranch": { "branch": "main", "kind": "retries", "count": 3 }
237
+ "medianDurationMs": 1840,
238
+ "topBranch": { "branch": "main", "kind": "retries", "count": 3 },
239
+ "variants": [
240
+ { "testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code", "project": "chromium", "runs": 8, "flakyRuns": 4 },
241
+ { "testName": "firefox > checkout.spec.ts > Checkout > applies a coupon code", "project": "firefox", "runs": 8, "flakyRuns": 0 }
242
+ ]
214
243
  }
215
244
  ]
216
245
  }
@@ -221,16 +250,19 @@ The three least stable tests of the last 30 days (abridged):
221
250
  ```json
222
251
  {
223
252
  "projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
224
- "testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code"
253
+ "testName": "checkout.spec.ts > Checkout > applies a coupon code"
225
254
  }
226
255
  ```
227
256
 
228
- The same numbers as that test's row, for one test (abridged):
257
+ The same numbers as that test's row, for one test across its projects
258
+ (abridged); add `"project": "firefox"` for firefox's variant alone:
229
259
 
230
260
  ```json
231
261
  {
232
262
  "days": 30,
233
263
  "branch": null,
264
+ "projectFilter": null,
265
+ "projects": ["chromium", "firefox"],
234
266
  "runs": 8,
235
267
  "stableRuns": 4,
236
268
  "flakyRuns": 4,
@@ -242,7 +274,8 @@ The same numbers as that test's row, for one test (abridged):
242
274
  }
243
275
  ```
244
276
 
245
- Pass the name exactly as Squally shows it. When it exists in several spec
277
+ Pass the name exactly as Squally shows it — without the project for the test
278
+ across projects, with it for one project. When it exists in several spec
246
279
  files, the tool answers with the `filePath` values to retry with; a test with
247
280
  results but no counted run in the period answers `runs: 0` and `null`
248
281
  rates — an answer, not an error.
package/dist/api.js CHANGED
@@ -52,6 +52,13 @@ export function buildUrl(apiBase, operation, args) {
52
52
  // empty string IS the value (EMPTY_IS_A_VALUE).
53
53
  if (value === undefined || value === null)
54
54
  continue;
55
+ // A list is the parameter repeated (?project=a&project=b, API
56
+ // 1.0.0-beta.19); an empty list is "not specified", like an empty string.
57
+ if (Array.isArray(value)) {
58
+ for (const item of value)
59
+ query.append(parameter.name, String(item));
60
+ continue;
61
+ }
55
62
  if (value === "" && !EMPTY_IS_A_VALUE.has(parameter.name))
56
63
  continue;
57
64
  query.set(parameter.name, String(value));
@@ -54,11 +54,28 @@
54
54
  // row. Said here because "which tests were flaky in this run" is the question
55
55
  // the run page's Fix prompt sends an agent with.
56
56
  //
57
+ // THE COUNTING RULE SINCE 0.9.0 (API 1.0.0-beta.19): a run counts like
58
+ // Playwright, one result per test and Playwright project; what is collected
59
+ // across runs counts a test once across its projects. Said here because the
60
+ // two tool families now name a test differently - with and without its
61
+ // project - and a model that mixes them asks squally-debug-failure for a name
62
+ // it cannot resolve, or adds up per-project rows as if they were tests. "Flaky
63
+ // is a count, not a run status" because a run's counts carry `flaky` beside
64
+ // four statuses, and a model reads a fifth status into it.
65
+ //
57
66
  // Exported as `squally-mcp/instructions` (package.json), so it stays a plain
58
67
  // constant - test/exports.test.js imports it through the package name.
59
68
  export const INSTRUCTIONS = [
60
69
  "Start with squally-list-projects; every other tool needs a projectId from it.",
61
70
  "For one test, use squally-get-test-metrics, not squally-list-tests.",
71
+ "A run counts like Playwright: squally-find-run, squally-get-run and squally-debug-failure",
72
+ "have one result per test and Playwright project, its name with the project",
73
+ '("chromium > cart.spec.ts > adds"). What is collected across runs - squally-list-tests,',
74
+ "squally-get-test-metrics, squally-list-errors - counts a test once across its Playwright",
75
+ "projects, by its name without the project, and names the projects: a run failed for the",
76
+ "test when it failed in any project, and passed only after a retry when it did so in some",
77
+ "project and failed in none. Flaky is a count, not a run status: a run's counts say how many",
78
+ "tests passed only after a retry, and its status never says flaky.",
62
79
  "Test metrics count the finished runs of the period, CI and local - a sharded run counts once -",
63
80
  "without runs that were a mass failure as a whole; pass source=ci or source=local",
64
81
  "to count only one of them. A run still in progress counts once all its shards have",