squally-mcp 0.3.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,7 +1,7 @@
1
1
  # squally-mcp
2
2
 
3
3
  A local [MCP](https://modelcontextprotocol.io) server that lets a coding agent
4
- read your [Squally](https://app.squally.dev) data: which CI runs happened, how
4
+ read your [Squally](https://app.squally.dev) data: which runs happened, how
5
5
  stable each test is, and why a particular test failed.
6
6
 
7
7
  It runs on your machine over stdio and talks to Squally's read API over HTTPS.
@@ -138,23 +138,38 @@ squally-mcp trusts your operating system's certificate store, like your browser
138
138
  | Tool | What it answers | Cost |
139
139
  |---|---|---|
140
140
  | `squally-list-projects` | Which projects exist. **Start here** — every other tool needs a `projectId` from it. | cheap |
141
- | `squally-find-run` | Which runs happened — latest, or by branch, commit SHA or status. Counters only. | cheap |
141
+ | `squally-find-run` | Which runs happened — latest, or by branch, commit SHA, status or source. Counters only. | cheap |
142
142
  | `squally-get-run` | One run with its per-test rows, across all shards. Which test is red. | cheap |
143
143
  | `squally-debug-failure` | Every attempt of one test in one run: error, stack, Copy-for-AI prompt. | cheap |
144
144
  | `squally-get-test-metrics` | One test's stability, flaky rate and failure rate over a period. | cheap — one test |
145
145
  | `squally-list-tests` | Every test with a CI run in the period, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
146
- | `squally-list-errors` | Error signatures in a period: what keeps failing, grouped. | cheap |
146
+ | `squally-list-errors` | Error signatures in a period: what keeps failing, grouped, and how its runs ended. | cheap |
147
147
 
148
148
  For a single test use `squally-get-test-metrics`, not `squally-list-tests` —
149
149
  the tool descriptions say so, and the server repeats it in its instructions.
150
150
 
151
+ ### Runs
152
+
153
+ `squally-find-run` lists every run, CI and local alike; `source` (`all` by
154
+ default, `ci` or `local`) narrows it, and each run says whether it was
155
+ `local`. A run's `status` is the dashboard's word: `passed`, `failed`,
156
+ `cancelled`, `incomplete` (a shard timed out, was stopped by the monthly
157
+ result limit or never reported) or `running` - and `?status=` filters by
158
+ any of them. A cancelled run is `cancelled`; `cancellation` says how, when
159
+ and by whom. A sharded run is one row, its counters over all its shards with
160
+ every test once, and the tests a cancel cut off count nowhere.
161
+
151
162
  ### What the numbers mean
152
163
 
153
- The two test tools count **CI runs whose shards all finished** in the period
154
- (7, 14, 30 or 90 days; 30 by default), on every branch unless you name one. A
155
- sharded run counts once, placed in the period by its first shard's start; a
156
- run that was a mass failure as a whole is left out. Local runs never count.
157
- Per test:
164
+ The two test tools count **finished CI runs** in the period (7, 14, 30 or 90
165
+ days; 30 by default), on every branch unless you name one. A run finishes when
166
+ all its shards have - or, once the run timeout has passed, with the results its
167
+ shards reported, like a cancelled run; a shard stopped by the monthly result
168
+ limit counts like one that timed out. A sharded run counts once, placed in the period
169
+ by its first shard's start; a run that was a mass failure as a whole is left
170
+ out. Local runs count only
171
+ when you ask for them: `source` is `ci` (the default), `local` or `all`, and
172
+ the response's `population` names which runs it counted. Per test:
158
173
 
159
174
  - **stability** — runs that passed on the first try ÷ runs;
160
175
  - **flakyRate** — runs that passed only after a retry ÷ runs;
@@ -17,16 +17,41 @@
17
17
  // runs whose shards all finished, each counted once ("ci_groups_finished_not_
18
18
  // mass_failure"); it was every completed shard on its own.
19
19
  //
20
+ // RUNS IN PROGRESS SINCE 0.4.0 (API 1.0.0-beta.7): said outright, because the
21
+ // newest run is the one an agent asks about, and it is missing from the
22
+ // numbers until its last shard has finished.
23
+ //
24
+ // THE SOURCE SINCE 0.4.0 (API 1.0.0-beta.8): CI runs by default; local runs
25
+ // count only when the agent asks for them - they were never counted before.
26
+ //
27
+ // THE RUN TIMEOUT (API 1.0.0-beta.10, since 0.5.0): a run whose shard died or
28
+ // never started counts once the run timeout has passed, with the shards that
29
+ // finished - "whose shards all finished" stopped being the whole rule.
30
+ //
31
+ // THE MONTHLY RESULT LIMIT (API 1.0.0-beta.11, since 0.5.0): a shard the limit
32
+ // stopped is treated like a timed-out one, so the sentence names it - and says
33
+ // "the results its shards reported", since such a shard did not finish.
34
+ //
35
+ // ONE RUN STATUS SINCE 0.5.0 (API 1.0.0-beta.14): a run's status is the
36
+ // dashboard's own word - passed, failed, cancelled, incomplete or running -
37
+ // and the `cancelled` boolean is gone (`cancellation` says how and when). Said
38
+ // here because a model that knew the old shape looks for a flag that no longer
39
+ // exists, or reads "cancelled" as an unknown value.
40
+ //
20
41
  // Exported as `squally-mcp/instructions` (package.json), so it stays a plain
21
42
  // constant - test/exports.test.js imports it through the package name.
22
43
  export const INSTRUCTIONS = [
23
44
  "Start with squally-list-projects; every other tool needs a projectId from it.",
24
45
  "For one test, use squally-get-test-metrics, not squally-list-tests.",
25
- "Test metrics count the CI runs of the period whose shards all finished - a sharded",
26
- "run counts once - without runs that were a mass failure as a whole; local runs never",
27
- "count. stability = runs that passed on the",
46
+ "Test metrics count the finished CI runs of the period - a sharded run counts once -",
47
+ "without runs that were a mass failure as a whole; pass source=local or source=all",
48
+ "to count local runs or both. A run still in progress counts once all its shards have",
49
+ "finished, or once the run timeout has passed, with the results its shards reported -",
50
+ "also when a shard was stopped by the monthly result limit.",
51
+ "stability = runs that passed on the",
28
52
  "first try / runs; flakyRate = runs that passed only after a retry / runs;",
29
53
  "failureRate = failed runs / runs. There is no verdict: the tools return numbers,",
30
54
  "and you judge them. If a test name is ambiguous, repeat with filePath from the",
31
- "error. Errors carry a code and an action; follow the action.",
55
+ "error. A run's status is passed, failed, cancelled, incomplete or running.",
56
+ "Errors carry a code and an action; follow the action.",
32
57
  ].join(" ");
@@ -3,7 +3,7 @@
3
3
  "jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
4
4
  "info": {
5
5
  "title": "Squally Read API",
6
- "version": "1.0.0-beta.6",
6
+ "version": "1.0.0-beta.14",
7
7
  "description": "Read access to a Squally organization's test results, for scripts, CI jobs and AI agents.\n\n**Authentication.** Every request carries an organization read key as `Authorization: Bearer sqly_ro_…`. A read key is organization-wide (every project), created by an organization admin in Organization › Read keys, and expires after 30, 60, 90, 180 days as chosen at creation. There is no renewal: create a new key and revoke the old one. Project ingest keys (`sqly_…`) are refused with `wrong_key_type`.\n\n**Expiry warning.** From 7 days before a key expires, every authenticated response carries `Squally-Key-Expires-At` with the expiry instant (ISO-8601).\n\n**Rate limit.** Currently 60 requests per 60 seconds per key, counted per `Authorization` header - a read key has its own budget, shared with nothing else. Over the limit the edge answers 429 without a JSON body.\n\n**Retries.** 429: wait 60 seconds, then retry. The edge counts requests per key in 60-second windows and sends no Retry-After header. 5xx: retry with exponential backoff, starting at 1 second and capped at 30 seconds. 400, 401, 403, 404: never retry - the same request gets the same answer.\n\n**Plans.** The read API is part of the Standard and Pro plans. A key of an organization on Free is answered 403 `tier_too_low`; keys are not revoked on a downgrade and work again after an upgrade.\n\n**Errors.** Every error is `{\"error\": \"<sentence>\", \"code\": \"<slug>\"}`. The sentence may be reworded; the slug may not. An id that does not exist and an id that belongs to another organization get the same 404, byte for byte.\n\n**Test names in paths.** Test names are Playwright title paths and often contain `/`: URL-encode them, including `/` as `%2F`.\n\n**Sharded runs** are returned as one logical run: /runs lists one row per logical run, and the run routes answer for all shards whichever shard's id you pass.\n\n**Stability.** v1, beta. The API may change until the beta ends; every change is listed in the changelog (docs/openapi/CHANGELOG.md)."
8
8
  },
9
9
  "servers": [
@@ -337,7 +337,7 @@
337
337
  "get": {
338
338
  "operationId": "listRuns",
339
339
  "summary": "List runs",
340
- "description": "Runs in a period, newest first - one row per logical run (a sharded CI run is one row). Filter by branch, commit SHA prefix or outcome; page with the cursors. Cheap to moderate: one bounded scan.",
340
+ "description": "Runs in a period, newest first - one row per logical run (a sharded CI run is one row), placed in the period and ordered by the start of its first shard (since 1.0.0-beta.13). CI and local runs both, unless source narrows it. Filter by branch, commit SHA prefix, status or source; page with the cursors. Cheap to moderate: bounded by the period's runs.",
341
341
  "tags": [
342
342
  "Runs"
343
343
  ],
@@ -392,15 +392,33 @@
392
392
  "name": "status",
393
393
  "in": "query",
394
394
  "required": false,
395
- "description": "Only runs with this outcome. A cancelled run matches neither value - only the unfiltered list has it.",
395
+ "description": "Only runs with this status - the status field's values. A cancelled run matches cancelled only (since 1.0.0-beta.14; passed and failed were the only values before it, and a cancelled run matched neither).",
396
396
  "schema": {
397
397
  "type": "string",
398
398
  "enum": [
399
399
  "passed",
400
- "failed"
400
+ "failed",
401
+ "cancelled",
402
+ "incomplete",
403
+ "running"
401
404
  ]
402
405
  }
403
406
  },
407
+ {
408
+ "name": "source",
409
+ "in": "query",
410
+ "required": false,
411
+ "description": "Which runs are listed, by where they ran: all - every run (the default); ci - runs a shard reported a CI provider for; local - runs no shard did (each run's local field). Since 1.0.0-beta.14.",
412
+ "schema": {
413
+ "type": "string",
414
+ "enum": [
415
+ "ci",
416
+ "local",
417
+ "all"
418
+ ],
419
+ "default": "all"
420
+ }
421
+ },
404
422
  {
405
423
  "name": "perPage",
406
424
  "in": "query",
@@ -474,8 +492,8 @@
474
492
  "testedRevision": null,
475
493
  "pullRequestNumber": null,
476
494
  "status": "passed",
477
- "cancelled": false,
478
495
  "cancellation": null,
496
+ "local": false,
479
497
  "startedAt": "2026-09-24T07:31:12.000Z",
480
498
  "finishedAt": "2026-09-24T07:39:47.000Z",
481
499
  "durationMs": 515000,
@@ -499,8 +517,8 @@
499
517
  "testedRevision": "c4f7a1e9b3d5c8e2f0a6b4d9e1c7f3a5b8d2e0f6",
500
518
  "pullRequestNumber": 318,
501
519
  "status": "failed",
502
- "cancelled": false,
503
520
  "cancellation": null,
521
+ "local": false,
504
522
  "startedAt": "2026-09-23T16:05:40.000Z",
505
523
  "finishedAt": "2026-09-23T16:12:02.000Z",
506
524
  "durationMs": 382000,
@@ -520,13 +538,13 @@
520
538
  "commitSha": "5f1d9b3e7a2c6e0f8b4d2a9c7e1f5b3d8a6c0e4f",
521
539
  "testedRevision": null,
522
540
  "pullRequestNumber": 318,
523
- "status": "failed",
524
- "cancelled": true,
541
+ "status": "cancelled",
525
542
  "cancellation": {
526
543
  "source": "reporter",
527
544
  "at": "2026-09-23T15:58:31.000Z",
528
545
  "byUserId": null
529
546
  },
547
+ "local": false,
530
548
  "startedAt": "2026-09-23T15:55:02.000Z",
531
549
  "finishedAt": "2026-09-23T15:58:31.000Z",
532
550
  "durationMs": 209000,
@@ -831,8 +849,8 @@
831
849
  "pullRequestTitle": null,
832
850
  "ciProvider": "github-actions",
833
851
  "status": "passed",
834
- "cancelled": false,
835
852
  "cancellation": null,
853
+ "local": false,
836
854
  "startedAt": "2026-09-24T07:31:12.000Z",
837
855
  "finishedAt": "2026-09-24T07:39:47.000Z",
838
856
  "durationMs": 515000,
@@ -1675,7 +1693,7 @@
1675
1693
  "get": {
1676
1694
  "operationId": "listTests",
1677
1695
  "summary": "List tests",
1678
- "description": "Every test with a CI run in the period and its metrics - stability, flaky rate, failure rate, runs, time lost, its last 20 runs, the branches it ran on and where it fails most - ranked by the chosen sort. Counts completed CI runs only, without mass-failure runs, on every branch unless branch names one (see population). Moderate: two queries over the period's CI results, paged in the database; use the metrics route for one test.",
1696
+ "description": "Every test with a run of the source (CI by default) in the period and its metrics - stability, flaky rate, failure rate, runs, time lost, its last 20 runs, the branches it ran on and where it fails most - ranked by the chosen sort. Counts completed runs of the source - CI by default - without mass-failure runs, on every branch unless branch names one (see population). Moderate: two queries over the period's CI results, paged in the database; use the metrics route for one test.",
1679
1697
  "tags": [
1680
1698
  "Tests"
1681
1699
  ],
@@ -1727,6 +1745,21 @@
1727
1745
  "maxLength": 200
1728
1746
  }
1729
1747
  },
1748
+ {
1749
+ "name": "source",
1750
+ "in": "query",
1751
+ "required": false,
1752
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
1753
+ "schema": {
1754
+ "type": "string",
1755
+ "enum": [
1756
+ "ci",
1757
+ "local",
1758
+ "all"
1759
+ ],
1760
+ "default": "ci"
1761
+ }
1762
+ },
1730
1763
  {
1731
1764
  "name": "browser",
1732
1765
  "in": "query",
@@ -2284,7 +2317,7 @@
2284
2317
  "get": {
2285
2318
  "operationId": "getTestMetrics",
2286
2319
  "summary": "Get test metrics",
2287
- "description": "One test's metrics over the period: the fields of its /tests row, flat, beside the same period. A test with results but no CI run in the period answers zeros and null rates rather than 404. Cheap: the list's queries, narrowed to one test; prefer it to /tests when the question is about one test.",
2320
+ "description": "One test's metrics over the period: the fields of its /tests row, flat, beside the same period. A test with results but no run of the source in the period answers zeros and null rates rather than 404. Cheap: the list's queries, narrowed to one test; prefer it to /tests when the question is about one test.",
2288
2321
  "tags": [
2289
2322
  "Tests"
2290
2323
  ],
@@ -2342,6 +2375,21 @@
2342
2375
  "minLength": 1,
2343
2376
  "maxLength": 200
2344
2377
  }
2378
+ },
2379
+ {
2380
+ "name": "source",
2381
+ "in": "query",
2382
+ "required": false,
2383
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
2384
+ "schema": {
2385
+ "type": "string",
2386
+ "enum": [
2387
+ "ci",
2388
+ "local",
2389
+ "all"
2390
+ ],
2391
+ "default": "ci"
2392
+ }
2345
2393
  }
2346
2394
  ],
2347
2395
  "responses": {
@@ -2695,7 +2743,7 @@
2695
2743
  "get": {
2696
2744
  "operationId": "getTestHistory",
2697
2745
  "summary": "Get test history",
2698
- "description": "Get a test's history across runs. One test in every run of a period, newest first: final status, duration, attempts and time spent on retries. Moderate.",
2746
+ "description": "Get a test's history across runs. One test in every run of a period, newest first: final status, duration, attempts and time spent on retries. Every run is listed; source decides which are counted. Moderate.",
2699
2747
  "tags": [
2700
2748
  "Tests"
2701
2749
  ],
@@ -2742,6 +2790,21 @@
2742
2790
  ],
2743
2791
  "default": 30
2744
2792
  }
2793
+ },
2794
+ {
2795
+ "name": "source",
2796
+ "in": "query",
2797
+ "required": false,
2798
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
2799
+ "schema": {
2800
+ "type": "string",
2801
+ "enum": [
2802
+ "ci",
2803
+ "local",
2804
+ "all"
2805
+ ],
2806
+ "default": "ci"
2807
+ }
2745
2808
  }
2746
2809
  ],
2747
2810
  "responses": {
@@ -2808,7 +2871,7 @@
2808
2871
  }
2809
2872
  },
2810
2873
  "400": {
2811
- "description": "Error: ambiguous_test, invalid_period",
2874
+ "description": "Error: ambiguous_test, invalid_parameter, invalid_period",
2812
2875
  "content": {
2813
2876
  "application/json": {
2814
2877
  "schema": {
@@ -2822,6 +2885,7 @@
2822
2885
  "type": "string",
2823
2886
  "enum": [
2824
2887
  "ambiguous_test",
2888
+ "invalid_parameter",
2825
2889
  "invalid_period"
2826
2890
  ],
2827
2891
  "description": "Stable slug - match on this, not on the sentence."
@@ -2840,6 +2904,13 @@
2840
2904
  "code": "ambiguous_test"
2841
2905
  }
2842
2906
  },
2907
+ "invalid_parameter": {
2908
+ "summary": "invalid_parameter",
2909
+ "value": {
2910
+ "error": "sha must be 4 to 40 hexadecimal characters (a commit SHA or a prefix of one).",
2911
+ "code": "invalid_parameter"
2912
+ }
2913
+ },
2843
2914
  "invalid_period": {
2844
2915
  "summary": "invalid_period",
2845
2916
  "value": {
@@ -3045,7 +3116,7 @@
3045
3116
  "get": {
3046
3117
  "operationId": "listErrors",
3047
3118
  "summary": "List errors",
3048
- "description": "List errors. The project's errors in a period - failures grouped by fingerprint - most frequent first. Every count covers the period; firstSeen is the first occurrence still stored, not bounded by it. Moderate: one aggregate query.",
3119
+ "description": "List errors. The project's errors in a period - failures grouped by fingerprint - most frequent first, over the finished runs of the source (see population), all shards of a run one run. Every count covers the period; firstSeen is the first occurrence still stored, not bounded by it. Moderate: one aggregate query.",
3049
3120
  "tags": [
3050
3121
  "Errors"
3051
3122
  ],
@@ -3075,6 +3146,21 @@
3075
3146
  "default": 30
3076
3147
  }
3077
3148
  },
3149
+ {
3150
+ "name": "source",
3151
+ "in": "query",
3152
+ "required": false,
3153
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
3154
+ "schema": {
3155
+ "type": "string",
3156
+ "enum": [
3157
+ "ci",
3158
+ "local",
3159
+ "all"
3160
+ ],
3161
+ "default": "ci"
3162
+ }
3163
+ },
3078
3164
  {
3079
3165
  "name": "page",
3080
3166
  "in": "query",
@@ -3120,6 +3206,7 @@
3120
3206
  "id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
3121
3207
  "name": "storefront-web"
3122
3208
  },
3209
+ "population": "ci_groups_finished",
3123
3210
  "window": {
3124
3211
  "days": 30,
3125
3212
  "from": "2026-08-25T08:00:00.000Z",
@@ -3145,12 +3232,15 @@
3145
3232
  ],
3146
3233
  "occurrenceCount": 9,
3147
3234
  "runCount": 7,
3235
+ "failedCount": 5,
3236
+ "flakyCount": 2,
3237
+ "massFailureRunCount": 0,
3148
3238
  "firstSeen": "2026-09-02T11:20:44.000Z",
3149
- "lastSeen": "2026-09-24T07:34:05.000Z",
3150
- "isNew": true
3239
+ "lastSeen": "2026-09-24T07:34:05.000Z"
3151
3240
  }
3152
3241
  ],
3153
- "unfingerprintedCount": 0
3242
+ "unfingerprintedCount": 0,
3243
+ "unfinishedRuns": 0
3154
3244
  }
3155
3245
  }
3156
3246
  }
@@ -3385,7 +3475,7 @@
3385
3475
  "get": {
3386
3476
  "operationId": "getError",
3387
3477
  "summary": "Get an error",
3388
- "description": "Get an error. One error: its counts, the tests it hits and the runs it occurred in. An error quiet in the period is answered over everything still stored (windowEmpty: true). Moderate.",
3478
+ "description": "Get an error. One error: its counts, the tests it hits and the runs it occurred in - one row per run and test, by the run's id as /runs gives it. An error quiet in the period is answered over everything still stored (windowEmpty: true). Moderate.",
3389
3479
  "tags": [
3390
3480
  "Errors"
3391
3481
  ],
@@ -3424,6 +3514,21 @@
3424
3514
  ],
3425
3515
  "default": 30
3426
3516
  }
3517
+ },
3518
+ {
3519
+ "name": "source",
3520
+ "in": "query",
3521
+ "required": false,
3522
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
3523
+ "schema": {
3524
+ "type": "string",
3525
+ "enum": [
3526
+ "ci",
3527
+ "local",
3528
+ "all"
3529
+ ],
3530
+ "default": "ci"
3531
+ }
3427
3532
  }
3428
3533
  ],
3429
3534
  "responses": {
@@ -3444,6 +3549,7 @@
3444
3549
  "id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
3445
3550
  "name": "storefront-web"
3446
3551
  },
3552
+ "population": "ci_groups_finished",
3447
3553
  "window": {
3448
3554
  "days": 30,
3449
3555
  "from": "2026-08-25T08:00:00.000Z",
@@ -3462,16 +3568,21 @@
3462
3568
  ],
3463
3569
  "occurrenceCount": 9,
3464
3570
  "runCount": 7,
3571
+ "failedCount": 5,
3572
+ "flakyCount": 2,
3573
+ "massFailureRunCount": 0,
3465
3574
  "firstSeen": "2026-09-02T11:20:44.000Z",
3466
- "lastSeen": "2026-09-24T07:34:05.000Z",
3467
- "isNew": true
3575
+ "lastSeen": "2026-09-24T07:34:05.000Z"
3468
3576
  },
3469
3577
  "windowEmpty": false,
3470
3578
  "tests": [
3471
3579
  {
3472
3580
  "testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code",
3473
3581
  "filePath": "tests/checkout.spec.ts",
3474
- "occurrenceCount": 9
3582
+ "occurrenceCount": 9,
3583
+ "runCount": 7,
3584
+ "failedCount": 5,
3585
+ "flakyCount": 2
3475
3586
  }
3476
3587
  ],
3477
3588
  "runs": [
@@ -3484,7 +3595,8 @@
3484
3595
  "filePath": "tests/checkout.spec.ts",
3485
3596
  "attempts": 2,
3486
3597
  "occurrences": 1,
3487
- "finalStatus": "passed"
3598
+ "finalStatus": "passed",
3599
+ "massFailure": false
3488
3600
  }
3489
3601
  ]
3490
3602
  }
@@ -3492,7 +3604,7 @@
3492
3604
  }
3493
3605
  },
3494
3606
  "400": {
3495
- "description": "Error: invalid_period",
3607
+ "description": "Error: invalid_parameter, invalid_period",
3496
3608
  "content": {
3497
3609
  "application/json": {
3498
3610
  "schema": {
@@ -3505,6 +3617,7 @@
3505
3617
  "code": {
3506
3618
  "type": "string",
3507
3619
  "enum": [
3620
+ "invalid_parameter",
3508
3621
  "invalid_period"
3509
3622
  ],
3510
3623
  "description": "Stable slug - match on this, not on the sentence."
@@ -3516,6 +3629,13 @@
3516
3629
  ]
3517
3630
  },
3518
3631
  "examples": {
3632
+ "invalid_parameter": {
3633
+ "summary": "invalid_parameter",
3634
+ "value": {
3635
+ "error": "sha must be 4 to 40 hexadecimal characters (a commit SHA or a prefix of one).",
3636
+ "code": "invalid_parameter"
3637
+ }
3638
+ },
3519
3639
  "invalid_period": {
3520
3640
  "summary": "invalid_period",
3521
3641
  "value": {
@@ -3721,7 +3841,7 @@
3721
3841
  "get": {
3722
3842
  "operationId": "getOverview",
3723
3843
  "summary": "Get dashboard figures",
3724
- "description": "The dashboard for a period: last run, suite stability, time lost, the two test-metric tiles (testMetrics), slowest tests, time sinks, recent runs. Suite stability and testMetrics count completed CI runs only. Moderate: one pass over the period's runs and one count over its CI results.",
3844
+ "description": "The dashboard for a period: last run, suite stability, time lost, the two test-metric tiles (testMetrics), slowest tests, time sinks, recent runs. Every figure counts finished runs of the source - CI by default - and testMetrics.population names it. Moderate: one pass over the period's runs and one count over its results.",
3725
3845
  "tags": [
3726
3846
  "Project"
3727
3847
  ],
@@ -3764,6 +3884,21 @@
3764
3884
  ],
3765
3885
  "default": "stable"
3766
3886
  }
3887
+ },
3888
+ {
3889
+ "name": "source",
3890
+ "in": "query",
3891
+ "required": false,
3892
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
3893
+ "schema": {
3894
+ "type": "string",
3895
+ "enum": [
3896
+ "ci",
3897
+ "local",
3898
+ "all"
3899
+ ],
3900
+ "default": "ci"
3901
+ }
3767
3902
  }
3768
3903
  ],
3769
3904
  "responses": {
@@ -3799,11 +3934,14 @@
3799
3934
  "startedAt": "2026-09-24T07:31:12.000Z",
3800
3935
  "finishedAt": "2026-09-24T07:39:47.000Z",
3801
3936
  "total": 412,
3802
- "passed": 408,
3937
+ "passed": 404,
3938
+ "flaky": 4,
3803
3939
  "failed": 0,
3940
+ "skipped": 4,
3804
3941
  "isFallbackBranch": false
3805
3942
  },
3806
3943
  "runsInWindow": 11,
3944
+ "unfinishedRuns": 1,
3807
3945
  "suiteSize": 412,
3808
3946
  "windowStability": 99,
3809
3947
  "timeLost": {
@@ -4469,19 +4607,15 @@
4469
4607
  ]
4470
4608
  },
4471
4609
  "status": {
4472
- "anyOf": [
4473
- {
4474
- "type": "string"
4475
- },
4476
- {
4477
- "type": "null"
4478
- }
4610
+ "type": "string",
4611
+ "enum": [
4612
+ "passed",
4613
+ "failed",
4614
+ "cancelled",
4615
+ "incomplete",
4616
+ "running"
4479
4617
  ],
4480
- "description": "passed | failed | running | incomplete - for a sharded run, the whole group's. Never \"cancelled\": a cancelled run keeps the outcome of what it reported, and says so in cancelled."
4481
- },
4482
- "cancelled": {
4483
- "type": "boolean",
4484
- "description": "True when the run was cancelled - for a sharded run, when any shard recorded a cancel. A cancelled run keeps its status (the outcome of what it reported before the stop) and never matches ?status=passed or ?status=failed."
4618
+ "description": "How the run ended, the dashboard's own word: cancelled - the run was cancelled (for a sharded run, any shard recorded a cancel; cancellation says how and when); incomplete - a shard timed out, was stopped by the monthly result limit or never reported; running - a shard is still running; failed - every shard finished and a test failed; passed - every shard finished and none failed. In that order: the first that applies. Since 1.0.0-beta.14 - before it, a cancelled run reported the outcome beneath the cancel and a cancelled boolean."
4485
4619
  },
4486
4620
  "cancellation": {
4487
4621
  "anyOf": [
@@ -4526,7 +4660,11 @@
4526
4660
  "type": "null"
4527
4661
  }
4528
4662
  ],
4529
- "description": "How and when the run was cancelled; null when it was not."
4663
+ "description": "How and when the run was cancelled; null when it was not - set exactly when status is cancelled."
4664
+ },
4665
+ "local": {
4666
+ "type": "boolean",
4667
+ "description": "No shard of the run reported a CI provider: a local run (since 1.0.0-beta.14). GET /runs lists both kinds unless source says otherwise."
4530
4668
  },
4531
4669
  "startedAt": {
4532
4670
  "type": "string",
@@ -4558,7 +4696,8 @@
4558
4696
  "type": "object",
4559
4697
  "properties": {
4560
4698
  "total": {
4561
- "type": "integer"
4699
+ "type": "integer",
4700
+ "description": "Tests in the run, every test once however many shards reported it - its final attempt across all of them decides (highest retry, then the lowest shard, then the first repetition). total = passed + failed + skipped + flaky. Since 1.0.0-beta.12; until then each shard's count was summed, so a test reported from two shards counted twice. In a cancelled run the tests the cancel interrupted - reported as skipped - count nowhere, so skipped there counts no test (since 1.0.0-beta.13); the run's tests list still has them."
4562
4701
  },
4563
4702
  "passed": {
4564
4703
  "type": "integer"
@@ -4620,8 +4759,8 @@
4620
4759
  "testedRevision",
4621
4760
  "pullRequestNumber",
4622
4761
  "status",
4623
- "cancelled",
4624
4762
  "cancellation",
4763
+ "local",
4625
4764
  "startedAt",
4626
4765
  "finishedAt",
4627
4766
  "durationMs",
@@ -4777,11 +4916,14 @@
4777
4916
  },
4778
4917
  "status": {
4779
4918
  "type": "string",
4780
- "description": "passed | failed | running | incomplete - the whole group's. Never \"cancelled\": a cancelled run keeps the outcome of what it reported, and says so in cancelled."
4781
- },
4782
- "cancelled": {
4783
- "type": "boolean",
4784
- "description": "True when the run was cancelled - for a sharded run, when any shard recorded a cancel. A cancelled run keeps its status (the outcome of what it reported before the stop) and never matches ?status=passed or ?status=failed."
4919
+ "enum": [
4920
+ "passed",
4921
+ "failed",
4922
+ "cancelled",
4923
+ "incomplete",
4924
+ "running"
4925
+ ],
4926
+ "description": "How the run ended, the dashboard's own word: cancelled - the run was cancelled (for a sharded run, any shard recorded a cancel; cancellation says how and when); incomplete - a shard timed out, was stopped by the monthly result limit or never reported; running - a shard is still running; failed - every shard finished and a test failed; passed - every shard finished and none failed. In that order: the first that applies. Since 1.0.0-beta.14 - before it, a cancelled run reported the outcome beneath the cancel and a cancelled boolean."
4785
4927
  },
4786
4928
  "cancellation": {
4787
4929
  "anyOf": [
@@ -4826,7 +4968,11 @@
4826
4968
  "type": "null"
4827
4969
  }
4828
4970
  ],
4829
- "description": "How and when the run was cancelled; null when it was not."
4971
+ "description": "How and when the run was cancelled; null when it was not - set exactly when status is cancelled."
4972
+ },
4973
+ "local": {
4974
+ "type": "boolean",
4975
+ "description": "No shard of the run reported a CI provider: a local run (since 1.0.0-beta.14). GET /runs lists both kinds unless source says otherwise."
4830
4976
  },
4831
4977
  "startedAt": {
4832
4978
  "type": "string",
@@ -4891,14 +5037,15 @@
4891
5037
  "type": "string"
4892
5038
  },
4893
5039
  "status": {
4894
- "anyOf": [
4895
- {
4896
- "type": "string"
4897
- },
4898
- {
4899
- "type": "null"
4900
- }
4901
- ]
5040
+ "type": "string",
5041
+ "enum": [
5042
+ "passed",
5043
+ "failed",
5044
+ "cancelled",
5045
+ "incomplete",
5046
+ "running"
5047
+ ],
5048
+ "description": "What this shard did, in the run status's words: cancelled when it recorded a cancel; else running or incomplete by its state; else failed when a test failed, passed otherwise. The shards panel's own rule (since 1.0.0-beta.14 - it was the shard's raw outcome, null until it finished)."
4902
5049
  }
4903
5050
  },
4904
5051
  "required": [
@@ -4933,8 +5080,8 @@
4933
5080
  "pullRequestTitle",
4934
5081
  "ciProvider",
4935
5082
  "status",
4936
- "cancelled",
4937
5083
  "cancellation",
5084
+ "local",
4938
5085
  "startedAt",
4939
5086
  "finishedAt",
4940
5087
  "durationMs",
@@ -4945,7 +5092,8 @@
4945
5092
  "type": "object",
4946
5093
  "properties": {
4947
5094
  "total": {
4948
- "type": "integer"
5095
+ "type": "integer",
5096
+ "description": "Tests in the run, every test once however many shards reported it - its final attempt across all of them decides (highest retry, then the lowest shard, then the first repetition). total = passed + failed + skipped + flaky. Since 1.0.0-beta.12; until then each shard's count was summed, so a test reported from two shards counted twice. In a cancelled run the tests the cancel interrupted - reported as skipped - count nowhere, so skipped there counts no test (since 1.0.0-beta.13); the run's tests list still has them."
4949
5097
  },
4950
5098
  "passed": {
4951
5099
  "type": "integer"
@@ -4991,7 +5139,8 @@
4991
5139
  "type": "string"
4992
5140
  },
4993
5141
  "attempts": {
4994
- "type": "integer"
5142
+ "type": "integer",
5143
+ "description": "Every attempt of the test in the run, across all its shards (since 1.0.0-beta.12; per shard before)."
4995
5144
  },
4996
5145
  "durationMs": {
4997
5146
  "anyOf": [
@@ -5082,7 +5231,8 @@
5082
5231
  {
5083
5232
  "type": "null"
5084
5233
  }
5085
- ]
5234
+ ],
5235
+ "description": "The shard the test's final attempt ran in; null for an unsharded run. One row per test since 1.0.0-beta.12 - a test reported from two shards was one row per shard before."
5086
5236
  }
5087
5237
  },
5088
5238
  "required": [
@@ -5352,7 +5502,8 @@
5352
5502
  {
5353
5503
  "type": "null"
5354
5504
  }
5355
- ]
5505
+ ],
5506
+ "description": "The shard the test's final attempt ran in; null for an unsharded run. One row per test since 1.0.0-beta.12 - a test reported from two shards was one row per shard before."
5356
5507
  }
5357
5508
  },
5358
5509
  "required": [
@@ -5427,8 +5578,12 @@
5427
5578
  },
5428
5579
  "population": {
5429
5580
  "type": "string",
5430
- "const": "ci_groups_finished_not_mass_failure",
5431
- "description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
5581
+ "enum": [
5582
+ "ci_groups_finished_not_mass_failure",
5583
+ "local_groups_finished_not_mass_failure",
5584
+ "all_groups_finished_not_mass_failure"
5585
+ ],
5586
+ "description": "Which runs every number counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. A run finishes when all its shards have, or - once it is overdue by the project's run timeout, counted from its latest shard's start - when every shard that started has completed, timed out or been stopped by the monthly result limit; such a run counts like a cancelled one, with the results its shards reported (since 1.0.0-beta.10; the monthly result limit since 1.0.0-beta.11). Runs still running are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.8; before it, ci_groups_finished_not_mass_failure was the only value, and since 1.0.0-beta.6 - it was ci_completed_not_excluded, which counted every completed shard on its own)."
5432
5587
  },
5433
5588
  "days": {
5434
5589
  "type": "integer",
@@ -5728,8 +5883,12 @@
5728
5883
  },
5729
5884
  "population": {
5730
5885
  "type": "string",
5731
- "const": "ci_groups_finished_not_mass_failure",
5732
- "description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
5886
+ "enum": [
5887
+ "ci_groups_finished_not_mass_failure",
5888
+ "local_groups_finished_not_mass_failure",
5889
+ "all_groups_finished_not_mass_failure"
5890
+ ],
5891
+ "description": "Which runs every number counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. A run finishes when all its shards have, or - once it is overdue by the project's run timeout, counted from its latest shard's start - when every shard that started has completed, timed out or been stopped by the monthly result limit; such a run counts like a cancelled one, with the results its shards reported (since 1.0.0-beta.10; the monthly result limit since 1.0.0-beta.11). Runs still running are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.8; before it, ci_groups_finished_not_mass_failure was the only value, and since 1.0.0-beta.6 - it was ci_completed_not_excluded, which counted every completed shard on its own)."
5733
5892
  },
5734
5893
  "days": {
5735
5894
  "type": "integer",
@@ -6033,11 +6192,11 @@
6033
6192
  },
6034
6193
  "local": {
6035
6194
  "type": "boolean",
6036
- "description": "No shard of the run reported a CI provider: a local run. Listed, never counted in a metric (since 1.0.0-beta.6)."
6195
+ "description": "No shard of the run reported a CI provider: a local run. Listed always; counted only under source=local or source=all (since 1.0.0-beta.8; never counted from 1.0.0-beta.6)."
6037
6196
  },
6038
6197
  "counted": {
6039
6198
  "type": "boolean",
6040
- "description": "Whether the run is in the metrics' population (see population on the metrics endpoints): a CI run whose shards all finished, not a mass failure, and the test not just skipped by a cancellation (since 1.0.0-beta.6)."
6199
+ "description": "Whether the run is in the metrics' population (see population on the metrics endpoints): a run of the source parameter's kind - CI by default - that finished (all its shards, or overdue with every shard that started completed, timed out or stopped by the monthly result limit - since 1.0.0-beta.10, the limit since 1.0.0-beta.11), not a mass failure, and the test not just skipped by a cancellation (since 1.0.0-beta.6; the source since 1.0.0-beta.8)."
6041
6200
  },
6042
6201
  "branch": {
6043
6202
  "anyOf": [
@@ -6150,6 +6309,15 @@
6150
6309
  "name"
6151
6310
  ]
6152
6311
  },
6312
+ "population": {
6313
+ "type": "string",
6314
+ "enum": [
6315
+ "ci_groups_finished",
6316
+ "local_groups_finished",
6317
+ "all_groups_finished"
6318
+ ],
6319
+ "description": "Which runs every error figure counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - runs that were a mass failure as a whole included (see massFailureRunCount). A run finishes when all its shards have, or - once it is overdue by the project's run timeout, counted from its latest shard's start - when every shard that started has completed, timed out or been stopped by the monthly result limit; such a run counts like a cancelled one, with the results its shards reported (since 1.0.0-beta.10; the monthly result limit since 1.0.0-beta.11). Runs still running are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.9)."
6320
+ },
6153
6321
  "window": {
6154
6322
  "type": "object",
6155
6323
  "properties": {
@@ -6237,19 +6405,29 @@
6237
6405
  "type": "integer"
6238
6406
  },
6239
6407
  "runCount": {
6240
- "type": "integer"
6408
+ "type": "integer",
6409
+ "description": "Runs of the population with at least one such attempt in the window - all shards of a run one run (since 1.0.0-beta.9; it counted every shard row)."
6410
+ },
6411
+ "failedCount": {
6412
+ "type": "integer",
6413
+ "description": "Of the (run, test) pairs the error hit, how many ended failed: the test's final attempt across all the run's shards failed (since 1.0.0-beta.9)."
6414
+ },
6415
+ "flakyCount": {
6416
+ "type": "integer",
6417
+ "description": "Of the same pairs, how many ended flaky: the final attempt passed after a retry. A pair whose final attempt skipped is neither (since 1.0.0-beta.9)."
6418
+ },
6419
+ "massFailureRunCount": {
6420
+ "type": "integer",
6421
+ "description": "Of runCount, the runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed) - counted, and named here (since 1.0.0-beta.9)."
6241
6422
  },
6242
6423
  "firstSeen": {
6243
6424
  "type": "string",
6244
6425
  "format": "date-time",
6245
- "description": "First seen in what is still stored: the earliest occurrence the plan's retention has kept, NOT bounded by the requested period. ISO-8601."
6426
+ "description": "First seen in what is still stored: the start of the earliest run of the population, as far back as the plan's retention keeps them, NOT bounded by the requested period. ISO-8601. (isNew, \"absent in the period before\", was removed in 1.0.0-beta.9: compare firstSeen with window.from.)"
6246
6427
  },
6247
6428
  "lastSeen": {
6248
6429
  "type": "string",
6249
6430
  "format": "date-time"
6250
- },
6251
- "isNew": {
6252
- "type": "boolean"
6253
6431
  }
6254
6432
  },
6255
6433
  "required": [
@@ -6263,22 +6441,31 @@
6263
6441
  "testNames",
6264
6442
  "occurrenceCount",
6265
6443
  "runCount",
6444
+ "failedCount",
6445
+ "flakyCount",
6446
+ "massFailureRunCount",
6266
6447
  "firstSeen",
6267
- "lastSeen",
6268
- "isNew"
6448
+ "lastSeen"
6269
6449
  ]
6270
6450
  }
6271
6451
  },
6272
6452
  "unfingerprintedCount": {
6273
- "type": "integer"
6453
+ "type": "integer",
6454
+ "description": "Failed attempts in the window's runs of the population that carry no signature and are not grouped (the window only since 1.0.0-beta.9; it read the period before too). Expected 0."
6455
+ },
6456
+ "unfinishedRuns": {
6457
+ "type": "integer",
6458
+ "description": "Runs of the source in the window that have not finished - a shard still running, or one timed out, stopped by the monthly result limit or never reported while the run is not yet overdue (an overdue one is finished and counted since 1.0.0-beta.10; a limit-stopped one since 1.0.0-beta.11). In no figure (since 1.0.0-beta.9)."
6274
6459
  }
6275
6460
  },
6276
6461
  "required": [
6277
6462
  "project",
6463
+ "population",
6278
6464
  "window",
6279
6465
  "page",
6280
6466
  "errors",
6281
- "unfingerprintedCount"
6467
+ "unfingerprintedCount",
6468
+ "unfinishedRuns"
6282
6469
  ]
6283
6470
  },
6284
6471
  "ErrorDetailResponse": {
@@ -6299,6 +6486,15 @@
6299
6486
  "name"
6300
6487
  ]
6301
6488
  },
6489
+ "population": {
6490
+ "type": "string",
6491
+ "enum": [
6492
+ "ci_groups_finished",
6493
+ "local_groups_finished",
6494
+ "all_groups_finished"
6495
+ ],
6496
+ "description": "Which runs every error figure counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - runs that were a mass failure as a whole included (see massFailureRunCount). A run finishes when all its shards have, or - once it is overdue by the project's run timeout, counted from its latest shard's start - when every shard that started has completed, timed out or been stopped by the monthly result limit; such a run counts like a cancelled one, with the results its shards reported (since 1.0.0-beta.10; the monthly result limit since 1.0.0-beta.11). Runs still running are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.9)."
6497
+ },
6302
6498
  "window": {
6303
6499
  "type": "object",
6304
6500
  "properties": {
@@ -6361,19 +6557,29 @@
6361
6557
  "type": "integer"
6362
6558
  },
6363
6559
  "runCount": {
6364
- "type": "integer"
6560
+ "type": "integer",
6561
+ "description": "Runs of the population with at least one such attempt in the window - all shards of a run one run (since 1.0.0-beta.9; it counted every shard row)."
6562
+ },
6563
+ "failedCount": {
6564
+ "type": "integer",
6565
+ "description": "Of the (run, test) pairs the error hit, how many ended failed: the test's final attempt across all the run's shards failed (since 1.0.0-beta.9)."
6566
+ },
6567
+ "flakyCount": {
6568
+ "type": "integer",
6569
+ "description": "Of the same pairs, how many ended flaky: the final attempt passed after a retry. A pair whose final attempt skipped is neither (since 1.0.0-beta.9)."
6570
+ },
6571
+ "massFailureRunCount": {
6572
+ "type": "integer",
6573
+ "description": "Of runCount, the runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed) - counted, and named here (since 1.0.0-beta.9)."
6365
6574
  },
6366
6575
  "firstSeen": {
6367
6576
  "type": "string",
6368
6577
  "format": "date-time",
6369
- "description": "First seen in what is still stored: the earliest occurrence the plan's retention has kept, NOT bounded by the requested period. ISO-8601."
6578
+ "description": "First seen in what is still stored: the start of the earliest run of the population, as far back as the plan's retention keeps them, NOT bounded by the requested period. ISO-8601. (isNew, \"absent in the period before\", was removed in 1.0.0-beta.9: compare firstSeen with window.from.)"
6370
6579
  },
6371
6580
  "lastSeen": {
6372
6581
  "type": "string",
6373
6582
  "format": "date-time"
6374
- },
6375
- "isNew": {
6376
- "type": "boolean"
6377
6583
  }
6378
6584
  },
6379
6585
  "required": [
@@ -6387,9 +6593,11 @@
6387
6593
  "testNames",
6388
6594
  "occurrenceCount",
6389
6595
  "runCount",
6596
+ "failedCount",
6597
+ "flakyCount",
6598
+ "massFailureRunCount",
6390
6599
  "firstSeen",
6391
- "lastSeen",
6392
- "isNew"
6600
+ "lastSeen"
6393
6601
  ]
6394
6602
  },
6395
6603
  "windowEmpty": {
@@ -6416,12 +6624,27 @@
6416
6624
  },
6417
6625
  "occurrenceCount": {
6418
6626
  "type": "integer"
6627
+ },
6628
+ "runCount": {
6629
+ "type": "integer",
6630
+ "description": "Runs in which this test hit the error (since 1.0.0-beta.9)."
6631
+ },
6632
+ "failedCount": {
6633
+ "type": "integer",
6634
+ "description": "Of those runs, the ones the test ended failed (since 1.0.0-beta.9)."
6635
+ },
6636
+ "flakyCount": {
6637
+ "type": "integer",
6638
+ "description": "Of those runs, the ones the test ended flaky - passed after a retry (since 1.0.0-beta.9)."
6419
6639
  }
6420
6640
  },
6421
6641
  "required": [
6422
6642
  "testName",
6423
6643
  "filePath",
6424
- "occurrenceCount"
6644
+ "occurrenceCount",
6645
+ "runCount",
6646
+ "failedCount",
6647
+ "flakyCount"
6425
6648
  ]
6426
6649
  }
6427
6650
  },
@@ -6431,7 +6654,8 @@
6431
6654
  "type": "object",
6432
6655
  "properties": {
6433
6656
  "runId": {
6434
- "type": "string"
6657
+ "type": "string",
6658
+ "description": "The run, by the id GET /projects/{projectId}/runs gives it - one row per run and test, however many shards the run had (since 1.0.0-beta.9; it was one row per shard, with the shard's id)."
6435
6659
  },
6436
6660
  "runNumber": {
6437
6661
  "type": "integer"
@@ -6464,13 +6688,20 @@
6464
6688
  ]
6465
6689
  },
6466
6690
  "attempts": {
6467
- "type": "integer"
6691
+ "type": "integer",
6692
+ "description": "Every attempt of the test in the run, retries and every shard included."
6468
6693
  },
6469
6694
  "occurrences": {
6470
- "type": "integer"
6695
+ "type": "integer",
6696
+ "description": "How many of those carry this error."
6471
6697
  },
6472
6698
  "finalStatus": {
6473
- "type": "string"
6699
+ "type": "string",
6700
+ "description": "The test's final attempt across all the run's shards, through test.fail(): passed, failed or skipped."
6701
+ },
6702
+ "massFailure": {
6703
+ "type": "boolean",
6704
+ "description": "The run was a mass failure as a whole (since 1.0.0-beta.9)."
6474
6705
  }
6475
6706
  },
6476
6707
  "required": [
@@ -6482,13 +6713,15 @@
6482
6713
  "filePath",
6483
6714
  "attempts",
6484
6715
  "occurrences",
6485
- "finalStatus"
6716
+ "finalStatus",
6717
+ "massFailure"
6486
6718
  ]
6487
6719
  }
6488
6720
  }
6489
6721
  },
6490
6722
  "required": [
6491
6723
  "project",
6724
+ "population",
6492
6725
  "window",
6493
6726
  "signature",
6494
6727
  "windowEmpty",
@@ -6551,7 +6784,8 @@
6551
6784
  "type": "object",
6552
6785
  "properties": {
6553
6786
  "id": {
6554
- "type": "string"
6787
+ "type": "string",
6788
+ "description": "The run's id as GET /projects/{projectId}/runs gives it - one id for all its shards."
6555
6789
  },
6556
6790
  "runNumber": {
6557
6791
  "type": "integer"
@@ -6567,18 +6801,19 @@
6567
6801
  ]
6568
6802
  },
6569
6803
  "status": {
6570
- "anyOf": [
6571
- {
6572
- "type": "string"
6573
- },
6574
- {
6575
- "type": "null"
6576
- }
6577
- ]
6804
+ "type": "string",
6805
+ "enum": [
6806
+ "passed",
6807
+ "failed",
6808
+ "cancelled",
6809
+ "incomplete"
6810
+ ],
6811
+ "description": "The run's status over all its shards, as GET /runs reports it - a finished run, so never running: passed, failed, cancelled, or incomplete for a run finished by the run timeout - a shard timed out, was stopped by the monthly result limit or never reported (incomplete since 1.0.0-beta.10, the limit since 1.0.0-beta.11; cancelled since 1.0.0-beta.14 - before it, a cancelled run reported the outcome beneath the cancel)."
6578
6812
  },
6579
6813
  "startedAt": {
6580
6814
  "type": "string",
6581
- "format": "date-time"
6815
+ "format": "date-time",
6816
+ "description": "Its first shard's start."
6582
6817
  },
6583
6818
  "finishedAt": {
6584
6819
  "anyOf": [
@@ -6589,17 +6824,28 @@
6589
6824
  {
6590
6825
  "type": "null"
6591
6826
  }
6592
- ]
6827
+ ],
6828
+ "description": "Its last shard's finish."
6593
6829
  },
6594
6830
  "total": {
6595
- "type": "integer"
6831
+ "type": "integer",
6832
+ "description": "Tests in the run, every test once over all shards: passed + flaky + failed + skipped."
6596
6833
  },
6597
6834
  "passed": {
6598
- "type": "integer"
6835
+ "type": "integer",
6836
+ "description": "Tests that passed on the first try."
6837
+ },
6838
+ "flaky": {
6839
+ "type": "integer",
6840
+ "description": "Tests that passed after a retry (since 1.0.0-beta.7)."
6599
6841
  },
6600
6842
  "failed": {
6601
6843
  "type": "integer"
6602
6844
  },
6845
+ "skipped": {
6846
+ "type": "integer",
6847
+ "description": "Since 1.0.0-beta.7."
6848
+ },
6603
6849
  "isFallbackBranch": {
6604
6850
  "type": "boolean"
6605
6851
  }
@@ -6613,21 +6859,29 @@
6613
6859
  "finishedAt",
6614
6860
  "total",
6615
6861
  "passed",
6862
+ "flaky",
6616
6863
  "failed",
6864
+ "skipped",
6617
6865
  "isFallbackBranch"
6618
6866
  ]
6619
6867
  },
6620
6868
  {
6621
6869
  "type": "null"
6622
6870
  }
6623
- ]
6871
+ ],
6872
+ "description": "The newest finished run of the source on branchScope, whatever the window - falling back to any branch when the stable branch has none (see the source parameter and testMetrics.population)."
6624
6873
  },
6625
6874
  "runsInWindow": {
6626
6875
  "type": "integer",
6627
- "description": "Completed CI runs in the window on branchScope, without mass-failure runs - the runs windowStability is counted over."
6876
+ "description": "Finished runs of the source in the window on branchScope (see the source parameter and testMetrics.population), each counted once however many shards it had, by the start of its first shard; mass-failure, cancelled and incomplete runs included (incomplete - finished by the run timeout - since 1.0.0-beta.10). The runs windowStability is counted over."
6877
+ },
6878
+ "unfinishedRuns": {
6879
+ "type": "integer",
6880
+ "description": "Runs of the source in the window on branchScope that have not finished - a shard still running, or one timed out, stopped by the monthly result limit or never reported while the run is not yet overdue (an overdue one is finished and counted since 1.0.0-beta.10; a limit-stopped one since 1.0.0-beta.11). In no other figure (since 1.0.0-beta.7; of the source since 1.0.0-beta.8)."
6628
6881
  },
6629
6882
  "suiteSize": {
6630
- "type": "integer"
6883
+ "type": "integer",
6884
+ "description": "The largest run in the window, every test once over all its shards."
6631
6885
  },
6632
6886
  "windowStability": {
6633
6887
  "anyOf": [
@@ -6638,7 +6892,7 @@
6638
6892
  "type": "null"
6639
6893
  }
6640
6894
  ],
6641
- "description": "Suite stability, 0-100: of the final test results in the window's completed CI runs on branchScope (mass-failure runs and skipped results left out), the share that passed on the first try, rounded. A result that passed only after a retry counts against it. null when no such result exists."
6895
+ "description": "Suite stability, 0-100: of the final test results in the window's finished runs of the source on branchScope (skipped results left out), the share that passed on the first try, rounded. Mass-failure runs count (since 1.0.0-beta.7). A result that passed only after a retry counts against it. null when no such result exists."
6642
6896
  },
6643
6897
  "timeLost": {
6644
6898
  "type": "object",
@@ -6668,7 +6922,8 @@
6668
6922
  "lostPercent",
6669
6923
  "wastedAttempts",
6670
6924
  "wallClockMs"
6671
- ]
6925
+ ],
6926
+ "description": "Time spent on attempts a later retry superseded, in the window's finished runs of the source on branchScope; wallClockMs is the time those runs used - each shard's own duration, summed - and lostPercent is lostMs of it."
6672
6927
  },
6673
6928
  "failingTests": {
6674
6929
  "type": "object",
@@ -6684,15 +6939,19 @@
6684
6939
  "failing",
6685
6940
  "testsRun"
6686
6941
  ],
6687
- "description": "Tests that failed at least once in the window on branchScope, out of the tests those runs reported - every run, local ones included. Not the dashboard's Failing tests tile: that is testMetrics.failingTests, over CI runs on every branch."
6942
+ "description": "Tests whose final attempt failed in at least one of the window's finished runs of the source on branchScope, out of the tests those runs reported (a skipped test counts as reported)."
6688
6943
  },
6689
6944
  "testMetrics": {
6690
6945
  "type": "object",
6691
6946
  "properties": {
6692
6947
  "population": {
6693
6948
  "type": "string",
6694
- "const": "ci_groups_finished_not_mass_failure",
6695
- "description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
6949
+ "enum": [
6950
+ "ci_groups_finished_not_mass_failure",
6951
+ "local_groups_finished_not_mass_failure",
6952
+ "all_groups_finished_not_mass_failure"
6953
+ ],
6954
+ "description": "Which runs every number counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. A run finishes when all its shards have, or - once it is overdue by the project's run timeout, counted from its latest shard's start - when every shard that started has completed, timed out or been stopped by the monthly result limit; such a run counts like a cancelled one, with the results its shards reported (since 1.0.0-beta.10; the monthly result limit since 1.0.0-beta.11). Runs still running are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.8; before it, ci_groups_finished_not_mass_failure was the only value, and since 1.0.0-beta.6 - it was ci_completed_not_excluded, which counted every completed shard on its own)."
6696
6955
  },
6697
6956
  "days": {
6698
6957
  "type": "integer",
@@ -6708,7 +6967,7 @@
6708
6967
  },
6709
6968
  "ciTests": {
6710
6969
  "type": "integer",
6711
- "description": "Tests with at least one CI run in the period."
6970
+ "description": "Tests with at least one run of the population in the period - CI runs by default; the name is from when only CI runs counted."
6712
6971
  }
6713
6972
  },
6714
6973
  "required": [
@@ -6718,7 +6977,7 @@
6718
6977
  "failingTests",
6719
6978
  "ciTests"
6720
6979
  ],
6721
- "description": "The dashboard's Tests with retries and Failing tests tiles and their denominator, counted like /tests: CI runs only, every branch - not narrowed by branchScope. The top-level failingTests counts differently: every run in the window on branchScope, local ones included."
6980
+ "description": "The dashboard's Tests with retries and Failing tests tiles and their denominator, counted like /tests on branchScope (since 1.0.0-beta.7; every branch before). Mass-failure runs are left out here, as in every per-test metric."
6722
6981
  },
6723
6982
  "slowestTests": {
6724
6983
  "type": "array",
@@ -6751,7 +7010,8 @@
6751
7010
  "medianMs",
6752
7011
  "runs"
6753
7012
  ]
6754
- }
7013
+ },
7014
+ "description": "Up to 5 tests by the median of their final-attempt durations, one per run, at least 3 runs; skipped results left out."
6755
7015
  },
6756
7016
  "timeSinks": {
6757
7017
  "type": "array",
@@ -6780,7 +7040,8 @@
6780
7040
  "filePath",
6781
7041
  "lostMs"
6782
7042
  ]
6783
- }
7043
+ },
7044
+ "description": "Up to 5 tests by the time their superseded attempts cost in the window (at least a second each), worst first."
6784
7045
  },
6785
7046
  "runsSeries": {
6786
7047
  "type": "array",
@@ -6814,7 +7075,7 @@
6814
7075
  "durationMs"
6815
7076
  ]
6816
7077
  },
6817
- "description": "The last 20 runs on branchScope, oldest first - the dashboard's Runs and Run duration charts. Cancelled runs are left out, every shard of a cancelled CI run included."
7078
+ "description": "The last 30 finished runs of the source on branchScope, whatever the window, oldest first - one point per run, passed counting first-try passes, durationMs from its first shard's start to its last shard's finish. Cancelled and incomplete runs are left out (incomplete since 1.0.0-beta.10)."
6818
7079
  }
6819
7080
  },
6820
7081
  "required": [
@@ -6824,6 +7085,7 @@
6824
7085
  "stableBranch",
6825
7086
  "lastRun",
6826
7087
  "runsInWindow",
7088
+ "unfinishedRuns",
6827
7089
  "suiteSize",
6828
7090
  "windowStability",
6829
7091
  "timeLost",
package/dist/tools.js CHANGED
@@ -23,10 +23,12 @@ export const TOOLS = [
23
23
  name: "squally-find-run",
24
24
  title: "Find runs",
25
25
  operationId: "listRuns",
26
- description: "Finds CI runs of a project - the latest, or filtered by branch, commit SHA or status. " +
27
- "One row per logical run: a sharded run is collapsed into a single entry. Returns summary " +
26
+ description: "Finds runs of a project, CI and local alike - the latest, or filtered by branch, commit SHA, " +
27
+ "status or source (all by default; ci or local narrows it, and each run says local). Status " +
28
+ "is passed, failed, cancelled, incomplete or running. One row per logical run: a sharded run " +
29
+ "is collapsed into a single entry, its counters over all its shards. Returns summary " +
28
30
  "counters only; use squally-get-run for the per-test rows.",
29
- answers: "Which runs happened — latest, or by branch, commit SHA or status. Counters only.",
31
+ answers: "Which runs happened — latest, or by branch, commit SHA, status or source. Counters only.",
30
32
  cost: "cheap",
31
33
  },
32
34
  {
@@ -34,7 +36,8 @@ export const TOOLS = [
34
36
  title: "Get one run",
35
37
  operationId: "getRun",
36
38
  description: "One run with its per-test result rows, across every shard. Use this to find out which " +
37
- "test is red in a run; the run list carries only the counters.",
39
+ "test is red in a run; the run list carries only the counters. Its status is cancelled " +
40
+ "for a cancelled run (cancellation says how and when), and local says it ran without CI.",
38
41
  answers: "One run with its per-test rows, across all shards. Which test is red.",
39
42
  cost: "cheap",
40
43
  },
@@ -56,10 +59,14 @@ export const TOOLS = [
56
59
  "/ runs), flakyRate (runs that passed only after a retry / runs), failureRate (failed " +
57
60
  "runs / runs), time lost to retries and failures, its last 20 runs (each with the run " +
58
61
  "id squally-find-run gives; runs it skipped included, as result \"skipped\" - shown, " +
59
- "never counted in runs or any rate) and the branches it ran on. Counts the CI runs of " +
60
- "the period whose shards all finished - a sharded run counts once, by its first " +
61
- "shard's start - without runs that were a mass failure as a whole; local runs never " +
62
- "count. There is no verdict - the tool returns the numbers and you judge them. runs = 0 " +
62
+ "never counted in runs or any rate) and the branches it ran on. Counts the finished " +
63
+ "CI runs of the period - a sharded run counts once, by its first shard's start - " +
64
+ "without runs that were a mass failure as a whole; source=local or source=all counts " +
65
+ "local runs or both instead (population names which). A run still in progress counts " +
66
+ "once all its shards have finished, or once the run timeout has passed, with the " +
67
+ "results its shards reported - also when a shard was stopped by the monthly result " +
68
+ "limit. There " +
69
+ "is no verdict - the tool returns the numbers and you judge them. runs = 0 " +
63
70
  "with null rates is a valid answer: no counted CI run of this test in the period. " +
64
71
  "Cheap; for one test use this, not squally-list-tests.",
65
72
  answers: "One test's stability, flaky rate and failure rate over a period.",
@@ -69,7 +76,7 @@ export const TOOLS = [
69
76
  name: "squally-list-tests",
70
77
  title: "List tests",
71
78
  operationId: "listTests",
72
- description: "Every test with a counted CI run in the period, with the same numbers per test as " +
79
+ description: "Every test with a counted CI run in the period (or local run, with source), with the same numbers per test as " +
73
80
  "squally-get-test-metrics - runs, stability, flakyRate, failureRate, time lost - ranked " +
74
81
  "worst first by the chosen sort (default: lowest stability). There is no verdict and no " +
75
82
  "status - the tool labels no test; you judge the numbers. Moderate - two queries over " +
@@ -83,8 +90,11 @@ export const TOOLS = [
83
90
  operationId: "listErrors",
84
91
  description: "The project's error signatures in a period: failures grouped by fingerprint, with how " +
85
92
  "often and how recently each occurred. Answers 'is this failure one instance of something " +
86
- "that keeps happening' - a grouping no agent can compute from the repository alone.",
87
- answers: "Error signatures in a period: what keeps failing, grouped.",
93
+ "that keeps happening' - a grouping no agent can compute from the repository alone. " +
94
+ "Counts finished CI runs by default (source=local or all for the others), a sharded run " +
95
+ "once, mass-failure runs included; failedCount and flakyCount say how the runs ended " +
96
+ "for the tests the error hit.",
97
+ answers: "Error signatures in a period: what keeps failing, grouped, and how its runs ended.",
88
98
  cost: "cheap",
89
99
  },
90
100
  ];
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "squally-mcp",
3
- "version": "0.3.0",
3
+ "version": "0.5.0",
4
4
  "description": "Local MCP server for Squally — read-only access to your Playwright test runs, flaky tests and failures.",
5
5
  "license": "MIT",
6
6
  "author": "Squally",