squally-mcp 0.2.3 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -143,16 +143,19 @@ squally-mcp trusts your operating system's certificate store, like your browser
143
143
  | `squally-debug-failure` | Every attempt of one test in one run: error, stack, Copy-for-AI prompt. | cheap |
144
144
  | `squally-get-test-metrics` | One test's stability, flaky rate and failure rate over a period. | cheap — one test |
145
145
  | `squally-list-tests` | Every test with a CI run in the period, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
146
- | `squally-list-errors` | Error signatures in a period: what keeps failing, grouped. | cheap |
146
+ | `squally-list-errors` | Error signatures in a period: what keeps failing, grouped, and how its runs ended. | cheap |
147
147
 
148
148
  For a single test use `squally-get-test-metrics`, not `squally-list-tests` —
149
149
  the tool descriptions say so, and the server repeats it in its instructions.
150
150
 
151
151
  ### What the numbers mean
152
152
 
153
- The two test tools count **completed CI runs** in the period (7, 14, 30 or 90
154
- days; 30 by default), on every branch unless you name one — without runs where
155
- most of the suite failed at once. Local runs never count. Per test:
153
+ The two test tools count **CI runs whose shards all finished** in the period
154
+ (7, 14, 30 or 90 days; 30 by default), on every branch unless you name one. A
155
+ sharded run counts once, placed in the period by its first shard's start; a
156
+ run that was a mass failure as a whole is left out. Local runs count only
157
+ when you ask for them: `source` is `ci` (the default), `local` or `all`, and
158
+ the response's `population` names which runs it counted. Per test:
156
159
 
157
160
  - **stability** — runs that passed on the first try ÷ runs;
158
161
  - **flakyRate** — runs that passed only after a retry ÷ runs;
@@ -172,7 +175,7 @@ The three least stable tests of the last 30 days (abridged):
172
175
 
173
176
  ```json
174
177
  {
175
- "population": "ci_completed_not_excluded",
178
+ "population": "ci_groups_finished_not_mass_failure",
176
179
  "days": 30,
177
180
  "sort": "stability",
178
181
  "total": 409,
@@ -219,7 +222,7 @@ The same numbers as that test's row, for one test (abridged):
219
222
 
220
223
  Pass the name exactly as Squally shows it. When it exists in several spec
221
224
  files, the tool answers with the `filePath` values to retry with; a test with
222
- results but no completed CI run in the period answers `runs: 0` and `null`
225
+ results but no counted CI run in the period answers `runs: 0` and `null`
223
226
  rates — an answer, not an error.
224
227
 
225
228
  ## Development
@@ -13,13 +13,27 @@
13
13
  // fractions of runs and nothing more; saying so here stops a model from
14
14
  // presenting a rate as Squally's judgement.
15
15
  //
16
+ // ONE RUN PER CI RUN SINCE 0.3.0 (API 1.0.0-beta.6): the population is the CI
17
+ // runs whose shards all finished, each counted once ("ci_groups_finished_not_
18
+ // mass_failure"); it was every completed shard on its own.
19
+ //
20
+ // RUNS IN PROGRESS SINCE 0.4.0 (API 1.0.0-beta.7): said outright, because the
21
+ // newest run is the one an agent asks about, and it is missing from the
22
+ // numbers until its last shard has finished.
23
+ //
24
+ // THE SOURCE SINCE 0.4.0 (API 1.0.0-beta.8): CI runs by default; local runs
25
+ // count only when the agent asks for them - they were never counted before.
26
+ //
16
27
  // Exported as `squally-mcp/instructions` (package.json), so it stays a plain
17
28
  // constant - test/exports.test.js imports it through the package name.
18
29
  export const INSTRUCTIONS = [
19
30
  "Start with squally-list-projects; every other tool needs a projectId from it.",
20
31
  "For one test, use squally-get-test-metrics, not squally-list-tests.",
21
- "Test metrics count completed CI runs in the period, without runs where most of the",
22
- "suite failed at once; local runs never count. stability = runs that passed on the",
32
+ "Test metrics count the CI runs of the period whose shards all finished - a sharded",
33
+ "run counts once - without runs that were a mass failure as a whole; pass source=local",
34
+ "or source=all to count local runs or both, and a run still in progress counts once",
35
+ "all its shards have finished.",
36
+ "stability = runs that passed on the",
23
37
  "first try / runs; flakyRate = runs that passed only after a retry / runs;",
24
38
  "failureRate = failed runs / runs. There is no verdict: the tools return numbers,",
25
39
  "and you judge them. If a test name is ambiguous, repeat with filePath from the",
@@ -3,7 +3,7 @@
3
3
  "jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
4
4
  "info": {
5
5
  "title": "Squally Read API",
6
- "version": "1.0.0-beta.5",
6
+ "version": "1.0.0-beta.9",
7
7
  "description": "Read access to a Squally organization's test results, for scripts, CI jobs and AI agents.\n\n**Authentication.** Every request carries an organization read key as `Authorization: Bearer sqly_ro_…`. A read key is organization-wide (every project), created by an organization admin in Organization › Read keys, and expires after 30, 60, 90, 180 days as chosen at creation. There is no renewal: create a new key and revoke the old one. Project ingest keys (`sqly_…`) are refused with `wrong_key_type`.\n\n**Expiry warning.** From 7 days before a key expires, every authenticated response carries `Squally-Key-Expires-At` with the expiry instant (ISO-8601).\n\n**Rate limit.** Currently 60 requests per 60 seconds per key, counted per `Authorization` header - a read key has its own budget, shared with nothing else. Over the limit the edge answers 429 without a JSON body.\n\n**Retries.** 429: wait 60 seconds, then retry. The edge counts requests per key in 60-second windows and sends no Retry-After header. 5xx: retry with exponential backoff, starting at 1 second and capped at 30 seconds. 400, 401, 403, 404: never retry - the same request gets the same answer.\n\n**Plans.** The read API is part of the Standard and Pro plans. A key of an organization on Free is answered 403 `tier_too_low`; keys are not revoked on a downgrade and work again after an upgrade.\n\n**Errors.** Every error is `{\"error\": \"<sentence>\", \"code\": \"<slug>\"}`. The sentence may be reworded; the slug may not. An id that does not exist and an id that belongs to another organization get the same 404, byte for byte.\n\n**Test names in paths.** Test names are Playwright title paths and often contain `/`: URL-encode them, including `/` as `%2F`.\n\n**Sharded runs** are returned as one logical run: /runs lists one row per logical run, and the run routes answer for all shards whichever shard's id you pass.\n\n**Stability.** v1, beta. The API may change until the beta ends; every change is listed in the changelog (docs/openapi/CHANGELOG.md)."
8
8
  },
9
9
  "servers": [
@@ -1675,7 +1675,7 @@
1675
1675
  "get": {
1676
1676
  "operationId": "listTests",
1677
1677
  "summary": "List tests",
1678
- "description": "Every test with a CI run in the period and its metrics - stability, flaky rate, failure rate, runs, time lost, its last 20 runs, the branches it ran on and where it fails most - ranked by the chosen sort. Counts completed CI runs only, without mass-failure runs, on every branch unless branch names one (see population). Moderate: two queries over the period's CI results, paged in the database; use the metrics route for one test.",
1678
+ "description": "Every test with a run of the source (CI by default) in the period and its metrics - stability, flaky rate, failure rate, runs, time lost, its last 20 runs, the branches it ran on and where it fails most - ranked by the chosen sort. Counts completed runs of the source - CI by default - without mass-failure runs, on every branch unless branch names one (see population). Moderate: two queries over the period's CI results, paged in the database; use the metrics route for one test.",
1679
1679
  "tags": [
1680
1680
  "Tests"
1681
1681
  ],
@@ -1727,6 +1727,21 @@
1727
1727
  "maxLength": 200
1728
1728
  }
1729
1729
  },
1730
+ {
1731
+ "name": "source",
1732
+ "in": "query",
1733
+ "required": false,
1734
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
1735
+ "schema": {
1736
+ "type": "string",
1737
+ "enum": [
1738
+ "ci",
1739
+ "local",
1740
+ "all"
1741
+ ],
1742
+ "default": "ci"
1743
+ }
1744
+ },
1730
1745
  {
1731
1746
  "name": "browser",
1732
1747
  "in": "query",
@@ -1797,7 +1812,7 @@
1797
1812
  "id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
1798
1813
  "name": "storefront-web"
1799
1814
  },
1800
- "population": "ci_completed_not_excluded",
1815
+ "population": "ci_groups_finished_not_mass_failure",
1801
1816
  "days": 30,
1802
1817
  "since": "2026-08-25T08:00:00.000Z",
1803
1818
  "until": "2026-09-24T08:00:00.000Z",
@@ -2284,7 +2299,7 @@
2284
2299
  "get": {
2285
2300
  "operationId": "getTestMetrics",
2286
2301
  "summary": "Get test metrics",
2287
- "description": "One test's metrics over the period: the fields of its /tests row, flat, beside the same period. A test with results but no CI run in the period answers zeros and null rates rather than 404. Cheap: the list's queries, narrowed to one test; prefer it to /tests when the question is about one test.",
2302
+ "description": "One test's metrics over the period: the fields of its /tests row, flat, beside the same period. A test with results but no run of the source in the period answers zeros and null rates rather than 404. Cheap: the list's queries, narrowed to one test; prefer it to /tests when the question is about one test.",
2288
2303
  "tags": [
2289
2304
  "Tests"
2290
2305
  ],
@@ -2342,6 +2357,21 @@
2342
2357
  "minLength": 1,
2343
2358
  "maxLength": 200
2344
2359
  }
2360
+ },
2361
+ {
2362
+ "name": "source",
2363
+ "in": "query",
2364
+ "required": false,
2365
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
2366
+ "schema": {
2367
+ "type": "string",
2368
+ "enum": [
2369
+ "ci",
2370
+ "local",
2371
+ "all"
2372
+ ],
2373
+ "default": "ci"
2374
+ }
2345
2375
  }
2346
2376
  ],
2347
2377
  "responses": {
@@ -2362,7 +2392,7 @@
2362
2392
  "id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
2363
2393
  "name": "storefront-web"
2364
2394
  },
2365
- "population": "ci_completed_not_excluded",
2395
+ "population": "ci_groups_finished_not_mass_failure",
2366
2396
  "days": 30,
2367
2397
  "since": "2026-08-25T08:00:00.000Z",
2368
2398
  "until": "2026-09-24T08:00:00.000Z",
@@ -2695,7 +2725,7 @@
2695
2725
  "get": {
2696
2726
  "operationId": "getTestHistory",
2697
2727
  "summary": "Get test history",
2698
- "description": "Get a test's history across runs. One test in every run of a period, newest first: final status, duration, attempts and time spent on retries. Moderate.",
2728
+ "description": "Get a test's history across runs. One test in every run of a period, newest first: final status, duration, attempts and time spent on retries. Every run is listed; source decides which are counted. Moderate.",
2699
2729
  "tags": [
2700
2730
  "Tests"
2701
2731
  ],
@@ -2742,6 +2772,21 @@
2742
2772
  ],
2743
2773
  "default": 30
2744
2774
  }
2775
+ },
2776
+ {
2777
+ "name": "source",
2778
+ "in": "query",
2779
+ "required": false,
2780
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
2781
+ "schema": {
2782
+ "type": "string",
2783
+ "enum": [
2784
+ "ci",
2785
+ "local",
2786
+ "all"
2787
+ ],
2788
+ "default": "ci"
2789
+ }
2745
2790
  }
2746
2791
  ],
2747
2792
  "responses": {
@@ -2775,6 +2820,8 @@
2775
2820
  {
2776
2821
  "runId": "b1e0c6d2-7a43-4f19-8d5e-0c3a9e6f2b71",
2777
2822
  "runNumber": 1284,
2823
+ "local": false,
2824
+ "counted": true,
2778
2825
  "branch": "main",
2779
2826
  "testedRevision": null,
2780
2827
  "startedAt": "2026-09-24T07:31:12.000Z",
@@ -2788,6 +2835,8 @@
2788
2835
  {
2789
2836
  "runId": "7d2f4b19-3c8e-4a61-9e07-5b1d8c2a6f34",
2790
2837
  "runNumber": 1283,
2838
+ "local": true,
2839
+ "counted": false,
2791
2840
  "branch": "feature/saved-carts",
2792
2841
  "testedRevision": "c4f7a1e9b3d5c8e2f0a6b4d9e1c7f3a5b8d2e0f6",
2793
2842
  "startedAt": "2026-09-23T16:05:40.000Z",
@@ -2804,7 +2853,7 @@
2804
2853
  }
2805
2854
  },
2806
2855
  "400": {
2807
- "description": "Error: ambiguous_test, invalid_period",
2856
+ "description": "Error: ambiguous_test, invalid_parameter, invalid_period",
2808
2857
  "content": {
2809
2858
  "application/json": {
2810
2859
  "schema": {
@@ -2818,6 +2867,7 @@
2818
2867
  "type": "string",
2819
2868
  "enum": [
2820
2869
  "ambiguous_test",
2870
+ "invalid_parameter",
2821
2871
  "invalid_period"
2822
2872
  ],
2823
2873
  "description": "Stable slug - match on this, not on the sentence."
@@ -2836,6 +2886,13 @@
2836
2886
  "code": "ambiguous_test"
2837
2887
  }
2838
2888
  },
2889
+ "invalid_parameter": {
2890
+ "summary": "invalid_parameter",
2891
+ "value": {
2892
+ "error": "sha must be 4 to 40 hexadecimal characters (a commit SHA or a prefix of one).",
2893
+ "code": "invalid_parameter"
2894
+ }
2895
+ },
2839
2896
  "invalid_period": {
2840
2897
  "summary": "invalid_period",
2841
2898
  "value": {
@@ -3041,7 +3098,7 @@
3041
3098
  "get": {
3042
3099
  "operationId": "listErrors",
3043
3100
  "summary": "List errors",
3044
- "description": "List errors. The project's errors in a period - failures grouped by fingerprint - most frequent first. Every count covers the period; firstSeen is the first occurrence still stored, not bounded by it. Moderate: one aggregate query.",
3101
+ "description": "List errors. The project's errors in a period - failures grouped by fingerprint - most frequent first, over the finished runs of the source (see population), all shards of a run one run. Every count covers the period; firstSeen is the first occurrence still stored, not bounded by it. Moderate: one aggregate query.",
3045
3102
  "tags": [
3046
3103
  "Errors"
3047
3104
  ],
@@ -3071,6 +3128,21 @@
3071
3128
  "default": 30
3072
3129
  }
3073
3130
  },
3131
+ {
3132
+ "name": "source",
3133
+ "in": "query",
3134
+ "required": false,
3135
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
3136
+ "schema": {
3137
+ "type": "string",
3138
+ "enum": [
3139
+ "ci",
3140
+ "local",
3141
+ "all"
3142
+ ],
3143
+ "default": "ci"
3144
+ }
3145
+ },
3074
3146
  {
3075
3147
  "name": "page",
3076
3148
  "in": "query",
@@ -3116,6 +3188,7 @@
3116
3188
  "id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
3117
3189
  "name": "storefront-web"
3118
3190
  },
3191
+ "population": "ci_groups_finished",
3119
3192
  "window": {
3120
3193
  "days": 30,
3121
3194
  "from": "2026-08-25T08:00:00.000Z",
@@ -3141,12 +3214,15 @@
3141
3214
  ],
3142
3215
  "occurrenceCount": 9,
3143
3216
  "runCount": 7,
3217
+ "failedCount": 5,
3218
+ "flakyCount": 2,
3219
+ "massFailureRunCount": 0,
3144
3220
  "firstSeen": "2026-09-02T11:20:44.000Z",
3145
- "lastSeen": "2026-09-24T07:34:05.000Z",
3146
- "isNew": true
3221
+ "lastSeen": "2026-09-24T07:34:05.000Z"
3147
3222
  }
3148
3223
  ],
3149
- "unfingerprintedCount": 0
3224
+ "unfingerprintedCount": 0,
3225
+ "unfinishedRuns": 0
3150
3226
  }
3151
3227
  }
3152
3228
  }
@@ -3381,7 +3457,7 @@
3381
3457
  "get": {
3382
3458
  "operationId": "getError",
3383
3459
  "summary": "Get an error",
3384
- "description": "Get an error. One error: its counts, the tests it hits and the runs it occurred in. An error quiet in the period is answered over everything still stored (windowEmpty: true). Moderate.",
3460
+ "description": "Get an error. One error: its counts, the tests it hits and the runs it occurred in - one row per run and test, by the run's id as /runs gives it. An error quiet in the period is answered over everything still stored (windowEmpty: true). Moderate.",
3385
3461
  "tags": [
3386
3462
  "Errors"
3387
3463
  ],
@@ -3420,6 +3496,21 @@
3420
3496
  ],
3421
3497
  "default": 30
3422
3498
  }
3499
+ },
3500
+ {
3501
+ "name": "source",
3502
+ "in": "query",
3503
+ "required": false,
3504
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
3505
+ "schema": {
3506
+ "type": "string",
3507
+ "enum": [
3508
+ "ci",
3509
+ "local",
3510
+ "all"
3511
+ ],
3512
+ "default": "ci"
3513
+ }
3423
3514
  }
3424
3515
  ],
3425
3516
  "responses": {
@@ -3440,6 +3531,7 @@
3440
3531
  "id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
3441
3532
  "name": "storefront-web"
3442
3533
  },
3534
+ "population": "ci_groups_finished",
3443
3535
  "window": {
3444
3536
  "days": 30,
3445
3537
  "from": "2026-08-25T08:00:00.000Z",
@@ -3458,16 +3550,21 @@
3458
3550
  ],
3459
3551
  "occurrenceCount": 9,
3460
3552
  "runCount": 7,
3553
+ "failedCount": 5,
3554
+ "flakyCount": 2,
3555
+ "massFailureRunCount": 0,
3461
3556
  "firstSeen": "2026-09-02T11:20:44.000Z",
3462
- "lastSeen": "2026-09-24T07:34:05.000Z",
3463
- "isNew": true
3557
+ "lastSeen": "2026-09-24T07:34:05.000Z"
3464
3558
  },
3465
3559
  "windowEmpty": false,
3466
3560
  "tests": [
3467
3561
  {
3468
3562
  "testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code",
3469
3563
  "filePath": "tests/checkout.spec.ts",
3470
- "occurrenceCount": 9
3564
+ "occurrenceCount": 9,
3565
+ "runCount": 7,
3566
+ "failedCount": 5,
3567
+ "flakyCount": 2
3471
3568
  }
3472
3569
  ],
3473
3570
  "runs": [
@@ -3480,7 +3577,8 @@
3480
3577
  "filePath": "tests/checkout.spec.ts",
3481
3578
  "attempts": 2,
3482
3579
  "occurrences": 1,
3483
- "finalStatus": "passed"
3580
+ "finalStatus": "passed",
3581
+ "massFailure": false
3484
3582
  }
3485
3583
  ]
3486
3584
  }
@@ -3488,7 +3586,7 @@
3488
3586
  }
3489
3587
  },
3490
3588
  "400": {
3491
- "description": "Error: invalid_period",
3589
+ "description": "Error: invalid_parameter, invalid_period",
3492
3590
  "content": {
3493
3591
  "application/json": {
3494
3592
  "schema": {
@@ -3501,6 +3599,7 @@
3501
3599
  "code": {
3502
3600
  "type": "string",
3503
3601
  "enum": [
3602
+ "invalid_parameter",
3504
3603
  "invalid_period"
3505
3604
  ],
3506
3605
  "description": "Stable slug - match on this, not on the sentence."
@@ -3512,6 +3611,13 @@
3512
3611
  ]
3513
3612
  },
3514
3613
  "examples": {
3614
+ "invalid_parameter": {
3615
+ "summary": "invalid_parameter",
3616
+ "value": {
3617
+ "error": "sha must be 4 to 40 hexadecimal characters (a commit SHA or a prefix of one).",
3618
+ "code": "invalid_parameter"
3619
+ }
3620
+ },
3515
3621
  "invalid_period": {
3516
3622
  "summary": "invalid_period",
3517
3623
  "value": {
@@ -3717,7 +3823,7 @@
3717
3823
  "get": {
3718
3824
  "operationId": "getOverview",
3719
3825
  "summary": "Get dashboard figures",
3720
- "description": "The dashboard for a period: last run, suite stability, time lost, the two test-metric tiles (testMetrics), slowest tests, time sinks, recent runs. Suite stability and testMetrics count completed CI runs only. Moderate: one pass over the period's runs and one count over its CI results.",
3826
+ "description": "The dashboard for a period: last run, suite stability, time lost, the two test-metric tiles (testMetrics), slowest tests, time sinks, recent runs. Every figure counts finished runs of the source - CI by default - and testMetrics.population names it. Moderate: one pass over the period's runs and one count over its results.",
3721
3827
  "tags": [
3722
3828
  "Project"
3723
3829
  ],
@@ -3760,6 +3866,21 @@
3760
3866
  ],
3761
3867
  "default": "stable"
3762
3868
  }
3869
+ },
3870
+ {
3871
+ "name": "source",
3872
+ "in": "query",
3873
+ "required": false,
3874
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
3875
+ "schema": {
3876
+ "type": "string",
3877
+ "enum": [
3878
+ "ci",
3879
+ "local",
3880
+ "all"
3881
+ ],
3882
+ "default": "ci"
3883
+ }
3763
3884
  }
3764
3885
  ],
3765
3886
  "responses": {
@@ -3795,11 +3916,14 @@
3795
3916
  "startedAt": "2026-09-24T07:31:12.000Z",
3796
3917
  "finishedAt": "2026-09-24T07:39:47.000Z",
3797
3918
  "total": 412,
3798
- "passed": 408,
3919
+ "passed": 404,
3920
+ "flaky": 4,
3799
3921
  "failed": 0,
3922
+ "skipped": 4,
3800
3923
  "isFallbackBranch": false
3801
3924
  },
3802
3925
  "runsInWindow": 11,
3926
+ "unfinishedRuns": 1,
3803
3927
  "suiteSize": 412,
3804
3928
  "windowStability": 99,
3805
3929
  "timeLost": {
@@ -3813,7 +3937,7 @@
3813
3937
  "testsRun": 412
3814
3938
  },
3815
3939
  "testMetrics": {
3816
- "population": "ci_completed_not_excluded",
3940
+ "population": "ci_groups_finished_not_mass_failure",
3817
3941
  "days": 30,
3818
3942
  "testsWithRetries": 7,
3819
3943
  "failingTests": 2,
@@ -5423,8 +5547,12 @@
5423
5547
  },
5424
5548
  "population": {
5425
5549
  "type": "string",
5426
- "const": "ci_completed_not_excluded",
5427
- "description": "Which runs every number counts: completed CI runs (reported with a CI provider) started inside the period, without the runs excluded as a mass failure (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count."
5550
+ "enum": [
5551
+ "ci_groups_finished_not_mass_failure",
5552
+ "local_groups_finished_not_mass_failure",
5553
+ "all_groups_finished_not_mass_failure"
5554
+ ],
5555
+ "description": "Which runs every number counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Runs that have not finished are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.8; before it, ci_groups_finished_not_mass_failure was the only value, and since 1.0.0-beta.6 - it was ci_completed_not_excluded, which counted every completed shard on its own)."
5428
5556
  },
5429
5557
  "days": {
5430
5558
  "type": "integer",
@@ -5480,7 +5608,7 @@
5480
5608
  },
5481
5609
  "total": {
5482
5610
  "type": "integer",
5483
- "description": "Tests matching the query across all pages: every test with at least one CI run in the period, narrowed by search, branch and browser."
5611
+ "description": "Tests matching the query across all pages: every test with at least one run of the population in the period, narrowed by search, branch and browser."
5484
5612
  },
5485
5613
  "pageCount": {
5486
5614
  "type": "integer"
@@ -5517,7 +5645,7 @@
5517
5645
  },
5518
5646
  "runs": {
5519
5647
  "type": "integer",
5520
- "description": "CI runs in the period in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns."
5648
+ "description": "Runs of the population in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns. A sharded run counts once; the final attempt is the one with the most retries, then from the lowest shard."
5521
5649
  },
5522
5650
  "stableRuns": {
5523
5651
  "type": "integer",
@@ -5574,7 +5702,8 @@
5574
5702
  "type": "object",
5575
5703
  "properties": {
5576
5704
  "runId": {
5577
- "type": "string"
5705
+ "type": "string",
5706
+ "description": "The run, by the id GET /projects/{projectId}/runs gives it - one id for all shards of a sharded run (since 1.0.0-beta.6; it was the id of the shard the test ran in)."
5578
5707
  },
5579
5708
  "runNumber": {
5580
5709
  "type": "integer"
@@ -5602,7 +5731,7 @@
5602
5731
  "failed",
5603
5732
  "skipped"
5604
5733
  ],
5605
- "description": "The run's final attempt: stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate."
5734
+ "description": "The run's final attempt - across all its shards - read through test.fail(): stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate. A test the run's cancellation interrupted appears nowhere."
5606
5735
  }
5607
5736
  },
5608
5737
  "required": [
@@ -5723,8 +5852,12 @@
5723
5852
  },
5724
5853
  "population": {
5725
5854
  "type": "string",
5726
- "const": "ci_completed_not_excluded",
5727
- "description": "Which runs every number counts: completed CI runs (reported with a CI provider) started inside the period, without the runs excluded as a mass failure (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count."
5855
+ "enum": [
5856
+ "ci_groups_finished_not_mass_failure",
5857
+ "local_groups_finished_not_mass_failure",
5858
+ "all_groups_finished_not_mass_failure"
5859
+ ],
5860
+ "description": "Which runs every number counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Runs that have not finished are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.8; before it, ci_groups_finished_not_mass_failure was the only value, and since 1.0.0-beta.6 - it was ci_completed_not_excluded, which counted every completed shard on its own)."
5728
5861
  },
5729
5862
  "days": {
5730
5863
  "type": "integer",
@@ -5778,7 +5911,7 @@
5778
5911
  },
5779
5912
  "runs": {
5780
5913
  "type": "integer",
5781
- "description": "CI runs in the period in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns."
5914
+ "description": "Runs of the population in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns. A sharded run counts once; the final attempt is the one with the most retries, then from the lowest shard."
5782
5915
  },
5783
5916
  "stableRuns": {
5784
5917
  "type": "integer",
@@ -5835,7 +5968,8 @@
5835
5968
  "type": "object",
5836
5969
  "properties": {
5837
5970
  "runId": {
5838
- "type": "string"
5971
+ "type": "string",
5972
+ "description": "The run, by the id GET /projects/{projectId}/runs gives it - one id for all shards of a sharded run (since 1.0.0-beta.6; it was the id of the shard the test ran in)."
5839
5973
  },
5840
5974
  "runNumber": {
5841
5975
  "type": "integer"
@@ -5863,7 +5997,7 @@
5863
5997
  "failed",
5864
5998
  "skipped"
5865
5999
  ],
5866
- "description": "The run's final attempt: stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate."
6000
+ "description": "The run's final attempt - across all its shards - read through test.fail(): stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate. A test the run's cancellation interrupted appears nowhere."
5867
6001
  }
5868
6002
  },
5869
6003
  "required": [
@@ -6019,11 +6153,20 @@
6019
6153
  "type": "object",
6020
6154
  "properties": {
6021
6155
  "runId": {
6022
- "type": "string"
6156
+ "type": "string",
6157
+ "description": "The run, by the id GET /projects/{projectId}/runs gives it - one occurrence and one id per run, however many shards it had (since 1.0.0-beta.6; it was one occurrence per shard)."
6023
6158
  },
6024
6159
  "runNumber": {
6025
6160
  "type": "integer"
6026
6161
  },
6162
+ "local": {
6163
+ "type": "boolean",
6164
+ "description": "No shard of the run reported a CI provider: a local run. Listed always; counted only under source=local or source=all (since 1.0.0-beta.8; never counted from 1.0.0-beta.6)."
6165
+ },
6166
+ "counted": {
6167
+ "type": "boolean",
6168
+ "description": "Whether the run is in the metrics' population (see population on the metrics endpoints): a run of the source parameter's kind - CI by default - whose shards all finished, not a mass failure, and the test not just skipped by a cancellation (since 1.0.0-beta.6; the source since 1.0.0-beta.8)."
6169
+ },
6027
6170
  "branch": {
6028
6171
  "anyOf": [
6029
6172
  {
@@ -6050,7 +6193,8 @@
6050
6193
  "format": "date-time"
6051
6194
  },
6052
6195
  "status": {
6053
- "type": "string"
6196
+ "type": "string",
6197
+ "description": "The final attempt's status across the run's shards, read through test.fail()."
6054
6198
  },
6055
6199
  "durationMs": {
6056
6200
  "anyOf": [
@@ -6094,6 +6238,8 @@
6094
6238
  "required": [
6095
6239
  "runId",
6096
6240
  "runNumber",
6241
+ "local",
6242
+ "counted",
6097
6243
  "branch",
6098
6244
  "testedRevision",
6099
6245
  "startedAt",
@@ -6132,6 +6278,15 @@
6132
6278
  "name"
6133
6279
  ]
6134
6280
  },
6281
+ "population": {
6282
+ "type": "string",
6283
+ "enum": [
6284
+ "ci_groups_finished",
6285
+ "local_groups_finished",
6286
+ "all_groups_finished"
6287
+ ],
6288
+ "description": "Which runs every error figure counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - runs that were a mass failure as a whole included (see massFailureRunCount). Runs that have not finished are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.9)."
6289
+ },
6135
6290
  "window": {
6136
6291
  "type": "object",
6137
6292
  "properties": {
@@ -6219,19 +6374,29 @@
6219
6374
  "type": "integer"
6220
6375
  },
6221
6376
  "runCount": {
6222
- "type": "integer"
6377
+ "type": "integer",
6378
+ "description": "Runs of the population with at least one such attempt in the window - all shards of a run one run (since 1.0.0-beta.9; it counted every shard row)."
6379
+ },
6380
+ "failedCount": {
6381
+ "type": "integer",
6382
+ "description": "Of the (run, test) pairs the error hit, how many ended failed: the test's final attempt across all the run's shards failed (since 1.0.0-beta.9)."
6383
+ },
6384
+ "flakyCount": {
6385
+ "type": "integer",
6386
+ "description": "Of the same pairs, how many ended flaky: the final attempt passed after a retry. A pair whose final attempt skipped is neither (since 1.0.0-beta.9)."
6387
+ },
6388
+ "massFailureRunCount": {
6389
+ "type": "integer",
6390
+ "description": "Of runCount, the runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed) - counted, and named here (since 1.0.0-beta.9)."
6223
6391
  },
6224
6392
  "firstSeen": {
6225
6393
  "type": "string",
6226
6394
  "format": "date-time",
6227
- "description": "First seen in what is still stored: the earliest occurrence the plan's retention has kept, NOT bounded by the requested period. ISO-8601."
6395
+ "description": "First seen in what is still stored: the start of the earliest run of the population, as far back as the plan's retention keeps them, NOT bounded by the requested period. ISO-8601. (isNew, \"absent in the period before\", was removed in 1.0.0-beta.9: compare firstSeen with window.from.)"
6228
6396
  },
6229
6397
  "lastSeen": {
6230
6398
  "type": "string",
6231
6399
  "format": "date-time"
6232
- },
6233
- "isNew": {
6234
- "type": "boolean"
6235
6400
  }
6236
6401
  },
6237
6402
  "required": [
@@ -6245,22 +6410,31 @@
6245
6410
  "testNames",
6246
6411
  "occurrenceCount",
6247
6412
  "runCount",
6413
+ "failedCount",
6414
+ "flakyCount",
6415
+ "massFailureRunCount",
6248
6416
  "firstSeen",
6249
- "lastSeen",
6250
- "isNew"
6417
+ "lastSeen"
6251
6418
  ]
6252
6419
  }
6253
6420
  },
6254
6421
  "unfingerprintedCount": {
6255
- "type": "integer"
6422
+ "type": "integer",
6423
+ "description": "Failed attempts in the window's runs of the population that carry no signature and are not grouped (the window only since 1.0.0-beta.9; it read the period before too). Expected 0."
6424
+ },
6425
+ "unfinishedRuns": {
6426
+ "type": "integer",
6427
+ "description": "Runs of the source in the window that have not finished - a shard still running, timed out or never reported. In no figure (since 1.0.0-beta.9)."
6256
6428
  }
6257
6429
  },
6258
6430
  "required": [
6259
6431
  "project",
6432
+ "population",
6260
6433
  "window",
6261
6434
  "page",
6262
6435
  "errors",
6263
- "unfingerprintedCount"
6436
+ "unfingerprintedCount",
6437
+ "unfinishedRuns"
6264
6438
  ]
6265
6439
  },
6266
6440
  "ErrorDetailResponse": {
@@ -6281,6 +6455,15 @@
6281
6455
  "name"
6282
6456
  ]
6283
6457
  },
6458
+ "population": {
6459
+ "type": "string",
6460
+ "enum": [
6461
+ "ci_groups_finished",
6462
+ "local_groups_finished",
6463
+ "all_groups_finished"
6464
+ ],
6465
+ "description": "Which runs every error figure counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - runs that were a mass failure as a whole included (see massFailureRunCount). Runs that have not finished are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.9)."
6466
+ },
6284
6467
  "window": {
6285
6468
  "type": "object",
6286
6469
  "properties": {
@@ -6343,19 +6526,29 @@
6343
6526
  "type": "integer"
6344
6527
  },
6345
6528
  "runCount": {
6346
- "type": "integer"
6529
+ "type": "integer",
6530
+ "description": "Runs of the population with at least one such attempt in the window - all shards of a run one run (since 1.0.0-beta.9; it counted every shard row)."
6531
+ },
6532
+ "failedCount": {
6533
+ "type": "integer",
6534
+ "description": "Of the (run, test) pairs the error hit, how many ended failed: the test's final attempt across all the run's shards failed (since 1.0.0-beta.9)."
6535
+ },
6536
+ "flakyCount": {
6537
+ "type": "integer",
6538
+ "description": "Of the same pairs, how many ended flaky: the final attempt passed after a retry. A pair whose final attempt skipped is neither (since 1.0.0-beta.9)."
6539
+ },
6540
+ "massFailureRunCount": {
6541
+ "type": "integer",
6542
+ "description": "Of runCount, the runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed) - counted, and named here (since 1.0.0-beta.9)."
6347
6543
  },
6348
6544
  "firstSeen": {
6349
6545
  "type": "string",
6350
6546
  "format": "date-time",
6351
- "description": "First seen in what is still stored: the earliest occurrence the plan's retention has kept, NOT bounded by the requested period. ISO-8601."
6547
+ "description": "First seen in what is still stored: the start of the earliest run of the population, as far back as the plan's retention keeps them, NOT bounded by the requested period. ISO-8601. (isNew, \"absent in the period before\", was removed in 1.0.0-beta.9: compare firstSeen with window.from.)"
6352
6548
  },
6353
6549
  "lastSeen": {
6354
6550
  "type": "string",
6355
6551
  "format": "date-time"
6356
- },
6357
- "isNew": {
6358
- "type": "boolean"
6359
6552
  }
6360
6553
  },
6361
6554
  "required": [
@@ -6369,9 +6562,11 @@
6369
6562
  "testNames",
6370
6563
  "occurrenceCount",
6371
6564
  "runCount",
6565
+ "failedCount",
6566
+ "flakyCount",
6567
+ "massFailureRunCount",
6372
6568
  "firstSeen",
6373
- "lastSeen",
6374
- "isNew"
6569
+ "lastSeen"
6375
6570
  ]
6376
6571
  },
6377
6572
  "windowEmpty": {
@@ -6398,12 +6593,27 @@
6398
6593
  },
6399
6594
  "occurrenceCount": {
6400
6595
  "type": "integer"
6596
+ },
6597
+ "runCount": {
6598
+ "type": "integer",
6599
+ "description": "Runs in which this test hit the error (since 1.0.0-beta.9)."
6600
+ },
6601
+ "failedCount": {
6602
+ "type": "integer",
6603
+ "description": "Of those runs, the ones the test ended failed (since 1.0.0-beta.9)."
6604
+ },
6605
+ "flakyCount": {
6606
+ "type": "integer",
6607
+ "description": "Of those runs, the ones the test ended flaky - passed after a retry (since 1.0.0-beta.9)."
6401
6608
  }
6402
6609
  },
6403
6610
  "required": [
6404
6611
  "testName",
6405
6612
  "filePath",
6406
- "occurrenceCount"
6613
+ "occurrenceCount",
6614
+ "runCount",
6615
+ "failedCount",
6616
+ "flakyCount"
6407
6617
  ]
6408
6618
  }
6409
6619
  },
@@ -6413,7 +6623,8 @@
6413
6623
  "type": "object",
6414
6624
  "properties": {
6415
6625
  "runId": {
6416
- "type": "string"
6626
+ "type": "string",
6627
+ "description": "The run, by the id GET /projects/{projectId}/runs gives it - one row per run and test, however many shards the run had (since 1.0.0-beta.9; it was one row per shard, with the shard's id)."
6417
6628
  },
6418
6629
  "runNumber": {
6419
6630
  "type": "integer"
@@ -6446,13 +6657,20 @@
6446
6657
  ]
6447
6658
  },
6448
6659
  "attempts": {
6449
- "type": "integer"
6660
+ "type": "integer",
6661
+ "description": "Every attempt of the test in the run, retries and every shard included."
6450
6662
  },
6451
6663
  "occurrences": {
6452
- "type": "integer"
6664
+ "type": "integer",
6665
+ "description": "How many of those carry this error."
6453
6666
  },
6454
6667
  "finalStatus": {
6455
- "type": "string"
6668
+ "type": "string",
6669
+ "description": "The test's final attempt across all the run's shards, through test.fail(): passed, failed or skipped."
6670
+ },
6671
+ "massFailure": {
6672
+ "type": "boolean",
6673
+ "description": "The run was a mass failure as a whole (since 1.0.0-beta.9)."
6456
6674
  }
6457
6675
  },
6458
6676
  "required": [
@@ -6464,13 +6682,15 @@
6464
6682
  "filePath",
6465
6683
  "attempts",
6466
6684
  "occurrences",
6467
- "finalStatus"
6685
+ "finalStatus",
6686
+ "massFailure"
6468
6687
  ]
6469
6688
  }
6470
6689
  }
6471
6690
  },
6472
6691
  "required": [
6473
6692
  "project",
6693
+ "population",
6474
6694
  "window",
6475
6695
  "signature",
6476
6696
  "windowEmpty",
@@ -6533,7 +6753,8 @@
6533
6753
  "type": "object",
6534
6754
  "properties": {
6535
6755
  "id": {
6536
- "type": "string"
6756
+ "type": "string",
6757
+ "description": "The run's id as GET /projects/{projectId}/runs gives it - one id for all its shards."
6537
6758
  },
6538
6759
  "runNumber": {
6539
6760
  "type": "integer"
@@ -6556,11 +6777,13 @@
6556
6777
  {
6557
6778
  "type": "null"
6558
6779
  }
6559
- ]
6780
+ ],
6781
+ "description": "The run's status over all its shards: passed or failed."
6560
6782
  },
6561
6783
  "startedAt": {
6562
6784
  "type": "string",
6563
- "format": "date-time"
6785
+ "format": "date-time",
6786
+ "description": "Its first shard's start."
6564
6787
  },
6565
6788
  "finishedAt": {
6566
6789
  "anyOf": [
@@ -6571,17 +6794,28 @@
6571
6794
  {
6572
6795
  "type": "null"
6573
6796
  }
6574
- ]
6797
+ ],
6798
+ "description": "Its last shard's finish."
6575
6799
  },
6576
6800
  "total": {
6577
- "type": "integer"
6801
+ "type": "integer",
6802
+ "description": "Tests in the run, every test once over all shards: passed + flaky + failed + skipped."
6578
6803
  },
6579
6804
  "passed": {
6580
- "type": "integer"
6805
+ "type": "integer",
6806
+ "description": "Tests that passed on the first try."
6807
+ },
6808
+ "flaky": {
6809
+ "type": "integer",
6810
+ "description": "Tests that passed after a retry (since 1.0.0-beta.7)."
6581
6811
  },
6582
6812
  "failed": {
6583
6813
  "type": "integer"
6584
6814
  },
6815
+ "skipped": {
6816
+ "type": "integer",
6817
+ "description": "Since 1.0.0-beta.7."
6818
+ },
6585
6819
  "isFallbackBranch": {
6586
6820
  "type": "boolean"
6587
6821
  }
@@ -6595,21 +6829,29 @@
6595
6829
  "finishedAt",
6596
6830
  "total",
6597
6831
  "passed",
6832
+ "flaky",
6598
6833
  "failed",
6834
+ "skipped",
6599
6835
  "isFallbackBranch"
6600
6836
  ]
6601
6837
  },
6602
6838
  {
6603
6839
  "type": "null"
6604
6840
  }
6605
- ]
6841
+ ],
6842
+ "description": "The newest finished run of the source on branchScope, whatever the window - falling back to any branch when the stable branch has none (see the source parameter and testMetrics.population)."
6606
6843
  },
6607
6844
  "runsInWindow": {
6608
6845
  "type": "integer",
6609
- "description": "Completed CI runs in the window on branchScope, without mass-failure runs - the runs windowStability is counted over."
6846
+ "description": "Finished runs of the source in the window on branchScope (see the source parameter and testMetrics.population), each counted once however many shards it had, by the start of its first shard; mass-failure and cancelled runs included. The runs windowStability is counted over."
6847
+ },
6848
+ "unfinishedRuns": {
6849
+ "type": "integer",
6850
+ "description": "Runs of the source in the window on branchScope that have not finished - a shard still running, timed out or never reported. In no other figure (since 1.0.0-beta.7; of the source since 1.0.0-beta.8)."
6610
6851
  },
6611
6852
  "suiteSize": {
6612
- "type": "integer"
6853
+ "type": "integer",
6854
+ "description": "The largest run in the window, every test once over all its shards."
6613
6855
  },
6614
6856
  "windowStability": {
6615
6857
  "anyOf": [
@@ -6620,7 +6862,7 @@
6620
6862
  "type": "null"
6621
6863
  }
6622
6864
  ],
6623
- "description": "Suite stability, 0-100: of the final test results in the window's completed CI runs on branchScope (mass-failure runs and skipped results left out), the share that passed on the first try, rounded. A result that passed only after a retry counts against it. null when no such result exists."
6865
+ "description": "Suite stability, 0-100: of the final test results in the window's finished runs of the source on branchScope (skipped results left out), the share that passed on the first try, rounded. Mass-failure runs count (since 1.0.0-beta.7). A result that passed only after a retry counts against it. null when no such result exists."
6624
6866
  },
6625
6867
  "timeLost": {
6626
6868
  "type": "object",
@@ -6650,7 +6892,8 @@
6650
6892
  "lostPercent",
6651
6893
  "wastedAttempts",
6652
6894
  "wallClockMs"
6653
- ]
6895
+ ],
6896
+ "description": "Time spent on attempts a later retry superseded, in the window's finished runs of the source on branchScope; wallClockMs is the time those runs used - each shard's own duration, summed - and lostPercent is lostMs of it."
6654
6897
  },
6655
6898
  "failingTests": {
6656
6899
  "type": "object",
@@ -6666,15 +6909,19 @@
6666
6909
  "failing",
6667
6910
  "testsRun"
6668
6911
  ],
6669
- "description": "Tests that failed at least once in the window on branchScope, out of the tests those runs reported - every run, local ones included. Not the dashboard's Failing tests tile: that is testMetrics.failingTests, over CI runs on every branch."
6912
+ "description": "Tests whose final attempt failed in at least one of the window's finished runs of the source on branchScope, out of the tests those runs reported (a skipped test counts as reported)."
6670
6913
  },
6671
6914
  "testMetrics": {
6672
6915
  "type": "object",
6673
6916
  "properties": {
6674
6917
  "population": {
6675
6918
  "type": "string",
6676
- "const": "ci_completed_not_excluded",
6677
- "description": "Which runs every number counts: completed CI runs (reported with a CI provider) started inside the period, without the runs excluded as a mass failure (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count."
6919
+ "enum": [
6920
+ "ci_groups_finished_not_mass_failure",
6921
+ "local_groups_finished_not_mass_failure",
6922
+ "all_groups_finished_not_mass_failure"
6923
+ ],
6924
+ "description": "Which runs every number counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Runs that have not finished are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.8; before it, ci_groups_finished_not_mass_failure was the only value, and since 1.0.0-beta.6 - it was ci_completed_not_excluded, which counted every completed shard on its own)."
6678
6925
  },
6679
6926
  "days": {
6680
6927
  "type": "integer",
@@ -6690,7 +6937,7 @@
6690
6937
  },
6691
6938
  "ciTests": {
6692
6939
  "type": "integer",
6693
- "description": "Tests with at least one CI run in the period."
6940
+ "description": "Tests with at least one run of the population in the period - CI runs by default; the name is from when only CI runs counted."
6694
6941
  }
6695
6942
  },
6696
6943
  "required": [
@@ -6700,7 +6947,7 @@
6700
6947
  "failingTests",
6701
6948
  "ciTests"
6702
6949
  ],
6703
- "description": "The dashboard's Tests with retries and Failing tests tiles and their denominator, counted like /tests: CI runs only, every branch - not narrowed by branchScope. The top-level failingTests counts differently: every run in the window on branchScope, local ones included."
6950
+ "description": "The dashboard's Tests with retries and Failing tests tiles and their denominator, counted like /tests on branchScope (since 1.0.0-beta.7; every branch before). Mass-failure runs are left out here, as in every per-test metric."
6704
6951
  },
6705
6952
  "slowestTests": {
6706
6953
  "type": "array",
@@ -6733,7 +6980,8 @@
6733
6980
  "medianMs",
6734
6981
  "runs"
6735
6982
  ]
6736
- }
6983
+ },
6984
+ "description": "Up to 5 tests by the median of their final-attempt durations, one per run, at least 3 runs; skipped results left out."
6737
6985
  },
6738
6986
  "timeSinks": {
6739
6987
  "type": "array",
@@ -6762,7 +7010,8 @@
6762
7010
  "filePath",
6763
7011
  "lostMs"
6764
7012
  ]
6765
- }
7013
+ },
7014
+ "description": "Up to 5 tests by the time their superseded attempts cost in the window (at least a second each), worst first."
6766
7015
  },
6767
7016
  "runsSeries": {
6768
7017
  "type": "array",
@@ -6796,7 +7045,7 @@
6796
7045
  "durationMs"
6797
7046
  ]
6798
7047
  },
6799
- "description": "The last 20 runs on branchScope, oldest first - the dashboard's Runs and Run duration charts. Cancelled runs are left out, every shard of a cancelled CI run included."
7048
+ "description": "The last 30 finished runs of the source on branchScope, whatever the window, oldest first - one point per run, passed counting first-try passes, durationMs from its first shard's start to its last shard's finish. Cancelled runs are left out."
6800
7049
  }
6801
7050
  },
6802
7051
  "required": [
@@ -6806,6 +7055,7 @@
6806
7055
  "stableBranch",
6807
7056
  "lastRun",
6808
7057
  "runsInWindow",
7058
+ "unfinishedRuns",
6809
7059
  "suiteSize",
6810
7060
  "windowStability",
6811
7061
  "timeLost",
package/dist/tools.js CHANGED
@@ -54,13 +54,16 @@ export const TOOLS = [
54
54
  operationId: "getTestMetrics",
55
55
  description: "One test's numbers over a period: runs, stability (runs that passed on the first try " +
56
56
  "/ runs), flakyRate (runs that passed only after a retry / runs), failureRate (failed " +
57
- "runs / runs), time lost to retries and failures, its last 20 runs (runs it skipped " +
58
- "included, as result \"skipped\" - shown, never counted in runs or any rate) and the " +
59
- "branches it ran on. Counts completed CI runs of the period, without runs where most " +
60
- "of the suite failed at once; local runs never count. There is no verdict - the tool " +
61
- "returns the numbers and you judge them. runs = 0 with null rates is a valid answer: " +
62
- "no completed CI run of this test in the period. Cheap; for one test use this, not " +
63
- "squally-list-tests.",
57
+ "runs / runs), time lost to retries and failures, its last 20 runs (each with the run " +
58
+ "id squally-find-run gives; runs it skipped included, as result \"skipped\" - shown, " +
59
+ "never counted in runs or any rate) and the branches it ran on. Counts the CI runs of " +
60
+ "the period whose shards all finished - a sharded run counts once, by its first " +
61
+ "shard's start - without runs that were a mass failure as a whole; source=local or " +
62
+ "source=all counts local runs or both instead (population names which), and a run " +
63
+ "still in progress counts once all its shards have finished. There " +
64
+ "is no verdict - the tool returns the numbers and you judge them. runs = 0 " +
65
+ "with null rates is a valid answer: no counted CI run of this test in the period. " +
66
+ "Cheap; for one test use this, not squally-list-tests.",
64
67
  answers: "One test's stability, flaky rate and failure rate over a period.",
65
68
  cost: "cheap",
66
69
  },
@@ -68,7 +71,7 @@ export const TOOLS = [
68
71
  name: "squally-list-tests",
69
72
  title: "List tests",
70
73
  operationId: "listTests",
71
- description: "Every test with a completed CI run in the period, with the same numbers per test as " +
74
+ description: "Every test with a counted CI run in the period (or local run, with source), with the same numbers per test as " +
72
75
  "squally-get-test-metrics - runs, stability, flakyRate, failureRate, time lost - ranked " +
73
76
  "worst first by the chosen sort (default: lowest stability). There is no verdict and no " +
74
77
  "status - the tool labels no test; you judge the numbers. Moderate - two queries over " +
@@ -82,8 +85,11 @@ export const TOOLS = [
82
85
  operationId: "listErrors",
83
86
  description: "The project's error signatures in a period: failures grouped by fingerprint, with how " +
84
87
  "often and how recently each occurred. Answers 'is this failure one instance of something " +
85
- "that keeps happening' - a grouping no agent can compute from the repository alone.",
86
- answers: "Error signatures in a period: what keeps failing, grouped.",
88
+ "that keeps happening' - a grouping no agent can compute from the repository alone. " +
89
+ "Counts finished CI runs by default (source=local or all for the others), a sharded run " +
90
+ "once, mass-failure runs included; failedCount and flakyCount say how the runs ended " +
91
+ "for the tests the error hit.",
92
+ answers: "Error signatures in a period: what keeps failing, grouped, and how its runs ended.",
87
93
  cost: "cheap",
88
94
  },
89
95
  ];
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "squally-mcp",
3
- "version": "0.2.3",
3
+ "version": "0.4.0",
4
4
  "description": "Local MCP server for Squally — read-only access to your Playwright test runs, flaky tests and failures.",
5
5
  "license": "MIT",
6
6
  "author": "Squally",