squally-mcp 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -143,7 +143,7 @@ squally-mcp trusts your operating system's certificate store, like your browser
143
143
  | `squally-debug-failure` | Every attempt of one test in one run: error, stack, Copy-for-AI prompt. | cheap |
144
144
  | `squally-get-test-metrics` | One test's stability, flaky rate and failure rate over a period. | cheap — one test |
145
145
  | `squally-list-tests` | Every test with a CI run in the period, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
146
- | `squally-list-errors` | Error signatures in a period: what keeps failing, grouped. | cheap |
146
+ | `squally-list-errors` | Error signatures in a period: what keeps failing, grouped, and how its runs ended. | cheap |
147
147
 
148
148
  For a single test use `squally-get-test-metrics`, not `squally-list-tests` —
149
149
  the tool descriptions say so, and the server repeats it in its instructions.
@@ -153,8 +153,9 @@ the tool descriptions say so, and the server repeats it in its instructions.
153
153
  The two test tools count **CI runs whose shards all finished** in the period
154
154
  (7, 14, 30 or 90 days; 30 by default), on every branch unless you name one. A
155
155
  sharded run counts once, placed in the period by its first shard's start; a
156
- run that was a mass failure as a whole is left out. Local runs never count.
157
- Per test:
156
+ run that was a mass failure as a whole is left out. Local runs count only
157
+ when you ask for them: `source` is `ci` (the default), `local` or `all`, and
158
+ the response's `population` names which runs it counted. Per test:
158
159
 
159
160
  - **stability** — runs that passed on the first try ÷ runs;
160
161
  - **flakyRate** — runs that passed only after a retry ÷ runs;
@@ -17,14 +17,23 @@
17
17
  // runs whose shards all finished, each counted once ("ci_groups_finished_not_
18
18
  // mass_failure"); it was every completed shard on its own.
19
19
  //
20
+ // RUNS IN PROGRESS SINCE 0.4.0 (API 1.0.0-beta.7): said outright, because the
21
+ // newest run is the one an agent asks about, and it is missing from the
22
+ // numbers until its last shard has finished.
23
+ //
24
+ // THE SOURCE SINCE 0.4.0 (API 1.0.0-beta.8): CI runs by default; local runs
25
+ // count only when the agent asks for them - they were never counted before.
26
+ //
20
27
  // Exported as `squally-mcp/instructions` (package.json), so it stays a plain
21
28
  // constant - test/exports.test.js imports it through the package name.
22
29
  export const INSTRUCTIONS = [
23
30
  "Start with squally-list-projects; every other tool needs a projectId from it.",
24
31
  "For one test, use squally-get-test-metrics, not squally-list-tests.",
25
32
  "Test metrics count the CI runs of the period whose shards all finished - a sharded",
26
- "run counts once - without runs that were a mass failure as a whole; local runs never",
27
- "count. stability = runs that passed on the",
33
+ "run counts once - without runs that were a mass failure as a whole; pass source=local",
34
+ "or source=all to count local runs or both, and a run still in progress counts once",
35
+ "all its shards have finished.",
36
+ "stability = runs that passed on the",
28
37
  "first try / runs; flakyRate = runs that passed only after a retry / runs;",
29
38
  "failureRate = failed runs / runs. There is no verdict: the tools return numbers,",
30
39
  "and you judge them. If a test name is ambiguous, repeat with filePath from the",
@@ -3,7 +3,7 @@
3
3
  "jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
4
4
  "info": {
5
5
  "title": "Squally Read API",
6
- "version": "1.0.0-beta.6",
6
+ "version": "1.0.0-beta.9",
7
7
  "description": "Read access to a Squally organization's test results, for scripts, CI jobs and AI agents.\n\n**Authentication.** Every request carries an organization read key as `Authorization: Bearer sqly_ro_…`. A read key is organization-wide (every project), created by an organization admin in Organization › Read keys, and expires after 30, 60, 90, 180 days as chosen at creation. There is no renewal: create a new key and revoke the old one. Project ingest keys (`sqly_…`) are refused with `wrong_key_type`.\n\n**Expiry warning.** From 7 days before a key expires, every authenticated response carries `Squally-Key-Expires-At` with the expiry instant (ISO-8601).\n\n**Rate limit.** Currently 60 requests per 60 seconds per key, counted per `Authorization` header - a read key has its own budget, shared with nothing else. Over the limit the edge answers 429 without a JSON body.\n\n**Retries.** 429: wait 60 seconds, then retry. The edge counts requests per key in 60-second windows and sends no Retry-After header. 5xx: retry with exponential backoff, starting at 1 second and capped at 30 seconds. 400, 401, 403, 404: never retry - the same request gets the same answer.\n\n**Plans.** The read API is part of the Standard and Pro plans. A key of an organization on Free is answered 403 `tier_too_low`; keys are not revoked on a downgrade and work again after an upgrade.\n\n**Errors.** Every error is `{\"error\": \"<sentence>\", \"code\": \"<slug>\"}`. The sentence may be reworded; the slug may not. An id that does not exist and an id that belongs to another organization get the same 404, byte for byte.\n\n**Test names in paths.** Test names are Playwright title paths and often contain `/`: URL-encode them, including `/` as `%2F`.\n\n**Sharded runs** are returned as one logical run: /runs lists one row per logical run, and the run routes answer for all shards whichever shard's id you pass.\n\n**Stability.** v1, beta. The API may change until the beta ends; every change is listed in the changelog (docs/openapi/CHANGELOG.md)."
8
8
  },
9
9
  "servers": [
@@ -1675,7 +1675,7 @@
1675
1675
  "get": {
1676
1676
  "operationId": "listTests",
1677
1677
  "summary": "List tests",
1678
- "description": "Every test with a CI run in the period and its metrics - stability, flaky rate, failure rate, runs, time lost, its last 20 runs, the branches it ran on and where it fails most - ranked by the chosen sort. Counts completed CI runs only, without mass-failure runs, on every branch unless branch names one (see population). Moderate: two queries over the period's CI results, paged in the database; use the metrics route for one test.",
1678
+ "description": "Every test with a run of the source (CI by default) in the period and its metrics - stability, flaky rate, failure rate, runs, time lost, its last 20 runs, the branches it ran on and where it fails most - ranked by the chosen sort. Counts completed runs of the source - CI by default - without mass-failure runs, on every branch unless branch names one (see population). Moderate: two queries over the period's CI results, paged in the database; use the metrics route for one test.",
1679
1679
  "tags": [
1680
1680
  "Tests"
1681
1681
  ],
@@ -1727,6 +1727,21 @@
1727
1727
  "maxLength": 200
1728
1728
  }
1729
1729
  },
1730
+ {
1731
+ "name": "source",
1732
+ "in": "query",
1733
+ "required": false,
1734
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
1735
+ "schema": {
1736
+ "type": "string",
1737
+ "enum": [
1738
+ "ci",
1739
+ "local",
1740
+ "all"
1741
+ ],
1742
+ "default": "ci"
1743
+ }
1744
+ },
1730
1745
  {
1731
1746
  "name": "browser",
1732
1747
  "in": "query",
@@ -2284,7 +2299,7 @@
2284
2299
  "get": {
2285
2300
  "operationId": "getTestMetrics",
2286
2301
  "summary": "Get test metrics",
2287
- "description": "One test's metrics over the period: the fields of its /tests row, flat, beside the same period. A test with results but no CI run in the period answers zeros and null rates rather than 404. Cheap: the list's queries, narrowed to one test; prefer it to /tests when the question is about one test.",
2302
+ "description": "One test's metrics over the period: the fields of its /tests row, flat, beside the same period. A test with results but no run of the source in the period answers zeros and null rates rather than 404. Cheap: the list's queries, narrowed to one test; prefer it to /tests when the question is about one test.",
2288
2303
  "tags": [
2289
2304
  "Tests"
2290
2305
  ],
@@ -2342,6 +2357,21 @@
2342
2357
  "minLength": 1,
2343
2358
  "maxLength": 200
2344
2359
  }
2360
+ },
2361
+ {
2362
+ "name": "source",
2363
+ "in": "query",
2364
+ "required": false,
2365
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
2366
+ "schema": {
2367
+ "type": "string",
2368
+ "enum": [
2369
+ "ci",
2370
+ "local",
2371
+ "all"
2372
+ ],
2373
+ "default": "ci"
2374
+ }
2345
2375
  }
2346
2376
  ],
2347
2377
  "responses": {
@@ -2695,7 +2725,7 @@
2695
2725
  "get": {
2696
2726
  "operationId": "getTestHistory",
2697
2727
  "summary": "Get test history",
2698
- "description": "Get a test's history across runs. One test in every run of a period, newest first: final status, duration, attempts and time spent on retries. Moderate.",
2728
+ "description": "Get a test's history across runs. One test in every run of a period, newest first: final status, duration, attempts and time spent on retries. Every run is listed; source decides which are counted. Moderate.",
2699
2729
  "tags": [
2700
2730
  "Tests"
2701
2731
  ],
@@ -2742,6 +2772,21 @@
2742
2772
  ],
2743
2773
  "default": 30
2744
2774
  }
2775
+ },
2776
+ {
2777
+ "name": "source",
2778
+ "in": "query",
2779
+ "required": false,
2780
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
2781
+ "schema": {
2782
+ "type": "string",
2783
+ "enum": [
2784
+ "ci",
2785
+ "local",
2786
+ "all"
2787
+ ],
2788
+ "default": "ci"
2789
+ }
2745
2790
  }
2746
2791
  ],
2747
2792
  "responses": {
@@ -2808,7 +2853,7 @@
2808
2853
  }
2809
2854
  },
2810
2855
  "400": {
2811
- "description": "Error: ambiguous_test, invalid_period",
2856
+ "description": "Error: ambiguous_test, invalid_parameter, invalid_period",
2812
2857
  "content": {
2813
2858
  "application/json": {
2814
2859
  "schema": {
@@ -2822,6 +2867,7 @@
2822
2867
  "type": "string",
2823
2868
  "enum": [
2824
2869
  "ambiguous_test",
2870
+ "invalid_parameter",
2825
2871
  "invalid_period"
2826
2872
  ],
2827
2873
  "description": "Stable slug - match on this, not on the sentence."
@@ -2840,6 +2886,13 @@
2840
2886
  "code": "ambiguous_test"
2841
2887
  }
2842
2888
  },
2889
+ "invalid_parameter": {
2890
+ "summary": "invalid_parameter",
2891
+ "value": {
2892
+ "error": "sha must be 4 to 40 hexadecimal characters (a commit SHA or a prefix of one).",
2893
+ "code": "invalid_parameter"
2894
+ }
2895
+ },
2843
2896
  "invalid_period": {
2844
2897
  "summary": "invalid_period",
2845
2898
  "value": {
@@ -3045,7 +3098,7 @@
3045
3098
  "get": {
3046
3099
  "operationId": "listErrors",
3047
3100
  "summary": "List errors",
3048
- "description": "List errors. The project's errors in a period - failures grouped by fingerprint - most frequent first. Every count covers the period; firstSeen is the first occurrence still stored, not bounded by it. Moderate: one aggregate query.",
3101
+ "description": "List errors. The project's errors in a period - failures grouped by fingerprint - most frequent first, over the finished runs of the source (see population), all shards of a run one run. Every count covers the period; firstSeen is the first occurrence still stored, not bounded by it. Moderate: one aggregate query.",
3049
3102
  "tags": [
3050
3103
  "Errors"
3051
3104
  ],
@@ -3075,6 +3128,21 @@
3075
3128
  "default": 30
3076
3129
  }
3077
3130
  },
3131
+ {
3132
+ "name": "source",
3133
+ "in": "query",
3134
+ "required": false,
3135
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
3136
+ "schema": {
3137
+ "type": "string",
3138
+ "enum": [
3139
+ "ci",
3140
+ "local",
3141
+ "all"
3142
+ ],
3143
+ "default": "ci"
3144
+ }
3145
+ },
3078
3146
  {
3079
3147
  "name": "page",
3080
3148
  "in": "query",
@@ -3120,6 +3188,7 @@
3120
3188
  "id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
3121
3189
  "name": "storefront-web"
3122
3190
  },
3191
+ "population": "ci_groups_finished",
3123
3192
  "window": {
3124
3193
  "days": 30,
3125
3194
  "from": "2026-08-25T08:00:00.000Z",
@@ -3145,12 +3214,15 @@
3145
3214
  ],
3146
3215
  "occurrenceCount": 9,
3147
3216
  "runCount": 7,
3217
+ "failedCount": 5,
3218
+ "flakyCount": 2,
3219
+ "massFailureRunCount": 0,
3148
3220
  "firstSeen": "2026-09-02T11:20:44.000Z",
3149
- "lastSeen": "2026-09-24T07:34:05.000Z",
3150
- "isNew": true
3221
+ "lastSeen": "2026-09-24T07:34:05.000Z"
3151
3222
  }
3152
3223
  ],
3153
- "unfingerprintedCount": 0
3224
+ "unfingerprintedCount": 0,
3225
+ "unfinishedRuns": 0
3154
3226
  }
3155
3227
  }
3156
3228
  }
@@ -3385,7 +3457,7 @@
3385
3457
  "get": {
3386
3458
  "operationId": "getError",
3387
3459
  "summary": "Get an error",
3388
- "description": "Get an error. One error: its counts, the tests it hits and the runs it occurred in. An error quiet in the period is answered over everything still stored (windowEmpty: true). Moderate.",
3460
+ "description": "Get an error. One error: its counts, the tests it hits and the runs it occurred in - one row per run and test, by the run's id as /runs gives it. An error quiet in the period is answered over everything still stored (windowEmpty: true). Moderate.",
3389
3461
  "tags": [
3390
3462
  "Errors"
3391
3463
  ],
@@ -3424,6 +3496,21 @@
3424
3496
  ],
3425
3497
  "default": 30
3426
3498
  }
3499
+ },
3500
+ {
3501
+ "name": "source",
3502
+ "in": "query",
3503
+ "required": false,
3504
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
3505
+ "schema": {
3506
+ "type": "string",
3507
+ "enum": [
3508
+ "ci",
3509
+ "local",
3510
+ "all"
3511
+ ],
3512
+ "default": "ci"
3513
+ }
3427
3514
  }
3428
3515
  ],
3429
3516
  "responses": {
@@ -3444,6 +3531,7 @@
3444
3531
  "id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
3445
3532
  "name": "storefront-web"
3446
3533
  },
3534
+ "population": "ci_groups_finished",
3447
3535
  "window": {
3448
3536
  "days": 30,
3449
3537
  "from": "2026-08-25T08:00:00.000Z",
@@ -3462,16 +3550,21 @@
3462
3550
  ],
3463
3551
  "occurrenceCount": 9,
3464
3552
  "runCount": 7,
3553
+ "failedCount": 5,
3554
+ "flakyCount": 2,
3555
+ "massFailureRunCount": 0,
3465
3556
  "firstSeen": "2026-09-02T11:20:44.000Z",
3466
- "lastSeen": "2026-09-24T07:34:05.000Z",
3467
- "isNew": true
3557
+ "lastSeen": "2026-09-24T07:34:05.000Z"
3468
3558
  },
3469
3559
  "windowEmpty": false,
3470
3560
  "tests": [
3471
3561
  {
3472
3562
  "testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code",
3473
3563
  "filePath": "tests/checkout.spec.ts",
3474
- "occurrenceCount": 9
3564
+ "occurrenceCount": 9,
3565
+ "runCount": 7,
3566
+ "failedCount": 5,
3567
+ "flakyCount": 2
3475
3568
  }
3476
3569
  ],
3477
3570
  "runs": [
@@ -3484,7 +3577,8 @@
3484
3577
  "filePath": "tests/checkout.spec.ts",
3485
3578
  "attempts": 2,
3486
3579
  "occurrences": 1,
3487
- "finalStatus": "passed"
3580
+ "finalStatus": "passed",
3581
+ "massFailure": false
3488
3582
  }
3489
3583
  ]
3490
3584
  }
@@ -3492,7 +3586,7 @@
3492
3586
  }
3493
3587
  },
3494
3588
  "400": {
3495
- "description": "Error: invalid_period",
3589
+ "description": "Error: invalid_parameter, invalid_period",
3496
3590
  "content": {
3497
3591
  "application/json": {
3498
3592
  "schema": {
@@ -3505,6 +3599,7 @@
3505
3599
  "code": {
3506
3600
  "type": "string",
3507
3601
  "enum": [
3602
+ "invalid_parameter",
3508
3603
  "invalid_period"
3509
3604
  ],
3510
3605
  "description": "Stable slug - match on this, not on the sentence."
@@ -3516,6 +3611,13 @@
3516
3611
  ]
3517
3612
  },
3518
3613
  "examples": {
3614
+ "invalid_parameter": {
3615
+ "summary": "invalid_parameter",
3616
+ "value": {
3617
+ "error": "sha must be 4 to 40 hexadecimal characters (a commit SHA or a prefix of one).",
3618
+ "code": "invalid_parameter"
3619
+ }
3620
+ },
3519
3621
  "invalid_period": {
3520
3622
  "summary": "invalid_period",
3521
3623
  "value": {
@@ -3721,7 +3823,7 @@
3721
3823
  "get": {
3722
3824
  "operationId": "getOverview",
3723
3825
  "summary": "Get dashboard figures",
3724
- "description": "The dashboard for a period: last run, suite stability, time lost, the two test-metric tiles (testMetrics), slowest tests, time sinks, recent runs. Suite stability and testMetrics count completed CI runs only. Moderate: one pass over the period's runs and one count over its CI results.",
3826
+ "description": "The dashboard for a period: last run, suite stability, time lost, the two test-metric tiles (testMetrics), slowest tests, time sinks, recent runs. Every figure counts finished runs of the source - CI by default - and testMetrics.population names it. Moderate: one pass over the period's runs and one count over its results.",
3725
3827
  "tags": [
3726
3828
  "Project"
3727
3829
  ],
@@ -3764,6 +3866,21 @@
3764
3866
  ],
3765
3867
  "default": "stable"
3766
3868
  }
3869
+ },
3870
+ {
3871
+ "name": "source",
3872
+ "in": "query",
3873
+ "required": false,
3874
+ "description": "Which runs count, by where they ran: ci - runs a shard reported a CI provider for (the default); local - runs no shard did; all - both. The response's population names the one counted.",
3875
+ "schema": {
3876
+ "type": "string",
3877
+ "enum": [
3878
+ "ci",
3879
+ "local",
3880
+ "all"
3881
+ ],
3882
+ "default": "ci"
3883
+ }
3767
3884
  }
3768
3885
  ],
3769
3886
  "responses": {
@@ -3799,11 +3916,14 @@
3799
3916
  "startedAt": "2026-09-24T07:31:12.000Z",
3800
3917
  "finishedAt": "2026-09-24T07:39:47.000Z",
3801
3918
  "total": 412,
3802
- "passed": 408,
3919
+ "passed": 404,
3920
+ "flaky": 4,
3803
3921
  "failed": 0,
3922
+ "skipped": 4,
3804
3923
  "isFallbackBranch": false
3805
3924
  },
3806
3925
  "runsInWindow": 11,
3926
+ "unfinishedRuns": 1,
3807
3927
  "suiteSize": 412,
3808
3928
  "windowStability": 99,
3809
3929
  "timeLost": {
@@ -5427,8 +5547,12 @@
5427
5547
  },
5428
5548
  "population": {
5429
5549
  "type": "string",
5430
- "const": "ci_groups_finished_not_mass_failure",
5431
- "description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
5550
+ "enum": [
5551
+ "ci_groups_finished_not_mass_failure",
5552
+ "local_groups_finished_not_mass_failure",
5553
+ "all_groups_finished_not_mass_failure"
5554
+ ],
5555
+ "description": "Which runs every number counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Runs that have not finished are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.8; before it, ci_groups_finished_not_mass_failure was the only value, and since 1.0.0-beta.6 - it was ci_completed_not_excluded, which counted every completed shard on its own)."
5432
5556
  },
5433
5557
  "days": {
5434
5558
  "type": "integer",
@@ -5728,8 +5852,12 @@
5728
5852
  },
5729
5853
  "population": {
5730
5854
  "type": "string",
5731
- "const": "ci_groups_finished_not_mass_failure",
5732
- "description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
5855
+ "enum": [
5856
+ "ci_groups_finished_not_mass_failure",
5857
+ "local_groups_finished_not_mass_failure",
5858
+ "all_groups_finished_not_mass_failure"
5859
+ ],
5860
+ "description": "Which runs every number counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Runs that have not finished are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.8; before it, ci_groups_finished_not_mass_failure was the only value, and since 1.0.0-beta.6 - it was ci_completed_not_excluded, which counted every completed shard on its own)."
5733
5861
  },
5734
5862
  "days": {
5735
5863
  "type": "integer",
@@ -6033,11 +6161,11 @@
6033
6161
  },
6034
6162
  "local": {
6035
6163
  "type": "boolean",
6036
- "description": "No shard of the run reported a CI provider: a local run. Listed, never counted in a metric (since 1.0.0-beta.6)."
6164
+ "description": "No shard of the run reported a CI provider: a local run. Listed always; counted only under source=local or source=all (since 1.0.0-beta.8; never counted from 1.0.0-beta.6)."
6037
6165
  },
6038
6166
  "counted": {
6039
6167
  "type": "boolean",
6040
- "description": "Whether the run is in the metrics' population (see population on the metrics endpoints): a CI run whose shards all finished, not a mass failure, and the test not just skipped by a cancellation (since 1.0.0-beta.6)."
6168
+ "description": "Whether the run is in the metrics' population (see population on the metrics endpoints): a run of the source parameter's kind - CI by default - whose shards all finished, not a mass failure, and the test not just skipped by a cancellation (since 1.0.0-beta.6; the source since 1.0.0-beta.8)."
6041
6169
  },
6042
6170
  "branch": {
6043
6171
  "anyOf": [
@@ -6150,6 +6278,15 @@
6150
6278
  "name"
6151
6279
  ]
6152
6280
  },
6281
+ "population": {
6282
+ "type": "string",
6283
+ "enum": [
6284
+ "ci_groups_finished",
6285
+ "local_groups_finished",
6286
+ "all_groups_finished"
6287
+ ],
6288
+ "description": "Which runs every error figure counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - runs that were a mass failure as a whole included (see massFailureRunCount). Runs that have not finished are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.9)."
6289
+ },
6153
6290
  "window": {
6154
6291
  "type": "object",
6155
6292
  "properties": {
@@ -6237,19 +6374,29 @@
6237
6374
  "type": "integer"
6238
6375
  },
6239
6376
  "runCount": {
6240
- "type": "integer"
6377
+ "type": "integer",
6378
+ "description": "Runs of the population with at least one such attempt in the window - all shards of a run one run (since 1.0.0-beta.9; it counted every shard row)."
6379
+ },
6380
+ "failedCount": {
6381
+ "type": "integer",
6382
+ "description": "Of the (run, test) pairs the error hit, how many ended failed: the test's final attempt across all the run's shards failed (since 1.0.0-beta.9)."
6383
+ },
6384
+ "flakyCount": {
6385
+ "type": "integer",
6386
+ "description": "Of the same pairs, how many ended flaky: the final attempt passed after a retry. A pair whose final attempt skipped is neither (since 1.0.0-beta.9)."
6387
+ },
6388
+ "massFailureRunCount": {
6389
+ "type": "integer",
6390
+ "description": "Of runCount, the runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed) - counted, and named here (since 1.0.0-beta.9)."
6241
6391
  },
6242
6392
  "firstSeen": {
6243
6393
  "type": "string",
6244
6394
  "format": "date-time",
6245
- "description": "First seen in what is still stored: the earliest occurrence the plan's retention has kept, NOT bounded by the requested period. ISO-8601."
6395
+ "description": "First seen in what is still stored: the start of the earliest run of the population, as far back as the plan's retention keeps them, NOT bounded by the requested period. ISO-8601. (isNew, \"absent in the period before\", was removed in 1.0.0-beta.9: compare firstSeen with window.from.)"
6246
6396
  },
6247
6397
  "lastSeen": {
6248
6398
  "type": "string",
6249
6399
  "format": "date-time"
6250
- },
6251
- "isNew": {
6252
- "type": "boolean"
6253
6400
  }
6254
6401
  },
6255
6402
  "required": [
@@ -6263,22 +6410,31 @@
6263
6410
  "testNames",
6264
6411
  "occurrenceCount",
6265
6412
  "runCount",
6413
+ "failedCount",
6414
+ "flakyCount",
6415
+ "massFailureRunCount",
6266
6416
  "firstSeen",
6267
- "lastSeen",
6268
- "isNew"
6417
+ "lastSeen"
6269
6418
  ]
6270
6419
  }
6271
6420
  },
6272
6421
  "unfingerprintedCount": {
6273
- "type": "integer"
6422
+ "type": "integer",
6423
+ "description": "Failed attempts in the window's runs of the population that carry no signature and are not grouped (the window only since 1.0.0-beta.9; it read the period before too). Expected 0."
6424
+ },
6425
+ "unfinishedRuns": {
6426
+ "type": "integer",
6427
+ "description": "Runs of the source in the window that have not finished - a shard still running, timed out or never reported. In no figure (since 1.0.0-beta.9)."
6274
6428
  }
6275
6429
  },
6276
6430
  "required": [
6277
6431
  "project",
6432
+ "population",
6278
6433
  "window",
6279
6434
  "page",
6280
6435
  "errors",
6281
- "unfingerprintedCount"
6436
+ "unfingerprintedCount",
6437
+ "unfinishedRuns"
6282
6438
  ]
6283
6439
  },
6284
6440
  "ErrorDetailResponse": {
@@ -6299,6 +6455,15 @@
6299
6455
  "name"
6300
6456
  ]
6301
6457
  },
6458
+ "population": {
6459
+ "type": "string",
6460
+ "enum": [
6461
+ "ci_groups_finished",
6462
+ "local_groups_finished",
6463
+ "all_groups_finished"
6464
+ ],
6465
+ "description": "Which runs every error figure counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - runs that were a mass failure as a whole included (see massFailureRunCount). Runs that have not finished are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.9)."
6466
+ },
6302
6467
  "window": {
6303
6468
  "type": "object",
6304
6469
  "properties": {
@@ -6361,19 +6526,29 @@
6361
6526
  "type": "integer"
6362
6527
  },
6363
6528
  "runCount": {
6364
- "type": "integer"
6529
+ "type": "integer",
6530
+ "description": "Runs of the population with at least one such attempt in the window - all shards of a run one run (since 1.0.0-beta.9; it counted every shard row)."
6531
+ },
6532
+ "failedCount": {
6533
+ "type": "integer",
6534
+ "description": "Of the (run, test) pairs the error hit, how many ended failed: the test's final attempt across all the run's shards failed (since 1.0.0-beta.9)."
6535
+ },
6536
+ "flakyCount": {
6537
+ "type": "integer",
6538
+ "description": "Of the same pairs, how many ended flaky: the final attempt passed after a retry. A pair whose final attempt skipped is neither (since 1.0.0-beta.9)."
6539
+ },
6540
+ "massFailureRunCount": {
6541
+ "type": "integer",
6542
+ "description": "Of runCount, the runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed) - counted, and named here (since 1.0.0-beta.9)."
6365
6543
  },
6366
6544
  "firstSeen": {
6367
6545
  "type": "string",
6368
6546
  "format": "date-time",
6369
- "description": "First seen in what is still stored: the earliest occurrence the plan's retention has kept, NOT bounded by the requested period. ISO-8601."
6547
+ "description": "First seen in what is still stored: the start of the earliest run of the population, as far back as the plan's retention keeps them, NOT bounded by the requested period. ISO-8601. (isNew, \"absent in the period before\", was removed in 1.0.0-beta.9: compare firstSeen with window.from.)"
6370
6548
  },
6371
6549
  "lastSeen": {
6372
6550
  "type": "string",
6373
6551
  "format": "date-time"
6374
- },
6375
- "isNew": {
6376
- "type": "boolean"
6377
6552
  }
6378
6553
  },
6379
6554
  "required": [
@@ -6387,9 +6562,11 @@
6387
6562
  "testNames",
6388
6563
  "occurrenceCount",
6389
6564
  "runCount",
6565
+ "failedCount",
6566
+ "flakyCount",
6567
+ "massFailureRunCount",
6390
6568
  "firstSeen",
6391
- "lastSeen",
6392
- "isNew"
6569
+ "lastSeen"
6393
6570
  ]
6394
6571
  },
6395
6572
  "windowEmpty": {
@@ -6416,12 +6593,27 @@
6416
6593
  },
6417
6594
  "occurrenceCount": {
6418
6595
  "type": "integer"
6596
+ },
6597
+ "runCount": {
6598
+ "type": "integer",
6599
+ "description": "Runs in which this test hit the error (since 1.0.0-beta.9)."
6600
+ },
6601
+ "failedCount": {
6602
+ "type": "integer",
6603
+ "description": "Of those runs, the ones the test ended failed (since 1.0.0-beta.9)."
6604
+ },
6605
+ "flakyCount": {
6606
+ "type": "integer",
6607
+ "description": "Of those runs, the ones the test ended flaky - passed after a retry (since 1.0.0-beta.9)."
6419
6608
  }
6420
6609
  },
6421
6610
  "required": [
6422
6611
  "testName",
6423
6612
  "filePath",
6424
- "occurrenceCount"
6613
+ "occurrenceCount",
6614
+ "runCount",
6615
+ "failedCount",
6616
+ "flakyCount"
6425
6617
  ]
6426
6618
  }
6427
6619
  },
@@ -6431,7 +6623,8 @@
6431
6623
  "type": "object",
6432
6624
  "properties": {
6433
6625
  "runId": {
6434
- "type": "string"
6626
+ "type": "string",
6627
+ "description": "The run, by the id GET /projects/{projectId}/runs gives it - one row per run and test, however many shards the run had (since 1.0.0-beta.9; it was one row per shard, with the shard's id)."
6435
6628
  },
6436
6629
  "runNumber": {
6437
6630
  "type": "integer"
@@ -6464,13 +6657,20 @@
6464
6657
  ]
6465
6658
  },
6466
6659
  "attempts": {
6467
- "type": "integer"
6660
+ "type": "integer",
6661
+ "description": "Every attempt of the test in the run, retries and every shard included."
6468
6662
  },
6469
6663
  "occurrences": {
6470
- "type": "integer"
6664
+ "type": "integer",
6665
+ "description": "How many of those carry this error."
6471
6666
  },
6472
6667
  "finalStatus": {
6473
- "type": "string"
6668
+ "type": "string",
6669
+ "description": "The test's final attempt across all the run's shards, through test.fail(): passed, failed or skipped."
6670
+ },
6671
+ "massFailure": {
6672
+ "type": "boolean",
6673
+ "description": "The run was a mass failure as a whole (since 1.0.0-beta.9)."
6474
6674
  }
6475
6675
  },
6476
6676
  "required": [
@@ -6482,13 +6682,15 @@
6482
6682
  "filePath",
6483
6683
  "attempts",
6484
6684
  "occurrences",
6485
- "finalStatus"
6685
+ "finalStatus",
6686
+ "massFailure"
6486
6687
  ]
6487
6688
  }
6488
6689
  }
6489
6690
  },
6490
6691
  "required": [
6491
6692
  "project",
6693
+ "population",
6492
6694
  "window",
6493
6695
  "signature",
6494
6696
  "windowEmpty",
@@ -6551,7 +6753,8 @@
6551
6753
  "type": "object",
6552
6754
  "properties": {
6553
6755
  "id": {
6554
- "type": "string"
6756
+ "type": "string",
6757
+ "description": "The run's id as GET /projects/{projectId}/runs gives it - one id for all its shards."
6555
6758
  },
6556
6759
  "runNumber": {
6557
6760
  "type": "integer"
@@ -6574,11 +6777,13 @@
6574
6777
  {
6575
6778
  "type": "null"
6576
6779
  }
6577
- ]
6780
+ ],
6781
+ "description": "The run's status over all its shards: passed or failed."
6578
6782
  },
6579
6783
  "startedAt": {
6580
6784
  "type": "string",
6581
- "format": "date-time"
6785
+ "format": "date-time",
6786
+ "description": "Its first shard's start."
6582
6787
  },
6583
6788
  "finishedAt": {
6584
6789
  "anyOf": [
@@ -6589,17 +6794,28 @@
6589
6794
  {
6590
6795
  "type": "null"
6591
6796
  }
6592
- ]
6797
+ ],
6798
+ "description": "Its last shard's finish."
6593
6799
  },
6594
6800
  "total": {
6595
- "type": "integer"
6801
+ "type": "integer",
6802
+ "description": "Tests in the run, every test once over all shards: passed + flaky + failed + skipped."
6596
6803
  },
6597
6804
  "passed": {
6598
- "type": "integer"
6805
+ "type": "integer",
6806
+ "description": "Tests that passed on the first try."
6807
+ },
6808
+ "flaky": {
6809
+ "type": "integer",
6810
+ "description": "Tests that passed after a retry (since 1.0.0-beta.7)."
6599
6811
  },
6600
6812
  "failed": {
6601
6813
  "type": "integer"
6602
6814
  },
6815
+ "skipped": {
6816
+ "type": "integer",
6817
+ "description": "Since 1.0.0-beta.7."
6818
+ },
6603
6819
  "isFallbackBranch": {
6604
6820
  "type": "boolean"
6605
6821
  }
@@ -6613,21 +6829,29 @@
6613
6829
  "finishedAt",
6614
6830
  "total",
6615
6831
  "passed",
6832
+ "flaky",
6616
6833
  "failed",
6834
+ "skipped",
6617
6835
  "isFallbackBranch"
6618
6836
  ]
6619
6837
  },
6620
6838
  {
6621
6839
  "type": "null"
6622
6840
  }
6623
- ]
6841
+ ],
6842
+ "description": "The newest finished run of the source on branchScope, whatever the window - falling back to any branch when the stable branch has none (see the source parameter and testMetrics.population)."
6624
6843
  },
6625
6844
  "runsInWindow": {
6626
6845
  "type": "integer",
6627
- "description": "Completed CI runs in the window on branchScope, without mass-failure runs - the runs windowStability is counted over."
6846
+ "description": "Finished runs of the source in the window on branchScope (see the source parameter and testMetrics.population), each counted once however many shards it had, by the start of its first shard; mass-failure and cancelled runs included. The runs windowStability is counted over."
6847
+ },
6848
+ "unfinishedRuns": {
6849
+ "type": "integer",
6850
+ "description": "Runs of the source in the window on branchScope that have not finished - a shard still running, timed out or never reported. In no other figure (since 1.0.0-beta.7; of the source since 1.0.0-beta.8)."
6628
6851
  },
6629
6852
  "suiteSize": {
6630
- "type": "integer"
6853
+ "type": "integer",
6854
+ "description": "The largest run in the window, every test once over all its shards."
6631
6855
  },
6632
6856
  "windowStability": {
6633
6857
  "anyOf": [
@@ -6638,7 +6862,7 @@
6638
6862
  "type": "null"
6639
6863
  }
6640
6864
  ],
6641
- "description": "Suite stability, 0-100: of the final test results in the window's completed CI runs on branchScope (mass-failure runs and skipped results left out), the share that passed on the first try, rounded. A result that passed only after a retry counts against it. null when no such result exists."
6865
+ "description": "Suite stability, 0-100: of the final test results in the window's finished runs of the source on branchScope (skipped results left out), the share that passed on the first try, rounded. Mass-failure runs count (since 1.0.0-beta.7). A result that passed only after a retry counts against it. null when no such result exists."
6642
6866
  },
6643
6867
  "timeLost": {
6644
6868
  "type": "object",
@@ -6668,7 +6892,8 @@
6668
6892
  "lostPercent",
6669
6893
  "wastedAttempts",
6670
6894
  "wallClockMs"
6671
- ]
6895
+ ],
6896
+ "description": "Time spent on attempts a later retry superseded, in the window's finished runs of the source on branchScope; wallClockMs is the time those runs used - each shard's own duration, summed - and lostPercent is lostMs of it."
6672
6897
  },
6673
6898
  "failingTests": {
6674
6899
  "type": "object",
@@ -6684,15 +6909,19 @@
6684
6909
  "failing",
6685
6910
  "testsRun"
6686
6911
  ],
6687
- "description": "Tests that failed at least once in the window on branchScope, out of the tests those runs reported - every run, local ones included. Not the dashboard's Failing tests tile: that is testMetrics.failingTests, over CI runs on every branch."
6912
+ "description": "Tests whose final attempt failed in at least one of the window's finished runs of the source on branchScope, out of the tests those runs reported (a skipped test counts as reported)."
6688
6913
  },
6689
6914
  "testMetrics": {
6690
6915
  "type": "object",
6691
6916
  "properties": {
6692
6917
  "population": {
6693
6918
  "type": "string",
6694
- "const": "ci_groups_finished_not_mass_failure",
6695
- "description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
6919
+ "enum": [
6920
+ "ci_groups_finished_not_mass_failure",
6921
+ "local_groups_finished_not_mass_failure",
6922
+ "all_groups_finished_not_mass_failure"
6923
+ ],
6924
+ "description": "Which runs every number counts: runs whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Runs that have not finished are not counted. The prefix names the source parameter: ci_ - CI runs (a shard reported a CI provider), the default; local_ - runs no shard reported a CI provider for; all_ - both (since 1.0.0-beta.8; before it, ci_groups_finished_not_mass_failure was the only value, and since 1.0.0-beta.6 - it was ci_completed_not_excluded, which counted every completed shard on its own)."
6696
6925
  },
6697
6926
  "days": {
6698
6927
  "type": "integer",
@@ -6708,7 +6937,7 @@
6708
6937
  },
6709
6938
  "ciTests": {
6710
6939
  "type": "integer",
6711
- "description": "Tests with at least one CI run in the period."
6940
+ "description": "Tests with at least one run of the population in the period - CI runs by default; the name is from when only CI runs counted."
6712
6941
  }
6713
6942
  },
6714
6943
  "required": [
@@ -6718,7 +6947,7 @@
6718
6947
  "failingTests",
6719
6948
  "ciTests"
6720
6949
  ],
6721
- "description": "The dashboard's Tests with retries and Failing tests tiles and their denominator, counted like /tests: CI runs only, every branch - not narrowed by branchScope. The top-level failingTests counts differently: every run in the window on branchScope, local ones included."
6950
+ "description": "The dashboard's Tests with retries and Failing tests tiles and their denominator, counted like /tests on branchScope (since 1.0.0-beta.7; every branch before). Mass-failure runs are left out here, as in every per-test metric."
6722
6951
  },
6723
6952
  "slowestTests": {
6724
6953
  "type": "array",
@@ -6751,7 +6980,8 @@
6751
6980
  "medianMs",
6752
6981
  "runs"
6753
6982
  ]
6754
- }
6983
+ },
6984
+ "description": "Up to 5 tests by the median of their final-attempt durations, one per run, at least 3 runs; skipped results left out."
6755
6985
  },
6756
6986
  "timeSinks": {
6757
6987
  "type": "array",
@@ -6780,7 +7010,8 @@
6780
7010
  "filePath",
6781
7011
  "lostMs"
6782
7012
  ]
6783
- }
7013
+ },
7014
+ "description": "Up to 5 tests by the time their superseded attempts cost in the window (at least a second each), worst first."
6784
7015
  },
6785
7016
  "runsSeries": {
6786
7017
  "type": "array",
@@ -6814,7 +7045,7 @@
6814
7045
  "durationMs"
6815
7046
  ]
6816
7047
  },
6817
- "description": "The last 20 runs on branchScope, oldest first - the dashboard's Runs and Run duration charts. Cancelled runs are left out, every shard of a cancelled CI run included."
7048
+ "description": "The last 30 finished runs of the source on branchScope, whatever the window, oldest first - one point per run, passed counting first-try passes, durationMs from its first shard's start to its last shard's finish. Cancelled runs are left out."
6818
7049
  }
6819
7050
  },
6820
7051
  "required": [
@@ -6824,6 +7055,7 @@
6824
7055
  "stableBranch",
6825
7056
  "lastRun",
6826
7057
  "runsInWindow",
7058
+ "unfinishedRuns",
6827
7059
  "suiteSize",
6828
7060
  "windowStability",
6829
7061
  "timeLost",
package/dist/tools.js CHANGED
@@ -58,8 +58,10 @@ export const TOOLS = [
58
58
  "id squally-find-run gives; runs it skipped included, as result \"skipped\" - shown, " +
59
59
  "never counted in runs or any rate) and the branches it ran on. Counts the CI runs of " +
60
60
  "the period whose shards all finished - a sharded run counts once, by its first " +
61
- "shard's start - without runs that were a mass failure as a whole; local runs never " +
62
- "count. There is no verdict - the tool returns the numbers and you judge them. runs = 0 " +
61
+ "shard's start - without runs that were a mass failure as a whole; source=local or " +
62
+ "source=all counts local runs or both instead (population names which), and a run " +
63
+ "still in progress counts once all its shards have finished. There " +
64
+ "is no verdict - the tool returns the numbers and you judge them. runs = 0 " +
63
65
  "with null rates is a valid answer: no counted CI run of this test in the period. " +
64
66
  "Cheap; for one test use this, not squally-list-tests.",
65
67
  answers: "One test's stability, flaky rate and failure rate over a period.",
@@ -69,7 +71,7 @@ export const TOOLS = [
69
71
  name: "squally-list-tests",
70
72
  title: "List tests",
71
73
  operationId: "listTests",
72
- description: "Every test with a counted CI run in the period, with the same numbers per test as " +
74
+ description: "Every test with a counted CI run in the period (or local run, with source), with the same numbers per test as " +
73
75
  "squally-get-test-metrics - runs, stability, flakyRate, failureRate, time lost - ranked " +
74
76
  "worst first by the chosen sort (default: lowest stability). There is no verdict and no " +
75
77
  "status - the tool labels no test; you judge the numbers. Moderate - two queries over " +
@@ -83,8 +85,11 @@ export const TOOLS = [
83
85
  operationId: "listErrors",
84
86
  description: "The project's error signatures in a period: failures grouped by fingerprint, with how " +
85
87
  "often and how recently each occurred. Answers 'is this failure one instance of something " +
86
- "that keeps happening' - a grouping no agent can compute from the repository alone.",
87
- answers: "Error signatures in a period: what keeps failing, grouped.",
88
+ "that keeps happening' - a grouping no agent can compute from the repository alone. " +
89
+ "Counts finished CI runs by default (source=local or all for the others), a sharded run " +
90
+ "once, mass-failure runs included; failedCount and flakyCount say how the runs ended " +
91
+ "for the tests the error hit.",
92
+ answers: "Error signatures in a period: what keeps failing, grouped, and how its runs ended.",
88
93
  cost: "cheap",
89
94
  },
90
95
  ];
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "squally-mcp",
3
- "version": "0.3.0",
3
+ "version": "0.4.0",
4
4
  "description": "Local MCP server for Squally — read-only access to your Playwright test runs, flaky tests and failures.",
5
5
  "license": "MIT",
6
6
  "author": "Squally",