squally-mcp 0.2.3 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -5
- package/dist/instructions.js +7 -2
- package/dist/openapi/v1.json +37 -19
- package/dist/tools.js +9 -8
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -150,9 +150,11 @@ the tool descriptions say so, and the server repeats it in its instructions.
|
|
|
150
150
|
|
|
151
151
|
### What the numbers mean
|
|
152
152
|
|
|
153
|
-
The two test tools count **
|
|
154
|
-
days; 30 by default), on every branch unless you name one
|
|
155
|
-
|
|
153
|
+
The two test tools count **CI runs whose shards all finished** in the period
|
|
154
|
+
(7, 14, 30 or 90 days; 30 by default), on every branch unless you name one. A
|
|
155
|
+
sharded run counts once, placed in the period by its first shard's start; a
|
|
156
|
+
run that was a mass failure as a whole is left out. Local runs never count.
|
|
157
|
+
Per test:
|
|
156
158
|
|
|
157
159
|
- **stability** — runs that passed on the first try ÷ runs;
|
|
158
160
|
- **flakyRate** — runs that passed only after a retry ÷ runs;
|
|
@@ -172,7 +174,7 @@ The three least stable tests of the last 30 days (abridged):
|
|
|
172
174
|
|
|
173
175
|
```json
|
|
174
176
|
{
|
|
175
|
-
"population": "
|
|
177
|
+
"population": "ci_groups_finished_not_mass_failure",
|
|
176
178
|
"days": 30,
|
|
177
179
|
"sort": "stability",
|
|
178
180
|
"total": 409,
|
|
@@ -219,7 +221,7 @@ The same numbers as that test's row, for one test (abridged):
|
|
|
219
221
|
|
|
220
222
|
Pass the name exactly as Squally shows it. When it exists in several spec
|
|
221
223
|
files, the tool answers with the `filePath` values to retry with; a test with
|
|
222
|
-
results but no
|
|
224
|
+
results but no counted CI run in the period answers `runs: 0` and `null`
|
|
223
225
|
rates — an answer, not an error.
|
|
224
226
|
|
|
225
227
|
## Development
|
package/dist/instructions.js
CHANGED
|
@@ -13,13 +13,18 @@
|
|
|
13
13
|
// fractions of runs and nothing more; saying so here stops a model from
|
|
14
14
|
// presenting a rate as Squally's judgement.
|
|
15
15
|
//
|
|
16
|
+
// ONE RUN PER CI RUN SINCE 0.3.0 (API 1.0.0-beta.6): the population is the CI
|
|
17
|
+
// runs whose shards all finished, each counted once ("ci_groups_finished_not_
|
|
18
|
+
// mass_failure"); it was every completed shard on its own.
|
|
19
|
+
//
|
|
16
20
|
// Exported as `squally-mcp/instructions` (package.json), so it stays a plain
|
|
17
21
|
// constant - test/exports.test.js imports it through the package name.
|
|
18
22
|
export const INSTRUCTIONS = [
|
|
19
23
|
"Start with squally-list-projects; every other tool needs a projectId from it.",
|
|
20
24
|
"For one test, use squally-get-test-metrics, not squally-list-tests.",
|
|
21
|
-
"Test metrics count
|
|
22
|
-
"
|
|
25
|
+
"Test metrics count the CI runs of the period whose shards all finished - a sharded",
|
|
26
|
+
"run counts once - without runs that were a mass failure as a whole; local runs never",
|
|
27
|
+
"count. stability = runs that passed on the",
|
|
23
28
|
"first try / runs; flakyRate = runs that passed only after a retry / runs;",
|
|
24
29
|
"failureRate = failed runs / runs. There is no verdict: the tools return numbers,",
|
|
25
30
|
"and you judge them. If a test name is ambiguous, repeat with filePath from the",
|
package/dist/openapi/v1.json
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
|
|
4
4
|
"info": {
|
|
5
5
|
"title": "Squally Read API",
|
|
6
|
-
"version": "1.0.0-beta.
|
|
6
|
+
"version": "1.0.0-beta.6",
|
|
7
7
|
"description": "Read access to a Squally organization's test results, for scripts, CI jobs and AI agents.\n\n**Authentication.** Every request carries an organization read key as `Authorization: Bearer sqly_ro_…`. A read key is organization-wide (every project), created by an organization admin in Organization › Read keys, and expires after 30, 60, 90, 180 days as chosen at creation. There is no renewal: create a new key and revoke the old one. Project ingest keys (`sqly_…`) are refused with `wrong_key_type`.\n\n**Expiry warning.** From 7 days before a key expires, every authenticated response carries `Squally-Key-Expires-At` with the expiry instant (ISO-8601).\n\n**Rate limit.** Currently 60 requests per 60 seconds per key, counted per `Authorization` header - a read key has its own budget, shared with nothing else. Over the limit the edge answers 429 without a JSON body.\n\n**Retries.** 429: wait 60 seconds, then retry. The edge counts requests per key in 60-second windows and sends no Retry-After header. 5xx: retry with exponential backoff, starting at 1 second and capped at 30 seconds. 400, 401, 403, 404: never retry - the same request gets the same answer.\n\n**Plans.** The read API is part of the Standard and Pro plans. A key of an organization on Free is answered 403 `tier_too_low`; keys are not revoked on a downgrade and work again after an upgrade.\n\n**Errors.** Every error is `{\"error\": \"<sentence>\", \"code\": \"<slug>\"}`. The sentence may be reworded; the slug may not. An id that does not exist and an id that belongs to another organization get the same 404, byte for byte.\n\n**Test names in paths.** Test names are Playwright title paths and often contain `/`: URL-encode them, including `/` as `%2F`.\n\n**Sharded runs** are returned as one logical run: /runs lists one row per logical run, and the run routes answer for all shards whichever shard's id you pass.\n\n**Stability.** v1, beta. The API may change until the beta ends; every change is listed in the changelog (docs/openapi/CHANGELOG.md)."
|
|
8
8
|
},
|
|
9
9
|
"servers": [
|
|
@@ -1797,7 +1797,7 @@
|
|
|
1797
1797
|
"id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
|
|
1798
1798
|
"name": "storefront-web"
|
|
1799
1799
|
},
|
|
1800
|
-
"population": "
|
|
1800
|
+
"population": "ci_groups_finished_not_mass_failure",
|
|
1801
1801
|
"days": 30,
|
|
1802
1802
|
"since": "2026-08-25T08:00:00.000Z",
|
|
1803
1803
|
"until": "2026-09-24T08:00:00.000Z",
|
|
@@ -2362,7 +2362,7 @@
|
|
|
2362
2362
|
"id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
|
|
2363
2363
|
"name": "storefront-web"
|
|
2364
2364
|
},
|
|
2365
|
-
"population": "
|
|
2365
|
+
"population": "ci_groups_finished_not_mass_failure",
|
|
2366
2366
|
"days": 30,
|
|
2367
2367
|
"since": "2026-08-25T08:00:00.000Z",
|
|
2368
2368
|
"until": "2026-09-24T08:00:00.000Z",
|
|
@@ -2775,6 +2775,8 @@
|
|
|
2775
2775
|
{
|
|
2776
2776
|
"runId": "b1e0c6d2-7a43-4f19-8d5e-0c3a9e6f2b71",
|
|
2777
2777
|
"runNumber": 1284,
|
|
2778
|
+
"local": false,
|
|
2779
|
+
"counted": true,
|
|
2778
2780
|
"branch": "main",
|
|
2779
2781
|
"testedRevision": null,
|
|
2780
2782
|
"startedAt": "2026-09-24T07:31:12.000Z",
|
|
@@ -2788,6 +2790,8 @@
|
|
|
2788
2790
|
{
|
|
2789
2791
|
"runId": "7d2f4b19-3c8e-4a61-9e07-5b1d8c2a6f34",
|
|
2790
2792
|
"runNumber": 1283,
|
|
2793
|
+
"local": true,
|
|
2794
|
+
"counted": false,
|
|
2791
2795
|
"branch": "feature/saved-carts",
|
|
2792
2796
|
"testedRevision": "c4f7a1e9b3d5c8e2f0a6b4d9e1c7f3a5b8d2e0f6",
|
|
2793
2797
|
"startedAt": "2026-09-23T16:05:40.000Z",
|
|
@@ -3813,7 +3817,7 @@
|
|
|
3813
3817
|
"testsRun": 412
|
|
3814
3818
|
},
|
|
3815
3819
|
"testMetrics": {
|
|
3816
|
-
"population": "
|
|
3820
|
+
"population": "ci_groups_finished_not_mass_failure",
|
|
3817
3821
|
"days": 30,
|
|
3818
3822
|
"testsWithRetries": 7,
|
|
3819
3823
|
"failingTests": 2,
|
|
@@ -5423,8 +5427,8 @@
|
|
|
5423
5427
|
},
|
|
5424
5428
|
"population": {
|
|
5425
5429
|
"type": "string",
|
|
5426
|
-
"const": "
|
|
5427
|
-
"description": "Which runs every number counts:
|
|
5430
|
+
"const": "ci_groups_finished_not_mass_failure",
|
|
5431
|
+
"description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
|
|
5428
5432
|
},
|
|
5429
5433
|
"days": {
|
|
5430
5434
|
"type": "integer",
|
|
@@ -5480,7 +5484,7 @@
|
|
|
5480
5484
|
},
|
|
5481
5485
|
"total": {
|
|
5482
5486
|
"type": "integer",
|
|
5483
|
-
"description": "Tests matching the query across all pages: every test with at least one
|
|
5487
|
+
"description": "Tests matching the query across all pages: every test with at least one run of the population in the period, narrowed by search, branch and browser."
|
|
5484
5488
|
},
|
|
5485
5489
|
"pageCount": {
|
|
5486
5490
|
"type": "integer"
|
|
@@ -5517,7 +5521,7 @@
|
|
|
5517
5521
|
},
|
|
5518
5522
|
"runs": {
|
|
5519
5523
|
"type": "integer",
|
|
5520
|
-
"description": "
|
|
5524
|
+
"description": "Runs of the population in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns. A sharded run counts once; the final attempt is the one with the most retries, then from the lowest shard."
|
|
5521
5525
|
},
|
|
5522
5526
|
"stableRuns": {
|
|
5523
5527
|
"type": "integer",
|
|
@@ -5574,7 +5578,8 @@
|
|
|
5574
5578
|
"type": "object",
|
|
5575
5579
|
"properties": {
|
|
5576
5580
|
"runId": {
|
|
5577
|
-
"type": "string"
|
|
5581
|
+
"type": "string",
|
|
5582
|
+
"description": "The run, by the id GET /projects/{projectId}/runs gives it - one id for all shards of a sharded run (since 1.0.0-beta.6; it was the id of the shard the test ran in)."
|
|
5578
5583
|
},
|
|
5579
5584
|
"runNumber": {
|
|
5580
5585
|
"type": "integer"
|
|
@@ -5602,7 +5607,7 @@
|
|
|
5602
5607
|
"failed",
|
|
5603
5608
|
"skipped"
|
|
5604
5609
|
],
|
|
5605
|
-
"description": "The run's final attempt: stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate."
|
|
5610
|
+
"description": "The run's final attempt - across all its shards - read through test.fail(): stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate. A test the run's cancellation interrupted appears nowhere."
|
|
5606
5611
|
}
|
|
5607
5612
|
},
|
|
5608
5613
|
"required": [
|
|
@@ -5723,8 +5728,8 @@
|
|
|
5723
5728
|
},
|
|
5724
5729
|
"population": {
|
|
5725
5730
|
"type": "string",
|
|
5726
|
-
"const": "
|
|
5727
|
-
"description": "Which runs every number counts:
|
|
5731
|
+
"const": "ci_groups_finished_not_mass_failure",
|
|
5732
|
+
"description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
|
|
5728
5733
|
},
|
|
5729
5734
|
"days": {
|
|
5730
5735
|
"type": "integer",
|
|
@@ -5778,7 +5783,7 @@
|
|
|
5778
5783
|
},
|
|
5779
5784
|
"runs": {
|
|
5780
5785
|
"type": "integer",
|
|
5781
|
-
"description": "
|
|
5786
|
+
"description": "Runs of the population in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns. A sharded run counts once; the final attempt is the one with the most retries, then from the lowest shard."
|
|
5782
5787
|
},
|
|
5783
5788
|
"stableRuns": {
|
|
5784
5789
|
"type": "integer",
|
|
@@ -5835,7 +5840,8 @@
|
|
|
5835
5840
|
"type": "object",
|
|
5836
5841
|
"properties": {
|
|
5837
5842
|
"runId": {
|
|
5838
|
-
"type": "string"
|
|
5843
|
+
"type": "string",
|
|
5844
|
+
"description": "The run, by the id GET /projects/{projectId}/runs gives it - one id for all shards of a sharded run (since 1.0.0-beta.6; it was the id of the shard the test ran in)."
|
|
5839
5845
|
},
|
|
5840
5846
|
"runNumber": {
|
|
5841
5847
|
"type": "integer"
|
|
@@ -5863,7 +5869,7 @@
|
|
|
5863
5869
|
"failed",
|
|
5864
5870
|
"skipped"
|
|
5865
5871
|
],
|
|
5866
|
-
"description": "The run's final attempt: stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate."
|
|
5872
|
+
"description": "The run's final attempt - across all its shards - read through test.fail(): stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate. A test the run's cancellation interrupted appears nowhere."
|
|
5867
5873
|
}
|
|
5868
5874
|
},
|
|
5869
5875
|
"required": [
|
|
@@ -6019,11 +6025,20 @@
|
|
|
6019
6025
|
"type": "object",
|
|
6020
6026
|
"properties": {
|
|
6021
6027
|
"runId": {
|
|
6022
|
-
"type": "string"
|
|
6028
|
+
"type": "string",
|
|
6029
|
+
"description": "The run, by the id GET /projects/{projectId}/runs gives it - one occurrence and one id per run, however many shards it had (since 1.0.0-beta.6; it was one occurrence per shard)."
|
|
6023
6030
|
},
|
|
6024
6031
|
"runNumber": {
|
|
6025
6032
|
"type": "integer"
|
|
6026
6033
|
},
|
|
6034
|
+
"local": {
|
|
6035
|
+
"type": "boolean",
|
|
6036
|
+
"description": "No shard of the run reported a CI provider: a local run. Listed, never counted in a metric (since 1.0.0-beta.6)."
|
|
6037
|
+
},
|
|
6038
|
+
"counted": {
|
|
6039
|
+
"type": "boolean",
|
|
6040
|
+
"description": "Whether the run is in the metrics' population (see population on the metrics endpoints): a CI run whose shards all finished, not a mass failure, and the test not just skipped by a cancellation (since 1.0.0-beta.6)."
|
|
6041
|
+
},
|
|
6027
6042
|
"branch": {
|
|
6028
6043
|
"anyOf": [
|
|
6029
6044
|
{
|
|
@@ -6050,7 +6065,8 @@
|
|
|
6050
6065
|
"format": "date-time"
|
|
6051
6066
|
},
|
|
6052
6067
|
"status": {
|
|
6053
|
-
"type": "string"
|
|
6068
|
+
"type": "string",
|
|
6069
|
+
"description": "The final attempt's status across the run's shards, read through test.fail()."
|
|
6054
6070
|
},
|
|
6055
6071
|
"durationMs": {
|
|
6056
6072
|
"anyOf": [
|
|
@@ -6094,6 +6110,8 @@
|
|
|
6094
6110
|
"required": [
|
|
6095
6111
|
"runId",
|
|
6096
6112
|
"runNumber",
|
|
6113
|
+
"local",
|
|
6114
|
+
"counted",
|
|
6097
6115
|
"branch",
|
|
6098
6116
|
"testedRevision",
|
|
6099
6117
|
"startedAt",
|
|
@@ -6673,8 +6691,8 @@
|
|
|
6673
6691
|
"properties": {
|
|
6674
6692
|
"population": {
|
|
6675
6693
|
"type": "string",
|
|
6676
|
-
"const": "
|
|
6677
|
-
"description": "Which runs every number counts:
|
|
6694
|
+
"const": "ci_groups_finished_not_mass_failure",
|
|
6695
|
+
"description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
|
|
6678
6696
|
},
|
|
6679
6697
|
"days": {
|
|
6680
6698
|
"type": "integer",
|
package/dist/tools.js
CHANGED
|
@@ -54,13 +54,14 @@ export const TOOLS = [
|
|
|
54
54
|
operationId: "getTestMetrics",
|
|
55
55
|
description: "One test's numbers over a period: runs, stability (runs that passed on the first try " +
|
|
56
56
|
"/ runs), flakyRate (runs that passed only after a retry / runs), failureRate (failed " +
|
|
57
|
-
"runs / runs), time lost to retries and failures, its last 20 runs (
|
|
58
|
-
"included, as result \"skipped\" - shown,
|
|
59
|
-
"branches it ran on. Counts
|
|
60
|
-
"
|
|
61
|
-
"
|
|
62
|
-
"
|
|
63
|
-
"
|
|
57
|
+
"runs / runs), time lost to retries and failures, its last 20 runs (each with the run " +
|
|
58
|
+
"id squally-find-run gives; runs it skipped included, as result \"skipped\" - shown, " +
|
|
59
|
+
"never counted in runs or any rate) and the branches it ran on. Counts the CI runs of " +
|
|
60
|
+
"the period whose shards all finished - a sharded run counts once, by its first " +
|
|
61
|
+
"shard's start - without runs that were a mass failure as a whole; local runs never " +
|
|
62
|
+
"count. There is no verdict - the tool returns the numbers and you judge them. runs = 0 " +
|
|
63
|
+
"with null rates is a valid answer: no counted CI run of this test in the period. " +
|
|
64
|
+
"Cheap; for one test use this, not squally-list-tests.",
|
|
64
65
|
answers: "One test's stability, flaky rate and failure rate over a period.",
|
|
65
66
|
cost: "cheap",
|
|
66
67
|
},
|
|
@@ -68,7 +69,7 @@ export const TOOLS = [
|
|
|
68
69
|
name: "squally-list-tests",
|
|
69
70
|
title: "List tests",
|
|
70
71
|
operationId: "listTests",
|
|
71
|
-
description: "Every test with a
|
|
72
|
+
description: "Every test with a counted CI run in the period, with the same numbers per test as " +
|
|
72
73
|
"squally-get-test-metrics - runs, stability, flakyRate, failureRate, time lost - ranked " +
|
|
73
74
|
"worst first by the chosen sort (default: lowest stability). There is no verdict and no " +
|
|
74
75
|
"status - the tool labels no test; you judge the numbers. Moderate - two queries over " +
|