squally-mcp 0.2.2 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -8
- package/dist/instructions.js +7 -2
- package/dist/openapi/v1.json +81 -63
- package/dist/tools.js +9 -8
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -150,9 +150,11 @@ the tool descriptions say so, and the server repeats it in its instructions.
|
|
|
150
150
|
|
|
151
151
|
### What the numbers mean
|
|
152
152
|
|
|
153
|
-
The two test tools count **
|
|
154
|
-
days;
|
|
155
|
-
|
|
153
|
+
The two test tools count **CI runs whose shards all finished** in the period
|
|
154
|
+
(7, 14, 30 or 90 days; 30 by default), on every branch unless you name one. A
|
|
155
|
+
sharded run counts once, placed in the period by its first shard's start; a
|
|
156
|
+
run that was a mass failure as a whole is left out. Local runs never count.
|
|
157
|
+
Per test:
|
|
156
158
|
|
|
157
159
|
- **stability** — runs that passed on the first try ÷ runs;
|
|
158
160
|
- **flakyRate** — runs that passed only after a retry ÷ runs;
|
|
@@ -168,12 +170,12 @@ healthy. The tools return the numbers; the agent — or you — judges them.
|
|
|
168
170
|
{ "projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15", "perPage": 3 }
|
|
169
171
|
```
|
|
170
172
|
|
|
171
|
-
The three least stable tests of the last
|
|
173
|
+
The three least stable tests of the last 30 days (abridged):
|
|
172
174
|
|
|
173
175
|
```json
|
|
174
176
|
{
|
|
175
|
-
"population": "
|
|
176
|
-
"days":
|
|
177
|
+
"population": "ci_groups_finished_not_mass_failure",
|
|
178
|
+
"days": 30,
|
|
177
179
|
"sort": "stability",
|
|
178
180
|
"total": 409,
|
|
179
181
|
"items": [
|
|
@@ -204,7 +206,7 @@ The same numbers as that test's row, for one test (abridged):
|
|
|
204
206
|
|
|
205
207
|
```json
|
|
206
208
|
{
|
|
207
|
-
"days":
|
|
209
|
+
"days": 30,
|
|
208
210
|
"branch": null,
|
|
209
211
|
"runs": 8,
|
|
210
212
|
"stableRuns": 4,
|
|
@@ -219,7 +221,7 @@ The same numbers as that test's row, for one test (abridged):
|
|
|
219
221
|
|
|
220
222
|
Pass the name exactly as Squally shows it. When it exists in several spec
|
|
221
223
|
files, the tool answers with the `filePath` values to retry with; a test with
|
|
222
|
-
results but no
|
|
224
|
+
results but no counted CI run in the period answers `runs: 0` and `null`
|
|
223
225
|
rates — an answer, not an error.
|
|
224
226
|
|
|
225
227
|
## Development
|
package/dist/instructions.js
CHANGED
|
@@ -13,13 +13,18 @@
|
|
|
13
13
|
// fractions of runs and nothing more; saying so here stops a model from
|
|
14
14
|
// presenting a rate as Squally's judgement.
|
|
15
15
|
//
|
|
16
|
+
// ONE RUN PER CI RUN SINCE 0.3.0 (API 1.0.0-beta.6): the population is the CI
|
|
17
|
+
// runs whose shards all finished, each counted once ("ci_groups_finished_not_
|
|
18
|
+
// mass_failure"); it was every completed shard on its own.
|
|
19
|
+
//
|
|
16
20
|
// Exported as `squally-mcp/instructions` (package.json), so it stays a plain
|
|
17
21
|
// constant - test/exports.test.js imports it through the package name.
|
|
18
22
|
export const INSTRUCTIONS = [
|
|
19
23
|
"Start with squally-list-projects; every other tool needs a projectId from it.",
|
|
20
24
|
"For one test, use squally-get-test-metrics, not squally-list-tests.",
|
|
21
|
-
"Test metrics count
|
|
22
|
-
"
|
|
25
|
+
"Test metrics count the CI runs of the period whose shards all finished - a sharded",
|
|
26
|
+
"run counts once - without runs that were a mass failure as a whole; local runs never",
|
|
27
|
+
"count. stability = runs that passed on the",
|
|
23
28
|
"first try / runs; flakyRate = runs that passed only after a retry / runs;",
|
|
24
29
|
"failureRate = failed runs / runs. There is no verdict: the tools return numbers,",
|
|
25
30
|
"and you judge them. If a test name is ambiguous, repeat with filePath from the",
|
package/dist/openapi/v1.json
CHANGED
|
@@ -3,8 +3,8 @@
|
|
|
3
3
|
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
|
|
4
4
|
"info": {
|
|
5
5
|
"title": "Squally Read API",
|
|
6
|
-
"version": "1.0.0-beta.
|
|
7
|
-
"description": "Read access to a Squally organization's test results, for scripts, CI jobs and AI agents.\n\n**Authentication.** Every request carries an organization read key as `Authorization: Bearer sqly_ro_…`. A read key is organization-wide (every project), created by an organization admin in
|
|
6
|
+
"version": "1.0.0-beta.6",
|
|
7
|
+
"description": "Read access to a Squally organization's test results, for scripts, CI jobs and AI agents.\n\n**Authentication.** Every request carries an organization read key as `Authorization: Bearer sqly_ro_…`. A read key is organization-wide (every project), created by an organization admin in Organization › Read keys, and expires after 30, 60, 90, 180 days as chosen at creation. There is no renewal: create a new key and revoke the old one. Project ingest keys (`sqly_…`) are refused with `wrong_key_type`.\n\n**Expiry warning.** From 7 days before a key expires, every authenticated response carries `Squally-Key-Expires-At` with the expiry instant (ISO-8601).\n\n**Rate limit.** Currently 60 requests per 60 seconds per key, counted per `Authorization` header - a read key has its own budget, shared with nothing else. Over the limit the edge answers 429 without a JSON body.\n\n**Retries.** 429: wait 60 seconds, then retry. The edge counts requests per key in 60-second windows and sends no Retry-After header. 5xx: retry with exponential backoff, starting at 1 second and capped at 30 seconds. 400, 401, 403, 404: never retry - the same request gets the same answer.\n\n**Plans.** The read API is part of the Standard and Pro plans. A key of an organization on Free is answered 403 `tier_too_low`; keys are not revoked on a downgrade and work again after an upgrade.\n\n**Errors.** Every error is `{\"error\": \"<sentence>\", \"code\": \"<slug>\"}`. The sentence may be reworded; the slug may not. An id that does not exist and an id that belongs to another organization get the same 404, byte for byte.\n\n**Test names in paths.** Test names are Playwright title paths and often contain `/`: URL-encode them, including `/` as `%2F`.\n\n**Sharded runs** are returned as one logical run: /runs lists one row per logical run, and the run routes answer for all shards whichever shard's id you pass.\n\n**Stability.** v1, beta. The API may change until the beta ends; every change is listed in the changelog (docs/openapi/CHANGELOG.md)."
|
|
8
8
|
},
|
|
9
9
|
"servers": [
|
|
10
10
|
{
|
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
},
|
|
32
32
|
{
|
|
33
33
|
"name": "Errors",
|
|
34
|
-
"description": "
|
|
34
|
+
"description": "The project's errors: failures that share a fingerprint, one entry each."
|
|
35
35
|
},
|
|
36
36
|
{
|
|
37
37
|
"name": "Project",
|
|
@@ -64,21 +64,21 @@
|
|
|
64
64
|
"code": "invalid_key",
|
|
65
65
|
"status": 401,
|
|
66
66
|
"when": "Invalid read key.",
|
|
67
|
-
"action": "Check the key for typos or create a new one in
|
|
67
|
+
"action": "Check the key for typos or create a new one in Organization › Read keys; do not retry.",
|
|
68
68
|
"hasBody": true
|
|
69
69
|
},
|
|
70
70
|
{
|
|
71
71
|
"code": "key_expired",
|
|
72
72
|
"status": 401,
|
|
73
|
-
"when": "This read key expired on 2026-09-01. Create a new one in
|
|
74
|
-
"action": "Create a new read key in
|
|
73
|
+
"when": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
74
|
+
"action": "Create a new read key in Organization › Read keys; keys are not renewed.",
|
|
75
75
|
"hasBody": true
|
|
76
76
|
},
|
|
77
77
|
{
|
|
78
78
|
"code": "key_revoked",
|
|
79
79
|
"status": 401,
|
|
80
80
|
"when": "This read key has been revoked.",
|
|
81
|
-
"action": "Create a new read key in
|
|
81
|
+
"action": "Create a new read key in Organization › Read keys; this one will not work again.",
|
|
82
82
|
"hasBody": true
|
|
83
83
|
},
|
|
84
84
|
{
|
|
@@ -91,7 +91,7 @@
|
|
|
91
91
|
{
|
|
92
92
|
"code": "wrong_key_type",
|
|
93
93
|
"status": 401,
|
|
94
|
-
"when": "This is a project ingest key. Read routes need an organization read key -
|
|
94
|
+
"when": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
95
95
|
"action": "Use an organization read key (sqly_ro_…), not a project ingest key; do not retry.",
|
|
96
96
|
"hasBody": true
|
|
97
97
|
},
|
|
@@ -226,7 +226,7 @@
|
|
|
226
226
|
"key_expired": {
|
|
227
227
|
"summary": "key_expired",
|
|
228
228
|
"value": {
|
|
229
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
229
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
230
230
|
"code": "key_expired"
|
|
231
231
|
}
|
|
232
232
|
},
|
|
@@ -247,7 +247,7 @@
|
|
|
247
247
|
"wrong_key_type": {
|
|
248
248
|
"summary": "wrong_key_type",
|
|
249
249
|
"value": {
|
|
250
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
250
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
251
251
|
"code": "wrong_key_type"
|
|
252
252
|
}
|
|
253
253
|
}
|
|
@@ -630,7 +630,7 @@
|
|
|
630
630
|
"key_expired": {
|
|
631
631
|
"summary": "key_expired",
|
|
632
632
|
"value": {
|
|
633
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
633
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
634
634
|
"code": "key_expired"
|
|
635
635
|
}
|
|
636
636
|
},
|
|
@@ -651,7 +651,7 @@
|
|
|
651
651
|
"wrong_key_type": {
|
|
652
652
|
"summary": "wrong_key_type",
|
|
653
653
|
"value": {
|
|
654
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
654
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
655
655
|
"code": "wrong_key_type"
|
|
656
656
|
}
|
|
657
657
|
}
|
|
@@ -938,7 +938,7 @@
|
|
|
938
938
|
"key_expired": {
|
|
939
939
|
"summary": "key_expired",
|
|
940
940
|
"value": {
|
|
941
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
941
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
942
942
|
"code": "key_expired"
|
|
943
943
|
}
|
|
944
944
|
},
|
|
@@ -959,7 +959,7 @@
|
|
|
959
959
|
"wrong_key_type": {
|
|
960
960
|
"summary": "wrong_key_type",
|
|
961
961
|
"value": {
|
|
962
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
962
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
963
963
|
"code": "wrong_key_type"
|
|
964
964
|
}
|
|
965
965
|
}
|
|
@@ -1183,7 +1183,7 @@
|
|
|
1183
1183
|
}
|
|
1184
1184
|
}
|
|
1185
1185
|
],
|
|
1186
|
-
"copyPrompt": "You have the repository open. Find the test below, explain the failure and propose a fix.\n\nFailing test: chromium > checkout.spec.ts > Checkout > applies a coupon code\nFile: tests/checkout.spec.ts\nBrowser: chromium\nRun branch: main\nAttempt: first try\nOutcome: a later retry of this test passed - a flaky run.\n\nError message:\nError: expect(locator).toHaveText(expected) failed\n\nLocator: getByTestId('order-total')\nExpected: \"€ 45.00\"\nReceived: \"€ 50.00\"\nTimeout: 5000ms\nStack (top frames):\nat /home/runner/work/storefront-web/tests/checkout.spec.ts:48:52\nAttempt duration: 6210 ms\nFailure category: Assertion\nLast
|
|
1186
|
+
"copyPrompt": "You have the repository open. Find the test below, explain the failure and propose a fix.\n\nFailing test: chromium > checkout.spec.ts > Checkout > applies a coupon code\nFile: tests/checkout.spec.ts\nBrowser: chromium\nRun branch: main\nAttempt: first try\nOutcome: a later retry of this test passed - a flaky run.\n\nError message:\nError: expect(locator).toHaveText(expected) failed\n\nLocator: getByTestId('order-total')\nExpected: \"€ 45.00\"\nReceived: \"€ 50.00\"\nTimeout: 5000ms\nStack (top frames):\nat /home/runner/work/storefront-web/tests/checkout.spec.ts:48:52\nAttempt duration: 6210 ms\nFailure category: Assertion\nLast 30 days on CI: flaky rate 50% (4/8), failure rate 0% (0/8)",
|
|
1187
1187
|
"shard": {
|
|
1188
1188
|
"index": 1,
|
|
1189
1189
|
"runId": "b1e0c6d2-7a43-4f19-8d5e-0c3a9e6f2b71"
|
|
@@ -1284,7 +1284,7 @@
|
|
|
1284
1284
|
"key_expired": {
|
|
1285
1285
|
"summary": "key_expired",
|
|
1286
1286
|
"value": {
|
|
1287
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
1287
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
1288
1288
|
"code": "key_expired"
|
|
1289
1289
|
}
|
|
1290
1290
|
},
|
|
@@ -1305,7 +1305,7 @@
|
|
|
1305
1305
|
"wrong_key_type": {
|
|
1306
1306
|
"summary": "wrong_key_type",
|
|
1307
1307
|
"value": {
|
|
1308
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
1308
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
1309
1309
|
"code": "wrong_key_type"
|
|
1310
1310
|
}
|
|
1311
1311
|
}
|
|
@@ -1528,7 +1528,7 @@
|
|
|
1528
1528
|
"key_expired": {
|
|
1529
1529
|
"summary": "key_expired",
|
|
1530
1530
|
"value": {
|
|
1531
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
1531
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
1532
1532
|
"code": "key_expired"
|
|
1533
1533
|
}
|
|
1534
1534
|
},
|
|
@@ -1549,7 +1549,7 @@
|
|
|
1549
1549
|
"wrong_key_type": {
|
|
1550
1550
|
"summary": "wrong_key_type",
|
|
1551
1551
|
"value": {
|
|
1552
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
1552
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
1553
1553
|
"code": "wrong_key_type"
|
|
1554
1554
|
}
|
|
1555
1555
|
}
|
|
@@ -1702,7 +1702,7 @@
|
|
|
1702
1702
|
30,
|
|
1703
1703
|
90
|
|
1704
1704
|
],
|
|
1705
|
-
"default":
|
|
1705
|
+
"default": 30
|
|
1706
1706
|
}
|
|
1707
1707
|
},
|
|
1708
1708
|
{
|
|
@@ -1797,9 +1797,9 @@
|
|
|
1797
1797
|
"id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
|
|
1798
1798
|
"name": "storefront-web"
|
|
1799
1799
|
},
|
|
1800
|
-
"population": "
|
|
1801
|
-
"days":
|
|
1802
|
-
"since": "2026-
|
|
1800
|
+
"population": "ci_groups_finished_not_mass_failure",
|
|
1801
|
+
"days": 30,
|
|
1802
|
+
"since": "2026-08-25T08:00:00.000Z",
|
|
1803
1803
|
"until": "2026-09-24T08:00:00.000Z",
|
|
1804
1804
|
"branch": null,
|
|
1805
1805
|
"browser": null,
|
|
@@ -2137,7 +2137,7 @@
|
|
|
2137
2137
|
"key_expired": {
|
|
2138
2138
|
"summary": "key_expired",
|
|
2139
2139
|
"value": {
|
|
2140
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
2140
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
2141
2141
|
"code": "key_expired"
|
|
2142
2142
|
}
|
|
2143
2143
|
},
|
|
@@ -2158,7 +2158,7 @@
|
|
|
2158
2158
|
"wrong_key_type": {
|
|
2159
2159
|
"summary": "wrong_key_type",
|
|
2160
2160
|
"value": {
|
|
2161
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
2161
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
2162
2162
|
"code": "wrong_key_type"
|
|
2163
2163
|
}
|
|
2164
2164
|
}
|
|
@@ -2329,7 +2329,7 @@
|
|
|
2329
2329
|
30,
|
|
2330
2330
|
90
|
|
2331
2331
|
],
|
|
2332
|
-
"default":
|
|
2332
|
+
"default": 30
|
|
2333
2333
|
}
|
|
2334
2334
|
},
|
|
2335
2335
|
{
|
|
@@ -2362,9 +2362,9 @@
|
|
|
2362
2362
|
"id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
|
|
2363
2363
|
"name": "storefront-web"
|
|
2364
2364
|
},
|
|
2365
|
-
"population": "
|
|
2366
|
-
"days":
|
|
2367
|
-
"since": "2026-
|
|
2365
|
+
"population": "ci_groups_finished_not_mass_failure",
|
|
2366
|
+
"days": 30,
|
|
2367
|
+
"since": "2026-08-25T08:00:00.000Z",
|
|
2368
2368
|
"until": "2026-09-24T08:00:00.000Z",
|
|
2369
2369
|
"branch": null,
|
|
2370
2370
|
"testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code",
|
|
@@ -2540,7 +2540,7 @@
|
|
|
2540
2540
|
"key_expired": {
|
|
2541
2541
|
"summary": "key_expired",
|
|
2542
2542
|
"value": {
|
|
2543
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
2543
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
2544
2544
|
"code": "key_expired"
|
|
2545
2545
|
}
|
|
2546
2546
|
},
|
|
@@ -2561,7 +2561,7 @@
|
|
|
2561
2561
|
"wrong_key_type": {
|
|
2562
2562
|
"summary": "wrong_key_type",
|
|
2563
2563
|
"value": {
|
|
2564
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
2564
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
2565
2565
|
"code": "wrong_key_type"
|
|
2566
2566
|
}
|
|
2567
2567
|
}
|
|
@@ -2775,6 +2775,8 @@
|
|
|
2775
2775
|
{
|
|
2776
2776
|
"runId": "b1e0c6d2-7a43-4f19-8d5e-0c3a9e6f2b71",
|
|
2777
2777
|
"runNumber": 1284,
|
|
2778
|
+
"local": false,
|
|
2779
|
+
"counted": true,
|
|
2778
2780
|
"branch": "main",
|
|
2779
2781
|
"testedRevision": null,
|
|
2780
2782
|
"startedAt": "2026-09-24T07:31:12.000Z",
|
|
@@ -2788,6 +2790,8 @@
|
|
|
2788
2790
|
{
|
|
2789
2791
|
"runId": "7d2f4b19-3c8e-4a61-9e07-5b1d8c2a6f34",
|
|
2790
2792
|
"runNumber": 1283,
|
|
2793
|
+
"local": true,
|
|
2794
|
+
"counted": false,
|
|
2791
2795
|
"branch": "feature/saved-carts",
|
|
2792
2796
|
"testedRevision": "c4f7a1e9b3d5c8e2f0a6b4d9e1c7f3a5b8d2e0f6",
|
|
2793
2797
|
"startedAt": "2026-09-23T16:05:40.000Z",
|
|
@@ -2886,7 +2890,7 @@
|
|
|
2886
2890
|
"key_expired": {
|
|
2887
2891
|
"summary": "key_expired",
|
|
2888
2892
|
"value": {
|
|
2889
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
2893
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
2890
2894
|
"code": "key_expired"
|
|
2891
2895
|
}
|
|
2892
2896
|
},
|
|
@@ -2907,7 +2911,7 @@
|
|
|
2907
2911
|
"wrong_key_type": {
|
|
2908
2912
|
"summary": "wrong_key_type",
|
|
2909
2913
|
"value": {
|
|
2910
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
2914
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
2911
2915
|
"code": "wrong_key_type"
|
|
2912
2916
|
}
|
|
2913
2917
|
}
|
|
@@ -3040,8 +3044,8 @@
|
|
|
3040
3044
|
"/projects/{projectId}/errors": {
|
|
3041
3045
|
"get": {
|
|
3042
3046
|
"operationId": "listErrors",
|
|
3043
|
-
"summary": "List
|
|
3044
|
-
"description": "List
|
|
3047
|
+
"summary": "List errors",
|
|
3048
|
+
"description": "List errors. The project's errors in a period - failures grouped by fingerprint - most frequent first. Every count covers the period; firstSeen is the first occurrence still stored, not bounded by it. Moderate: one aggregate query.",
|
|
3045
3049
|
"tags": [
|
|
3046
3050
|
"Errors"
|
|
3047
3051
|
],
|
|
@@ -3234,7 +3238,7 @@
|
|
|
3234
3238
|
"key_expired": {
|
|
3235
3239
|
"summary": "key_expired",
|
|
3236
3240
|
"value": {
|
|
3237
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
3241
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
3238
3242
|
"code": "key_expired"
|
|
3239
3243
|
}
|
|
3240
3244
|
},
|
|
@@ -3255,7 +3259,7 @@
|
|
|
3255
3259
|
"wrong_key_type": {
|
|
3256
3260
|
"summary": "wrong_key_type",
|
|
3257
3261
|
"value": {
|
|
3258
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
3262
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
3259
3263
|
"code": "wrong_key_type"
|
|
3260
3264
|
}
|
|
3261
3265
|
}
|
|
@@ -3380,8 +3384,8 @@
|
|
|
3380
3384
|
"/projects/{projectId}/errors/{fingerprint}": {
|
|
3381
3385
|
"get": {
|
|
3382
3386
|
"operationId": "getError",
|
|
3383
|
-
"summary": "Get an error
|
|
3384
|
-
"description": "Get an error
|
|
3387
|
+
"summary": "Get an error",
|
|
3388
|
+
"description": "Get an error. One error: its counts, the tests it hits and the runs it occurred in. An error quiet in the period is answered over everything still stored (windowEmpty: true). Moderate.",
|
|
3385
3389
|
"tags": [
|
|
3386
3390
|
"Errors"
|
|
3387
3391
|
],
|
|
@@ -3399,7 +3403,7 @@
|
|
|
3399
3403
|
"name": "fingerprint",
|
|
3400
3404
|
"in": "path",
|
|
3401
3405
|
"required": true,
|
|
3402
|
-
"description": "An error
|
|
3406
|
+
"description": "An error's fingerprint, from GET /projects/{projectId}/errors.",
|
|
3403
3407
|
"schema": {
|
|
3404
3408
|
"type": "string",
|
|
3405
3409
|
"pattern": "^[0-9A-Za-z_-]{1,128}$"
|
|
@@ -3562,7 +3566,7 @@
|
|
|
3562
3566
|
"key_expired": {
|
|
3563
3567
|
"summary": "key_expired",
|
|
3564
3568
|
"value": {
|
|
3565
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
3569
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
3566
3570
|
"code": "key_expired"
|
|
3567
3571
|
}
|
|
3568
3572
|
},
|
|
@@ -3583,7 +3587,7 @@
|
|
|
3583
3587
|
"wrong_key_type": {
|
|
3584
3588
|
"summary": "wrong_key_type",
|
|
3585
3589
|
"value": {
|
|
3586
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
3590
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
3587
3591
|
"code": "wrong_key_type"
|
|
3588
3592
|
}
|
|
3589
3593
|
}
|
|
@@ -3813,7 +3817,7 @@
|
|
|
3813
3817
|
"testsRun": 412
|
|
3814
3818
|
},
|
|
3815
3819
|
"testMetrics": {
|
|
3816
|
-
"population": "
|
|
3820
|
+
"population": "ci_groups_finished_not_mass_failure",
|
|
3817
3821
|
"days": 30,
|
|
3818
3822
|
"testsWithRetries": 7,
|
|
3819
3823
|
"failingTests": 2,
|
|
@@ -3935,7 +3939,7 @@
|
|
|
3935
3939
|
"key_expired": {
|
|
3936
3940
|
"summary": "key_expired",
|
|
3937
3941
|
"value": {
|
|
3938
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
3942
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
3939
3943
|
"code": "key_expired"
|
|
3940
3944
|
}
|
|
3941
3945
|
},
|
|
@@ -3956,7 +3960,7 @@
|
|
|
3956
3960
|
"wrong_key_type": {
|
|
3957
3961
|
"summary": "wrong_key_type",
|
|
3958
3962
|
"value": {
|
|
3959
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
3963
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
3960
3964
|
"code": "wrong_key_type"
|
|
3961
3965
|
}
|
|
3962
3966
|
}
|
|
@@ -4171,7 +4175,7 @@
|
|
|
4171
4175
|
"key_expired": {
|
|
4172
4176
|
"summary": "key_expired",
|
|
4173
4177
|
"value": {
|
|
4174
|
-
"error": "This read key expired on 2026-09-01. Create a new one in
|
|
4178
|
+
"error": "This read key expired on 2026-09-01. Create a new one in Organization › Read keys.",
|
|
4175
4179
|
"code": "key_expired"
|
|
4176
4180
|
}
|
|
4177
4181
|
},
|
|
@@ -4192,7 +4196,7 @@
|
|
|
4192
4196
|
"wrong_key_type": {
|
|
4193
4197
|
"summary": "wrong_key_type",
|
|
4194
4198
|
"value": {
|
|
4195
|
-
"error": "This is a project ingest key. Read routes need an organization read key -
|
|
4199
|
+
"error": "This is a project ingest key. Read routes need an organization read key - Organization › Read keys.",
|
|
4196
4200
|
"code": "wrong_key_type"
|
|
4197
4201
|
}
|
|
4198
4202
|
}
|
|
@@ -4321,7 +4325,7 @@
|
|
|
4321
4325
|
"type": "http",
|
|
4322
4326
|
"scheme": "bearer",
|
|
4323
4327
|
"bearerFormat": "sqly_ro_…",
|
|
4324
|
-
"description": "An organization read key (sqly_ro_…), created by an organization admin in
|
|
4328
|
+
"description": "An organization read key (sqly_ro_…), created by an organization admin in Organization › Read keys. Expires after 30, 60, 90, 180 days; no renewal."
|
|
4325
4329
|
}
|
|
4326
4330
|
},
|
|
4327
4331
|
"headers": {
|
|
@@ -5423,8 +5427,8 @@
|
|
|
5423
5427
|
},
|
|
5424
5428
|
"population": {
|
|
5425
5429
|
"type": "string",
|
|
5426
|
-
"const": "
|
|
5427
|
-
"description": "Which runs every number counts:
|
|
5430
|
+
"const": "ci_groups_finished_not_mass_failure",
|
|
5431
|
+
"description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
|
|
5428
5432
|
},
|
|
5429
5433
|
"days": {
|
|
5430
5434
|
"type": "integer",
|
|
@@ -5480,7 +5484,7 @@
|
|
|
5480
5484
|
},
|
|
5481
5485
|
"total": {
|
|
5482
5486
|
"type": "integer",
|
|
5483
|
-
"description": "Tests matching the query across all pages: every test with at least one
|
|
5487
|
+
"description": "Tests matching the query across all pages: every test with at least one run of the population in the period, narrowed by search, branch and browser."
|
|
5484
5488
|
},
|
|
5485
5489
|
"pageCount": {
|
|
5486
5490
|
"type": "integer"
|
|
@@ -5517,7 +5521,7 @@
|
|
|
5517
5521
|
},
|
|
5518
5522
|
"runs": {
|
|
5519
5523
|
"type": "integer",
|
|
5520
|
-
"description": "
|
|
5524
|
+
"description": "Runs of the population in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns. A sharded run counts once; the final attempt is the one with the most retries, then from the lowest shard."
|
|
5521
5525
|
},
|
|
5522
5526
|
"stableRuns": {
|
|
5523
5527
|
"type": "integer",
|
|
@@ -5574,7 +5578,8 @@
|
|
|
5574
5578
|
"type": "object",
|
|
5575
5579
|
"properties": {
|
|
5576
5580
|
"runId": {
|
|
5577
|
-
"type": "string"
|
|
5581
|
+
"type": "string",
|
|
5582
|
+
"description": "The run, by the id GET /projects/{projectId}/runs gives it - one id for all shards of a sharded run (since 1.0.0-beta.6; it was the id of the shard the test ran in)."
|
|
5578
5583
|
},
|
|
5579
5584
|
"runNumber": {
|
|
5580
5585
|
"type": "integer"
|
|
@@ -5602,7 +5607,7 @@
|
|
|
5602
5607
|
"failed",
|
|
5603
5608
|
"skipped"
|
|
5604
5609
|
],
|
|
5605
|
-
"description": "The run's final attempt: stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate."
|
|
5610
|
+
"description": "The run's final attempt - across all its shards - read through test.fail(): stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate. A test the run's cancellation interrupted appears nowhere."
|
|
5606
5611
|
}
|
|
5607
5612
|
},
|
|
5608
5613
|
"required": [
|
|
@@ -5723,8 +5728,8 @@
|
|
|
5723
5728
|
},
|
|
5724
5729
|
"population": {
|
|
5725
5730
|
"type": "string",
|
|
5726
|
-
"const": "
|
|
5727
|
-
"description": "Which runs every number counts:
|
|
5731
|
+
"const": "ci_groups_finished_not_mass_failure",
|
|
5732
|
+
"description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
|
|
5728
5733
|
},
|
|
5729
5734
|
"days": {
|
|
5730
5735
|
"type": "integer",
|
|
@@ -5778,7 +5783,7 @@
|
|
|
5778
5783
|
},
|
|
5779
5784
|
"runs": {
|
|
5780
5785
|
"type": "integer",
|
|
5781
|
-
"description": "
|
|
5786
|
+
"description": "Runs of the population in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns. A sharded run counts once; the final attempt is the one with the most retries, then from the lowest shard."
|
|
5782
5787
|
},
|
|
5783
5788
|
"stableRuns": {
|
|
5784
5789
|
"type": "integer",
|
|
@@ -5835,7 +5840,8 @@
|
|
|
5835
5840
|
"type": "object",
|
|
5836
5841
|
"properties": {
|
|
5837
5842
|
"runId": {
|
|
5838
|
-
"type": "string"
|
|
5843
|
+
"type": "string",
|
|
5844
|
+
"description": "The run, by the id GET /projects/{projectId}/runs gives it - one id for all shards of a sharded run (since 1.0.0-beta.6; it was the id of the shard the test ran in)."
|
|
5839
5845
|
},
|
|
5840
5846
|
"runNumber": {
|
|
5841
5847
|
"type": "integer"
|
|
@@ -5863,7 +5869,7 @@
|
|
|
5863
5869
|
"failed",
|
|
5864
5870
|
"skipped"
|
|
5865
5871
|
],
|
|
5866
|
-
"description": "The run's final attempt: stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate."
|
|
5872
|
+
"description": "The run's final attempt - across all its shards - read through test.fail(): stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate. A test the run's cancellation interrupted appears nowhere."
|
|
5867
5873
|
}
|
|
5868
5874
|
},
|
|
5869
5875
|
"required": [
|
|
@@ -6019,11 +6025,20 @@
|
|
|
6019
6025
|
"type": "object",
|
|
6020
6026
|
"properties": {
|
|
6021
6027
|
"runId": {
|
|
6022
|
-
"type": "string"
|
|
6028
|
+
"type": "string",
|
|
6029
|
+
"description": "The run, by the id GET /projects/{projectId}/runs gives it - one occurrence and one id per run, however many shards it had (since 1.0.0-beta.6; it was one occurrence per shard)."
|
|
6023
6030
|
},
|
|
6024
6031
|
"runNumber": {
|
|
6025
6032
|
"type": "integer"
|
|
6026
6033
|
},
|
|
6034
|
+
"local": {
|
|
6035
|
+
"type": "boolean",
|
|
6036
|
+
"description": "No shard of the run reported a CI provider: a local run. Listed, never counted in a metric (since 1.0.0-beta.6)."
|
|
6037
|
+
},
|
|
6038
|
+
"counted": {
|
|
6039
|
+
"type": "boolean",
|
|
6040
|
+
"description": "Whether the run is in the metrics' population (see population on the metrics endpoints): a CI run whose shards all finished, not a mass failure, and the test not just skipped by a cancellation (since 1.0.0-beta.6)."
|
|
6041
|
+
},
|
|
6027
6042
|
"branch": {
|
|
6028
6043
|
"anyOf": [
|
|
6029
6044
|
{
|
|
@@ -6050,7 +6065,8 @@
|
|
|
6050
6065
|
"format": "date-time"
|
|
6051
6066
|
},
|
|
6052
6067
|
"status": {
|
|
6053
|
-
"type": "string"
|
|
6068
|
+
"type": "string",
|
|
6069
|
+
"description": "The final attempt's status across the run's shards, read through test.fail()."
|
|
6054
6070
|
},
|
|
6055
6071
|
"durationMs": {
|
|
6056
6072
|
"anyOf": [
|
|
@@ -6094,6 +6110,8 @@
|
|
|
6094
6110
|
"required": [
|
|
6095
6111
|
"runId",
|
|
6096
6112
|
"runNumber",
|
|
6113
|
+
"local",
|
|
6114
|
+
"counted",
|
|
6097
6115
|
"branch",
|
|
6098
6116
|
"testedRevision",
|
|
6099
6117
|
"startedAt",
|
|
@@ -6673,8 +6691,8 @@
|
|
|
6673
6691
|
"properties": {
|
|
6674
6692
|
"population": {
|
|
6675
6693
|
"type": "string",
|
|
6676
|
-
"const": "
|
|
6677
|
-
"description": "Which runs every number counts:
|
|
6694
|
+
"const": "ci_groups_finished_not_mass_failure",
|
|
6695
|
+
"description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
|
|
6678
6696
|
},
|
|
6679
6697
|
"days": {
|
|
6680
6698
|
"type": "integer",
|
package/dist/tools.js
CHANGED
|
@@ -54,13 +54,14 @@ export const TOOLS = [
|
|
|
54
54
|
operationId: "getTestMetrics",
|
|
55
55
|
description: "One test's numbers over a period: runs, stability (runs that passed on the first try " +
|
|
56
56
|
"/ runs), flakyRate (runs that passed only after a retry / runs), failureRate (failed " +
|
|
57
|
-
"runs / runs), time lost to retries and failures, its last 20 runs (
|
|
58
|
-
"included, as result \"skipped\" - shown,
|
|
59
|
-
"branches it ran on. Counts
|
|
60
|
-
"
|
|
61
|
-
"
|
|
62
|
-
"
|
|
63
|
-
"
|
|
57
|
+
"runs / runs), time lost to retries and failures, its last 20 runs (each with the run " +
|
|
58
|
+
"id squally-find-run gives; runs it skipped included, as result \"skipped\" - shown, " +
|
|
59
|
+
"never counted in runs or any rate) and the branches it ran on. Counts the CI runs of " +
|
|
60
|
+
"the period whose shards all finished - a sharded run counts once, by its first " +
|
|
61
|
+
"shard's start - without runs that were a mass failure as a whole; local runs never " +
|
|
62
|
+
"count. There is no verdict - the tool returns the numbers and you judge them. runs = 0 " +
|
|
63
|
+
"with null rates is a valid answer: no counted CI run of this test in the period. " +
|
|
64
|
+
"Cheap; for one test use this, not squally-list-tests.",
|
|
64
65
|
answers: "One test's stability, flaky rate and failure rate over a period.",
|
|
65
66
|
cost: "cheap",
|
|
66
67
|
},
|
|
@@ -68,7 +69,7 @@ export const TOOLS = [
|
|
|
68
69
|
name: "squally-list-tests",
|
|
69
70
|
title: "List tests",
|
|
70
71
|
operationId: "listTests",
|
|
71
|
-
description: "Every test with a
|
|
72
|
+
description: "Every test with a counted CI run in the period, with the same numbers per test as " +
|
|
72
73
|
"squally-get-test-metrics - runs, stability, flakyRate, failureRate, time lost - ranked " +
|
|
73
74
|
"worst first by the chosen sort (default: lowest stability). There is no verdict and no " +
|
|
74
75
|
"status - the tool labels no test; you judge the numbers. Moderate - two queries over " +
|