squally-mcp 0.2.3 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -150,9 +150,11 @@ the tool descriptions say so, and the server repeats it in its instructions.
150
150
 
151
151
  ### What the numbers mean
152
152
 
153
- The two test tools count **completed CI runs** in the period (7, 14, 30 or 90
154
- days; 30 by default), on every branch unless you name one — without runs where
155
- most of the suite failed at once. Local runs never count. Per test:
153
+ The two test tools count **CI runs whose shards all finished** in the period
154
+ (7, 14, 30 or 90 days; 30 by default), on every branch unless you name one. A
155
+ sharded run counts once, placed in the period by its first shard's start; a
156
+ run that was a mass failure as a whole is left out. Local runs never count.
157
+ Per test:
156
158
 
157
159
  - **stability** — runs that passed on the first try ÷ runs;
158
160
  - **flakyRate** — runs that passed only after a retry ÷ runs;
@@ -172,7 +174,7 @@ The three least stable tests of the last 30 days (abridged):
172
174
 
173
175
  ```json
174
176
  {
175
- "population": "ci_completed_not_excluded",
177
+ "population": "ci_groups_finished_not_mass_failure",
176
178
  "days": 30,
177
179
  "sort": "stability",
178
180
  "total": 409,
@@ -219,7 +221,7 @@ The same numbers as that test's row, for one test (abridged):
219
221
 
220
222
  Pass the name exactly as Squally shows it. When it exists in several spec
221
223
  files, the tool answers with the `filePath` values to retry with; a test with
222
- results but no completed CI run in the period answers `runs: 0` and `null`
224
+ results but no counted CI run in the period answers `runs: 0` and `null`
223
225
  rates — an answer, not an error.
224
226
 
225
227
  ## Development
@@ -13,13 +13,18 @@
13
13
  // fractions of runs and nothing more; saying so here stops a model from
14
14
  // presenting a rate as Squally's judgement.
15
15
  //
16
+ // ONE RUN PER CI RUN SINCE 0.3.0 (API 1.0.0-beta.6): the population is the CI
17
+ // runs whose shards all finished, each counted once ("ci_groups_finished_not_
18
+ // mass_failure"); it was every completed shard on its own.
19
+ //
16
20
  // Exported as `squally-mcp/instructions` (package.json), so it stays a plain
17
21
  // constant - test/exports.test.js imports it through the package name.
18
22
  export const INSTRUCTIONS = [
19
23
  "Start with squally-list-projects; every other tool needs a projectId from it.",
20
24
  "For one test, use squally-get-test-metrics, not squally-list-tests.",
21
- "Test metrics count completed CI runs in the period, without runs where most of the",
22
- "suite failed at once; local runs never count. stability = runs that passed on the",
25
+ "Test metrics count the CI runs of the period whose shards all finished - a sharded",
26
+ "run counts once - without runs that were a mass failure as a whole; local runs never",
27
+ "count. stability = runs that passed on the",
23
28
  "first try / runs; flakyRate = runs that passed only after a retry / runs;",
24
29
  "failureRate = failed runs / runs. There is no verdict: the tools return numbers,",
25
30
  "and you judge them. If a test name is ambiguous, repeat with filePath from the",
@@ -3,7 +3,7 @@
3
3
  "jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
4
4
  "info": {
5
5
  "title": "Squally Read API",
6
- "version": "1.0.0-beta.5",
6
+ "version": "1.0.0-beta.6",
7
7
  "description": "Read access to a Squally organization's test results, for scripts, CI jobs and AI agents.\n\n**Authentication.** Every request carries an organization read key as `Authorization: Bearer sqly_ro_…`. A read key is organization-wide (every project), created by an organization admin in Organization › Read keys, and expires after 30, 60, 90, 180 days as chosen at creation. There is no renewal: create a new key and revoke the old one. Project ingest keys (`sqly_…`) are refused with `wrong_key_type`.\n\n**Expiry warning.** From 7 days before a key expires, every authenticated response carries `Squally-Key-Expires-At` with the expiry instant (ISO-8601).\n\n**Rate limit.** Currently 60 requests per 60 seconds per key, counted per `Authorization` header - a read key has its own budget, shared with nothing else. Over the limit the edge answers 429 without a JSON body.\n\n**Retries.** 429: wait 60 seconds, then retry. The edge counts requests per key in 60-second windows and sends no Retry-After header. 5xx: retry with exponential backoff, starting at 1 second and capped at 30 seconds. 400, 401, 403, 404: never retry - the same request gets the same answer.\n\n**Plans.** The read API is part of the Standard and Pro plans. A key of an organization on Free is answered 403 `tier_too_low`; keys are not revoked on a downgrade and work again after an upgrade.\n\n**Errors.** Every error is `{\"error\": \"<sentence>\", \"code\": \"<slug>\"}`. The sentence may be reworded; the slug may not. An id that does not exist and an id that belongs to another organization get the same 404, byte for byte.\n\n**Test names in paths.** Test names are Playwright title paths and often contain `/`: URL-encode them, including `/` as `%2F`.\n\n**Sharded runs** are returned as one logical run: /runs lists one row per logical run, and the run routes answer for all shards whichever shard's id you pass.\n\n**Stability.** v1, beta. The API may change until the beta ends; every change is listed in the changelog (docs/openapi/CHANGELOG.md)."
8
8
  },
9
9
  "servers": [
@@ -1797,7 +1797,7 @@
1797
1797
  "id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
1798
1798
  "name": "storefront-web"
1799
1799
  },
1800
- "population": "ci_completed_not_excluded",
1800
+ "population": "ci_groups_finished_not_mass_failure",
1801
1801
  "days": 30,
1802
1802
  "since": "2026-08-25T08:00:00.000Z",
1803
1803
  "until": "2026-09-24T08:00:00.000Z",
@@ -2362,7 +2362,7 @@
2362
2362
  "id": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
2363
2363
  "name": "storefront-web"
2364
2364
  },
2365
- "population": "ci_completed_not_excluded",
2365
+ "population": "ci_groups_finished_not_mass_failure",
2366
2366
  "days": 30,
2367
2367
  "since": "2026-08-25T08:00:00.000Z",
2368
2368
  "until": "2026-09-24T08:00:00.000Z",
@@ -2775,6 +2775,8 @@
2775
2775
  {
2776
2776
  "runId": "b1e0c6d2-7a43-4f19-8d5e-0c3a9e6f2b71",
2777
2777
  "runNumber": 1284,
2778
+ "local": false,
2779
+ "counted": true,
2778
2780
  "branch": "main",
2779
2781
  "testedRevision": null,
2780
2782
  "startedAt": "2026-09-24T07:31:12.000Z",
@@ -2788,6 +2790,8 @@
2788
2790
  {
2789
2791
  "runId": "7d2f4b19-3c8e-4a61-9e07-5b1d8c2a6f34",
2790
2792
  "runNumber": 1283,
2793
+ "local": true,
2794
+ "counted": false,
2791
2795
  "branch": "feature/saved-carts",
2792
2796
  "testedRevision": "c4f7a1e9b3d5c8e2f0a6b4d9e1c7f3a5b8d2e0f6",
2793
2797
  "startedAt": "2026-09-23T16:05:40.000Z",
@@ -3813,7 +3817,7 @@
3813
3817
  "testsRun": 412
3814
3818
  },
3815
3819
  "testMetrics": {
3816
- "population": "ci_completed_not_excluded",
3820
+ "population": "ci_groups_finished_not_mass_failure",
3817
3821
  "days": 30,
3818
3822
  "testsWithRetries": 7,
3819
3823
  "failingTests": 2,
@@ -5423,8 +5427,8 @@
5423
5427
  },
5424
5428
  "population": {
5425
5429
  "type": "string",
5426
- "const": "ci_completed_not_excluded",
5427
- "description": "Which runs every number counts: completed CI runs (reported with a CI provider) started inside the period, without the runs excluded as a mass failure (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count."
5430
+ "const": "ci_groups_finished_not_mass_failure",
5431
+ "description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
5428
5432
  },
5429
5433
  "days": {
5430
5434
  "type": "integer",
@@ -5480,7 +5484,7 @@
5480
5484
  },
5481
5485
  "total": {
5482
5486
  "type": "integer",
5483
- "description": "Tests matching the query across all pages: every test with at least one CI run in the period, narrowed by search, branch and browser."
5487
+ "description": "Tests matching the query across all pages: every test with at least one run of the population in the period, narrowed by search, branch and browser."
5484
5488
  },
5485
5489
  "pageCount": {
5486
5490
  "type": "integer"
@@ -5517,7 +5521,7 @@
5517
5521
  },
5518
5522
  "runs": {
5519
5523
  "type": "integer",
5520
- "description": "CI runs in the period in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns."
5524
+ "description": "Runs of the population in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns. A sharded run counts once; the final attempt is the one with the most retries, then from the lowest shard."
5521
5525
  },
5522
5526
  "stableRuns": {
5523
5527
  "type": "integer",
@@ -5574,7 +5578,8 @@
5574
5578
  "type": "object",
5575
5579
  "properties": {
5576
5580
  "runId": {
5577
- "type": "string"
5581
+ "type": "string",
5582
+ "description": "The run, by the id GET /projects/{projectId}/runs gives it - one id for all shards of a sharded run (since 1.0.0-beta.6; it was the id of the shard the test ran in)."
5578
5583
  },
5579
5584
  "runNumber": {
5580
5585
  "type": "integer"
@@ -5602,7 +5607,7 @@
5602
5607
  "failed",
5603
5608
  "skipped"
5604
5609
  ],
5605
- "description": "The run's final attempt: stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate."
5610
+ "description": "The run's final attempt - across all its shards - read through test.fail(): stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate. A test the run's cancellation interrupted appears nowhere."
5606
5611
  }
5607
5612
  },
5608
5613
  "required": [
@@ -5723,8 +5728,8 @@
5723
5728
  },
5724
5729
  "population": {
5725
5730
  "type": "string",
5726
- "const": "ci_completed_not_excluded",
5727
- "description": "Which runs every number counts: completed CI runs (reported with a CI provider) started inside the period, without the runs excluded as a mass failure (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count."
5731
+ "const": "ci_groups_finished_not_mass_failure",
5732
+ "description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
5728
5733
  },
5729
5734
  "days": {
5730
5735
  "type": "integer",
@@ -5778,7 +5783,7 @@
5778
5783
  },
5779
5784
  "runs": {
5780
5785
  "type": "integer",
5781
- "description": "CI runs in the period in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns."
5786
+ "description": "Runs of the population in which the test has a final attempt that did not skip: stableRuns + flakyRuns + failedRuns. A sharded run counts once; the final attempt is the one with the most retries, then from the lowest shard."
5782
5787
  },
5783
5788
  "stableRuns": {
5784
5789
  "type": "integer",
@@ -5835,7 +5840,8 @@
5835
5840
  "type": "object",
5836
5841
  "properties": {
5837
5842
  "runId": {
5838
- "type": "string"
5843
+ "type": "string",
5844
+ "description": "The run, by the id GET /projects/{projectId}/runs gives it - one id for all shards of a sharded run (since 1.0.0-beta.6; it was the id of the shard the test ran in)."
5839
5845
  },
5840
5846
  "runNumber": {
5841
5847
  "type": "integer"
@@ -5863,7 +5869,7 @@
5863
5869
  "failed",
5864
5870
  "skipped"
5865
5871
  ],
5866
- "description": "The run's final attempt: stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate."
5872
+ "description": "The run's final attempt - across all its shards - read through test.fail(): stable (passed on the first try), flaky (passed after at least one retry), failed, or skipped. A skipped run appears only here: it is not a run of the test, so it counts in neither runs nor any rate. A test the run's cancellation interrupted appears nowhere."
5867
5873
  }
5868
5874
  },
5869
5875
  "required": [
@@ -6019,11 +6025,20 @@
6019
6025
  "type": "object",
6020
6026
  "properties": {
6021
6027
  "runId": {
6022
- "type": "string"
6028
+ "type": "string",
6029
+ "description": "The run, by the id GET /projects/{projectId}/runs gives it - one occurrence and one id per run, however many shards it had (since 1.0.0-beta.6; it was one occurrence per shard)."
6023
6030
  },
6024
6031
  "runNumber": {
6025
6032
  "type": "integer"
6026
6033
  },
6034
+ "local": {
6035
+ "type": "boolean",
6036
+ "description": "No shard of the run reported a CI provider: a local run. Listed, never counted in a metric (since 1.0.0-beta.6)."
6037
+ },
6038
+ "counted": {
6039
+ "type": "boolean",
6040
+ "description": "Whether the run is in the metrics' population (see population on the metrics endpoints): a CI run whose shards all finished, not a mass failure, and the test not just skipped by a cancellation (since 1.0.0-beta.6)."
6041
+ },
6027
6042
  "branch": {
6028
6043
  "anyOf": [
6029
6044
  {
@@ -6050,7 +6065,8 @@
6050
6065
  "format": "date-time"
6051
6066
  },
6052
6067
  "status": {
6053
- "type": "string"
6068
+ "type": "string",
6069
+ "description": "The final attempt's status across the run's shards, read through test.fail()."
6054
6070
  },
6055
6071
  "durationMs": {
6056
6072
  "anyOf": [
@@ -6094,6 +6110,8 @@
6094
6110
  "required": [
6095
6111
  "runId",
6096
6112
  "runNumber",
6113
+ "local",
6114
+ "counted",
6097
6115
  "branch",
6098
6116
  "testedRevision",
6099
6117
  "startedAt",
@@ -6673,8 +6691,8 @@
6673
6691
  "properties": {
6674
6692
  "population": {
6675
6693
  "type": "string",
6676
- "const": "ci_completed_not_excluded",
6677
- "description": "Which runs every number counts: completed CI runs (reported with a CI provider) started inside the period, without the runs excluded as a mass failure (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count."
6694
+ "const": "ci_groups_finished_not_mass_failure",
6695
+ "description": "Which runs every number counts: CI runs (reported with a CI provider) whose shards all finished, each counted once however many shards it ran in, placed in the period by the start of their first shard - without runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed). Every branch unless the branch parameter names one. Local runs never count; runs that have not finished are not counted. Since 1.0.0-beta.6; it was ci_completed_not_excluded, which counted every completed shard on its own."
6678
6696
  },
6679
6697
  "days": {
6680
6698
  "type": "integer",
package/dist/tools.js CHANGED
@@ -54,13 +54,14 @@ export const TOOLS = [
54
54
  operationId: "getTestMetrics",
55
55
  description: "One test's numbers over a period: runs, stability (runs that passed on the first try " +
56
56
  "/ runs), flakyRate (runs that passed only after a retry / runs), failureRate (failed " +
57
- "runs / runs), time lost to retries and failures, its last 20 runs (runs it skipped " +
58
- "included, as result \"skipped\" - shown, never counted in runs or any rate) and the " +
59
- "branches it ran on. Counts completed CI runs of the period, without runs where most " +
60
- "of the suite failed at once; local runs never count. There is no verdict - the tool " +
61
- "returns the numbers and you judge them. runs = 0 with null rates is a valid answer: " +
62
- "no completed CI run of this test in the period. Cheap; for one test use this, not " +
63
- "squally-list-tests.",
57
+ "runs / runs), time lost to retries and failures, its last 20 runs (each with the run " +
58
+ "id squally-find-run gives; runs it skipped included, as result \"skipped\" - shown, " +
59
+ "never counted in runs or any rate) and the branches it ran on. Counts the CI runs of " +
60
+ "the period whose shards all finished - a sharded run counts once, by its first " +
61
+ "shard's start - without runs that were a mass failure as a whole; local runs never " +
62
+ "count. There is no verdict - the tool returns the numbers and you judge them. runs = 0 " +
63
+ "with null rates is a valid answer: no counted CI run of this test in the period. " +
64
+ "Cheap; for one test use this, not squally-list-tests.",
64
65
  answers: "One test's stability, flaky rate and failure rate over a period.",
65
66
  cost: "cheap",
66
67
  },
@@ -68,7 +69,7 @@ export const TOOLS = [
68
69
  name: "squally-list-tests",
69
70
  title: "List tests",
70
71
  operationId: "listTests",
71
- description: "Every test with a completed CI run in the period, with the same numbers per test as " +
72
+ description: "Every test with a counted CI run in the period, with the same numbers per test as " +
72
73
  "squally-get-test-metrics - runs, stability, flakyRate, failureRate, time lost - ranked " +
73
74
  "worst first by the chosen sort (default: lowest stability). There is no verdict and no " +
74
75
  "status - the tool labels no test; you judge the numbers. Moderate - two queries over " +
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "squally-mcp",
3
- "version": "0.2.3",
3
+ "version": "0.3.0",
4
4
  "description": "Local MCP server for Squally — read-only access to your Playwright test runs, flaky tests and failures.",
5
5
  "license": "MIT",
6
6
  "author": "Squally",