squally-mcp 0.8.1 → 0.8.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +321 -321
- package/dist/api.d.ts +5 -0
- package/dist/api.js +11 -1
- package/dist/instructions.js +9 -0
- package/dist/openapi/v1.json +39 -15
- package/dist/result.d.ts +8 -0
- package/dist/result.js +17 -0
- package/dist/tools.js +7 -3
- package/package.json +63 -63
package/README.md
CHANGED
|
@@ -1,321 +1,321 @@
|
|
|
1
|
-
# squally-mcp
|
|
2
|
-
|
|
3
|
-
A local [MCP](https://modelcontextprotocol.io) server that lets a coding agent
|
|
4
|
-
read your [Squally](https://app.squally.dev) data: which runs happened, how
|
|
5
|
-
stable each test is, and why a particular test failed.
|
|
6
|
-
|
|
7
|
-
It runs on your machine over stdio and talks to Squally's read API over HTTPS.
|
|
8
|
-
Point Claude Code, Codex, Cursor or Claude Desktop at it and ask "why is
|
|
9
|
-
`checkout.spec.ts` failing on main?".
|
|
10
|
-
|
|
11
|
-
## Read-only, and structurally so
|
|
12
|
-
|
|
13
|
-
The server issues **HTTP GETs and nothing else**. There is no code path that
|
|
14
|
-
writes, and every tool is annotated `readOnlyHint: true`, so a client can run
|
|
15
|
-
them without asking you to confirm each one.
|
|
16
|
-
|
|
17
|
-
It also cannot reach past your own organization: the key is scoped to one
|
|
18
|
-
organization, and scoping, plan checks and secret exclusion are all enforced
|
|
19
|
-
by the API, not by this client.
|
|
20
|
-
|
|
21
|
-
## What an agent can see
|
|
22
|
-
|
|
23
|
-
Worth knowing before you hand a key to an assistant. Through this server an
|
|
24
|
-
agent can read:
|
|
25
|
-
|
|
26
|
-
- your **projects** and their names;
|
|
27
|
-
- your **runs**, CI and local — commit SHA and message, branch, author, pull
|
|
28
|
-
request, job key, a link to the CI run, timing, pass/fail counts;
|
|
29
|
-
- **per-test results** for a run, including which shard ran what;
|
|
30
|
-
- for a failing attempt: the **error message and stack**, the **code snippet**
|
|
31
|
-
— **verbatim source code from your test file** — and the **full-context
|
|
32
|
-
prompt** (the text behind Copy the full context in Squally), which carries the same snippet plus an **ARIA snapshot of your
|
|
33
|
-
application at the moment of failure** (whatever was on screen, truncated to
|
|
34
|
-
3000 characters);
|
|
35
|
-
- **per-test metrics**: how often each test passed on the first try, passed
|
|
36
|
-
only after a retry, or failed, and how much time retries and failures cost;
|
|
37
|
-
- **error signatures**: failures grouped across runs and branches.
|
|
38
|
-
|
|
39
|
-
It cannot see screenshots, videos or traces — artifacts are deliberately out of
|
|
40
|
-
scope — and it cannot see any key, secret or webhook URL.
|
|
41
|
-
|
|
42
|
-
The key reaches **every project in the organization**. Create one per machine so
|
|
43
|
-
a single revocation does not lock out everything, and revoke it under
|
|
44
|
-
Organization › Read keys when the machine is retired.
|
|
45
|
-
|
|
46
|
-
## Setup
|
|
47
|
-
|
|
48
|
-
You need an **organization read key** (`sqly_ro_…`): Squally → Organization ›
|
|
49
|
-
Read keys → *Create read key*. It is shown once. It is not the same thing as the
|
|
50
|
-
project ingest key your Playwright reporter uses — this server refuses that one
|
|
51
|
-
by name.
|
|
52
|
-
|
|
53
|
-
Requires **Node 22 or newer**. Read keys are available from the **Standard**
|
|
54
|
-
plan.
|
|
55
|
-
|
|
56
|
-
### Claude Code
|
|
57
|
-
|
|
58
|
-
```bash
|
|
59
|
-
claude mcp add squally --scope user \
|
|
60
|
-
--env SQUALLY_API_KEY=sqly_ro_your_key_here \
|
|
61
|
-
-- npx -y squally-mcp
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
### Codex
|
|
65
|
-
|
|
66
|
-
`~/.codex/config.toml`:
|
|
67
|
-
|
|
68
|
-
```toml
|
|
69
|
-
[mcp_servers.squally]
|
|
70
|
-
command = "npx"
|
|
71
|
-
args = ["-y", "squally-mcp"]
|
|
72
|
-
env = { SQUALLY_API_KEY = "sqly_ro_your_key_here" }
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
### Cursor
|
|
76
|
-
|
|
77
|
-
`~/.cursor/mcp.json` (or `.cursor/mcp.json` in a project):
|
|
78
|
-
|
|
79
|
-
```json
|
|
80
|
-
{
|
|
81
|
-
"mcpServers": {
|
|
82
|
-
"squally": {
|
|
83
|
-
"command": "npx",
|
|
84
|
-
"args": ["-y", "squally-mcp"],
|
|
85
|
-
"env": { "SQUALLY_API_KEY": "sqly_ro_your_key_here" }
|
|
86
|
-
}
|
|
87
|
-
}
|
|
88
|
-
}
|
|
89
|
-
```
|
|
90
|
-
|
|
91
|
-
### Claude Desktop
|
|
92
|
-
|
|
93
|
-
`claude_desktop_config.json` — macOS:
|
|
94
|
-
`~/Library/Application Support/Claude/`, Windows: `%APPDATA%\Claude\` — then
|
|
95
|
-
restart the app:
|
|
96
|
-
|
|
97
|
-
```json
|
|
98
|
-
{
|
|
99
|
-
"mcpServers": {
|
|
100
|
-
"squally": {
|
|
101
|
-
"command": "npx",
|
|
102
|
-
"args": ["-y", "squally-mcp"],
|
|
103
|
-
"env": { "SQUALLY_API_KEY": "sqly_ro_your_key_here" }
|
|
104
|
-
}
|
|
105
|
-
}
|
|
106
|
-
}
|
|
107
|
-
```
|
|
108
|
-
|
|
109
|
-
## Environment
|
|
110
|
-
|
|
111
|
-
| Variable | Required | Meaning |
|
|
112
|
-
|---|---|---|
|
|
113
|
-
| `SQUALLY_API_KEY` | yes | Organization read key, `sqly_ro_…`. Missing or an ingest key by mistake, and the server exits with one line saying so. |
|
|
114
|
-
| `SQUALLY_API_URL` | no | Base URL, default `https://app.squally.dev`. A trailing `/` or `/api/v1` is tolerated. Useful against a local Squally: `http://localhost:3000`. |
|
|
115
|
-
| `SQUALLY_USE_SYSTEM_CA` | no | `0` keeps Node's bundled certificates only. |
|
|
116
|
-
|
|
117
|
-
squally-mcp trusts your operating system's certificate store, like your browser does; set SQUALLY_USE_SYSTEM_CA=0 to use Node's bundled certificates only.
|
|
118
|
-
|
|
119
|
-
## Troubleshooting
|
|
120
|
-
|
|
121
|
-
### Claude Desktop
|
|
122
|
-
|
|
123
|
-
- **The server does not start in time.** The first start through `npx`
|
|
124
|
-
downloads the package, which can take longer than Claude Desktop's 60-second
|
|
125
|
-
start-up limit. Install it once beforehand with `npm install -g squally-mcp`;
|
|
126
|
-
`npx -y squally-mcp` then runs that copy without a download. Update it with
|
|
127
|
-
the same command.
|
|
128
|
-
- **Where the log is.** The Microsoft Store build of Claude Desktop writes it to
|
|
129
|
-
`%LOCALAPPDATA%\Claude\logs\mcp-server-squally.log`. Its first lines say which
|
|
130
|
-
Node runs the server and whether it trusts the system's certificate store;
|
|
131
|
-
a request that never arrives is logged there with its cause.
|
|
132
|
-
- **Edit the config only while the app is closed.** Quit Claude Desktop
|
|
133
|
-
completely, including from the system tray, before changing
|
|
134
|
-
`claude_desktop_config.json`.
|
|
135
|
-
|
|
136
|
-
## The tools
|
|
137
|
-
|
|
138
|
-
| Tool | What it answers | Cost |
|
|
139
|
-
|---|---|---|
|
|
140
|
-
| `squally-list-projects` | Which projects exist. **Start here** — every other tool needs a `projectId` from it. | cheap |
|
|
141
|
-
| `squally-find-run` | Which runs happened — latest, or by branch, commit SHA, status or source. Counters only. | cheap |
|
|
142
|
-
| `squally-get-run` | One run with its per-test rows, across all shards. Which test is red. | cheap |
|
|
143
|
-
| `squally-debug-failure` | Every attempt of one test in one run: error, stack, full-context prompt. | cheap |
|
|
144
|
-
| `squally-get-test-metrics` | One test's stability, flaky rate and failure rate over a period. | cheap — one test |
|
|
145
|
-
| `squally-list-tests` | Every test with a run in the period, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
|
|
146
|
-
| `squally-list-errors` | Error signatures in a period: what keeps failing, grouped, and how its runs ended. | cheap |
|
|
147
|
-
|
|
148
|
-
For a single test use `squally-get-test-metrics`, not `squally-list-tests` —
|
|
149
|
-
the tool descriptions say so, and the server repeats it in its instructions.
|
|
150
|
-
|
|
151
|
-
### Runs
|
|
152
|
-
|
|
153
|
-
`squally-find-run` lists every run, CI and local alike; `source` (`all` by
|
|
154
|
-
default, `ci` or `local`) narrows it, and each run says whether it was
|
|
155
|
-
`local`. A run's `status` is the dashboard's word: `passed`, `failed`,
|
|
156
|
-
`cancelled`, `incomplete` (a shard timed out, was stopped by the monthly
|
|
157
|
-
result limit or never reported) or `running` - and `?status=` filters by
|
|
158
|
-
any of them. A cancelled run is `cancelled`; `cancellation` says how, when
|
|
159
|
-
and by whom. A sharded run is one row, its counters over all its shards with
|
|
160
|
-
every test once, and the tests a cancel cut off count nowhere.
|
|
161
|
-
|
|
162
|
-
Each run also says whether it is `settled` - final and counted; an
|
|
163
|
-
`incomplete` run that is not settled still waits for a shard within the run
|
|
164
|
-
timeout - and, from squally-reporter 0.11.0, which job of its CI run it is
|
|
165
|
-
(`jobKey`, from `SQUALLY_CI_JOB_KEY`; each job is its own run), which
|
|
166
|
-
`attempt`, which run it re-ran (`rerunOf`) and where the CI run is
|
|
167
|
-
(`ciRunUrl`; for GitHub Actions also for older reporters when the project
|
|
168
|
-
has one linked repository).
|
|
169
|
-
|
|
170
|
-
### What the numbers mean
|
|
171
|
-
|
|
172
|
-
The two test tools count **finished runs, CI and local,** in the period (7, 14, 30 or 90
|
|
173
|
-
days; 30 by default), on every branch unless you name one. A run finishes when
|
|
174
|
-
all its shards have - or, once the run timeout has passed, with the results its
|
|
175
|
-
shards reported, like a cancelled run; a shard stopped by the monthly result
|
|
176
|
-
limit counts like one that timed out. A sharded run counts once, placed in the period
|
|
177
|
-
by its first shard's start; a run that was a mass failure as a whole is left
|
|
178
|
-
out. `source` is `all` (the default since read API 1.0.0-beta.18; `ci`
|
|
179
|
-
before), `ci` or `local` - send `source=ci` for CI runs only - and the
|
|
180
|
-
response's `population` names which runs it counted. Per test:
|
|
181
|
-
|
|
182
|
-
- **stability** — runs that passed on the first try ÷ runs;
|
|
183
|
-
- **flakyRate** — runs that passed only after a retry ÷ runs;
|
|
184
|
-
- **failureRate** — failed runs ÷ runs.
|
|
185
|
-
|
|
186
|
-
Each is a fraction from 0 to 1, and `null` when the test has no run in the
|
|
187
|
-
period. **There is no verdict**: nothing calls a test flaky, broken or
|
|
188
|
-
healthy. The tools return the numbers; the agent — or you — judges them.
|
|
189
|
-
|
|
190
|
-
### Example: `squally-list-tests`
|
|
191
|
-
|
|
192
|
-
```json
|
|
193
|
-
{ "projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15", "perPage": 3 }
|
|
194
|
-
```
|
|
195
|
-
|
|
196
|
-
The three least stable tests of the last 30 days (abridged):
|
|
197
|
-
|
|
198
|
-
```json
|
|
199
|
-
{
|
|
200
|
-
"population": "ci_groups_finished_not_mass_failure",
|
|
201
|
-
"days": 30,
|
|
202
|
-
"sort": "stability",
|
|
203
|
-
"total": 409,
|
|
204
|
-
"items": [
|
|
205
|
-
{
|
|
206
|
-
"testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code",
|
|
207
|
-
"filePath": "tests/checkout.spec.ts",
|
|
208
|
-
"runs": 8,
|
|
209
|
-
"stability": 0.5,
|
|
210
|
-
"flakyRate": 0.5,
|
|
211
|
-
"failureRate": 0,
|
|
212
|
-
"timeLostMs": 7361,
|
|
213
|
-
"topBranch": { "branch": "main", "kind": "retries", "count": 3 }
|
|
214
|
-
}
|
|
215
|
-
]
|
|
216
|
-
}
|
|
217
|
-
```
|
|
218
|
-
|
|
219
|
-
### Example: `squally-get-test-metrics`
|
|
220
|
-
|
|
221
|
-
```json
|
|
222
|
-
{
|
|
223
|
-
"projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
|
|
224
|
-
"testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code"
|
|
225
|
-
}
|
|
226
|
-
```
|
|
227
|
-
|
|
228
|
-
The same numbers as that test's row, for one test (abridged):
|
|
229
|
-
|
|
230
|
-
```json
|
|
231
|
-
{
|
|
232
|
-
"days": 30,
|
|
233
|
-
"branch": null,
|
|
234
|
-
"runs": 8,
|
|
235
|
-
"stableRuns": 4,
|
|
236
|
-
"flakyRuns": 4,
|
|
237
|
-
"failedRuns": 0,
|
|
238
|
-
"stability": 0.5,
|
|
239
|
-
"flakyRate": 0.5,
|
|
240
|
-
"failureRate": 0,
|
|
241
|
-
"branches": ["feature/saved-carts", "main"]
|
|
242
|
-
}
|
|
243
|
-
```
|
|
244
|
-
|
|
245
|
-
Pass the name exactly as Squally shows it. When it exists in several spec
|
|
246
|
-
files, the tool answers with the `filePath` values to retry with; a test with
|
|
247
|
-
results but no counted run in the period answers `runs: 0` and `null`
|
|
248
|
-
rates — an answer, not an error.
|
|
249
|
-
|
|
250
|
-
## Development
|
|
251
|
-
|
|
252
|
-
```bash
|
|
253
|
-
npm install
|
|
254
|
-
npm run build # tsc, then copy the vendored OpenAPI document into dist/
|
|
255
|
-
npm test # builds, then runs the suite
|
|
256
|
-
npm run vendor:openapi # re-fetch the API document; review the diff, then re-pin
|
|
257
|
-
```
|
|
258
|
-
|
|
259
|
-
The tool schemas are **derived from Squally's published OpenAPI document**,
|
|
260
|
-
vendored at `src/openapi/v1.json` — an input schema is the operation's
|
|
261
|
-
parameters, an output schema is its 200 response. Nothing is transcribed by
|
|
262
|
-
hand, so the two cannot drift. `test/drift.test.js` compares the vendored copy
|
|
263
|
-
against the live document and fails when an operation or parameter has moved;
|
|
264
|
-
it skips loudly when offline, and while the live API is still an older version
|
|
265
|
-
than the vendored one — a release that has not been deployed yet.
|
|
266
|
-
|
|
267
|
-
## Releasing
|
|
268
|
-
|
|
269
|
-
Publishing is **staged**: CI uploads the tarball, a maintainer approves it, and
|
|
270
|
-
only then is the version installable. `npm stage publish` never asks for 2FA,
|
|
271
|
-
which is what makes it usable from a workflow; the 2FA prompt moves to the
|
|
272
|
-
approval step, so a compromised workflow can stage a version but cannot put one
|
|
273
|
-
in front of users. Direct `npm publish` is not permitted for this package's
|
|
274
|
-
trusted publisher, so this is the only route.
|
|
275
|
-
|
|
276
|
-
> **0.1.0 is the exception.** It was published by hand with `npm publish`
|
|
277
|
-
> before the trusted publisher existed, so it carries no provenance
|
|
278
|
-
> attestation. Every version from 0.1.1 on goes through the steps below.
|
|
279
|
-
|
|
280
|
-
1. **Bump and tag.** The lockfile belongs to the release commit, so let `npm
|
|
281
|
-
version` write both and commit them together:
|
|
282
|
-
|
|
283
|
-
```bash
|
|
284
|
-
npm install # only if package.json changed by hand
|
|
285
|
-
npm version patch # or minor / major - writes package.json + lockfile, makes the tag
|
|
286
|
-
git push --follow-tags
|
|
287
|
-
```
|
|
288
|
-
|
|
289
|
-
2. **The tag starts the workflow.** `.github/workflows/publish.yml` runs on
|
|
290
|
-
`v*`: it tests, checks that the tag and `package.json` agree, and runs
|
|
291
|
-
|
|
292
|
-
```bash
|
|
293
|
-
npm stage publish --provenance --access public
|
|
294
|
-
```
|
|
295
|
-
|
|
296
|
-
through npm's trusted publishing (OIDC) — no npm token in GitHub secrets.
|
|
297
|
-
`--provenance` attaches a signed statement linking the tarball to that
|
|
298
|
-
workflow run and commit.
|
|
299
|
-
|
|
300
|
-
3. **Approve the staged version.** On
|
|
301
|
-
[npmjs.com/package/squally-mcp](https://www.npmjs.com/package/squally-mcp) →
|
|
302
|
-
**Staged Packages** → **Approve**, which asks for 2FA. Or from a terminal:
|
|
303
|
-
|
|
304
|
-
```bash
|
|
305
|
-
npm stage list squally-mcp
|
|
306
|
-
npm stage view <stage-id> # what is in the tarball
|
|
307
|
-
npm stage approve <stage-id> # also asks for 2FA
|
|
308
|
-
npm stage reject <stage-id> # if something is wrong
|
|
309
|
-
```
|
|
310
|
-
|
|
311
|
-
Until this step the version exists in the registry but installs nothing.
|
|
312
|
-
|
|
313
|
-
**Requirements for step 3 on your machine:** npm **11.15.0 or later** and Node
|
|
314
|
-
**22.14.0 or higher** ([npm docs](https://docs.npmjs.com/staged-publishing/)).
|
|
315
|
-
Node 22 still bundles npm 10.9.9, so check `npm --version` rather than assuming
|
|
316
|
-
your Node version brought a new enough npm. The workflow uses Node 24 for the
|
|
317
|
-
same reason and fails with that requirement if a runner ever ships an older npm.
|
|
318
|
-
|
|
319
|
-
## License
|
|
320
|
-
|
|
321
|
-
MIT
|
|
1
|
+
# squally-mcp
|
|
2
|
+
|
|
3
|
+
A local [MCP](https://modelcontextprotocol.io) server that lets a coding agent
|
|
4
|
+
read your [Squally](https://app.squally.dev) data: which runs happened, how
|
|
5
|
+
stable each test is, and why a particular test failed.
|
|
6
|
+
|
|
7
|
+
It runs on your machine over stdio and talks to Squally's read API over HTTPS.
|
|
8
|
+
Point Claude Code, Codex, Cursor or Claude Desktop at it and ask "why is
|
|
9
|
+
`checkout.spec.ts` failing on main?".
|
|
10
|
+
|
|
11
|
+
## Read-only, and structurally so
|
|
12
|
+
|
|
13
|
+
The server issues **HTTP GETs and nothing else**. There is no code path that
|
|
14
|
+
writes, and every tool is annotated `readOnlyHint: true`, so a client can run
|
|
15
|
+
them without asking you to confirm each one.
|
|
16
|
+
|
|
17
|
+
It also cannot reach past your own organization: the key is scoped to one
|
|
18
|
+
organization, and scoping, plan checks and secret exclusion are all enforced
|
|
19
|
+
by the API, not by this client.
|
|
20
|
+
|
|
21
|
+
## What an agent can see
|
|
22
|
+
|
|
23
|
+
Worth knowing before you hand a key to an assistant. Through this server an
|
|
24
|
+
agent can read:
|
|
25
|
+
|
|
26
|
+
- your **projects** and their names;
|
|
27
|
+
- your **runs**, CI and local — commit SHA and message, branch, author, pull
|
|
28
|
+
request, job key, a link to the CI run, timing, pass/fail counts;
|
|
29
|
+
- **per-test results** for a run, including which shard ran what;
|
|
30
|
+
- for a failing attempt: the **error message and stack**, the **code snippet**
|
|
31
|
+
— **verbatim source code from your test file** — and the **full-context
|
|
32
|
+
prompt** (the text behind Copy the full context in Squally), which carries the same snippet plus an **ARIA snapshot of your
|
|
33
|
+
application at the moment of failure** (whatever was on screen, truncated to
|
|
34
|
+
3000 characters);
|
|
35
|
+
- **per-test metrics**: how often each test passed on the first try, passed
|
|
36
|
+
only after a retry, or failed, and how much time retries and failures cost;
|
|
37
|
+
- **error signatures**: failures grouped across runs and branches.
|
|
38
|
+
|
|
39
|
+
It cannot see screenshots, videos or traces — artifacts are deliberately out of
|
|
40
|
+
scope — and it cannot see any key, secret or webhook URL.
|
|
41
|
+
|
|
42
|
+
The key reaches **every project in the organization**. Create one per machine so
|
|
43
|
+
a single revocation does not lock out everything, and revoke it under
|
|
44
|
+
Organization › Read keys when the machine is retired.
|
|
45
|
+
|
|
46
|
+
## Setup
|
|
47
|
+
|
|
48
|
+
You need an **organization read key** (`sqly_ro_…`): Squally → Organization ›
|
|
49
|
+
Read keys → *Create read key*. It is shown once. It is not the same thing as the
|
|
50
|
+
project ingest key your Playwright reporter uses — this server refuses that one
|
|
51
|
+
by name.
|
|
52
|
+
|
|
53
|
+
Requires **Node 22 or newer**. Read keys are available from the **Standard**
|
|
54
|
+
plan.
|
|
55
|
+
|
|
56
|
+
### Claude Code
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
claude mcp add squally --scope user \
|
|
60
|
+
--env SQUALLY_API_KEY=sqly_ro_your_key_here \
|
|
61
|
+
-- npx -y squally-mcp
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
### Codex
|
|
65
|
+
|
|
66
|
+
`~/.codex/config.toml`:
|
|
67
|
+
|
|
68
|
+
```toml
|
|
69
|
+
[mcp_servers.squally]
|
|
70
|
+
command = "npx"
|
|
71
|
+
args = ["-y", "squally-mcp"]
|
|
72
|
+
env = { SQUALLY_API_KEY = "sqly_ro_your_key_here" }
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### Cursor
|
|
76
|
+
|
|
77
|
+
`~/.cursor/mcp.json` (or `.cursor/mcp.json` in a project):
|
|
78
|
+
|
|
79
|
+
```json
|
|
80
|
+
{
|
|
81
|
+
"mcpServers": {
|
|
82
|
+
"squally": {
|
|
83
|
+
"command": "npx",
|
|
84
|
+
"args": ["-y", "squally-mcp"],
|
|
85
|
+
"env": { "SQUALLY_API_KEY": "sqly_ro_your_key_here" }
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### Claude Desktop
|
|
92
|
+
|
|
93
|
+
`claude_desktop_config.json` — macOS:
|
|
94
|
+
`~/Library/Application Support/Claude/`, Windows: `%APPDATA%\Claude\` — then
|
|
95
|
+
restart the app:
|
|
96
|
+
|
|
97
|
+
```json
|
|
98
|
+
{
|
|
99
|
+
"mcpServers": {
|
|
100
|
+
"squally": {
|
|
101
|
+
"command": "npx",
|
|
102
|
+
"args": ["-y", "squally-mcp"],
|
|
103
|
+
"env": { "SQUALLY_API_KEY": "sqly_ro_your_key_here" }
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
## Environment
|
|
110
|
+
|
|
111
|
+
| Variable | Required | Meaning |
|
|
112
|
+
|---|---|---|
|
|
113
|
+
| `SQUALLY_API_KEY` | yes | Organization read key, `sqly_ro_…`. Missing or an ingest key by mistake, and the server exits with one line saying so. |
|
|
114
|
+
| `SQUALLY_API_URL` | no | Base URL, default `https://app.squally.dev`. A trailing `/` or `/api/v1` is tolerated. Useful against a local Squally: `http://localhost:3000`. |
|
|
115
|
+
| `SQUALLY_USE_SYSTEM_CA` | no | `0` keeps Node's bundled certificates only. |
|
|
116
|
+
|
|
117
|
+
squally-mcp trusts your operating system's certificate store, like your browser does; set SQUALLY_USE_SYSTEM_CA=0 to use Node's bundled certificates only.
|
|
118
|
+
|
|
119
|
+
## Troubleshooting
|
|
120
|
+
|
|
121
|
+
### Claude Desktop
|
|
122
|
+
|
|
123
|
+
- **The server does not start in time.** The first start through `npx`
|
|
124
|
+
downloads the package, which can take longer than Claude Desktop's 60-second
|
|
125
|
+
start-up limit. Install it once beforehand with `npm install -g squally-mcp`;
|
|
126
|
+
`npx -y squally-mcp` then runs that copy without a download. Update it with
|
|
127
|
+
the same command.
|
|
128
|
+
- **Where the log is.** The Microsoft Store build of Claude Desktop writes it to
|
|
129
|
+
`%LOCALAPPDATA%\Claude\logs\mcp-server-squally.log`. Its first lines say which
|
|
130
|
+
Node runs the server and whether it trusts the system's certificate store;
|
|
131
|
+
a request that never arrives is logged there with its cause.
|
|
132
|
+
- **Edit the config only while the app is closed.** Quit Claude Desktop
|
|
133
|
+
completely, including from the system tray, before changing
|
|
134
|
+
`claude_desktop_config.json`.
|
|
135
|
+
|
|
136
|
+
## The tools
|
|
137
|
+
|
|
138
|
+
| Tool | What it answers | Cost |
|
|
139
|
+
|---|---|---|
|
|
140
|
+
| `squally-list-projects` | Which projects exist. **Start here** — every other tool needs a `projectId` from it. | cheap |
|
|
141
|
+
| `squally-find-run` | Which runs happened — latest, or by branch, commit SHA, status or source. Counters only. | cheap |
|
|
142
|
+
| `squally-get-run` | One run with its per-test rows, across all shards. Which test is red. | cheap |
|
|
143
|
+
| `squally-debug-failure` | Every attempt of one test in one run: error, stack, full-context prompt. | cheap |
|
|
144
|
+
| `squally-get-test-metrics` | One test's stability, flaky rate and failure rate over a period. | cheap — one test |
|
|
145
|
+
| `squally-list-tests` | Every test with a run in the period, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
|
|
146
|
+
| `squally-list-errors` | Error signatures in a period: what keeps failing, grouped, and how its runs ended. | cheap |
|
|
147
|
+
|
|
148
|
+
For a single test use `squally-get-test-metrics`, not `squally-list-tests` —
|
|
149
|
+
the tool descriptions say so, and the server repeats it in its instructions.
|
|
150
|
+
|
|
151
|
+
### Runs
|
|
152
|
+
|
|
153
|
+
`squally-find-run` lists every run, CI and local alike; `source` (`all` by
|
|
154
|
+
default, `ci` or `local`) narrows it, and each run says whether it was
|
|
155
|
+
`local`. A run's `status` is the dashboard's word: `passed`, `failed`,
|
|
156
|
+
`cancelled`, `incomplete` (a shard timed out, was stopped by the monthly
|
|
157
|
+
result limit or never reported) or `running` - and `?status=` filters by
|
|
158
|
+
any of them. A cancelled run is `cancelled`; `cancellation` says how, when
|
|
159
|
+
and by whom. A sharded run is one row, its counters over all its shards with
|
|
160
|
+
every test once, and the tests a cancel cut off count nowhere.
|
|
161
|
+
|
|
162
|
+
Each run also says whether it is `settled` - final and counted; an
|
|
163
|
+
`incomplete` run that is not settled still waits for a shard within the run
|
|
164
|
+
timeout - and, from squally-reporter 0.11.0, which job of its CI run it is
|
|
165
|
+
(`jobKey`, from `SQUALLY_CI_JOB_KEY`; each job is its own run), which
|
|
166
|
+
`attempt`, which run it re-ran (`rerunOf`) and where the CI run is
|
|
167
|
+
(`ciRunUrl`; for GitHub Actions also for older reporters when the project
|
|
168
|
+
has one linked repository).
|
|
169
|
+
|
|
170
|
+
### What the numbers mean
|
|
171
|
+
|
|
172
|
+
The two test tools count **finished runs, CI and local,** in the period (7, 14, 30 or 90
|
|
173
|
+
days; 30 by default), on every branch unless you name one. A run finishes when
|
|
174
|
+
all its shards have - or, once the run timeout has passed, with the results its
|
|
175
|
+
shards reported, like a cancelled run; a shard stopped by the monthly result
|
|
176
|
+
limit counts like one that timed out. A sharded run counts once, placed in the period
|
|
177
|
+
by its first shard's start; a run that was a mass failure as a whole is left
|
|
178
|
+
out. `source` is `all` (the default since read API 1.0.0-beta.18; `ci`
|
|
179
|
+
before), `ci` or `local` - send `source=ci` for CI runs only - and the
|
|
180
|
+
response's `population` names which runs it counted. Per test:
|
|
181
|
+
|
|
182
|
+
- **stability** — runs that passed on the first try ÷ runs;
|
|
183
|
+
- **flakyRate** — runs that passed only after a retry ÷ runs;
|
|
184
|
+
- **failureRate** — failed runs ÷ runs.
|
|
185
|
+
|
|
186
|
+
Each is a fraction from 0 to 1, and `null` when the test has no run in the
|
|
187
|
+
period. **There is no verdict**: nothing calls a test flaky, broken or
|
|
188
|
+
healthy. The tools return the numbers; the agent — or you — judges them.
|
|
189
|
+
|
|
190
|
+
### Example: `squally-list-tests`
|
|
191
|
+
|
|
192
|
+
```json
|
|
193
|
+
{ "projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15", "perPage": 3 }
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
The three least stable tests of the last 30 days (abridged):
|
|
197
|
+
|
|
198
|
+
```json
|
|
199
|
+
{
|
|
200
|
+
"population": "ci_groups_finished_not_mass_failure",
|
|
201
|
+
"days": 30,
|
|
202
|
+
"sort": "stability",
|
|
203
|
+
"total": 409,
|
|
204
|
+
"items": [
|
|
205
|
+
{
|
|
206
|
+
"testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code",
|
|
207
|
+
"filePath": "tests/checkout.spec.ts",
|
|
208
|
+
"runs": 8,
|
|
209
|
+
"stability": 0.5,
|
|
210
|
+
"flakyRate": 0.5,
|
|
211
|
+
"failureRate": 0,
|
|
212
|
+
"timeLostMs": 7361,
|
|
213
|
+
"topBranch": { "branch": "main", "kind": "retries", "count": 3 }
|
|
214
|
+
}
|
|
215
|
+
]
|
|
216
|
+
}
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
### Example: `squally-get-test-metrics`
|
|
220
|
+
|
|
221
|
+
```json
|
|
222
|
+
{
|
|
223
|
+
"projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
|
|
224
|
+
"testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code"
|
|
225
|
+
}
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
The same numbers as that test's row, for one test (abridged):
|
|
229
|
+
|
|
230
|
+
```json
|
|
231
|
+
{
|
|
232
|
+
"days": 30,
|
|
233
|
+
"branch": null,
|
|
234
|
+
"runs": 8,
|
|
235
|
+
"stableRuns": 4,
|
|
236
|
+
"flakyRuns": 4,
|
|
237
|
+
"failedRuns": 0,
|
|
238
|
+
"stability": 0.5,
|
|
239
|
+
"flakyRate": 0.5,
|
|
240
|
+
"failureRate": 0,
|
|
241
|
+
"branches": ["feature/saved-carts", "main"]
|
|
242
|
+
}
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
Pass the name exactly as Squally shows it. When it exists in several spec
|
|
246
|
+
files, the tool answers with the `filePath` values to retry with; a test with
|
|
247
|
+
results but no counted run in the period answers `runs: 0` and `null`
|
|
248
|
+
rates — an answer, not an error.
|
|
249
|
+
|
|
250
|
+
## Development
|
|
251
|
+
|
|
252
|
+
```bash
|
|
253
|
+
npm install
|
|
254
|
+
npm run build # tsc, then copy the vendored OpenAPI document into dist/
|
|
255
|
+
npm test # builds, then runs the suite
|
|
256
|
+
npm run vendor:openapi # re-fetch the API document; review the diff, then re-pin
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
The tool schemas are **derived from Squally's published OpenAPI document**,
|
|
260
|
+
vendored at `src/openapi/v1.json` — an input schema is the operation's
|
|
261
|
+
parameters, an output schema is its 200 response. Nothing is transcribed by
|
|
262
|
+
hand, so the two cannot drift. `test/drift.test.js` compares the vendored copy
|
|
263
|
+
against the live document and fails when an operation or parameter has moved;
|
|
264
|
+
it skips loudly when offline, and while the live API is still an older version
|
|
265
|
+
than the vendored one — a release that has not been deployed yet.
|
|
266
|
+
|
|
267
|
+
## Releasing
|
|
268
|
+
|
|
269
|
+
Publishing is **staged**: CI uploads the tarball, a maintainer approves it, and
|
|
270
|
+
only then is the version installable. `npm stage publish` never asks for 2FA,
|
|
271
|
+
which is what makes it usable from a workflow; the 2FA prompt moves to the
|
|
272
|
+
approval step, so a compromised workflow can stage a version but cannot put one
|
|
273
|
+
in front of users. Direct `npm publish` is not permitted for this package's
|
|
274
|
+
trusted publisher, so this is the only route.
|
|
275
|
+
|
|
276
|
+
> **0.1.0 is the exception.** It was published by hand with `npm publish`
|
|
277
|
+
> before the trusted publisher existed, so it carries no provenance
|
|
278
|
+
> attestation. Every version from 0.1.1 on goes through the steps below.
|
|
279
|
+
|
|
280
|
+
1. **Bump and tag.** The lockfile belongs to the release commit, so let `npm
|
|
281
|
+
version` write both and commit them together:
|
|
282
|
+
|
|
283
|
+
```bash
|
|
284
|
+
npm install # only if package.json changed by hand
|
|
285
|
+
npm version patch # or minor / major - writes package.json + lockfile, makes the tag
|
|
286
|
+
git push --follow-tags
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
2. **The tag starts the workflow.** `.github/workflows/publish.yml` runs on
|
|
290
|
+
`v*`: it tests, checks that the tag and `package.json` agree, and runs
|
|
291
|
+
|
|
292
|
+
```bash
|
|
293
|
+
npm stage publish --provenance --access public
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
through npm's trusted publishing (OIDC) — no npm token in GitHub secrets.
|
|
297
|
+
`--provenance` attaches a signed statement linking the tarball to that
|
|
298
|
+
workflow run and commit.
|
|
299
|
+
|
|
300
|
+
3. **Approve the staged version.** On
|
|
301
|
+
[npmjs.com/package/squally-mcp](https://www.npmjs.com/package/squally-mcp) →
|
|
302
|
+
**Staged Packages** → **Approve**, which asks for 2FA. Or from a terminal:
|
|
303
|
+
|
|
304
|
+
```bash
|
|
305
|
+
npm stage list squally-mcp
|
|
306
|
+
npm stage view <stage-id> # what is in the tarball
|
|
307
|
+
npm stage approve <stage-id> # also asks for 2FA
|
|
308
|
+
npm stage reject <stage-id> # if something is wrong
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
Until this step the version exists in the registry but installs nothing.
|
|
312
|
+
|
|
313
|
+
**Requirements for step 3 on your machine:** npm **11.15.0 or later** and Node
|
|
314
|
+
**22.14.0 or higher** ([npm docs](https://docs.npmjs.com/staged-publishing/)).
|
|
315
|
+
Node 22 still bundles npm 10.9.9, so check `npm --version` rather than assuming
|
|
316
|
+
your Node version brought a new enough npm. The workflow uses Node 24 for the
|
|
317
|
+
same reason and fails with that requirement if a runner ever ships an older npm.
|
|
318
|
+
|
|
319
|
+
## License
|
|
320
|
+
|
|
321
|
+
MIT
|
package/dist/api.d.ts
CHANGED
|
@@ -16,6 +16,11 @@ export type ApiFailure = {
|
|
|
16
16
|
message: string | null;
|
|
17
17
|
/** The API's stable slug, when it sent a body. */
|
|
18
18
|
code: string | null;
|
|
19
|
+
/**
|
|
20
|
+
* `x-vercel-mitigated`, when Vercel's firewall in front of the API stopped
|
|
21
|
+
* the request itself (0.8.3): "challenge" or "deny", otherwise null.
|
|
22
|
+
*/
|
|
23
|
+
mitigated: "challenge" | "deny" | null;
|
|
19
24
|
};
|
|
20
25
|
/** The request never reached the API, or the answer was not JSON. */
|
|
21
26
|
export type ApiUnreachable = {
|
package/dist/api.js
CHANGED
|
@@ -1,6 +1,16 @@
|
|
|
1
1
|
import { errorCauses } from "./diagnostics.js";
|
|
2
2
|
/** Set by the API from 7 days before the key expires (OpenAPI components.headers). */
|
|
3
3
|
export const KEY_EXPIRY_HEADER = "Squally-Key-Expires-At";
|
|
4
|
+
/**
|
|
5
|
+
* Vercel's firewall answering instead of the API (0.8.3). Measured 30.10.2026:
|
|
6
|
+
* this server's Node fetch got 403 with `x-vercel-mitigated: challenge` and
|
|
7
|
+
* Vercel's own `{"error":{"code":"challenge"}}` while curl with the same key
|
|
8
|
+
* got 200 - a temporary mitigation for one machine, not a refusal of the key.
|
|
9
|
+
*/
|
|
10
|
+
function edgeMitigation(headers) {
|
|
11
|
+
const value = (headers.get("x-vercel-mitigated") ?? "").trim().toLowerCase();
|
|
12
|
+
return value === "challenge" || value === "deny" ? value : null;
|
|
13
|
+
}
|
|
4
14
|
/**
|
|
5
15
|
* Query parameters for which an EMPTY STRING is a value, not an absence.
|
|
6
16
|
*
|
|
@@ -113,5 +123,5 @@ export async function callOperation(config, operation, args, fetcher = fetch) {
|
|
|
113
123
|
catch {
|
|
114
124
|
// Left null; the caller has a sentence for every status either way.
|
|
115
125
|
}
|
|
116
|
-
return { kind: "failure", status: response.status, message, code };
|
|
126
|
+
return { kind: "failure", status: response.status, message, code, mitigated: edgeMitigation(response.headers) };
|
|
117
127
|
}
|
package/dist/instructions.js
CHANGED
|
@@ -48,6 +48,12 @@
|
|
|
48
48
|
// unsettled "incomplete" read as final, or two runs of one commit read as a
|
|
49
49
|
// duplicate, is the mistake.
|
|
50
50
|
//
|
|
51
|
+
// THE FINAL ATTEMPT (0.8.2, text only, squally-app #54): squally-get-run's
|
|
52
|
+
// rows are each test's final attempt, so a test that passed after a retry
|
|
53
|
+
// reads as passed - attempts > 1 is the only sign, and its error is not in the
|
|
54
|
+
// row. Said here because "which tests were flaky in this run" is the question
|
|
55
|
+
// the run page's Fix prompt sends an agent with.
|
|
56
|
+
//
|
|
51
57
|
// Exported as `squally-mcp/instructions` (package.json), so it stays a plain
|
|
52
58
|
// constant - test/exports.test.js imports it through the package name.
|
|
53
59
|
export const INSTRUCTIONS = [
|
|
@@ -65,5 +71,8 @@ export const INSTRUCTIONS = [
|
|
|
65
71
|
"error. A run's status is passed, failed, cancelled, incomplete or running;",
|
|
66
72
|
"a run that is not settled can still change. Each job of a CI run that sets a",
|
|
67
73
|
"job key is its own run (jobKey).",
|
|
74
|
+
"squally-get-run reports each test's final attempt: one that passed only after a",
|
|
75
|
+
"retry is passed with attempts above 1, and squally-debug-failure has the errors",
|
|
76
|
+
"of its failed attempts.",
|
|
68
77
|
"Errors carry a code and an action; follow the action.",
|
|
69
78
|
].join(" ");
|
package/dist/openapi/v1.json
CHANGED
|
@@ -56,7 +56,7 @@
|
|
|
56
56
|
{
|
|
57
57
|
"code": "invalid_period",
|
|
58
58
|
"status": 400,
|
|
59
|
-
"when": "days=90 is beyond this plan's 60-day retention. Use
|
|
59
|
+
"when": "days=90 is beyond this plan's 60-day retention. Use one of 7, 14, 30.",
|
|
60
60
|
"action": "Use days=7, 14, 30 or 90, within the plan's retention; do not retry.",
|
|
61
61
|
"hasBody": true
|
|
62
62
|
},
|
|
@@ -628,7 +628,7 @@
|
|
|
628
628
|
"invalid_period": {
|
|
629
629
|
"summary": "invalid_period",
|
|
630
630
|
"value": {
|
|
631
|
-
"error": "days=90 is beyond this plan's 60-day retention. Use
|
|
631
|
+
"error": "days=90 is beyond this plan's 60-day retention. Use one of 7, 14, 30.",
|
|
632
632
|
"code": "invalid_period"
|
|
633
633
|
}
|
|
634
634
|
}
|
|
@@ -2187,7 +2187,7 @@
|
|
|
2187
2187
|
"invalid_period": {
|
|
2188
2188
|
"summary": "invalid_period",
|
|
2189
2189
|
"value": {
|
|
2190
|
-
"error": "days=90 is beyond this plan's 60-day retention. Use
|
|
2190
|
+
"error": "days=90 is beyond this plan's 60-day retention. Use one of 7, 14, 30.",
|
|
2191
2191
|
"code": "invalid_period"
|
|
2192
2192
|
}
|
|
2193
2193
|
}
|
|
@@ -2613,7 +2613,7 @@
|
|
|
2613
2613
|
"invalid_period": {
|
|
2614
2614
|
"summary": "invalid_period",
|
|
2615
2615
|
"value": {
|
|
2616
|
-
"error": "days=90 is beyond this plan's 60-day retention. Use
|
|
2616
|
+
"error": "days=90 is beyond this plan's 60-day retention. Use one of 7, 14, 30.",
|
|
2617
2617
|
"code": "invalid_period"
|
|
2618
2618
|
}
|
|
2619
2619
|
}
|
|
@@ -2994,7 +2994,7 @@
|
|
|
2994
2994
|
"invalid_period": {
|
|
2995
2995
|
"summary": "invalid_period",
|
|
2996
2996
|
"value": {
|
|
2997
|
-
"error": "days=90 is beyond this plan's 60-day retention. Use
|
|
2997
|
+
"error": "days=90 is beyond this plan's 60-day retention. Use one of 7, 14, 30.",
|
|
2998
2998
|
"code": "invalid_period"
|
|
2999
2999
|
}
|
|
3000
3000
|
}
|
|
@@ -3322,6 +3322,8 @@
|
|
|
3322
3322
|
"runCount": 7,
|
|
3323
3323
|
"failedCount": 5,
|
|
3324
3324
|
"flakyCount": 2,
|
|
3325
|
+
"failedRunCount": 5,
|
|
3326
|
+
"flakyRunCount": 2,
|
|
3325
3327
|
"massFailureRunCount": 0,
|
|
3326
3328
|
"firstSeen": "2026-09-02T11:20:44.000Z",
|
|
3327
3329
|
"lastSeen": "2026-09-24T07:34:05.000Z"
|
|
@@ -3369,7 +3371,7 @@
|
|
|
3369
3371
|
"invalid_period": {
|
|
3370
3372
|
"summary": "invalid_period",
|
|
3371
3373
|
"value": {
|
|
3372
|
-
"error": "days=90 is beyond this plan's 60-day retention. Use
|
|
3374
|
+
"error": "days=90 is beyond this plan's 60-day retention. Use one of 7, 14, 30.",
|
|
3373
3375
|
"code": "invalid_period"
|
|
3374
3376
|
}
|
|
3375
3377
|
}
|
|
@@ -3666,6 +3668,8 @@
|
|
|
3666
3668
|
"runCount": 7,
|
|
3667
3669
|
"failedCount": 5,
|
|
3668
3670
|
"flakyCount": 2,
|
|
3671
|
+
"failedRunCount": 5,
|
|
3672
|
+
"flakyRunCount": 2,
|
|
3669
3673
|
"massFailureRunCount": 0,
|
|
3670
3674
|
"firstSeen": "2026-09-02T11:20:44.000Z",
|
|
3671
3675
|
"lastSeen": "2026-09-24T07:34:05.000Z"
|
|
@@ -3735,7 +3739,7 @@
|
|
|
3735
3739
|
"invalid_period": {
|
|
3736
3740
|
"summary": "invalid_period",
|
|
3737
3741
|
"value": {
|
|
3738
|
-
"error": "days=90 is beyond this plan's 60-day retention. Use
|
|
3742
|
+
"error": "days=90 is beyond this plan's 60-day retention. Use one of 7, 14, 30.",
|
|
3739
3743
|
"code": "invalid_period"
|
|
3740
3744
|
}
|
|
3741
3745
|
}
|
|
@@ -4129,7 +4133,7 @@
|
|
|
4129
4133
|
"invalid_period": {
|
|
4130
4134
|
"summary": "invalid_period",
|
|
4131
4135
|
"value": {
|
|
4132
|
-
"error": "days=90 is beyond this plan's 60-day retention. Use
|
|
4136
|
+
"error": "days=90 is beyond this plan's 60-day retention. Use one of 7, 14, 30.",
|
|
4133
4137
|
"code": "invalid_period"
|
|
4134
4138
|
}
|
|
4135
4139
|
}
|
|
@@ -4710,7 +4714,7 @@
|
|
|
4710
4714
|
"incomplete",
|
|
4711
4715
|
"running"
|
|
4712
4716
|
],
|
|
4713
|
-
"description": "How the run ended, the dashboard's own word: cancelled - the run was cancelled (for a sharded run, any shard recorded a cancel; cancellation says how and when); incomplete - a shard timed out, was stopped by the monthly result limit, or never reported by the time the run is overdue by the project's run timeout, counted from its latest
|
|
4717
|
+
"description": "How the run ended, the dashboard's own word: cancelled - the run was cancelled (for a sharded run, any shard recorded a cancel, or Squally marked the whole run cancelled; cancellation says how and when); incomplete - a shard timed out, was stopped by the monthly result limit, or never reported by the time the run is overdue by the project's run timeout, counted from its shards' last activity (a begin, a batch of results or a completion; their latest start until 2026-10-27); running - a shard is still running, or the run still waits for a shard within that time; failed - every shard finished and a test failed; passed - every shard finished and none failed. In that order: the first that applies. Since 1.0.0-beta.14 - before it, a cancelled run reported the outcome beneath the cancel and a cancelled boolean. Waiting for a shard is running since 2026-10-20 - before it, incomplete."
|
|
4714
4718
|
},
|
|
4715
4719
|
"cancellation": {
|
|
4716
4720
|
"anyOf": [
|
|
@@ -4727,7 +4731,7 @@
|
|
|
4727
4731
|
"at": {
|
|
4728
4732
|
"type": "string",
|
|
4729
4733
|
"format": "date-time",
|
|
4730
|
-
"description": "When the run was cancelled. For a sharded run, the first shard to record it."
|
|
4734
|
+
"description": "When the run was cancelled. For a sharded run, the first shard to record it - or, when the cancel marked the whole run before any shard recorded one (shards that had not begun, since 2026-10-22), when it was marked."
|
|
4731
4735
|
},
|
|
4732
4736
|
"byUserId": {
|
|
4733
4737
|
"description": "The user who cancelled the run, when source is \"user\"; null otherwise.",
|
|
@@ -4755,7 +4759,7 @@
|
|
|
4755
4759
|
},
|
|
4756
4760
|
"settled": {
|
|
4757
4761
|
"type": "boolean",
|
|
4758
|
-
"description": "Whether the run is settled: its status and counts are final and it counts in the dashboard's figures. false while a shard is running, and while the run still waits for a shard within the run time limit - once that passes, the hourly check settles it as incomplete. The dashboard's unfinished runs are the ones not settled. Since 1.0.0-beta.15."
|
|
4762
|
+
"description": "Whether the run is settled: its status and counts are final and it counts in the dashboard's figures. false while a shard is running, and while the run still waits for a shard within the run time limit, counted from its shards' last activity - once that passes, the hourly check settles it as incomplete. The dashboard's unfinished runs are the ones not settled. Since 1.0.0-beta.15."
|
|
4759
4763
|
},
|
|
4760
4764
|
"jobKey": {
|
|
4761
4765
|
"description": "The job key the reporter sent (SQUALLY_CI_JOB_KEY, squally-reporter 0.11.0): which job of a CI run this is, for example the browser of a matrix. Each job is its own run with its own GitHub check, Squally · <key>. null when the job set none. Since 1.0.0-beta.15.",
|
|
@@ -5033,7 +5037,7 @@
|
|
|
5033
5037
|
"incomplete",
|
|
5034
5038
|
"running"
|
|
5035
5039
|
],
|
|
5036
|
-
"description": "How the run ended, the dashboard's own word: cancelled - the run was cancelled (for a sharded run, any shard recorded a cancel; cancellation says how and when); incomplete - a shard timed out, was stopped by the monthly result limit, or never reported by the time the run is overdue by the project's run timeout, counted from its latest
|
|
5040
|
+
"description": "How the run ended, the dashboard's own word: cancelled - the run was cancelled (for a sharded run, any shard recorded a cancel, or Squally marked the whole run cancelled; cancellation says how and when); incomplete - a shard timed out, was stopped by the monthly result limit, or never reported by the time the run is overdue by the project's run timeout, counted from its shards' last activity (a begin, a batch of results or a completion; their latest start until 2026-10-27); running - a shard is still running, or the run still waits for a shard within that time; failed - every shard finished and a test failed; passed - every shard finished and none failed. In that order: the first that applies. Since 1.0.0-beta.14 - before it, a cancelled run reported the outcome beneath the cancel and a cancelled boolean. Waiting for a shard is running since 2026-10-20 - before it, incomplete."
|
|
5037
5041
|
},
|
|
5038
5042
|
"cancellation": {
|
|
5039
5043
|
"anyOf": [
|
|
@@ -5050,7 +5054,7 @@
|
|
|
5050
5054
|
"at": {
|
|
5051
5055
|
"type": "string",
|
|
5052
5056
|
"format": "date-time",
|
|
5053
|
-
"description": "When the run was cancelled. For a sharded run, the first shard to record it."
|
|
5057
|
+
"description": "When the run was cancelled. For a sharded run, the first shard to record it - or, when the cancel marked the whole run before any shard recorded one (shards that had not begun, since 2026-10-22), when it was marked."
|
|
5054
5058
|
},
|
|
5055
5059
|
"byUserId": {
|
|
5056
5060
|
"description": "The user who cancelled the run, when source is \"user\"; null otherwise.",
|
|
@@ -5078,7 +5082,7 @@
|
|
|
5078
5082
|
},
|
|
5079
5083
|
"settled": {
|
|
5080
5084
|
"type": "boolean",
|
|
5081
|
-
"description": "Whether the run is settled: its status and counts are final and it counts in the dashboard's figures. false while a shard is running, and while the run still waits for a shard within the run time limit - once that passes, the hourly check settles it as incomplete. The dashboard's unfinished runs are the ones not settled. Since 1.0.0-beta.15."
|
|
5085
|
+
"description": "Whether the run is settled: its status and counts are final and it counts in the dashboard's figures. false while a shard is running, and while the run still waits for a shard within the run time limit, counted from its shards' last activity - once that passes, the hourly check settles it as incomplete. The dashboard's unfinished runs are the ones not settled. Since 1.0.0-beta.15."
|
|
5082
5086
|
},
|
|
5083
5087
|
"jobKey": {
|
|
5084
5088
|
"description": "The job key the reporter sent (SQUALLY_CI_JOB_KEY, squally-reporter 0.11.0): which job of a CI run this is, for example the browser of a matrix. Each job is its own run with its own GitHub check, Squally · <key>. null when the job set none. Since 1.0.0-beta.15.",
|
|
@@ -5578,7 +5582,7 @@
|
|
|
5578
5582
|
}
|
|
5579
5583
|
},
|
|
5580
5584
|
"copyPrompt": {
|
|
5581
|
-
"description": "A prompt for an assistant that has the repository open, byte-identical to the text that Copy
|
|
5585
|
+
"description": "A prompt for an assistant that has the repository open, byte-identical to the text that Copy prompt copies under Fix in your agent, Without MCP, on the attempt page. null for an attempt without a real failure (a pass, a test.fail() marker, no error text).",
|
|
5582
5586
|
"type": [
|
|
5583
5587
|
"string",
|
|
5584
5588
|
"null"
|
|
@@ -6429,6 +6433,14 @@
|
|
|
6429
6433
|
"type": "integer",
|
|
6430
6434
|
"description": "Of the same pairs, how many ended flaky: the final attempt passed after a retry. A pair whose final attempt skipped is neither (since 1.0.0-beta.9)."
|
|
6431
6435
|
},
|
|
6436
|
+
"failedRunCount": {
|
|
6437
|
+
"type": "integer",
|
|
6438
|
+
"description": "Of runCount, the runs in which at least one test the error hit ended failed - each run counted once, however many of its tests failed (since 1.0.0-beta.18, additive)."
|
|
6439
|
+
},
|
|
6440
|
+
"flakyRunCount": {
|
|
6441
|
+
"type": "integer",
|
|
6442
|
+
"description": "Of runCount, the runs in which no test the error hit ended failed and at least one ended flaky - passed after a retry. failedRunCount + flakyRunCount <= runCount (since 1.0.0-beta.18, additive)."
|
|
6443
|
+
},
|
|
6432
6444
|
"massFailureRunCount": {
|
|
6433
6445
|
"type": "integer",
|
|
6434
6446
|
"description": "Of runCount, the runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed) - counted, and named here (since 1.0.0-beta.9)."
|
|
@@ -6456,6 +6468,8 @@
|
|
|
6456
6468
|
"runCount",
|
|
6457
6469
|
"failedCount",
|
|
6458
6470
|
"flakyCount",
|
|
6471
|
+
"failedRunCount",
|
|
6472
|
+
"flakyRunCount",
|
|
6459
6473
|
"massFailureRunCount",
|
|
6460
6474
|
"firstSeen",
|
|
6461
6475
|
"lastSeen"
|
|
@@ -6577,6 +6591,14 @@
|
|
|
6577
6591
|
"type": "integer",
|
|
6578
6592
|
"description": "Of the same pairs, how many ended flaky: the final attempt passed after a retry. A pair whose final attempt skipped is neither (since 1.0.0-beta.9)."
|
|
6579
6593
|
},
|
|
6594
|
+
"failedRunCount": {
|
|
6595
|
+
"type": "integer",
|
|
6596
|
+
"description": "Of runCount, the runs in which at least one test the error hit ended failed - each run counted once, however many of its tests failed (since 1.0.0-beta.18, additive)."
|
|
6597
|
+
},
|
|
6598
|
+
"flakyRunCount": {
|
|
6599
|
+
"type": "integer",
|
|
6600
|
+
"description": "Of runCount, the runs in which no test the error hit ended failed and at least one ended flaky - passed after a retry. failedRunCount + flakyRunCount <= runCount (since 1.0.0-beta.18, additive)."
|
|
6601
|
+
},
|
|
6580
6602
|
"massFailureRunCount": {
|
|
6581
6603
|
"type": "integer",
|
|
6582
6604
|
"description": "Of runCount, the runs that were a mass failure as a whole (at least 20 tests, over 80% of them failed) - counted, and named here (since 1.0.0-beta.9)."
|
|
@@ -6604,6 +6626,8 @@
|
|
|
6604
6626
|
"runCount",
|
|
6605
6627
|
"failedCount",
|
|
6606
6628
|
"flakyCount",
|
|
6629
|
+
"failedRunCount",
|
|
6630
|
+
"flakyRunCount",
|
|
6607
6631
|
"massFailureRunCount",
|
|
6608
6632
|
"firstSeen",
|
|
6609
6633
|
"lastSeen"
|
package/dist/result.d.ts
CHANGED
|
@@ -3,6 +3,14 @@ import type { ApiFailure, ApiResult, ApiUnreachable } from "./api.js";
|
|
|
3
3
|
import type { ErrorCode } from "./openapi.js";
|
|
4
4
|
/** The one message for a rate limit, which arrives without a body. */
|
|
5
5
|
export declare const RATE_LIMITED_TEXT = "Rate limited - wait 60 seconds, then retry.";
|
|
6
|
+
/**
|
|
7
|
+
* Vercel's firewall stopped the request (0.8.3): 403 with
|
|
8
|
+
* `x-vercel-mitigated`. Until 0.8.2 this read "The Squally API answered HTTP
|
|
9
|
+
* 403." and nothing else - which sends an agent and its user after the key.
|
|
10
|
+
* One retry after a wait, not a loop: retrying into the challenge is what
|
|
11
|
+
* keeps it going.
|
|
12
|
+
*/
|
|
13
|
+
export declare function edgeMitigatedText(mitigated: "challenge" | "deny"): string;
|
|
6
14
|
/**
|
|
7
15
|
* The line appended when the key is within 7 days of expiring.
|
|
8
16
|
*
|
package/dist/result.js
CHANGED
|
@@ -1,6 +1,18 @@
|
|
|
1
1
|
import { describeFailure } from "./diagnostics.js";
|
|
2
2
|
/** The one message for a rate limit, which arrives without a body. */
|
|
3
3
|
export const RATE_LIMITED_TEXT = "Rate limited - wait 60 seconds, then retry.";
|
|
4
|
+
/**
|
|
5
|
+
* Vercel's firewall stopped the request (0.8.3): 403 with
|
|
6
|
+
* `x-vercel-mitigated`. Until 0.8.2 this read "The Squally API answered HTTP
|
|
7
|
+
* 403." and nothing else - which sends an agent and its user after the key.
|
|
8
|
+
* One retry after a wait, not a loop: retrying into the challenge is what
|
|
9
|
+
* keeps it going.
|
|
10
|
+
*/
|
|
11
|
+
export function edgeMitigatedText(mitigated) {
|
|
12
|
+
return (`Vercel's firewall in front of Squally ${mitigated === "deny" ? "blocked" : "challenged"} this request ` +
|
|
13
|
+
`(HTTP 403, x-vercel-mitigated: ${mitigated}). This is not a key problem - the key was not checked. ` +
|
|
14
|
+
"Wait a few minutes, then retry once; do not retry in a loop. If it persists, tell the user.");
|
|
15
|
+
}
|
|
4
16
|
function text(value) {
|
|
5
17
|
return { type: "text", text: value };
|
|
6
18
|
}
|
|
@@ -55,6 +67,11 @@ export function failureResult(failure, errorCodes) {
|
|
|
55
67
|
if (failure.status === 429) {
|
|
56
68
|
return { content: [text(RATE_LIMITED_TEXT)], isError: true };
|
|
57
69
|
}
|
|
70
|
+
// Before the API's own 403s (organization_deleted, tier_too_low): those
|
|
71
|
+
// come without the header and keep their sentence and action.
|
|
72
|
+
if (failure.status === 403 && failure.mitigated !== null) {
|
|
73
|
+
return { content: [text(edgeMitigatedText(failure.mitigated))], isError: true };
|
|
74
|
+
}
|
|
58
75
|
const known = failure.code ? errorCodes.get(failure.code) : undefined;
|
|
59
76
|
const lines = [];
|
|
60
77
|
lines.push(failure.message ?? `The Squally API answered HTTP ${failure.status}.`);
|
package/dist/tools.js
CHANGED
|
@@ -42,7 +42,9 @@ export const TOOLS = [
|
|
|
42
42
|
"test is red in a run; the run list carries only the counters. Its status is cancelled " +
|
|
43
43
|
"for a cancelled run (cancellation says how and when), and local says it ran without CI. " +
|
|
44
44
|
"Like squally-find-run it carries settled, jobKey, attempt, rerunOf and ciRunUrl - for a " +
|
|
45
|
-
"re-run, rerunOf.id is the run it re-ran, to compare with."
|
|
45
|
+
"re-run, rerunOf.id is the run it re-ran, to compare with. Each test row is the test's " +
|
|
46
|
+
"final attempt: a test that passed only after a retry has status passed and attempts " +
|
|
47
|
+
"above 1, and the errors of its failed attempts come only from squally-debug-failure.",
|
|
46
48
|
answers: "One run with its per-test rows, across all shards. Which test is red.",
|
|
47
49
|
cost: "cheap",
|
|
48
50
|
},
|
|
@@ -98,8 +100,10 @@ export const TOOLS = [
|
|
|
98
100
|
"often and how recently each occurred. Answers 'is this failure one instance of something " +
|
|
99
101
|
"that keeps happening' - a grouping no agent can compute from the repository alone. " +
|
|
100
102
|
"Counts finished runs, CI and local, by default (source=ci or source=local for one of them), a sharded run " +
|
|
101
|
-
"once, mass-failure runs included
|
|
102
|
-
"
|
|
103
|
+
"once, mass-failure runs included. failedRunCount and flakyRunCount say how the runs " +
|
|
104
|
+
"ended: runs in which a test the error hit failed, and runs in which none failed and " +
|
|
105
|
+
"one passed only after a retry. failedCount and flakyCount count (run, test) pairs " +
|
|
106
|
+
"instead - a run in which the error failed three tests counts three times there.",
|
|
103
107
|
answers: "Error signatures in a period: what keeps failing, grouped, and how its runs ended.",
|
|
104
108
|
cost: "cheap",
|
|
105
109
|
},
|
package/package.json
CHANGED
|
@@ -1,63 +1,63 @@
|
|
|
1
|
-
{
|
|
2
|
-
"name": "squally-mcp",
|
|
3
|
-
"version": "0.8.
|
|
4
|
-
"description": "Local MCP server for Squally — read-only access to your Playwright test runs, flaky tests and failures.",
|
|
5
|
-
"license": "MIT",
|
|
6
|
-
"author": "Squally",
|
|
7
|
-
"homepage": "https://docs.squally.dev/agents/mcp-server/",
|
|
8
|
-
"repository": {
|
|
9
|
-
"type": "git",
|
|
10
|
-
"url": "git+https://github.com/squallydev-admin/squally-mcp.git"
|
|
11
|
-
},
|
|
12
|
-
"bugs": {
|
|
13
|
-
"url": "https://github.com/squallydev-admin/squally-mcp/issues"
|
|
14
|
-
},
|
|
15
|
-
"keywords": [
|
|
16
|
-
"mcp",
|
|
17
|
-
"modelcontextprotocol",
|
|
18
|
-
"playwright",
|
|
19
|
-
"flaky-tests",
|
|
20
|
-
"squally"
|
|
21
|
-
],
|
|
22
|
-
"type": "module",
|
|
23
|
-
"bin": {
|
|
24
|
-
"squally-mcp": "dist/index.js"
|
|
25
|
-
},
|
|
26
|
-
"main": "dist/index.js",
|
|
27
|
-
"types": "dist/index.d.ts",
|
|
28
|
-
"exports": {
|
|
29
|
-
".": {
|
|
30
|
-
"types": "./dist/index.d.ts",
|
|
31
|
-
"default": "./dist/index.js"
|
|
32
|
-
},
|
|
33
|
-
"./tools": {
|
|
34
|
-
"types": "./dist/tools.d.ts",
|
|
35
|
-
"default": "./dist/tools.js"
|
|
36
|
-
},
|
|
37
|
-
"./instructions": {
|
|
38
|
-
"types": "./dist/instructions.d.ts",
|
|
39
|
-
"default": "./dist/instructions.js"
|
|
40
|
-
},
|
|
41
|
-
"./package.json": "./package.json"
|
|
42
|
-
},
|
|
43
|
-
"files": [
|
|
44
|
-
"dist"
|
|
45
|
-
],
|
|
46
|
-
"engines": {
|
|
47
|
-
"node": ">=22"
|
|
48
|
-
},
|
|
49
|
-
"scripts": {
|
|
50
|
-
"build": "tsc && node scripts/copy-assets.mjs",
|
|
51
|
-
"vendor:openapi": "node scripts/vendor-openapi.mjs",
|
|
52
|
-
"test": "npm run build && node --test \"test/**/*.test.js\"",
|
|
53
|
-
"prepublishOnly": "npm run build"
|
|
54
|
-
},
|
|
55
|
-
"dependencies": {
|
|
56
|
-
"@modelcontextprotocol/sdk": "^1.32.2",
|
|
57
|
-
"zod": "^4.6.7"
|
|
58
|
-
},
|
|
59
|
-
"devDependencies": {
|
|
60
|
-
"@types/node": "^22.10.0",
|
|
61
|
-
"typescript": "^5.6.0"
|
|
62
|
-
}
|
|
63
|
-
}
|
|
1
|
+
{
|
|
2
|
+
"name": "squally-mcp",
|
|
3
|
+
"version": "0.8.3",
|
|
4
|
+
"description": "Local MCP server for Squally — read-only access to your Playwright test runs, flaky tests and failures.",
|
|
5
|
+
"license": "MIT",
|
|
6
|
+
"author": "Squally",
|
|
7
|
+
"homepage": "https://docs.squally.dev/agents/mcp-server/",
|
|
8
|
+
"repository": {
|
|
9
|
+
"type": "git",
|
|
10
|
+
"url": "git+https://github.com/squallydev-admin/squally-mcp.git"
|
|
11
|
+
},
|
|
12
|
+
"bugs": {
|
|
13
|
+
"url": "https://github.com/squallydev-admin/squally-mcp/issues"
|
|
14
|
+
},
|
|
15
|
+
"keywords": [
|
|
16
|
+
"mcp",
|
|
17
|
+
"modelcontextprotocol",
|
|
18
|
+
"playwright",
|
|
19
|
+
"flaky-tests",
|
|
20
|
+
"squally"
|
|
21
|
+
],
|
|
22
|
+
"type": "module",
|
|
23
|
+
"bin": {
|
|
24
|
+
"squally-mcp": "dist/index.js"
|
|
25
|
+
},
|
|
26
|
+
"main": "dist/index.js",
|
|
27
|
+
"types": "dist/index.d.ts",
|
|
28
|
+
"exports": {
|
|
29
|
+
".": {
|
|
30
|
+
"types": "./dist/index.d.ts",
|
|
31
|
+
"default": "./dist/index.js"
|
|
32
|
+
},
|
|
33
|
+
"./tools": {
|
|
34
|
+
"types": "./dist/tools.d.ts",
|
|
35
|
+
"default": "./dist/tools.js"
|
|
36
|
+
},
|
|
37
|
+
"./instructions": {
|
|
38
|
+
"types": "./dist/instructions.d.ts",
|
|
39
|
+
"default": "./dist/instructions.js"
|
|
40
|
+
},
|
|
41
|
+
"./package.json": "./package.json"
|
|
42
|
+
},
|
|
43
|
+
"files": [
|
|
44
|
+
"dist"
|
|
45
|
+
],
|
|
46
|
+
"engines": {
|
|
47
|
+
"node": ">=22"
|
|
48
|
+
},
|
|
49
|
+
"scripts": {
|
|
50
|
+
"build": "tsc && node scripts/copy-assets.mjs",
|
|
51
|
+
"vendor:openapi": "node scripts/vendor-openapi.mjs",
|
|
52
|
+
"test": "npm run build && node --test \"test/**/*.test.js\"",
|
|
53
|
+
"prepublishOnly": "npm run build"
|
|
54
|
+
},
|
|
55
|
+
"dependencies": {
|
|
56
|
+
"@modelcontextprotocol/sdk": "^1.32.2",
|
|
57
|
+
"zod": "^4.6.7"
|
|
58
|
+
},
|
|
59
|
+
"devDependencies": {
|
|
60
|
+
"@types/node": "^22.10.0",
|
|
61
|
+
"typescript": "^5.6.0"
|
|
62
|
+
}
|
|
63
|
+
}
|