squally-mcp 0.1.3 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,8 +1,8 @@
1
1
  # squally-mcp
2
2
 
3
3
  A local [MCP](https://modelcontextprotocol.io) server that lets a coding agent
4
- read your [Squally](https://app.squally.dev) data: which CI runs happened, which
5
- tests are flaky, and why a particular test failed.
4
+ read your [Squally](https://app.squally.dev) data: which CI runs happened, how
5
+ stable each test is, and why a particular test failed.
6
6
 
7
7
  It runs on your machine over stdio and talks to Squally's read API over HTTPS.
8
8
  Point Claude Code, Codex, Cursor or Claude Desktop at it and ask "why is
@@ -32,7 +32,8 @@ agent can read:
32
32
  prompt**, which carries the same snippet plus an **ARIA snapshot of your
33
33
  application at the moment of failure** (whatever was on screen, truncated to
34
34
  3000 characters);
35
- - **flakiness verdicts** and how much time each flaky test has cost;
35
+ - **per-test metrics**: how often each test passed on the first try, passed
36
+ only after a retry, or failed, and how much time retries and failures cost;
36
37
  - **error signatures**: failures grouped across runs and branches.
37
38
 
38
39
  It cannot see screenshots, videos or traces — artifacts are deliberately out of
@@ -140,14 +141,86 @@ squally-mcp trusts your operating system's certificate store, like your browser
140
141
  | `squally-find-run` | Which runs happened — latest, or by branch, commit SHA or status. Counters only. | cheap |
141
142
  | `squally-get-run` | One run with its per-test rows, across all shards. Which test is red. | cheap |
142
143
  | `squally-debug-failure` | Every attempt of one test in one run: error, stack, Copy-for-AI prompt. | cheap |
143
- | `squally-get-test-status` | The stored flakiness status of **one** test. | cheap — one lookup |
144
- | `squally-list-flaky-tests` | The ranked flaky/broken list with time lost. | **expensive — one engine pass** |
144
+ | `squally-get-test-metrics` | One test's stability, flaky rate and failure rate over a period. | cheap — one test |
145
+ | `squally-list-tests` | Every test with a CI run in the period, ranked by stability, flaky rate or failures. | cheap — two queries, paged |
145
146
  | `squally-list-errors` | Error signatures in a period: what keeps failing, grouped. | cheap |
146
147
 
147
- For a single test use `squally-get-test-status`, not
148
- `squally-list-flaky-tests` — the tool descriptions say so, and the server
149
- repeats it in its instructions, because the difference is one database lookup
150
- against a pass over the project's recent runs.
148
+ For a single test use `squally-get-test-metrics`, not `squally-list-tests` —
149
+ the tool descriptions say so, and the server repeats it in its instructions.
150
+
151
+ ### What the numbers mean
152
+
153
+ The two test tools count **completed CI runs** in the period (7, 14, 30 or 90
154
+ days; 14 by default), on every branch unless you name one — without runs where
155
+ most of the suite failed at once. Local runs never count. Per test:
156
+
157
+ - **stability** — runs that passed on the first try ÷ runs;
158
+ - **flakyRate** — runs that passed only after a retry ÷ runs;
159
+ - **failureRate** — failed runs ÷ runs.
160
+
161
+ Each is a fraction from 0 to 1, and `null` when the test has no run in the
162
+ period. **There is no verdict**: nothing calls a test flaky, broken or
163
+ healthy. The tools return the numbers; the agent — or you — judges them.
164
+
165
+ ### Example: `squally-list-tests`
166
+
167
+ ```json
168
+ { "projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15", "perPage": 3 }
169
+ ```
170
+
171
+ The three least stable tests of the last 14 days (abridged):
172
+
173
+ ```json
174
+ {
175
+ "population": "ci_completed_not_excluded",
176
+ "days": 14,
177
+ "sort": "stability",
178
+ "total": 409,
179
+ "items": [
180
+ {
181
+ "testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code",
182
+ "filePath": "tests/checkout.spec.ts",
183
+ "runs": 8,
184
+ "stability": 0.5,
185
+ "flakyRate": 0.5,
186
+ "failureRate": 0,
187
+ "timeLostMs": 7361,
188
+ "topBranch": { "branch": "main", "kind": "retries", "count": 3 }
189
+ }
190
+ ]
191
+ }
192
+ ```
193
+
194
+ ### Example: `squally-get-test-metrics`
195
+
196
+ ```json
197
+ {
198
+ "projectId": "3f7c2a91-5b8e-4d0a-9c61-2e4b7f0d8a15",
199
+ "testName": "chromium > checkout.spec.ts > Checkout > applies a coupon code"
200
+ }
201
+ ```
202
+
203
+ The same numbers as that test's row, for one test (abridged):
204
+
205
+ ```json
206
+ {
207
+ "days": 14,
208
+ "branch": null,
209
+ "runs": 8,
210
+ "stableRuns": 4,
211
+ "flakyRuns": 4,
212
+ "failedRuns": 0,
213
+ "stability": 0.5,
214
+ "flakyRate": 0.5,
215
+ "failureRate": 0,
216
+ "branches": ["feature/saved-carts", "main"]
217
+ }
218
+ ```
219
+
220
+ Pass the name exactly as Squally shows it. When it exists in several spec
221
+ files, the tool answers with the `filePath` values to retry with; a test with
222
+ results but no completed CI run in the period answers `runs: 0` and `null`
223
+ rates — an answer, not an error.
151
224
 
152
225
  ## Development
153
226
 
@@ -163,7 +236,8 @@ vendored at `src/openapi/v1.json` — an input schema is the operation's
163
236
  parameters, an output schema is its 200 response. Nothing is transcribed by
164
237
  hand, so the two cannot drift. `test/drift.test.js` compares the vendored copy
165
238
  against the live document and fails when an operation or parameter has moved;
166
- it skips loudly when offline.
239
+ it skips loudly when offline, and while the live API is still an older version
240
+ than the vendored one — a release that has not been deployed yet.
167
241
 
168
242
  ## Releasing
169
243
 
package/dist/api.js CHANGED
@@ -1,6 +1,18 @@
1
1
  import { errorCauses } from "./diagnostics.js";
2
2
  /** Set by the API from 7 days before the key expires (OpenAPI components.headers). */
3
3
  export const KEY_EXPIRY_HEADER = "Squally-Key-Expires-At";
4
+ /**
5
+ * Query parameters for which an EMPTY STRING is a value, not an absence.
6
+ *
7
+ * filePath: "" names a test with no file, and the API tells the two apart
8
+ * (`?filePath=` present and empty vs. not sent - squally-app's test-scoped
9
+ * routes read `searchParams.get("filePath")`, so "" and null differ). Until
10
+ * 0.2.0 this client dropped the empty value like any other, so a tool
11
+ * description that promised "an empty value means a test with no file"
12
+ * silently asked about the name alone - and got ambiguous_test back for the
13
+ * very case the value exists to settle.
14
+ */
15
+ const EMPTY_IS_A_VALUE = new Set(["filePath"]);
4
16
  /**
5
17
  * Builds the request URL.
6
18
  *
@@ -26,8 +38,11 @@ export function buildUrl(apiBase, operation, args) {
26
38
  const value = args[parameter.name];
27
39
  // Omitted rather than sent empty: the API is strict about query values and
28
40
  // answers invalid_parameter rather than falling back to a default, so an
29
- // empty string would turn "not specified" into a 400.
30
- if (value === undefined || value === null || value === "")
41
+ // empty string would turn "not specified" into a 400. Except where the
42
+ // empty string IS the value (EMPTY_IS_A_VALUE).
43
+ if (value === undefined || value === null)
44
+ continue;
45
+ if (value === "" && !EMPTY_IS_A_VALUE.has(parameter.name))
31
46
  continue;
32
47
  query.set(parameter.name, String(value));
33
48
  }
@@ -1,16 +1,27 @@
1
1
  // The server's `instructions` - what a client puts in front of the model once,
2
2
  // before any tool is called (decided 24.09.).
3
3
  //
4
- // Four sentences, each preventing one concrete mistake: starting without a
5
- // projectId (which every other tool needs), reaching for the expensive ranking
6
- // to ask about one test, guessing at an ambiguous test name instead of passing
7
- // filePath, and ignoring the action the API already put in the error.
4
+ // Each sentence prevents one concrete mistake: starting without a projectId
5
+ // (which every other tool needs), listing the whole suite to ask about one
6
+ // test, reading a rate without knowing what it divides by or which runs it
7
+ // counts, reading a verdict into numbers that carry none, guessing at an
8
+ // ambiguous test name instead of passing filePath, and ignoring the action the
9
+ // API already put in the error.
10
+ //
11
+ // NO VERDICT SINCE 0.2.0 (API 1.0.0-beta.2): squally-app removed the engine
12
+ // that called tests flaky, broken or healthy (25.09.2026). The metrics are
13
+ // fractions of runs and nothing more; saying so here stops a model from
14
+ // presenting a rate as Squally's judgement.
8
15
  //
9
16
  // Exported as `squally-mcp/instructions` (package.json), so it stays a plain
10
17
  // constant - test/exports.test.js imports it through the package name.
11
18
  export const INSTRUCTIONS = [
12
19
  "Start with squally-list-projects; every other tool needs a projectId from it.",
13
- "For one test, use squally-get-test-status (cheap), not squally-list-flaky-tests",
14
- "(expensive, one engine pass). If a test name is ambiguous, repeat with filePath",
15
- "from the error. Errors carry a code and an action; follow the action.",
20
+ "For one test, use squally-get-test-metrics, not squally-list-tests.",
21
+ "Test metrics count completed CI runs in the period, without runs where most of the",
22
+ "suite failed at once; local runs never count. stability = runs that passed on the",
23
+ "first try / runs; flakyRate = runs that passed only after a retry / runs;",
24
+ "failureRate = failed runs / runs. There is no verdict: the tools return numbers,",
25
+ "and you judge them. If a test name is ambiguous, repeat with filePath from the",
26
+ "error. Errors carry a code and an action; follow the action.",
16
27
  ].join(" ");