@keepsnappy/agent 0.1.0-alpha.11 → 0.1.0-alpha.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -6
- package/dist/appium-driver.js +129 -3
- package/dist/keepsnappy.js +8 -11
- package/examples/farm/flows/perflab/send-markers.js +2 -3
- package/examples/farm/workflows/perflab-measure.yml +3 -3
- package/examples/github-actions.yml +4 -4
- package/examples/keepsnappy.yml +1 -1
- package/maestro/perflab/send-markers.js +2 -3
- package/maestro/perflab/step.js +6 -7
- package/package.json +2 -3
- package/examples/perflab.yml +0 -36
package/README.md
CHANGED
|
@@ -22,7 +22,7 @@ perflab doctor # xcrun, simctl, devicect
|
|
|
22
22
|
- `--arms`: `NAME=<.app/.ipa path>` (installed with `simctl install` / `devicectl device install app`, only when the arm changes) or `NAME=<bundle id>` (already installed). Without `--arms` there is one arm `A`: plain repetition, usable for `keepsnappy aa`.
|
|
23
23
|
- `--arm-env 'A:KEY=VAL'` (repeatable): extra env for the test command on that arm's runs, also exported as `MAESTRO_KEY` (Maestro forwards only `MAESTRO_*` to flows). Used for same-binary arms that differ only in knobs, e.g. `--arms A=dev.perflab.reference,B=dev.perflab.reference --arm-env 'A:REF_KNOBS=…' --arm-env 'B:REF_KNOBS=…'`. The arms with their env are written to `<out>/arms.json`. Order stays ABBA.
|
|
24
24
|
- `--platform ios-device` needs `--device <udid>`. The simulator default target is `booted`.
|
|
25
|
-
- The test command runs as argv (no shell) with `
|
|
25
|
+
- The test command runs as argv (no shell) with `KEEPSNAPPY_RUN_ID`, `KEEPSNAPPY_ARM`, `KEEPSNAPPY_RUN_DIR`, `KEEPSNAPPY_MARKER_URL` and `MAESTRO_KEEPSNAPPY_MARKER_URL` / `MAESTRO_KEEPSNAPPY_RUN_ID` in its environment.
|
|
26
26
|
- Declared steps (Maestro): call `maestro/perflab/step.yaml` with `STEP`/`PHASE` (start|end) from your flow; `measure` adds `step:<name>.*` rows (source=declared). `--no-markers` turns the endpoint off, `--clock-offset-ms` overrides the clock estimate. See docs/research/28-step-markers-prototype.md.
|
|
27
27
|
|
|
28
28
|
## `keepsnappy run` and `keepsnappy.yml` (customer entry point)
|
|
@@ -148,9 +148,9 @@ that arrives after a run was finished is added by the server and the run re-proc
|
|
|
148
148
|
result, `3` with `--fail-on-worse` on "More work", `2` for configuration, `1` when the API fails or nothing could be
|
|
149
149
|
judged ("Blocked": no iteration sent any data).
|
|
150
150
|
|
|
151
|
-
Slack: set `
|
|
151
|
+
Slack: set `KEEPSNAPPY_SLACK_WEBHOOK` (an incoming-webhook URL kept as a CI secret) and `farm close --wait` posts the
|
|
152
152
|
result: a pull request only when something got worse (a table of the 3 biggest changes, main vs PR), main only when
|
|
153
|
-
something changed (`
|
|
153
|
+
something changed (`KEEPSNAPPY_SLACK_ALWAYS=1` also posts "no change"), and a measurement that sent no data always. The
|
|
154
154
|
commit's title and author come from `git log` in the checkout. A Slack failure is printed and never changes the exit code.
|
|
155
155
|
|
|
156
156
|
Declared steps on the farm: `maestro/perflab/step.yaml` / `step.js` / `send-markers.js` (copy them into the suite).
|
|
@@ -162,9 +162,9 @@ run is invalid through its missing completion marker, and its markers stay for d
|
|
|
162
162
|
(`examples/farm/flows/perflab/`) ships its own copy of `send-markers.js`. Without a token, `step.js` posts to the local
|
|
163
163
|
CLI endpoint at once; there a step started but never ended invalidates its run (`unclosed-steps`).
|
|
164
164
|
|
|
165
|
-
Detox (Jest): `packages/detox` (`@keepsnappy/detox`) gives `step(name, fn)` / `stepStart` / `stepEnd`, posting the same markers: to the local endpoint from `
|
|
165
|
+
Detox (Jest): `packages/detox` (`@keepsnappy/detox`) gives `step(name, fn)` / `stepStart` / `stepEnd`, posting the same markers: to the local endpoint from `KEEPSNAPPY_MARKER_URL` + `KEEPSNAPPY_RUN_ID` (set by `keepsnappy run`/`measure`), or on the farm to `KEEPSNAPPY_API_URL/v1/markers` with `KEEPSNAPPY_RUN_TOKEN` as the bearer token. Optional per-test step names via `@keepsnappy/detox/auto-steps`. See its README.
|
|
166
166
|
|
|
167
|
-
Farm e2e against a local API: `KEEPSNAPPY_KEY=…
|
|
167
|
+
Farm e2e against a local API: `KEEPSNAPPY_KEY=… KEEPSNAPPY_INGEST_KEY=… e2e/farm/run.sh <new out dir>` (fake session using
|
|
168
168
|
the SDK push client, one late batch, one empty iteration).
|
|
169
169
|
|
|
170
170
|
## `keepsnappy findings` and friends (read-only questions)
|
|
@@ -258,4 +258,4 @@ Memory slopes (Theil-Sen over the cycle index, 3+ cycles of `--mem-window`, defa
|
|
|
258
258
|
`bun run typecheck` and `bun run test` first build the workspace packages the CLI imports (`bun run deps`: stats, derive,
|
|
259
259
|
verdicts, profile-analysis, mcp), so both work from a clean checkout.
|
|
260
260
|
|
|
261
|
-
|
|
261
|
+
Only `keepsnappy` and `keepsnappy.yml` / `keepsnappy.yaml` are supported. Configure `KEEPSNAPPY_*` variables. Explicit `--key-env`, `project.key_env`, and `--config` remain arbitrary user-supplied names and paths.
|
package/dist/appium-driver.js
CHANGED
|
@@ -448,9 +448,12 @@ class Driver {
|
|
|
448
448
|
}
|
|
449
449
|
async tapElement(el) {
|
|
450
450
|
const r = this.rects.get(el) ?? await this.cmd("GET", `/element/${el}/rect`);
|
|
451
|
+
await this.tapPoint({ x: r.x + r.width / 2, y: r.y + r.height / 2 });
|
|
452
|
+
}
|
|
453
|
+
async tapPoint(p) {
|
|
451
454
|
const w = await this.screen();
|
|
452
|
-
const x = Math.round(Math.min(Math.max(
|
|
453
|
-
const y = Math.round(Math.min(Math.max(
|
|
455
|
+
const x = Math.round(Math.min(Math.max(p.x, 1), w.width - 1));
|
|
456
|
+
const y = Math.round(Math.min(Math.max(p.y, 1), w.height - 1));
|
|
454
457
|
await this.cmd("POST", "/actions", {
|
|
455
458
|
actions: [
|
|
456
459
|
{
|
|
@@ -674,6 +677,13 @@ var MARKERS_PER_REQUEST = 500;
|
|
|
674
677
|
var MARKER_TRIES = 3;
|
|
675
678
|
var HOST_CLOCK_TRIPS = 6;
|
|
676
679
|
var RETRY_BASE_MS = 1000;
|
|
680
|
+
var FLING_MS = 80;
|
|
681
|
+
var FLING_FROM = 0.8;
|
|
682
|
+
var FLING_TO = 0.3;
|
|
683
|
+
var FLING_TIMEOUT_MS = 30000;
|
|
684
|
+
var STALE_REFINDS = 3;
|
|
685
|
+
var FRAME_POLL_MS = 50;
|
|
686
|
+
var sameRect = (a, b) => a.x === b.x && a.y === b.y && a.width === b.width && a.height === b.height;
|
|
677
687
|
var RETRY_AFTER_CAP_MS = 30000;
|
|
678
688
|
async function post(fetchImpl, url, token, body, now) {
|
|
679
689
|
const t0 = now();
|
|
@@ -780,6 +790,10 @@ class Session {
|
|
|
780
790
|
endGc;
|
|
781
791
|
gcN = 0;
|
|
782
792
|
stepDepth = 0;
|
|
793
|
+
refinds = 0;
|
|
794
|
+
get staleRefinds() {
|
|
795
|
+
return this.refinds;
|
|
796
|
+
}
|
|
783
797
|
constructor(o) {
|
|
784
798
|
this.driver = o.driver;
|
|
785
799
|
this.scheme = o.scheme;
|
|
@@ -992,6 +1006,118 @@ class Session {
|
|
|
992
1006
|
await this.swipe([50, 50], direction === "down" ? [50, 10] : [50, 90], durationMs);
|
|
993
1007
|
}
|
|
994
1008
|
}
|
|
1009
|
+
async frameOf(sel, el) {
|
|
1010
|
+
for (let refinds = 0;; refinds++) {
|
|
1011
|
+
try {
|
|
1012
|
+
return { el, rect: await this.driver.cmd("GET", `/element/${el}/rect`) };
|
|
1013
|
+
} catch (e) {
|
|
1014
|
+
if (!(e instanceof WdError && e.wdError === "stale element reference"))
|
|
1015
|
+
throw e;
|
|
1016
|
+
const what = describeSelector(toSelector(sel));
|
|
1017
|
+
if (refinds >= STALE_REFINDS)
|
|
1018
|
+
throw new Error(`${what} went stale ${refinds + 1} times in a row (${refinds} re-finds): ${e.message}`);
|
|
1019
|
+
this.refinds++;
|
|
1020
|
+
try {
|
|
1021
|
+
el = await this.waitVisible(sel);
|
|
1022
|
+
} catch (lost) {
|
|
1023
|
+
throw new Error(`${what} went stale and was not found again: ${lost instanceof Error ? lost.message : String(lost)}`);
|
|
1024
|
+
}
|
|
1025
|
+
}
|
|
1026
|
+
}
|
|
1027
|
+
}
|
|
1028
|
+
async settledFrame(sel, timeoutMs) {
|
|
1029
|
+
const start = this.now();
|
|
1030
|
+
let at = await this.frameOf(sel, await this.waitVisible(sel));
|
|
1031
|
+
for (;; ) {
|
|
1032
|
+
await this.driver.sleep(FRAME_POLL_MS);
|
|
1033
|
+
const now = await this.frameOf(sel, at.el);
|
|
1034
|
+
if (sameRect(at.rect, now.rect))
|
|
1035
|
+
return now;
|
|
1036
|
+
if (this.now() - start > timeoutMs)
|
|
1037
|
+
throw new Error(`${describeSelector(toSelector(sel))} did not stop moving within ${timeoutMs} ms`);
|
|
1038
|
+
at = now;
|
|
1039
|
+
}
|
|
1040
|
+
}
|
|
1041
|
+
async flingIn(r, toward, durationMs) {
|
|
1042
|
+
const [from, to] = toward === "end" ? [FLING_FROM, FLING_TO] : [FLING_TO, FLING_FROM];
|
|
1043
|
+
const x = Math.round(r.x + r.width / 2);
|
|
1044
|
+
await this.driver.drag({ x, y: Math.round(r.y + r.height * from) }, { x, y: Math.round(r.y + r.height * to) }, durationMs);
|
|
1045
|
+
}
|
|
1046
|
+
async flingTimes(list, times, toward, { durationMs = FLING_MS, timeoutMs = FLING_TIMEOUT_MS } = {}) {
|
|
1047
|
+
if (!Number.isInteger(times) || times < 1)
|
|
1048
|
+
throw new Error(`flingTimes needs a whole number of flings >= 1, got ${times}`);
|
|
1049
|
+
await this.ready();
|
|
1050
|
+
const start = this.now();
|
|
1051
|
+
const refindsBefore = this.refinds;
|
|
1052
|
+
let flings = 0;
|
|
1053
|
+
let status = 1;
|
|
1054
|
+
try {
|
|
1055
|
+
const { rect } = await this.settledFrame(list, timeoutMs);
|
|
1056
|
+
for (;flings < times; flings++)
|
|
1057
|
+
await this.flingIn(rect, toward, durationMs);
|
|
1058
|
+
status = 0;
|
|
1059
|
+
} finally {
|
|
1060
|
+
this.driver.log.push({ ...this.driver.context, cmd: `fling-${toward}-times`, t: this.now(), ms: this.now() - start, status, flings, refinds: this.refinds - refindsBefore });
|
|
1061
|
+
}
|
|
1062
|
+
}
|
|
1063
|
+
async flingUntil(list, target, toward, { blind = 0, durationMs = FLING_MS, timeoutMs = FLING_TIMEOUT_MS } = {}) {
|
|
1064
|
+
await this.ready();
|
|
1065
|
+
const start = this.now();
|
|
1066
|
+
const refindsBefore = this.refinds;
|
|
1067
|
+
let flings = 0;
|
|
1068
|
+
let checks = 0;
|
|
1069
|
+
let status = 1;
|
|
1070
|
+
try {
|
|
1071
|
+
let at = await this.settledFrame(list, timeoutMs);
|
|
1072
|
+
const fling = async () => {
|
|
1073
|
+
if (flings > 0)
|
|
1074
|
+
at = await this.frameOf(list, at.el);
|
|
1075
|
+
await this.flingIn(at.rect, toward, durationMs);
|
|
1076
|
+
flings++;
|
|
1077
|
+
};
|
|
1078
|
+
while (flings < blind)
|
|
1079
|
+
await fling();
|
|
1080
|
+
const sel = toSelector(target);
|
|
1081
|
+
while (checks++, !await this.driver.findShown(sel, true)) {
|
|
1082
|
+
if (this.now() - start > timeoutMs)
|
|
1083
|
+
throw new Error(`the ${toward} of the list did not show within ${timeoutMs} ms (${flings} flings)`);
|
|
1084
|
+
await fling();
|
|
1085
|
+
}
|
|
1086
|
+
status = 0;
|
|
1087
|
+
return flings;
|
|
1088
|
+
} finally {
|
|
1089
|
+
this.driver.log.push({ ...this.driver.context, cmd: `fling-to-${toward}`, t: this.now(), ms: this.now() - start, status, flings, checks, blind, refinds: this.refinds - refindsBefore });
|
|
1090
|
+
}
|
|
1091
|
+
}
|
|
1092
|
+
async stableCentres(sels, timeoutMs = FLING_TIMEOUT_MS) {
|
|
1093
|
+
await this.ready();
|
|
1094
|
+
const start = this.now();
|
|
1095
|
+
const els = [];
|
|
1096
|
+
for (const sel of sels)
|
|
1097
|
+
els.push(await this.waitVisible(sel));
|
|
1098
|
+
const frames = async () => {
|
|
1099
|
+
const out = [];
|
|
1100
|
+
for (let i = 0;i < sels.length; i++) {
|
|
1101
|
+
const f = await this.frameOf(sels[i], els[i]);
|
|
1102
|
+
els[i] = f.el;
|
|
1103
|
+
out.push(f.rect);
|
|
1104
|
+
}
|
|
1105
|
+
return out;
|
|
1106
|
+
};
|
|
1107
|
+
for (let before = await frames();; ) {
|
|
1108
|
+
await this.driver.sleep(FRAME_POLL_MS);
|
|
1109
|
+
const now = await frames();
|
|
1110
|
+
if (now.every((r, i) => sameRect(r, before[i])))
|
|
1111
|
+
return now.map((r) => ({ x: Math.round(r.x + r.width / 2), y: Math.round(r.y + r.height / 2) }));
|
|
1112
|
+
if (this.now() - start > timeoutMs)
|
|
1113
|
+
throw new Error(`the elements did not stop moving within ${timeoutMs} ms`);
|
|
1114
|
+
before = now;
|
|
1115
|
+
}
|
|
1116
|
+
}
|
|
1117
|
+
async tapAt(point) {
|
|
1118
|
+
await this.ready();
|
|
1119
|
+
await this.driver.tapPoint(point);
|
|
1120
|
+
}
|
|
995
1121
|
async type(sel, value) {
|
|
996
1122
|
await this.ready();
|
|
997
1123
|
const el = await this.waitVisible(sel);
|
|
@@ -1081,7 +1207,7 @@ var DRIVER_OPTIONS = {
|
|
|
1081
1207
|
repeat: { type: "string", default: "1" },
|
|
1082
1208
|
device: { type: "string" },
|
|
1083
1209
|
"local-identifier": { type: "string" },
|
|
1084
|
-
"build-name": { type: "string", default: "
|
|
1210
|
+
"build-name": { type: "string", default: "keepsnappy-appium" },
|
|
1085
1211
|
"appium-version": { type: "string", default: "2.19.0" },
|
|
1086
1212
|
out: { type: "string", default: "farm-run" },
|
|
1087
1213
|
platform: { type: "string", default: "ios" },
|
package/dist/keepsnappy.js
CHANGED
|
@@ -3,12 +3,12 @@
|
|
|
3
3
|
// apps/agent/src/compat.ts
|
|
4
4
|
import { resolve } from "node:path";
|
|
5
5
|
function envValue(env, name) {
|
|
6
|
-
return env[`KEEPSNAPPY_${name}`]
|
|
6
|
+
return env[`KEEPSNAPPY_${name}`];
|
|
7
7
|
}
|
|
8
8
|
function projectKey(env, name = "KEEPSNAPPY_KEY") {
|
|
9
9
|
return name === "KEEPSNAPPY_KEY" ? envValue(env, "KEY") : env[name];
|
|
10
10
|
}
|
|
11
|
-
var CONFIG_NAMES = ["keepsnappy.yml", "keepsnappy.yaml"
|
|
11
|
+
var CONFIG_NAMES = ["keepsnappy.yml", "keepsnappy.yaml"];
|
|
12
12
|
async function readProjectConfig(read, explicit) {
|
|
13
13
|
if (explicit !== undefined)
|
|
14
14
|
return read(resolve(explicit));
|
|
@@ -5248,10 +5248,7 @@ function startMarkerServer(now = Date.now) {
|
|
|
5248
5248
|
var markerEnv = (url, runId) => ({
|
|
5249
5249
|
KEEPSNAPPY_MARKER_URL: url,
|
|
5250
5250
|
MAESTRO_KEEPSNAPPY_MARKER_URL: url,
|
|
5251
|
-
MAESTRO_KEEPSNAPPY_RUN_ID: runId
|
|
5252
|
-
PERFLAB_MARKER_URL: url,
|
|
5253
|
-
MAESTRO_PERFLAB_MARKER_URL: url,
|
|
5254
|
-
MAESTRO_PERFLAB_RUN_ID: runId
|
|
5251
|
+
MAESTRO_KEEPSNAPPY_RUN_ID: runId
|
|
5255
5252
|
});
|
|
5256
5253
|
|
|
5257
5254
|
// apps/agent/src/upload.ts
|
|
@@ -9124,7 +9121,7 @@ async function measure(deps, source, installer, opts) {
|
|
|
9124
9121
|
let exit = null;
|
|
9125
9122
|
if (!reason) {
|
|
9126
9123
|
const r = await spawner.run(cmd, args, {
|
|
9127
|
-
env: { ...armEnv.get(p.arm), KEEPSNAPPY_RUN_ID: p.runId, KEEPSNAPPY_ARM: p.arm, KEEPSNAPPY_RUN_DIR: dir,
|
|
9124
|
+
env: { ...armEnv.get(p.arm), KEEPSNAPPY_RUN_ID: p.runId, KEEPSNAPPY_ARM: p.arm, KEEPSNAPPY_RUN_DIR: dir, ...opts.markers ? markerEnv(opts.markers.url, p.runId) : {} },
|
|
9128
9125
|
...opts.testTimeoutMs ? { timeoutMs: opts.testTimeoutMs } : {}
|
|
9129
9126
|
});
|
|
9130
9127
|
exit = r.code;
|
|
@@ -10072,7 +10069,7 @@ import { existsSync, readFileSync, statSync } from "node:fs";
|
|
|
10072
10069
|
import { dirname as dirname2, join as join12, relative as relative2, resolve as resolve4 } from "node:path";
|
|
10073
10070
|
import { fileURLToPath } from "node:url";
|
|
10074
10071
|
function driverVersion() {
|
|
10075
|
-
const built =
|
|
10072
|
+
const built = "0.1.0-alpha.12";
|
|
10076
10073
|
if (built)
|
|
10077
10074
|
return built;
|
|
10078
10075
|
let dir = dirname2(fileURLToPath(import.meta.url));
|
|
@@ -10082,7 +10079,7 @@ function driverVersion() {
|
|
|
10082
10079
|
continue;
|
|
10083
10080
|
try {
|
|
10084
10081
|
const pkg = JSON.parse(readFileSync(file, "utf8"));
|
|
10085
|
-
if (
|
|
10082
|
+
if (pkg.name === "@keepsnappy/appium-driver" && pkg.version)
|
|
10086
10083
|
return pkg.version;
|
|
10087
10084
|
} catch {}
|
|
10088
10085
|
}
|
|
@@ -11144,7 +11141,7 @@ function substitute(value, s) {
|
|
|
11144
11141
|
return value.replace(/\{tunnel:(\d+)\}/g, (_m, port) => `http://bs-local.com:${port}`).replace(/\{run_id\}/g, s.runId ?? "{run_id}").replace(/\{out\}/g, s.out ?? "{out}").replace(/\{tokens_js\}/g, s.tokensJs ?? "{tokens_js}").replace(/\{run_set_id\}/g, s.runSetId ?? "{run_set_id}");
|
|
11145
11142
|
}
|
|
11146
11143
|
function secretsOf(p, f, env) {
|
|
11147
|
-
const out = [env.KEEPSNAPPY_KEY, env.
|
|
11144
|
+
const out = [env.KEEPSNAPPY_KEY, env.KEEPSNAPPY_SLACK_WEBHOOK, env.BROWSERSTACK_ACCESS_KEY, projectKey(env, f["key-env"]), env.EXPO_TOKEN, envValue(env, "SLACK_WEBHOOK")];
|
|
11148
11145
|
for (const [k, v] of p.env)
|
|
11149
11146
|
if (SECRETISH.test(k))
|
|
11150
11147
|
out.push(v);
|
|
@@ -11634,7 +11631,7 @@ async function get(t, path, query = {}) {
|
|
|
11634
11631
|
if (!r.ok) {
|
|
11635
11632
|
const code = typeof body.error === "string" ? body.error : String(r.status);
|
|
11636
11633
|
const message = typeof body.message === "string" ? body.message : code;
|
|
11637
|
-
const hint = r.status === 401 ? " (check KEEPSNAPPY_KEY
|
|
11634
|
+
const hint = r.status === 401 ? " (check KEEPSNAPPY_KEY)" : r.status === 403 ? " (the key needs the read or run scope)" : "";
|
|
11638
11635
|
throw new ApiError(r.status, code, `${message}${hint}`);
|
|
11639
11636
|
}
|
|
11640
11637
|
return body;
|
|
@@ -13,8 +13,7 @@
|
|
|
13
13
|
try {
|
|
14
14
|
var g = typeof globalThis !== 'undefined' ? globalThis : this;
|
|
15
15
|
var pick = function (name, dflt) {
|
|
16
|
-
var
|
|
17
|
-
var names = modern === name ? [name, 'MAESTRO_' + name] : [modern, 'MAESTRO_' + modern, name, 'MAESTRO_' + name];
|
|
16
|
+
var names = [name, 'MAESTRO_' + name];
|
|
18
17
|
for (var i = 0; i < names.length; i++) {
|
|
19
18
|
var v = g[names[i]];
|
|
20
19
|
if (typeof v === 'string' && (v !== '' || names[i].indexOf('KEEPSNAPPY_') >= 0)) return v;
|
|
@@ -29,7 +28,7 @@ try {
|
|
|
29
28
|
var out = typeof output !== 'undefined' && output ? output : {};
|
|
30
29
|
var list = out.perflabMarkers || [];
|
|
31
30
|
out.perflabMarkers = [];
|
|
32
|
-
var api = pick('
|
|
31
|
+
var api = pick('KEEPSNAPPY_API_URL', typeof out.perflabApi === 'string' ? out.perflabApi : '').replace(/\/+$/, '');
|
|
33
32
|
var wait = function (ms) {
|
|
34
33
|
try {
|
|
35
34
|
java.lang.Thread.sleep(ms);
|
|
@@ -37,7 +37,7 @@ jobs:
|
|
|
37
37
|
BROWSERSTACK_ACCESS_KEY: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}
|
|
38
38
|
KEEPSNAPPY_KEY: ${{ secrets.KEEPSNAPPY_KEY }} # run scope; never leaves this runner
|
|
39
39
|
KEEPSNAPPY_API: https://perflab-api-864539138788.europe-west1.run.app
|
|
40
|
-
|
|
40
|
+
KEEPSNAPPY_CLI: '@keepsnappy/agent@0.1.0-alpha.1' # first version with `farm` (not published yet)
|
|
41
41
|
OUT_DIR: ${{ github.workspace }}/farm-run
|
|
42
42
|
RUN_ID: perflab-${{ inputs.platform }}-${{ github.run_id }}-${{ github.run_attempt }}
|
|
43
43
|
JOURNEYS: e2e-community-and-live-chat # = the JOURNEY values in run-perflab.yaml
|
|
@@ -71,7 +71,7 @@ jobs:
|
|
|
71
71
|
APP_COMMIT: ${{ steps.app.outputs.app_commit }}
|
|
72
72
|
run: |
|
|
73
73
|
[ -n "$APP_COMMIT" ] || { echo '::error::the artifact carries no commit'; exit 1; }
|
|
74
|
-
npx --yes "$
|
|
74
|
+
npx --yes "$KEEPSNAPPY_CLI" farm open \
|
|
75
75
|
--api "$KEEPSNAPPY_API" --branch main --commit "$APP_COMMIT" \
|
|
76
76
|
--device-model 'iPhone 15 Pro' --device-os 17 --device-provider browserstack \
|
|
77
77
|
--journeys "$JOURNEYS" --repeat "$REPEAT" \
|
|
@@ -99,7 +99,7 @@ jobs:
|
|
|
99
99
|
- name: Close the run set and print the verdict
|
|
100
100
|
if: always() && hashFiles('farm-run/perflab-farm.json') != ''
|
|
101
101
|
run: |
|
|
102
|
-
npx --yes "$
|
|
102
|
+
npx --yes "$KEEPSNAPPY_CLI" farm close \
|
|
103
103
|
--state "$OUT_DIR/perflab-farm.json" \
|
|
104
104
|
--bs-build "$OUT_DIR/browserstack.${{ inputs.platform }}.json" --settle-seconds 60 --wait
|
|
105
105
|
|
|
@@ -1,14 +1,14 @@
|
|
|
1
|
-
# .github/workflows/
|
|
1
|
+
# .github/workflows/keepsnappy.yml: measure every pull request against main, and grow main's history on every merge.
|
|
2
2
|
#
|
|
3
3
|
# Needs: a perf build of your app (built with PERFLAB=1, so the KeepSnappy SDK writes its data), Maestro flows, a
|
|
4
|
-
#
|
|
5
|
-
# secret KEEPSNAPPY_KEY. The key never appears in logs;
|
|
4
|
+
# keepsnappy.yml in the repo root (see examples/keepsnappy.yml), and a project key with `run` scope stored as the repository
|
|
5
|
+
# secret KEEPSNAPPY_KEY. The key never appears in logs; keepsnappy reads it from the environment variable keepsnappy.yml names.
|
|
6
6
|
#
|
|
7
7
|
# What you get: on main, the runs join the history of each journey (one journey per Maestro flow file). On a pull
|
|
8
8
|
# request, each journey is compared with main's recent history; the verdict ("More work", "Less work", "No change",
|
|
9
9
|
# "Not enough data yet") is printed in the log and written to the job summary with a link to the comparison page.
|
|
10
10
|
# The job does not fail on "More work" (no blocking checks for now); add --fail-on-worse to make it exit 3 instead.
|
|
11
|
-
# Exit 2 = configuration problem (
|
|
11
|
+
# Exit 2 = configuration problem (keepsnappy.yml, flags), exit 1 = infrastructure (device log, upload).
|
|
12
12
|
|
|
13
13
|
name: perflab
|
|
14
14
|
|
package/examples/keepsnappy.yml
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
#
|
|
1
|
+
# keepsnappy.yml: the only file you maintain for KeepSnappy. Run it with `keepsnappy run`.
|
|
2
2
|
|
|
3
3
|
project:
|
|
4
4
|
key_env: KEEPSNAPPY_KEY # NAME of the env var holding the project key (a CI secret), never the key itself
|
|
@@ -13,8 +13,7 @@
|
|
|
13
13
|
try {
|
|
14
14
|
var g = typeof globalThis !== 'undefined' ? globalThis : this;
|
|
15
15
|
var pick = function (name, dflt) {
|
|
16
|
-
var
|
|
17
|
-
var names = modern === name ? [name, 'MAESTRO_' + name] : [modern, 'MAESTRO_' + modern, name, 'MAESTRO_' + name];
|
|
16
|
+
var names = [name, 'MAESTRO_' + name];
|
|
18
17
|
for (var i = 0; i < names.length; i++) {
|
|
19
18
|
var v = g[names[i]];
|
|
20
19
|
if (typeof v === 'string' && (v !== '' || names[i].indexOf('KEEPSNAPPY_') >= 0)) return v;
|
|
@@ -29,7 +28,7 @@ try {
|
|
|
29
28
|
var out = typeof output !== 'undefined' && output ? output : {};
|
|
30
29
|
var list = out.perflabMarkers || [];
|
|
31
30
|
out.perflabMarkers = [];
|
|
32
|
-
var api = pick('
|
|
31
|
+
var api = pick('KEEPSNAPPY_API_URL', typeof out.perflabApi === 'string' ? out.perflabApi : '').replace(/\/+$/, '');
|
|
33
32
|
var wait = function (ms) {
|
|
34
33
|
try {
|
|
35
34
|
java.lang.Thread.sleep(ms);
|
package/maestro/perflab/step.js
CHANGED
|
@@ -2,17 +2,16 @@
|
|
|
2
2
|
// Rhino). Must not throw: a marker is never a test failure.
|
|
3
3
|
// Inputs: STEP, PHASE (from step.yaml). t_host is taken now, when the action happens, whenever the marker is sent.
|
|
4
4
|
// Farm mode (`perflab farm open`): the flow has set `output.perflabToken` (one run token per iteration, from the
|
|
5
|
-
// generated tokens.js) and the API base comes from `output.perflabApi` or
|
|
5
|
+
// generated tokens.js) and the API base comes from `output.perflabApi` or KEEPSNAPPY_API_URL. The marker is only recorded
|
|
6
6
|
// in the flow's memory; send-markers.js (end-run.yaml) posts the iteration's markers to <api>/v1/markers after it
|
|
7
7
|
// ended, so the test never waits on the network between actions.
|
|
8
|
-
// Local mode (`
|
|
8
|
+
// Local mode (`keepsnappy run` / `measure`): KEEPSNAPPY_MARKER_URL, KEEPSNAPPY_RUN_ID point at the CLI's own endpoint on this
|
|
9
9
|
// host, posted at once (local flows have no end-run step). Maestro only forwards host env vars with the MAESTRO_ prefix,
|
|
10
10
|
// unprefixed, so both spellings are read (docs/research/28).
|
|
11
11
|
try {
|
|
12
12
|
var g = typeof globalThis !== 'undefined' ? globalThis : this;
|
|
13
13
|
var pick = function (name, dflt) {
|
|
14
|
-
var
|
|
15
|
-
var names = modern === name ? [name, 'MAESTRO_' + name] : [modern, 'MAESTRO_' + modern, name, 'MAESTRO_' + name];
|
|
14
|
+
var names = [name, 'MAESTRO_' + name];
|
|
16
15
|
for (var i = 0; i < names.length; i++) {
|
|
17
16
|
var v = g[names[i]];
|
|
18
17
|
if (typeof v === 'string' && (v !== '' || names[i].indexOf('KEEPSNAPPY_') >= 0)) return v;
|
|
@@ -23,15 +22,15 @@ try {
|
|
|
23
22
|
var step = pick('STEP', '');
|
|
24
23
|
var phase = pick('PHASE', '');
|
|
25
24
|
var token = typeof out.perflabToken === 'string' ? out.perflabToken : '';
|
|
26
|
-
var api = pick('
|
|
25
|
+
var api = pick('KEEPSNAPPY_API_URL', typeof out.perflabApi === 'string' ? out.perflabApi : '');
|
|
27
26
|
if (token !== '' && api !== '') {
|
|
28
27
|
out.perflabMarkers = out.perflabMarkers || [];
|
|
29
28
|
out.perflabMarkers.push({ token: token, step: step, phase: phase, t_host: Date.now() });
|
|
30
29
|
} else {
|
|
31
|
-
var base = pick('
|
|
30
|
+
var base = pick('KEEPSNAPPY_MARKER_URL', 'http://127.0.0.1:47321');
|
|
32
31
|
http.post(base + '/marker', {
|
|
33
32
|
headers: { 'Content-Type': 'application/json' },
|
|
34
|
-
body: JSON.stringify({ run: pick('
|
|
33
|
+
body: JSON.stringify({ run: pick('KEEPSNAPPY_RUN_ID', ''), step: step, phase: phase, t_host: Date.now() }),
|
|
35
34
|
});
|
|
36
35
|
}
|
|
37
36
|
} catch (e) {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@keepsnappy/agent",
|
|
3
|
-
"version": "0.1.0-alpha.
|
|
3
|
+
"version": "0.1.0-alpha.12",
|
|
4
4
|
"description": "KeepSnappy CLI: runs your app journeys, measures from inside the app and reports More work / Less work / No change per step.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "UNLICENSED",
|
|
@@ -16,8 +16,7 @@
|
|
|
16
16
|
"node": ">=22"
|
|
17
17
|
},
|
|
18
18
|
"bin": {
|
|
19
|
-
"keepsnappy": "dist/keepsnappy.js"
|
|
20
|
-
"perflab": "dist/keepsnappy.js"
|
|
19
|
+
"keepsnappy": "dist/keepsnappy.js"
|
|
21
20
|
},
|
|
22
21
|
"files": [
|
|
23
22
|
"dist",
|
package/examples/perflab.yml
DELETED
|
@@ -1,36 +0,0 @@
|
|
|
1
|
-
# perflab.yml: the only file you maintain for KeepSnappy. Run it with `keepsnappy run`.
|
|
2
|
-
|
|
3
|
-
project:
|
|
4
|
-
key_env: KEEPSNAPPY_KEY # NAME of the env var holding the project key (a CI secret), never the key itself
|
|
5
|
-
ingest_url: https://ingest.example.com # optional; only used with `keepsnappy run --upload`
|
|
6
|
-
|
|
7
|
-
app:
|
|
8
|
-
bundle_id: com.example.app # the app under test; its perf-lab data is read from the device after every run
|
|
9
|
-
|
|
10
|
-
platform: ios-sim # ios-sim | ios-device
|
|
11
|
-
device: booted # simulator UDID or "booted"; required (UDID) for ios-device
|
|
12
|
-
|
|
13
|
-
repetitions: 10 # how many times the whole flow set runs
|
|
14
|
-
baseline_branch: main # the branch your baseline comes from (CI passes --baseline <its out dir>)
|
|
15
|
-
|
|
16
|
-
flows:
|
|
17
|
-
runner: maestro # maestro: build `maestro test ...` from this file | command: run flows.command
|
|
18
|
-
maestro_version: 2.6.1 # note only: printed at the start, so a drifting Maestro is visible in logs
|
|
19
|
-
tags: [perf] # maestro --include-tags=perf
|
|
20
|
-
exclude_tags: [flaky] # maestro --exclude-tags=flaky
|
|
21
|
-
paths: # files, directories or globs (*, ?, **); default "." when only tags are given
|
|
22
|
-
- e2e/flows/**/*.yaml
|
|
23
|
-
# - path: e2e/flows/chat/open.yaml # one flow file with its dashboard title (else its `name:` / opening comment)
|
|
24
|
-
# title: Open the chat
|
|
25
|
-
# description: Community screen -> chat sheet, first message visible
|
|
26
|
-
# command: bun run e2e # runner: command. Anything that exits 0 on success (Appium, Detox, ...)
|
|
27
|
-
|
|
28
|
-
steps:
|
|
29
|
-
rename: # stable names in history; keys are window/step names as they appear in metrics.csv
|
|
30
|
-
"screen:(app)/community/[handle]/(tabs)": community.open
|
|
31
|
-
pin: [community.open] # always listed in the summary, even without a change
|
|
32
|
-
|
|
33
|
-
output:
|
|
34
|
-
dir: perflab-out # runs.ndjson, metrics.csv, <run>/ logs, verdict (compare/verdict.json)
|
|
35
|
-
|
|
36
|
-
timeout_seconds: 600 # per run of the command; optional
|