@ihyeon/flow-qa 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.ko.md +7 -1
- package/README.md +7 -1
- package/dist/src/case-selection.js +50 -0
- package/dist/src/cli.js +25 -4
- package/dist/src/engine.js +15 -9
- package/dist/src/i18n.js +2 -0
- package/dist/src/report.js +20 -5
- package/dist/src/run-comparison.js +13 -0
- package/dist/src/ui/public/app.js +51 -13
- package/dist/src/ui/public/i18n.js +11 -0
- package/dist/src/ui/public/index.html +2 -1
- package/dist/src/ui/public/style.css +11 -0
- package/dist/src/ui/server.js +18 -5
- package/docs/architecture.md +3 -1
- package/docs/mobile.md +1 -1
- package/docs/releases.md +12 -1
- package/docs/ui.md +11 -1
- package/package.json +2 -2
package/README.ko.md
CHANGED
|
@@ -16,7 +16,7 @@ CLI와 `127.0.0.1` 전용 웹 UI를 제공합니다. UI는 영어로 시작하
|
|
|
16
16
|
Node.js 24.x(24.22.1 이상) 또는 26.12.1 이상이 필요합니다. 브라우저는 첫 웹 실행 전에 설치합니다.
|
|
17
17
|
|
|
18
18
|
```sh
|
|
19
|
-
npm install -g @ihyeon/flow-qa@0.
|
|
19
|
+
npm install -g @ihyeon/flow-qa@0.4.0
|
|
20
20
|
flow-qa browser install
|
|
21
21
|
flow-qa ui --demo
|
|
22
22
|
```
|
|
@@ -62,6 +62,12 @@ flow-qa plan flows/my-flow.md --workspace my-qa
|
|
|
62
62
|
|
|
63
63
|
`flow-qa doctor --workspace my-qa`로 로컬 의존성과 모델 인증용 환경 변수의 설정 여부를 확인할 수 있습니다. 키 값은 출력하지 않고 모델 서버도 호출하지 않습니다. 환경 변수가 있어도 실제 인증 성공을 확인한 것은 아닙니다.
|
|
64
64
|
|
|
65
|
+
## 선택한 케이스만 실행
|
|
66
|
+
|
|
67
|
+
`0.4.0`부터 **시나리오 → 검토 후 실행 → 실행할 케이스**에서 필요한 케이스만 고를 수 있습니다. 처음에는 모두 선택됩니다. 선택을 바꾸면 검토 확인을 다시 해야 하며, 저장한 원본 케이스 파일은 그대로 유지됩니다. 다른 케이스를 준비 단계로 자동 실행하지 않으므로 각 케이스에 필요한 시작 상태를 직접 구성하세요.
|
|
68
|
+
|
|
69
|
+
CLI에서는 `flow-qa run cases/task-flow.json --target demo --case TC-001 --case TC-003`처럼 ID를 반복합니다. `--case`를 생략하면 전체를 실행합니다. 부분 실행의 통과는 선택한 케이스에만 해당하며, 보고서와 비교 화면에도 선택 범위를 표시합니다. 자세한 동작은 [UI 가이드](docs/ui.md#run-selected-cases)를 참고하세요.
|
|
70
|
+
|
|
65
71
|
## 실행 결과 비교
|
|
66
72
|
|
|
67
73
|
**실행 기록 → 실행 2건 비교**에서 같은 스위트와 대상의 두 기록을 선택하면 통과→실패, 실패→통과, 그 외 상태 변경, 동일 상태, 추가·누락 케이스를 확인할 수 있습니다. 분류별로 보거나 전체 비교를 Markdown·JSON으로 내려받으세요. 테스트 내용 변경과 불완전한 실행은 별도로 표시하며, 상태 변화만으로 제품 버그의 발생·해결을 확정하지 않습니다. 자세한 동작은 [UI 가이드](docs/ui.md#compare-two-runs)를 참고하세요.
|
package/README.md
CHANGED
|
@@ -16,7 +16,7 @@ Flow QA is available as a CLI and a web UI bound to `127.0.0.1`. Provide a targe
|
|
|
16
16
|
Node.js 24.x (24.22.1 or later) or 26.12.1 or later is required. Install Chromium before the first web run:
|
|
17
17
|
|
|
18
18
|
```sh
|
|
19
|
-
npm install -g @ihyeon/flow-qa@0.
|
|
19
|
+
npm install -g @ihyeon/flow-qa@0.4.0
|
|
20
20
|
flow-qa browser install
|
|
21
21
|
flow-qa ui --demo
|
|
22
22
|
```
|
|
@@ -62,6 +62,12 @@ For authenticated models, add `--key-env FLOW_QA_API_KEY` to the model command a
|
|
|
62
62
|
|
|
63
63
|
`flow-qa doctor --workspace my-qa` checks local dependencies and whether configured model-key environment variables are present. It does not print key values or call the model server; a present key does not confirm remote authentication.
|
|
64
64
|
|
|
65
|
+
## Run selected cases
|
|
66
|
+
|
|
67
|
+
From `0.4.0`, choose cases under **Scenario → Review, then run → Cases to run**. Every case starts selected. Clear or change the selection, review it, then run against your target. Selecting cases never edits the saved suite or implicitly runs prerequisite cases. Each case must establish its own starting state.
|
|
68
|
+
|
|
69
|
+
The CLI supports repeated IDs: `flow-qa run cases/task-flow.json --target demo --case TC-001 --case TC-003`. Omit `--case` to run the complete suite. Reports and comparisons label partial runs explicitly; a selected-run pass does not evaluate omitted cases. See the [UI guide](docs/ui.md#run-selected-cases).
|
|
70
|
+
|
|
65
71
|
## Compare recorded runs
|
|
66
72
|
|
|
67
73
|
In **Run history → Compare two runs**, select two reports from the same suite and target to see passed-to-failed, failed-to-passed, other status changes, unchanged, added, and missing cases. Filter the comparison or download all results as Markdown/JSON. Changed test definitions and incomplete runs carry explicit warnings; observed status changes do not prove a product defect was introduced or fixed. See the [UI guide](docs/ui.md#compare-two-runs).
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
export class CaseSelectionError extends Error {
|
|
2
|
+
}
|
|
3
|
+
/** Resolve requested IDs against the complete suite without changing its order or contents. */
|
|
4
|
+
export function selectCases(suite, caseIds) {
|
|
5
|
+
if (caseIds === undefined)
|
|
6
|
+
return { suite };
|
|
7
|
+
if (!Array.isArray(caseIds) || caseIds.length === 0)
|
|
8
|
+
throw new CaseSelectionError('Select at least one case ID.');
|
|
9
|
+
const available = new Set(suite.cases.map((testCase) => testCase.id));
|
|
10
|
+
const requested = new Set();
|
|
11
|
+
for (const id of caseIds) {
|
|
12
|
+
if (typeof id !== 'string' || !available.has(id))
|
|
13
|
+
throw new CaseSelectionError(`Unknown case ID: ${String(id)}`);
|
|
14
|
+
if (requested.has(id))
|
|
15
|
+
throw new CaseSelectionError(`Duplicate selected case ID: ${id}`);
|
|
16
|
+
requested.add(id);
|
|
17
|
+
}
|
|
18
|
+
if (requested.size === suite.cases.length)
|
|
19
|
+
return { suite };
|
|
20
|
+
const cases = suite.cases.filter((testCase) => requested.has(testCase.id));
|
|
21
|
+
const requirementIds = new Set(cases.flatMap((testCase) => testCase.requirementIds));
|
|
22
|
+
return {
|
|
23
|
+
suite: { ...suite, cases, requirements: suite.requirements.filter((requirement) => requirementIds.has(requirement.id)) },
|
|
24
|
+
selection: { caseIds: cases.map((testCase) => testCase.id), totalCases: suite.cases.length },
|
|
25
|
+
};
|
|
26
|
+
}
|
|
27
|
+
/** Check optional additive report metadata before treating a report as comparable. */
|
|
28
|
+
export function isReportSelectionValid(cases, selection) {
|
|
29
|
+
const caseIds = cases.map((item) => item.id);
|
|
30
|
+
if (caseIds.some((id) => typeof id !== 'string' || !id.trim()) || new Set(caseIds).size !== caseIds.length)
|
|
31
|
+
return false;
|
|
32
|
+
if (selection === undefined)
|
|
33
|
+
return true;
|
|
34
|
+
if (selection === null || typeof selection !== 'object' || Array.isArray(selection))
|
|
35
|
+
return false;
|
|
36
|
+
const prototype = Object.getPrototypeOf(selection);
|
|
37
|
+
if (prototype !== Object.prototype && prototype !== null)
|
|
38
|
+
return false;
|
|
39
|
+
if (Reflect.ownKeys(selection).length !== 2 || !Object.hasOwn(selection, 'caseIds') || !Object.hasOwn(selection, 'totalCases'))
|
|
40
|
+
return false;
|
|
41
|
+
const value = selection;
|
|
42
|
+
if (!Number.isInteger(value.totalCases) || value.totalCases > 100 || value.totalCases <= caseIds.length)
|
|
43
|
+
return false;
|
|
44
|
+
if (!Array.isArray(value.caseIds) || value.caseIds.length === 0 || value.caseIds.length !== caseIds.length)
|
|
45
|
+
return false;
|
|
46
|
+
if (value.caseIds.some((id) => typeof id !== 'string' || !id.trim()))
|
|
47
|
+
return false;
|
|
48
|
+
const selected = new Set(value.caseIds);
|
|
49
|
+
return selected.size === value.caseIds.length && caseIds.every((id) => selected.has(id));
|
|
50
|
+
}
|
package/dist/src/cli.js
CHANGED
|
@@ -10,6 +10,7 @@ import { configSchema, idSchema, modelSchema, targetSchema } from "./schema.js";
|
|
|
10
10
|
import { initWorkspace, readConfig, readSuite, sha256, verifySuiteSource, writeConfig, writeJSON } from "./workspace.js";
|
|
11
11
|
import { buildPlanningPrompt, planFlow } from "./planner.js";
|
|
12
12
|
import { compileSuite } from "./compiler.js";
|
|
13
|
+
import { CaseSelectionError, isReportSelectionValid, selectCases } from "./case-selection.js";
|
|
13
14
|
import { runSuite } from "./engine.js";
|
|
14
15
|
import { renderMarkdown } from "./report.js";
|
|
15
16
|
import { targetFingerprint, targetLabel } from "./targets.js";
|
|
@@ -29,6 +30,7 @@ Usage: flow-qa <command> [arguments] [options]
|
|
|
29
30
|
validate <cases.json> Check case structure and assertions
|
|
30
31
|
generate <cases.json> [--out test.ts] Generate e2e test code
|
|
31
32
|
run <cases.json> --target <name> Run against a registered target and save a QA report
|
|
33
|
+
[--case <id> ...] Run only the selected case IDs (repeat --case for more)
|
|
32
34
|
report <run-directory> View a saved QA report
|
|
33
35
|
compare <run-a> <run-b> Compare case results
|
|
34
36
|
ui [--demo] [--port 4310] Start the local QA UI
|
|
@@ -54,6 +56,7 @@ Local example: flow-qa ui --demo
|
|
|
54
56
|
validate <cases.json> 케이스 구조와 검증 조건 확인
|
|
55
57
|
generate <cases.json> [--out test.ts] e2e 테스트 코드 생성
|
|
56
58
|
run <cases.json> --target <name> 연결한 대상에서 실행하고 QA 리포트 저장
|
|
59
|
+
[--case <id> ...] 선택한 케이스만 실행 (--case를 반복해 여러 개 선택)
|
|
57
60
|
report <run-directory> 저장된 QA 리포트 보기
|
|
58
61
|
compare <run-a> <run-b> 케이스별 결과 비교
|
|
59
62
|
ui [--demo] [--port 4310] 로컬 QA 화면 실행
|
|
@@ -126,6 +129,7 @@ export async function main(args) {
|
|
|
126
129
|
const { positionals, values } = parseArgs({ args, allowPositionals: true, strict: true, options: {
|
|
127
130
|
workspace: { type: 'string' }, url: { type: 'string' }, model: { type: 'string' },
|
|
128
131
|
'key-env': { type: 'string' }, out: { type: 'string' }, target: { type: 'string' },
|
|
132
|
+
case: { type: 'string', multiple: true },
|
|
129
133
|
platform: { type: 'string' }, 'bundle-id': { type: 'string' }, device: { type: 'string' },
|
|
130
134
|
'prompt-only': { type: 'boolean' }, help: { type: 'boolean', short: 'h' },
|
|
131
135
|
demo: { type: 'boolean' }, port: { type: 'string' }, 'with-deps': { type: 'boolean' }, lang: { type: 'string' },
|
|
@@ -145,7 +149,7 @@ export async function main(args) {
|
|
|
145
149
|
(values['bundle-id'] !== undefined || values.device !== undefined) && command !== 'connect') {
|
|
146
150
|
throw new Error(localized(locale, 'Platform and device options are only supported by connect and plan. See --help.', '플랫폼·기기 옵션은 connect와 plan에서만 사용할 수 있습니다. --help를 참고하세요.'));
|
|
147
151
|
}
|
|
148
|
-
if (values.demo && command !== 'ui' || values.port && command !== 'ui' ||
|
|
152
|
+
if (values.case !== undefined && command !== 'run' || values.demo && command !== 'ui' || values.port && command !== 'ui' ||
|
|
149
153
|
values['with-deps'] && (command !== 'browser' || arg !== 'install'))
|
|
150
154
|
throw new Error(localized(locale, 'This option is not supported for the command. See --help.', '이 명령에서 사용할 수 없는 옵션입니다. --help를 참고하세요.'));
|
|
151
155
|
if (command === 'ui') {
|
|
@@ -246,12 +250,24 @@ export async function main(args) {
|
|
|
246
250
|
}
|
|
247
251
|
const config = await readConfig(workspace);
|
|
248
252
|
const targetName = values.target ?? (Object.keys(config.targets).length === 1 ? Object.keys(config.targets)[0] : undefined);
|
|
253
|
+
let caseIds;
|
|
254
|
+
try {
|
|
255
|
+
caseIds = selectCases(suite, values.case).selection?.caseIds;
|
|
256
|
+
}
|
|
257
|
+
catch (error) {
|
|
258
|
+
if (error instanceof CaseSelectionError)
|
|
259
|
+
throw new Error(localizeProductMessage('Select one or more unique cases from this suite.', locale));
|
|
260
|
+
throw error;
|
|
261
|
+
}
|
|
249
262
|
const controller = new AbortController();
|
|
250
263
|
const interrupt = () => controller.abort();
|
|
251
264
|
process.once('SIGINT', interrupt);
|
|
252
265
|
process.once('SIGTERM', interrupt);
|
|
253
266
|
try {
|
|
254
|
-
const { report, runDir } = await runSuite({ workspace, suite, config, targetName: required(targetName, '--target', locale), signal: controller.signal
|
|
267
|
+
const { report, runDir } = await runSuite({ workspace, suite, config, targetName: required(targetName, '--target', locale), signal: controller.signal,
|
|
268
|
+
...(caseIds ? { caseIds } : {}) });
|
|
269
|
+
if (report.selection)
|
|
270
|
+
console.log(localized(locale, `Selected run: ${report.selection.caseIds.length}/${report.selection.totalCases} cases; the full suite was not evaluated.`, `선택 실행: ${report.selection.totalCases}개 중 ${report.selection.caseIds.length}개. 전체 케이스를 검증한 결과가 아닙니다.`));
|
|
255
271
|
console.log(localized(locale, `Run result: ${report.status}\nQA report: ${resolve(runDir, 'qa-report.md')}\nRaw result: ${resolve(runDir, 'qa-report.json')}`, `실행 결과: ${report.status}\nQA 리포트: ${resolve(runDir, 'qa-report.md')}\n원본 결과: ${resolve(runDir, 'qa-report.json')}`));
|
|
256
272
|
process.exitCode = report.status === 'passed' ? 0 : report.status === 'interrupted' ? 130 : 1;
|
|
257
273
|
}
|
|
@@ -283,11 +299,16 @@ export async function main(args) {
|
|
|
283
299
|
const { name: _targetName, ...identity } = data.target;
|
|
284
300
|
if (!targetSchema.safeParse(identity).success)
|
|
285
301
|
throw new Error(localized(locale, 'This is not a valid QA report.', '유효한 QA 리포트가 아닙니다.'));
|
|
302
|
+
if (!isReportSelectionValid(data.cases, data.selection))
|
|
303
|
+
throw new Error(localized(locale, 'This is not a valid QA report.', '유효한 QA 리포트가 아닙니다.'));
|
|
286
304
|
return data;
|
|
287
305
|
};
|
|
288
306
|
const before = await load(arg), after = await load(second);
|
|
289
|
-
const
|
|
290
|
-
|
|
307
|
+
const sameSelection = JSON.stringify(before.cases.map((item) => item.id).sort()) === JSON.stringify(after.cases.map((item) => item.id).sort());
|
|
308
|
+
const comparable = before.suiteId === after.suiteId && before.suiteHash === after.suiteHash && targetFingerprint(before.target) === targetFingerprint(after.target) && sameSelection;
|
|
309
|
+
console.log(comparable ? localized(locale, 'Comparing the same suite, target and case selection (check external data state separately)', '동일 명세·대상·케이스 범위 비교 (외부 데이터 상태는 별도 확인)') : localized(locale, 'Suite, target or case selection differs; no direct regression verdict is made.', '명세·대상 또는 케이스 범위가 달라 직접적인 회귀 판정은 하지 않습니다.'));
|
|
310
|
+
if (before.selection || after.selection)
|
|
311
|
+
console.log(localized(locale, 'This comparison includes selected cases only; omitted cases were not evaluated.', '이 비교에는 선택 실행이 포함되어 있습니다. 미선택 케이스는 검증되지 않았습니다.'));
|
|
291
312
|
const previous = new Map(before.cases.map((c) => [c.id, c.status]));
|
|
292
313
|
const current = new Map(after.cases.map((c) => [c.id, c.status]));
|
|
293
314
|
for (const id of new Set([...previous.keys(), ...current.keys()]))
|
package/dist/src/engine.js
CHANGED
|
@@ -4,6 +4,7 @@ import { createWriteStream } from 'node:fs';
|
|
|
4
4
|
import { lstat, mkdir, readFile, realpath, stat, writeFile } from 'node:fs/promises';
|
|
5
5
|
import path from 'node:path';
|
|
6
6
|
import { fileURLToPath } from 'node:url';
|
|
7
|
+
import { selectCases } from "./case-selection.js";
|
|
7
8
|
import { compileSuite } from "./compiler.js";
|
|
8
9
|
import { acquireMobileDevice, mobileReadiness } from "./mobile-runtime.js";
|
|
9
10
|
import { failureReport, normalizeReport1, renderMarkdown, stoppedReport } from "./report.js";
|
|
@@ -158,6 +159,10 @@ function runnableSuite(suite, blocked) {
|
|
|
158
159
|
}
|
|
159
160
|
/** Run one immutable suite snapshot through the locally installed e2e CLI. */
|
|
160
161
|
export async function runSuite(options) {
|
|
162
|
+
// A malformed selection must not leave a run directory behind.
|
|
163
|
+
const selected = selectCases(options.suite, options.caseIds);
|
|
164
|
+
let executionSuite = selected.suite;
|
|
165
|
+
let selection = selected.selection;
|
|
161
166
|
const workspace = await realpath(options.workspace);
|
|
162
167
|
const qaRoot = await ownedDirectory(workspace, '.flowqa');
|
|
163
168
|
const runsRoot = await ownedDirectory(qaRoot, 'runs');
|
|
@@ -183,7 +188,7 @@ export async function runSuite(options) {
|
|
|
183
188
|
let blocked = new Map();
|
|
184
189
|
let target = { name: options.targetName, url: '' };
|
|
185
190
|
let releaseMobileDevice;
|
|
186
|
-
const context = () => ({ suite:
|
|
191
|
+
const context = () => ({ suite: executionSuite, selection, runId, suiteHash, target, startedAt, finishedAt: new Date().toISOString(), blockedReasons: blocked });
|
|
187
192
|
const blockAll = (suite, reason) => {
|
|
188
193
|
for (const testCase of suite.cases)
|
|
189
194
|
if (!blocked.has(testCase.id))
|
|
@@ -193,6 +198,7 @@ export async function runSuite(options) {
|
|
|
193
198
|
try {
|
|
194
199
|
const suite = suiteSchema.parse(options.suite);
|
|
195
200
|
await verifySuiteSource(suite, workspace);
|
|
201
|
+
({ suite: executionSuite, selection } = selectCases(suite, options.caseIds));
|
|
196
202
|
const rawTarget = options.config?.targets?.[options.targetName];
|
|
197
203
|
if (rawTarget && typeof rawTarget === 'object' && !Array.isArray(rawTarget)) {
|
|
198
204
|
const raw = rawTarget;
|
|
@@ -200,8 +206,8 @@ export async function runSuite(options) {
|
|
|
200
206
|
if ((raw.platform === 'ios' || raw.platform === 'android') && typeof raw.bundleId === 'string') {
|
|
201
207
|
target = { name: options.targetName, kind: 'mobile', platform: raw.platform, bundleId: raw.bundleId, device: '' };
|
|
202
208
|
}
|
|
203
|
-
blocked = blockedCases(
|
|
204
|
-
blockAll(
|
|
209
|
+
blocked = blockedCases(executionSuite, options.config);
|
|
210
|
+
blockAll(executionSuite, 'A mobile target requires an explicit device.');
|
|
205
211
|
}
|
|
206
212
|
}
|
|
207
213
|
const config = configSchema.parse(options.config);
|
|
@@ -209,23 +215,23 @@ export async function runSuite(options) {
|
|
|
209
215
|
if (!selectedTarget)
|
|
210
216
|
throw new Error(`Target not found: ${options.targetName}`);
|
|
211
217
|
target = reportTarget(options.targetName, selectedTarget);
|
|
212
|
-
blocked = blockedCases(
|
|
218
|
+
blocked = blockedCases(executionSuite, config);
|
|
213
219
|
if ((suite.platform ?? 'web') !== targetKind(selectedTarget))
|
|
214
|
-
blockAll(
|
|
215
|
-
const runnable =
|
|
220
|
+
blockAll(executionSuite, 'The suite platform does not match the selected target.');
|
|
221
|
+
const runnable = executionSuite.cases.filter((testCase) => !blocked.has(testCase.id));
|
|
216
222
|
let sdkDevice;
|
|
217
223
|
if (selectedTarget.kind === 'mobile' && runnable.length > 0 && !options.signal?.aborted) {
|
|
218
224
|
const readiness = await mobileReadiness(selectedTarget, options.listMobileDevices);
|
|
219
225
|
if (!readiness.ready)
|
|
220
|
-
blockAll(
|
|
226
|
+
blockAll(executionSuite, readiness.reason);
|
|
221
227
|
else {
|
|
222
228
|
sdkDevice = readiness.sdkDevice;
|
|
223
229
|
releaseMobileDevice = acquireMobileDevice(selectedTarget.platform, readiness.deviceId);
|
|
224
230
|
}
|
|
225
231
|
if (!releaseMobileDevice)
|
|
226
|
-
blockAll(
|
|
232
|
+
blockAll(executionSuite, 'Selected mobile device is in use.');
|
|
227
233
|
}
|
|
228
|
-
const executable = runnableSuite(
|
|
234
|
+
const executable = runnableSuite(executionSuite, blocked);
|
|
229
235
|
const testFile = path.join(runDir, TEST_FILE);
|
|
230
236
|
const configFile = path.join(runDir, CONFIG_FILE);
|
|
231
237
|
const agentAvailable = runnable.some(agentCase);
|
package/dist/src/i18n.js
CHANGED
|
@@ -68,6 +68,8 @@ const koreanMessages = {
|
|
|
68
68
|
'Both reports must use the same target.': '두 리포트의 실행 대상이 같아야 합니다.',
|
|
69
69
|
'Both reports need a non-empty suite hash.': '두 리포트에 비어 있지 않은 케이스 모음 해시가 있어야 합니다.',
|
|
70
70
|
'Both reports need unique, non-empty case IDs.': '두 리포트의 케이스 ID는 비어 있지 않고 중복되지 않아야 합니다.',
|
|
71
|
+
'The report selection does not match its recorded cases.': '보고서의 선택 범위가 기록된 케이스와 일치하지 않습니다.',
|
|
72
|
+
'Select one or more unique cases from this suite.': '이 케이스 모음에서 중복 없이 하나 이상의 케이스를 선택하세요.',
|
|
71
73
|
'Invalid evidence selection.': '증거 파일 선택이 올바르지 않습니다.',
|
|
72
74
|
'The request is not allowed.': '허용되지 않은 요청입니다.',
|
|
73
75
|
'Invalid request token.': '요청 토큰이 올바르지 않습니다.',
|
package/dist/src/report.js
CHANGED
|
@@ -1,8 +1,21 @@
|
|
|
1
1
|
import { realpathSync } from 'node:fs';
|
|
2
2
|
import path from 'node:path';
|
|
3
3
|
import { expectedChecks } from "./compiler.js";
|
|
4
|
+
import { isReportSelectionValid } from "./case-selection.js";
|
|
4
5
|
import { localized, localizeProductMessage } from "./i18n.js";
|
|
5
6
|
import { targetLabel } from "./targets.js";
|
|
7
|
+
function scopedCases(context) {
|
|
8
|
+
if (!context.selection)
|
|
9
|
+
return context.suite.cases;
|
|
10
|
+
const selected = new Set(context.selection.caseIds);
|
|
11
|
+
const cases = context.suite.cases.filter((testCase) => selected.has(testCase.id));
|
|
12
|
+
if (!isReportSelectionValid(cases, context.selection))
|
|
13
|
+
throw new Error('Report selection does not match the current suite.');
|
|
14
|
+
return cases;
|
|
15
|
+
}
|
|
16
|
+
function reportSelection(context) {
|
|
17
|
+
return context.selection ? { selection: { caseIds: [...context.selection.caseIds], totalCases: context.selection.totalCases } } : {};
|
|
18
|
+
}
|
|
6
19
|
function object(value, label) {
|
|
7
20
|
if (value === null || typeof value !== 'object' || Array.isArray(value))
|
|
8
21
|
throw new Error(`Invalid raw report field: ${label}.`);
|
|
@@ -151,7 +164,8 @@ export function normalizeReport1(raw, context) {
|
|
|
151
164
|
const rawResults = array(run.results, 'run.results');
|
|
152
165
|
const rawErrors = array(run.errors, 'run.errors').map(rawError);
|
|
153
166
|
const sourceName = path.basename(context.testFile);
|
|
154
|
-
const
|
|
167
|
+
const selectedCases = scopedCases(context);
|
|
168
|
+
const knownTitles = new Map(selectedCases.map((testCase) => [`[${testCase.id}] ${testCase.title}`, testCase.id]));
|
|
155
169
|
const mapped = new Map();
|
|
156
170
|
for (const value of rawResults) {
|
|
157
171
|
const result = object(value, 'result');
|
|
@@ -185,7 +199,7 @@ export function normalizeReport1(raw, context) {
|
|
|
185
199
|
mapped.set(id, { status, attempts, ...(skip === undefined ? {} : { reason: string(skip.reason, 'result.skip.reason') }) });
|
|
186
200
|
}
|
|
187
201
|
const errors = rawErrors.map((error) => `${error.code}: ${error.message}`);
|
|
188
|
-
const cases =
|
|
202
|
+
const cases = selectedCases.map((testCase) => {
|
|
189
203
|
const expected = expectedChecks(testCase);
|
|
190
204
|
const result = mapped.get(testCase.id);
|
|
191
205
|
const blockedReason = context.blockedReasons.get(testCase.id);
|
|
@@ -220,7 +234,7 @@ export function normalizeReport1(raw, context) {
|
|
|
220
234
|
return {
|
|
221
235
|
schemaVersion: 'flow-qa/report-1', runId: context.runId, suiteId: context.suite.id,
|
|
222
236
|
suiteHash: context.suiteHash, target: context.target, startedAt: context.startedAt,
|
|
223
|
-
finishedAt: context.finishedAt, status, cases, errors, usage: reportUsage,
|
|
237
|
+
finishedAt: context.finishedAt, status, cases, errors, usage: reportUsage, ...reportSelection(context),
|
|
224
238
|
};
|
|
225
239
|
}
|
|
226
240
|
/** Keep every recorded attempt when the adapter stops a still-running child. */
|
|
@@ -236,13 +250,13 @@ export function failureReport(context, status, error) {
|
|
|
236
250
|
schemaVersion: 'flow-qa/report-1', runId: context.runId, suiteId: context.suite.id,
|
|
237
251
|
suiteHash: context.suiteHash, target: context.target, startedAt: context.startedAt,
|
|
238
252
|
finishedAt: context.finishedAt, status,
|
|
239
|
-
cases: context.
|
|
253
|
+
cases: scopedCases(context).map((testCase) => ({
|
|
240
254
|
id: testCase.id, title: testCase.title,
|
|
241
255
|
status: context.blockedReasons.has(testCase.id) ? 'blocked' : 'error',
|
|
242
256
|
expected: expectedChecks(testCase), attempts: [],
|
|
243
257
|
reason: context.blockedReasons.get(testCase.id) ?? error,
|
|
244
258
|
})),
|
|
245
|
-
errors: [error],
|
|
259
|
+
errors: [error], ...reportSelection(context),
|
|
246
260
|
};
|
|
247
261
|
}
|
|
248
262
|
function safeText(value, locale) {
|
|
@@ -277,6 +291,7 @@ export function renderMarkdown(report, locale = 'en') {
|
|
|
277
291
|
? safe(targetLabel(report.target))
|
|
278
292
|
: label('(URL available in the local JSON report)', '(주소는 로컬 JSON 보고서에서 확인)')}`,
|
|
279
293
|
`- ${label('Status', '상태')}: ${safe(report.status)}`,
|
|
294
|
+
...(report.selection ? [`- ${label('Scope', '범위')}: ${label(`Selected ${report.selection.caseIds.length} / ${report.selection.totalCases} cases; full suite not evaluated.`, `선택 ${report.selection.caseIds.length} / 전체 ${report.selection.totalCases}개 케이스; 전체 케이스 모음은 평가하지 않았습니다.`)}`] : []),
|
|
280
295
|
`- ${label('Started', '시작')}: ${safe(report.startedAt)}`,
|
|
281
296
|
`- ${label('Finished', '종료')}: ${safe(report.finishedAt)}`,
|
|
282
297
|
`- ${label('Results', '결과')}: ${label('passed', '통과')} ${counts.get('passed') ?? 0}, ${label('failed', '실패')} ${counts.get('failed') ?? 0}, ${label('flaky', '재시도 후 통과')} ${counts.get('flaky') ?? 0}, ${label('blocked', '차단')} ${counts.get('blocked') ?? 0}, ${label('skipped', '건너뜀')} ${counts.get('skipped') ?? 0}, ${label('error', '오류')} ${counts.get('error') ?? 0}`,
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { localized } from "./i18n.js";
|
|
2
|
+
import { isReportSelectionValid } from "./case-selection.js";
|
|
2
3
|
import { targetFingerprint, targetLabel } from "./targets.js";
|
|
3
4
|
export const comparisonCategories = ['newly-failed', 'now-passed', 'changed', 'unchanged', 'added', 'missing'];
|
|
4
5
|
export class ComparisonError extends Error {
|
|
@@ -20,6 +21,7 @@ function runSummary(report) {
|
|
|
20
21
|
runId: report.runId, suiteId: report.suiteId, suiteHash: report.suiteHash,
|
|
21
22
|
target: { ...report.target }, startedAt: report.startedAt,
|
|
22
23
|
finishedAt: report.finishedAt, status: report.status,
|
|
24
|
+
...(report.selection ? { selection: { caseIds: [...report.selection.caseIds], totalCases: report.selection.totalCases } } : {}),
|
|
23
25
|
};
|
|
24
26
|
}
|
|
25
27
|
function caseSnapshot(item) {
|
|
@@ -57,9 +59,17 @@ export function compareReports(baseline, current) {
|
|
|
57
59
|
}
|
|
58
60
|
const before = validateCaseIds(baseline);
|
|
59
61
|
const after = validateCaseIds(current);
|
|
62
|
+
if (!isReportSelectionValid(baseline.cases, baseline.selection) || !isReportSelectionValid(current.cases, current.selection)) {
|
|
63
|
+
throw new ComparisonError(409, 'The report selection does not match its recorded cases.');
|
|
64
|
+
}
|
|
60
65
|
const warnings = [];
|
|
61
66
|
if (baseline.suiteHash !== current.suiteHash)
|
|
62
67
|
warnings.push('suite-changed');
|
|
68
|
+
if (baseline.selection || current.selection) {
|
|
69
|
+
warnings.push('partial-run');
|
|
70
|
+
if (before.size !== after.size || [...before.keys()].some((id) => !after.has(id)))
|
|
71
|
+
warnings.push('selection-changed');
|
|
72
|
+
}
|
|
63
73
|
if (Date.parse(current.startedAt) < Date.parse(baseline.startedAt))
|
|
64
74
|
warnings.push('reverse-order');
|
|
65
75
|
if (baseline.status === 'interrupted' || baseline.status === 'error' || current.status === 'interrupted' || current.status === 'error') {
|
|
@@ -106,10 +116,13 @@ export function renderComparisonMarkdown(comparison, locale = 'en') {
|
|
|
106
116
|
`- ${label('Started', '시작')}: ${safe(item.startedAt)}`,
|
|
107
117
|
`- ${label('Finished', '종료')}: ${safe(item.finishedAt)}`,
|
|
108
118
|
`- ${label('Status', '상태')}: ${safe(item.status)}`,
|
|
119
|
+
...(item.selection ? [`- ${label('Scope', '범위')}: ${label(`Selected ${item.selection.caseIds.length} / ${item.selection.totalCases} cases; full suite not evaluated.`, `선택 ${item.selection.caseIds.length} / 전체 ${item.selection.totalCases}개 케이스; 전체 케이스 모음은 평가하지 않았습니다.`)}`] : []),
|
|
109
120
|
'',
|
|
110
121
|
];
|
|
111
122
|
const warningText = {
|
|
112
123
|
'suite-changed': label('Steps or expectations may differ because the suite changed; these outcomes do not prove a regression or fix.', '케이스 모음이 변경되어 단계나 기대 결과가 다를 수 있습니다. 이 결과만으로 회귀나 수정이 입증되지는 않습니다.'),
|
|
124
|
+
'partial-run': label('At least one run evaluated only selected cases; the full suite was not evaluated.', '하나 이상의 실행에서 선택한 케이스만 평가했습니다. 전체 케이스 모음은 평가하지 않았습니다.'),
|
|
125
|
+
'selection-changed': label('The runs selected different case sets; added or missing entries may reflect scope rather than a changed outcome.', '실행마다 선택한 케이스가 다릅니다. 추가 또는 누락 항목은 결과 변화가 아닌 실행 범위 차이일 수 있습니다.'),
|
|
113
126
|
'reverse-order': label('The selected current run started before the baseline run.', '선택한 현재 실행이 기준 실행보다 먼저 시작됐습니다.'),
|
|
114
127
|
'incomplete-run': label('At least one run ended with error or interruption; case results may be incomplete.', '하나 이상의 실행이 오류 또는 중단으로 끝나 케이스 결과가 불완전할 수 있습니다.'),
|
|
115
128
|
'mobile-identity-unverified': label('The recorded app and device settings match, but these reports do not verify that both runs used the same physical device.', '기록된 앱과 기기 설정은 같지만, 두 실행이 같은 실제 기기에서 진행됐는지는 이 리포트로 확인할 수 없습니다.'),
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
const i18n = window.FlowQAI18n;
|
|
5
5
|
const t = i18n.t;
|
|
6
6
|
const names = { scenario: 'nav.scenario', library: 'nav.library', history: 'nav.history', settings: 'nav.settings' };
|
|
7
|
-
const app = { state: null, connected: false, displayTargets: [], draft: null, script: '', savedFile: '', savedHash: '', origin: 'none', sourceFlow: '', boundSource: null, stale: false, dirty: false, busy: false, planning: false, planRequest: 0, stopping: false, report: null, reportMarkdown: '', selectedRun: '', comparison: null, comparisonRequest: 0, comparisonBusy: false, comparisonStatus: 'comparison.selectBoth', view: 'scenario' };
|
|
7
|
+
const app = { state: null, connected: false, displayTargets: [], draft: null, selectedCaseIds: new Set(), script: '', savedFile: '', savedHash: '', origin: 'none', sourceFlow: '', boundSource: null, stale: false, dirty: false, busy: false, planning: false, planRequest: 0, stopping: false, report: null, reportMarkdown: '', selectedRun: '', comparison: null, comparisonRequest: 0, comparisonBusy: false, comparisonStatus: 'comparison.selectBoth', view: 'scenario' };
|
|
8
8
|
|
|
9
9
|
function el(tag, className, value) {
|
|
10
10
|
const node = document.createElement(tag);
|
|
@@ -61,6 +61,37 @@
|
|
|
61
61
|
function targetLabel(target) { return target?.kind === 'mobile' ? `${target.platform === 'ios' ? 'iOS' : 'Android'} · ${target.bundleId} · ${target.device}` : target?.url || ''; }
|
|
62
62
|
function targetDisplay(target) { return `${target?.name || t('app.noTarget')} · ${targetLabel(target)}`; }
|
|
63
63
|
function compatibleTarget() { const target = selectedTarget(); return target && targetPlatform(target) === suitePlatform(); }
|
|
64
|
+
function selectedCases() { return (app.draft?.cases || []).filter((item) => app.selectedCaseIds.has(item.id)); }
|
|
65
|
+
function selectionScope(selection) {
|
|
66
|
+
return selection && Array.isArray(selection.caseIds) && Number.isInteger(selection.totalCases) && selection.caseIds.length < selection.totalCases
|
|
67
|
+
? t('app.selectedScope', { selected: selection.caseIds.length, total: selection.totalCases }) : '';
|
|
68
|
+
}
|
|
69
|
+
function renderCaseSelection() {
|
|
70
|
+
const host = $('case-selection-list'); replace(host);
|
|
71
|
+
for (const item of app.draft?.cases || []) {
|
|
72
|
+
const label = el('label', 'case-selection-item');
|
|
73
|
+
const input = el('input'); input.type = 'checkbox'; input.dataset.caseId = item.id; input.checked = app.selectedCaseIds.has(item.id);
|
|
74
|
+
label.append(input, el('span', '', `${item.id} · ${item.title}`)); host.append(label);
|
|
75
|
+
}
|
|
76
|
+
updateCaseSelection();
|
|
77
|
+
}
|
|
78
|
+
function updateCaseSelection() {
|
|
79
|
+
const total = app.draft?.cases?.length || 0;
|
|
80
|
+
$('case-selection-summary').textContent = t('scenario.selectionSummary', { selected: selectedCases().length, total });
|
|
81
|
+
const disabled = !app.draft || app.busy || !!app.state?.active;
|
|
82
|
+
for (const input of $('case-selection-list').querySelectorAll('input[data-case-id]')) {
|
|
83
|
+
input.checked = app.selectedCaseIds.has(input.dataset.caseId);
|
|
84
|
+
input.disabled = disabled;
|
|
85
|
+
}
|
|
86
|
+
$('select-all-cases').disabled = disabled || selectedCases().length === total;
|
|
87
|
+
$('clear-case-selection').disabled = disabled || !app.selectedCaseIds.size;
|
|
88
|
+
}
|
|
89
|
+
function changeCaseSelection(ids) {
|
|
90
|
+
if (!app.draft || app.busy || app.state?.active) return;
|
|
91
|
+
const next = new Set(ids);
|
|
92
|
+
if (next.size === app.selectedCaseIds.size && [...next].every((id) => app.selectedCaseIds.has(id))) return;
|
|
93
|
+
app.selectedCaseIds = next; $('review-checkbox').checked = false; readiness();
|
|
94
|
+
}
|
|
64
95
|
function canDownloadSavedCases() {
|
|
65
96
|
return Boolean(app.draft && app.script && app.savedFile && app.savedHash && !app.dirty && !app.stale && !app.busy && !hasUnsavedScenario());
|
|
66
97
|
}
|
|
@@ -68,7 +99,8 @@
|
|
|
68
99
|
function readiness() {
|
|
69
100
|
const active = app.state?.active;
|
|
70
101
|
const ready = !!app.draft && !!app.script && !app.stale && !app.dirty;
|
|
71
|
-
$('run-btn').disabled = !app.connected || !ready || !app.savedFile || !app.savedHash || !$('review-checkbox').checked || !compatibleTarget() || !!active || app.busy;
|
|
102
|
+
$('run-btn').disabled = !app.connected || !ready || !app.savedFile || !app.savedHash || !selectedCases().length || !$('review-checkbox').checked || !compatibleTarget() || !!active || app.busy;
|
|
103
|
+
updateCaseSelection();
|
|
72
104
|
$('save-btn').disabled = !ready || app.busy;
|
|
73
105
|
$('suite-download').disabled = !canDownloadSavedCases();
|
|
74
106
|
$('flow-download').disabled = !canDownloadSourceMarkdown();
|
|
@@ -95,12 +127,13 @@
|
|
|
95
127
|
: app.dirty ? 'app.dirty'
|
|
96
128
|
: !ready ? 'app.needValidated'
|
|
97
129
|
: !app.savedFile || !app.savedHash ? 'app.needSaved'
|
|
130
|
+
: !selectedCases().length ? 'app.needCaseSelection'
|
|
98
131
|
: !$('target-select').value ? 'app.needTarget'
|
|
99
132
|
: !compatibleTarget() ? 'app.platformMismatch'
|
|
100
133
|
: !$('review-checkbox').checked ? 'app.needReview' : 'app.ready');
|
|
101
|
-
const cases =
|
|
134
|
+
const cases = selectedCases();
|
|
102
135
|
const blocked = cases.filter((item) => item.questions?.length || (item.steps.some((step) => ['act', 'assertAgent'].includes(step.type)) && !app.state?.models?.agent?.keyReady)).length;
|
|
103
|
-
$('case-readiness').textContent = ready ? t('app.caseReadiness', { total: cases.length, ready: cases.length - blocked, blocked, hint: blocked ? t('app.blockedHint') : '' }) : '';
|
|
136
|
+
$('case-readiness').textContent = ready ? t(cases.length < (app.draft?.cases.length || 0) ? 'app.selectedCaseReadiness' : 'app.caseReadiness', { total: cases.length, ready: cases.length - blocked, blocked, hint: blocked ? t('app.blockedHint') : '' }) : '';
|
|
104
137
|
const pill = $('draft-status');
|
|
105
138
|
pill.textContent = t(app.stale ? 'app.regenerate' : app.dirty ? 'app.revalidate' : ready ? 'app.validated' : 'scenario.noDraft');
|
|
106
139
|
pill.className = 'status-pill' + (ready ? ' ready' : app.stale || app.dirty ? ' dirty' : '');
|
|
@@ -169,7 +202,8 @@
|
|
|
169
202
|
$('suite-editor').value = JSON.stringify(suite, null, 2);
|
|
170
203
|
$('scenario-platform').value = suite.platform || 'web';
|
|
171
204
|
$('review-checkbox').checked = false;
|
|
172
|
-
|
|
205
|
+
app.selectedCaseIds = new Set((suite.cases || []).map((item) => item.id));
|
|
206
|
+
summarize(suite); renderCaseSelection(); readiness();
|
|
173
207
|
}
|
|
174
208
|
function targetOptions() {
|
|
175
209
|
const selected = $('target-select').value;
|
|
@@ -236,12 +270,13 @@
|
|
|
236
270
|
if (!matchingRuns.length) { host.append(el('p', 'empty', t(query ? 'app.historyNoSearchMatches' : 'app.historyNoStatusMatches'))); return; }
|
|
237
271
|
for (const run of matchingRuns) {
|
|
238
272
|
const row = el('div', 'list-row'); const copy = el('div'); copy.append(el('strong', '', `${run.suiteId} · ${i18n.status(run.status)}`), el('small', '', t('app.runMeta', { date: formatDate(run.startedAt), target: targetDisplay(run.target), passed: run.passedCount ?? 0, total: run.caseCount ?? 0, id: run.runId })));
|
|
273
|
+
const scope = selectionScope(run.selection); if (scope) copy.append(el('small', 'selection-scope', scope));
|
|
239
274
|
const button = el('button', 'button', t('app.viewReport')); button.type = 'button'; button.disabled = run.status === 'running' || run.status === 'stopping'; button.addEventListener('click', () => loadReport(run.runId)); row.append(copy, button); host.append(row);
|
|
240
275
|
}
|
|
241
276
|
}
|
|
242
277
|
const comparisonCategories = ['newly-failed', 'now-passed', 'changed', 'unchanged', 'added', 'missing'];
|
|
243
278
|
const comparisonCategoryKeys = { 'newly-failed': 'comparison.newlyFailed', 'now-passed': 'comparison.nowPassed', changed: 'comparison.changed', unchanged: 'comparison.unchanged', added: 'comparison.added', missing: 'comparison.missing' };
|
|
244
|
-
const comparisonWarningKeys = { 'suite-changed': 'comparison.warningSuite', 'reverse-order': 'comparison.warningReverse', 'incomplete-run': 'comparison.warningIncomplete', 'mobile-identity-unverified': 'comparison.warningMobileIdentity' };
|
|
279
|
+
const comparisonWarningKeys = { 'suite-changed': 'comparison.warningSuite', 'reverse-order': 'comparison.warningReverse', 'incomplete-run': 'comparison.warningIncomplete', 'mobile-identity-unverified': 'comparison.warningMobileIdentity', 'partial-run': 'comparison.warningPartial', 'selection-changed': 'comparison.warningSelectionChanged' };
|
|
245
280
|
function comparisonPair() { return { baseline: $('compare-baseline').value, current: $('compare-current').value }; }
|
|
246
281
|
function comparisonValid() { const { baseline, current } = comparisonPair(); return !!baseline && !!current && baseline !== current; }
|
|
247
282
|
function comparisonStatus(key, params = {}, error = false) {
|
|
@@ -268,7 +303,7 @@
|
|
|
268
303
|
const select = $(id); const selected = select.value;
|
|
269
304
|
select.replaceChildren(el('option', '', t('comparison.choose'))); select.firstChild.value = '';
|
|
270
305
|
for (const run of runs) {
|
|
271
|
-
const option = el('option', '', t('comparison.runLabel', { id: run.runId, shortId: run.runId.slice(-8), date: formatDate(run.startedAt), suite: run.suiteId, target: targetDisplay(run.target), status: i18n.status(run.status) }));
|
|
306
|
+
const option = el('option', '', `${t('comparison.runLabel', { id: run.runId, shortId: run.runId.slice(-8), date: formatDate(run.startedAt), suite: run.suiteId, target: targetDisplay(run.target), status: i18n.status(run.status) })}${selectionScope(run.selection) ? ` · ${selectionScope(run.selection)}` : ''}`);
|
|
272
307
|
option.value = run.runId; select.append(option);
|
|
273
308
|
}
|
|
274
309
|
select.value = runs.some((run) => run.runId === selected) ? selected : '';
|
|
@@ -297,7 +332,7 @@
|
|
|
297
332
|
const warnings = $('comparison-warnings'); replace(warnings);
|
|
298
333
|
for (const code of result.warnings || []) warnings.append(el('p', 'comparison-warning', t(comparisonWarningKeys[code] || 'comparison.warningUnknown', { code })));
|
|
299
334
|
const summary = $('comparison-summary'); replace(summary);
|
|
300
|
-
const meta = el('p', 'comparison-meta', `${result.baseline.runId} · ${formatDate(result.baseline.startedAt)} → ${result.current.runId} · ${formatDate(result.current.startedAt)}`); summary.append(meta);
|
|
335
|
+
const meta = el('p', 'comparison-meta', `${result.baseline.runId} · ${formatDate(result.baseline.startedAt)}${selectionScope(result.baseline.selection) ? ` · ${selectionScope(result.baseline.selection)}` : ''} → ${result.current.runId} · ${formatDate(result.current.startedAt)}${selectionScope(result.current.selection) ? ` · ${selectionScope(result.current.selection)}` : ''}`); summary.append(meta);
|
|
301
336
|
const counts = el('div', 'comparison-counts');
|
|
302
337
|
for (const category of comparisonCategories) {
|
|
303
338
|
const item = el('div', 'comparison-count');
|
|
@@ -423,7 +458,7 @@
|
|
|
423
458
|
noticeKey('app.compileComplete');
|
|
424
459
|
}); }
|
|
425
460
|
async function save() { await work(async () => { if (!app.draft || app.dirty || app.stale) throw localizedError('app.needValidated'); const body = { suite: app.draft }; if (!app.boundSource && app.sourceFlow && (app.origin === 'imported' ? editableFlow(app.sourceFlow) === $('flow-input').value : app.sourceFlow === $('flow-input').value.trim())) body.flow = app.sourceFlow; const result = await api('/api/save', body); const saved = await api(`/api/suite?file=${encodeURIComponent(result.file)}`); $('flow-input').value = saved.flow || ''; setDraft(saved.suite, saved.script, 'library', saved.flow || '', result.file, saved.suiteHash || ''); noticeKey('app.savedSuite', { file: result.file }); await refresh(false); }); }
|
|
426
|
-
async function run() { await work(async () => { if (!app.draft || !app.script || !app.savedFile || !app.savedHash || app.dirty || app.stale || !$('review-checkbox').checked || !compatibleTarget()) throw localizedError('app.runNotReady'); const target = selectedTarget(); const
|
|
461
|
+
async function run() { await work(async () => { if (!app.draft || !app.script || !app.savedFile || !app.savedHash || app.dirty || app.stale || !selectedCases().length || !$('review-checkbox').checked || !compatibleTarget()) throw localizedError('app.runNotReady'); const target = selectedTarget(); const body = { suite: app.savedFile, suiteHash: app.savedHash, targetName: target.name, targetFingerprint: target.targetFingerprint }; if (selectedCases().length < app.draft.cases.length) body.caseIds = selectedCases().map((item) => item.id); const result = await api('/api/run', body); noticeKey('app.runStarted', { id: result.operationId }); await refresh(false); }); }
|
|
427
462
|
async function stop() {
|
|
428
463
|
if (app.stopping || !app.state?.active) return;
|
|
429
464
|
if (app.state.active.kind === 'plan' && app.planning) app.planRequest += 1;
|
|
@@ -447,7 +482,7 @@
|
|
|
447
482
|
const report = app.report; if (!report) return;
|
|
448
483
|
$('report-heading').textContent = `${report.runId} · ${i18n.status(report.status)} · ${targetDisplay(report.target)}`;
|
|
449
484
|
const host = $('report-content'); replace(host);
|
|
450
|
-
const meta = el('div', 'report-meta'); for (const value of [t('app.reportStatus', { status: i18n.status(report.status) }), t('app.reportTarget', { target: targetDisplay(report.target) }), t('app.reportStarted', { date: formatDate(report.startedAt) }), t('app.reportFinished', { date: formatDate(report.finishedAt) })]) meta.append(el('span', '', value)); host.append(meta);
|
|
485
|
+
const meta = el('div', 'report-meta'); for (const value of [t('app.reportStatus', { status: i18n.status(report.status) }), t('app.reportTarget', { target: targetDisplay(report.target) }), t('app.reportStarted', { date: formatDate(report.startedAt) }), t('app.reportFinished', { date: formatDate(report.finishedAt) })]) meta.append(el('span', '', value)); const scope = selectionScope(report.selection); if (scope) meta.append(el('span', 'selection-scope', scope)); host.append(meta);
|
|
451
486
|
if (report.usage) host.append(el('p', 'hint', t('app.usage', { tokens: report.usage.modelTokens ?? t('app.unavailable'), cost: typeof report.usage.estimatedCostUsd === 'number' ? `$${report.usage.estimatedCostUsd.toFixed(4)}` : t('app.unavailable') })));
|
|
452
487
|
if (report.errors?.length) host.append(el('p', 'report-errors', report.errors.join('\n')));
|
|
453
488
|
for (const item of report.cases || []) {
|
|
@@ -497,7 +532,7 @@
|
|
|
497
532
|
const input = form?.elements.namedItem(focus.name);
|
|
498
533
|
if (input) { input.focus(); if (Number.isInteger(focus.start)) input.setSelectionRange(focus.start, focus.end); }
|
|
499
534
|
}
|
|
500
|
-
summarize(app.draft, true);
|
|
535
|
+
summarize(app.draft, true); renderCaseSelection();
|
|
501
536
|
renderActive(); targetDescription(); renderLibrary(); renderHistory(); renderComparisonOptions(); renderComparison(); renderReport();
|
|
502
537
|
if (app.comparisonStatus) comparisonStatus(app.comparisonStatus, app.comparisonStatusParams || {}, app.comparisonStatus === 'comparison.error');
|
|
503
538
|
app.reportMarkdown = '';
|
|
@@ -525,10 +560,10 @@
|
|
|
525
560
|
const json = $('suite-editor').value;
|
|
526
561
|
const unsavedJson = json.trim() && (!app.savedFile || json !== JSON.stringify(app.draft, null, 2));
|
|
527
562
|
if (unsavedJson && !confirmScenarioReplacement('app.confirmNewDraft')) return;
|
|
528
|
-
app.draft = null; app.script = ''; app.savedFile = ''; app.savedHash = '';
|
|
563
|
+
app.draft = null; app.selectedCaseIds = new Set(); app.script = ''; app.savedFile = ''; app.savedHash = '';
|
|
529
564
|
app.origin = 'none'; app.sourceFlow = ''; app.boundSource = null; app.stale = false; app.dirty = false;
|
|
530
565
|
$('suite-editor').value = ''; $('review-checkbox').checked = false;
|
|
531
|
-
summarize(null); readiness();
|
|
566
|
+
summarize(null); renderCaseSelection(); readiness();
|
|
532
567
|
noticeKey('app.newDraftNotice');
|
|
533
568
|
$('suite-editor').focus();
|
|
534
569
|
});
|
|
@@ -536,6 +571,9 @@
|
|
|
536
571
|
$('scenario-platform').addEventListener('change', () => { app.planRequest += 1; if (app.draft) { app.stale = true; app.savedHash = ''; } $('review-checkbox').checked = false; readiness(); });
|
|
537
572
|
$('suite-editor').addEventListener('input', () => { app.dirty = true; app.script = ''; app.savedFile = ''; app.savedHash = ''; if (app.origin === 'none') app.origin = 'manual'; $('review-checkbox').checked = false; summarize(null); readiness(); });
|
|
538
573
|
$('review-checkbox').addEventListener('change', readiness); $('target-select').addEventListener('change', () => { $('review-checkbox').checked = false; targetDescription(); });
|
|
574
|
+
$('case-selection-list').addEventListener('change', (event) => { const input = event.target.closest('input[data-case-id]'); if (!input) return; const next = new Set(app.selectedCaseIds); if (input.checked) next.add(input.dataset.caseId); else next.delete(input.dataset.caseId); changeCaseSelection(next); });
|
|
575
|
+
$('select-all-cases').addEventListener('click', () => changeCaseSelection((app.draft?.cases || []).map((item) => item.id)));
|
|
576
|
+
$('clear-case-selection').addEventListener('click', () => changeCaseSelection([]));
|
|
539
577
|
$('plan-btn').addEventListener('click', generate); $('prompt-btn').addEventListener('click', prompt); $('compile-btn').addEventListener('click', compile); $('save-btn').addEventListener('click', save); $('run-btn').addEventListener('click', run); $('stop-btn').addEventListener('click', stop);
|
|
540
578
|
$('import-btn').addEventListener('click', importFiles);
|
|
541
579
|
$('global-stop-btn').addEventListener('click', stop);
|
|
@@ -56,6 +56,12 @@
|
|
|
56
56
|
'scenario.newDraft': ['New JSON draft', '새 JSON 초안'],
|
|
57
57
|
'scenario.compile': ['Validate JSON · Generate script', 'JSON 검증 · 스크립트 생성'],
|
|
58
58
|
'scenario.script': ['Run script', '실행 스크립트'],
|
|
59
|
+
'scenario.fullSuiteScript': ['Script preview and download include the full suite, regardless of the run selection.', '스크립트 미리보기와 다운로드에는 실행 선택과 관계없이 전체 스위트가 포함됩니다.'],
|
|
60
|
+
'scenario.selectionTitle': ['Cases to run', '실행할 케이스'],
|
|
61
|
+
'scenario.selectionSummary': ['{selected} selected / {total} total', '선택 {selected}개 / 전체 {total}개'],
|
|
62
|
+
'scenario.selectionHint': ['Each selected case must establish its own starting state. Excluded cases are not assessed.', '선택한 각 케이스는 자체 시작 상태를 설정해야 합니다. 제외한 케이스는 평가하지 않습니다.'],
|
|
63
|
+
'scenario.selectAllCases': ['Select all', '전체 선택'],
|
|
64
|
+
'scenario.clearSelection': ['Clear selection', '선택 해제'],
|
|
59
65
|
'scenario.downloadScript': ['Download script ↓', '스크립트 다운로드 ↓'],
|
|
60
66
|
'scenario.scriptPlaceholder': ['The script for validated cases will appear here.', '검증된 케이스의 스크립트가 여기에 표시됩니다.'],
|
|
61
67
|
'scenario.runTitle': ['Review, then run', '검토 후 실행'],
|
|
@@ -112,6 +118,8 @@
|
|
|
112
118
|
'comparison.warningReverse': ['The selected current run started before the baseline run.', '선택한 현재 실행이 기준 실행보다 먼저 시작됐습니다.'],
|
|
113
119
|
'comparison.warningIncomplete': ['At least one run is incomplete. Interpret case outcomes with its detailed report.', '완료되지 않은 실행이 있습니다. 상세 보고서와 함께 결과를 해석하세요.'],
|
|
114
120
|
'comparison.warningMobileIdentity': ['Mobile comparison matches recorded app and device settings. It does not verify that both runs used the same physical device.', '모바일 비교는 기록된 앱과 기기 설정을 기준으로 합니다. 두 실행이 같은 실제 기기를 사용했는지는 확인하지 않습니다.'],
|
|
121
|
+
'comparison.warningPartial': ['At least one run assessed only selected cases. Missing outcomes do not mean those cases passed.', '하나 이상의 실행에서 선택한 케이스만 평가했습니다. 결과가 없는 케이스를 통과로 해석할 수 없습니다.'],
|
|
122
|
+
'comparison.warningSelectionChanged': ['The runs selected different case sets. Compare only outcomes for cases assessed in both runs.', '두 실행에서 선택한 케이스가 다릅니다. 두 실행 모두에서 평가한 케이스의 결과만 비교하세요.'],
|
|
115
123
|
'comparison.warningUnknown': ['Comparison warning: {code}', '비교 주의: {code}'],
|
|
116
124
|
'comparison.pair': ['{baseline} → {current}', '{baseline} → {current}'],
|
|
117
125
|
'comparison.notPresent': ['not present', '없음'],
|
|
@@ -192,7 +200,10 @@
|
|
|
192
200
|
'app.needTarget': ['Select a run target.', '실행 대상을 선택하세요.'],
|
|
193
201
|
'app.platformMismatch': ['The case platform and target type must match.', '케이스 플랫폼과 대상 유형이 일치해야 합니다.'],
|
|
194
202
|
'app.needReview': ['Review the contents, then select the checkbox.', '내용 검토 후 확인란을 선택하세요.'],
|
|
203
|
+
'app.needCaseSelection': ['Select at least one case to run.', '실행할 케이스를 하나 이상 선택하세요.'],
|
|
195
204
|
'app.ready': ['Ready to run.', '실행할 준비가 됐습니다.'],
|
|
205
|
+
'app.selectedScope': ['Selected {selected}/{total} cases · excluded cases not assessed', '선택 {selected}/{total}개 · 제외한 케이스는 평가하지 않음'],
|
|
206
|
+
'app.selectedCaseReadiness': ['{total} selected · {ready} runnable · {blocked} blocked before run{hint}', '선택 {total}개 · 실행 가능 {ready}개 · 사전 차단 {blocked}개{hint}'],
|
|
196
207
|
'app.caseReadiness': ['{total} total · {ready} runnable · {blocked} blocked before run{hint}', '전체 {total}개 · 실행 가능 {ready}개 · 사전 차단 {blocked}개{hint}'],
|
|
197
208
|
'app.blockedHint': [' — Check open questions or run agent settings.', ' — 확인 질문이나 실행 모델 설정을 확인하세요.'],
|
|
198
209
|
'app.regenerate': ['Regenerate required', '재생성 필요'],
|
|
@@ -72,6 +72,7 @@
|
|
|
72
72
|
<textarea id="suite-editor" class="code-editor" rows="13" spellcheck="false" placeholder="Load a local demo case from the library or enter suite JSON." data-i18n-placeholder="scenario.suitePlaceholder"></textarea>
|
|
73
73
|
<div class="actions"><button id="new-draft-btn" type="button" class="button quiet" data-i18n="scenario.newDraft">New JSON draft</button><button id="compile-btn" type="button" class="button" data-i18n="scenario.compile">Validate JSON · Generate script</button></div>
|
|
74
74
|
<div class="script-head"><h3 data-i18n="scenario.script">Run script</h3><button type="button" id="script-download" class="text-button" disabled data-i18n="scenario.downloadScript">Download script ↓</button></div>
|
|
75
|
+
<p class="hint" data-i18n="scenario.fullSuiteScript">Script preview and download include the full suite, regardless of the run selection.</p>
|
|
75
76
|
<pre id="script-preview" class="script-preview" data-i18n="scenario.scriptPlaceholder">The script for validated cases will appear here.</pre>
|
|
76
77
|
</div>
|
|
77
78
|
</details>
|
|
@@ -81,7 +82,7 @@
|
|
|
81
82
|
<div class="section-head"><span class="section-number">03</span><div><h2 id="run-title" data-i18n="scenario.runTitle">Review, then run</h2><p data-i18n="scenario.runHelp">Select a target and confirm you have reviewed the cases.</p></div></div>
|
|
82
83
|
<div class="execution-grid">
|
|
83
84
|
<div class="target-field"><label for="target-select" class="field-label" data-i18n="scenario.target">Run target</label><select id="target-select"><option value="" data-i18n="scenario.selectTarget">Select a target</option></select><p id="target-description" class="hint" data-i18n="scenario.targetHint">You can add a target in Connections.</p><p id="mobile-setup-hint" class="hint" hidden data-i18n="scenario.mobileSetup">For iOS, prepare Xcode, a simulator or signed device, and app signing. For Android, prepare the Android SDK and an emulator or authorized device. Configuration alone does not verify device access.</p></div>
|
|
84
|
-
<div class="execution-confirm"><p id="case-readiness" class="hint"></p><label class="review-line"><input type="checkbox" id="review-checkbox"><span data-i18n="scenario.reviewConfirm">I have reviewed the requirements, case steps, and expected results.</span></label><div class="run-bar"><span id="run-readiness" data-i18n="scenario.runInitial">Validated cases and a run target are required.</span><div class="actions"><button type="button" id="stop-btn" class="button quiet" hidden data-i18n="scenario.stop">Stop run</button><button type="button" id="run-btn" class="button primary" disabled><span data-i18n="scenario.run">Run QA</span><span aria-hidden="true">→</span></button></div></div></div>
|
|
85
|
+
<div class="execution-confirm"><div class="case-selection"><div class="case-selection-head"><h3 id="case-selection-title" data-i18n="scenario.selectionTitle">Cases to run</h3><span id="case-selection-summary" role="status" aria-live="polite"></span></div><p class="hint" data-i18n="scenario.selectionHint">Each selected case must establish its own starting state. Excluded cases are not assessed.</p><div class="actions"><button type="button" id="select-all-cases" class="button quiet" data-i18n="scenario.selectAllCases">Select all</button><button type="button" id="clear-case-selection" class="button quiet" data-i18n="scenario.clearSelection">Clear selection</button></div><div id="case-selection-list" role="group" aria-labelledby="case-selection-title"></div></div><p id="case-readiness" class="hint"></p><label class="review-line"><input type="checkbox" id="review-checkbox"><span data-i18n="scenario.reviewConfirm">I have reviewed the requirements, case steps, and expected results.</span></label><div class="run-bar"><span id="run-readiness" data-i18n="scenario.runInitial">Validated cases and a run target are required.</span><div class="actions"><button type="button" id="stop-btn" class="button quiet" hidden data-i18n="scenario.stop">Stop run</button><button type="button" id="run-btn" class="button primary" disabled><span data-i18n="scenario.run">Run QA</span><span aria-hidden="true">→</span></button></div></div></div>
|
|
85
86
|
</div>
|
|
86
87
|
</section>
|
|
87
88
|
</div>
|
|
@@ -126,6 +126,17 @@ button:focus-visible, input:focus-visible, select:focus-visible, textarea:focus-
|
|
|
126
126
|
.target-field { min-width: 0; }
|
|
127
127
|
.target-field select { min-height: 45px; }
|
|
128
128
|
.execution-confirm { display: flex; flex-direction: column; min-width: 0; }
|
|
129
|
+
.case-selection { min-width: 0; margin-bottom: 16px; padding-bottom: 16px; border-bottom: 1px solid var(--line); }
|
|
130
|
+
.case-selection-head { display: flex; align-items: baseline; justify-content: space-between; flex-wrap: wrap; gap: 4px 12px; }
|
|
131
|
+
.case-selection-head h3 { margin: 0; font-size: 14px; }
|
|
132
|
+
#case-selection-summary { color: var(--muted); font-size: 12px; }
|
|
133
|
+
.case-selection .actions { margin: 12px 0; }
|
|
134
|
+
#case-selection-list { display: grid; gap: 0; max-height: 250px; overflow-y: auto; border: 1px solid var(--line); border-radius: var(--radius-control); }
|
|
135
|
+
.case-selection-item { display: flex; align-items: flex-start; gap: 10px; min-width: 0; padding: 10px 12px; border-bottom: 1px solid var(--line); cursor: pointer; font-size: 13px; }
|
|
136
|
+
.case-selection-item:last-child { border-bottom: 0; }
|
|
137
|
+
.case-selection-item input { width: 18px; height: 18px; flex: none; margin: 1px 0 0; accent-color: var(--accent-solid); }
|
|
138
|
+
.case-selection-item span { min-width: 0; overflow-wrap: anywhere; }
|
|
139
|
+
.selection-scope { color: var(--accent-text) !important; font-weight: 700; }
|
|
129
140
|
.review-line { display: flex; align-items: flex-start; gap: 10px; font-size: 13px; line-height: 1.55; cursor: pointer; }
|
|
130
141
|
.review-line input { width: 18px; height: 18px; flex: none; margin: 2px 0 0; accent-color: var(--accent-solid); }
|
|
131
142
|
.run-bar { display: flex; align-items: end; justify-content: space-between; flex-wrap: wrap; gap: 12px; margin-top: auto; padding-top: 18px; }
|
package/dist/src/ui/server.js
CHANGED
|
@@ -8,6 +8,7 @@ import { z } from 'zod';
|
|
|
8
8
|
import { createPreferenceScript } from '@hyeonse/design-system/preferences';
|
|
9
9
|
import { chromium } from 'playwright';
|
|
10
10
|
import { compileSuite } from "../compiler.js";
|
|
11
|
+
import { CaseSelectionError, isReportSelectionValid, selectCases } from "../case-selection.js";
|
|
11
12
|
import { runSuite } from "../engine.js";
|
|
12
13
|
import { localeFromAcceptLanguage, localizeProductMessage } from "../i18n.js";
|
|
13
14
|
import { EvidenceError, readEvidence } from "./evidence.js";
|
|
@@ -38,7 +39,8 @@ const reportEnvelope = z.object({
|
|
|
38
39
|
cases: z.array(z.object({ id: z.string(), title: z.string(), status: z.enum(['passed', 'failed', 'flaky', 'blocked', 'skipped', 'error']),
|
|
39
40
|
expected: z.array(z.string()), attempts: z.array(z.unknown()), reason: z.string().optional() }).passthrough()),
|
|
40
41
|
errors: z.array(z.string()), usage: z.object({ modelTokens: z.number(), modelCachedTokens: z.number().optional(), estimatedCostUsd: z.number().optional() }).optional(),
|
|
41
|
-
|
|
42
|
+
selection: z.unknown().optional(),
|
|
43
|
+
}).passthrough().refine((report) => isReportSelectionValid(report.cases, report.selection));
|
|
42
44
|
const PUBLIC_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), 'public');
|
|
43
45
|
const STATIC_FILES = {
|
|
44
46
|
'/': { file: path.join(PUBLIC_DIR, 'index.html'), type: 'text/html; charset=utf-8' },
|
|
@@ -72,7 +74,7 @@ const SCHEMA_FIELDS = new Set([
|
|
|
72
74
|
'requirementIds', 'preconditions', 'questions', 'steps', 'type', 'path', 'instruction',
|
|
73
75
|
'locator', 'by', 'value', 'name', 'visible', 'expected', 'match', 'flow', 'role', 'url',
|
|
74
76
|
'description', 'baseURL', 'model', 'apiKeyEnv', 'clear', 'targetName', 'targetURL', 'targetFingerprint', 'suiteHash',
|
|
75
|
-
'kind', 'platform', 'bundleId', 'device', 'direction',
|
|
77
|
+
'kind', 'platform', 'bundleId', 'device', 'direction', 'caseIds',
|
|
76
78
|
]);
|
|
77
79
|
const KNOWN_SCHEMA_MESSAGES = new Map([
|
|
78
80
|
['IDs must be unique.', 'IDs must be unique.'],
|
|
@@ -402,7 +404,7 @@ export async function startUIServer(options) {
|
|
|
402
404
|
history.push({ runId: item.runId, suiteId: item.suiteId, status: item.status, startedAt: item.startedAt,
|
|
403
405
|
finishedAt: item.finishedAt, caseCount: item.cases.length,
|
|
404
406
|
passedCount: item.cases.filter((entry) => entry.status === 'passed' || entry.status === 'flaky').length,
|
|
405
|
-
target: item.target });
|
|
407
|
+
target: item.target, ...(item.selection ? { selection: item.selection } : {}) });
|
|
406
408
|
}
|
|
407
409
|
catch { /* incomplete or unsafe run */ }
|
|
408
410
|
}
|
|
@@ -708,14 +710,24 @@ export async function startUIServer(options) {
|
|
|
708
710
|
if (url.pathname === '/api/run') {
|
|
709
711
|
enterMutation();
|
|
710
712
|
try {
|
|
711
|
-
const { suite: file, suiteHash, targetName, targetURL, targetFingerprint: reviewedFingerprint } = parse(z.object({
|
|
713
|
+
const { suite: file, suiteHash, targetName, targetURL, targetFingerprint: reviewedFingerprint, caseIds } = parse(z.object({
|
|
712
714
|
suite: z.string(), suiteHash: z.string().regex(/^[a-f0-9]{64}$/), targetName: idSchema,
|
|
713
715
|
targetURL: z.string().optional(), targetFingerprint: z.string().regex(/^[a-f0-9]{64}$/).optional(),
|
|
716
|
+
caseIds: z.array(idSchema).min(1).max(100).optional(),
|
|
714
717
|
}).strict(), body);
|
|
715
718
|
const selected = await getSuite(file, true);
|
|
716
719
|
if (selected.suiteHash !== suiteHash)
|
|
717
720
|
throw new HTTPError(409, 'The reviewed case file changed. Open it again to review.');
|
|
718
721
|
const { suite } = selected;
|
|
722
|
+
let selectedIds;
|
|
723
|
+
try {
|
|
724
|
+
selectedIds = selectCases(suite, caseIds).selection?.caseIds;
|
|
725
|
+
}
|
|
726
|
+
catch (error) {
|
|
727
|
+
if (error instanceof CaseSelectionError)
|
|
728
|
+
invalid('Select one or more unique cases from this suite.');
|
|
729
|
+
throw error;
|
|
730
|
+
}
|
|
719
731
|
const settings = await config();
|
|
720
732
|
const target = settings.targets[targetName];
|
|
721
733
|
if (!target)
|
|
@@ -739,7 +751,8 @@ export async function startUIServer(options) {
|
|
|
739
751
|
lastError = undefined;
|
|
740
752
|
const suiteSnapshot = structuredClone(suite);
|
|
741
753
|
const configSnapshot = structuredClone(settings);
|
|
742
|
-
runTask = Promise.resolve().then(() => run({ workspace, suite: suiteSnapshot, config: configSnapshot, targetName, signal: abort.signal
|
|
754
|
+
runTask = Promise.resolve().then(() => run({ workspace, suite: suiteSnapshot, config: configSnapshot, targetName, signal: abort.signal,
|
|
755
|
+
...(selectedIds ? { caseIds: [...selectedIds] } : {}) }))
|
|
743
756
|
.then(() => undefined)
|
|
744
757
|
.catch(() => { lastError = 'Could not complete the run. Check the local records.'; })
|
|
745
758
|
.finally(() => { active = null; controller = undefined; busy = false; runTask = undefined; });
|
package/docs/architecture.md
CHANGED
|
@@ -22,7 +22,9 @@ flowchart LR
|
|
|
22
22
|
|
|
23
23
|
**Only a defined vocabulary compiles.** The schema checks requirement references, assertions, step types, and inputs. It does not execute arbitrary code or shell commands from model output. Cases with unresolved `questions`, or steps that need an unavailable agent model, are blocked. `preconditions` describe required state; they are not an automatic setup contract.
|
|
24
24
|
|
|
25
|
-
**Source and evidence stay linked.** Planning records the source flow's SHA-256 hash in the suite. `validate`, `generate`, and `run` check it again and stop when the flow changed. Each execution writes a new directory with a suite snapshot, generated tests, raw engine output, and Markdown/JSON reports. Previous evidence is not reused as a current success. `compare` permits a direct comparison only for the same suite hash and target identity (web URL or native platform/app/device); it cannot establish that external data was unchanged.
|
|
25
|
+
**Source and evidence stay linked.** Planning records the source flow's SHA-256 hash in the suite. `validate`, `generate`, and `run` check it again and stop when the flow changed. Each execution writes a new directory with a suite snapshot, generated tests, raw engine output, and Markdown/JSON reports. Previous evidence is not reused as a current success. CLI `compare` permits a direct comparison only for the same suite ID, suite hash, case selection, and target identity (web URL or native platform/app/device); it cannot establish that external data was unchanged.
|
|
26
|
+
|
|
27
|
+
**Selection changes execution scope, not source identity.** The full suite and source are validated and the full original snapshot/hash remain in each run directory. Selected cases and their referenced requirements form the executable subset. Partial reports carry explicit selection metadata and contain only selected results, including blocked, error, and interrupted results. UI comparisons flag partial/differing scopes; missing results never imply a pass. Selection changes require fresh UI review and cannot bypass the target or file fingerprint checks.
|
|
26
28
|
|
|
27
29
|
**Targets can be configured after authoring.** A suite's optional platform distinguishes web from mobile. Omitted platform preserves legacy web suites and their hashes. Native app IDs and explicit device names belong to target settings, not generated code from the planner. UI execution checks a fingerprint of the reviewed target identity. Mobile runtime readiness is checked before device execution; configuration alone does not prove connectivity.
|
|
28
30
|
|
package/docs/mobile.md
CHANGED
|
@@ -9,7 +9,7 @@ Choose **Mobile** on the scenario screen, describe the flow, and generate or dow
|
|
|
9
9
|
With the installed CLI:
|
|
10
10
|
|
|
11
11
|
```sh
|
|
12
|
-
npm install -g @ihyeon/flow-qa@0.
|
|
12
|
+
npm install -g @ihyeon/flow-qa@0.4.0
|
|
13
13
|
flow-qa init ./my-qa
|
|
14
14
|
# Write your mobile flow in my-qa/flows/my-flow.md first.
|
|
15
15
|
flow-qa plan flows/my-flow.md --platform mobile --prompt-only --workspace ./my-qa
|
package/docs/releases.md
CHANGED
|
@@ -2,6 +2,17 @@
|
|
|
2
2
|
|
|
3
3
|
The alpha entries below are historical release records. They document earlier package versions and checks.
|
|
4
4
|
|
|
5
|
+
## 0.4.0: run selected cases
|
|
6
|
+
|
|
7
|
+
Adds case selection to the execution panel and repeated `--case <id>` flags to CLI runs. Selecting or clearing cases resets UI review, and execution still requires the reviewed file and target fingerprints. Unknown, duplicate, and empty selections are rejected. Selection is preserved across navigation, language changes, and polling, while a new draft starts with all cases selected.
|
|
8
|
+
|
|
9
|
+
Each run retains the full original suite snapshot/hash and verifies its source, while compiling and assessing only the chosen cases. Partial reports add explicit `selection` metadata to `flow-qa/report-1`; unselected cases are neither executed nor reported as passed/skipped. Selected blocked/error/interrupted cases retain their normal meaning. History, report downloads, and comparisons display the partial scope; comparisons warn when selected case sets differ. Existing full runs and their report shape are preserved.
|
|
10
|
+
|
|
11
|
+
Locally verified on macOS with Node.js 24.22.1 on 2026-11-03: 115 tests, type checking, UI syntax and design-system audit, and existing demo/UI checks passed. The dedicated Chromium smoke ran two selected cases successfully, confirmed zero requests from an excluded sentinel case, and kept a selected unresolved-question case blocked without attempts. It covered review invalidation, bulk selection, navigation/language preservation, 390px layout, original snapshot/hash preservation, and report/comparison scope. A fresh tarball install executed two full runs plus a one-case partial run (seven passing browser cases) and compared their reports. No model calls or native-device execution were used.
|
|
12
|
+
|
|
13
|
+
- Package: [`@ihyeon/flow-qa@0.4.0`](https://www.npmjs.com/package/@ihyeon/flow-qa/v/0.4.0)
|
|
14
|
+
- Release: [GitHub 0.4.0](https://github.com/hyeonse-swing/flow-qa/releases/tag/v0.4.0)
|
|
15
|
+
|
|
5
16
|
## 0.3.0: compare recorded runs
|
|
6
17
|
|
|
7
18
|
Adds a two-run comparison workspace to Run history. Select a baseline and current report, swap their direction, filter by case transition, open either detailed report, and export the complete comparison as Markdown or JSON. Cases are matched by ID, with distinct categories for passed-to-failed, failed-to-passed, other changes, equal statuses, added, and missing. Flaky, blocked, skipped, and execution-error results retain their meaning.
|
|
@@ -111,7 +122,7 @@ Locally verified on macOS with Node.js 24.14.1 on 2026-10-12: 89 tests, type che
|
|
|
111
122
|
|
|
112
123
|
A push to `main` publishes only when `package.json` changes to a new plain `X.Y.Z` version. Pull requests and pushes without a version change run verification without publishing. The package and lockfile versions must agree; the version must increase, and registry errors stop the release. Rerunning a successful publication skips the existing immutable npm version.
|
|
113
124
|
|
|
114
|
-
The `publish` job in [ci.yml](../.github/workflows/ci.yml) waits for the full verification matrix on Ubuntu and macOS, then publishes through npm trusted publishing (OIDC). It uses the `latest` tag, requires the registry's `gitHead` to match the workflow commit, and checks the published package with a fresh consumer installation, two
|
|
125
|
+
The `publish` job in [ci.yml](../.github/workflows/ci.yml) waits for the full verification matrix on Ubuntu and macOS, then publishes through npm trusted publishing (OIDC). It uses the `latest` tag, requires the registry's `gitHead` to match the workflow commit, and checks the published package with a fresh consumer installation, two full Chromium demo runs plus a one-case selected run (seven cases), and comparison of their saved reports. The commit check also applies when retrying an already published version. No npm token is stored in GitHub secrets. After the published package passes its consumer smoke check, a separate job creates the matching GitHub tag and release at the exact verified commit. It preserves existing releases and rejects a tag pointing to another commit. Rerunning an older release does not replace a newer latest release.
|
|
115
126
|
|
|
116
127
|
For the next release, update the release notes and current installation examples, then bump the version before committing and pushing:
|
|
117
128
|
|
package/docs/ui.md
CHANGED
|
@@ -7,7 +7,7 @@ Describe a user flow, review cases, and configure a target when ready to run. We
|
|
|
7
7
|
Node.js 24.x (24.22.1 or later) or 26.12.1 or later is required. Node 25 is unsupported. Upgrade the Node runtime and restart any running UI process; updating Flow QA alone does not patch Node. Install Flow QA and start the web demo:
|
|
8
8
|
|
|
9
9
|
```sh
|
|
10
|
-
npm install -g @ihyeon/flow-qa@0.
|
|
10
|
+
npm install -g @ihyeon/flow-qa@0.4.0
|
|
11
11
|
flow-qa browser install
|
|
12
12
|
flow-qa ui --demo
|
|
13
13
|
```
|
|
@@ -46,6 +46,16 @@ From version `0.2.2`, the UI requests the browser's leave-page confirmation when
|
|
|
46
46
|
|
|
47
47
|
From version `0.2.4`, loading a suite from the library or importing valid case files also asks before replacing unsaved scenario or case content. **New JSON draft** asks before clearing unsaved JSON and keeps the scenario text. Cancel keeps the current content and selected import files. Unchanged saved cases and ordinary navigation between screens do not prompt; invalid imports still show their validation error without replacing anything.
|
|
48
48
|
|
|
49
|
+
### Run selected cases
|
|
50
|
+
|
|
51
|
+
From version `0.4.0`, **Scenario → Review, then run → Cases to run** lists the validated suite's cases. Initially all are selected. Use individual checkboxes, **Select all**, or **Clear selection** to choose a nonempty set. A changed selection clears the review confirmation; choose the target, inspect the selected cases, and confirm review again before execution. Selection controls are disabled during a run or another busy operation. Navigation, language changes, and polling keep the selection. Loading/importing/validating a new draft resets it to all cases. It is not saved across page reloads.
|
|
52
|
+
|
|
53
|
+
The selected cases alone determine runnable/blocked counts and the run result. A selected case with unresolved questions or an unavailable required model stays blocked. Unselected cases do not execute and are not reported as passed or skipped. Flow QA does not infer prerequisites: each selected case must establish its own starting state. The source flow and full saved suite still pass existing validation and review-hash checks. Selection does not edit the saved file; the script preview and script/case downloads continue to include the full suite.
|
|
54
|
+
|
|
55
|
+
The run directory keeps the complete original `suite.json` and its hash. A partial `qa-report.json` retains `flow-qa/report-1` and adds `selection: { caseIds, totalCases }`; its `cases` list contains exactly the selected IDs in original suite order. A full run, including explicitly selecting every case, omits this field as before. History, detailed reports, and Markdown exports show selected/total counts and explain that omitted cases were not assessed. Comparison exports retain each run's scope, warn on partial runs and differing selections, and describe absent cases as missing results rather than recovered defects. Invalid selection metadata is not accepted as a valid report.
|
|
56
|
+
|
|
57
|
+
For the CLI, repeat `--case`: `flow-qa run cases/task-flow.json --target demo --case TC-001 --case TC-003`. Unknown, duplicate, or empty IDs are rejected before execution. The command keeps existing exit semantics: a partial pass is exit 0, a failure/block is exit 1, interrupted is 130, and invalid input is 2. If CI requires complete-suite coverage, omit `--case` and check that report selection metadata is absent.
|
|
58
|
+
|
|
49
59
|
### Expand or collapse case review
|
|
50
60
|
|
|
51
61
|
From version `0.2.8`, **Expand all cases** and **Collapse all cases** control the case details in Scenario. Individual case titles still toggle their own details. Language changes and navigation preserve which cases are open; loading a new suite keeps the original first-case-open default. These controls only change the view: they do not approve cases, modify the draft, or change the JSON editor disclosure. Review confirmation is still required before execution.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ihyeon/flow-qa",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"author": "Hyeonse Im",
|
|
6
6
|
"repository": {
|
|
@@ -57,7 +57,7 @@
|
|
|
57
57
|
"check:ui": "node --check src/ui/public/app.js && node --check src/ui/public/display.js && node --check src/ui/public/i18n.js && hds-audit .",
|
|
58
58
|
"demo:serve": "node examples/demo/server.ts",
|
|
59
59
|
"demo:smoke": "node scripts/demo-smoke.ts",
|
|
60
|
-
"smoke:ui": "node scripts/ui-i18n-smoke.mjs && node scripts/ui-comparison-smoke.mjs"
|
|
60
|
+
"smoke:ui": "node scripts/ui-i18n-smoke.mjs && node scripts/ui-comparison-smoke.mjs && node scripts/ui-selection-smoke.mjs"
|
|
61
61
|
},
|
|
62
62
|
"dependencies": {
|
|
63
63
|
"@ai-sdk/openai-compatible": "3.0.53",
|