@ihyeon/flow-qa 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.ko.md +9 -1
- package/README.md +9 -1
- package/dist/src/case-selection.js +50 -0
- package/dist/src/cli.js +25 -4
- package/dist/src/engine.js +15 -9
- package/dist/src/i18n.js +10 -0
- package/dist/src/report.js +20 -5
- package/dist/src/run-comparison.js +13 -0
- package/dist/src/ui/public/app.js +91 -17
- package/dist/src/ui/public/i18n.js +16 -0
- package/dist/src/ui/public/index.html +4 -3
- package/dist/src/ui/public/style.css +11 -0
- package/dist/src/ui/server.js +71 -5
- package/docs/architecture.md +3 -1
- package/docs/mobile.md +1 -1
- package/docs/releases.md +23 -1
- package/docs/ui.md +21 -1
- package/package.json +2 -2
package/README.ko.md
CHANGED
|
@@ -16,7 +16,7 @@ CLI와 `127.0.0.1` 전용 웹 UI를 제공합니다. UI는 영어로 시작하
|
|
|
16
16
|
Node.js 24.x(24.22.1 이상) 또는 26.12.1 이상이 필요합니다. 브라우저는 첫 웹 실행 전에 설치합니다.
|
|
17
17
|
|
|
18
18
|
```sh
|
|
19
|
-
npm install -g @ihyeon/flow-qa@0.
|
|
19
|
+
npm install -g @ihyeon/flow-qa@0.4.0
|
|
20
20
|
flow-qa browser install
|
|
21
21
|
flow-qa ui --demo
|
|
22
22
|
```
|
|
@@ -62,6 +62,14 @@ flow-qa plan flows/my-flow.md --workspace my-qa
|
|
|
62
62
|
|
|
63
63
|
`flow-qa doctor --workspace my-qa`로 로컬 의존성과 모델 인증용 환경 변수의 설정 여부를 확인할 수 있습니다. 키 값은 출력하지 않고 모델 서버도 호출하지 않습니다. 환경 변수가 있어도 실제 인증 성공을 확인한 것은 아닙니다.
|
|
64
64
|
|
|
65
|
+
## 선택한 케이스만 실행
|
|
66
|
+
|
|
67
|
+
`0.4.0`부터 **시나리오 → 검토 후 실행 → 실행할 케이스**에서 필요한 케이스만 고를 수 있습니다. 처음에는 모두 선택됩니다. 선택을 바꾸면 검토 확인을 다시 해야 하며, 저장한 원본 케이스 파일은 그대로 유지됩니다. 다른 케이스를 준비 단계로 자동 실행하지 않으므로 각 케이스에 필요한 시작 상태를 직접 구성하세요.
|
|
68
|
+
|
|
69
|
+
CLI에서는 `flow-qa run cases/task-flow.json --target demo --case TC-001 --case TC-003`처럼 ID를 반복합니다. `--case`를 생략하면 전체를 실행합니다. 부분 실행의 통과는 선택한 케이스에만 해당하며, 보고서와 비교 화면에도 선택 범위를 표시합니다. 자세한 동작은 [UI 가이드](docs/ui.md#run-selected-cases)를 참고하세요.
|
|
70
|
+
|
|
71
|
+
`0.5.0`부터 상세 보고서에서 **실패·오류 케이스 준비**를 누르면 실행 당시와 같은 저장된 케이스를 불러와 `실패`와 `실행 오류`만 선택합니다. 준비 후 케이스와 대상을 다시 검토하고 직접 실행하세요. 케이스·원본 파일이 없거나 바뀌었으면 준비를 중단하며, 실행 대상이 없거나 바뀌었으면 다시 선택해야 합니다. 자세한 동작은 [실패 케이스 준비](docs/ui.md#prepare-failed-cases)를 참고하세요.
|
|
72
|
+
|
|
65
73
|
## 실행 결과 비교
|
|
66
74
|
|
|
67
75
|
**실행 기록 → 실행 2건 비교**에서 같은 스위트와 대상의 두 기록을 선택하면 통과→실패, 실패→통과, 그 외 상태 변경, 동일 상태, 추가·누락 케이스를 확인할 수 있습니다. 분류별로 보거나 전체 비교를 Markdown·JSON으로 내려받으세요. 테스트 내용 변경과 불완전한 실행은 별도로 표시하며, 상태 변화만으로 제품 버그의 발생·해결을 확정하지 않습니다. 자세한 동작은 [UI 가이드](docs/ui.md#compare-two-runs)를 참고하세요.
|
package/README.md
CHANGED
|
@@ -16,7 +16,7 @@ Flow QA is available as a CLI and a web UI bound to `127.0.0.1`. Provide a targe
|
|
|
16
16
|
Node.js 24.x (24.22.1 or later) or 26.12.1 or later is required. Install Chromium before the first web run:
|
|
17
17
|
|
|
18
18
|
```sh
|
|
19
|
-
npm install -g @ihyeon/flow-qa@0.
|
|
19
|
+
npm install -g @ihyeon/flow-qa@0.4.0
|
|
20
20
|
flow-qa browser install
|
|
21
21
|
flow-qa ui --demo
|
|
22
22
|
```
|
|
@@ -62,6 +62,14 @@ For authenticated models, add `--key-env FLOW_QA_API_KEY` to the model command a
|
|
|
62
62
|
|
|
63
63
|
`flow-qa doctor --workspace my-qa` checks local dependencies and whether configured model-key environment variables are present. It does not print key values or call the model server; a present key does not confirm remote authentication.
|
|
64
64
|
|
|
65
|
+
## Run selected cases
|
|
66
|
+
|
|
67
|
+
From `0.4.0`, choose cases under **Scenario → Review, then run → Cases to run**. Every case starts selected. Clear or change the selection, review it, then run against your target. Selecting cases never edits the saved suite or implicitly runs prerequisite cases. Each case must establish its own starting state.
|
|
68
|
+
|
|
69
|
+
The CLI supports repeated IDs: `flow-qa run cases/task-flow.json --target demo --case TC-001 --case TC-003`. Omit `--case` to run the complete suite. Reports and comparisons label partial runs explicitly; a selected-run pass does not evaluate omitted cases. See the [UI guide](docs/ui.md#run-selected-cases).
|
|
70
|
+
|
|
71
|
+
From `0.5.0`, open a detailed report and choose **Prepare failed/error cases** to load its unchanged saved suite with only `failed` and `error` cases selected. Preparation never starts a run: review the cases and target, then run explicitly. Changed or missing case/source files block preparation; a changed or missing target must be selected again. See [preparing failed cases](docs/ui.md#prepare-failed-cases).
|
|
72
|
+
|
|
65
73
|
## Compare recorded runs
|
|
66
74
|
|
|
67
75
|
In **Run history → Compare two runs**, select two reports from the same suite and target to see passed-to-failed, failed-to-passed, other status changes, unchanged, added, and missing cases. Filter the comparison or download all results as Markdown/JSON. Changed test definitions and incomplete runs carry explicit warnings; observed status changes do not prove a product defect was introduced or fixed. See the [UI guide](docs/ui.md#compare-two-runs).
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
export class CaseSelectionError extends Error {
|
|
2
|
+
}
|
|
3
|
+
/** Resolve requested IDs against the complete suite without changing its order or contents. */
|
|
4
|
+
export function selectCases(suite, caseIds) {
|
|
5
|
+
if (caseIds === undefined)
|
|
6
|
+
return { suite };
|
|
7
|
+
if (!Array.isArray(caseIds) || caseIds.length === 0)
|
|
8
|
+
throw new CaseSelectionError('Select at least one case ID.');
|
|
9
|
+
const available = new Set(suite.cases.map((testCase) => testCase.id));
|
|
10
|
+
const requested = new Set();
|
|
11
|
+
for (const id of caseIds) {
|
|
12
|
+
if (typeof id !== 'string' || !available.has(id))
|
|
13
|
+
throw new CaseSelectionError(`Unknown case ID: ${String(id)}`);
|
|
14
|
+
if (requested.has(id))
|
|
15
|
+
throw new CaseSelectionError(`Duplicate selected case ID: ${id}`);
|
|
16
|
+
requested.add(id);
|
|
17
|
+
}
|
|
18
|
+
if (requested.size === suite.cases.length)
|
|
19
|
+
return { suite };
|
|
20
|
+
const cases = suite.cases.filter((testCase) => requested.has(testCase.id));
|
|
21
|
+
const requirementIds = new Set(cases.flatMap((testCase) => testCase.requirementIds));
|
|
22
|
+
return {
|
|
23
|
+
suite: { ...suite, cases, requirements: suite.requirements.filter((requirement) => requirementIds.has(requirement.id)) },
|
|
24
|
+
selection: { caseIds: cases.map((testCase) => testCase.id), totalCases: suite.cases.length },
|
|
25
|
+
};
|
|
26
|
+
}
|
|
27
|
+
/** Check optional additive report metadata before treating a report as comparable. */
|
|
28
|
+
export function isReportSelectionValid(cases, selection) {
|
|
29
|
+
const caseIds = cases.map((item) => item.id);
|
|
30
|
+
if (caseIds.some((id) => typeof id !== 'string' || !id.trim()) || new Set(caseIds).size !== caseIds.length)
|
|
31
|
+
return false;
|
|
32
|
+
if (selection === undefined)
|
|
33
|
+
return true;
|
|
34
|
+
if (selection === null || typeof selection !== 'object' || Array.isArray(selection))
|
|
35
|
+
return false;
|
|
36
|
+
const prototype = Object.getPrototypeOf(selection);
|
|
37
|
+
if (prototype !== Object.prototype && prototype !== null)
|
|
38
|
+
return false;
|
|
39
|
+
if (Reflect.ownKeys(selection).length !== 2 || !Object.hasOwn(selection, 'caseIds') || !Object.hasOwn(selection, 'totalCases'))
|
|
40
|
+
return false;
|
|
41
|
+
const value = selection;
|
|
42
|
+
if (!Number.isInteger(value.totalCases) || value.totalCases > 100 || value.totalCases <= caseIds.length)
|
|
43
|
+
return false;
|
|
44
|
+
if (!Array.isArray(value.caseIds) || value.caseIds.length === 0 || value.caseIds.length !== caseIds.length)
|
|
45
|
+
return false;
|
|
46
|
+
if (value.caseIds.some((id) => typeof id !== 'string' || !id.trim()))
|
|
47
|
+
return false;
|
|
48
|
+
const selected = new Set(value.caseIds);
|
|
49
|
+
return selected.size === value.caseIds.length && caseIds.every((id) => selected.has(id));
|
|
50
|
+
}
|
package/dist/src/cli.js
CHANGED
|
@@ -10,6 +10,7 @@ import { configSchema, idSchema, modelSchema, targetSchema } from "./schema.js";
|
|
|
10
10
|
import { initWorkspace, readConfig, readSuite, sha256, verifySuiteSource, writeConfig, writeJSON } from "./workspace.js";
|
|
11
11
|
import { buildPlanningPrompt, planFlow } from "./planner.js";
|
|
12
12
|
import { compileSuite } from "./compiler.js";
|
|
13
|
+
import { CaseSelectionError, isReportSelectionValid, selectCases } from "./case-selection.js";
|
|
13
14
|
import { runSuite } from "./engine.js";
|
|
14
15
|
import { renderMarkdown } from "./report.js";
|
|
15
16
|
import { targetFingerprint, targetLabel } from "./targets.js";
|
|
@@ -29,6 +30,7 @@ Usage: flow-qa <command> [arguments] [options]
|
|
|
29
30
|
validate <cases.json> Check case structure and assertions
|
|
30
31
|
generate <cases.json> [--out test.ts] Generate e2e test code
|
|
31
32
|
run <cases.json> --target <name> Run against a registered target and save a QA report
|
|
33
|
+
[--case <id> ...] Run only the selected case IDs (repeat --case for more)
|
|
32
34
|
report <run-directory> View a saved QA report
|
|
33
35
|
compare <run-a> <run-b> Compare case results
|
|
34
36
|
ui [--demo] [--port 4310] Start the local QA UI
|
|
@@ -54,6 +56,7 @@ Local example: flow-qa ui --demo
|
|
|
54
56
|
validate <cases.json> 케이스 구조와 검증 조건 확인
|
|
55
57
|
generate <cases.json> [--out test.ts] e2e 테스트 코드 생성
|
|
56
58
|
run <cases.json> --target <name> 연결한 대상에서 실행하고 QA 리포트 저장
|
|
59
|
+
[--case <id> ...] 선택한 케이스만 실행 (--case를 반복해 여러 개 선택)
|
|
57
60
|
report <run-directory> 저장된 QA 리포트 보기
|
|
58
61
|
compare <run-a> <run-b> 케이스별 결과 비교
|
|
59
62
|
ui [--demo] [--port 4310] 로컬 QA 화면 실행
|
|
@@ -126,6 +129,7 @@ export async function main(args) {
|
|
|
126
129
|
const { positionals, values } = parseArgs({ args, allowPositionals: true, strict: true, options: {
|
|
127
130
|
workspace: { type: 'string' }, url: { type: 'string' }, model: { type: 'string' },
|
|
128
131
|
'key-env': { type: 'string' }, out: { type: 'string' }, target: { type: 'string' },
|
|
132
|
+
case: { type: 'string', multiple: true },
|
|
129
133
|
platform: { type: 'string' }, 'bundle-id': { type: 'string' }, device: { type: 'string' },
|
|
130
134
|
'prompt-only': { type: 'boolean' }, help: { type: 'boolean', short: 'h' },
|
|
131
135
|
demo: { type: 'boolean' }, port: { type: 'string' }, 'with-deps': { type: 'boolean' }, lang: { type: 'string' },
|
|
@@ -145,7 +149,7 @@ export async function main(args) {
|
|
|
145
149
|
(values['bundle-id'] !== undefined || values.device !== undefined) && command !== 'connect') {
|
|
146
150
|
throw new Error(localized(locale, 'Platform and device options are only supported by connect and plan. See --help.', '플랫폼·기기 옵션은 connect와 plan에서만 사용할 수 있습니다. --help를 참고하세요.'));
|
|
147
151
|
}
|
|
148
|
-
if (values.demo && command !== 'ui' || values.port && command !== 'ui' ||
|
|
152
|
+
if (values.case !== undefined && command !== 'run' || values.demo && command !== 'ui' || values.port && command !== 'ui' ||
|
|
149
153
|
values['with-deps'] && (command !== 'browser' || arg !== 'install'))
|
|
150
154
|
throw new Error(localized(locale, 'This option is not supported for the command. See --help.', '이 명령에서 사용할 수 없는 옵션입니다. --help를 참고하세요.'));
|
|
151
155
|
if (command === 'ui') {
|
|
@@ -246,12 +250,24 @@ export async function main(args) {
|
|
|
246
250
|
}
|
|
247
251
|
const config = await readConfig(workspace);
|
|
248
252
|
const targetName = values.target ?? (Object.keys(config.targets).length === 1 ? Object.keys(config.targets)[0] : undefined);
|
|
253
|
+
let caseIds;
|
|
254
|
+
try {
|
|
255
|
+
caseIds = selectCases(suite, values.case).selection?.caseIds;
|
|
256
|
+
}
|
|
257
|
+
catch (error) {
|
|
258
|
+
if (error instanceof CaseSelectionError)
|
|
259
|
+
throw new Error(localizeProductMessage('Select one or more unique cases from this suite.', locale));
|
|
260
|
+
throw error;
|
|
261
|
+
}
|
|
249
262
|
const controller = new AbortController();
|
|
250
263
|
const interrupt = () => controller.abort();
|
|
251
264
|
process.once('SIGINT', interrupt);
|
|
252
265
|
process.once('SIGTERM', interrupt);
|
|
253
266
|
try {
|
|
254
|
-
const { report, runDir } = await runSuite({ workspace, suite, config, targetName: required(targetName, '--target', locale), signal: controller.signal
|
|
267
|
+
const { report, runDir } = await runSuite({ workspace, suite, config, targetName: required(targetName, '--target', locale), signal: controller.signal,
|
|
268
|
+
...(caseIds ? { caseIds } : {}) });
|
|
269
|
+
if (report.selection)
|
|
270
|
+
console.log(localized(locale, `Selected run: ${report.selection.caseIds.length}/${report.selection.totalCases} cases; the full suite was not evaluated.`, `선택 실행: ${report.selection.totalCases}개 중 ${report.selection.caseIds.length}개. 전체 케이스를 검증한 결과가 아닙니다.`));
|
|
255
271
|
console.log(localized(locale, `Run result: ${report.status}\nQA report: ${resolve(runDir, 'qa-report.md')}\nRaw result: ${resolve(runDir, 'qa-report.json')}`, `실행 결과: ${report.status}\nQA 리포트: ${resolve(runDir, 'qa-report.md')}\n원본 결과: ${resolve(runDir, 'qa-report.json')}`));
|
|
256
272
|
process.exitCode = report.status === 'passed' ? 0 : report.status === 'interrupted' ? 130 : 1;
|
|
257
273
|
}
|
|
@@ -283,11 +299,16 @@ export async function main(args) {
|
|
|
283
299
|
const { name: _targetName, ...identity } = data.target;
|
|
284
300
|
if (!targetSchema.safeParse(identity).success)
|
|
285
301
|
throw new Error(localized(locale, 'This is not a valid QA report.', '유효한 QA 리포트가 아닙니다.'));
|
|
302
|
+
if (!isReportSelectionValid(data.cases, data.selection))
|
|
303
|
+
throw new Error(localized(locale, 'This is not a valid QA report.', '유효한 QA 리포트가 아닙니다.'));
|
|
286
304
|
return data;
|
|
287
305
|
};
|
|
288
306
|
const before = await load(arg), after = await load(second);
|
|
289
|
-
const
|
|
290
|
-
|
|
307
|
+
const sameSelection = JSON.stringify(before.cases.map((item) => item.id).sort()) === JSON.stringify(after.cases.map((item) => item.id).sort());
|
|
308
|
+
const comparable = before.suiteId === after.suiteId && before.suiteHash === after.suiteHash && targetFingerprint(before.target) === targetFingerprint(after.target) && sameSelection;
|
|
309
|
+
console.log(comparable ? localized(locale, 'Comparing the same suite, target and case selection (check external data state separately)', '동일 명세·대상·케이스 범위 비교 (외부 데이터 상태는 별도 확인)') : localized(locale, 'Suite, target or case selection differs; no direct regression verdict is made.', '명세·대상 또는 케이스 범위가 달라 직접적인 회귀 판정은 하지 않습니다.'));
|
|
310
|
+
if (before.selection || after.selection)
|
|
311
|
+
console.log(localized(locale, 'This comparison includes selected cases only; omitted cases were not evaluated.', '이 비교에는 선택 실행이 포함되어 있습니다. 미선택 케이스는 검증되지 않았습니다.'));
|
|
291
312
|
const previous = new Map(before.cases.map((c) => [c.id, c.status]));
|
|
292
313
|
const current = new Map(after.cases.map((c) => [c.id, c.status]));
|
|
293
314
|
for (const id of new Set([...previous.keys(), ...current.keys()]))
|
package/dist/src/engine.js
CHANGED
|
@@ -4,6 +4,7 @@ import { createWriteStream } from 'node:fs';
|
|
|
4
4
|
import { lstat, mkdir, readFile, realpath, stat, writeFile } from 'node:fs/promises';
|
|
5
5
|
import path from 'node:path';
|
|
6
6
|
import { fileURLToPath } from 'node:url';
|
|
7
|
+
import { selectCases } from "./case-selection.js";
|
|
7
8
|
import { compileSuite } from "./compiler.js";
|
|
8
9
|
import { acquireMobileDevice, mobileReadiness } from "./mobile-runtime.js";
|
|
9
10
|
import { failureReport, normalizeReport1, renderMarkdown, stoppedReport } from "./report.js";
|
|
@@ -158,6 +159,10 @@ function runnableSuite(suite, blocked) {
|
|
|
158
159
|
}
|
|
159
160
|
/** Run one immutable suite snapshot through the locally installed e2e CLI. */
|
|
160
161
|
export async function runSuite(options) {
|
|
162
|
+
// A malformed selection must not leave a run directory behind.
|
|
163
|
+
const selected = selectCases(options.suite, options.caseIds);
|
|
164
|
+
let executionSuite = selected.suite;
|
|
165
|
+
let selection = selected.selection;
|
|
161
166
|
const workspace = await realpath(options.workspace);
|
|
162
167
|
const qaRoot = await ownedDirectory(workspace, '.flowqa');
|
|
163
168
|
const runsRoot = await ownedDirectory(qaRoot, 'runs');
|
|
@@ -183,7 +188,7 @@ export async function runSuite(options) {
|
|
|
183
188
|
let blocked = new Map();
|
|
184
189
|
let target = { name: options.targetName, url: '' };
|
|
185
190
|
let releaseMobileDevice;
|
|
186
|
-
const context = () => ({ suite:
|
|
191
|
+
const context = () => ({ suite: executionSuite, selection, runId, suiteHash, target, startedAt, finishedAt: new Date().toISOString(), blockedReasons: blocked });
|
|
187
192
|
const blockAll = (suite, reason) => {
|
|
188
193
|
for (const testCase of suite.cases)
|
|
189
194
|
if (!blocked.has(testCase.id))
|
|
@@ -193,6 +198,7 @@ export async function runSuite(options) {
|
|
|
193
198
|
try {
|
|
194
199
|
const suite = suiteSchema.parse(options.suite);
|
|
195
200
|
await verifySuiteSource(suite, workspace);
|
|
201
|
+
({ suite: executionSuite, selection } = selectCases(suite, options.caseIds));
|
|
196
202
|
const rawTarget = options.config?.targets?.[options.targetName];
|
|
197
203
|
if (rawTarget && typeof rawTarget === 'object' && !Array.isArray(rawTarget)) {
|
|
198
204
|
const raw = rawTarget;
|
|
@@ -200,8 +206,8 @@ export async function runSuite(options) {
|
|
|
200
206
|
if ((raw.platform === 'ios' || raw.platform === 'android') && typeof raw.bundleId === 'string') {
|
|
201
207
|
target = { name: options.targetName, kind: 'mobile', platform: raw.platform, bundleId: raw.bundleId, device: '' };
|
|
202
208
|
}
|
|
203
|
-
blocked = blockedCases(
|
|
204
|
-
blockAll(
|
|
209
|
+
blocked = blockedCases(executionSuite, options.config);
|
|
210
|
+
blockAll(executionSuite, 'A mobile target requires an explicit device.');
|
|
205
211
|
}
|
|
206
212
|
}
|
|
207
213
|
const config = configSchema.parse(options.config);
|
|
@@ -209,23 +215,23 @@ export async function runSuite(options) {
|
|
|
209
215
|
if (!selectedTarget)
|
|
210
216
|
throw new Error(`Target not found: ${options.targetName}`);
|
|
211
217
|
target = reportTarget(options.targetName, selectedTarget);
|
|
212
|
-
blocked = blockedCases(
|
|
218
|
+
blocked = blockedCases(executionSuite, config);
|
|
213
219
|
if ((suite.platform ?? 'web') !== targetKind(selectedTarget))
|
|
214
|
-
blockAll(
|
|
215
|
-
const runnable =
|
|
220
|
+
blockAll(executionSuite, 'The suite platform does not match the selected target.');
|
|
221
|
+
const runnable = executionSuite.cases.filter((testCase) => !blocked.has(testCase.id));
|
|
216
222
|
let sdkDevice;
|
|
217
223
|
if (selectedTarget.kind === 'mobile' && runnable.length > 0 && !options.signal?.aborted) {
|
|
218
224
|
const readiness = await mobileReadiness(selectedTarget, options.listMobileDevices);
|
|
219
225
|
if (!readiness.ready)
|
|
220
|
-
blockAll(
|
|
226
|
+
blockAll(executionSuite, readiness.reason);
|
|
221
227
|
else {
|
|
222
228
|
sdkDevice = readiness.sdkDevice;
|
|
223
229
|
releaseMobileDevice = acquireMobileDevice(selectedTarget.platform, readiness.deviceId);
|
|
224
230
|
}
|
|
225
231
|
if (!releaseMobileDevice)
|
|
226
|
-
blockAll(
|
|
232
|
+
blockAll(executionSuite, 'Selected mobile device is in use.');
|
|
227
233
|
}
|
|
228
|
-
const executable = runnableSuite(
|
|
234
|
+
const executable = runnableSuite(executionSuite, blocked);
|
|
229
235
|
const testFile = path.join(runDir, TEST_FILE);
|
|
230
236
|
const configFile = path.join(runDir, CONFIG_FILE);
|
|
231
237
|
const agentAvailable = runnable.some(agentCase);
|
package/dist/src/i18n.js
CHANGED
|
@@ -68,6 +68,16 @@ const koreanMessages = {
|
|
|
68
68
|
'Both reports must use the same target.': '두 리포트의 실행 대상이 같아야 합니다.',
|
|
69
69
|
'Both reports need a non-empty suite hash.': '두 리포트에 비어 있지 않은 케이스 모음 해시가 있어야 합니다.',
|
|
70
70
|
'Both reports need unique, non-empty case IDs.': '두 리포트의 케이스 ID는 비어 있지 않고 중복되지 않아야 합니다.',
|
|
71
|
+
'The report selection does not match its recorded cases.': '보고서의 선택 범위가 기록된 케이스와 일치하지 않습니다.',
|
|
72
|
+
'Select one or more unique cases from this suite.': '이 케이스 모음에서 중복 없이 하나 이상의 케이스를 선택하세요.',
|
|
73
|
+
'No failed or errored cases are available to rerun.': '다시 실행할 실패 또는 실행 오류 케이스가 없습니다.',
|
|
74
|
+
'No saved case file matches this report.': '이 보고서에 해당하는 저장된 케이스 파일이 없습니다.',
|
|
75
|
+
'Multiple saved case files match this report.': '이 보고서에 해당하는 케이스 파일이 여러 개입니다. 보관함에서 사용할 파일을 직접 선택하세요.',
|
|
76
|
+
'The saved case file or source flow was not found.': '저장된 케이스 파일 또는 원본 플로우를 찾을 수 없습니다.',
|
|
77
|
+
'The saved case file or source flow is invalid.': '저장된 케이스 파일 또는 원본 플로우가 올바르지 않습니다.',
|
|
78
|
+
'The saved case file changed since this run. Review it before rerunning.': '실행 이후 케이스 파일이 변경됐습니다. 보관함에서 다시 열어 검토하세요.',
|
|
79
|
+
'The report cases do not match the saved case file.': '보고서의 케이스가 저장된 케이스 파일과 일치하지 않습니다.',
|
|
80
|
+
'The report selection does not match the saved case file.': '보고서의 선택 범위가 저장된 케이스 파일과 일치하지 않습니다.',
|
|
71
81
|
'Invalid evidence selection.': '증거 파일 선택이 올바르지 않습니다.',
|
|
72
82
|
'The request is not allowed.': '허용되지 않은 요청입니다.',
|
|
73
83
|
'Invalid request token.': '요청 토큰이 올바르지 않습니다.',
|
package/dist/src/report.js
CHANGED
|
@@ -1,8 +1,21 @@
|
|
|
1
1
|
import { realpathSync } from 'node:fs';
|
|
2
2
|
import path from 'node:path';
|
|
3
3
|
import { expectedChecks } from "./compiler.js";
|
|
4
|
+
import { isReportSelectionValid } from "./case-selection.js";
|
|
4
5
|
import { localized, localizeProductMessage } from "./i18n.js";
|
|
5
6
|
import { targetLabel } from "./targets.js";
|
|
7
|
+
function scopedCases(context) {
|
|
8
|
+
if (!context.selection)
|
|
9
|
+
return context.suite.cases;
|
|
10
|
+
const selected = new Set(context.selection.caseIds);
|
|
11
|
+
const cases = context.suite.cases.filter((testCase) => selected.has(testCase.id));
|
|
12
|
+
if (!isReportSelectionValid(cases, context.selection))
|
|
13
|
+
throw new Error('Report selection does not match the current suite.');
|
|
14
|
+
return cases;
|
|
15
|
+
}
|
|
16
|
+
function reportSelection(context) {
|
|
17
|
+
return context.selection ? { selection: { caseIds: [...context.selection.caseIds], totalCases: context.selection.totalCases } } : {};
|
|
18
|
+
}
|
|
6
19
|
function object(value, label) {
|
|
7
20
|
if (value === null || typeof value !== 'object' || Array.isArray(value))
|
|
8
21
|
throw new Error(`Invalid raw report field: ${label}.`);
|
|
@@ -151,7 +164,8 @@ export function normalizeReport1(raw, context) {
|
|
|
151
164
|
const rawResults = array(run.results, 'run.results');
|
|
152
165
|
const rawErrors = array(run.errors, 'run.errors').map(rawError);
|
|
153
166
|
const sourceName = path.basename(context.testFile);
|
|
154
|
-
const
|
|
167
|
+
const selectedCases = scopedCases(context);
|
|
168
|
+
const knownTitles = new Map(selectedCases.map((testCase) => [`[${testCase.id}] ${testCase.title}`, testCase.id]));
|
|
155
169
|
const mapped = new Map();
|
|
156
170
|
for (const value of rawResults) {
|
|
157
171
|
const result = object(value, 'result');
|
|
@@ -185,7 +199,7 @@ export function normalizeReport1(raw, context) {
|
|
|
185
199
|
mapped.set(id, { status, attempts, ...(skip === undefined ? {} : { reason: string(skip.reason, 'result.skip.reason') }) });
|
|
186
200
|
}
|
|
187
201
|
const errors = rawErrors.map((error) => `${error.code}: ${error.message}`);
|
|
188
|
-
const cases =
|
|
202
|
+
const cases = selectedCases.map((testCase) => {
|
|
189
203
|
const expected = expectedChecks(testCase);
|
|
190
204
|
const result = mapped.get(testCase.id);
|
|
191
205
|
const blockedReason = context.blockedReasons.get(testCase.id);
|
|
@@ -220,7 +234,7 @@ export function normalizeReport1(raw, context) {
|
|
|
220
234
|
return {
|
|
221
235
|
schemaVersion: 'flow-qa/report-1', runId: context.runId, suiteId: context.suite.id,
|
|
222
236
|
suiteHash: context.suiteHash, target: context.target, startedAt: context.startedAt,
|
|
223
|
-
finishedAt: context.finishedAt, status, cases, errors, usage: reportUsage,
|
|
237
|
+
finishedAt: context.finishedAt, status, cases, errors, usage: reportUsage, ...reportSelection(context),
|
|
224
238
|
};
|
|
225
239
|
}
|
|
226
240
|
/** Keep every recorded attempt when the adapter stops a still-running child. */
|
|
@@ -236,13 +250,13 @@ export function failureReport(context, status, error) {
|
|
|
236
250
|
schemaVersion: 'flow-qa/report-1', runId: context.runId, suiteId: context.suite.id,
|
|
237
251
|
suiteHash: context.suiteHash, target: context.target, startedAt: context.startedAt,
|
|
238
252
|
finishedAt: context.finishedAt, status,
|
|
239
|
-
cases: context.
|
|
253
|
+
cases: scopedCases(context).map((testCase) => ({
|
|
240
254
|
id: testCase.id, title: testCase.title,
|
|
241
255
|
status: context.blockedReasons.has(testCase.id) ? 'blocked' : 'error',
|
|
242
256
|
expected: expectedChecks(testCase), attempts: [],
|
|
243
257
|
reason: context.blockedReasons.get(testCase.id) ?? error,
|
|
244
258
|
})),
|
|
245
|
-
errors: [error],
|
|
259
|
+
errors: [error], ...reportSelection(context),
|
|
246
260
|
};
|
|
247
261
|
}
|
|
248
262
|
function safeText(value, locale) {
|
|
@@ -277,6 +291,7 @@ export function renderMarkdown(report, locale = 'en') {
|
|
|
277
291
|
? safe(targetLabel(report.target))
|
|
278
292
|
: label('(URL available in the local JSON report)', '(주소는 로컬 JSON 보고서에서 확인)')}`,
|
|
279
293
|
`- ${label('Status', '상태')}: ${safe(report.status)}`,
|
|
294
|
+
...(report.selection ? [`- ${label('Scope', '범위')}: ${label(`Selected ${report.selection.caseIds.length} / ${report.selection.totalCases} cases; full suite not evaluated.`, `선택 ${report.selection.caseIds.length} / 전체 ${report.selection.totalCases}개 케이스; 전체 케이스 모음은 평가하지 않았습니다.`)}`] : []),
|
|
280
295
|
`- ${label('Started', '시작')}: ${safe(report.startedAt)}`,
|
|
281
296
|
`- ${label('Finished', '종료')}: ${safe(report.finishedAt)}`,
|
|
282
297
|
`- ${label('Results', '결과')}: ${label('passed', '통과')} ${counts.get('passed') ?? 0}, ${label('failed', '실패')} ${counts.get('failed') ?? 0}, ${label('flaky', '재시도 후 통과')} ${counts.get('flaky') ?? 0}, ${label('blocked', '차단')} ${counts.get('blocked') ?? 0}, ${label('skipped', '건너뜀')} ${counts.get('skipped') ?? 0}, ${label('error', '오류')} ${counts.get('error') ?? 0}`,
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { localized } from "./i18n.js";
|
|
2
|
+
import { isReportSelectionValid } from "./case-selection.js";
|
|
2
3
|
import { targetFingerprint, targetLabel } from "./targets.js";
|
|
3
4
|
export const comparisonCategories = ['newly-failed', 'now-passed', 'changed', 'unchanged', 'added', 'missing'];
|
|
4
5
|
export class ComparisonError extends Error {
|
|
@@ -20,6 +21,7 @@ function runSummary(report) {
|
|
|
20
21
|
runId: report.runId, suiteId: report.suiteId, suiteHash: report.suiteHash,
|
|
21
22
|
target: { ...report.target }, startedAt: report.startedAt,
|
|
22
23
|
finishedAt: report.finishedAt, status: report.status,
|
|
24
|
+
...(report.selection ? { selection: { caseIds: [...report.selection.caseIds], totalCases: report.selection.totalCases } } : {}),
|
|
23
25
|
};
|
|
24
26
|
}
|
|
25
27
|
function caseSnapshot(item) {
|
|
@@ -57,9 +59,17 @@ export function compareReports(baseline, current) {
|
|
|
57
59
|
}
|
|
58
60
|
const before = validateCaseIds(baseline);
|
|
59
61
|
const after = validateCaseIds(current);
|
|
62
|
+
if (!isReportSelectionValid(baseline.cases, baseline.selection) || !isReportSelectionValid(current.cases, current.selection)) {
|
|
63
|
+
throw new ComparisonError(409, 'The report selection does not match its recorded cases.');
|
|
64
|
+
}
|
|
60
65
|
const warnings = [];
|
|
61
66
|
if (baseline.suiteHash !== current.suiteHash)
|
|
62
67
|
warnings.push('suite-changed');
|
|
68
|
+
if (baseline.selection || current.selection) {
|
|
69
|
+
warnings.push('partial-run');
|
|
70
|
+
if (before.size !== after.size || [...before.keys()].some((id) => !after.has(id)))
|
|
71
|
+
warnings.push('selection-changed');
|
|
72
|
+
}
|
|
63
73
|
if (Date.parse(current.startedAt) < Date.parse(baseline.startedAt))
|
|
64
74
|
warnings.push('reverse-order');
|
|
65
75
|
if (baseline.status === 'interrupted' || baseline.status === 'error' || current.status === 'interrupted' || current.status === 'error') {
|
|
@@ -106,10 +116,13 @@ export function renderComparisonMarkdown(comparison, locale = 'en') {
|
|
|
106
116
|
`- ${label('Started', '시작')}: ${safe(item.startedAt)}`,
|
|
107
117
|
`- ${label('Finished', '종료')}: ${safe(item.finishedAt)}`,
|
|
108
118
|
`- ${label('Status', '상태')}: ${safe(item.status)}`,
|
|
119
|
+
...(item.selection ? [`- ${label('Scope', '범위')}: ${label(`Selected ${item.selection.caseIds.length} / ${item.selection.totalCases} cases; full suite not evaluated.`, `선택 ${item.selection.caseIds.length} / 전체 ${item.selection.totalCases}개 케이스; 전체 케이스 모음은 평가하지 않았습니다.`)}`] : []),
|
|
109
120
|
'',
|
|
110
121
|
];
|
|
111
122
|
const warningText = {
|
|
112
123
|
'suite-changed': label('Steps or expectations may differ because the suite changed; these outcomes do not prove a regression or fix.', '케이스 모음이 변경되어 단계나 기대 결과가 다를 수 있습니다. 이 결과만으로 회귀나 수정이 입증되지는 않습니다.'),
|
|
124
|
+
'partial-run': label('At least one run evaluated only selected cases; the full suite was not evaluated.', '하나 이상의 실행에서 선택한 케이스만 평가했습니다. 전체 케이스 모음은 평가하지 않았습니다.'),
|
|
125
|
+
'selection-changed': label('The runs selected different case sets; added or missing entries may reflect scope rather than a changed outcome.', '실행마다 선택한 케이스가 다릅니다. 추가 또는 누락 항목은 결과 변화가 아닌 실행 범위 차이일 수 있습니다.'),
|
|
113
126
|
'reverse-order': label('The selected current run started before the baseline run.', '선택한 현재 실행이 기준 실행보다 먼저 시작됐습니다.'),
|
|
114
127
|
'incomplete-run': label('At least one run ended with error or interruption; case results may be incomplete.', '하나 이상의 실행이 오류 또는 중단으로 끝나 케이스 결과가 불완전할 수 있습니다.'),
|
|
115
128
|
'mobile-identity-unverified': label('The recorded app and device settings match, but these reports do not verify that both runs used the same physical device.', '기록된 앱과 기기 설정은 같지만, 두 실행이 같은 실제 기기에서 진행됐는지는 이 리포트로 확인할 수 없습니다.'),
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
const i18n = window.FlowQAI18n;
|
|
5
5
|
const t = i18n.t;
|
|
6
6
|
const names = { scenario: 'nav.scenario', library: 'nav.library', history: 'nav.history', settings: 'nav.settings' };
|
|
7
|
-
const app = { state: null, connected: false, displayTargets: [], draft: null, script: '', savedFile: '', savedHash: '', origin: 'none', sourceFlow: '', boundSource: null, stale: false, dirty: false, busy: false, planning: false, planRequest: 0, stopping: false, report: null, reportMarkdown: '', selectedRun: '', comparison: null, comparisonRequest: 0, comparisonBusy: false, comparisonStatus: 'comparison.selectBoth', view: 'scenario' };
|
|
7
|
+
const app = { state: null, connected: false, displayTargets: [], draft: null, selectedCaseIds: new Set(), script: '', savedFile: '', savedHash: '', origin: 'none', sourceFlow: '', boundSource: null, stale: false, dirty: false, busy: false, planning: false, planRequest: 0, stopping: false, report: null, reportMarkdown: '', selectedRun: '', rerunOrigin: null, rerunTargetMismatch: false, navigationVersion: 0, comparison: null, comparisonRequest: 0, comparisonBusy: false, comparisonStatus: 'comparison.selectBoth', view: 'scenario' };
|
|
8
8
|
|
|
9
9
|
function el(tag, className, value) {
|
|
10
10
|
const node = document.createElement(tag);
|
|
@@ -44,6 +44,7 @@
|
|
|
44
44
|
return result;
|
|
45
45
|
}
|
|
46
46
|
function view(name) {
|
|
47
|
+
app.navigationVersion += 1;
|
|
47
48
|
app.view = name;
|
|
48
49
|
for (const key of Object.keys(names)) $('view-' + key).hidden = key !== name;
|
|
49
50
|
document.querySelectorAll('[data-view]').forEach((button) => {
|
|
@@ -61,6 +62,37 @@
|
|
|
61
62
|
function targetLabel(target) { return target?.kind === 'mobile' ? `${target.platform === 'ios' ? 'iOS' : 'Android'} · ${target.bundleId} · ${target.device}` : target?.url || ''; }
|
|
62
63
|
function targetDisplay(target) { return `${target?.name || t('app.noTarget')} · ${targetLabel(target)}`; }
|
|
63
64
|
function compatibleTarget() { const target = selectedTarget(); return target && targetPlatform(target) === suitePlatform(); }
|
|
65
|
+
function selectedCases() { return (app.draft?.cases || []).filter((item) => app.selectedCaseIds.has(item.id)); }
|
|
66
|
+
function selectionScope(selection) {
|
|
67
|
+
return selection && Array.isArray(selection.caseIds) && Number.isInteger(selection.totalCases) && selection.caseIds.length < selection.totalCases
|
|
68
|
+
? t('app.selectedScope', { selected: selection.caseIds.length, total: selection.totalCases }) : '';
|
|
69
|
+
}
|
|
70
|
+
function renderCaseSelection() {
|
|
71
|
+
const host = $('case-selection-list'); replace(host);
|
|
72
|
+
for (const item of app.draft?.cases || []) {
|
|
73
|
+
const label = el('label', 'case-selection-item');
|
|
74
|
+
const input = el('input'); input.type = 'checkbox'; input.dataset.caseId = item.id; input.checked = app.selectedCaseIds.has(item.id);
|
|
75
|
+
label.append(input, el('span', '', `${item.id} · ${item.title}`)); host.append(label);
|
|
76
|
+
}
|
|
77
|
+
updateCaseSelection();
|
|
78
|
+
}
|
|
79
|
+
function updateCaseSelection() {
|
|
80
|
+
const total = app.draft?.cases?.length || 0;
|
|
81
|
+
$('case-selection-summary').textContent = t('scenario.selectionSummary', { selected: selectedCases().length, total });
|
|
82
|
+
const disabled = !app.draft || app.busy || !!app.state?.active;
|
|
83
|
+
for (const input of $('case-selection-list').querySelectorAll('input[data-case-id]')) {
|
|
84
|
+
input.checked = app.selectedCaseIds.has(input.dataset.caseId);
|
|
85
|
+
input.disabled = disabled;
|
|
86
|
+
}
|
|
87
|
+
$('select-all-cases').disabled = disabled || selectedCases().length === total;
|
|
88
|
+
$('clear-case-selection').disabled = disabled || !app.selectedCaseIds.size;
|
|
89
|
+
}
|
|
90
|
+
function changeCaseSelection(ids) {
|
|
91
|
+
if (!app.draft || app.busy || app.state?.active) return;
|
|
92
|
+
const next = new Set(ids);
|
|
93
|
+
if (next.size === app.selectedCaseIds.size && [...next].every((id) => app.selectedCaseIds.has(id))) return;
|
|
94
|
+
app.selectedCaseIds = next; $('review-checkbox').checked = false; readiness();
|
|
95
|
+
}
|
|
64
96
|
function canDownloadSavedCases() {
|
|
65
97
|
return Boolean(app.draft && app.script && app.savedFile && app.savedHash && !app.dirty && !app.stale && !app.busy && !hasUnsavedScenario());
|
|
66
98
|
}
|
|
@@ -68,7 +100,8 @@
|
|
|
68
100
|
function readiness() {
|
|
69
101
|
const active = app.state?.active;
|
|
70
102
|
const ready = !!app.draft && !!app.script && !app.stale && !app.dirty;
|
|
71
|
-
$('run-btn').disabled = !app.connected || !ready || !app.savedFile || !app.savedHash || !$('review-checkbox').checked || !compatibleTarget() || !!active || app.busy;
|
|
103
|
+
$('run-btn').disabled = !app.connected || !ready || !app.savedFile || !app.savedHash || !selectedCases().length || !$('review-checkbox').checked || !compatibleTarget() || !!active || app.busy;
|
|
104
|
+
updateCaseSelection();
|
|
72
105
|
$('save-btn').disabled = !ready || app.busy;
|
|
73
106
|
$('suite-download').disabled = !canDownloadSavedCases();
|
|
74
107
|
$('flow-download').disabled = !canDownloadSourceMarkdown();
|
|
@@ -88,6 +121,9 @@
|
|
|
88
121
|
$('suite-editor').disabled = app.busy;
|
|
89
122
|
$('prompt-btn').disabled = app.busy;
|
|
90
123
|
$('scenario-platform').disabled = app.busy;
|
|
124
|
+
$('target-select').disabled = app.busy;
|
|
125
|
+
$('review-checkbox').disabled = app.busy;
|
|
126
|
+
$('report-rerun').disabled = !app.connected || !!active || app.busy || !app.selectedRun || !(app.report?.cases || []).some((item) => item.status === 'failed' || item.status === 'error');
|
|
91
127
|
$('mobile-setup-hint').hidden = suitePlatform() !== 'mobile' && $('scenario-platform').value !== 'mobile';
|
|
92
128
|
$('run-readiness').textContent = t(!app.connected ? 'app.connectionRequired'
|
|
93
129
|
: active ? (active.status === 'stopping' ? 'app.stopping' : active.kind === 'plan' ? 'app.planning' : 'app.running')
|
|
@@ -95,12 +131,13 @@
|
|
|
95
131
|
: app.dirty ? 'app.dirty'
|
|
96
132
|
: !ready ? 'app.needValidated'
|
|
97
133
|
: !app.savedFile || !app.savedHash ? 'app.needSaved'
|
|
134
|
+
: !selectedCases().length ? 'app.needCaseSelection'
|
|
98
135
|
: !$('target-select').value ? 'app.needTarget'
|
|
99
136
|
: !compatibleTarget() ? 'app.platformMismatch'
|
|
100
137
|
: !$('review-checkbox').checked ? 'app.needReview' : 'app.ready');
|
|
101
|
-
const cases =
|
|
138
|
+
const cases = selectedCases();
|
|
102
139
|
const blocked = cases.filter((item) => item.questions?.length || (item.steps.some((step) => ['act', 'assertAgent'].includes(step.type)) && !app.state?.models?.agent?.keyReady)).length;
|
|
103
|
-
$('case-readiness').textContent = ready ? t('app.caseReadiness', { total: cases.length, ready: cases.length - blocked, blocked, hint: blocked ? t('app.blockedHint') : '' }) : '';
|
|
140
|
+
$('case-readiness').textContent = ready ? t(cases.length < (app.draft?.cases.length || 0) ? 'app.selectedCaseReadiness' : 'app.caseReadiness', { total: cases.length, ready: cases.length - blocked, blocked, hint: blocked ? t('app.blockedHint') : '' }) : '';
|
|
104
141
|
const pill = $('draft-status');
|
|
105
142
|
pill.textContent = t(app.stale ? 'app.regenerate' : app.dirty ? 'app.revalidate' : ready ? 'app.validated' : 'scenario.noDraft');
|
|
106
143
|
pill.className = 'status-pill' + (ready ? ' ready' : app.stale || app.dirty ? ' dirty' : '');
|
|
@@ -165,11 +202,13 @@
|
|
|
165
202
|
}
|
|
166
203
|
}
|
|
167
204
|
function setDraft(suite, script, origin, sourceFlow = '', savedFile = '', savedHash = '') {
|
|
205
|
+
app.rerunOrigin = null; app.rerunTargetMismatch = false; renderRerunHints();
|
|
168
206
|
app.draft = suite; app.script = script; app.origin = origin; app.sourceFlow = sourceFlow; app.boundSource = suite.source ? { ...suite.source } : null; app.savedFile = savedFile; app.savedHash = savedHash; app.stale = false; app.dirty = false;
|
|
169
207
|
$('suite-editor').value = JSON.stringify(suite, null, 2);
|
|
170
208
|
$('scenario-platform').value = suite.platform || 'web';
|
|
171
209
|
$('review-checkbox').checked = false;
|
|
172
|
-
|
|
210
|
+
app.selectedCaseIds = new Set((suite.cases || []).map((item) => item.id));
|
|
211
|
+
summarize(suite); renderCaseSelection(); readiness();
|
|
173
212
|
}
|
|
174
213
|
function targetOptions() {
|
|
175
214
|
const selected = $('target-select').value;
|
|
@@ -185,6 +224,12 @@
|
|
|
185
224
|
for (const target of app.state?.targets || []) { const row = el('div', 'settings-entry'); row.append(el('strong', '', target.name), document.createTextNode(` · ${targetLabel(target)}${target.description ? ` · ${target.description}` : ''}`)); list.append(row); }
|
|
186
225
|
}
|
|
187
226
|
function targetDescription() { const target = selectedTarget(); $('target-description').textContent = target ? `${targetLabel(target)}${target.description ? ` · ${target.description}` : ''}` : t('scenario.targetHint'); readiness(); }
|
|
227
|
+
function renderRerunHints() {
|
|
228
|
+
$('rerun-origin-hint').hidden = !app.rerunOrigin;
|
|
229
|
+
if (app.rerunOrigin) $('rerun-origin-hint').textContent = t('app.rerunPrepared', app.rerunOrigin);
|
|
230
|
+
$('rerun-target-hint').hidden = !app.rerunTargetMismatch;
|
|
231
|
+
if (app.rerunTargetMismatch) $('rerun-target-hint').textContent = t('app.rerunTargetMismatch');
|
|
232
|
+
}
|
|
188
233
|
function modelDescription(role) {
|
|
189
234
|
const model = app.state?.models?.[role] || {};
|
|
190
235
|
const explanation = t(role === 'planner' ? 'app.modelPlannerHelp' : 'app.modelAgentHelp');
|
|
@@ -236,12 +281,13 @@
|
|
|
236
281
|
if (!matchingRuns.length) { host.append(el('p', 'empty', t(query ? 'app.historyNoSearchMatches' : 'app.historyNoStatusMatches'))); return; }
|
|
237
282
|
for (const run of matchingRuns) {
|
|
238
283
|
const row = el('div', 'list-row'); const copy = el('div'); copy.append(el('strong', '', `${run.suiteId} · ${i18n.status(run.status)}`), el('small', '', t('app.runMeta', { date: formatDate(run.startedAt), target: targetDisplay(run.target), passed: run.passedCount ?? 0, total: run.caseCount ?? 0, id: run.runId })));
|
|
284
|
+
const scope = selectionScope(run.selection); if (scope) copy.append(el('small', 'selection-scope', scope));
|
|
239
285
|
const button = el('button', 'button', t('app.viewReport')); button.type = 'button'; button.disabled = run.status === 'running' || run.status === 'stopping'; button.addEventListener('click', () => loadReport(run.runId)); row.append(copy, button); host.append(row);
|
|
240
286
|
}
|
|
241
287
|
}
|
|
242
288
|
const comparisonCategories = ['newly-failed', 'now-passed', 'changed', 'unchanged', 'added', 'missing'];
|
|
243
289
|
const comparisonCategoryKeys = { 'newly-failed': 'comparison.newlyFailed', 'now-passed': 'comparison.nowPassed', changed: 'comparison.changed', unchanged: 'comparison.unchanged', added: 'comparison.added', missing: 'comparison.missing' };
|
|
244
|
-
const comparisonWarningKeys = { 'suite-changed': 'comparison.warningSuite', 'reverse-order': 'comparison.warningReverse', 'incomplete-run': 'comparison.warningIncomplete', 'mobile-identity-unverified': 'comparison.warningMobileIdentity' };
|
|
290
|
+
const comparisonWarningKeys = { 'suite-changed': 'comparison.warningSuite', 'reverse-order': 'comparison.warningReverse', 'incomplete-run': 'comparison.warningIncomplete', 'mobile-identity-unverified': 'comparison.warningMobileIdentity', 'partial-run': 'comparison.warningPartial', 'selection-changed': 'comparison.warningSelectionChanged' };
|
|
245
291
|
function comparisonPair() { return { baseline: $('compare-baseline').value, current: $('compare-current').value }; }
|
|
246
292
|
function comparisonValid() { const { baseline, current } = comparisonPair(); return !!baseline && !!current && baseline !== current; }
|
|
247
293
|
function comparisonStatus(key, params = {}, error = false) {
|
|
@@ -268,7 +314,7 @@
|
|
|
268
314
|
const select = $(id); const selected = select.value;
|
|
269
315
|
select.replaceChildren(el('option', '', t('comparison.choose'))); select.firstChild.value = '';
|
|
270
316
|
for (const run of runs) {
|
|
271
|
-
const option = el('option', '', t('comparison.runLabel', { id: run.runId, shortId: run.runId.slice(-8), date: formatDate(run.startedAt), suite: run.suiteId, target: targetDisplay(run.target), status: i18n.status(run.status) }));
|
|
317
|
+
const option = el('option', '', `${t('comparison.runLabel', { id: run.runId, shortId: run.runId.slice(-8), date: formatDate(run.startedAt), suite: run.suiteId, target: targetDisplay(run.target), status: i18n.status(run.status) })}${selectionScope(run.selection) ? ` · ${selectionScope(run.selection)}` : ''}`);
|
|
272
318
|
option.value = run.runId; select.append(option);
|
|
273
319
|
}
|
|
274
320
|
select.value = runs.some((run) => run.runId === selected) ? selected : '';
|
|
@@ -297,7 +343,7 @@
|
|
|
297
343
|
const warnings = $('comparison-warnings'); replace(warnings);
|
|
298
344
|
for (const code of result.warnings || []) warnings.append(el('p', 'comparison-warning', t(comparisonWarningKeys[code] || 'comparison.warningUnknown', { code })));
|
|
299
345
|
const summary = $('comparison-summary'); replace(summary);
|
|
300
|
-
const meta = el('p', 'comparison-meta', `${result.baseline.runId} · ${formatDate(result.baseline.startedAt)} → ${result.current.runId} · ${formatDate(result.current.startedAt)}`); summary.append(meta);
|
|
346
|
+
const meta = el('p', 'comparison-meta', `${result.baseline.runId} · ${formatDate(result.baseline.startedAt)}${selectionScope(result.baseline.selection) ? ` · ${selectionScope(result.baseline.selection)}` : ''} → ${result.current.runId} · ${formatDate(result.current.startedAt)}${selectionScope(result.current.selection) ? ` · ${selectionScope(result.current.selection)}` : ''}`); summary.append(meta);
|
|
301
347
|
const counts = el('div', 'comparison-counts');
|
|
302
348
|
for (const category of comparisonCategories) {
|
|
303
349
|
const item = el('div', 'comparison-count');
|
|
@@ -423,7 +469,7 @@
|
|
|
423
469
|
noticeKey('app.compileComplete');
|
|
424
470
|
}); }
|
|
425
471
|
async function save() { await work(async () => { if (!app.draft || app.dirty || app.stale) throw localizedError('app.needValidated'); const body = { suite: app.draft }; if (!app.boundSource && app.sourceFlow && (app.origin === 'imported' ? editableFlow(app.sourceFlow) === $('flow-input').value : app.sourceFlow === $('flow-input').value.trim())) body.flow = app.sourceFlow; const result = await api('/api/save', body); const saved = await api(`/api/suite?file=${encodeURIComponent(result.file)}`); $('flow-input').value = saved.flow || ''; setDraft(saved.suite, saved.script, 'library', saved.flow || '', result.file, saved.suiteHash || ''); noticeKey('app.savedSuite', { file: result.file }); await refresh(false); }); }
|
|
426
|
-
async function run() { await work(async () => { if (!app.draft || !app.script || !app.savedFile || !app.savedHash || app.dirty || app.stale || !$('review-checkbox').checked || !compatibleTarget()) throw localizedError('app.runNotReady'); const target = selectedTarget(); const
|
|
472
|
+
async function run() { await work(async () => { if (!app.draft || !app.script || !app.savedFile || !app.savedHash || app.dirty || app.stale || !selectedCases().length || !$('review-checkbox').checked || !compatibleTarget()) throw localizedError('app.runNotReady'); const target = selectedTarget(); const body = { suite: app.savedFile, suiteHash: app.savedHash, targetName: target.name, targetFingerprint: target.targetFingerprint }; if (selectedCases().length < app.draft.cases.length) body.caseIds = selectedCases().map((item) => item.id); const result = await api('/api/run', body); noticeKey('app.runStarted', { id: result.operationId }); await refresh(false); }); }
|
|
427
473
|
async function stop() {
|
|
428
474
|
if (app.stopping || !app.state?.active) return;
|
|
429
475
|
if (app.state.active.kind === 'plan' && app.planning) app.planRequest += 1;
|
|
@@ -432,7 +478,31 @@
|
|
|
432
478
|
catch (error) { showError(error); }
|
|
433
479
|
finally { app.stopping = false; readiness(); }
|
|
434
480
|
}
|
|
435
|
-
async function loadReport(runId) { await work(async () => { const result = await api(`/api/report?runId=${encodeURIComponent(runId)}`); app.report = result.report; app.reportMarkdown = result.markdown; app.selectedRun = runId; renderReport(); $('report-panel').hidden = false; $('report-panel').scrollIntoView({ behavior: 'smooth', block: 'start' }); }); }
|
|
481
|
+
async function loadReport(runId) { await work(async () => { const result = await api(`/api/report?runId=${encodeURIComponent(runId)}`); app.report = result.report; app.reportMarkdown = result.markdown; app.selectedRun = runId; renderReport(); readiness(); $('report-panel').hidden = false; $('report-panel').scrollIntoView({ behavior: 'smooth', block: 'start' }); }); }
|
|
482
|
+
async function prepareFailedCases() { await work(async () => {
|
|
483
|
+
const runId = app.selectedRun;
|
|
484
|
+
const report = app.report;
|
|
485
|
+
const navigationVersion = app.navigationVersion;
|
|
486
|
+
if (!app.connected || app.state?.active || !runId || !report?.cases?.some((item) => item.status === 'failed' || item.status === 'error')) return;
|
|
487
|
+
const result = await api(`/api/rerun?runId=${encodeURIComponent(runId)}`);
|
|
488
|
+
if (app.selectedRun !== runId || app.report !== report || app.view !== 'history' || app.navigationVersion !== navigationVersion) return;
|
|
489
|
+
await refresh(false);
|
|
490
|
+
if (app.selectedRun !== runId || app.report !== report || app.view !== 'history' || app.navigationVersion !== navigationVersion || !app.connected || app.state?.active) return;
|
|
491
|
+
if (!confirmScenarioReplacement('app.confirmReplaceDraft')) return;
|
|
492
|
+
targetOptions();
|
|
493
|
+
$('flow-input').value = result.flow || '';
|
|
494
|
+
setDraft(result.suite, result.script, 'library', result.flow || '', result.file, result.suiteHash);
|
|
495
|
+
app.selectedCaseIds = new Set(result.caseIds);
|
|
496
|
+
renderCaseSelection();
|
|
497
|
+
const target = app.displayTargets.find((item) => item.name === result.targetName);
|
|
498
|
+
const sameTarget = result.targetMatches === true && target?.targetFingerprint === result.targetFingerprint;
|
|
499
|
+
$('target-select').value = sameTarget ? target.name : '';
|
|
500
|
+
app.rerunTargetMismatch = !sameTarget;
|
|
501
|
+
app.rerunOrigin = { runId: result.originRunId, count: result.caseIds.length };
|
|
502
|
+
renderRerunHints(); targetDescription();
|
|
503
|
+
noticeKey(sameTarget ? 'app.rerunPrepared' : 'app.rerunPreparedTarget', app.rerunOrigin);
|
|
504
|
+
view('scenario');
|
|
505
|
+
}); }
|
|
436
506
|
async function downloadReportMarkdown() {
|
|
437
507
|
const runId = app.selectedRun; const locale = i18n.locale();
|
|
438
508
|
if (!runId || !app.report) return;
|
|
@@ -447,7 +517,7 @@
|
|
|
447
517
|
const report = app.report; if (!report) return;
|
|
448
518
|
$('report-heading').textContent = `${report.runId} · ${i18n.status(report.status)} · ${targetDisplay(report.target)}`;
|
|
449
519
|
const host = $('report-content'); replace(host);
|
|
450
|
-
const meta = el('div', 'report-meta'); for (const value of [t('app.reportStatus', { status: i18n.status(report.status) }), t('app.reportTarget', { target: targetDisplay(report.target) }), t('app.reportStarted', { date: formatDate(report.startedAt) }), t('app.reportFinished', { date: formatDate(report.finishedAt) })]) meta.append(el('span', '', value)); host.append(meta);
|
|
520
|
+
const meta = el('div', 'report-meta'); for (const value of [t('app.reportStatus', { status: i18n.status(report.status) }), t('app.reportTarget', { target: targetDisplay(report.target) }), t('app.reportStarted', { date: formatDate(report.startedAt) }), t('app.reportFinished', { date: formatDate(report.finishedAt) })]) meta.append(el('span', '', value)); const scope = selectionScope(report.selection); if (scope) meta.append(el('span', 'selection-scope', scope)); host.append(meta);
|
|
451
521
|
if (report.usage) host.append(el('p', 'hint', t('app.usage', { tokens: report.usage.modelTokens ?? t('app.unavailable'), cost: typeof report.usage.estimatedCostUsd === 'number' ? `$${report.usage.estimatedCostUsd.toFixed(4)}` : t('app.unavailable') })));
|
|
452
522
|
if (report.errors?.length) host.append(el('p', 'report-errors', report.errors.join('\n')));
|
|
453
523
|
for (const item of report.cases || []) {
|
|
@@ -497,8 +567,8 @@
|
|
|
497
567
|
const input = form?.elements.namedItem(focus.name);
|
|
498
568
|
if (input) { input.focus(); if (Number.isInteger(focus.start)) input.setSelectionRange(focus.start, focus.end); }
|
|
499
569
|
}
|
|
500
|
-
summarize(app.draft, true);
|
|
501
|
-
renderActive(); targetDescription(); renderLibrary(); renderHistory(); renderComparisonOptions(); renderComparison(); renderReport();
|
|
570
|
+
summarize(app.draft, true); renderCaseSelection();
|
|
571
|
+
renderActive(); targetDescription(); renderRerunHints(); renderLibrary(); renderHistory(); renderComparisonOptions(); renderComparison(); renderReport();
|
|
502
572
|
if (app.comparisonStatus) comparisonStatus(app.comparisonStatus, app.comparisonStatusParams || {}, app.comparisonStatus === 'comparison.error');
|
|
503
573
|
app.reportMarkdown = '';
|
|
504
574
|
if (app.notice) $('notice').textContent = t(app.notice.key, app.notice.params);
|
|
@@ -525,17 +595,20 @@
|
|
|
525
595
|
const json = $('suite-editor').value;
|
|
526
596
|
const unsavedJson = json.trim() && (!app.savedFile || json !== JSON.stringify(app.draft, null, 2));
|
|
527
597
|
if (unsavedJson && !confirmScenarioReplacement('app.confirmNewDraft')) return;
|
|
528
|
-
app.draft = null; app.script = ''; app.savedFile = ''; app.savedHash = '';
|
|
529
|
-
app.origin = 'none'; app.sourceFlow = ''; app.boundSource = null; app.stale = false; app.dirty = false;
|
|
598
|
+
app.draft = null; app.selectedCaseIds = new Set(); app.script = ''; app.savedFile = ''; app.savedHash = '';
|
|
599
|
+
app.origin = 'none'; app.sourceFlow = ''; app.boundSource = null; app.stale = false; app.dirty = false; app.rerunOrigin = null; app.rerunTargetMismatch = false; renderRerunHints();
|
|
530
600
|
$('suite-editor').value = ''; $('review-checkbox').checked = false;
|
|
531
|
-
summarize(null); readiness();
|
|
601
|
+
summarize(null); renderCaseSelection(); readiness();
|
|
532
602
|
noticeKey('app.newDraftNotice');
|
|
533
603
|
$('suite-editor').focus();
|
|
534
604
|
});
|
|
535
605
|
$('flow-input').addEventListener('input', () => { if (app.origin === 'imported' ? $('flow-input').value !== editableFlow(app.sourceFlow) : (app.sourceFlow || app.boundSource) && $('flow-input').value.trim() !== editableFlow(app.sourceFlow).trim()) { app.stale = true; app.savedHash = ''; $('review-checkbox').checked = false; } readiness(); });
|
|
536
606
|
$('scenario-platform').addEventListener('change', () => { app.planRequest += 1; if (app.draft) { app.stale = true; app.savedHash = ''; } $('review-checkbox').checked = false; readiness(); });
|
|
537
607
|
$('suite-editor').addEventListener('input', () => { app.dirty = true; app.script = ''; app.savedFile = ''; app.savedHash = ''; if (app.origin === 'none') app.origin = 'manual'; $('review-checkbox').checked = false; summarize(null); readiness(); });
|
|
538
|
-
$('review-checkbox').addEventListener('change', readiness); $('target-select').addEventListener('change', () => { $('review-checkbox').checked = false; targetDescription(); });
|
|
608
|
+
$('review-checkbox').addEventListener('change', readiness); $('target-select').addEventListener('change', () => { $('review-checkbox').checked = false; if ($('target-select').value) { app.rerunTargetMismatch = false; renderRerunHints(); } targetDescription(); });
|
|
609
|
+
$('case-selection-list').addEventListener('change', (event) => { const input = event.target.closest('input[data-case-id]'); if (!input) return; const next = new Set(app.selectedCaseIds); if (input.checked) next.add(input.dataset.caseId); else next.delete(input.dataset.caseId); changeCaseSelection(next); });
|
|
610
|
+
$('select-all-cases').addEventListener('click', () => changeCaseSelection((app.draft?.cases || []).map((item) => item.id)));
|
|
611
|
+
$('clear-case-selection').addEventListener('click', () => changeCaseSelection([]));
|
|
539
612
|
$('plan-btn').addEventListener('click', generate); $('prompt-btn').addEventListener('click', prompt); $('compile-btn').addEventListener('click', compile); $('save-btn').addEventListener('click', save); $('run-btn').addEventListener('click', run); $('stop-btn').addEventListener('click', stop);
|
|
540
613
|
$('import-btn').addEventListener('click', importFiles);
|
|
541
614
|
$('global-stop-btn').addEventListener('click', stop);
|
|
@@ -545,6 +618,7 @@
|
|
|
545
618
|
$('script-download').addEventListener('click', () => { if (app.draft && !app.dirty && !app.stale) download(`${app.draft.id || 'flow-qa'}.test.mjs`, app.script, 'text/javascript'); });
|
|
546
619
|
$('report-md').addEventListener('click', downloadReportMarkdown);
|
|
547
620
|
$('report-json').addEventListener('click', () => { if (app.report) download(`${app.selectedRun}.json`, JSON.stringify(app.report, null, 2), 'application/json'); });
|
|
621
|
+
$('report-rerun').addEventListener('click', prepareFailedCases);
|
|
548
622
|
$('target-form').addEventListener('submit', saveTarget);
|
|
549
623
|
$('target-kind').addEventListener('change', targetFields); targetFields();
|
|
550
624
|
$('language-select').addEventListener('change', (event) => i18n.setLocale(event.currentTarget.value));
|
|
@@ -56,6 +56,12 @@
|
|
|
56
56
|
'scenario.newDraft': ['New JSON draft', '새 JSON 초안'],
|
|
57
57
|
'scenario.compile': ['Validate JSON · Generate script', 'JSON 검증 · 스크립트 생성'],
|
|
58
58
|
'scenario.script': ['Run script', '실행 스크립트'],
|
|
59
|
+
'scenario.fullSuiteScript': ['Script preview and download include the full suite, regardless of the run selection.', '스크립트 미리보기와 다운로드에는 실행 선택과 관계없이 전체 스위트가 포함됩니다.'],
|
|
60
|
+
'scenario.selectionTitle': ['Cases to run', '실행할 케이스'],
|
|
61
|
+
'scenario.selectionSummary': ['{selected} selected / {total} total', '선택 {selected}개 / 전체 {total}개'],
|
|
62
|
+
'scenario.selectionHint': ['Each selected case must establish its own starting state. Excluded cases are not assessed.', '선택한 각 케이스는 자체 시작 상태를 설정해야 합니다. 제외한 케이스는 평가하지 않습니다.'],
|
|
63
|
+
'scenario.selectAllCases': ['Select all', '전체 선택'],
|
|
64
|
+
'scenario.clearSelection': ['Clear selection', '선택 해제'],
|
|
59
65
|
'scenario.downloadScript': ['Download script ↓', '스크립트 다운로드 ↓'],
|
|
60
66
|
'scenario.scriptPlaceholder': ['The script for validated cases will appear here.', '검증된 케이스의 스크립트가 여기에 표시됩니다.'],
|
|
61
67
|
'scenario.runTitle': ['Review, then run', '검토 후 실행'],
|
|
@@ -112,6 +118,8 @@
|
|
|
112
118
|
'comparison.warningReverse': ['The selected current run started before the baseline run.', '선택한 현재 실행이 기준 실행보다 먼저 시작됐습니다.'],
|
|
113
119
|
'comparison.warningIncomplete': ['At least one run is incomplete. Interpret case outcomes with its detailed report.', '완료되지 않은 실행이 있습니다. 상세 보고서와 함께 결과를 해석하세요.'],
|
|
114
120
|
'comparison.warningMobileIdentity': ['Mobile comparison matches recorded app and device settings. It does not verify that both runs used the same physical device.', '모바일 비교는 기록된 앱과 기기 설정을 기준으로 합니다. 두 실행이 같은 실제 기기를 사용했는지는 확인하지 않습니다.'],
|
|
121
|
+
'comparison.warningPartial': ['At least one run assessed only selected cases. Missing outcomes do not mean those cases passed.', '하나 이상의 실행에서 선택한 케이스만 평가했습니다. 결과가 없는 케이스를 통과로 해석할 수 없습니다.'],
|
|
122
|
+
'comparison.warningSelectionChanged': ['The runs selected different case sets. Compare only outcomes for cases assessed in both runs.', '두 실행에서 선택한 케이스가 다릅니다. 두 실행 모두에서 평가한 케이스의 결과만 비교하세요.'],
|
|
115
123
|
'comparison.warningUnknown': ['Comparison warning: {code}', '비교 주의: {code}'],
|
|
116
124
|
'comparison.pair': ['{baseline} → {current}', '{baseline} → {current}'],
|
|
117
125
|
'comparison.notPresent': ['not present', '없음'],
|
|
@@ -123,6 +131,8 @@
|
|
|
123
131
|
'comparison.error': ['Comparison failed: {message}', '비교 실패: {message}'],
|
|
124
132
|
'comparison.runLabel': ['{date} · {suite} · {shortId} · {target} · {status} · {id}', '{date} · {suite} · {shortId} · {target} · {status} · {id}'],
|
|
125
133
|
'history.report': ['Detailed report', '상세 보고서'],
|
|
134
|
+
'history.prepareFailed': ['Prepare failed/error cases', '실패·오류 케이스 준비'],
|
|
135
|
+
'history.prepareHint': ['Only failed and error cases are selected. Passed, flaky, blocked, and skipped cases are excluded. Preparation never starts a run.', '실패·오류 케이스만 선택합니다. 통과·간헐 실패·차단·건너뜀 케이스는 제외됩니다. 준비만으로 실행은 시작되지 않습니다.'],
|
|
126
136
|
'history.downloadMarkdown': ['Download Markdown', 'Markdown 다운로드'],
|
|
127
137
|
'history.downloadJson': ['Download JSON', 'JSON 다운로드'],
|
|
128
138
|
'history.evidenceHint': ['The report summarizes results. Open or download screenshots and recordings from each attempt. Evidence files are not included in report downloads.', '보고서는 결과 요약입니다. 스크린샷·화면 기록은 각 시도에서 열거나 개별 다운로드하세요. 증거 파일은 보고서 다운로드에 포함되지 않습니다.'],
|
|
@@ -192,7 +202,10 @@
|
|
|
192
202
|
'app.needTarget': ['Select a run target.', '실행 대상을 선택하세요.'],
|
|
193
203
|
'app.platformMismatch': ['The case platform and target type must match.', '케이스 플랫폼과 대상 유형이 일치해야 합니다.'],
|
|
194
204
|
'app.needReview': ['Review the contents, then select the checkbox.', '내용 검토 후 확인란을 선택하세요.'],
|
|
205
|
+
'app.needCaseSelection': ['Select at least one case to run.', '실행할 케이스를 하나 이상 선택하세요.'],
|
|
195
206
|
'app.ready': ['Ready to run.', '실행할 준비가 됐습니다.'],
|
|
207
|
+
'app.selectedScope': ['Selected {selected}/{total} cases · excluded cases not assessed', '선택 {selected}/{total}개 · 제외한 케이스는 평가하지 않음'],
|
|
208
|
+
'app.selectedCaseReadiness': ['{total} selected · {ready} runnable · {blocked} blocked before run{hint}', '선택 {total}개 · 실행 가능 {ready}개 · 사전 차단 {blocked}개{hint}'],
|
|
196
209
|
'app.caseReadiness': ['{total} total · {ready} runnable · {blocked} blocked before run{hint}', '전체 {total}개 · 실행 가능 {ready}개 · 사전 차단 {blocked}개{hint}'],
|
|
197
210
|
'app.blockedHint': [' — Check open questions or run agent settings.', ' — 확인 질문이나 실행 모델 설정을 확인하세요.'],
|
|
198
211
|
'app.regenerate': ['Regenerate required', '재생성 필요'],
|
|
@@ -295,6 +308,9 @@
|
|
|
295
308
|
'app.savedSuite': ['Cases saved to a separate file: {file}. Review them again before running.', '케이스를 별도 파일로 저장했습니다: {file}. 실행 전 내용을 다시 검토하세요.'],
|
|
296
309
|
'app.runNotReady': ['Save or reload cases, select a target, and review the contents before running.', '케이스 저장 또는 다시 불러오기, 대상 선택, 직접 검토가 필요합니다.'],
|
|
297
310
|
'app.runStarted': ['Run started. Task ID: {id}', '실행을 시작했습니다. 작업 ID: {id}'],
|
|
311
|
+
'app.rerunPrepared': ['Prepared {count} failed/error cases from run {runId}. Review the cases and target before starting a new run.', '실행 {runId}의 실패·오류 케이스 {count}개를 준비했습니다. 새 실행 전 케이스와 대상을 검토하세요.'],
|
|
312
|
+
'app.rerunPreparedTarget': ['Prepared {count} failed/error cases from run {runId}. The target changed or is unavailable. Select and review a target before starting a new run.', '실행 {runId}의 실패·오류 케이스 {count}개를 준비했습니다. 대상이 변경되었거나 없습니다. 새 실행 전 대상을 선택하고 검토하세요.'],
|
|
313
|
+
'app.rerunTargetMismatch': ['The report target changed or is unavailable. Select a current target and review it before running.', '보고서의 대상이 변경되었거나 없습니다. 현재 대상을 선택하고 검토한 뒤 실행하세요.'],
|
|
298
314
|
'app.stopRequested': ['Stop request sent. Checking the final state.', '중지 요청을 보냈습니다. 실제 종료 상태를 확인 중입니다.'],
|
|
299
315
|
'app.reportStatus': ['Status {status}', '상태 {status}'],
|
|
300
316
|
'app.reportTarget': ['Target {target}', '대상 {target}'],
|
|
@@ -72,6 +72,7 @@
|
|
|
72
72
|
<textarea id="suite-editor" class="code-editor" rows="13" spellcheck="false" placeholder="Load a local demo case from the library or enter suite JSON." data-i18n-placeholder="scenario.suitePlaceholder"></textarea>
|
|
73
73
|
<div class="actions"><button id="new-draft-btn" type="button" class="button quiet" data-i18n="scenario.newDraft">New JSON draft</button><button id="compile-btn" type="button" class="button" data-i18n="scenario.compile">Validate JSON · Generate script</button></div>
|
|
74
74
|
<div class="script-head"><h3 data-i18n="scenario.script">Run script</h3><button type="button" id="script-download" class="text-button" disabled data-i18n="scenario.downloadScript">Download script ↓</button></div>
|
|
75
|
+
<p class="hint" data-i18n="scenario.fullSuiteScript">Script preview and download include the full suite, regardless of the run selection.</p>
|
|
75
76
|
<pre id="script-preview" class="script-preview" data-i18n="scenario.scriptPlaceholder">The script for validated cases will appear here.</pre>
|
|
76
77
|
</div>
|
|
77
78
|
</details>
|
|
@@ -80,8 +81,8 @@
|
|
|
80
81
|
<section class="execution-section" aria-labelledby="run-title">
|
|
81
82
|
<div class="section-head"><span class="section-number">03</span><div><h2 id="run-title" data-i18n="scenario.runTitle">Review, then run</h2><p data-i18n="scenario.runHelp">Select a target and confirm you have reviewed the cases.</p></div></div>
|
|
82
83
|
<div class="execution-grid">
|
|
83
|
-
<div class="target-field"><label for="target-select" class="field-label" data-i18n="scenario.target">Run target</label><select id="target-select"><option value="" data-i18n="scenario.selectTarget">Select a target</option></select><p id="target-description" class="hint" data-i18n="scenario.targetHint">You can add a target in Connections.</p><p id="mobile-setup-hint" class="hint" hidden data-i18n="scenario.mobileSetup">For iOS, prepare Xcode, a simulator or signed device, and app signing. For Android, prepare the Android SDK and an emulator or authorized device. Configuration alone does not verify device access.</p></div>
|
|
84
|
-
<div class="execution-confirm"><p id="case-readiness" class="hint"></p><label class="review-line"><input type="checkbox" id="review-checkbox"><span data-i18n="scenario.reviewConfirm">I have reviewed the requirements, case steps, and expected results.</span></label><div class="run-bar"><span id="run-readiness" data-i18n="scenario.runInitial">Validated cases and a run target are required.</span><div class="actions"><button type="button" id="stop-btn" class="button quiet" hidden data-i18n="scenario.stop">Stop run</button><button type="button" id="run-btn" class="button primary" disabled><span data-i18n="scenario.run">Run QA</span><span aria-hidden="true">→</span></button></div></div></div>
|
|
84
|
+
<div class="target-field"><label for="target-select" class="field-label" data-i18n="scenario.target">Run target</label><select id="target-select"><option value="" data-i18n="scenario.selectTarget">Select a target</option></select><p id="target-description" class="hint" data-i18n="scenario.targetHint">You can add a target in Connections.</p><p id="rerun-target-hint" class="hint" hidden></p><p id="mobile-setup-hint" class="hint" hidden data-i18n="scenario.mobileSetup">For iOS, prepare Xcode, a simulator or signed device, and app signing. For Android, prepare the Android SDK and an emulator or authorized device. Configuration alone does not verify device access.</p></div>
|
|
85
|
+
<div class="execution-confirm"><div class="case-selection"><div class="case-selection-head"><h3 id="case-selection-title" data-i18n="scenario.selectionTitle">Cases to run</h3><span id="case-selection-summary" role="status" aria-live="polite"></span></div><p id="rerun-origin-hint" class="hint" hidden></p><p class="hint" data-i18n="scenario.selectionHint">Each selected case must establish its own starting state. Excluded cases are not assessed.</p><div class="actions"><button type="button" id="select-all-cases" class="button quiet" data-i18n="scenario.selectAllCases">Select all</button><button type="button" id="clear-case-selection" class="button quiet" data-i18n="scenario.clearSelection">Clear selection</button></div><div id="case-selection-list" role="group" aria-labelledby="case-selection-title"></div></div><p id="case-readiness" class="hint"></p><label class="review-line"><input type="checkbox" id="review-checkbox"><span data-i18n="scenario.reviewConfirm">I have reviewed the requirements, case steps, and expected results.</span></label><div class="run-bar"><span id="run-readiness" data-i18n="scenario.runInitial">Validated cases and a run target are required.</span><div class="actions"><button type="button" id="stop-btn" class="button quiet" hidden data-i18n="scenario.stop">Stop run</button><button type="button" id="run-btn" class="button primary" disabled><span data-i18n="scenario.run">Run QA</span><span aria-hidden="true">→</span></button></div></div></div>
|
|
85
86
|
</div>
|
|
86
87
|
</section>
|
|
87
88
|
</div>
|
|
@@ -103,7 +104,7 @@
|
|
|
103
104
|
<div id="comparison-results" hidden><div id="comparison-warnings"></div><div id="comparison-summary"></div><div class="comparison-filter-row"><label for="comparison-filter" data-i18n="comparison.filter">Show cases</label><select id="comparison-filter" aria-controls="comparison-content"><option value="" data-i18n="comparison.all">All categories</option><option value="newly-failed" data-i18n="comparison.newlyFailed">Newly failed</option><option value="now-passed" data-i18n="comparison.nowPassed">Now passed</option><option value="changed" data-i18n="comparison.changed">Changed status</option><option value="unchanged" data-i18n="comparison.unchanged">Same status</option><option value="added" data-i18n="comparison.added">Added to current</option><option value="missing" data-i18n="comparison.missing">Missing from current</option></select></div><div class="actions comparison-actions"><button type="button" id="comparison-md" class="button" data-i18n="comparison.downloadMarkdown" disabled>Download comparison Markdown</button><button type="button" id="comparison-json" class="button quiet" data-i18n="comparison.downloadJson" disabled>Download comparison JSON</button><button type="button" id="compare-open-baseline" class="button quiet" data-i18n="comparison.openBaseline" disabled>View baseline report</button><button type="button" id="compare-open-current" class="button quiet" data-i18n="comparison.openCurrent" disabled>View current report</button></div><div id="comparison-content"></div></div>
|
|
104
105
|
</div>
|
|
105
106
|
</section>
|
|
106
|
-
<section id="report-panel" class="report-panel" hidden><div class="section-head"><span class="section-number">03</span><div><h2 data-i18n="history.report">Detailed report</h2><p id="report-heading"></p></div></div><div class="section-body"><div class="actions"><button type="button" id="report-md" class="button" data-i18n="history.downloadMarkdown">Download Markdown</button><button type="button" id="report-json" class="button quiet" data-i18n="history.downloadJson">Download JSON</button></div><p class="hint" data-i18n="history.evidenceHint">The report summarizes results. Open or download screenshots and recordings from each attempt. Evidence files are not included in report downloads.</p><div id="report-content"></div></div></section>
|
|
107
|
+
<section id="report-panel" class="report-panel" hidden><div class="section-head"><span class="section-number">03</span><div><h2 data-i18n="history.report">Detailed report</h2><p id="report-heading"></p></div></div><div class="section-body"><div class="actions"><button type="button" id="report-md" class="button" data-i18n="history.downloadMarkdown">Download Markdown</button><button type="button" id="report-json" class="button quiet" data-i18n="history.downloadJson">Download JSON</button><button type="button" id="report-rerun" class="button primary" disabled data-i18n="history.prepareFailed">Prepare failed/error cases</button></div><p class="hint" data-i18n="history.prepareHint">Passed, flaky, blocked, and skipped cases are excluded. Preparation does not start a run.</p><p class="hint" data-i18n="history.evidenceHint">The report summarizes results. Open or download screenshots and recordings from each attempt. Evidence files are not included in report downloads.</p><div id="report-content"></div></div></section>
|
|
107
108
|
</section>
|
|
108
109
|
<section id="view-settings" class="view" hidden>
|
|
109
110
|
<div class="page-heading"><p class="eyebrow" data-i18n="settings.eyebrow">04 / CONNECTIONS</p><h1 data-i18n="nav.settings">Connections</h1><p data-i18n="settings.intro">Configure run targets and models separately. Key values are read only from server environment variables.</p></div>
|
|
@@ -126,6 +126,17 @@ button:focus-visible, input:focus-visible, select:focus-visible, textarea:focus-
|
|
|
126
126
|
.target-field { min-width: 0; }
|
|
127
127
|
.target-field select { min-height: 45px; }
|
|
128
128
|
.execution-confirm { display: flex; flex-direction: column; min-width: 0; }
|
|
129
|
+
.case-selection { min-width: 0; margin-bottom: 16px; padding-bottom: 16px; border-bottom: 1px solid var(--line); }
|
|
130
|
+
.case-selection-head { display: flex; align-items: baseline; justify-content: space-between; flex-wrap: wrap; gap: 4px 12px; }
|
|
131
|
+
.case-selection-head h3 { margin: 0; font-size: 14px; }
|
|
132
|
+
#case-selection-summary { color: var(--muted); font-size: 12px; }
|
|
133
|
+
.case-selection .actions { margin: 12px 0; }
|
|
134
|
+
#case-selection-list { display: grid; gap: 0; max-height: 250px; overflow-y: auto; border: 1px solid var(--line); border-radius: var(--radius-control); }
|
|
135
|
+
.case-selection-item { display: flex; align-items: flex-start; gap: 10px; min-width: 0; padding: 10px 12px; border-bottom: 1px solid var(--line); cursor: pointer; font-size: 13px; }
|
|
136
|
+
.case-selection-item:last-child { border-bottom: 0; }
|
|
137
|
+
.case-selection-item input { width: 18px; height: 18px; flex: none; margin: 1px 0 0; accent-color: var(--accent-solid); }
|
|
138
|
+
.case-selection-item span { min-width: 0; overflow-wrap: anywhere; }
|
|
139
|
+
.selection-scope { color: var(--accent-text) !important; font-weight: 700; }
|
|
129
140
|
.review-line { display: flex; align-items: flex-start; gap: 10px; font-size: 13px; line-height: 1.55; cursor: pointer; }
|
|
130
141
|
.review-line input { width: 18px; height: 18px; flex: none; margin: 2px 0 0; accent-color: var(--accent-solid); }
|
|
131
142
|
.run-bar { display: flex; align-items: end; justify-content: space-between; flex-wrap: wrap; gap: 12px; margin-top: auto; padding-top: 18px; }
|
package/dist/src/ui/server.js
CHANGED
|
@@ -8,6 +8,7 @@ import { z } from 'zod';
|
|
|
8
8
|
import { createPreferenceScript } from '@hyeonse/design-system/preferences';
|
|
9
9
|
import { chromium } from 'playwright';
|
|
10
10
|
import { compileSuite } from "../compiler.js";
|
|
11
|
+
import { CaseSelectionError, isReportSelectionValid, selectCases } from "../case-selection.js";
|
|
11
12
|
import { runSuite } from "../engine.js";
|
|
12
13
|
import { localeFromAcceptLanguage, localizeProductMessage } from "../i18n.js";
|
|
13
14
|
import { EvidenceError, readEvidence } from "./evidence.js";
|
|
@@ -38,7 +39,8 @@ const reportEnvelope = z.object({
|
|
|
38
39
|
cases: z.array(z.object({ id: z.string(), title: z.string(), status: z.enum(['passed', 'failed', 'flaky', 'blocked', 'skipped', 'error']),
|
|
39
40
|
expected: z.array(z.string()), attempts: z.array(z.unknown()), reason: z.string().optional() }).passthrough()),
|
|
40
41
|
errors: z.array(z.string()), usage: z.object({ modelTokens: z.number(), modelCachedTokens: z.number().optional(), estimatedCostUsd: z.number().optional() }).optional(),
|
|
41
|
-
|
|
42
|
+
selection: z.unknown().optional(),
|
|
43
|
+
}).passthrough().refine((report) => isReportSelectionValid(report.cases, report.selection));
|
|
42
44
|
const PUBLIC_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), 'public');
|
|
43
45
|
const STATIC_FILES = {
|
|
44
46
|
'/': { file: path.join(PUBLIC_DIR, 'index.html'), type: 'text/html; charset=utf-8' },
|
|
@@ -72,7 +74,7 @@ const SCHEMA_FIELDS = new Set([
|
|
|
72
74
|
'requirementIds', 'preconditions', 'questions', 'steps', 'type', 'path', 'instruction',
|
|
73
75
|
'locator', 'by', 'value', 'name', 'visible', 'expected', 'match', 'flow', 'role', 'url',
|
|
74
76
|
'description', 'baseURL', 'model', 'apiKeyEnv', 'clear', 'targetName', 'targetURL', 'targetFingerprint', 'suiteHash',
|
|
75
|
-
'kind', 'platform', 'bundleId', 'device', 'direction',
|
|
77
|
+
'kind', 'platform', 'bundleId', 'device', 'direction', 'caseIds',
|
|
76
78
|
]);
|
|
77
79
|
const KNOWN_SCHEMA_MESSAGES = new Map([
|
|
78
80
|
['IDs must be unique.', 'IDs must be unique.'],
|
|
@@ -402,7 +404,7 @@ export async function startUIServer(options) {
|
|
|
402
404
|
history.push({ runId: item.runId, suiteId: item.suiteId, status: item.status, startedAt: item.startedAt,
|
|
403
405
|
finishedAt: item.finishedAt, caseCount: item.cases.length,
|
|
404
406
|
passedCount: item.cases.filter((entry) => entry.status === 'passed' || entry.status === 'flaky').length,
|
|
405
|
-
target: item.target });
|
|
407
|
+
target: item.target, ...(item.selection ? { selection: item.selection } : {}) });
|
|
406
408
|
}
|
|
407
409
|
catch { /* incomplete or unsafe run */ }
|
|
408
410
|
}
|
|
@@ -469,6 +471,59 @@ export async function startUIServer(options) {
|
|
|
469
471
|
sendJSON(response, 200, await report(runId, true, locale));
|
|
470
472
|
return;
|
|
471
473
|
}
|
|
474
|
+
if (url.pathname === '/api/rerun') {
|
|
475
|
+
const runId = url.searchParams.get('runId');
|
|
476
|
+
if (!runId)
|
|
477
|
+
invalid('A run ID is required.');
|
|
478
|
+
const { report: previous } = await report(runId, false);
|
|
479
|
+
const matches = (await suites()).filter((item) => item.id === previous.suiteId);
|
|
480
|
+
if (matches.length === 0)
|
|
481
|
+
throw new HTTPError(409, 'No saved case file matches this report.');
|
|
482
|
+
if (matches.length > 1)
|
|
483
|
+
throw new HTTPError(409, 'Multiple saved case files match this report.');
|
|
484
|
+
const file = matches[0].file;
|
|
485
|
+
let selected;
|
|
486
|
+
try {
|
|
487
|
+
selected = await getSuite(file, true);
|
|
488
|
+
}
|
|
489
|
+
catch (error) {
|
|
490
|
+
if (error instanceof HTTPError && error.status === 404) {
|
|
491
|
+
throw new HTTPError(409, 'The saved case file or source flow was not found.');
|
|
492
|
+
}
|
|
493
|
+
if (error instanceof HTTPError && error.status === 400) {
|
|
494
|
+
throw new HTTPError(409, 'The saved case file or source flow is invalid.');
|
|
495
|
+
}
|
|
496
|
+
throw error;
|
|
497
|
+
}
|
|
498
|
+
const { suite } = selected;
|
|
499
|
+
if (sha256(`${JSON.stringify(suite, null, 2)}\n`) !== previous.suiteHash) {
|
|
500
|
+
throw new HTTPError(409, 'The saved case file changed since this run. Review it before rerunning.');
|
|
501
|
+
}
|
|
502
|
+
const available = new Set(suite.cases.map((item) => item.id));
|
|
503
|
+
if (previous.cases.some((item) => !item.id.trim() || !available.has(item.id)) ||
|
|
504
|
+
new Set(previous.cases.map((item) => item.id)).size !== previous.cases.length ||
|
|
505
|
+
(!previous.selection && previous.cases.length !== suite.cases.length)) {
|
|
506
|
+
throw new HTTPError(409, 'The report cases do not match the saved case file.');
|
|
507
|
+
}
|
|
508
|
+
if (previous.selection && (previous.selection.totalCases !== suite.cases.length ||
|
|
509
|
+
!isReportSelectionValid(previous.cases, previous.selection))) {
|
|
510
|
+
throw new HTTPError(409, 'The report selection does not match the saved case file.');
|
|
511
|
+
}
|
|
512
|
+
const failing = new Set(previous.cases.filter((item) => item.status === 'failed' || item.status === 'error').map((item) => item.id));
|
|
513
|
+
const caseIds = suite.cases.filter((item) => failing.has(item.id)).map((item) => item.id);
|
|
514
|
+
if (caseIds.length === 0)
|
|
515
|
+
throw new HTTPError(409, 'No failed or errored cases are available to rerun.');
|
|
516
|
+
const settings = await config();
|
|
517
|
+
const targetName = previous.target.name;
|
|
518
|
+
const expectedFingerprint = targetFingerprint(previous.target);
|
|
519
|
+
const currentTarget = Object.hasOwn(settings.targets, targetName) ? settings.targets[targetName] : undefined;
|
|
520
|
+
sendJSON(response, 200, { originRunId: runId, file, ...selected, caseIds, targetName,
|
|
521
|
+
targetFingerprint: expectedFingerprint,
|
|
522
|
+
targetMatches: Boolean(currentTarget &&
|
|
523
|
+
(suite.platform ?? 'web') === (currentTarget.kind === 'mobile' ? 'mobile' : 'web') &&
|
|
524
|
+
targetFingerprint(currentTarget) === expectedFingerprint) });
|
|
525
|
+
return;
|
|
526
|
+
}
|
|
472
527
|
if (url.pathname === '/api/comparison') {
|
|
473
528
|
const baseline = url.searchParams.get('baseline');
|
|
474
529
|
const current = url.searchParams.get('current');
|
|
@@ -708,14 +763,24 @@ export async function startUIServer(options) {
|
|
|
708
763
|
if (url.pathname === '/api/run') {
|
|
709
764
|
enterMutation();
|
|
710
765
|
try {
|
|
711
|
-
const { suite: file, suiteHash, targetName, targetURL, targetFingerprint: reviewedFingerprint } = parse(z.object({
|
|
766
|
+
const { suite: file, suiteHash, targetName, targetURL, targetFingerprint: reviewedFingerprint, caseIds } = parse(z.object({
|
|
712
767
|
suite: z.string(), suiteHash: z.string().regex(/^[a-f0-9]{64}$/), targetName: idSchema,
|
|
713
768
|
targetURL: z.string().optional(), targetFingerprint: z.string().regex(/^[a-f0-9]{64}$/).optional(),
|
|
769
|
+
caseIds: z.array(idSchema).min(1).max(100).optional(),
|
|
714
770
|
}).strict(), body);
|
|
715
771
|
const selected = await getSuite(file, true);
|
|
716
772
|
if (selected.suiteHash !== suiteHash)
|
|
717
773
|
throw new HTTPError(409, 'The reviewed case file changed. Open it again to review.');
|
|
718
774
|
const { suite } = selected;
|
|
775
|
+
let selectedIds;
|
|
776
|
+
try {
|
|
777
|
+
selectedIds = selectCases(suite, caseIds).selection?.caseIds;
|
|
778
|
+
}
|
|
779
|
+
catch (error) {
|
|
780
|
+
if (error instanceof CaseSelectionError)
|
|
781
|
+
invalid('Select one or more unique cases from this suite.');
|
|
782
|
+
throw error;
|
|
783
|
+
}
|
|
719
784
|
const settings = await config();
|
|
720
785
|
const target = settings.targets[targetName];
|
|
721
786
|
if (!target)
|
|
@@ -739,7 +804,8 @@ export async function startUIServer(options) {
|
|
|
739
804
|
lastError = undefined;
|
|
740
805
|
const suiteSnapshot = structuredClone(suite);
|
|
741
806
|
const configSnapshot = structuredClone(settings);
|
|
742
|
-
runTask = Promise.resolve().then(() => run({ workspace, suite: suiteSnapshot, config: configSnapshot, targetName, signal: abort.signal
|
|
807
|
+
runTask = Promise.resolve().then(() => run({ workspace, suite: suiteSnapshot, config: configSnapshot, targetName, signal: abort.signal,
|
|
808
|
+
...(selectedIds ? { caseIds: [...selectedIds] } : {}) }))
|
|
743
809
|
.then(() => undefined)
|
|
744
810
|
.catch(() => { lastError = 'Could not complete the run. Check the local records.'; })
|
|
745
811
|
.finally(() => { active = null; controller = undefined; busy = false; runTask = undefined; });
|
package/docs/architecture.md
CHANGED
|
@@ -22,7 +22,9 @@ flowchart LR
|
|
|
22
22
|
|
|
23
23
|
**Only a defined vocabulary compiles.** The schema checks requirement references, assertions, step types, and inputs. It does not execute arbitrary code or shell commands from model output. Cases with unresolved `questions`, or steps that need an unavailable agent model, are blocked. `preconditions` describe required state; they are not an automatic setup contract.
|
|
24
24
|
|
|
25
|
-
**Source and evidence stay linked.** Planning records the source flow's SHA-256 hash in the suite. `validate`, `generate`, and `run` check it again and stop when the flow changed. Each execution writes a new directory with a suite snapshot, generated tests, raw engine output, and Markdown/JSON reports. Previous evidence is not reused as a current success. `compare` permits a direct comparison only for the same suite hash and target identity (web URL or native platform/app/device); it cannot establish that external data was unchanged.
|
|
25
|
+
**Source and evidence stay linked.** Planning records the source flow's SHA-256 hash in the suite. `validate`, `generate`, and `run` check it again and stop when the flow changed. Each execution writes a new directory with a suite snapshot, generated tests, raw engine output, and Markdown/JSON reports. Previous evidence is not reused as a current success. CLI `compare` permits a direct comparison only for the same suite ID, suite hash, case selection, and target identity (web URL or native platform/app/device); it cannot establish that external data was unchanged.
|
|
26
|
+
|
|
27
|
+
**Selection changes execution scope, not source identity.** The full suite and source are validated and the full original snapshot/hash remain in each run directory. Selected cases and their referenced requirements form the executable subset. Partial reports carry explicit selection metadata and contain only selected results, including blocked, error, and interrupted results. UI comparisons flag partial/differing scopes; missing results never imply a pass. Selection changes require fresh UI review and cannot bypass the target or file fingerprint checks.
|
|
26
28
|
|
|
27
29
|
**Targets can be configured after authoring.** A suite's optional platform distinguishes web from mobile. Omitted platform preserves legacy web suites and their hashes. Native app IDs and explicit device names belong to target settings, not generated code from the planner. UI execution checks a fingerprint of the reviewed target identity. Mobile runtime readiness is checked before device execution; configuration alone does not prove connectivity.
|
|
28
30
|
|
package/docs/mobile.md
CHANGED
|
@@ -9,7 +9,7 @@ Choose **Mobile** on the scenario screen, describe the flow, and generate or dow
|
|
|
9
9
|
With the installed CLI:
|
|
10
10
|
|
|
11
11
|
```sh
|
|
12
|
-
npm install -g @ihyeon/flow-qa@0.
|
|
12
|
+
npm install -g @ihyeon/flow-qa@0.4.0
|
|
13
13
|
flow-qa init ./my-qa
|
|
14
14
|
# Write your mobile flow in my-qa/flows/my-flow.md first.
|
|
15
15
|
flow-qa plan flows/my-flow.md --platform mobile --prompt-only --workspace ./my-qa
|
package/docs/releases.md
CHANGED
|
@@ -2,6 +2,28 @@
|
|
|
2
2
|
|
|
3
3
|
The alpha entries below are historical release records. They document earlier package versions and checks.
|
|
4
4
|
|
|
5
|
+
## 0.5.0: prepare failed cases from a report
|
|
6
|
+
|
|
7
|
+
Adds **Prepare failed/error cases** to detailed reports. It loads the unchanged saved suite, selects only recorded `failed` and `error` cases in suite order, and opens Scenario with a fresh review requirement. Passed, flaky, blocked, and skipped cases are excluded. Preparation is read-only and never starts a run or calls a model. Unsaved drafts require confirmation; failed preparation and late responses after navigation preserve the draft.
|
|
8
|
+
|
|
9
|
+
Preparation requires one matching saved suite, the exact parsed suite snapshot hash, a valid linked source, and coherent full/partial report IDs. Missing, changed, or ambiguous inputs stop preparation with guidance. A changed or missing target is cleared for explicit selection; even a matching target requires renewed review. Existing execution-time checks remain in force. The next run writes its own ordinary selected-scope report; the original report remains unchanged, and no persistent parent-run link is added.
|
|
10
|
+
|
|
11
|
+
Locally verified on macOS with Node.js 24.22.1 on 2026-11-03: 117 tests, type checking, UI syntax and design-system audit, and demo/UI checks passed. The dedicated Chromium smoke covered all six case statuses, unsaved-draft acceptance/cancellation, changed suite and target, navigation during a delayed response, English/Korean, and a 390px layout. It executed two prepared cases successfully with zero requests from the excluded sentinel. A fresh tarball installation prepared a synthetic failed report and ran that selection in a real browser, alongside full-run/comparison checks (seven passing browser cases). The existing import smoke now waits for save completion before asserting download readiness. No model calls, external logins, or native-device runs were used.
|
|
12
|
+
|
|
13
|
+
- Package: [`@ihyeon/flow-qa@0.5.0`](https://www.npmjs.com/package/@ihyeon/flow-qa/v/0.5.0)
|
|
14
|
+
- Release: [GitHub 0.5.0](https://github.com/hyeonse-swing/flow-qa/releases/tag/v0.5.0)
|
|
15
|
+
|
|
16
|
+
## 0.4.0: run selected cases
|
|
17
|
+
|
|
18
|
+
Adds case selection to the execution panel and repeated `--case <id>` flags to CLI runs. Selecting or clearing cases resets UI review, and execution still requires the reviewed file and target fingerprints. Unknown, duplicate, and empty selections are rejected. Selection is preserved across navigation, language changes, and polling, while a new draft starts with all cases selected.
|
|
19
|
+
|
|
20
|
+
Each run retains the full original suite snapshot/hash and verifies its source, while compiling and assessing only the chosen cases. Partial reports add explicit `selection` metadata to `flow-qa/report-1`; unselected cases are neither executed nor reported as passed/skipped. Selected blocked/error/interrupted cases retain their normal meaning. History, report downloads, and comparisons display the partial scope; comparisons warn when selected case sets differ. Existing full runs and their report shape are preserved.
|
|
21
|
+
|
|
22
|
+
Locally verified on macOS with Node.js 24.22.1 on 2026-11-03: 115 tests, type checking, UI syntax and design-system audit, and existing demo/UI checks passed. The dedicated Chromium smoke ran two selected cases successfully, confirmed zero requests from an excluded sentinel case, and kept a selected unresolved-question case blocked without attempts. It covered review invalidation, bulk selection, navigation/language preservation, 390px layout, original snapshot/hash preservation, and report/comparison scope. A fresh tarball install executed two full runs plus a one-case partial run (seven passing browser cases) and compared their reports. No model calls or native-device execution were used.
|
|
23
|
+
|
|
24
|
+
- Package: [`@ihyeon/flow-qa@0.4.0`](https://www.npmjs.com/package/@ihyeon/flow-qa/v/0.4.0)
|
|
25
|
+
- Release: [GitHub 0.4.0](https://github.com/hyeonse-swing/flow-qa/releases/tag/v0.4.0)
|
|
26
|
+
|
|
5
27
|
## 0.3.0: compare recorded runs
|
|
6
28
|
|
|
7
29
|
Adds a two-run comparison workspace to Run history. Select a baseline and current report, swap their direction, filter by case transition, open either detailed report, and export the complete comparison as Markdown or JSON. Cases are matched by ID, with distinct categories for passed-to-failed, failed-to-passed, other changes, equal statuses, added, and missing. Flaky, blocked, skipped, and execution-error results retain their meaning.
|
|
@@ -111,7 +133,7 @@ Locally verified on macOS with Node.js 24.14.1 on 2026-10-12: 89 tests, type che
|
|
|
111
133
|
|
|
112
134
|
A push to `main` publishes only when `package.json` changes to a new plain `X.Y.Z` version. Pull requests and pushes without a version change run verification without publishing. The package and lockfile versions must agree; the version must increase, and registry errors stop the release. Rerunning a successful publication skips the existing immutable npm version.
|
|
113
135
|
|
|
114
|
-
The `publish` job in [ci.yml](../.github/workflows/ci.yml) waits for the full verification matrix on Ubuntu and macOS, then publishes through npm trusted publishing (OIDC). It uses the `latest` tag, requires the registry's `gitHead` to match the workflow commit, and checks the published package with a fresh consumer installation, two
|
|
136
|
+
The `publish` job in [ci.yml](../.github/workflows/ci.yml) waits for the full verification matrix on Ubuntu and macOS, then publishes through npm trusted publishing (OIDC). It uses the `latest` tag, requires the registry's `gitHead` to match the workflow commit, and checks the published package with a fresh consumer installation, two full Chromium demo runs plus a one-case selected run prepared from a synthetic failed report (seven cases), and comparison of their saved reports. The commit check also applies when retrying an already published version. No npm token is stored in GitHub secrets. After the published package passes its consumer smoke check, a separate job creates the matching GitHub tag and release at the exact verified commit. It preserves existing releases and rejects a tag pointing to another commit. Rerunning an older release does not replace a newer latest release.
|
|
115
137
|
|
|
116
138
|
For the next release, update the release notes and current installation examples, then bump the version before committing and pushing:
|
|
117
139
|
|
package/docs/ui.md
CHANGED
|
@@ -7,7 +7,7 @@ Describe a user flow, review cases, and configure a target when ready to run. We
|
|
|
7
7
|
Node.js 24.x (24.22.1 or later) or 26.12.1 or later is required. Node 25 is unsupported. Upgrade the Node runtime and restart any running UI process; updating Flow QA alone does not patch Node. Install Flow QA and start the web demo:
|
|
8
8
|
|
|
9
9
|
```sh
|
|
10
|
-
npm install -g @ihyeon/flow-qa@0.
|
|
10
|
+
npm install -g @ihyeon/flow-qa@0.4.0
|
|
11
11
|
flow-qa browser install
|
|
12
12
|
flow-qa ui --demo
|
|
13
13
|
```
|
|
@@ -46,6 +46,16 @@ From version `0.2.2`, the UI requests the browser's leave-page confirmation when
|
|
|
46
46
|
|
|
47
47
|
From version `0.2.4`, loading a suite from the library or importing valid case files also asks before replacing unsaved scenario or case content. **New JSON draft** asks before clearing unsaved JSON and keeps the scenario text. Cancel keeps the current content and selected import files. Unchanged saved cases and ordinary navigation between screens do not prompt; invalid imports still show their validation error without replacing anything.
|
|
48
48
|
|
|
49
|
+
### Run selected cases
|
|
50
|
+
|
|
51
|
+
From version `0.4.0`, **Scenario → Review, then run → Cases to run** lists the validated suite's cases. Initially all are selected. Use individual checkboxes, **Select all**, or **Clear selection** to choose a nonempty set. A changed selection clears the review confirmation; choose the target, inspect the selected cases, and confirm review again before execution. Selection controls are disabled during a run or another busy operation. Navigation, language changes, and polling keep the selection. Loading/importing/validating a new draft resets it to all cases. It is not saved across page reloads.
|
|
52
|
+
|
|
53
|
+
The selected cases alone determine runnable/blocked counts and the run result. A selected case with unresolved questions or an unavailable required model stays blocked. Unselected cases do not execute and are not reported as passed or skipped. Flow QA does not infer prerequisites: each selected case must establish its own starting state. The source flow and full saved suite still pass existing validation and review-hash checks. Selection does not edit the saved file; the script preview and script/case downloads continue to include the full suite.
|
|
54
|
+
|
|
55
|
+
The run directory keeps the complete original `suite.json` and its hash. A partial `qa-report.json` retains `flow-qa/report-1` and adds `selection: { caseIds, totalCases }`; its `cases` list contains exactly the selected IDs in original suite order. A full run, including explicitly selecting every case, omits this field as before. History, detailed reports, and Markdown exports show selected/total counts and explain that omitted cases were not assessed. Comparison exports retain each run's scope, warn on partial runs and differing selections, and describe absent cases as missing results rather than recovered defects. Invalid selection metadata is not accepted as a valid report.
|
|
56
|
+
|
|
57
|
+
For the CLI, repeat `--case`: `flow-qa run cases/task-flow.json --target demo --case TC-001 --case TC-003`. Unknown, duplicate, or empty IDs are rejected before execution. The command keeps existing exit semantics: a partial pass is exit 0, a failure/block is exit 1, interrupted is 130, and invalid input is 2. If CI requires complete-suite coverage, omit `--case` and check that report selection metadata is absent.
|
|
58
|
+
|
|
49
59
|
### Expand or collapse case review
|
|
50
60
|
|
|
51
61
|
From version `0.2.8`, **Expand all cases** and **Collapse all cases** control the case details in Scenario. Individual case titles still toggle their own details. Language changes and navigation preserve which cases are open; loading a new suite keeps the original first-case-open default. These controls only change the view: they do not approve cases, modify the draft, or change the JSON editor disclosure. Review confirmation is still required before execution.
|
|
@@ -62,6 +72,16 @@ From version `0.2.7`, **Run history → Run status** filters the loaded reports
|
|
|
62
72
|
|
|
63
73
|
From version `0.2.9`, **Run history → Search runs** matches part of a suite ID, run ID, or target name. Matching ignores letter case and surrounding whitespace, and supports Korean target names. Search combines with **Run status**; clear the search to return to the selected status. The query stays through navigation, language changes, and automatic report updates, and resets on page reload. Search covers the most recent 50 reports loaded by the UI, not older records, target URLs, or individual case content. Changing the query keeps the open report and its downloads available.
|
|
64
74
|
|
|
75
|
+
### Prepare failed cases
|
|
76
|
+
|
|
77
|
+
From `0.5.0`, open **Run history → Detailed report → Prepare failed/error cases**. Only cases whose recorded status is `failed` or `error` are selected; `passed`, `flaky`, `blocked`, and `skipped` are excluded. The button is unavailable when no eligible cases exist or an operation is active. Preparation does not execute tests, call a model, or change files.
|
|
78
|
+
|
|
79
|
+
The UI finds exactly one saved case file with the report's suite ID, verifies its complete parsed content against the run's snapshot hash, and verifies its linked source Markdown. Missing, changed, ambiguous, or unreadable inputs stop preparation without replacing your draft. Open the current case file from the library and review it manually when the test definition has changed. An unsaved draft requires confirmation before replacement.
|
|
80
|
+
|
|
81
|
+
Successful preparation opens Scenario with the original source and failed/error selection. The prior target is selected only when its current name and settings still match the recorded target. Otherwise the target is cleared for explicit selection. Review confirmation is always cleared; inspect the selected cases and target before pressing **Run QA**. The normal execution-time file, source, and target checks still apply. Each selected case must establish its own starting state. Mobile target matching compares configured selectors, not physical device identity.
|
|
82
|
+
|
|
83
|
+
The preparation notice identifies the original run. The next execution creates a separate ordinary report with its selected scope; it does not overwrite the original report or store a persistent parent-run link. Selection stays through navigation, language changes, and polling, but resets on page reload.
|
|
84
|
+
|
|
65
85
|
### Compare two runs
|
|
66
86
|
|
|
67
87
|
From version `0.3.0`, open **Run history → Compare two runs**, choose a **Baseline run** and **Current run**, then compare. Both selectors use the most recent 50 loaded reports, independently of the history search and status filter. The reports must belong to the same suite ID and recorded target settings: a matching web URL, or mobile platform, app ID, and device. Target display names may differ. Mobile reports record the configured device selector, which can be a name; all mobile comparisons warn that matching selectors do not establish that the same physical device was used. Comparing does not run tests, call a model, or change case review.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ihyeon/flow-qa",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.5.0",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"author": "Hyeonse Im",
|
|
6
6
|
"repository": {
|
|
@@ -57,7 +57,7 @@
|
|
|
57
57
|
"check:ui": "node --check src/ui/public/app.js && node --check src/ui/public/display.js && node --check src/ui/public/i18n.js && hds-audit .",
|
|
58
58
|
"demo:serve": "node examples/demo/server.ts",
|
|
59
59
|
"demo:smoke": "node scripts/demo-smoke.ts",
|
|
60
|
-
"smoke:ui": "node scripts/ui-i18n-smoke.mjs && node scripts/ui-comparison-smoke.mjs"
|
|
60
|
+
"smoke:ui": "node scripts/ui-i18n-smoke.mjs && node scripts/ui-comparison-smoke.mjs && node scripts/ui-selection-smoke.mjs && node scripts/ui-rerun-smoke.mjs"
|
|
61
61
|
},
|
|
62
62
|
"dependencies": {
|
|
63
63
|
"@ai-sdk/openai-compatible": "3.0.53",
|