pipwave-ekyc-uikit 3.3.0 → 3.3.1-beta.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -21,7 +21,41 @@
21
21
  "Bash(ffprobe -v error -select_streams a -show_entries stream=codec_name -of default=noprint_wrappers=1 \"/Users/chechin/Desktop/Screen Recording 2026-08-12 at 7.27.54 AM.mov\")",
22
22
  "Bash(grep -i -E \"\\\\.\\(mov|mp4|m4v\\)$\")",
23
23
  "Bash(~/.claude/skills/compress-video/compress-video.sh --help)",
24
- "mcp__claude_ai_G2G_Shortcut__stories-get-by-id"
24
+ "mcp__claude_ai_G2G_Shortcut__stories-get-by-id",
25
+ "Bash(python3 -c ' *)",
26
+ "Bash(python3 -)",
27
+ "mcp__claude_ai_Figma__get_screenshot",
28
+ "Bash(cd \"/Users/chechin/Documents/2. pipwave Projects/pw-ekyc-web-ui /pw-ekyc-web-ui\" && sed -n '/^### Overlap resolution/,/^---/p' docs/plan-epic-342056-selfie-action-checking.md)",
29
+ "Bash(mv \"support-facematch-selfie-fields-in-web-sdk.md\" \"a-support-facematch-selfie-fields-in-web-sdk.md\")",
30
+ "Bash(mv \"selfie-flow-and-orchestration.md\" \"b-selfie-flow-and-orchestration.md\")",
31
+ "Bash(python3 -c \"print\\('A .5 + B1 2.5 + C 3 + D 3 + E 3.25 + F 5.25 + G 1.5 + B2 3 =', 0.5+2.5+3+3+3.25+5.25+1.5+3\\)\")",
32
+ "Bash(grep -n \"story \\\\*\\\\*B\\\\*\\\\*\\\\|\\(B\\)\\\\b\\\\|B's mocks\\\\|B's debug\\\\|B's \\\\`\" docs/plan-epic-342056-selfie-action-checking.md)",
33
+ "Bash(grep -n \"^#\\\\{2,4\\\\} \" plan-epic-342056-selfie-action-checking.md)",
34
+ "Bash(grep -v \"are$\\\\|dropped\\\\|superseded\")",
35
+ "Bash(cd \"/Users/chechin/Documents/2. pipwave Projects/pw-ekyc-web-ui /pw-ekyc-web-ui/docs\" && echo \"=== §7.0 refs ===\" && sed -n '1532,1533p' plan-epic-342056-selfie-action-checking.md | grep -o \"\\\\*\\\\*[FG]\\\\*\\\\*[^.]*\"; echo; echo \"=== §6.3 sequencing prose ===\" && sed -n '/^### 6.3 Sequencing/,/^### 6.4/p' plan-epic-342056-selfie-action-checking.md | grep -n \"\\\\*\\\\*[FG]\\\\*\\\\*\\\\|── [FG] \\\\|[FG] visibility\\\\|[FG] sequencer\" | sed 's/\\\\\\(.\\\\{130\\\\}\\\\\\).*/\\\\1…/')",
36
+ "Bash(cd \"/Users/chechin/Documents/2. pipwave Projects/pw-ekyc-web-ui /pw-ekyc-web-ui/docs\" && sed -n '/^### 6.3 Sequencing/,/^### 6.4/p' plan-epic-342056-selfie-action-checking.md | grep -n \"\\\\bF\\\\b\\\\|\\\\bG\\\\b\" | sed 's/\\\\\\(.\\\\{150\\\\}\\\\\\).*/\\\\1…/')",
37
+ "mcp__claude_ai_G2G_Shortcut__stories-search",
38
+ "mcp__claude_ai_Figma__get_metadata",
39
+ "Bash(cd \"/Users/chechin/Documents/2. pipwave Projects/pw-ekyc-web-ui /pw-ekyc-web-ui/docs\" && sed -n '/Invalidate with a dirty flag/,/^### /p' plan-epic-342056-selfie-action-checking.md | sed -n '8,30p')",
40
+ "Bash(cd \"/Users/chechin/Documents/2. pipwave Projects/pw-ekyc-web-ui /pw-ekyc-web-ui/docs/..\" && grep -rn \"requestAnimationFrame\\\\|cancelAnimationFrame\" src/plugins/*.ts | sed 's/\\\\\\(.\\\\{110\\\\}\\\\\\).*/\\\\1…/')",
41
+ "Read(//Users/chechin/Documents/2. pipwave Projects/pw-ekyc-web-ui /pw-ekyc-web-ui/**)",
42
+ "Bash(— a *)",
43
+ "Bash(grep -n \"config.ts` — a skeleton\\\\|config.ts` — skeleton\" stories/b-selfie-flow-scaffold.md plan-epic-342056-selfie-action-checking.md)",
44
+ "Bash(cd \"/Users/chechin/Documents/2. pipwave Projects/pw-ekyc-web-ui /pw-ekyc-web-ui\" && grep -n \"^_Rev 2[4-9]\" docs/plan-epic-342056-selfie-action-checking.md | head && sed -n '/^_Rev 26/,/^_Rev 25/p' docs/plan-epic-342056-selfie-action-checking.md | head -30)",
45
+ "Bash(awk '/^\\\\| \\\\| Shortcut \\\\| Draft/,/^$/' docs/plan-epic-342056-selfie-action-checking.md)",
46
+ "Bash(awk -F'|' '{print NF-2\" cols: \"$2}')",
47
+ "Bash(perl -i -pe 'if \\(/^_/\\) { s/the flow scaffold story/the flow scaffold story \\(sc-354838\\)/; s/the flow scaffold\\(?! story\\)\\(?! \\\\\\(sc\\)/the flow scaffold \\(sc-354838\\)/; }' docs/stories/c-selfie-camera-screen-ui.md docs/stories/d-selfie-non-camera-screens.md docs/stories/f-visibility-prompts-and-fallback.md docs/stories/h-selfie-submission-and-result.md docs/stories/tickets/c-selfie-camera-screen-ui.md docs/stories/tickets/d-selfie-non-camera-screens.md docs/stories/tickets/f-visibility-prompts-and-fallback.md docs/stories/tickets/h-selfie-submission-and-result.md)",
48
+ "Bash(perl -i -pe 's/^As a Frontend Developer, I want the SDK package to accept the facematch selfie fields, so that the UI can submit action-stage images\\\\.$/As a Frontend Developer, I want the pw-ekyc-web-sdk to accept the facematch selfie fields, so that the pw-ekyc-web-ui can submit action-stage images./' docs/stories/a-support-facematch-selfie-fields-in-web-sdk.md docs/stories/tickets/a-web-sdk-facematch-selfie-fields.md)",
49
+ "Bash(awk '/^# Documents$/{f=1} f' e-signal-package-adapter-and-geometry.md)",
50
+ "Bash(awk '/^# Documents$/{f=1} f' g-action-sequencer-and-capture.md)",
51
+ "Bash(perl -0777 -i -pe ' *)",
52
+ "Bash(perl -0777 -i -pe 's/- Only one image is captured and uploaded; the new flow sends three\\\\./- Only one image is captured and uploaded; the new flow sends 3./' g-action-sequencer-and-capture.md)",
53
+ "mcp__claude_ai_G2G_Shortcut__stories-create",
54
+ "mcp__claude_ai_G2G_Shortcut__stories-update",
55
+ "Bash(set -e)"
56
+ ],
57
+ "additionalDirectories": [
58
+ "/Users/chechin/Documents/2. pipwave Projects/pw-ekyc-web-ui /pw-ekyc-web-ui/docs"
25
59
  ]
26
60
  }
27
61
  }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pipwave-ekyc-uikit",
3
- "version": "3.3.0",
3
+ "version": "3.3.1-beta.1",
4
4
  "description": "",
5
5
  "main": "dist/pw-bundle.js",
6
6
  "module": "dist/pw-bundle.mjs",
@@ -1,463 +0,0 @@
1
- # What this branch does (plain English + code)
2
-
3
- **Branch:** `feature-teojunjie-sc-349453-fix-passport-detection`
4
- **Story:** sc-349453 — passport detection failing
5
- **Own work:** one commit, `6337605`, plus a merge of `master` (hotfix 3.2.2) to stay current.
6
- **Files touched:** [card-detection.ts](../src/plugins/card-detection.ts), [card-detection.inference.ts](../src/plugins/card-detection.inference.ts), and their two test files. Nothing else.
7
-
8
- Every snippet below is the code as it stands on this branch, with a link to the live line numbers.
9
-
10
- ---
11
-
12
- ## The problem it fixes
13
-
14
- When a user tried to capture a **passport**, the app could never let them take the photo. Two rules had to pass at the same time, and no way of holding the passport satisfied both.
15
-
16
- Both rules live in `_evaluateBox` — [card-detection.ts:548-603](../src/plugins/card-detection.ts#L548-L603):
17
-
18
- ```ts
19
- // Rule 1: is this a document at all? (confidence gate, with hysteresis)
20
- const gate = this.cardPresent ? CONF_EXIT : CONF_ENTER; // 0.40 once locked on, else 0.55
21
- if (!box || conf <= gate) {
22
- this.cardPresent = false;
23
- this._trace('no-card', conf, null);
24
- return 'no-card';
25
- }
26
-
27
- // Rule 2a: does it fill enough of the guide? (too far away → low fill)
28
- const inter = Math.max(0, Math.min(box.x2, gx2) - Math.max(box.x1, gx1))
29
- * Math.max(0, Math.min(box.y2, gy2) - Math.max(box.y1, gy1));
30
- const fill = inter / gArea;
31
-
32
- // Rule 2b: does it stay inside the guide (+ a little slack)?
33
- const eps = CONTAINMENT_SLACK_RATIO * gMin; // 0.05
34
- const contained = box.x1 >= gx1 - eps && box.y1 >= gy1 - eps
35
- && box.x2 <= gx2 + eps && box.y2 <= gy2 + eps;
36
-
37
- if (!contained) { result = 'invalid'; reason = 'out-of-box'; }
38
- else if (fill < MIN_FILL_RATIO) { result = 'invalid'; reason = 'too-small'; } // 0.85
39
- else { result = 'valid'; reason = 'VALID'; }
40
- ```
41
-
42
- The catch was **what the model was looking at**. Before this branch, `runFrame` was handed the *entire* camera frame:
43
-
44
- ```ts
45
- // BEFORE
46
- engine.runFrame(this.videoElement, this.videoElement.videoWidth, this.videoElement.videoHeight)
47
- ```
48
-
49
- But the guide box on screen covers only about **29% of that frame**. The branch added a test that pins that number down — [card-detection.test.ts](../src/plugins/card-detection.test.ts):
50
-
51
- ```ts
52
- it("reproduces the field-of-view mismatch that caused the bug", () => {
53
- // 1280x960 stream in a 1516x759 element, guide 440x330 centred
54
- const guide = (sdk as any)._computeNormalizedGuide();
55
- expect(guide.width).toBeCloseTo(0.290, 2); // the user fills < 1/3 of what the model sees
56
- expect(guide.height).toBeCloseTo(0.290, 2);
57
- });
58
- ```
59
-
60
- So:
61
-
62
- - Hold the passport so it fills the guide → to the model it is a small object in a big picture → score below `CONF_ENTER` → **rule 1 fails**, `no-card`.
63
- - Hold it close enough to score well → it is now far too big for the guide → **rule 2b fails**, `out-of-box`.
64
-
65
- No position satisfied both. The border flipped, the shutter stayed locked, users gave up.
66
-
67
- Second, smaller problem: the old model was trained mostly on **card-shaped** documents. A passport is an open booklet with a different aspect ratio, glossy page, glare and a visible spine, so it scored poorly even when held perfectly.
68
-
69
- ---
70
-
71
- ## Change 1 — show the model a zoomed-in square around the guide
72
-
73
- A new method computes the sub-rectangle of the raw camera frame to hand the model — [card-detection.ts:665-692](../src/plugins/card-detection.ts#L665-L692):
74
-
75
- ```ts
76
- private _computeDetectCropRect(): { x: number; y: number; width: number; height: number } | null {
77
- if (!this.videoElement) return null;
78
- const VW = this.videoElement.videoWidth, VH = this.videoElement.videoHeight;
79
- if (!VW || !VH) return null;
80
-
81
- const guide = this._computeNormalizedGuide();
82
- if (!guide) return null;
83
-
84
- const marginRatio = /* debug override */ ?? CARD_DETECTION_CONFIG.DETECT_CROP_MARGIN_RATIO; // 0.35
85
-
86
- const gw = guide.width * VW; // guide size in RAW FRAME PIXELS
87
- const gh = guide.height * VH;
88
- const margin = marginRatio * Math.min(gw, gh);
89
- // Can't exceed the frame; a guide wider than the frame is short is the limiting case.
90
- const side = Math.round(Math.min(Math.max(gw, gh) + 2 * margin, VW, VH));
91
- if (side < 2) return null; // overlay collapsed mid-transition
92
-
93
- // Slide (don't shrink) to stay in frame, so a guide near an edge keeps the same scale and
94
- // only loses margin on one side.
95
- const cx = (guide.left + guide.width / 2) * VW;
96
- const cy = (guide.top + guide.height / 2) * VH;
97
- const x = Math.max(0, Math.min(Math.round(cx - side / 2), VW - side));
98
- const y = Math.max(0, Math.min(Math.round(cy - side / 2), VH - side));
99
- return { x, y, width: side, height: side };
100
- }
101
- ```
102
-
103
- Three design decisions worth noticing:
104
-
105
- **It is a square.** `runFrame` letterboxes whatever it is given into a square `INPUT_SIZE × INPUT_SIZE` tensor. Any non-square crop spends part of that tensor on grey padding — a 4:3 crop fits as 256×192 and wastes 25% of the input, making the document ~11% smaller to the model for nothing.
106
-
107
- **The margin is measured in pixels, not in the normalized fractions.** `guide.width` is a fraction of frame *width* and `guide.height` a fraction of frame *height*, so `Math.min` on those two compares different units. On a 1280×960 stream that yields 202px on one axis and 151px on the other. The code converts to pixels first (`gw`, `gh`) before taking `Math.min`. There is a regression test named after this exact trap.
108
-
109
- **The crop is strictly bigger than the guide on every side.** That is what keeps rule 2b working — if the crop hugged the guide, an oversized document would clip at the crop edge and look like a perfect fit:
110
-
111
- ```ts
112
- it("leaves room outside the guide so an oversized document still reads as spilling out", () => {
113
- expect(crop.x).toBeLessThan(guide.left * 1280);
114
- expect(crop.x + crop.width).toBeGreaterThan((guide.left + guide.width) * 1280);
115
- // …same for y
116
- });
117
- ```
118
-
119
- The crop is then passed to the engine each tick — [card-detection.ts:489-498](../src/plugins/card-detection.ts#L489-L498):
120
-
121
- ```ts
122
- const crop = this._computeDetectCropRect();
123
- this.lastCrop = crop;
124
- engine
125
- .runFrame(
126
- this.videoElement,
127
- this.videoElement.videoWidth,
128
- this.videoElement.videoHeight,
129
- this.type === CARD_TYPES.PASSPORT ? MODEL_CLASS_INDEX.PASSPORT : MODEL_CLASS_INDEX.CARD,
130
- crop,
131
- )
132
- ```
133
-
134
- And on the engine side, the crop simply changes the source rectangle of the `drawImage` call — [card-detection.inference.ts:141-166](../src/plugins/card-detection.inference.ts#L141-L166):
135
-
136
- ```ts
137
- // Absent/zero crop degrades to the full frame rather than dividing by zero.
138
- const cw = crop && crop.width > 0 ? crop.width : srcW;
139
- const ch = crop && crop.height > 0 ? crop.height : srcH;
140
- const cx0 = crop && crop.width > 0 ? crop.x : 0;
141
- const cy0 = crop && crop.height > 0 ? crop.y : 0;
142
-
143
- const scale = Math.min(N / cw, N / ch); // letterbox scale, now based on the CROP
144
- const newW = Math.round(cw * scale);
145
- const newH = Math.round(ch * scale);
146
- const padX = (N - newW) / 2;
147
- const padY = (N - newH) / 2;
148
-
149
- ctx.fillStyle = "#727272";
150
- ctx.fillRect(0, 0, N, N);
151
- ctx.drawImage(source, cx0, cy0, cw, ch, padX, padY, newW, newH);
152
- // ^^^^^^^^^^^^^^^^ source rect: was 0, 0, srcW, srcH
153
- ```
154
-
155
- Crucially, the box the model returns is translated back into full-frame coordinates before it leaves the engine, so nothing downstream needs to know a crop happened — [card-detection.inference.ts:190-195](../src/plugins/card-detection.inference.ts#L190-L195):
156
-
157
- ```ts
158
- const { score, box } = decodeBestBox(o.data, nA, scale, padX, padY, nC, classIdx);
159
- // crop-relative -> raw-frame, so callers never need to know a crop was applied.
160
- if (box && (cx0 || cy0)) {
161
- box.x1 += cx0; box.x2 += cx0;
162
- box.y1 += cy0; box.y2 += cy0;
163
- }
164
- ```
165
-
166
- That is why `_evaluateBox` above is unchanged in its geometry — it still compares raw-frame box against raw-frame guide.
167
-
168
- ---
169
-
170
- ## Change 2 — a new model that knows what a passport is
171
-
172
- The config swap — [card-detection.ts:13-26](../src/plugins/card-detection.ts#L13-L26):
173
-
174
- ```ts
175
- // Fine-tuned YOLO11n (ReLU), two classes {0:'card', 1:'passport'}, INT8. Output is
176
- // [1, 6, N]: cx, cy, w, h, card-score, passport-score per anchor.
177
- //
178
- // Exported at imgsz=256 rather than 320. Inference is synchronous on the main thread, so
179
- // the model's cost IS the UI's stutter; 256 measures 72ms p50 single-threaded vs 107ms for
180
- // the same model at 320 and 139ms for the old production model.
181
- MODEL_URL: `${process.env.CARD_MODEL || '/card-models'}/doc_v2_relu_640-int8-256.onnx`,
182
- INPUT_SIZE: 256, // keep in sync with the model's imgsz
183
- ```
184
-
185
- Old model: `document-detection-int8-320.onnx`, one generic `document` class, output `[1, 5, N]`.
186
- New model: two classes, output `[1, 6, N]` — one extra score row.
187
-
188
- Which row gets read depends on the document being captured — [card-detection.ts:73-77](../src/plugins/card-detection.ts#L73-L77):
189
-
190
- ```ts
191
- // Which of the model's score rows to read for the document currently being captured.
192
- // Scoring only the matching class means a passport held up on an ID step (or vice versa)
193
- // stays below CONF_ENTER and won't auto-capture. Falls back to class 0 inside
194
- // decodeBestBox if the loaded model is single-class.
195
- const MODEL_CLASS_INDEX = { CARD: 0, PASSPORT: 1 };
196
- ```
197
-
198
- The decoder gained two optional parameters — [card-detection.inference.ts:38-56](../src/plugins/card-detection.inference.ts#L38-L56):
199
-
200
- ```ts
201
- export function decodeBestBox(
202
- outputData: ArrayLike<number>,
203
- nA: number, // number of anchors
204
- scale: number, padX: number, padY: number,
205
- nC = 1, // number of class rows the loaded model actually has
206
- classIdx = 0, // which one to read
207
- ): { score: number; box: FrameBox | null } {
208
- let bestScore = 0;
209
- let bestIdx = -1;
210
- const sBase = (4 + (classIdx < nC ? classIdx : 0)) * nA; // was: 4 * nA
211
- for (let a = 0; a < nA; a++) {
212
- const s = outputData[sBase + a];
213
- if (s > bestScore) { bestScore = s; bestIdx = a; }
214
- }
215
- …
216
- }
217
- ```
218
-
219
- The output is flat and row-major: rows are `cx, cy, w, h, class0, class1`, each `nA` long. So `4 * nA` was "start of the first score row", and `(4 + classIdx) * nA` is "start of the row I want". The `classIdx < nC ? classIdx : 0` guard is the single-class fallback — ask for row 1 on a model that only has row 0 and you get row 0 rather than reading garbage past the end.
220
-
221
- `nC` is derived from the model itself rather than hardcoded — [card-detection.inference.ts:188-189](../src/plugins/card-detection.inference.ts#L188-L189):
222
-
223
- ```ts
224
- const nC = o.dims[1] - 4; // 6 - 4 = 2 classes; the old model gives 5 - 4 = 1
225
- const nA = o.dims[2];
226
- ```
227
-
228
- The tests build a tiny fake output to prove both rows are read correctly:
229
-
230
- ```ts
231
- const twoClass = Float32Array.from([
232
- ...[10, 20], // cx
233
- ...[11, 21], // cy
234
- ...[4, 6], // w
235
- ...[5, 7], // h
236
- ...[0.9, 0.1], // card score
237
- ...[0.2, 0.8], // passport score
238
- ]);
239
-
240
- it("reads the passport row for classIdx 1, picking a different anchor than class 0", () => {
241
- const result = decodeBestBox(twoClass, 2, 1, 0, 0, 2, 1);
242
- expect(result.score).toBeCloseTo(0.8); // not 0.9
243
- expect(result.box).toEqual({ x1: 17, y1: 17.5, x2: 23, y2: 24.5 }); // the second anchor
244
- });
245
- ```
246
-
247
- ---
248
-
249
- ## Change 3 — the detection loop paces itself
250
-
251
- Inference runs synchronously on the main thread, so while the model thinks, the video freezes. The old loop was a fixed 8 FPS cap:
252
-
253
- ```ts
254
- // BEFORE
255
- if (now - this.lastFrameTime < 1000 / CARD_DETECTION_CONFIG.THROTTLE_FPS) return;
256
- ```
257
-
258
- At 8 FPS the budget per frame is 125ms. If inference costs 120ms, the main thread is ~95% busy and the preview visibly stutters. The interval is now derived from the last frame's real cost — [card-detection.ts:455-469](../src/plugins/card-detection.ts#L455-L469):
259
-
260
- ```ts
261
- // Minimum gap before the next inference. Floors at the THROTTLE_FPS cap, then stretches
262
- // in proportion to how long the last frame actually took so inference can't monopolise
263
- // the main thread on a slow device. Clamped by MAX_DETECTION_INTERVAL_MS.
264
- private get detectionIntervalMs(): number {
265
- const floor = 1000 / CARD_DETECTION_CONFIG.THROTTLE_FPS; // 125ms
266
- if (!this.lastFrameMs) return floor;
267
-
268
- const duty = /* debug override */ ?? CARD_DETECTION_CONFIG.MAIN_THREAD_DUTY_MAX; // 0.5
269
- const maxI = /* debug override */ ?? CARD_DETECTION_CONFIG.MAX_DETECTION_INTERVAL_MS; // 400
270
-
271
- return Math.min(Math.max(floor, this.lastFrameMs / duty), maxI);
272
- }
273
- ```
274
-
275
- Read it as: *wait at least 125ms; if the last frame cost 120ms, wait 120/0.5 = 240ms instead; never wait more than 400ms.* The engine now reports total frame cost (not just the ORT call) so this has something to divide:
276
-
277
- ```ts
278
- // card-detection.inference.ts — runFrame
279
- const frameStart = performance.now();
280
- …
281
- return { score, box, W: srcW, H: srcH, inferMs, frameMs: performance.now() - frameStart, classScores };
282
- ```
283
-
284
- The tests pin the three regimes:
285
-
286
- ```ts
287
- (sdk as any).lastFrameMs = 120; expect(detectionIntervalMs).toBeCloseTo(240, 5); // stretched
288
- (sdk as any).lastFrameMs = 20; expect(detectionIntervalMs).toBeCloseTo(125, 5); // floored
289
- (sdk as any).lastFrameMs = 2000; expect(detectionIntervalMs).toBe(400); // clamped
290
- ```
291
-
292
- ---
293
-
294
- ## Change 4 — timers switched from frame counts to real time
295
-
296
- Once the loop speed varies, "wait for 2 good frames" stops meaning a fixed amount of time — at 400ms/frame it is 800ms instead of 250ms, and the border lags a second or more behind the card. So every counter became a millisecond accumulator — [card-detection.ts:132-138](../src/plugins/card-detection.ts#L132-L138):
297
-
298
- ```ts
299
- // BEFORE // AFTER
300
- private consecutiveValidFrames = 0; private validForMs = 0;
301
- private consecutiveInvalidFrames = 0; private invalidForMs = 0;
302
- private candidateBorderCount = 0; private candidateBorderMs = 0;
303
- ```
304
-
305
- The elapsed time per result is measured in `_onResult` — [card-detection.ts:520-536](../src/plugins/card-detection.ts#L520-L536):
306
-
307
- ```ts
308
- // Elapsed wall-clock since the previous result — every debounce/EMA decision below is
309
- // expressed in time, not frame count, so they behave the same at 2.5 FPS as at 8 FPS.
310
- // Clamped so a long gap (tab backgrounded, camera paused and resumed) can't satisfy a
311
- // whole debounce window in a single frame.
312
- const nowMs = performance.now();
313
- const dt = this.lastResultAt
314
- ? Math.max(0, Math.min(nowMs - this.lastResultAt, CARD_DETECTION_CONFIG.MAX_DETECTION_INTERVAL_MS))
315
- : CARD_DETECTION_CONFIG.TUNING_REF_INTERVAL_MS;
316
- this.lastResultAt = nowMs;
317
-
318
- const state = this._evaluateBox(m.score, m.box, m.W, m.H, dt);
319
- this._updateBorderDebounced(state, dt);
320
- this._updateDebounce(state, dt);
321
- ```
322
-
323
- `dt` is then threaded through every decision. The existing `*_FRAMES` constants were kept and reinterpreted as multiples of a 125ms reference interval — [card-detection.ts:830-848](../src/plugins/card-detection.ts#L830-L848):
324
-
325
- ```ts
326
- private _updateBorderDebounced(rawState, dtMs: number): void {
327
- …
328
- if (target === this.candidateBorder) {
329
- this.candidateBorderMs += dtMs; // was: candidateBorderCount++
330
- } else {
331
- this.candidateBorder = target;
332
- this.candidateBorderMs = dtMs; // was: = 1
333
- }
334
- const needed = CARD_DETECTION_CONFIG.TUNING_REF_INTERVAL_MS * (target === 'valid'
335
- ? CARD_DETECTION_CONFIG.BORDER_STABLE_FRAMES // 2 → 250ms of sustained 'valid'
336
- : CARD_DETECTION_CONFIG.BORDER_CLEAR_FRAMES); // 1 → 125ms to drop it
337
- if (this.candidateBorderMs >= needed && target !== this.activeBorder) {
338
- this.activeBorder = target;
339
- this.setBorderState(target);
340
- }
341
- }
342
- ```
343
-
344
- > Reading tip: the constants are still *named* `*_FRAMES` but are no longer frame counts — they are multiplied by `TUNING_REF_INTERVAL_MS` (125ms) at every use site. Changing `BORDER_STABLE_FRAMES` from 2 to 3 changes 250ms to 375ms. This naming is flagged as F5 in the code review.
345
-
346
- The confidence smoothing needed the same treatment. A plain EMA applies the same alpha per *frame*, so a slow 250ms frame would smooth as heavily as a fast 125ms one and the score would take ~3× longer in wall-clock to cross the gate. The alpha is now compounded over elapsed time — [card-detection.ts:557-564](../src/plugins/card-detection.ts#L557-L564):
347
-
348
- ```ts
349
- // CONF_EMA_ALPHA is defined per TUNING_REF_INTERVAL_MS, so compound it over however
350
- // long this frame actually took.
351
- const a = 1 - Math.pow(
352
- 1 - CARD_DETECTION_CONFIG.CONF_EMA_ALPHA, // 0.5
353
- dtMs / CARD_DETECTION_CONFIG.TUNING_REF_INTERVAL_MS,
354
- );
355
- this.scoreEma = a * (box ? score : 0) + (1 - a) * this.scoreEma;
356
- ```
357
-
358
- At `dtMs = 125` that gives back exactly `a = 0.5`. At `dtMs = 250` it gives `a = 0.75`, which lands the score where two 125ms frames would have. The test states it directly:
359
-
360
- ```ts
361
- // One 125ms frame at alpha 0.5 → ema = 0.5 * 0.8 = 0.4
362
- (sdk as any)._evaluateBox(0.8, box, 640, 480, 125);
363
- expect((sdk as any).scoreEma).toBeCloseTo(0.4, 5);
364
-
365
- // A single 250ms frame must land where two 125ms frames would (0.6), not at 0.4.
366
- (slow as any)._evaluateBox(0.8, box, 640, 480, 250);
367
- expect((slow as any).scoreEma).toBeCloseTo(0.6, 5);
368
- ```
369
-
370
- ---
371
-
372
- ## Change 5 — fail closed when the guide isn't laid out
373
-
374
- One line, and the most safety-relevant change in the branch — [card-detection.ts:577-578](../src/plugins/card-detection.ts#L577-L578):
375
-
376
- ```ts
377
- const guide = this._computeNormalizedGuide();
378
-
379
- // BEFORE — no guide → trust confidence alone → auto-capture could fire on a document
380
- // sitting anywhere in view
381
- if (!guide || !W || !H) { this._trace('VALID', conf, null); return 'valid'; }
382
-
383
- // AFTER
384
- if (!guide || !W || !H) { this._trace('no-guide', conf, null); return 'no-card'; }
385
- ```
386
-
387
- ```ts
388
- it("reports no-card when the overlay rect is unavailable, even on a strong score", () => {
389
- const sdk = new CARDDETECTIONSDK({ videoElement }); // no overlayElement
390
- const state = (sdk as any)._evaluateBox(0.99, { x1: 10, y1: 10, x2: 200, y2: 200 }, 1280, 720, 125);
391
- expect(state).toBe("no-card");
392
- });
393
- ```
394
-
395
- > The code review flags (F2) that this path currently has no escape hatch: if the guide were *persistently* unavailable the shutter would never unlock, and the manual-capture fallback in `card-camera.ts` only triggers on model-load failure.
396
-
397
- ---
398
-
399
- ## Change 6 — on-device debugging tools
400
-
401
- All of this is gated on `enableImageDebug`, the flag the integrator passes to `MainPage`. None of it runs in a normal production session.
402
-
403
- **Richer trace line** — [card-detection.ts:728-767](../src/plugins/card-detection.ts#L728-L767). `_pacingTrace()` builds a single line with the raw (pre-EMA) score, every class score with a `*` on the row actually being read, the source and crop sizes, how much of the square input the crop fills, the guide as a fraction of the frame, the `object-fit: cover` loss, and the measured FPS:
404
-
405
- ```ts
406
- return `raw=${this.lastRawScore.toFixed(2)} cls=[${cls}] `
407
- + `src=${this.lastSrcW}x${this.lastSrcH} fit=${fit}/${N} `
408
- + `crop=${c ? `${c.width}x${c.height}@${c.x},${c.y}` : 'full'} `
409
- + `disp=${vw}x${vh} cover=${cover} guide=[${gTxt}] `
410
- + `infer=${this.lastInferMs.toFixed(0)}ms frame=${this.lastFrameMs.toFixed(0)}ms `
411
- + `every=${this.detectionIntervalMs.toFixed(0)}ms fps=${fps.toFixed(1)} duty=${duty}`;
412
- ```
413
-
414
- The `cls=[...]` field is the one to watch for this story — if the *unread* row is the high one, the class index is wrong for the document type.
415
-
416
- **Live preview of what the model sees** — [card-detection.ts:811-825](../src/plugins/card-detection.ts#L811-L825) mirrors the letterboxed OffscreenCanvas into a 200px corner panel. The trace tells you a frame's geometry and cost but not its content; when scores are ~0 with correct geometry, only the pixels (blur, exposure, stale frame) explain it.
417
-
418
- **`window.__pwCardSnap()`** — [card-detection.ts:772-790](../src/plugins/card-detection.ts#L772-L790) downloads the exact frame the model scored as a PNG, so a real in-app failure can be replayed against other models instead of re-staged by hand.
419
-
420
- **`window.__pwCardPacing`** — [card-detection.ts:79-96](../src/plugins/card-detection.ts#L79-L96) exposes `duty`, `maxIntervalMs` and `cropMargin` for console A/B without a rebuild. Note the getter that reads them refuses to do so in production — [card-detection.ts:445-450](../src/plugins/card-detection.ts#L445-L450):
421
-
422
- ```ts
423
- // Only consulted when enableImageDebug is on — a stray global can never change
424
- // production pacing.
425
- private get pacingOverrides() {
426
- if (!enableImageDebug || typeof window === 'undefined') return null;
427
- return (window as any).__pwCardPacing || null;
428
- }
429
- ```
430
-
431
- …and there is a test that a stray `__pwCardPacing` on a merchant page changes nothing:
432
-
433
- ```ts
434
- it("ignores the debug pacing override when enableImageDebug is off", async () => {
435
- vi.stubGlobal("window", { __pwCardPacing: { duty: 1.0, maxIntervalMs: 0 } });
436
- …
437
- expect((sdk as any).detectionIntervalMs).toBeCloseTo(240, 5); // configured 0.5, not 1.0
438
- });
439
- ```
440
-
441
- ---
442
-
443
- ## Tests
444
-
445
- 18 new tests: crop geometry (square, pixel-space margin, centring, edge clamping, sliding not shrinking), the crop reaching `runFrame`, two-class row decoding in both directions plus the single-class fallback, adaptive pacing in all three regimes, frame-rate-invariant border/shutter/EMA behaviour, the long-gap and negative-`dt` clamps, and the new fail-closed path.
446
-
447
- Whole suite: **49 tests pass**, `tsc --noEmit` clean.
448
-
449
- Note: the repo's default Node (v16) cannot start vitest — `node:util` has no `styleText` export. Run under Node 20+ (`nvm use 22`).
450
-
451
- ---
452
-
453
- ## What is NOT done yet
454
-
455
- Three release steps the branch has not completed:
456
-
457
- 1. **The new model file exists nowhere the code looks for it.** `public/card-models/` still contains only `document-detection-int8-320.onnx`, and `git ls-files` confirms `doc_v2_relu_640-int8-256.onnx` is untracked. Production reads from `https://assets.pipwave.com/ekyc/sdk/card-models`, so it also has to be uploaded to the CDN.
458
- 2. **`dist/pw-bundle.js` has not been rebuilt** — both the committed and the working-tree copy still contain `document-detection-int8-320`.
459
- 3. **The SDK version is still 3.2.2** — no bump.
460
-
461
- Until those are done the change has no effect for anyone consuming the published SDK, and running it as-is fails to load a model at all — which drops *every* document type to manual capture, not just passport.
462
-
463
- Full findings, including the `dt`-clamp bug on slow devices and the missing escape hatch on the fail-closed path, are in [code-review-sc-349453.md](code-review-sc-349453.md).
@@ -1,176 +0,0 @@
1
- # Code review — sc-349453 passport detection fix
2
-
3
- **Branch:** `feature-teojunjie-sc-349453-fix-passport-detection`
4
- **Reviewed range:** `bccdff5..HEAD` (effectively the single commit `6337605`; the other commit is a merge of `master`)
5
- **Files:** `src/plugins/card-detection.ts`, `src/plugins/card-detection.inference.ts`, `src/plugins/card-detection.test.ts`, `src/plugins/card-detection.inference.test.ts`
6
-
7
- ## Verification performed
8
-
9
- | Check | Result |
10
- | --- | --- |
11
- | `npx vitest run` (Node 22) | 49/49 pass, 4 files |
12
- | `npx tsc --noEmit -p tsconfig.json` | clean |
13
- | `npx vitest run` (repo default Node 16) | **cannot start** — `node:util` has no export `styleText`; pre-existing, not caused by this branch |
14
- | New model present in `public/card-models/` | **no** — only `document-detection-int8-320.onnx` |
15
- | `dist/pw-bundle.js` contains the new model URL | **no** — still `document-detection-int8-320` in both HEAD and the working tree |
16
-
17
- ## Overall
18
-
19
- The diagnosis is correct and unusually well argued: the guide occupies ~29% of the frame, so whole-frame scoring made "big enough to detect" and "small enough to fit the guide" mutually exclusive. Cropping to a square around the guide is the right fix, and doing it *square* (rather than matching the guide's aspect) to avoid wasting the letterboxed input is a genuinely good detail. The comments explain *why* rather than *what*, and the accompanying conversion of the debounce/EMA logic from frame counts to wall-clock time is a real correctness fix, not incidental churn.
20
-
21
- Test coverage of the new geometry is strong. The gaps are in release mechanics, one time-accounting bug on slow devices, and a fail-closed path with no escape hatch.
22
-
23
- ---
24
-
25
- ## Blockers
26
-
27
- ### B1. The new model artifact does not exist on any path the code loads from
28
-
29
- `card-detection.ts:24` now resolves to `${CARD_MODEL}/doc_v2_relu_640-int8-256.onnx`.
30
-
31
- - **Dev** (`CARD_MODEL=/card-models`) serves `public/`, which contains only `document-detection-int8-320.onnx`. `git ls-files public/card-models/` confirms the new file is not tracked. Local dev is broken on this branch today.
32
- - **Production** (`CARD_MODEL=https://assets.pipwave.com/ekyc/sdk/card-models`) is a CDN — committing the file is not enough, it has to be uploaded there too.
33
-
34
- Failure mode if this ships unresolved: `session-create` throws → `classifyFailure` → after `MAX_LOAD_RETRIES` → `modelReady = false` → `onUnavailable()` → the "scanner unavailable" dialog and manual capture, for **every** document type, including ID and licence which work fine today. That is a much larger regression than the bug being fixed.
35
-
36
- Also: do not delete `document-detection-int8-320.onnx` from the CDN as part of this change. Cached copies of older SDK bundles will keep requesting it.
37
-
38
- ### B2. `dist/pw-bundle.js` is stale, and the version is not bumped
39
-
40
- `dist/pw-bundle.js` is the package `main` and is tracked in git (despite `dist` appearing in `.gitignore`). Both the committed and the working-tree copy still contain `document-detection-int8-320` and no reference to `doc_v2`. Anyone consuming the published package gets none of this change.
41
-
42
- Separately, there is an **uncommitted** modification to `dist/pw-bundle.js` sitting in the working tree that also predates these source changes. Rebuild deliberately and commit the result rather than shipping whatever is currently there.
43
-
44
- `package.json` is still at `3.2.2`. The story explicitly lists rebuild + version bump as steps.
45
-
46
- ---
47
-
48
- ## Findings
49
-
50
- ### F1. The `dt` clamp defeats frame-rate invariance on exactly the slow devices this targets — *medium*
51
-
52
- `card-detection.ts:525-527` clamps the measured inter-result gap with `MAX_DETECTION_INTERVAL_MS` (400ms). But that constant is the maximum **wait between frames**, not the maximum **cycle time**. The real cycle is `interval + frameMs`.
53
-
54
- Worked example, a slow mobile with `frameMs = 300`:
55
-
56
- ```
57
- detectionIntervalMs = min(max(125, 300/0.5), 400) = 400
58
- actual result-to-result gap ≈ 400 + 300 = 700ms
59
- dt handed to the state machine = min(700, 400) = 400
60
- ```
61
-
62
- The state machine believes 400ms elapsed when 700ms did. Consequences on that device:
63
-
64
- - Border/shutter transitions take ~1.75× longer in wall-clock than the intended 250ms.
65
- - The EMA alpha is computed from the understated `dt`, so the score is smoothed *more* than intended and takes correspondingly longer to cross `CONF_ENTER`.
66
-
67
- This is the same class of lag the commit set out to remove, just reintroduced at the slow end. The clamp's stated purpose (comment at 522-523) is to stop a backgrounded-tab resume from satisfying a whole debounce window in one frame — that only needs an upper bound on the order of 1–2 seconds. Suggest a separate constant:
68
-
69
- ```ts
70
- MAX_DEBOUNCE_DT_MS: 1500, // resume-gap guard only; not the pacing clamp
71
- ```
72
-
73
- The existing test `"clamps a long gap so a resumed loop can't satisfy a whole window in one frame"` would just assert the new value.
74
-
75
- ### F2. Fail-closed on a missing guide has no escape hatch — *medium*
76
-
77
- `card-detection.ts:578` changed `no guide → 'valid'` to `no guide → 'no-card'`. The safety reasoning is right — the old behaviour could auto-capture a document sitting anywhere in view.
78
-
79
- But the recovery path is missing. If `_computeNormalizedGuide()` returns null persistently (overlay hidden, zero-size, or `getBoundingClientRect` returning zeros mid-transition), the user is in a hard stuck state:
80
-
81
- - Every frame is `no-card`, so the shutter never unlocks.
82
- - `card-camera.ts:557-562` only wires the manual fallback to `onUnavailable`, which fires solely on **model load** failure.
83
- - `detectionAvailable` stays `true`, so the retry dialog is never offered either.
84
-
85
- In practice the overlay is unhidden on `loadedmetadata` (`card-camera.ts:667`) before detection starts, so the probability is low — but the cost is a completely blocked flow with no user-visible cause. Consider: if the guide has been unavailable for more than a few seconds while the loop is running, call `enableManualCapture()` (or the existing unavailable path) so the user is never trapped.
86
-
87
- ### F3. Documents held outside the guide may now report `no-card` instead of `invalid` — *medium, needs on-device confirmation*
88
-
89
- The model can only see the crop. A document held substantially outside the guide is now largely or entirely outside the model's field of view, so the likely outcome is a low score → `no-card` → border `'none'`, where whole-frame scoring previously found it and reported `out-of-box` → border `'invalid'` (red).
90
-
91
- That is a real feedback regression: "no border at all" tells the user nothing, whereas red told them to reposition. The 35% margin softens it, but a user holding the passport off to one side is exactly the case that used to get corrective feedback. Worth checking on-device what the hint copy shows in that state before sign-off, and whether the margin should be larger.
92
-
93
- ### F4. `MAIN_THREAD_DUTY_MAX` does not measure what its name and comment claim — *low*
94
-
95
- Two separate inaccuracies:
96
-
97
- 1. `lastFrameTime` is stamped at the *start* of the tick (`card-detection.ts:484`), so the real cycle is `interval + frameMs`. At the measured 72ms p50, `interval = max(125, 144) = 144`, and true occupancy is `72 / 216 ≈ 33%`, not 50%. Conservative, so harmless, but the constant overpromises.
98
- 2. When the 400ms clamp binds, the duty target is silently abandoned: a 300ms frame at a 400ms interval is 300/700 ≈ 43%, and with the fix in F1 the numbers move again. The comment "inference never occupies more than this fraction of wall-clock time" is not something the code guarantees.
99
-
100
- Suggest rewording the comment to "target duty; the `MAX_DETECTION_INTERVAL_MS` clamp takes precedence" rather than stating it as an invariant.
101
-
102
- ### F5. `*_FRAMES` constants are no longer frame counts — *low, maintenance hazard*
103
-
104
- `STABLE_FRAMES_TO_ENABLE`, `STABLE_FRAMES_TO_DISABLE`, `BORDER_STABLE_FRAMES`, `BORDER_CLEAR_FRAMES` are now multiplied by `TUNING_REF_INTERVAL_MS` at every use site (`:841-843`, `:858`, `:865`). Their inline comments still say "valid frames in a row" and "frames before the border goes green".
105
-
106
- Someone tuning "2 frames → 3 frames" is actually changing 250ms → 375ms, which is not what the name says. Rename to `BORDER_STABLE_MS: 250` etc., or `BORDER_STABLE_INTERVALS` if the 125ms reference unit is worth keeping explicit.
107
-
108
- ### F6. `THROTTLE_FPS` comment is now wrong — *low*
109
-
110
- `card-detection.ts:55` reads `// detection loop FPS floor`. It is still an FPS **ceiling** — `1000 / THROTTLE_FPS` is the *interval* floor, i.e. the fastest the loop may run. Easy to misread as "we guarantee at least 8 FPS", which the code does not.
111
-
112
- ### F7. Debug hooks leak and go stale across camera sessions — *low, debug-only*
113
-
114
- `installPacingDebugHook()` and `_installSnapshotHook()` attach `window.__pwCardPacing` and `window.__pwCardSnap`, but `dispose()` (`:401-409`) removes only the DOM debug elements. Two consequences:
115
-
116
- - `__pwCardSnap` closes over `this`, so the SDK instance and its engine are retained after teardown.
117
- - The `if (w.__pwCardSnap) return` guard means a **second** camera session keeps the *first* (disposed) instance's closure. `this.engine` is null by then, so `__pwCardSnap()` just logs "no frame captured yet" — the hook silently stops working precisely when someone is trying to debug a repeat attempt.
118
-
119
- Fix: delete both globals in `dispose()`.
120
-
121
- ### F8. `snapshotInput()` can hang — *low, debug-only*
122
-
123
- `card-detection.inference.ts:216-224` wraps `FileReader` in a promise with an `onload` handler and no `onerror`/`onabort`. A read failure leaves the awaiting caller pending forever. Add a rejecting (or `resolve(null)`) error handler.
124
-
125
- ### F9. Comments point at files that are not in the repo — *low*
126
-
127
- - `card-detection.ts:20` cites `tmp/bench-model.mjs` for the 72ms/107ms/139ms benchmark numbers. `tmp/` does not exist.
128
- - `card-detection.ts:770` cites `public/debug-model-compare.html`. `public/` contains only `debug-camera.html`.
129
-
130
- The benchmark numbers are load-bearing justification for the 256px choice — either commit the harness (it is small and clearly useful for the next model swap) or reword so the citation is not a dead pointer. Same for the compare page.
131
-
132
- ### F10. The model filename carries two conflicting sizes — *low*
133
-
134
- `doc_v2_relu_640-int8-256.onnx` contains both `640` and `256`, and `INPUT_SIZE: 256` is commented `// keep in sync with the model's imgsz (the -256 in the filename)`. That instruction is ambiguous the moment a reader notices the `640`. Since `MODEL_URL` and `INPUT_SIZE` *must* move together (a mismatch throws at `run()`), the filename should be unambiguous — e.g. `doc-v2-relu-int8-256.onnx`, with the 640 training resolution recorded in the comment instead.
135
-
136
- ### F11. Removed documentation on two now-more-complex functions — *low*
137
-
138
- The diff deletes the explanatory comment above `decodeBestBox` (the `[1,5,nA]` layout and the letterbox→frame mapping) and above `runFrame` (why the raw `<video>` is used at full resolution). Both functions **gained** parameters in this change — `runFrame` went from 3 to 5, including an optional crop with non-obvious coordinate semantics. Those are the two places a docstring earned its keep. Suggest restoring them, updated for the new signature, particularly documenting that `crop` is in raw-frame pixels and that returned boxes are always raw-frame regardless.
139
-
140
- Also, `card-detection.inference.ts:34` has trailing whitespace after `inferMs: number;`, and the blank line between `FrameResult` and `decodeBestBox` was removed.
141
-
142
- ### F12. Silent class-index fallback deserves a warning — *low*
143
-
144
- `decodeBestBox` (`inference.ts:49`) falls back to class 0 when `classIdx >= nC`. Correct and deliberate for a single-class model. But it also means shipping a model with the class order swapped, or accidentally reverting to a one-class export, degrades to "score everything as a card" with no signal at all — which is precisely the bug this story is fixing, returning silently. A one-time `console.warn` when the fallback engages (even under `enableImageDebug` only) would make that self-diagnosing. The `cls=[...]` trace with the `*` marker helps, but only if someone is looking.
145
-
146
- ### F13. `classScores` is computed in production for debug-only output — *trivial*
147
-
148
- `inference.ts:197-204` runs a full extra pass over every class row on every frame; `lastClassScores` is read only by `_pacingTrace()`, which is debug-gated. The comment ("~2 × nA compares — not worth gating") is a fair call on cost, but it is still dead work plus an allocation in the hot path. Gating on a flag passed into `runFrame` would cost nothing and remove the question.
149
-
150
- ---
151
-
152
- ## Test review
153
-
154
- **Well covered:** crop is square; crop sized from the longer guide side with the margin taken in *pixel* space (the unit-mixing trap is explicitly regression-tested); crop stays centred; crop strictly encloses the guide so oversized documents still read as spilling; clamping and sliding at frame edges; null crop with no overlay; crop reaching `runFrame`; two-class row selection in both directions; single-class fallback; adaptive interval floor/stretch/clamp; the debug override being ignored in production; frame-rate-invariant border, shutter and EMA behaviour; long-gap and negative-`dt` clamping; the new fail-closed path.
155
-
156
- **Gaps:**
157
-
158
- 1. **The headline wiring is untested.** No test asserts that a `PASSPORT`-typed SDK passes `classIdx = 1` and an ID-typed one passes `0`. The one test that could (`"passes the crop through to the engine's runFrame"`, `card-detection.test.ts:486`) destructures the argument as `_classIdx` and never asserts it. `decodeBestBox` is tested with explicit indices, but the `this.type === CARD_TYPES.PASSPORT ? PASSPORT : CARD` mapping at `card-detection.ts:496` — the actual fix for the reported bug — has no coverage. Two assertions on `runFrameArgs[0][3]` would close this.
159
- 2. **`FrameResult` fixtures are incomplete.** Test objects are built as `{ score, box, W, H, inferMs, frameMs }` with no `classScores`, passed through `as any`, so TypeScript never checks them. `_onResult` then assigns `undefined` to `lastClassScores`, and `_pacingTrace()` would throw on `.map` if `enableImageDebug` were true in a test. A shared `makeFrameResult(overrides)` helper would keep the fixtures honest and catch the next added field.
160
- 3. **No coverage of F1's slow-device path** — a `frameMs` large enough that the real cycle exceeds the clamp. Worth adding alongside any fix.
161
- 4. **`_computeDetectCropRect` on mobile.** All crop tests use the desktop path (`isMobileDevice()` false → X mirrored). The mobile branch skips the mirror, and passport capture is overwhelmingly mobile. One mobile-path crop test would be cheap insurance.
162
-
163
- ## Minor / stylistic
164
-
165
- - `_computeNormalizedGuide()` is called twice per detection frame (`_computeDetectCropRect` and `_evaluateBox`), three times with debug on (`_pacingTrace`). Each is a forced synchronous layout. At ≤8 FPS this is noise, but caching the result per tick is free and removes any question about layout thrash on low-end devices.
166
- - `card-detection.ts:51` `DETECT_CROP_MARGIN_RATIO` is placed under the "Framing geometry" block with no comment of its own, while every neighbouring constant has one. The rationale lives on `_computeDetectCropRect` instead — a one-line pointer at the constant would help.
167
- - The commit message body has hard-wrapped lines breaking mid-phrase ("whole-frame scoring\ngeometry", "fail closed instead of\nopen"), which reads awkwardly in `git log --oneline` and in the Shortcut integration.
168
-
169
- ## Suggested sequence before merge
170
-
171
- 1. Commit the model to `public/card-models/` **and** upload it to `assets.pipwave.com/ekyc/sdk/card-models/`; verify both dev and a production-config build actually load it (B1).
172
- 2. Fix the `dt` clamp (F1) — it is a behaviour bug on the slowest devices, and it is a two-line change.
173
- 3. Decide on the missing-guide escape hatch (F2) and the out-of-guide feedback question (F3); both want a look on a real device rather than a code decision.
174
- 4. Add the class-index assertions (test gap 1) — the headline fix should not be untested.
175
- 5. Rebuild `dist/pw-bundle.js`, bump the version, commit both (B2).
176
- 6. Sweep the low-severity items (F5–F12) — all small, none blocking.
@@ -1,439 +0,0 @@
1
- # FE Plan — eKYC SDK Selfie Action Checking Enhancement (Epic sc-342056)
2
-
3
- _Analysis date: 2026-08-12 · Repo version: `pipwave-ekyc-uikit@3.3.0-beta.3` · Branch at time of writing: `feature-teojunjie-sc-349453-fix-passport-detection`_
4
-
5
- Scope of this doc: **frontend only**. Model training (sc-348062, sc-348317) and backend
6
- (sc-348061, sc-340376) are other people's stories and are referenced only where they block FE.
7
-
8
- ---
9
-
10
- ## Sources read
11
-
12
- | Source | ID / link |
13
- |---|---|
14
- | Epic | [sc-342056 — eKYC SDK — Selfie Action Checking Enhancement](https://app.shortcut.com/pipwave/epic/342056) |
15
- | Story (given) | [sc-342060 — follow the actions shown in the SDK (liveness)](https://app.shortcut.com/pipwave/story/342060) — 3 days est. |
16
- | Story (given) | [sc-348063 — detect the six liveness actions client-side with MediaPipe](https://app.shortcut.com/pipwave/story/348063) — no est. |
17
- | Story (given) | [sc-346306 — capture my face while I hold still](https://app.shortcut.com/pipwave/story/346306) — 4 days est. (+2 if retraining needed) |
18
- | Story (related FE) | [sc-342058 — animated video guide before I start](https://app.shortcut.com/pipwave/story/342058) |
19
- | Story (related FE) | [sc-342059 — tell me my face is centered and not blocked](https://app.shortcut.com/pipwave/story/342059) — 6 days est. |
20
- | Story (related FE) | [sc-342061 — know my result and what to do next](https://app.shortcut.com/pipwave/story/342061) — 4 days est. |
21
- | Story (related FE) | [sc-342062 — verification link single-use](https://app.shortcut.com/pipwave/story/342062) |
22
- | Story (related FE) | [sc-348318 — integrate the face-visibility ONNX model](https://app.shortcut.com/pipwave/story/348318) |
23
- | Story (BE) | [sc-348061 — accept action-stage images + anti-face-swap](https://app.shortcut.com/pipwave/story/348061) — already in *Ready for QA (Staging)* |
24
- | Figma (approved) | file `quWrR270oHorR9i9ZPGMxk`, page **`8671:9427` — "✅ Ready for Dev \| Approved - Face Scan \| 2026-07-10"** |
25
- | Figma (archived) | page `9290:28008` — "Ardhived \| Iteration - Face Scan \| Before 2026-07-10" |
26
-
27
- Epic deadline: **2026-08-31**. Epic stats at time of writing: 12 stories, 0 done, 20 points unstarted.
28
-
29
- ---
30
-
31
- ## 0. Two things to flag first
32
-
33
- ### 0.1 The Figma link circulated in the stories is the archived page
34
-
35
- All six stories link `node-id=8671-9428`, which resolves under the canvas
36
- **`9290:28008` = "Ardhived | Iteration - Face Scan | Before 2026-07-10"**.
37
-
38
- The authoritative page is **`8671:9427` — "✅ Ready for Dev | Approved - Face Scan | 2026-07-10"**.
39
-
40
- Differences that change the build:
41
-
42
- | Archived (v1/v2) | Approved (2026-07-10) |
43
- |---|---|
44
- | `countdown 3s` section | **`hold still (3s) (take selfie)`** — countdown removed |
45
- | 7 actions incl. **shake head** | 6 actions, **shake dropped** |
46
- | `KB article` / `Client's KB article` | **`guide-video-dialog`** component |
47
- | — | **QR scan** section (desktop) + hosting-page frames |
48
- | `verification failed endpoint (expired)` | adds **`duplicate submission`** result screen |
49
- | mobile only | **mobile + full desktop** section (1280×720) |
50
-
51
- **Plan against `8671:9427`.** Worth getting the story links updated so nobody builds the countdown.
52
-
53
- ### 0.2 The three stories cannot be built independently
54
-
55
- They are three stages of one continuous camera session, sharing one video stream, one detection
56
- loop and one image buffer. The other three FE stories in the epic (sc-342058 video guide,
57
- sc-342059 real-time check, sc-342061 result/attempts) sit in the same session. Building them as
58
- six separate features will produce six competing detection loops — which is precisely the class of
59
- bug already fixed once in this repo (see the `isStartingDetection` guard comment at
60
- [`face-detection.ts:629-645`](../src/plugins/face-detection.ts#L629-L645)).
61
-
62
- Ownership note: sc-348063 and sc-348318 are currently assigned to **Teo Jun Jie**; sc-342059 /
63
- 342060 / 342061 / 346306 are unassigned with suggested branch `feature-chechin-*`.
64
-
65
- ---
66
-
67
- ## 1. Target flow (approved Figma `8671:9427`)
68
-
69
- ```
70
- launch SDK → face scan intro
71
- ├── (overlay) face scan guide [guide-video-dialog]
72
- └── (overlay) not saved [error-dialog variant=not-saved]
73
- │
74
- ┌── SINGLE ATTEMPT ────────▼──────────────────────────────────────────┐
75
- │ real-time checking (continuous, non-blocking, skip if good) │
76
- │ └── (overlay) idle │
77
- │ hold still (3s) → captures image 1 (reference) │
78
- │ └── (overlay) photo issue - can retry │
79
- │ └── (overlay) failed 3 attempts - cannot retry │
80
- │ action checking (15s) → 2-of-3 random actions, pool of 6 │
81
- │ └── each action result: success / failed │
82
- │ └── (overlay) action issue - can retry │
83
- │ └── (overlay) face mismatch - can retry │
84
- │ └── (overlay) failed 3 attempts - cannot retry │
85
- └─────────────────────────────────────────────────────────────────────┘
86
- │
87
- face scan session result: analyzing
88
- ├── verification success session → redirect ~5s
89
- ├── verification failed - 3 limit reached
90
- ├── verification failed - expired
91
- └── verification failed - duplicate submission
92
- generic error (out of epic scope — already covered by current flow)
93
- ```
94
-
95
- Figma section title carries an unresolved design question verbatim:
96
- `action checking (15s) (must success 2 out of 3 random actions) (take selfie when starting 2nd action?)`
97
-
98
- ### 1.1 Components already spec'd in Figma
99
-
100
- Under `Designer use` (`8808:9882`):
101
-
102
- | Component | Variants |
103
- |---|---|
104
- | `comp/ekyc-sdk/oval-static` | `countdown=false,result=false` / `countdown=true,result=false` / `countdown=true,result=true` |
105
- | `comp/ekyc-sdk/oval-motion` | (animated ring) |
106
- | `comp/ekyc-sdk/capture` | `success=true` / `success=false` |
107
- | `comp/ekyc-sdk/action-icon` | `Nod your head`, `Turn your head left`, `Turn your head right`, `Smile and show your teeth`, `Open your mouth wide`, `Blink your eyes` |
108
- | `comp/ekyc-sdk/camera` | 30 variants: `static`, `position-1/2`, `face-1/2`, `eye-1/2`, `nod-1/2/3`, `turn-left-1/2/3`, `turn-right-1/2/3`, `smile-1/2/3`, `open-mouth-1/2/3`, `blink-1/2/3`, `move-1/2/3` |
109
- | `comp/base/video-button` | `play` / `pause` / `skip` × `enabled` / `hover` / `active` |
110
- | `comp/ekyc-sdk/guide-video` | `optimisation=tablet-mobile\|desktop` × `show-button` × `playing` |
111
- | `dialog-template/ekyc-sdk/error-dialog` | `not-saved`, `selfie-can-retry`, `selfie-cannot-retry`, `idle` × mobile/desktop |
112
- | `page-template/ekyc-sdk/ekyc-main` | `launching`, `intro`, `analyzing`, `success-endpoint`, `failed-endpoint`, `generic-error` × mobile/desktop |
113
- | `dialog-template/ekyc-sdk/QR-dialog` | — |
114
-
115
- The 3-step `-1/-2/-3` camera variants confirm each action is designed as an **animated
116
- demonstration**, not a static icon.
117
-
118
- ### 1.2 Exact copy (from the Figma annotation on `real-time checking`)
119
-
120
- > **To keep existing (updated copy):**
121
- > - `face_neutral_expression`: "Show a neutral expression"
122
- > - `face_looks_straight`: "Look straight at the camera"
123
- > - `face_within_frame`: "Place your face within the frame"
124
- > - `face_too_far`: "Move closer"
125
- > - `face_too_close`: "Move further"
126
- > - `face_not_centered`: "Center your face in the frame"
127
- >
128
- > **To remove existing:**
129
- > - ~~`face_smile_at_camera`: "Please smile at the camera."~~
130
- >
131
- > **New:**
132
- > - `no_forehead`: "Keep your forehead in view"
133
- > - `no_eye`: "Keep your eyes in view"
134
- > - `no_nose`: "Keep your nose in view"
135
- > - `no_mouth`: "Keep your mouth in view"
136
- > - `no_jaw`: "Keep your chin in view"
137
-
138
- `face_smile_seconds` ("Please keep smiling for 3 seconds.") follows `face_smile_at_camera` out.
139
-
140
- Note sc-342059's AC proposes *different* wording for the same keys ("Make sure your full face is
141
- visible", "Show the top of your face", "Make sure your chin is visible"). **Figma's annotation is
142
- newer (2026-07-10) — use Figma's, and get the story updated.**
143
-
144
- Other copy read off the frames:
145
- - Hold-still header: `Hold still`; success state shows a green tick + `Done` inside the oval.
146
- - Photo-issue dialog: "We couldn't get a clear photo of your face." + Tips list (position within
147
- frame / keep face in focus and fully in frame / remove sunglasses, hats, accessories / plain
148
- light-coloured background) + `View guide` button + `Try again`.
149
- - Cannot-retry dialog: "We couldn't verify your face scan." + `Continue`.
150
- - Idle dialog: "You have been idle for too long." + `Try again`.
151
-
152
- ### 1.3 Structural UI change
153
-
154
- Guidance text moves **into the header bar** (`< Place your face within the frame`). Today it is a
155
- floating absolutely-positioned div above the oval —
156
- [`face-camera.ts:299-312`](../src/plugins/face-camera.ts#L299-L312). That div gets deleted.
157
-
158
- ---
159
-
160
- ## 2. Requirements, consolidated
161
-
162
- ### sc-346306 — Hold still & reference capture
163
-
164
- - Remove the 5-second countdown entirely (kills the `createCountdown` call at
165
- [`face-detection.ts:652`](../src/plugins/face-detection.ts#L652)).
166
- - New "hold still" stage: a stability window (~3s per Figma) during which the face must stay
167
- framed, centred and still.
168
- - The oval frame renders as a **countdown-timer ring**: white by default, turning green while
169
- running / capturing, then green tick + "Done".
170
- - Capture **image 1 = the reference face**, verified clean (in frame, not blurred) before the flow
171
- advances to actions.
172
-
173
- ### sc-342060 — Randomised liveness actions
174
-
175
- - Remove standalone smile detection.
176
- - **Pool of 6:** nod head up/down · turn head left · turn head right · blink eyes · open mouth wide
177
- and close · smile (big smile, teeth showing).
178
- - Random selection **and** random order, per attempt — sequence must not be predictable or
179
- pre-recordable.
180
- - **Head must be centred before a turn-left / turn-right action is accepted.**
181
- - Frame turns green when an action is detected.
182
- - Capture the action images (one per action performed); these plus image 1 go to BE.
183
- - **Pass/fail:** must pass 2 actions, max 3 shown.
184
-
185
- | Action 1 | Action 2 | Action 3 | Result |
186
- |---|---|---|---|
187
- | pass | pass | — | **PASS** |
188
- | fail | pass | pass | **PASS** |
189
- | pass | fail | pass | **PASS** |
190
- | pass | fail | fail | FAIL |
191
- | fail | pass | fail | FAIL |
192
- | fail | fail | — | FAIL |
193
-
194
- - An action-stage **FAIL counts as a failed attempt** toward the max-3 overall limit (sc-342061).
195
-
196
- ### sc-348063 — MediaPipe client-side detection
197
-
198
- - Every action is a movement that **completes back to neutral**:
199
- nod (down → up → centre) · turn L/R (turn → back to centre) · blink (close → open) ·
200
- open mouth (open → close). **Smile is the exception — detected on smile, no neutral return.**
201
- - Enforce the head-centred precondition before a turn is accepted.
202
- - Frame turns green when each action completes.
203
- - Tune per-action thresholds and minimum durations across devices, lighting, glasses and skin
204
- tones; distinguish involuntary from deliberate (the blink case).
205
- - Must run real-time on a **mid-range mobile device**.
206
- - **Out of scope:** the random 2-of-6 selection flow (that's sc-342060) and anti-spoofing /
207
- video-replay (server-side, sc-348061).
208
-
209
- ### Overlap resolution
210
-
211
- sc-342060 and sc-348063 are the same feature at two altitudes:
212
-
213
- - **sc-348063** = the six detectors + signal processing + threshold tuning (the "can we see it?" layer)
214
- - **sc-342060** = orchestration, randomisation, pass/fail table, image capture, UI (the "what do we ask and what does it mean?" layer)
215
-
216
- Clean seam — split along it.
217
-
218
- **sc-346306** overlaps both only through shared infrastructure: one camera session, one detection
219
- loop, one image buffer, one overlay component whose ring state is driven by whichever stage is
220
- active. That shared infrastructure is the foundation story (F0 below), not a copy in each story.
221
-
222
- ---
223
-
224
- ## 3. Open questions that block coding
225
-
226
- Answer these early — several are hard blockers, not polish items.
227
-
228
- 1. **Engine contradiction.** sc-348063 says MediaPipe, explicitly "with no trained model".
229
- sc-342059 / sc-348318 / sc-348062 say a trained **ONNX face-visibility model** delivered by the
230
- AI team. Running face-api.js + MediaPipe + ONNX concurrently will not hold frame rate on a
231
- mid-range Android. → See §4.2 for the recommendation.
232
-
233
- 2. **API contract missing — hard blocker for sc-342061.**
234
- `UploadSelfieParam` in `pipwave-ekyc-sdk@2.1.0`
235
- (`node_modules/pipwave-ekyc-sdk/dist/types/modules/proofs/model.d.ts:39`) is:
236
-
237
- ```ts
238
- export interface UploadSelfieParam {
239
- selfie_uri?: string;
240
- selfie_uri_full?: string;
241
- selfie_base64?: Base64;
242
- selfie_base64_full?: Base64;
243
- liveness_score?: number;
244
- }
245
- ```
246
-
247
- **There is no field for action images.** BE story sc-348061 is already in *Ready for QA
248
- (Staging)*, so the server side likely exists — but the npm package needs a version bump with the
249
- new param shape before FE can submit anything.
250
-
251
- 3. **How many images actually go up?** sc-342060 says one image per action (up to 3); the epic says
252
- "1 reference + action images"; BE verifies "1st + last first". Figma asks in a section title:
253
- *"(take selfie when starting 2nd action?)"*. This determines payload size, capture timing and
254
- whether we upload once or progressively.
255
-
256
- 4. **Timings.** Figma implies hold-still 3s and action stage 15s total. Neither story states a
257
- number. Need: per-action timeout, and what a timeout means (action fail? or re-prompt?).
258
-
259
- 5. **What is `liveness_score` now?** Today it's face-api's *detection confidence* passed straight
260
- through ([`face-detection.ts:514`](../src/plugins/face-detection.ts#L514) →
261
- [`StartSelfie.ts:142`](../src/pageClass/StartSelfie.ts#L142)) — it was never a liveness score.
262
- Under the new flow: derive one from action outcomes, or retire the field?
263
-
264
- 6. **Desktop + QR flow** is fully designed in the approved Figma and is in the epic (sc-342062) but
265
- appears in none of the three stories. Confirm it's in this release.
266
-
267
- 7. **Idle timeout** is currently 90s (`cameraTimeLimit` in
268
- [`face-camera.ts:27`](../src/plugins/face-camera.ts#L27)). The new flow is longer
269
- (guide + real-time + 3s hold + 15s actions). Confirm the value and whether the timer resets per
270
- stage.
271
-
272
- ---
273
-
274
- ## 4. Current code — what survives, what gets rewritten
275
-
276
- ### 4.1 File-by-file verdict
277
-
278
- | File | Lines | Verdict |
279
- |---|---|---|
280
- | [`plugins/face-detection.ts`](../src/plugins/face-detection.ts) | 861 | **Full rewrite.** Monolith: face-api calls + geometry + hint debounce + smile/liveness heuristic + model loading + warmup + overlay sizing + countdown trigger, all in one class. `detectLiveness` / `detectSimilarFace` / the whole smile branch (≈ lines 326-523) are **deleted outright** — expression-change-as-liveness is replaced by real actions, and the client-side descriptor match is replaced by the BE anti-swap check (sc-348061). |
281
- | [`plugins/face-camera.ts`](../src/plugins/face-camera.ts) | 598 | **~60% keep.** `loadVideoStream`, `cameraInit`, `resizeElement` (aspect-fit) and `stopVideoStream` teardown are recently hardened — extract and keep. `captureImage` (lines 51-145, incl. the overlay-rect crop maths and `CROP_TOP/BOTTOM_PADDING_RATIO`) is correct — extract to `capture/frame-capture.ts`. The 8 `listenTo*` smile handlers, the description div and the green-tick block are replaced by stage-driven rendering. |
282
- | [`pageClass/StartSelfie.ts`](../src/pageClass/StartSelfie.ts) | 149 | **Rewrite.** Becomes the flow host: attempt counter, stage transitions, dialog variants, submission. Its error-type mapping (`openApi/max-selfie-attempt`, `openApi/img-exposed`, …) at lines 68-103 is worth preserving as-is. |
283
- | Geometry helpers in `face-detection.ts` (`getFaceXY`, `getCircleXY`, `isFaceDetected`, `isFaceCentered`, `isFaceLargeEnough`, `isFaceTooClose`) | ~180 | **Port, don't rewrite.** The constants (`MIN/MAX_FACE_FILL_RATIO`, `FOREHEAD_PADDING_RATIO`, `MAX_FACE_CENTER_OFFSET`, `TOP_CONTAINMENT_SLACK_RATIO`) and their documented reasoning are hard-won on-device tuning. `FOREHEAD_PADDING_RATIO` exists purely because face-api's 68 landmarks stop at the eyebrows — **MediaPipe's mesh includes the forehead, so that whole correction and its slack constant disappear.** |
284
- | Hint debounce (`emitHint`, `confirmPositionState`, `HINT_MIN_INTERVAL_MS`, `STABLE_FRAMES_TO_CHANGE_POSITION`) | ~60 | **Keep the concept, generalise it.** Anti-flicker is still needed, and will need a second instance for action state. |
285
- | [`plugins/card-detection.ts`](../src/plugins/card-detection.ts) + [`card-detection.inference.ts`](../src/plugins/card-detection.inference.ts) | 1369 | **Untouched — but copy the pattern.** `CARD_DETECTION_CONFIG` (one config object, every threshold documented with *why*) and the engine/consumer split are exactly the shape the selfie side should adopt. Also the dual-threshold + EMA smoothing idea (`CONF_ENTER` / `CONF_EXIT` / `CONF_EMA_ALPHA`) transfers directly to action detection. |
286
- | [`translations/en.ts`](../src/translations/en.ts) | 227 | Edit: reword 6 keys, delete 2, add 5 (see §1.2). |
287
- | [`components/components.ts`](../src/components/components.ts) `createCountdown` | ~40 | **Delete** (or keep for card flow if used there — verify). Replaced by the ring component. |
288
-
289
- ### 4.2 Engine recommendation — one engine, MediaPipe FaceLandmarker
290
-
291
- **Use MediaPipe FaceLandmarker as the single engine and retire face-api.js.**
292
-
293
- From *one* inference per frame it gives all three consumers what they need:
294
-
295
- | Need | MediaPipe output |
296
- |---|---|
297
- | Framing, centring, per-region presence (forehead / eyes / nose / mouth / chin) — sc-342059 | 478 landmarks, incl. forehead |
298
- | nod / turn left / turn right — sc-348063 | `facialTransformationMatrix` → true yaw / pitch / roll |
299
- | blink / open mouth / smile — sc-348063 | 52 blendshapes: `eyeBlinkLeft/Right`, `jawOpen`, `mouthSmileLeft/Right` |
300
-
301
- Compared with today: `isAngleDetected`
302
- ([`face-detection.ts:200-233`](../src/plugins/face-detection.ts#L200-L233)) infers head angle from
303
- 2D landmark arctangents — a hack that cannot cleanly separate a deliberate 25° turn from a tilt.
304
- And the 7-class expression classifier cannot distinguish an involuntary blink from a deliberate one
305
- at all, which sc-348063 explicitly requires.
306
-
307
- **Bundle win:** `public/face-models/` is currently **15 MB**, of which
308
- `face_recognition_model` shards are **6.4 MB** — loaded solely for `detectSimilarFace`, which this
309
- epic deletes. MediaPipe's `face_landmarker.task` is ~3.7 MB. Meaningful mobile load-time
310
- improvement, and it removes the `face-api.js` dependency plus the `declare const faceapi` /
311
- `require()` dual-path hack at [`face-detection.ts:49-54`](../src/plugins/face-detection.ts#L49-L54).
312
-
313
- If the AI team's ONNX visibility model (sc-348062) lands and measurably beats landmark presence,
314
- slot it in behind the same `engine/visibility.ts` interface. **Do not run both.** The ORT plumbing
315
- to reuse already exists in
316
- [`card-detection.inference.ts`](../src/plugins/card-detection.inference.ts) — including the
317
- hard-won note that Chromium refuses cross-origin `new Worker(url)`, so inference runs on the main
318
- thread via dynamic `import()`.
319
-
320
- ### 4.3 Proposed module layout
321
-
322
- ```
323
- src/plugins/selfie/
324
- config.ts // every threshold + timing, CARD_DETECTION_CONFIG style
325
- engine/face-mesh.ts // MediaPipe wrapper: load, warmup, throttled detect loop
326
- engine/signals.ts // one frame -> {yaw,pitch,roll, ear, mar, smile, regions, box}
327
- engine/visibility.ts // signals -> RealtimeIssue (sc-342059 / sc-348318)
328
- actions/types.ts // ActionId, ActionSpec, ActionResult
329
- actions/detectors/nod.ts
330
- actions/detectors/turn-left.ts
331
- actions/detectors/turn-right.ts
332
- actions/detectors/blink.ts
333
- actions/detectors/open-mouth.ts
334
- actions/detectors/smile.ts // 6 small state machines (sc-348063)
335
- actions/action-runner.ts // arm -> precondition -> detect -> pass / timeout
336
- actions/action-sequencer.ts // random 2-of-6, pass/fail table, verdict (sc-342060)
337
- stages/hold-still.ts // stability window + reference capture (sc-346306)
338
- capture/frame-capture.ts // lifted from face-camera.captureImage
339
- capture/image-buffer.ts // labelled reference + action images
340
- ui/selfie-overlay.ts // oval ring states: white / countdown / green / red
341
- ui/action-prompt.ts // header copy + action icon + animated demo
342
- selfie-flow.ts // REALTIME -> HOLD_STILL -> ACTIONS -> SUBMIT -> RESULT
343
- debug.ts // extends the existing [pw-face] gated logging + on-screen HUD
344
- ```
345
-
346
- **The key structural change: one detection loop, one `signals` object per frame, many consumers.**
347
-
348
- Today every check independently re-derives geometry from raw landmarks — `getFaceXY` alone is
349
- called 5× per frame across the position checks. That's what makes the current file unmaintainable,
350
- and it is what will melt the CPU budget once six action detectors are added on top.
351
-
352
- Each detector should be a pure `(signals, state) -> state` step with no DOM access, so they are
353
- unit-testable without a camera — the repo already has this pattern in
354
- [`card-detection.test.ts`](../src/plugins/card-detection.test.ts) (885 lines) and
355
- [`face-detection.test.ts`](../src/plugins/face-detection.test.ts) (227 lines) under vitest.
356
-
357
- ### 4.4 Carry-over gotchas already solved once — do not regress
358
-
359
- Documented in the current code and worth preserving in the rewrite:
360
-
361
- - **Stacked detection loops.** `isStartingDetection` guard
362
- ([`face-detection.ts:629-645`](../src/plugins/face-detection.ts#L629-L645)) — two "camera ready"
363
- signals each starting their own loop starved the iOS Safari main thread.
364
- - **Warm-up.** First inference compiles shaders and blocks for seconds; do it behind the spinner
365
- ([`face-detection.ts:745-787`](../src/plugins/face-detection.ts#L745-L787)). MediaPipe needs the
366
- equivalent.
367
- - **Teardown ordering.** `stopVideoStream` must cancel the loop *before* dropping the stream
368
- ([`face-camera.ts:464-473`](../src/plugins/face-camera.ts#L464-L473)).
369
- - **iOS autoplay.** Explicit `videoElement.play()` after `getUserMedia`
370
- ([`face-camera.ts:415-420`](../src/plugins/face-camera.ts#L415-L420)).
371
- - **Inference throttle.** `DETECT_MIN_INTERVAL_MS = 200` (~5 fps) — a floor, not a cap. Actions may
372
- need faster sampling than 5 fps (blink especially); re-measure, don't assume.
373
- - **Debug flag.** `window.__PW_FACE_DEBUG__` / `enableImageDebug` gating and the `[pw-face]` log
374
- prefix — extend, don't replace.
375
-
376
- ---
377
-
378
- ## 5. Sub-story split + ETA
379
-
380
- Draft titles only, FE point of view. Day counts assume **full AI-assisted coding** and include
381
- review + on-device verification, which is where the real time goes.
382
-
383
- | # | Draft FE story title | Days | Maps to |
384
- |---|---|---|---|
385
- | F0 | Selfie flow foundation — stage state machine, single detection loop, config module, debug HUD | 2.0 | prereq for all |
386
- | F1 | Swap face detection engine to MediaPipe FaceLandmarker; retire face-api.js and its 15 MB models | 2.0 | sc-348063 |
387
- | F2 | Real-time face-visibility + framing check with new prompt copy (forehead/eyes/nose/mouth/chin) | 2.0 | sc-342059 / sc-348318 |
388
- | F3 | Hold-still stage: countdown ring (white→green), stability gate, reference-image capture | 1.5 | **sc-346306** |
389
- | F4 | Six liveness action detectors with return-to-neutral rule + per-action thresholds | 2.5 | **sc-348063** |
390
- | F5 | On-device threshold tuning pass (devices / lighting / glasses / skin tone) | 2.0 | **sc-348063** |
391
- | F6 | Action sequencer: random 2-of-6, centred-before-turn gate, 2-pass/3-action verdict, per-action capture | 2.0 | **sc-342060** |
392
- | F7 | Action UI: coloured ring states, action icons + animated demos, per-action success/fail feedback | 1.5 | **sc-342060** + Figma |
393
- | F8 | Multi-image submission + analysing/result screens + 3-attempt limit & retry dialogs | 2.0 | sc-342061 *(blocked on API)* |
394
- | F9 | Video guide dialog (play/pause/skip, auto re-pop after failure) | 1.0 | sc-342058 |
395
- | F10 | Desktop layout + QR-scan variants of the new selfie screens | 1.5 | sc-342062 / Figma |
396
- | F11 | i18n copy migration + unit tests for detectors & sequencer | 1.0 | all |
397
- | F12 | Cross-device QA, perf profiling, bug-fix buffer | 2.0 | all |
398
-
399
- **Totals**
400
-
401
- - The three given stories only (F3 + F4 + F5 + F6 + F7) = **9.5 days**
402
- - …but they require F0 + F1 underneath = **+4.0 days → ~13.5 days realistic**
403
- - **Whole FE epic ≈ 23 days** against a **2026-08-31** deadline
404
-
405
- **Sequencing**
406
-
407
- - F0 + F1 are the critical path — nothing else starts cleanly until they land.
408
- - F2 can run in parallel with F3 once F1 is done (different consumers of the same signals).
409
- - **F5 is calendar-bound**, not effort-bound — it needs real devices in real lighting. Start it the
410
- moment F4's first detector works, not after all six are written.
411
- - F8 is gated on the `pipwave-ekyc-sdk` package bump (§3.2). If that slips, build F8 against a
412
- mocked contract and swap the call site.
413
- - F10 is the safest thing to cut if the deadline compresses; F9 is next safest.
414
-
415
- ---
416
-
417
- ## 6. Immediate next actions
418
-
419
- 1. **Decide the engine** — MediaPipe as the single engine vs. keeping the ONNX visibility model as a
420
- second inference (§4.2). Blocks F1, which blocks everything.
421
- 2. **Get the `uploadSelfie` payload shape** for action images from the BE team and a
422
- `pipwave-ekyc-sdk` version bump (§3.2). Blocks F8.
423
- 3. **Update the six stories' Figma links** from the archived `8671-9428` to the approved page
424
- `8671:9427` (§0.1), and reconcile sc-342059's prompt copy with the Figma annotation (§1.2).
425
- 4. **Confirm image count + capture timing** — Figma still asks it in a section title (§3.3).
426
-
427
- ---
428
-
429
- ## Appendix — coverage caveat
430
-
431
- The Figma MCP hit its per-seat call limit during this analysis. Read in full detail:
432
- `real-time checking` (`9158:6586`) and `hold still` (`9158:6652`), including the copy annotation.
433
- Read as **structure only, not pixels**: the six action frames (`9158:6639`), each-action-result
434
- (`9158:6634`), the failed-attempt dialogs (`9158:6597`, `9158:6653`), the session-result screens
435
- (`9158:6505`), face-scan intro (`9158:6418`), and the entire desktop section (`9235:7565`).
436
-
437
- Component names, variant names, section names, frame names and the full node tree for the approved
438
- page **are** confirmed — that's what §1.1 is built from. The visual detail of the action screens
439
- (ring colours per state, icon placement, animation timing) still needs a pass before F7.