yunti-browser-runtime 0.1.3 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,808 @@
1
+ # Next Major Plan: 0.2.0 Best Browser Automation Runtime
2
+
3
+ This document is the durable planning contract for the next major development
4
+ cycle after `yunti-browser-runtime@0.1.3`.
5
+
6
+ ## Goal
7
+
8
+ Build the most useful browser automation operation layer for AI agents and
9
+ developers: local-first, LLM-agnostic, MCP-native, and able to operate the
10
+ user's real Chrome/Edge browser through fine-grained, inspectable tools.
11
+
12
+ `0.2.0` should strengthen Yunti's own direction while absorbing the best
13
+ practical ideas from the broader browser automation ecosystem:
14
+
15
+ - observe pages in a compact, agent-friendly text form;
16
+ - act through stable element ids and robust DOM action semantics;
17
+ - verify each step with clear recovery hints;
18
+ - keep sensitive page content redacted by default;
19
+ - combine page-level DOM automation with browser-level CDP, tabs, screenshots,
20
+ network, console, file upload, and diagnostics;
21
+ - make both human operators and external agents confident about what happened
22
+ and what to do next;
23
+ - preserve the current local-first, zero-config install path.
24
+
25
+ ## Product Promise
26
+
27
+ `0.2.0` should let any MCP agent operate the user's already-open local
28
+ Chrome/Edge browser in an observable, recoverable, and auditable way without
29
+ asking the user to understand CDP, CSS selectors, browser-extension tokens, or
30
+ complex setup.
31
+
32
+ The practical promise is:
33
+
34
+ - fewer mis-clicks because agents act through fresh observed uids;
35
+ - fewer blind retries because actions return recovery hints;
36
+ - less user setup friction because the local zero-config path remains the
37
+ default;
38
+ - clearer safety boundaries because observation, diagnostics, and screenshots
39
+ document what they do and do not redact;
40
+ - stronger real-browser workflows because Yunti keeps tabs, CDP, screenshots,
41
+ network, console, file upload, and diagnostics alongside page-level DOM
42
+ automation.
43
+
44
+ Execution should stay incremental. The first implementation slice should focus
45
+ on Page Agent / browser-use style page observation and indexed actions. Other
46
+ ecosystem lessons are backlog inputs for later phases after the observe/action
47
+ spine is stable.
48
+
49
+ ## Planning Discipline Before Code
50
+
51
+ Before each implementation slice, update this document and
52
+ `docs/EXECUTION_PLAN.md` with the concrete scope, non-goals, acceptance checks,
53
+ and the reason the slice strengthens Yunti's own product direction.
54
+
55
+ This prevents two failure modes:
56
+
57
+ - copying a reference project's architecture instead of extracting useful
58
+ concepts;
59
+ - starting a broad rewrite before the current Yunti tool surface has a stable
60
+ incremental path forward.
61
+
62
+ For `0.2.0`, code should move only after the relevant phase has:
63
+
64
+ - a user-facing capability statement;
65
+ - a stable tool/API contract or compatibility rule;
66
+ - an explicit "not this phase" list;
67
+ - tests or smoke checks that prove the slice works;
68
+ - documentation updates that an external agent can follow.
69
+
70
+ ## Yunti-First Principle
71
+
72
+ Yunti should not become a clone of any single existing tool. Every browser
73
+ automation project is a source of ingredients, not the target architecture.
74
+
75
+ Yunti's distinctive strengths are:
76
+
77
+ - local MCP bridge that any agent can connect to;
78
+ - fine-grained `yunti_*` tools instead of a single opaque task runner;
79
+ - real user Chrome/Edge state, tabs, login sessions, screenshots, CDP, network
80
+ observations, and console diagnostics;
81
+ - zero LLM API dependency inside the runtime;
82
+ - zero-config local install path after extension loading;
83
+ - security boundaries based on local loopback, optional token hardening, and
84
+ explicit redaction.
85
+
86
+ ## Ecosystem Lessons To Absorb
87
+
88
+ The order matters. Start with Page Agent / browser-use for P6.1 and P6.2, then
89
+ fold in other automation-system lessons step by step.
90
+
91
+ ### browser-use / Page Agent
92
+
93
+ What to absorb first:
94
+
95
+ - text-oriented DOM observation;
96
+ - indexed interactive elements;
97
+ - scroll hints and scrollable-container metadata;
98
+ - better DOM action semantics;
99
+ - task history/activity concepts for optional diagnostics;
100
+ - content masking hooks.
101
+
102
+ How Yunti should differ:
103
+
104
+ - do not move the LLM loop into the runtime core;
105
+ - keep Yunti as the browser hands/eyes for any external agent;
106
+ - preserve fine-grained tools instead of only natural-language task execution.
107
+
108
+ Reference audit from `alibaba/page-agent`:
109
+
110
+ - `PageController.getBrowserState()` is the closest reference for P6.1:
111
+ combine URL/title, page metrics, scroll hints, and compact interactive DOM
112
+ text into one agent-facing observation.
113
+ - `updateTree()` separates observation refresh from action execution. Yunti
114
+ should mirror the idea by making `yunti_observe_page` the explicit refresh
115
+ step before uid-based actions.
116
+ - `selectorMap` and simplified HTML show why indexed elements are easier for
117
+ agents than raw CSS selectors. Yunti should adapt this into stable `uid`
118
+ fields while keeping its current `yunti_*` tool style.
119
+ - `data-scrollable` metadata is worth absorbing because multi-panel business
120
+ apps often need container scrolling instead of page scrolling.
121
+ - Action code around click, input, select, and scroll is useful as an event
122
+ sequencing reference, especially for contenteditable and scrollable
123
+ containers.
124
+ - The prompt rules around evaluating previous action results, avoiding blind
125
+ retries, and scrolling only when there are pixels above/below should inform
126
+ Yunti skill/tool hints.
127
+
128
+ What not to copy:
129
+
130
+ - Page Agent's built-in LLM loop is not Yunti's runtime responsibility.
131
+ - Page Agent's hub/sidebar/operator UI should not become required for Yunti's
132
+ default local path.
133
+ - Page Agent's single-page-task framing should not remove Yunti's tab, CDP,
134
+ network, console, screenshot, file upload, and diagnostics capabilities.
135
+ - Numeric indexes can inspire the UX, but Yunti should keep `uid` naming
136
+ aligned with existing `yunti_take_snapshot` and action tools.
137
+
138
+ Page Agent / browser-use should remain the first DOM observation/action
139
+ reference only. Yunti's product boundary remains broader: MCP connection,
140
+ real-browser sessions, tab routing, CDP escape hatches, network/console
141
+ diagnostics, screenshots, uploads, local installation, and explicit safety
142
+ controls.
143
+
144
+ ### Playwright / Puppeteer / Selenium
145
+
146
+ What to absorb:
147
+
148
+ - reliable action semantics for click, fill, select, keyboard, upload, wait, and
149
+ navigation;
150
+ - clear locator strategy and auto-waiting behavior;
151
+ - trace-style debugging and reproducible action logs;
152
+ - screenshot/video/artifact thinking for failures;
153
+ - browser context and tab management discipline.
154
+
155
+ How Yunti should differ:
156
+
157
+ - operate the user's already-open real browser instead of forcing a separate
158
+ test-runner profile;
159
+ - expose capabilities through MCP tools that external agents can compose;
160
+ - keep install and runtime local-first.
161
+
162
+ ### Chrome DevTools Protocol
163
+
164
+ What to absorb:
165
+
166
+ - complete browser-level power: targets, runtime evaluation, screenshots,
167
+ network, console, performance, DOM, file upload, and emulation;
168
+ - precise low-level escape hatches when high-level DOM actions are not enough.
169
+
170
+ How Yunti should differ:
171
+
172
+ - wrap CDP with safer, friendlier workflows and recovery hints;
173
+ - avoid forcing agents to know raw CDP for common tasks.
174
+
175
+ ### BrowserGym / Web Evaluation Harnesses
176
+
177
+ What to absorb:
178
+
179
+ - scenario-based task evaluation;
180
+ - deterministic smoke pages and fixtures;
181
+ - measurable success/failure criteria for browser actions;
182
+ - regression suites for common interaction patterns.
183
+
184
+ How Yunti should differ:
185
+
186
+ - prioritize real local browser usefulness over benchmark-only behavior.
187
+
188
+ ### Browser Extensions And Local Runtimes
189
+
190
+ What to absorb:
191
+
192
+ - easy install/update paths;
193
+ - visible runtime health and connection status;
194
+ - permission transparency;
195
+ - optional debugging UI that does not block first use.
196
+
197
+ How Yunti should differ:
198
+
199
+ - keep the default popup zero-config;
200
+ - make richer UI optional and diagnostic, not mandatory.
201
+
202
+ When a design choice conflicts with Yunti's current strengths, keep Yunti's
203
+ current strengths.
204
+
205
+ ## Reference Absorption Ladder
206
+
207
+ Yunti should absorb browser automation ideas in this order:
208
+
209
+ 1. **Observe/action spine**: Page Agent and browser-use inspire compact page
210
+ observation, indexed interactive elements, and scroll hints.
211
+ 2. **Action reliability**: Playwright, Puppeteer, and Selenium inspire more
212
+ predictable clicking, filling, selecting, waiting, and upload behavior.
213
+ 3. **Diagnostics and escape hatches**: CDP remains Yunti's deep browser-control
214
+ layer for screenshots, network, console, targets, runtime evaluation, and
215
+ lower-level recovery.
216
+ 4. **Regression confidence**: BrowserGym-style fixtures and scenario checks
217
+ help measure whether common browser actions keep working.
218
+ 5. **Operator experience**: extension/runtime patterns help reduce install
219
+ friction and make connection health visible without adding mandatory UI.
220
+
221
+ Each rung must be additive. If a new idea requires removing fine-grained
222
+ `yunti_*` tools, hiding browser state behind an opaque task runner, or requiring
223
+ an LLM provider key inside the runtime, it does not belong in the default
224
+ `0.2.0` path.
225
+
226
+ ## Non Goals
227
+
228
+ - Do not require an LLM API key inside Yunti Browser Runtime.
229
+ - Do not replace fine-grained `yunti_*` MCP tools with a single black-box
230
+ natural-language `execute_task` tool.
231
+ - Do not make a hub tab, side panel, or UI console mandatory for first use.
232
+ - Do not mix remote multi-user mode into the local single-user core.
233
+ - Do not remove CDP, network, console, screenshot, or tab-level capabilities in
234
+ favor of a narrower page-only agent abstraction.
235
+ - Do not treat compatibility or API parity with any reference project as a goal.
236
+ - Do not optimize only for benchmarks while making real user-browser automation
237
+ harder.
238
+ - Do not weaken the local loopback security boundary or publish real secrets in
239
+ docs, config output, logs, or tests.
240
+
241
+ ## Release Theme
242
+
243
+ `0.2.0` should be a compatibility-preserving enhancement release that makes
244
+ Yunti feel like the most practical browser automation layer for agents. Existing
245
+ tools continue to work, while agents are guided toward a stronger default
246
+ workflow:
247
+
248
+ ```text
249
+ yunti_observe_page -> yunti_click/fill/select/scroll by uid -> observe/verify
250
+ ```
251
+
252
+ `yunti_take_snapshot` remains available. Newer docs and skill instructions should
253
+ prefer `yunti_observe_page` once implemented.
254
+
255
+ Success metrics:
256
+
257
+ - an agent can identify the active browser page and interact with it without
258
+ asking the user for tokens or selectors;
259
+ - `observe -> action by uid -> observe` becomes the documented default path;
260
+ - common failures return actionable next steps instead of opaque errors;
261
+ - sensitive credential-like values are not exposed by default observation;
262
+ - existing `yunti_get_page_snapshot`, `yunti_take_snapshot`, CDP, screenshot,
263
+ network, console, upload, and tab tools remain compatible;
264
+ - at least one deterministic fixture proves
265
+ `observe -> click/fill/select/scroll -> observe verify -> recover`.
266
+
267
+ Tool choice:
268
+
269
+ - Use `yunti_observe_page` for the normal agent loop once available.
270
+ - Use `yunti_get_page_snapshot` for lightweight route/title/text/auth overview.
271
+ - Use `yunti_take_snapshot` as the compatibility path for existing uid-based
272
+ action workflows.
273
+ - Use screenshot tools for visual inspection, with the assumption that
274
+ screenshots may contain all visible sensitive page content.
275
+ - Use CDP tools for low-level browser recovery or capabilities not covered by
276
+ high-level DOM tools.
277
+
278
+ ## Phase Plan
279
+
280
+ ### P6.0 Product Direction Guardrail
281
+
282
+ Status: complete.
283
+
284
+ The release direction is Yunti-first and ecosystem-informed:
285
+
286
+ - keep Yunti local-first, LLM-agnostic, MCP-native, and real-browser-oriented;
287
+ - absorb useful ideas from Playwright, Puppeteer, Selenium, CDP, browser-use,
288
+ Page Agent, BrowserGym, and extension runtimes;
289
+ - do not clone a single project or weaken Yunti's current strengths.
290
+
291
+ ### P6.1 Agent-Friendly Page Observation
292
+
293
+ Status: in progress; P6.1.4 real-browser closure is pending Playwright/Chromium
294
+ availability.
295
+
296
+ Create `yunti_observe_page`, a higher-level observation tool. Use Page Agent's
297
+ PageController/browser-state approach as the first concrete reference, adapted
298
+ to Yunti's MCP/extension architecture. It returns:
299
+
300
+ - current `browserSessionId`, URL, title, origin, and captured timestamp;
301
+ - viewport size, page size, scroll position, pages above/below, and pixels
302
+ above/below;
303
+ - compact text DOM tree for visible interactive elements;
304
+ - stable uid for each actionable element;
305
+ - role, tag, name/text, label, placeholder, value preview policy, rect, disabled
306
+ state, and visibility state;
307
+ - scrollable container metadata, including vertical/horizontal scroll capacity;
308
+ - optional full-page mode with bounded element and text limits;
309
+ - clear next-step hints when no elements are visible, the page is restricted, or
310
+ the content script is stale.
311
+
312
+ First slice:
313
+
314
+ - add the tool as an additive replacement path, not a breaking change to
315
+ `yunti_get_page_snapshot` or `yunti_take_snapshot`;
316
+ - prefer content-script DOM collection first so basic observation works without
317
+ CDP attachment;
318
+ - return both structured fields and a compact text tree so agents can choose
319
+ programmatic or prompt-oriented consumption;
320
+ - keep uid generation scoped to the latest observation and make stale uid errors
321
+ point back to `yunti_observe_page`;
322
+ - redact sensitive input values by default, even before the full P6.4 policy is
323
+ complete.
324
+
325
+ Input contract:
326
+
327
+ - `browserSessionId`: optional target browser session.
328
+ - `mode`: `viewport` by default, with bounded `fullPage` support later.
329
+ - `maxElements`: bounded element limit.
330
+ - `maxTextLength`: bounded text limit.
331
+ - `includeHidden`: defaults to false.
332
+ - `includeTextTree`: defaults to true.
333
+ - `includeRects`: defaults to true.
334
+ - `redaction`: defaults to `balanced`; `strict` is allowed for stronger
335
+ privacy; `off` is reserved for explicit local debugging only and must not be
336
+ used by default agent workflows.
337
+
338
+ Output contract:
339
+
340
+ - `observationId`, `browserSessionId`, `capturedAt`, and `uidMapVersion`.
341
+ - `page`: URL, sanitized URL preview, title, origin, and document ready state.
342
+ - `viewport`: width, height, device pixel ratio.
343
+ - `scroll`: x/y position, page size, pixels/pages above and below.
344
+ - `elements[]`: `uid`, `role`, `tag`, `name`, `text`, `label`,
345
+ `placeholder`, `valuePreview`, `valueRedacted`, `type`, `hrefPreview`,
346
+ `rect`, `visible`, `disabled`, `editable`, `checked`, `selected`,
347
+ `scrollable`, and `containerUid` where applicable.
348
+ - `textTree`: compact line-oriented representation of visible interactive
349
+ elements.
350
+ - `scrollableContainers[]`: `uid`, `tag`, `name`, `rect`, `scrollTop`,
351
+ `scrollHeight`, `clientHeight`, `canScrollVertical`,
352
+ `canScrollHorizontal`, and pixels available in each direction.
353
+ - `limits`: returned/max element and text counts plus truncation flags.
354
+ - `redactions`: redaction mode, count, categories, and whether values were
355
+ suppressed.
356
+ - `hints[]` and `warnings[]`: next-step and degraded-observation guidance.
357
+
358
+ Uid lifecycle:
359
+
360
+ - uids from `yunti_observe_page` are valid for the latest observation in the
361
+ current `browserSessionId`, not permanent selectors.
362
+ - Actions should accept uids from either `yunti_observe_page` or
363
+ `yunti_take_snapshot`, but errors must explain which refresh step is needed.
364
+ - Stale uid errors should use a structured code such as
365
+ `STALE_OBSERVATION_UID` and recommend calling `yunti_observe_page` again.
366
+ - uid maps must stay in memory only; they must not be written to learning
367
+ memory, task history, or persistent diagnostics.
368
+
369
+ Minimum P6.1 redaction baseline:
370
+
371
+ - default `balanced` redaction hides password fields, hidden token-like fields,
372
+ token/secret/auth/cookie/session/api-key/credential/otp-like values, JWT-like
373
+ values, Bearer-like values, private-key-like values, and long random-looking
374
+ strings;
375
+ - observation uses an attribute allowlist rather than returning arbitrary DOM
376
+ attributes;
377
+ - URL query values are summarized or partially redacted by default;
378
+ - screenshot redaction is not implied by DOM redaction;
379
+ - P6.4 will deepen the policy with optional strict PII redaction and aligned
380
+ DOM/network/console behavior.
381
+
382
+ Restricted-page and degraded-state hints:
383
+
384
+ - content script not available;
385
+ - unsupported URL such as browser internal pages;
386
+ - cross-origin iframe content omitted;
387
+ - stale browser session;
388
+ - tab changed or no active browser page;
389
+ - no visible interactive elements;
390
+ - page still loading or action verification uncertain.
391
+
392
+ Page Agent mapping:
393
+
394
+ - `BrowserState.header` maps to Yunti's `page`, `viewport`, `scroll`, and
395
+ `hints` fields.
396
+ - `BrowserState.content` maps to Yunti's `textTree` plus structured
397
+ `elements[]`.
398
+ - `BrowserState.footer` maps to Yunti's above/below scroll hints.
399
+ - `selectorMap` maps to Yunti's latest observation uid map.
400
+ - Page Agent's `data-scrollable` string maps to Yunti's structured
401
+ `scrollableContainers[]`.
402
+
403
+ Not this phase:
404
+
405
+ - no LLM task loop inside the runtime;
406
+ - no Page Agent hub tab, side panel, or visual console requirement;
407
+ - no full Playwright-style locator engine;
408
+ - no benchmark harness beyond a focused fixture/smoke test;
409
+ - no remote multi-user browser orchestration.
410
+
411
+ Acceptance:
412
+
413
+ - `yunti_observe_page` works on a simple page without CDP attachment.
414
+ - It returns a readable tree that an LLM can use without raw selectors.
415
+ - It includes scroll hints comparable to "pixels below" and "pages below".
416
+ - It redacts sensitive values by default.
417
+ - Unit tests cover schema, routing, stale-session recovery, and a fixture page.
418
+ - E2E smoke can use `observe -> click uid -> observe` on the test page.
419
+ - P6.1 is complete only after schema tests, bridge routing tests, content
420
+ fixture tests, real-browser smoke, skill guidance, and tool usage hints are
421
+ updated.
422
+
423
+ ### P6.2 Robust DOM Action Layer
424
+
425
+ Status: completed through real-browser validation.
426
+
427
+ Extract DOM actions from the current content/tool handler code into a focused
428
+ module such as `extension/dom-actions.js`. Use Page Agent's action sequencing
429
+ as the first reference for click/input/select/scroll behavior, then evolve with
430
+ additional lessons from Playwright/CDP in later increments.
431
+
432
+ Scope:
433
+
434
+ - use consistent pointer/mouse event order for click and hover;
435
+ - improve text input for input, textarea, select, and contenteditable;
436
+ - support select-by-visible-text and select-by-value;
437
+ - improve uid-targeted fill so contenteditable and rich text editors get better
438
+ fallback behavior;
439
+ - support scrolling the document or a scrollable container by uid;
440
+ - return structured action results with before/after summary where useful.
441
+
442
+ Action priority:
443
+
444
+ - P0: click by uid, fill input/textarea/contenteditable by uid, and verify by
445
+ observing again.
446
+ - P0: page and container scrolling recovery.
447
+ - P1: select-by-visible-text, select-by-value, dropdown recovery, tab selection,
448
+ and wait/verify templates.
449
+ - P2: deeper diagnostics that connect console/network/screenshot/CDP evidence
450
+ to action recovery hints.
451
+ - Not now: canvas automation, CAPTCHA solving, complex drag workflows, or a
452
+ black-box task runner.
453
+
454
+ First slice:
455
+
456
+ - keep existing `yunti_click`, `yunti_hover`, `yunti_fill`, `yunti_select`,
457
+ `yunti_type_text`, `yunti_press_key`, and `yunti_scroll` names stable;
458
+ - make uid-based actions work naturally after `yunti_observe_page`;
459
+ - improve errors so agents understand whether to observe again, scroll, wait, or
460
+ switch tabs;
461
+ - keep selector and coordinate fallbacks available for recovery and debugging.
462
+
463
+ Action result contract:
464
+
465
+ - return the action name, `browserSessionId`, target uid/selector/coordinate,
466
+ success flag or clear failure code, and a human-readable `nextStepHint`;
467
+ - include before/after summaries when useful, such as scroll position, input
468
+ value length, selected option, URL/title change, or detected toast/popup;
469
+ - mark recoverable failures explicitly so agents know whether to observe,
470
+ scroll, wait, switch tabs, or ask the user;
471
+ - update `yunti_select` planning to support uid and visible text, while keeping
472
+ the current selector/value path compatible.
473
+
474
+ Page Agent mapping:
475
+
476
+ - `clickElement()` informs Yunti's pointer/mouse/focus/click event order.
477
+ - `inputTextElement()` informs contenteditable fallback sequencing and
478
+ verification.
479
+ - `selectOptionElement()` informs select-by-visible-text behavior.
480
+ - `scrollVertically()` and `scrollHorizontally()` inform uid-targeted container
481
+ scroll behavior and reached-edge result messages.
482
+
483
+ Not this phase:
484
+
485
+ - no large action-runner abstraction that hides individual tool calls;
486
+ - no mandatory replay/trace UI;
487
+ - no removal of CDP-based fallback behavior.
488
+
489
+ Acceptance:
490
+
491
+ - Existing `yunti_click`, `yunti_hover`, `yunti_fill`, `yunti_select`,
492
+ `yunti_type_text`, `yunti_press_key`, and `yunti_scroll` tests keep passing.
493
+ - New tests cover contenteditable, select option text, scrollable containers,
494
+ missing uid, stale uid, and action result shape.
495
+ - Tool errors recommend running `yunti_observe_page` before retrying.
496
+
497
+ ### P6.3 Agent Workflow Contract
498
+
499
+ Status: planned.
500
+
501
+ Make the recommended agent workflow explicit in docs, skill, and tool hints.
502
+
503
+ Scope:
504
+
505
+ - update `skills/yunti-browser-runtime/SKILL.md` with `observe -> act -> verify`;
506
+ - add guidance to evaluate the previous action before retrying;
507
+ - discourage repeated blind retries and coordinate-only actions;
508
+ - add examples for form filling, tab switching, dropdowns, and scroll recovery;
509
+ - add a copyable external-agent prompt for the default Yunti workflow;
510
+ - define verification templates for URL changes, visible text, toast messages,
511
+ element disappearance, input value changes, list/table updates, and dialogs;
512
+ - define recovery taxonomy for stale sessions, stale uids, hidden/disabled
513
+ elements, covered elements, pending navigation, wrong tab, and restricted
514
+ pages;
515
+ - consider local task-history tools:
516
+ - `yunti_begin_task`
517
+ - `yunti_record_step`
518
+ - `yunti_get_task_history`
519
+ - `yunti_end_task`
520
+
521
+ Acceptance:
522
+
523
+ - Skill and tool hints consistently prefer `yunti_observe_page`.
524
+ - Agents can recover from failed action by observing again and using a fresh uid.
525
+ - `yunti_observe_page` release includes updated `yunti_get_tool_usage_hints`;
526
+ this cannot wait until late P6.3.
527
+ - Optional task-history tools, if implemented, store no secrets and are scoped to
528
+ the local user.
529
+
530
+ ### P6.4 DOM Redaction And Page Content Policy
531
+
532
+ Status: complete for the current 0.2.0 planning scope. P6.4.1 DOM strict
533
+ redaction, P6.4.2 memory/console diagnostic sanitization, and P6.4.3 raw
534
+ diagnostics / screenshot non-redaction guidance are implemented.
535
+
536
+ Promote DOM snapshot redaction to a first-class runtime capability.
537
+
538
+ Scope:
539
+
540
+ - redact password fields, hidden token-like fields, auth-like attributes, and
541
+ long credential-looking strings;
542
+ - in `strict`, redact likely email, phone, address-like, and Luhn-valid
543
+ payment-card-like values across page titles, labels, names, visible text,
544
+ placeholders, values, selected option text, and option text;
545
+ - add local configuration for redaction modes:
546
+ - `strict`
547
+ - `balanced`
548
+ - `off` for local debugging only;
549
+ - show redaction metadata in observation output;
550
+ - keep network, console, and learning-memory redaction behavior aligned with
551
+ DOM redaction.
552
+
553
+ P6.1 already carries the minimum DOM observation baseline. P6.4 is for expanding
554
+ coverage, making strict mode robust, aligning DOM/network/console terminology,
555
+ and documenting cleanup/diagnostic retention behavior.
556
+
557
+ Implemented slices:
558
+
559
+ - P6.4.1: strict DOM observation redaction for common PII-like text surfaces.
560
+ - P6.4.2: shared MCP text sanitization before learning-memory writes and
561
+ console diagnostic caching, with raw CDP events documented as the explicit
562
+ low-level exception.
563
+ - P6.4.3: raw CDP and screenshot tools now expose usage guidance for sanitized
564
+ tool preference, method/limit scoping, CDP cleanup, screenshot non-redaction,
565
+ viewport preference, and safe summarization.
566
+
567
+ Security caveats to keep explicit:
568
+
569
+ - DOM redaction is not complete data-loss prevention; visible business data can
570
+ still be returned.
571
+ - Screenshots are real visible pixels and should be treated as sensitive unless
572
+ future pixel masking exists.
573
+ - External agents may transmit observed page content according to their own
574
+ policies; Yunti can keep the runtime local but cannot promise what a third
575
+ party agent does with returned content.
576
+
577
+ Acceptance:
578
+
579
+ - Sensitive input values are not returned by default observation tools.
580
+ - Redaction behavior is deterministic and unit tested.
581
+ - Docs explain what is and is not redacted.
582
+
583
+ ### P6.5 Optional Local Runtime Console
584
+
585
+ Status: planned.
586
+
587
+ Create a non-required local console for debugging and operator confidence.
588
+
589
+ Scope:
590
+
591
+ - list bridge status and connected browser sessions;
592
+ - show recent MCP tool calls, action results, errors, and stale-session events;
593
+ - provide a stop/cancel affordance for pending browser tasks;
594
+ - display extension version and bridge version mismatch warnings;
595
+ - keep popup zero-config and small.
596
+
597
+ Acceptance:
598
+
599
+ - The console is optional and never blocks the first-run path.
600
+ - It exposes no raw secrets.
601
+ - It shows sanitized summaries by default, not raw DOM, screenshots, console
602
+ payloads, network bodies, or credential-like values.
603
+ - It helps diagnose "extension loaded but no page connected" without asking the
604
+ user to inspect logs manually.
605
+
606
+ P6.5.1 completed:
607
+
608
+ - Optional `/console` page served by the local bridge.
609
+ - Protected `/console/state` JSON with sanitized session, pending request,
610
+ diagnostic-count, and recent activity summaries.
611
+ - Protected `/console/cancel-pending` action for queued/runtime-pending browser
612
+ requests, with an explicit boundary that it does not undo browser-side
613
+ effects that already happened.
614
+ - `yunti-browser-runtime console` CLI entry that starts the bridge and prints
615
+ the console URL.
616
+
617
+ P6.5.2 completed:
618
+
619
+ - `HEAD /console` works for simple local probes.
620
+ - Console state includes runtime version, expected extension version, connected
621
+ session extension versions, and warning entries.
622
+ - Extension sessions report `client.extensionVersion`.
623
+ - Console UI renders version metadata and warning cards.
624
+ - `doctor` JSON and human summary include the optional console URL.
625
+
626
+ P6.5.3 completed:
627
+
628
+ - Real-browser E2E validates `HEAD /console`, `/console`, `/console/state`,
629
+ runtime/extension version alignment, connected page state, and absence of
630
+ no-page/version-mismatch warnings after a page registers.
631
+ - Remaining stale-session and extension-not-loaded UX refinement can continue as
632
+ polish, but P6.5 no longer blocks the 0.2.0 direction.
633
+
634
+ ### P6.6 Extension Distribution Readiness
635
+
636
+ Status: complete for `0.2.0` npm/unpacked release scope. P6.6.1, P6.6.2, and
637
+ P6.6.3 are complete. P6.6.4 is deferred to the post-0.2 store-candidate track.
638
+
639
+ Prepare for browser-store distribution to reduce manual loading friction.
640
+
641
+ Scope:
642
+
643
+ - review Chrome Web Store and Edge Add-ons permission text;
644
+ - write public privacy and permission rationale docs;
645
+ - decide whether broad host permissions remain required or can be narrowed;
646
+ - produce store-ready icons, screenshots, and package artifacts;
647
+ - keep unpacked extension support for developers and npm installs.
648
+
649
+ Acceptance:
650
+
651
+ - A store submission checklist exists.
652
+ - Permission rationale is understandable to non-developers.
653
+ - Release packaging can produce both npm package and extension-store artifact.
654
+
655
+ P6.6.1 completed:
656
+
657
+ - Added `docs/EXTENSION_DISTRIBUTION.md` as the durable browser-store readiness
658
+ audit.
659
+ - Audited the current `extension/manifest.json` permissions and broad host
660
+ access without changing runtime behavior.
661
+ - Captured Chrome Web Store and Edge Add-ons policy references, store material
662
+ checklist, privacy disclosure topics, and permission narrowing options.
663
+ - Linked the audit from README, Security notes, and Roadmap so it stays visible
664
+ before any store-distributed release.
665
+
666
+ P6.6.2 scope:
667
+
668
+ - Draft store-facing permission and privacy copy for `debugger`, `tabs`,
669
+ `webRequest`, broad `http` / `https` access, localhost bridge access,
670
+ screenshots, DOM observation, network/console diagnostics, and learning
671
+ memory.
672
+ - Keep this as copy and decision support first; manifest narrowing remains a
673
+ later tested implementation slice.
674
+
675
+ P6.6.2 completed:
676
+
677
+ - Added `docs/EXTENSION_STORE_COPY.md` as the draft store copy pack.
678
+ - Drafted short and long descriptions, permission rationale, privacy-policy
679
+ language, store data disclosure guidance, reviewer notes, and final review
680
+ checklist.
681
+ - Kept the current extension manifest and runtime behavior unchanged.
682
+
683
+ P6.6.3 scope:
684
+
685
+ - Decide the pre-store permission strategy for broad host permissions and
686
+ `webRequest`: keep broad install-time access, move to optional host
687
+ permissions, optionalize diagnostics, or explicitly defer store submission.
688
+ - If behavior changes, update `extension/manifest.json`, extension UX,
689
+ `mcp/tools.js`, Tool Guide, packaged skill, Security notes, and tests in one
690
+ small compatibility-preserving slice.
691
+
692
+ P6.6.3 completed:
693
+
694
+ - Added `docs/EXTENSION_PERMISSION_STRATEGY.md` as the pre-store permission
695
+ decision record.
696
+ - Kept the npm/unpacked developer path unchanged for `0.2.0`.
697
+ - Decided that the current broad-permission manifest should not be submitted to
698
+ browser stores unchanged by default.
699
+ - Defined a future store-candidate track around optional host access and
700
+ optional network diagnostics, without changing runtime behavior yet.
701
+
702
+ P6.6.4 deferred:
703
+
704
+ - Design the store-candidate permission UX and technical prototype plan:
705
+ optional host access, missing-permission recovery, optional diagnostics, and
706
+ the exact docs/tool/skill/test surfaces that would change.
707
+ - This is not a `0.2.0` npm/unpacked release blocker.
708
+
709
+ ### P7.1 0.2.0 Release Closure
710
+
711
+ Status: in progress.
712
+
713
+ Scope:
714
+
715
+ - bump package and extension versions to `0.2.0`;
716
+ - preserve the npm/unpacked developer path and current local runtime behavior;
717
+ - run metadata, release, real-browser E2E, and dry-run gates;
718
+ - ask for explicit user confirmation before formal npm publish.
719
+
720
+ ## Suggested Implementation Order
721
+
722
+ 1. P6.1 `yunti_observe_page`
723
+ 2. P6.2 robust DOM action layer
724
+ 3. P6.3 skill/tool workflow updates
725
+ 4. P6.4 DOM redaction policy
726
+ 5. P6.5 optional console
727
+ 6. P6.6 store distribution readiness
728
+
729
+ The first two phases should be treated as the technical core of `0.2.0`, but
730
+ they must remain additive enhancements to Yunti's MCP tool surface.
731
+
732
+ For the first implementation cycle, do not attempt to absorb every ecosystem
733
+ lesson at once. Complete the Page-Agent-informed observe/action foundation
734
+ first, then add Playwright/CDP/BrowserGym-style reliability and diagnostics in
735
+ separate follow-up increments.
736
+
737
+ ## Compatibility Rules
738
+
739
+ - Keep existing `yunti_*` tools unless a replacement has shipped and docs have a
740
+ migration path.
741
+ - Add aliases only when they reduce agent confusion.
742
+ - Avoid breaking package install, MCP config printer, doctor, and bridge startup.
743
+ - Preserve Yunti's CDP, tab, network, console, screenshot, and fine-grained DOM
744
+ action capabilities.
745
+ - Maintain Node.js `>=22`.
746
+ - Keep local default tokenless loopback behavior unless binding is non-loopback
747
+ or `YUNTI_BROWSER_BRIDGE_TOKEN` is explicitly configured.
748
+
749
+ ## Validation Gates
750
+
751
+ For every phase:
752
+
753
+ ```bash
754
+ npm run check
755
+ npm test
756
+ ```
757
+
758
+ When observation/action behavior changes:
759
+
760
+ ```bash
761
+ npm run release:check
762
+ YUNTI_E2E=1 npm run test:e2e
763
+ ```
764
+
765
+ P6.1/P6.2 fixture matrix:
766
+
767
+ - `observe-basic.fixture.html`: title, links, buttons, labels, inputs,
768
+ textarea, select, and visible element metadata.
769
+ - `observe-redaction.fixture.html`: password, hidden token, ordinary search
770
+ input, credential-looking strings, and redaction metadata.
771
+ - `observe-scroll.fixture.html`: long page and nested scroll container.
772
+ - `observe-dynamic.fixture.html`: DOM change after action and stale uid
773
+ recovery.
774
+ - `actions-form.fixture.html`: click, fill, select-by-text, select-by-value,
775
+ and event counters.
776
+ - `actions-contenteditable.fixture.html`: contenteditable edit and verification.
777
+ - `actions-scroll-container.fixture.html`: uid-targeted container scrolling and
778
+ reached-edge feedback.
779
+
780
+ The fixture suite is for deterministic local regression, not benchmark
781
+ optimization.
782
+
783
+ Before publishing `0.2.0`:
784
+
785
+ ```bash
786
+ npm run release:check
787
+ npm run release:publish
788
+ npm run release:verify-published
789
+ ```
790
+
791
+ ## Current Next Step
792
+
793
+ Start with docs-first P6.1 preparation:
794
+
795
+ 1. Confirm the `yunti_observe_page` output contract in this plan and
796
+ `docs/EXECUTION_PLAN.md`.
797
+ 2. Confirm how its uid map relates to existing `yunti_take_snapshot` uid
798
+ behavior.
799
+ 3. Confirm the first smoke fixture: `observe -> click uid -> observe`.
800
+ 4. Confirm default redaction rules for password/token-like values.
801
+ 5. Only then begin the implementation slice:
802
+
803
+ - define the `yunti_observe_page` schema in `mcp/tools.js`;
804
+ - implement content-script collection for page metrics and interactive text
805
+ tree;
806
+ - add routing through the extension dispatcher;
807
+ - add unit tests for schema/routing and an E2E smoke update;
808
+ - update skill and docs to prefer observe-first workflows.