@akshayram1/omnibrowser-agent 0.2.8 → 0.2.26

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@akshayram1/omnibrowser-agent",
-  "version": "0.2.8",
+  "version": "0.2.26",
   "private": false,
   "type": "module",
   "main": "./dist/lib.js",

package/plan.md

@@ -0,0 +1,114 @@
+---
+name: Local LLM Website Control
+overview: Improve the WebLLM integration so an on-device LLM can actually control a website reliably — fixing the thin prompt, adding a ready-made bridge helper, making JSON output robust, and fixing the extension memory bug.
+todos:
+  - id: prompt-module
+    content: Create src/core/prompt.ts with buildSystemPrompt() and buildUserMessage() that serialize the full PageSnapshot, history, memory, lastError, and action schema for the LLM
+    status: pending
+  - id: webllm-bridge-helper
+    content: Create src/core/webllm-bridge.ts with createWebLLMBridge(engine) factory and export it from src/lib/index.ts
+    status: pending
+  - id: planner-rich-prompt
+    content: Update planner.ts webllm path to use the rich prompt and add JSON parse retry logic using parsePlannerResult
+    status: pending
+  - id: fix-extension-memory
+    content: Fix memory and lastError persistence in content/index.ts and background/index.ts so reflection state survives across extension ticks
+    status: pending
+  - id: popup-modelid
+    content: Add model name input to popup UI (shown when webllm is selected), forwarded in planner.modelId
+    status: pending
+isProject: false
+---
+
+# Local LLM Website Control — WebLLM Improvements
+
+## Current State
+
+The library has two planners: `heuristic` (regex, no LLM) and `webllm` (delegates to `window.__browserAgentWebLLM`, manually wired by the user). The example bridge in `[docs/EMBEDDING.md](docs/EMBEDDING.md)` only passes `goal` and `history` to the LLM — the rich page snapshot (interactive elements with selectors, page text, URL) is completely ignored, and there is no error recovery or memory. This means WebLLM can't reliably pick selectors or multi-step through a site.
+
+## Architecture of improved WebLLM flow
+
+```mermaid
+flowchart TD
+    UserGoal["User Goal"] --> BrowserAgent
+    BrowserAgent --> collectSnapshot["collectSnapshot()\nobserver.ts"]
+    collectSnapshot --> Snapshot["PageSnapshot\n(url, title, textPreview, candidates[])"]
+    Snapshot --> planNextAction["planNextAction()\nplanner.ts"]
+    planNextAction -->|kind=heuristic| HeuristicPlan["Regex planner (existing)"]
+    planNextAction -->|kind=webllm| BridgeLookup["window.__browserAgentWebLLM"]
+    BridgeLookup -->|manual wiring| UserBridge["User-written bridge (existing)"]
+    BridgeLookup -->|createWebLLMBridge NEW| BuiltInBridge["createWebLLMBridge(engine)\nbuilt-in helper"]
+    BuiltInBridge --> RichPrompt["prompt.ts NEW\nbuildSystemPrompt()\nbuildUserMessage(input)\n- All 8 action types + schema\n- Candidates list with selectors\n- history + memory + lastError"]
+    RichPrompt --> WebLLMEngine["@mlc-ai/web-llm engine\n(runs fully on-device)"]
+    WebLLMEngine --> JSONOutput["Raw LLM output"]
+    JSONOutput --> ParseRetry["parsePlannerResult()\n+ retry on bad JSON"]
+    ParseRetry --> executeAction["executeAction()\nexecutor.ts"]
+```
+
+
+
+## Improvements
+
+### 1. Rich prompt module
+
+**New file:** `[src/core/prompt.ts](src/core/prompt.ts)`
+
+The LLM currently has no idea what elements are on the page. This module creates two functions:
+
+- `buildSystemPrompt(customPrompt?)` — explains all 8 action types (`click`, `type`, `navigate`, `extract`, `scroll`, `focus`, `wait`, `done`) with their exact JSON fields, explains the `PlannerResult` wrapper (`evaluation`, `memory`, `nextGoal`), and instructs the model to output only valid JSON
+- `buildUserMessage(input: PlannerInput)` — serializes the full context:
+  - Page URL, title, text preview (first 800 chars)
+  - Numbered list of candidates: `[1] button | selector: "button.submit" | text: "Submit" | label: "Submit form"`
+  - Step history
+  - Working memory (if any)
+  - Last error (if any), asking the LLM to recover
+
+### 2. `createWebLLMBridge(engine)` helper
+
+**New file:** `[src/core/webllm-bridge.ts](src/core/webllm-bridge.ts)`, exported from `[src/lib/index.ts](src/lib/index.ts)`
+
+Replaces the fragile manual bridge from EMBEDDING.md. Users just do:
+
+```ts
+import * as webllm from "@mlc-ai/web-llm";
+import { createWebLLMBridge } from "@akshayram1/omnibrowser-agent";
+
+const engine = await webllm.CreateMLCEngine("Llama-3.2-3B-Instruct-q4f16_1-MLC");
+window.__browserAgentWebLLM = createWebLLMBridge(engine);
+```
+
+Internally it calls `buildSystemPrompt` + `buildUserMessage`, sends them to `engine.chat.completions.create`, and runs `parsePlannerResult` on the output. If JSON parsing fails it sends one correction follow-up before giving up.
+
+### 3. JSON parse retry in planner
+
+**File:** `[src/core/planner.ts](src/core/planner.ts)`
+
+The existing `webllm` path passes the raw result directly to `toPlannerResult` with no error handling. Update it to:
+
+- Use `parsePlannerResult` from `[src/shared/parse-action.ts](src/shared/parse-action.ts)` (already handles fenced JSON)
+- On failure, send a correction message `"Invalid JSON. Reply with only a valid JSON object."` and retry once via the bridge's chat history
+- Return `{ action: { type: "done", reason: "..." } }` if retry also fails
+
+### 4. Fix extension memory persistence bug
+
+**Files:** `[src/content/index.ts](src/content/index.ts)`, `[src/background/index.ts](src/background/index.ts)`
+
+The content script returns `ContentResult` (which includes `reflection.memory`) but the background never writes it back into the session before the next `AGENT_TICK`. So WebLLM's cross-step memory is silently lost in the extension. Fix: after receiving a tick result, background updates `session.memory = result.reflection?.memory ?? session.memory` and `session.lastError = result.status === "error" ? result.message : undefined`.
+
+### 5. Model name input in extension popup
+
+**Files:** `[src/popup/index.html](src/popup/index.html)`, `[src/popup/index.ts](src/popup/index.ts)`
+
+When `webllm` is selected, show a text input for the model ID (e.g. `Llama-3.2-1B-Instruct-q4f16_1-MLC`) and forward it as `planner.modelId` in the session. Currently the popup hardcodes no model ID.
+
+## File change summary
+
+- `[src/core/prompt.ts](src/core/prompt.ts)` — New: `buildSystemPrompt()` + `buildUserMessage()`
+- `[src/core/webllm-bridge.ts](src/core/webllm-bridge.ts)` — New: `createWebLLMBridge(engine)` factory
+- `[src/lib/index.ts](src/lib/index.ts)` — Export `createWebLLMBridge`
+- `[src/core/planner.ts](src/core/planner.ts)` — Use rich prompt in webllm path + JSON retry
+- `[src/content/index.ts](src/content/index.ts)` — Return memory/lastError in tick response
+- `[src/background/index.ts](src/background/index.ts)` — Persist memory/lastError into session between ticks
+- `[src/popup/index.html](src/popup/index.html)` + `[src/popup/index.ts](src/popup/index.ts)` — Add model ID input
+- `[docs/EMBEDDING.md](docs/EMBEDDING.md)` — Update example to use `createWebLLMBridge`
+