@bobfrankston/msgapidefs 0.1.37 → 0.1.39

package/README.md

@@ -75,13 +75,18 @@ msger + msga cover every platform, so the Electron host has no platform to itsel
 **Do not do feature-parity work on msgview.** New features land in msger (and msga
 where it makes sense); msgview catches up only on demand, if a real need appears.
 
-What Electron alone could still do, and why it isn't a reason to keep syncing it:
-
-- **Unpacked Chrome extensions** (`-extension`) — wry has no extension API and
-  WebKitGTK has none at all. If Chrome features are needed, use Chrome directly.
-- **`-render` off Windows** — msger's capture goes through WebView2 DevTools, so it
-  is Windows-only; msgview's `capturePage()` works everywhere. Portable to
-  WebKitGTK's snapshot API if it ever matters.
+**As of 2026-08-01 msgview has no capability msger lacks.** The two things that
+used to be Electron-only are both settled:
+
+- **Unpacked Chrome extensions** (`-extension`) — not how add-ins are done here.
+  **`window.msgapi` (this package) plus injected init scripts is the add-in
+  mechanism**: `initScript` / `initScriptPath` inject app JS into every document
+  before page scripts run, with the full msgapi surface available — host-native,
+  works in msger and msga, no Chromium required. If something genuinely needs
+  Chrome extension APIs, run it in Chrome.
+- **`-render` off Windows** — ported. msger now captures on Linux/Pi through
+  WebKitGTK's snapshot API (see the render row above), verified on a Pi 5 for both
+  PNG and JPEG output.
 
 Engine currency is *not* a reason: a feature probe on Pi 5 / WebKitGTK 2.50.6
 (2026-07-31) passed 23 of 24 modern checks — `:has()`, CSS nesting, container
@@ -145,11 +150,12 @@ nothing about capability.
 | `tcp.close()` | ✅ | ✅ | ✅ | Close connection |
 | **HTTP Fetch** | | | | **`window.msgapi.http.*`** |
 | `http.fetch()` | ✅ | ✅ | ✅ | Native HTTP — bypasses CORS/mixed-content |
+| **Media capture (mic/camera)** | ❌ | ❌ | ❌ | Not an msgapi API — `getUserMedia` is the engine's. What's missing is the permission grant, which no host makes: wry's WebKitGTK backend has no `permission-request` handling (WebKitGTK denies by default), msger ignores WebView2's `PermissionRequested`, msga lacks `RECORD_AUDIO`. Deferred; direction is an external speech package, not in-page capture — see [msgapi-plan.md](./msgapi-plan.md#not-planned-recorded-so-it-doesnt-get-re-investigated). |
 | **Host Launcher Features** | | | | *Not part of `window.msgapi`; CLI / launcher-side* |
 | Markdown rendering (`.md`/`.markdown` auto-render) | ✅ | ✅ | ❌ | Local `file://` URL → rendered HTML via `msgcommon/markdown` (marked + light/dark CSS). msga could match it in `MainPage.xaml.cs` URL interception — not done yet. |
-| Unpacked Chrome extension loading (`-extension <dir>`) | ✅ | ❌ | ❌ | msgview-only via `session.extensions.loadExtension`. wry has no public extension API; WebKitGTK has none at all. msga would need a separate Android/WebView2 plumbing — not planned. |
+| Unpacked Chrome extension loading (`-extension <dir>`) | ✅ | ❌ | ❌ | msgview-only via `session.extensions.loadExtension`. wry has no public extension API; WebKitGTK has none at all. **Not the add-in mechanism here** — app add-ins are `window.msgapi` + injected init scripts (`initScript` / `initScriptPath`), which work in msger and msga without Chromium. Chrome-extension-specific needs go to Chrome. |
 | Headless mode (`-noshow` without `-save`) | ✅ | ✅ | ❌ | msger creates the WebView invisible (`with_visible(false)`) but still loads HTML and runs scripts — an embedded `<script>` can query the engine and `window.ipc.postMessage` a result back. Combined with `-result <fieldname>` for clean stdout, this lets msger act as a headless CSS/JS resolver (`contrast-color()`, `getComputedStyle()`, `CSS.supports()`). msgview matches with `BrowserWindow({show:false})` + `backgroundThrottling:false`. |
-| Render to bitmap (`-render [file]`, `render` option) | ✅ | ✅ | ❌ | Headless page→image capture: the window is never shown; the page loads, settles (`renderDelay`, default 100ms after `load` + two rAFs), is screenshotted, and the host exits. `-render <file>` writes the image (format from extension: `.png`/`.jpg`/`.bmp`) and the JSON result carries `render: {path, width, height, format}`; bare `-render` (or API `render: true`) returns it as `render: {data (base64), width, height, format}`. Library API: `showMessageBox({render: "shot.png" \| true, renderDelay?, renderFormat?})`; a failed capture rejects (msger) or sets `renderError` in the result. Both hosts fall back to capturing whatever rendered if the page hasn't loaded within the timeout (default 30s in render mode). msger: WebView2 DevTools `Page.captureScreenshot` — **Windows only** for now; msgview: `webContents.capturePage()` — all Electron platforms. Width/height are physical pixels (include DPI scale). |
+| Render to bitmap (`-render [file]`, `render` option) | ✅ | ✅ | ❌ | Headless page→image capture: the window is never shown; the page loads, settles (`renderDelay`, default 100ms after `load` + two rAFs), is screenshotted, and the host exits. `-render <file>` writes the image (format from extension: `.png`/`.jpg`/`.bmp`) and the JSON result carries `render: {path, width, height, format}`; bare `-render` (or API `render: true`) returns it as `render: {data (base64), width, height, format}`. Library API: `showMessageBox({render: "shot.png" \| true, renderDelay?, renderFormat?})`; a failed capture rejects (msger) or sets `renderError` in the result. Both hosts fall back to capturing whatever rendered if the page hasn't loaded within the timeout (default 30s in render mode). msger: WebView2 DevTools `Page.captureScreenshot` on Windows, `webkit_web_view_get_snapshot()` → cairo PNG on Linux/Pi (2026-08-01); msgview: `webContents.capturePage()`. Both msger paths produce the same base64 PNG, so format transcoding and file writing are shared. **Linux caveat:** WebKitGTK cannot snapshot a window that was never mapped, so on Linux the render window IS mapped — made invisible by going fully transparent and skipping the taskbar, since Wayland has no way to park a window off-screen. Width/height are physical pixels (include DPI scale) and cover the WebView viewport, not the window frame. |
 
 > **msger** now injects `window.msgapi` via `msger-api.js` with window control, UDP, and HTTP. File system and shell are not yet implemented in msger or msgview.
 >

package/msgapi-plan.md

@@ -242,6 +242,53 @@ interface MsgAPI {
 
 ---
 
+## Not Planned (recorded so it doesn't get re-investigated)
+
+### Microphone / camera capture — deferred, direction is an external package
+
+**Status 2026-08-01: not in msgapi, and no host grants the permission.** Recorded
+after the wallclock app showed *"Initialization failed: Microphone constraints not
+supported"* under msger on a Pi 5.
+
+**Why it isn't an msgapi API.** The mic is reached with the plain web API
+`navigator.mediaDevices.getUserMedia` — the engine already provides it. msgapi
+exists for what a browser *can't* do. What's actually missing is a **permission
+decision**, and none of the hosts makes one:
+
+| host | today |
+|------|-------|
+| msger / WebKitGTK (Linux, Pi) | wry 0.54's WebKitGTK backend has no permission-request handling at all; WebKitGTK denies an unhandled `permission-request` by default |
+| msger / WebView2 (Windows) | msger never handles `PermissionRequested`; whatever WebView2 defaults to applies |
+| msga (Android) | manifest doesn't declare `RECORD_AUDIO` (the only hits are inside `Mono.Android.dll`); no `onPermissionRequest` handler |
+
+**Unresolved:** whether the Pi failure is the permission gate or the app's
+constraints — WebKitGTK is pickier than Chromium about `echoCancellation`,
+`sampleRate`, etc. One probe settles it: call `getUserMedia({audio: true})` with
+bare constraints via `initScript` and read the error name — `NotAllowedError`
+means permission, `OverconstrainedError` means constraints.
+
+**If it ever becomes a priority**, the work is:
+
+1. **msger (Rust)** — connect WebKitGTK's `permission-request` signal and WebView2's
+   `PermissionRequested` event, granting media for content the caller trusts.
+   Opt-in per the no-regression rule in CLAUDE.md (e.g. `permissions:
+   ["microphone","camera"]` / `-allow mic`), defaulting to today's behavior so no
+   existing caller changes.
+2. **msga** — `RECORD_AUDIO` in the manifest, the Android runtime permission
+   request, and a `WebChromeClient.onPermissionRequest` handler.
+3. **msgapidefs** — a matrix row for which host grants what. No new
+   `window.msgapi` call is needed.
+
+**Direction (Bob, 2026-08-01): don't build this speculatively.** The original
+motivation was speech processing, which never worked well enough in-browser to be
+worth it. The likely future is delegating to an **external speech package** —
+msger/msga hand off to a native or networked processor (reachable today through
+`msgapi.shell` / `msgapi.http` / `msgapi.udp`) rather than capturing audio in the
+page. In-page mic support is only worth building if a concrete use case shows up
+that genuinely needs the audio inside the WebView.
+
+---
+
 ## Implementation Priorities
 
 ### Phase 1: msga msgapi + UDP ✅ Done

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bobfrankston/msgapidefs",
-  "version": "0.1.37",
+  "version": "0.1.39",
   "description": "TypeScript definitions for msgapi JavaScript API (msgview/msger)",
   "type": "module",
   "main": "./msgapidefs.js",