@rmrdeveloper/sideroom-pi 8.12.0 → 8.14.0

package/README.md

@@ -141,6 +141,22 @@ session. The compact view shows at most five recent paths below an active
 board; the extended `F8` view scrolls through everything. Paths outside the
 project are labelled `external`. A new session starts clean.
 
+### Browser context — point to the element
+
+Press `F7` to authorize the bundled browser companion once per Pi installation.
+It reconnects automatically and remembers Windows and WSL independently; choose
+a destination if several Pi sessions are open. Press `Alt+Shift+S` to select
+several elements without opening the popup; Escape or the shortcut exits.
+Review their local source candidates in Pi and send your
+instruction. The click prepares context; it never starts the agent. No project
+configuration or framework plugin is needed. Source ranges are real, but the
+DOM-to-source relationship is inferred and stays labelled as a candidate.
+Download [the companion ZIP](https://github.com/RMRdeveloper/sideroom-pi/releases/latest/download/sideroom-browser-extension.zip)
+from GitHub Releases, extract it, and load its folder in your browser's developer mode.
+No extension store is required; installation and updates are manual. The same
+companion ships in the npm package. See [browser context](docs/extensions/visual-context.md)
+for installation, Windows/WSL pairing and supported source searches.
+
 ### Grill — settle the words first
 
 An on-demand skill for fuzzy plans. It interviews you in `sideroom_ask`

package/assets/visual-context/browser/INSTALL.md

@@ -0,0 +1,69 @@
+# Sideroom Visual Context
+
+## English
+
+This is a manually installed companion for Chromium-based browsers 116 or
+later, such as Chrome, Edge or Brave. It is not a store installation. Firefox is
+not supported.
+
+1. Extract the ZIP. Keep the `sideroom-browser-extension` folder in a permanent
+   location; the browser loads the extension from this directory.
+2. Open `chrome://extensions` (or `edge://extensions`), enable developer mode,
+   choose **Load unpacked**, and select the folder containing `manifest.json`.
+3. Install Sideroom Pi with `pi install npm:@rmrdeveloper/sideroom-pi`. Start Pi
+   in your project, press `F7`, and choose **Pair browser**.
+4. Open the companion, paste the private pairing address from Pi, and choose
+   **Connect to Pi**. Keep that address private. Choose a destination if several
+   Pi sessions are open. Repeat pairing for a separate Windows or WSL installation.
+5. On a normal web page, press `Alt+Shift+S` to start selecting. Click several
+   elements. Press the shortcut again or Escape to return to normal browsing.
+   Review pending elements with `F7` in Pi, then send your instruction.
+
+Change or assign the shortcut in `chrome://extensions/shortcuts` or
+`edge://extensions/shortcuts`. Browser and operating-system shortcuts can take
+priority. Internal browser pages and other protected pages cannot be inspected.
+Managed browsers can prohibit developer mode.
+
+Updates are manual. Download the ZIP from the new Sideroom GitHub Release,
+replace the files in the same folder, and reload the extension from the browser's
+extensions page. Do not remove and reinstall it unless you want to pair again.
+The extension connects only to authorized local Pi installations; it captures
+DOM descriptions, not form values or editable text. Source locations are inferred
+candidates, never proven mappings.
+
+## Español
+
+Esta extensión se instala manualmente en navegadores basados en Chromium 116 o
+posterior, como Chrome, Edge o Brave. No se instala desde una tienda. Firefox no
+está soportado.
+
+1. Descomprime el ZIP. Guarda la carpeta `sideroom-browser-extension` en una
+   ubicación permanente; el navegador carga la extensión desde esa carpeta.
+2. Abre `chrome://extensions` (o `edge://extensions`), activa el modo
+   desarrollador, elige **Cargar desempaquetada** y selecciona la carpeta que
+   contiene `manifest.json`.
+3. Instala Sideroom Pi con `pi install npm:@rmrdeveloper/sideroom-pi`. Inicia Pi
+   en tu proyecto, pulsa `F7` y elige **Pair browser**.
+4. Abre la extensión, pega la dirección privada de emparejamiento de Pi y elige
+   **Connect to Pi**. No compartas esa dirección. Elige un destino si hay varias
+   sesiones abiertas. Repite el emparejamiento para otra instalación de Windows o WSL.
+5. En una página web normal, pulsa `Alt+Shift+S` para seleccionar. Haz clic en
+   varios elementos. Pulsa de nuevo el atajo o Escape para navegar normalmente.
+   Revisa los elementos con `F7` en Pi y envía tu instrucción.
+
+Cambia o asigna el atajo en `chrome://extensions/shortcuts` o
+`edge://extensions/shortcuts`. Los atajos del navegador o del sistema pueden
+tener prioridad. No puedes seleccionar elementos de páginas internas del
+navegador ni de otras páginas protegidas. Un navegador administrado puede
+prohibir el modo desarrollador.
+
+Las actualizaciones son manuales. Descarga el ZIP de la nueva versión de Sideroom
+en GitHub Releases, reemplaza los archivos en la misma carpeta y recarga la
+extensión desde la página de extensiones del navegador. No la elimines y
+reinstales salvo que quieras volver a emparejarla.
+La extensión solo se conecta a instalaciones locales de Pi que hayas autorizado;
+recoge descripciones de los elementos, no valores de formularios ni texto editable.
+Los archivos de origen son candidatos inferidos, no correspondencias demostradas.
+
+Full guide / Guía completa:
+https://github.com/RMRdeveloper/sideroom-pi/blob/main/docs/extensions/visual-context.md

package/assets/visual-context/browser/background.js

@@ -0,0 +1,578 @@
+const PROTOCOL_VERSION = 2;
+const BRIDGE_PATH = '/sideroom-visual-context';
+const HEARTBEAT_MS = 20_000;
+const RESPONSE_TIMEOUT_MS = 10_000;
+const RETRY_MS = 2_000;
+const RETRY_ALARM = 'sideroom-reconnect';
+const MAX_INSTALLATIONS = 4;
+const INSPECTOR_COMMAND = 'toggle-inspector';
+const IDLE_TITLE = 'Select an element for Pi';
+const ACTIVE_TITLE =
+  'Selecting elements for Pi. Press Escape or the selection shortcut to stop.';
+let inspectorCommands = Promise.resolve();
+const MESSAGE = {
+  authenticate: 'authenticate',
+  ready: 'ready',
+  sessions: 'sessions',
+  selection: 'selection',
+  received: 'received',
+  error: 'error',
+  ping: 'ping',
+};
+const WAITING_NOTICE = 'Waiting for Pi. Reconnection is automatic.';
+const PAUSED_NOTICE = 'Disconnected. Your authorizations are remembered.';
+const activeTabs = new Map();
+const receipts = new Map();
+const connections = new Map();
+let paused = false;
+let sessions = [];
+let selected;
+let preferredId;
+let automaticSelection = false;
+let lastAddress = '';
+let lastNotice = 'Pair once with each Pi installation to select elements.';
+const isConnected = () =>
+  [...connections.values()].some((entry) => entry.connected);
+const availableSessions = () =>
+  [...connections.values()].flatMap((entry) => entry.sessions);
+
+function parseUrl(address, failureMessage) {
+  try {
+    return new URL(address);
+  } catch (cause) {
+    throw new Error(failureMessage, { cause });
+  }
+}
+
+function parsePairingAddress(address) {
+  const uri = parseUrl(
+    address.trim(),
+    'Paste the local pairing address shown by Pi (F7).',
+  );
+  if (
+    uri.protocol !== 'ws:' ||
+    !['127.0.0.1', 'localhost'].includes(uri.hostname) ||
+    !uri.port ||
+    uri.pathname !== BRIDGE_PATH ||
+    uri.search ||
+    uri.username ||
+    uri.password ||
+    !/^#[a-f0-9]{64}$/u.test(uri.hash)
+  ) {
+    throw new Error('Paste the local pairing address shown by Pi (F7).');
+  }
+  const token = uri.hash.slice(1);
+  uri.hash = '';
+  return { endpoint: uri.href, token };
+}
+
+async function sendToTab(tabId, message) {
+  try {
+    await chrome.tabs.sendMessage(tabId, message);
+  } catch (cause) {
+    lastNotice = `Inspector stopped: ${cause.message}`;
+    activeTabs.delete(tabId);
+  }
+}
+
+async function showInspectorIndicator(tabId, { text, title }) {
+  await chrome.action.setBadgeBackgroundColor({ tabId, color: '#ffb340' });
+  await chrome.action.setBadgeText({ tabId, text });
+  await chrome.action.setTitle({ tabId, title });
+}
+
+async function stopInspector(tabId) {
+  activeTabs.delete(tabId);
+  await sendToTab(tabId, { action: 'stop' });
+  await showInspectorIndicator(tabId, { text: '', title: IDLE_TITLE });
+}
+
+async function stopInspectors() {
+  const tabIds = [...activeTabs.keys()];
+  activeTabs.clear();
+  receipts.clear();
+  for (const tabId of tabIds) {
+    await stopInspector(tabId);
+  }
+}
+
+async function startInspector(tab) {
+  if (!selected || !isConnected()) {
+    throw new Error('Choose an available Pi session first.');
+  }
+  if (!tab?.id || !/^https?:\/\//u.test(tab.url ?? '')) {
+    throw new Error('Open the web page you want to inspect.');
+  }
+  const target = selected;
+  await chrome.scripting.executeScript({
+    target: { tabId: tab.id },
+    files: ['content.js'],
+  });
+  const commands = await chrome.commands.getAll();
+  const shortcut = commands.find(
+    (command) => command.name === INSPECTOR_COMMAND,
+  )?.shortcut;
+  if (selected !== target) {
+    throw new Error('The destination changed. Start the inspector again.');
+  }
+  activeTabs.set(tab.id, target);
+  try {
+    await chrome.tabs.sendMessage(tab.id, { action: 'start', shortcut });
+    await showInspectorIndicator(tab.id, { text: 'ON', title: ACTIVE_TITLE });
+  } catch (cause) {
+    await stopInspector(tab.id);
+    throw cause;
+  }
+}
+
+async function toggleInspector(commandTab) {
+  let tab = commandTab;
+  try {
+    await initialized;
+    if (!tab) {
+      [tab] = await chrome.tabs.query({ active: true, currentWindow: true });
+    }
+    if (tab?.id && activeTabs.has(tab.id)) {
+      await stopInspector(tab.id);
+      return;
+    }
+    await startInspector(tab);
+  } catch (cause) {
+    lastNotice = `${cause.message} Open the Sideroom extension for details.`;
+    await showInspectorIndicator(tab?.id, { text: '!', title: lastNotice });
+  }
+}
+
+function updateSessions(available) {
+  const previous = selected;
+  sessions = available;
+  selected = sessions.find(
+    (session) => session.id === (preferredId ?? previous?.id),
+  );
+  if (automaticSelection && sessions.length > 1) {
+    selected = undefined;
+    automaticSelection = false;
+  }
+  if (!selected && !preferredId && sessions.length === 1) {
+    selected = sessions[0];
+    automaticSelection = true;
+  }
+  if (
+    previous?.id !== selected?.id ||
+    previous?.revision !== selected?.revision
+  ) {
+    void stopInspectors().catch((cause) => {
+      console.error('Sideroom could not stop its inspectors:', cause);
+    });
+  }
+  lastNotice = selected
+    ? `Connected to ${selected.name}.`
+    : sessions.length
+      ? 'Choose which Pi session receives your selections.'
+      : WAITING_NOTICE;
+}
+
+function closeConnection(entry) {
+  const previous = entry.socket;
+  entry.socket = undefined;
+  entry.connected = false;
+  entry.sessions = [];
+  clearInterval(entry.heartbeat);
+  clearTimeout(entry.retry);
+  entry.cancel?.(new Error('The connection was closed.'));
+  entry.cancel = undefined;
+  previous?.close();
+}
+
+async function disconnect() {
+  for (const entry of connections.values()) {
+    closeConnection(entry);
+  }
+  sessions = [];
+  selected = undefined;
+  preferredId = undefined;
+  await stopInspectors();
+}
+
+function scheduleReconnect(entry) {
+  if (paused || entry.blocked || connections.get(entry.endpoint) !== entry) {
+    return;
+  }
+  clearTimeout(entry.retry);
+  entry.retry = setTimeout(() => {
+    void reconnect(entry);
+  }, RETRY_MS);
+  // Alarms wake the worker after it has been suspended while Pi is closed.
+  void chrome.alarms.create(RETRY_ALARM, { periodInMinutes: 1 });
+}
+
+async function openConnection(entry) {
+  closeConnection(entry);
+  const socket = new WebSocket(entry.endpoint);
+  entry.socket = socket;
+  await new Promise((resolve, reject) => {
+    entry.cancel = reject;
+    const timeout = setTimeout(() => {
+      socket.close();
+      reject(new Error(WAITING_NOTICE));
+    }, RESPONSE_TIMEOUT_MS);
+    socket.onopen = () =>
+      socket.send(
+        JSON.stringify({
+          type: MESSAGE.authenticate,
+          role: 'browser',
+          version: PROTOCOL_VERSION,
+          token: entry.token,
+        }),
+      );
+    socket.onmessage = async (event) => {
+      if (entry.socket !== socket) {
+        return;
+      }
+      try {
+        const message = JSON.parse(event.data);
+        if (message.type === MESSAGE.ready) {
+          if (message.version !== PROTOCOL_VERSION) {
+            throw new Error('Reload the updated Sideroom browser extension.');
+          }
+          clearTimeout(timeout);
+          entry.connected = true;
+          entry.cancel = undefined;
+          entry.sessions = message.sessions;
+          updateSessions(availableSessions());
+          if (
+            [...connections.values()].every(
+              (connection) => connection.connected,
+            )
+          ) {
+            await chrome.alarms.clear(RETRY_ALARM);
+          }
+          entry.heartbeat = setInterval(() => {
+            if (socket.readyState === WebSocket.OPEN) {
+              socket.send(JSON.stringify({ type: MESSAGE.ping }));
+            }
+          }, HEARTBEAT_MS);
+          resolve();
+        }
+        if (message.type === MESSAGE.sessions) {
+          entry.sessions = message.sessions;
+          updateSessions(availableSessions());
+        }
+        if (message.type === MESSAGE.received) {
+          lastNotice = `${String(message.candidates)} source candidates. Review in Pi (F7).`;
+          const receipt = receipts.get(message.id);
+          receipts.delete(message.id);
+          if (receipt?.endpoint === entry.endpoint) {
+            await sendToTab(receipt.tabId, {
+              action: 'receipt',
+              message: lastNotice,
+            });
+          }
+        }
+        if (message.type === MESSAGE.error) {
+          lastNotice = message.message;
+          for (const [selectionId, receipt] of receipts) {
+            if (receipt.endpoint === entry.endpoint) {
+              receipts.delete(selectionId);
+              await sendToTab(receipt.tabId, {
+                action: 'receipt',
+                message: lastNotice,
+              });
+            }
+          }
+        }
+      } catch (cause) {
+        lastNotice = `Invalid bridge response: ${cause.message}`;
+        clearTimeout(timeout);
+        socket.close();
+        reject(new Error(lastNotice));
+      }
+    };
+    socket.onerror = () => {
+      clearTimeout(timeout);
+      reject(
+        new Error(
+          'Waiting for Pi. Windows must be able to reach WSL localhost when using Pi in WSL.',
+        ),
+      );
+    };
+    socket.onclose = (event) => {
+      clearTimeout(timeout);
+      if (entry.socket !== socket) {
+        return;
+      }
+      entry.socket = undefined;
+      entry.connected = false;
+      entry.sessions = [];
+      entry.blocked = event?.code === 1008;
+      clearInterval(entry.heartbeat);
+      for (const [selectionId, receipt] of receipts) {
+        if (receipt.endpoint === entry.endpoint) {
+          receipts.delete(selectionId);
+        }
+      }
+      updateSessions(availableSessions());
+      reject(
+        new Error(
+          entry.blocked
+            ? 'Authorization rejected. Pair that installation again from F7.'
+            : WAITING_NOTICE,
+        ),
+      );
+      scheduleReconnect(entry);
+    };
+  });
+}
+
+async function reconnect(entry) {
+  if (paused || entry.blocked || entry.connected) {
+    return;
+  }
+  if (!entry.attempt) {
+    entry.attempt = openConnection(entry)
+      .catch((cause) => {
+        if (!selected) {
+          lastNotice = cause.message;
+        }
+        scheduleReconnect(entry);
+      })
+      .finally(() => {
+        entry.attempt = undefined;
+      });
+  }
+  await entry.attempt;
+}
+
+function remember(address) {
+  const pairing = parsePairingAddress(address);
+  const existing = connections.get(pairing.endpoint);
+  if (existing && existing.token === pairing.token) {
+    existing.blocked = false;
+    return existing;
+  }
+  if (!existing && connections.size >= MAX_INSTALLATIONS) {
+    throw new Error(
+      'Four installations are already authorized. Forget them before adding another.',
+    );
+  }
+  if (existing) {
+    closeConnection(existing);
+  }
+  const entry = {
+    ...pairing,
+    address: address.trim(),
+    connected: false,
+    sessions: [],
+    blocked: false,
+  };
+  connections.set(entry.endpoint, entry);
+  return entry;
+}
+
+const initialized = chrome.storage.local
+  .setAccessLevel({ accessLevel: 'TRUSTED_CONTEXTS' })
+  .then(async () => {
+    const saved = await chrome.storage.local.get([
+      'pairingAddresses',
+      'pairingAddress',
+      'paused',
+    ]);
+    const addresses =
+      saved.pairingAddresses ??
+      (saved.pairingAddress ? [saved.pairingAddress] : []);
+    paused = saved.paused === true;
+    for (const address of addresses) {
+      const entry = remember(address);
+      lastAddress = address;
+      void reconnect(entry);
+    }
+    if (paused) {
+      lastNotice = PAUSED_NOTICE;
+    }
+  });
+
+async function handleMessage(message, sender) {
+  if (sender.id !== chrome.runtime.id) {
+    throw new Error('Unknown inspector sender.');
+  }
+  await initialized;
+  const fromPopup = sender.url === chrome.runtime.getURL('popup.html');
+  if (!fromPopup) {
+    const target = activeTabs.get(sender.tab?.id);
+    if (message.action === 'stopped') {
+      if (sender.tab?.id) {
+        activeTabs.delete(sender.tab.id);
+        await showInspectorIndicator(sender.tab.id, {
+          text: '',
+          title: IDLE_TITLE,
+        });
+      }
+      return { ok: true };
+    }
+    if (message.action !== 'selection' || !target || !isConnected()) {
+      throw new Error('Connect to Pi and start the inspector first.');
+    }
+    if (target.id !== selected?.id || target.revision !== selected?.revision) {
+      throw new Error('The destination changed. Start the inspector again.');
+    }
+    const entry = [...connections.values()].find(
+      (connection) =>
+        connection.connected &&
+        connection.sessions.some(
+          (session) =>
+            session.id === target.id && session.revision === target.revision,
+        ),
+    );
+    if (!entry || entry.socket?.readyState !== WebSocket.OPEN) {
+      throw new Error('The destination is no longer available.');
+    }
+    if (receipts.size >= 8) {
+      throw new Error('Pi is still processing selections. Try again shortly.');
+    }
+    const page = parseUrl(
+      sender.tab.url,
+      'The selected page has an invalid address.',
+    );
+    const selection = {
+      ...message.selection,
+      page: `${page.origin}${page.pathname}`,
+    };
+    receipts.set(selection.id, {
+      tabId: sender.tab.id,
+      endpoint: entry.endpoint,
+    });
+    entry.socket.send(
+      JSON.stringify({ type: MESSAGE.selection, selection, target }),
+    );
+    return { ok: true };
+  }
+  if (message.action === 'status') {
+    for (const entry of connections.values()) {
+      void reconnect(entry);
+    }
+    return {
+      ok: true,
+      connected: isConnected(),
+      notice: lastNotice,
+      address: lastAddress,
+      sessions,
+      selectedId: selected?.id ?? '',
+      paused,
+      installations: connections.size,
+    };
+  }
+  if (message.action === 'connect') {
+    const entry = remember(message.address);
+    lastAddress = entry.address;
+    paused = false;
+    await chrome.storage.local.set({
+      pairingAddresses: [...connections.values()].map(
+        (connection) => connection.address,
+      ),
+      paused,
+    });
+    await chrome.storage.local.remove('pairingAddress');
+    await reconnect(entry);
+    for (const connection of connections.values()) {
+      if (connection !== entry) {
+        void reconnect(connection);
+      }
+    }
+    return {
+      ok: true,
+      connected: isConnected(),
+      notice: lastNotice,
+      sessions,
+      selectedId: selected?.id ?? '',
+      address: lastAddress,
+    };
+  }
+  if (message.action === 'disconnect' || message.action === 'forget') {
+    paused = true;
+    await disconnect();
+    await chrome.alarms.clear(RETRY_ALARM);
+    if (message.action === 'forget') {
+      connections.clear();
+      lastAddress = '';
+      await chrome.storage.local.remove(['pairingAddress', 'pairingAddresses']);
+    }
+    await chrome.storage.local.set({ paused });
+    lastNotice = lastAddress
+      ? PAUSED_NOTICE
+      : 'Authorizations forgotten. Pair again from F7.';
+    return {
+      ok: true,
+      connected: false,
+      notice: lastNotice,
+      address: lastAddress,
+      sessions: [],
+      selectedId: '',
+    };
+  }
+  if (message.action === 'choose') {
+    const destination = sessions.find((session) => session.id === message.id);
+    if (!destination) {
+      throw new Error('That Pi session is no longer available.');
+    }
+    await stopInspectors();
+    selected = destination;
+    preferredId = destination.id;
+    automaticSelection = false;
+    lastNotice = `Connected to ${selected.name}.`;
+    return {
+      ok: true,
+      connected: isConnected(),
+      notice: lastNotice,
+      sessions,
+      selectedId: selected.id,
+    };
+  }
+  if (message.action === 'inspect') {
+    const [tab] = await chrome.tabs.query({
+      active: true,
+      currentWindow: true,
+    });
+    await startInspector(tab);
+    return { ok: true };
+  }
+  throw new Error('Unknown inspector action.');
+}
+
+chrome.runtime.onMessage.addListener((message, sender, respond) => {
+  handleMessage(message, sender).then(respond, (cause) => {
+    lastNotice = cause.message;
+    respond({ ok: false, message: lastNotice });
+  });
+  return true;
+});
+chrome.commands.onCommand.addListener((command, tab) => {
+  if (command !== INSPECTOR_COMMAND) {
+    return;
+  }
+  inspectorCommands = inspectorCommands
+    .then(() => toggleInspector(tab))
+    .catch((cause) => {
+      console.error('Sideroom could not update its inspector:', cause);
+    });
+  return inspectorCommands;
+});
+chrome.tabs.onRemoved.addListener((tabId) => activeTabs.delete(tabId));
+chrome.tabs.onUpdated.addListener((tabId, change) => {
+  if (change.status === 'loading' && activeTabs.has(tabId)) {
+    void stopInspector(tabId).catch((cause) => {
+      console.error(
+        'Sideroom could not clear its inspector after navigation:',
+        cause,
+      );
+    });
+  }
+});
+chrome.alarms.onAlarm.addListener((alarm) => {
+  if (alarm.name === RETRY_ALARM) {
+    void initialized
+      .then(() => Promise.all([...connections.values()].map(reconnect)))
+      .catch((cause) => {
+        lastNotice = cause.message;
+      });
+  }
+});