@staff0rd/assist 0.305.4 → 0.307.0

package/README.md

@@ -224,6 +224,8 @@ The first backlog command in a repository that still has a local `.assist/backlo
 - `assist sql mutate "<sql>" [connection]` - Execute a mutating SQL statement and print rows affected (rejects non-mutating statements like pure SELECTs)
 - `assist sql tables [connection]` - List tables in the connected database (via INFORMATION_SCHEMA.TABLES)
 - `assist sql columns <table> [connection]` - List columns for a table (use `schema.table` for non-default schema; via INFORMATION_SCHEMA.COLUMNS)
+- `assist netcap [-p, --port <port>] [-o, --out <dir>] [-f, --filter <pattern>]` - Start a local receiver that captures browser network traffic to a JSONL file (`capture.jsonl` under `--out`, default `~/.assist/netcap`), paired with the netcap browser extension. Sends permissive CORS, logs each ping/capture live, and runs until Ctrl-C, then prints a capture count (default port `8723`). `--filter` bakes a URL substring into the extension so only matching requests forward. See [netcap browser extension](#netcap-browser-extension)
+- `assist netcap extract [file]` - Parse a netcap capture file into structured posts (`text`, `markdown` with mentions linked, `author`, `mentions`, `hashtags`, `links`, `relatedPosts`, and `postedAt` decoded from the activity id) and write them to `posts.json` beside the capture (defaults to `~/.assist/netcap/capture.jsonl`). Reads both the LinkedIn SDUI (rsc-action) responses rendered on first paint and the voyager GraphQL profile-updates responses loaded on scroll, deduped by activity urn keeping the richest copy of each post
 - `assist screenshot <process>` - Capture a screenshot of a running application window (e.g. `assist screenshot notepad`). Output directory is configurable via `screenshot.outputDir` (default `./screenshots`)
 - `assist handover save --summary <s>` - Save a session handover note to the backlog DB (content read from stdin), scoped by the repo's git origin. Appends a new row each call; `--summary` is the one-line description shown when recalling
 - `assist handover list` - List unrecalled handovers for this repo, most recent first, one per line as tab-separated `id`, ISO-8601 created timestamp, and one-line summary. Prints nothing when none are pending
@@ -273,3 +275,16 @@ When iterating on assist itself: web server changes only need the `assist sessio
 
 When `commit.pull` is enabled in config, `assist draft`, `assist bug`, `assist refine`, `assist next`, and `assist backlog run` run `git pull --ff-only` before doing anything else; if the pull fails the command aborts. `assist next` pulls once per invocation, not per item in its loop.
 
+
+## netcap browser extension
+
+`assist netcap` only runs the receiver; the browser side is a raw Manifest V3 extension (no build step) under `netcap-extension/`. A MAIN-world content script patches `fetch`/`XMLHttpRequest` to capture `{url, method, status, requestBody, responseBody, timestamp}`, relays each entry to the background service worker (`window.postMessage` → `chrome.runtime.sendMessage`), and the background worker POSTs it to the receiver. The XHR patch reads the response across every `responseType` (`json`, `blob`, `arraybuffer`, `document`, not just `text`) — LinkedIn's voyager GraphQL calls use `responseType: "json"`, which exposes no `responseText`, so reading `responseText` alone captured those bodies empty. Forwarding happens in the background context, so the page's CSP (`connect-src`) never blocks it — the limitation that killed the earlier console-paste approach.
+
+1. Run `assist netcap` — it prints the receiver URL, the capture file path, and the absolute path to the extension directory to load. The receiver host/port is baked into the extension's `background.js` at this point. Under WSL (where the browser runs on the Windows host, cannot load from a WSL path, and cannot reach the receiver on `localhost` because WSL2 localhost forwarding is unreliable) it copies the extension to `C:\tools\netcap-extension`, targets the WSL VM's IP, and prints that Windows path instead. Re-run `assist netcap` after a reboot (the WSL IP can change) and reload the extension.
+2. Load the unpacked extension:
+   - **Firefox**: open `about:debugging#/runtime/this-firefox` → **Load Temporary Add-on…** → pick `manifest.json` inside the printed extension directory. (Requires Firefox 128+ for MAIN-world content scripts.)
+   - **Chrome**: open `chrome://extensions`, enable **Developer mode**, click **Load unpacked**, and select the extension directory.
+3. On load the background worker pings the receiver; `ping from extension` appears in the `assist netcap` log, confirming browser→server connectivity.
+4. Browse a site (e.g. LinkedIn activity); matching requests append to the capture file live and survive page refreshes (re-loading the extension appends to the same file). Press Ctrl-C to stop the receiver; it prints how many entries were captured.
+
+`assist netcap` bakes the receiver host/port and the optional `--filter` substring into the extension's `background.js` at startup, so you don't normally edit it by hand. With `--filter <pattern>`, the background worker only forwards requests whose URL contains `<pattern>` (case-sensitive substring); the matching logic mirrors `matchesNetcapFilter` in `src/commands/netcap`. Use `--out <dir>` to write `capture.jsonl` into a different directory.

package/dist/commands/netcap/netcap-extension/background.js

@@ -0,0 +1,30 @@
+// Background service worker: the only context that talks to the receiver. It
+// runs outside any page, so the page's CSP does not restrict its fetches. On
+// startup it pings the receiver (the connectivity spike) and thereafter
+// forwards every captured entry relayed from the content scripts.
+const RECEIVER = "http://127.0.0.1:8723/";
+const FILTER = "";
+
+function ping() {
+	fetch(`${RECEIVER}ping`, { method: "GET" }).catch(() => {});
+}
+
+// why: duplicates matchesNetcapFilter (src/commands/netcap) — plain JS, no TS import
+function matchesFilter(url) {
+	if (!FILTER) return true;
+	return typeof url === "string" && url.includes(FILTER);
+}
+
+ping();
+chrome.runtime.onInstalled?.addListener(ping);
+chrome.runtime.onStartup?.addListener(ping);
+
+chrome.runtime.onMessage.addListener((message) => {
+	if (!message || message.type !== "netcap-entry") return;
+	if (!matchesFilter(message.entry && message.entry.url)) return;
+	fetch(RECEIVER, {
+		method: "POST",
+		headers: { "Content-Type": "application/json" },
+		body: JSON.stringify(message.entry),
+	}).catch(() => {});
+});

package/dist/commands/netcap/netcap-extension/interceptor.js

@@ -0,0 +1,114 @@
+// MAIN-world content script: runs in the page's own JS context so it can patch
+// the page's fetch/XMLHttpRequest before page code uses them. Captured entries
+// are handed to the isolated-world relay via window.postMessage; this script
+// never talks to the receiver itself (the background worker does, outside CSP).
+(() => {
+	const SOURCE = "assist-netcap";
+
+	function post(entry) {
+		try {
+			window.postMessage({ source: SOURCE, entry }, "*");
+		} catch {
+			// ignore: structured-clone failures on exotic bodies
+		}
+	}
+
+	function bodyToString(body) {
+		return typeof body === "string" ? body : null;
+	}
+
+	const originalFetch = window.fetch;
+	if (typeof originalFetch === "function") {
+		window.fetch = function (...args) {
+			const input = args[0];
+			const init = args[1] || {};
+			const url =
+				typeof input === "string" || input instanceof URL
+					? String(input)
+					: input && input.url
+						? input.url
+						: "";
+			const method = (
+				init.method ||
+				(input && input.method) ||
+				"GET"
+			).toUpperCase();
+			const requestBody = bodyToString(init.body);
+			const result = originalFetch.apply(this, args);
+			result
+				.then((response) => {
+					response
+						.clone()
+						.text()
+						.then((responseBody) => {
+							post({
+								url,
+								method,
+								status: response.status,
+								requestBody,
+								responseBody,
+								timestamp: Date.now(),
+							});
+						})
+						.catch(() => {});
+				})
+				.catch(() => {});
+			return result;
+		};
+	}
+
+	// why: voyager GraphQL uses responseType "json"/"blob" (no responseText), so reading responseText alone captured them empty
+	function xhrResponseBody(xhr) {
+		try {
+			const type = xhr.responseType;
+			if (type === "" || type === "text") return xhr.responseText ?? "";
+			if (type === "json")
+				return xhr.response == null ? "" : JSON.stringify(xhr.response);
+			if (type === "arraybuffer")
+				return new TextDecoder().decode(xhr.response);
+			if (type === "document")
+				return xhr.response?.documentElement?.outerHTML ?? "";
+			if (type === "blob") return xhr.response;
+		} catch {
+			// ignore: response unavailable for this responseType
+		}
+		return "";
+	}
+
+	const OriginalXHR = window.XMLHttpRequest;
+	if (OriginalXHR) {
+		const open = OriginalXHR.prototype.open;
+		const send = OriginalXHR.prototype.send;
+		OriginalXHR.prototype.open = function (method, url, ...rest) {
+			this.__netcap = { method: String(method).toUpperCase(), url: String(url) };
+			return open.call(this, method, url, ...rest);
+		};
+		OriginalXHR.prototype.send = function (body) {
+			const meta = this.__netcap;
+			if (meta) {
+				meta.requestBody = bodyToString(body);
+				this.addEventListener("loadend", () => {
+					const emit = (responseBody) =>
+						post({
+							url: meta.url,
+							method: meta.method,
+							status: this.status,
+							requestBody: meta.requestBody ?? null,
+							responseBody,
+							timestamp: Date.now(),
+						});
+					const value = xhrResponseBody(this);
+					if (value instanceof Blob) {
+						value
+							.text()
+							.then(emit)
+							.catch(() => emit(""));
+					} else {
+						emit(value);
+					}
+				});
+			}
+			return send.call(this, body);
+		};
+	}
+})();

package/dist/commands/netcap/netcap-extension/manifest.json

@@ -0,0 +1,25 @@
+{
+	"manifest_version": 3,
+	"name": "assist netcap",
+	"version": "1.0.0",
+	"description": "Captures fetch/XHR traffic and forwards it to the local assist netcap receiver.",
+	"host_permissions": ["http://*/*"],
+	"background": {
+		"service_worker": "background.js",
+		"scripts": ["background.js"]
+	},
+	"content_scripts": [
+		{
+			"matches": ["<all_urls>"],
+			"js": ["interceptor.js"],
+			"run_at": "document_start",
+			"world": "MAIN"
+		},
+		{
+			"matches": ["<all_urls>"],
+			"js": ["relay.js"],
+			"run_at": "document_start",
+			"world": "ISOLATED"
+		}
+	]
+}

package/dist/commands/netcap/netcap-extension/relay.js

@@ -0,0 +1,14 @@
+// Isolated-world content script: bridges the MAIN-world interceptor's
+// window.postMessage events to the background service worker via runtime
+// messaging. The background worker, not the page, posts to the receiver, so the
+// page's CSP (connect-src) never applies.
+window.addEventListener("message", (event) => {
+	if (event.source !== window) return;
+	const data = event.data;
+	if (!data || data.source !== "assist-netcap" || !data.entry) return;
+	try {
+		chrome.runtime.sendMessage({ type: "netcap-entry", entry: data.entry });
+	} catch {
+		// ignore: extension context invalidated (e.g. after reload)
+	}
+});