@staff0rd/assist 0.306.0 → 0.307.0

package/README.md

@@ -224,7 +224,8 @@ The first backlog command in a repository that still has a local `.assist/backlo
 - `assist sql mutate "<sql>" [connection]` - Execute a mutating SQL statement and print rows affected (rejects non-mutating statements like pure SELECTs)
 - `assist sql tables [connection]` - List tables in the connected database (via INFORMATION_SCHEMA.TABLES)
 - `assist sql columns <table> [connection]` - List columns for a table (use `schema.table` for non-default schema; via INFORMATION_SCHEMA.COLUMNS)
-- `assist netcap [-p, --port <port>]` - Start a local receiver that captures browser network traffic to a JSONL file (default `~/.assist/netcap/capture.jsonl`), paired with the netcap browser extension. Sends permissive CORS, logs each ping/capture live, and runs until Ctrl-C (default port `8723`). See [netcap browser extension](#netcap-browser-extension)
+- `assist netcap [-p, --port <port>] [-o, --out <dir>] [-f, --filter <pattern>]` - Start a local receiver that captures browser network traffic to a JSONL file (`capture.jsonl` under `--out`, default `~/.assist/netcap`), paired with the netcap browser extension. Sends permissive CORS, logs each ping/capture live, and runs until Ctrl-C, then prints a capture count (default port `8723`). `--filter` bakes a URL substring into the extension so only matching requests forward. See [netcap browser extension](#netcap-browser-extension)
+- `assist netcap extract [file]` - Parse a netcap capture file into structured posts (`text`, `markdown` with mentions linked, `author`, `mentions`, `hashtags`, `links`, `relatedPosts`, and `postedAt` decoded from the activity id) and write them to `posts.json` beside the capture (defaults to `~/.assist/netcap/capture.jsonl`). Reads both the LinkedIn SDUI (rsc-action) responses rendered on first paint and the voyager GraphQL profile-updates responses loaded on scroll, deduped by activity urn keeping the richest copy of each post
 - `assist screenshot <process>` - Capture a screenshot of a running application window (e.g. `assist screenshot notepad`). Output directory is configurable via `screenshot.outputDir` (default `./screenshots`)
 - `assist handover save --summary <s>` - Save a session handover note to the backlog DB (content read from stdin), scoped by the repo's git origin. Appends a new row each call; `--summary` is the one-line description shown when recalling
 - `assist handover list` - List unrecalled handovers for this repo, most recent first, one per line as tab-separated `id`, ISO-8601 created timestamp, and one-line summary. Prints nothing when none are pending
@@ -277,11 +278,13 @@ When `commit.pull` is enabled in config, `assist draft`, `assist bug`, `assist r
 
 ## netcap browser extension
 
-`assist netcap` only runs the receiver; the browser side is a raw Manifest V3 extension (no build step) under `netcap-extension/`. A MAIN-world content script patches `fetch`/`XMLHttpRequest` to capture `{url, method, status, requestBody, responseBody, timestamp}`, relays each entry to the background service worker (`window.postMessage` → `chrome.runtime.sendMessage`), and the background worker POSTs it to the receiver. Forwarding happens in the background context, so the page's CSP (`connect-src`) never blocks it — the limitation that killed the earlier console-paste approach.
+`assist netcap` only runs the receiver; the browser side is a raw Manifest V3 extension (no build step) under `netcap-extension/`. A MAIN-world content script patches `fetch`/`XMLHttpRequest` to capture `{url, method, status, requestBody, responseBody, timestamp}`, relays each entry to the background service worker (`window.postMessage` → `chrome.runtime.sendMessage`), and the background worker POSTs it to the receiver. The XHR patch reads the response across every `responseType` (`json`, `blob`, `arraybuffer`, `document`, not just `text`) — LinkedIn's voyager GraphQL calls use `responseType: "json"`, which exposes no `responseText`, so reading `responseText` alone captured those bodies empty. Forwarding happens in the background context, so the page's CSP (`connect-src`) never blocks it — the limitation that killed the earlier console-paste approach.
 
 1. Run `assist netcap` — it prints the receiver URL, the capture file path, and the absolute path to the extension directory to load. The receiver host/port is baked into the extension's `background.js` at this point. Under WSL (where the browser runs on the Windows host, cannot load from a WSL path, and cannot reach the receiver on `localhost` because WSL2 localhost forwarding is unreliable) it copies the extension to `C:\tools\netcap-extension`, targets the WSL VM's IP, and prints that Windows path instead. Re-run `assist netcap` after a reboot (the WSL IP can change) and reload the extension.
 2. Load the unpacked extension:
    - **Firefox**: open `about:debugging#/runtime/this-firefox` → **Load Temporary Add-on…** → pick `manifest.json` inside the printed extension directory. (Requires Firefox 128+ for MAIN-world content scripts.)
    - **Chrome**: open `chrome://extensions`, enable **Developer mode**, click **Load unpacked**, and select the extension directory.
 3. On load the background worker pings the receiver; `ping from extension` appears in the `assist netcap` log, confirming browser→server connectivity.
-4. Browse a site (e.g. LinkedIn activity); matching requests append to the capture file live and survive page refreshes. The extension targets `http://127.0.0.1:8723`; if you change `--port`, edit `RECEIVER` in `netcap-extension/background.js`.
+4. Browse a site (e.g. LinkedIn activity); matching requests append to the capture file live and survive page refreshes (re-loading the extension appends to the same file). Press Ctrl-C to stop the receiver; it prints how many entries were captured.
+
+`assist netcap` bakes the receiver host/port and the optional `--filter` substring into the extension's `background.js` at startup, so you don't normally edit it by hand. With `--filter <pattern>`, the background worker only forwards requests whose URL contains `<pattern>` (case-sensitive substring); the matching logic mirrors `matchesNetcapFilter` in `src/commands/netcap`. Use `--out <dir>` to write `capture.jsonl` into a different directory.

package/dist/commands/netcap/netcap-extension/background.js

@@ -3,17 +3,25 @@
 // startup it pings the receiver (the connectivity spike) and thereafter
 // forwards every captured entry relayed from the content scripts.
 const RECEIVER = "http://127.0.0.1:8723/";
+const FILTER = "";
 
 function ping() {
 	fetch(`${RECEIVER}ping`, { method: "GET" }).catch(() => {});
 }
 
+// why: duplicates matchesNetcapFilter (src/commands/netcap) — plain JS, no TS import
+function matchesFilter(url) {
+	if (!FILTER) return true;
+	return typeof url === "string" && url.includes(FILTER);
+}
+
 ping();
 chrome.runtime.onInstalled?.addListener(ping);
 chrome.runtime.onStartup?.addListener(ping);
 
 chrome.runtime.onMessage.addListener((message) => {
 	if (!message || message.type !== "netcap-entry") return;
+	if (!matchesFilter(message.entry && message.entry.url)) return;
 	fetch(RECEIVER, {
 		method: "POST",
 		headers: { "Content-Type": "application/json" },

package/dist/commands/netcap/netcap-extension/interceptor.js

@@ -57,6 +57,24 @@
 		};
 	}
 
+	// why: voyager GraphQL uses responseType "json"/"blob" (no responseText), so reading responseText alone captured them empty
+	function xhrResponseBody(xhr) {
+		try {
+			const type = xhr.responseType;
+			if (type === "" || type === "text") return xhr.responseText ?? "";
+			if (type === "json")
+				return xhr.response == null ? "" : JSON.stringify(xhr.response);
+			if (type === "arraybuffer")
+				return new TextDecoder().decode(xhr.response);
+			if (type === "document")
+				return xhr.response?.documentElement?.outerHTML ?? "";
+			if (type === "blob") return xhr.response;
+		} catch {
+			// ignore: response unavailable for this responseType
+		}
+		return "";
+	}
+
 	const OriginalXHR = window.XMLHttpRequest;
 	if (OriginalXHR) {
 		const open = OriginalXHR.prototype.open;
@@ -70,22 +88,24 @@
 			if (meta) {
 				meta.requestBody = bodyToString(body);
 				this.addEventListener("loadend", () => {
-					let responseBody = "";
-					try {
-						if (this.responseType === "" || this.responseType === "text") {
-							responseBody = this.responseText;
-						}
-					} catch {
-						// ignore: responseText unavailable for this responseType
+					const emit = (responseBody) =>
+						post({
+							url: meta.url,
+							method: meta.method,
+							status: this.status,
+							requestBody: meta.requestBody ?? null,
+							responseBody,
+							timestamp: Date.now(),
+						});
+					const value = xhrResponseBody(this);
+					if (value instanceof Blob) {
+						value
+							.text()
+							.then(emit)
+							.catch(() => emit(""));
+					} else {
+						emit(value);
 					}
-					post({
-						url: meta.url,
-						method: meta.method,
-						status: this.status,
-						requestBody: meta.requestBody ?? null,
-						responseBody,
-						timestamp: Date.now(),
-					});
 				});
 			}
 			return send.call(this, body);