@replayio/self-healing-capture 0.1.0 → 0.1.3

package/README.md

@@ -1,73 +1,32 @@
 # @replayio/self-healing-capture
 
-Browser capture for FullStory sessions. This package owns the producers for network exchanges,
-clicks, input/change values, paste text, keyboard events, user identity, activity counts and page/session
-context. Install the package rather than copying a recorder into your app.
+Temporary browser capture support for Self Healing while Subtext gains the accessors needed for
+network exchanges, interactions, identity, metrics, and session context.
 
-```sh
-npm install @replayio/self-healing-capture @fullstory/browser
-```
-
-Call once in your production browser entry point, before rendering the application:
+## Updating an existing installation
 
-```ts
-import { initCapture } from '@replayio/self-healing-capture'
+**Agents: read [UPGRADING.md](./UPGRADING.md) before updating this dependency.**
+It ships in the npm package at
+`node_modules/@replayio/self-healing-capture/UPGRADING.md` and covers changes to
+application-owned initialization and forwarding code, not just the lockfile.
+No original installer conversation is needed. In particular, old installations
+may contain a 256 KiB forwarding check that a dependency update cannot remove.
 
-export const capture = initCapture({
-  orgId: '<your FullStory organization ID>',
-  endpoint: '/api/self-healing/session', // same-origin server route
-  onError: error => console.error('Session capture failed', error),
-})
-
-// When the application authenticates a user:
-capture.identify({ id: user.id, name: user.name, email: user.email })
-```
+**Installation and integration instructions live in the
+[Self Healing setup skill](https://self-healing.replay.io/api/v1/skills/setup-self-healing/SKILL.md).**
+Follow that skill for account provisioning, browser initialization, server forwarding, automatic reviews,
+and verification. This package is an implementation detail of that setup, not a separate installer.
 
-The package initializes FullStory itself. Remove the previous recorder/FullStory initialization
-when migrating; do not install a second fetch wrapper. Do not call `initCapture` during SSR.
-Repeated calls with the same org/endpoint return the existing controller. It captures fetch traffic
-(browser-visible headers and bodies), not XMLHttpRequest or WebSockets. It preserves the existing
-QA capture fields without introducing a redaction policy.
-
-## Server route
-
-The browser has no provider or account credentials. Its same-origin POST endpoint uses the app's
-existing access controls and forwards the JSON body to Self Healing's
-`POST /api/v1/connection/sessions`, adding `Authorization: Bearer <SELF_HEALING_API_KEY>` server-side.
-Preserve the upstream response status so the recorder can detect failures.
-
-For a standalone QA integration, configure the endpoint to the app's QA registration proxy instead.
-The wire format is QA's existing `{session_url, auxiliary_data}` envelope. No Self Healing server or
-package import is required in QA's ingestion/review implementation.
-
-## Uploads and lifecycle
-
-- Network, interaction and page-context artifacts use QA's version-1 namespace/key contracts.
-  `session/capture-producer` additionally identifies this package and version. That is provenance,
-  not authentication. QA recognizes network/interaction data by their artifact keys.
-- The package batches whole events into UTF-8 JSON requests of at most 256 KiB. Uploads use the
-  original fetch, so they do not capture themselves. Network errors, 429, server errors and
-  Self Healing's `upload_busy` retry up to three attempts with identical bodies and event IDs.
-- A single event larger than the request limit fails explicitly through `onError` and `flush()`;
-  it is not truncated to fit. Bodies above 1 MB are null and the per-page/session network budget
-  is 8 MB, matching the existing producer. Dropped counts are included in capture context.
-  Network and interaction counts each stop at 5,000 entries per page/session.
-- `await capture.flush()` waits for current in-flight captures and pending uploads. It rejects
-  if FullStory has no session yet or any capture/upload has failed. There is no durable offline queue.
-- `await capture.stop()` stops new capture, removes interaction listeners and flushes outstanding
-  data. It does not stop FullStory recording or seal the server's session, and it cannot be restarted
-  on the same page. `identify()` is a no-op after stopping.
-- Session rollover preserves ownership of requests already in flight and resets producer counters.
-  `captured_at` is absolute milliseconds; `source_timestamp` is page-relative milliseconds.
-
-Finalization is a server operation. Once a whole session has ended and all pages' uploads succeeded,
-send `{session_url, auxiliary_data: [], complete: true}` to Self Healing. A page flush or unload alone
-cannot establish that a multi-page FullStory session has ended.
+QA consumes compatible versioned artifacts and recognizes the package/version provenance. Its
+ingestion and review code does not import this package. When Subtext supplies the required accessors,
+the Self Healing skills will describe the replacement and migration.
 
 ## Development and release
 
-The source of truth is `packages/capture/src` in `replayio/self-healing`. Changes to producers belong
-here. QA owns its independent ingestion schemas and compatibility tests, not another installer copy.
+The capture implementation lives in `packages/capture/src`. Change producers here and keep QA’s
+independent ingestion schemas and compatibility tests compatible. Initial installation instructions live in the Self Healing skills. Every release
+that changes application integration requirements must add a versioned migration
+to UPGRADING.md and include verification steps.
 
 Publish locally with interactive npm authentication and 2FA:
 
@@ -83,10 +42,24 @@ artifact. npm owns the browser login/security-key/2FA prompts. Credentials and O
 arguments or repository secrets. Follow npm's authentication prompt when it appears.
 
 Use `npm run capture:publish -- --dry-run` to test building and packaging without login or publication.
-Registry errors fail the version check; already-published versions are skipped. After publishing,
-the script checks that the version is visible on npm. If visibility verification fails, rerun: an
-existing version is skipped, never overwritten. Temporary tarballs are removed when the script exits.
+Registry errors fail the version check; already-published versions are skipped. After npm accepts publication,
+the script briefly checks registry visibility. Delayed processing or registry read failures report
+visibility as pending without treating the accepted publication as a failure. Use the printed
+`npm view` command to check later; do not republish. Existing versions are skipped, never overwritten. Temporary tarballs are removed when the script exits.
 
 For later releases, update the package version and emitted producer metadata together, update the
 root lockfile, and run the script from the reviewed revision. The producer test checks that metadata
 matches the manifest. Installers receive updates through dependency upgrades and their lockfiles.
+
+## Capture failure behavior
+
+Version 0.1.1 preserves valid UTF-8 body text exactly. Binary, invalid UTF-8, or NUL-containing
+bodies are recorded as unavailable (`null`), retaining the exchange metadata. Both request and response
+bodies follow this rule, including compressed requests advertised as `text/plain`.
+
+An upload failure is reported through `onError` and retained for retry on the next capture upload or
+explicit `flush()`. Later batches are still attempted. Retries preserve event IDs for server deduplication;
+current session context and metrics are sent after retries so an older batch cannot leave stale counters.
+`flush()` rejects while any batch remains undelivered, and succeeds once they have all been accepted.
+
+Release tests and builds print concise progress; their full output is shown only if they fail.

package/UPGRADING.md

@@ -0,0 +1,72 @@
+# Upgrading an existing Self Healing integration
+
+This guide ships with `@replayio/self-healing-capture`. Coding agents should read it
+when asked to update the package and reconcile the application integration as well
+as its dependency. The original installation conversation is not required.
+
+## Upgrade procedure
+
+1. Read the installed package version and the version being installed. Use the
+   application's package manager to update the dependency and lockfile together.
+   Apply every migration below between those versions. If the old version is
+   unknown, inspect the integration against all migrations.
+2. Locate the existing `initCapture` call, identity hook, configured same-origin
+   capture endpoint, and server forwarding handler. Search for
+   `@replayio/self-healing-capture`, `initCapture`, `SELF_HEALING_URL`, and
+   `/api/v1/connection/sessions`. Follow the actual configured endpoint rather than
+   assuming the example route name.
+3. Update application-owned installation code as described below. Keep the existing
+   account, project connection, credentials, identity integration, and application
+   access controls. Do not repeat provisioning or rotate credentials for an upgrade.
+4. Run the application's relevant checks. Verify through its actual forwarding
+   route that captured network and interaction artifacts reach Self Healing.
+   Once deployed, exercise a real browser session and confirm HTTP 200 with
+   `status: "stored"` and a nonempty `session_id`. An initial metadata-only upload
+   or a FullStory recording alone does not verify auxiliary capture.
+5. Report the dependency version, application-code changes, checks, and session
+   delivery evidence. Distinguish local verification from deployed verification.
+   Follow the application's normal review/deployment process; do not claim the
+   deployed integration is updated until it has been verified.
+
+For the current complete setup contract, see
+https://self-healing.replay.io/api/v1/skills/setup-self-healing/SKILL.md.
+Use it to reconcile the existing integration; do not treat an upgrade as a new setup.
+
+## 0.1.3 — Packaged upgrade instructions
+
+No runtime integration changes beyond the migrations below. This release includes
+this guide so upgrades can be performed without access to the installer conversation.
+
+## 0.1.2 — One configurable limit per network exchange
+
+- The embedder option is `maxNetworkCaptureBytes`, defaulting to 1,000,000 bytes.
+  It measures the UTF-8 JSON encoding of each exchange, including both bodies,
+  headers, and event metadata. Omit the option to use the default; preserve an
+  intentional application-specific setting. If installation code uses the
+  pre-release `maxBatchBytes` option, replace it with this per-exchange policy.
+- Oversized exchanges are skipped and counted in `dropped_network_count`; later
+  captures continue. Batching is internal and includes space for envelope overhead.
+  A full-size exchange therefore produces an upload slightly larger than its limit.
+- Earlier setup examples installed a **256 KiB request-size check in the app's
+  forwarding handler**. Inspect for `256 * 1024`, `262144`, `413`, or the message
+  `Capture exceeds Self Healing request limit` along the capture route. Remove that
+  capture-specific check if present. Do not replace it with a 1 MB upload check.
+  Inspect capture-route body-parser configuration for the same obsolete restriction;
+  do not change unrelated application routes.
+- Forward the original JSON body to `/api/v1/connection/sessions` using server-held
+  Self Healing credentials and return the upstream status/body. Do not truncate
+  capture fields, reimplement batching, or expose credentials in the browser.
+- Verify an exchange with a body larger than 256 KiB but below the configured
+  exchange limit (for example, 500 KB under the default) is actually delivered
+  through the forwarding route with its body intact. Also verify a later small
+  exchange after an oversized exchange, and check the dropped counter increases.
+
+Updating the npm dependency cannot modify an application's existing forwarding
+handler. These application-code migrations are part of the package upgrade.
+
+## 0.1.1 — Capture decoding and retry fixes
+
+No application-code migration is required. Keep capture initialization and the
+existing identity hook. Failed uploads remain retryable; later batches continue.
+Binary, invalid UTF-8, and NUL-containing bodies are represented as unavailable
+while their exchange metadata remains captured.

package/dist/index.d.ts

@@ -2,6 +2,8 @@ export interface CaptureOptions {
     orgId: string;
     /** Same-origin POST route holding the server-side credential. */
     endpoint?: string;
+    /** Maximum UTF-8 JSON bytes per network exchange (bodies, headers and metadata). Defaults to 1,000,000. */
+    maxNetworkCaptureBytes?: number;
     onError?: (error: Error) => void;
 }
 export interface CaptureController {

package/dist/index.js

@@ -1,5 +1,5 @@
 import { FullStory, init } from "@fullstory/browser";
-import { splitBatches } from "./transport.js";
+import { decodeCaptureBody, splitBatches, DEFAULT_MAX_NETWORK_CAPTURE_BYTES, } from "./transport.js";
 let active;
 let activeOptions;
 const CAPTURED_SESSION_INTERACTION_EVENTS = [
@@ -17,7 +17,6 @@ function countsAsSessionInteraction(event) {
             event.type === "keydown" &&
             ["Enter", "Escape"].includes(event.key)));
 }
-const MAX_BODY_BYTES = 1_000_000;
 const MAX_CAPTURE_BYTES = 8_000_000;
 const MAX_CAPTURED_INTERACTIONS = 5_000;
 const ACTIONABLE_SELECTOR = 'button, a[href], input, select, textarea, summary, [role="button"], [role="link"], [role="checkbox"], [role="menuitem"], [role="option"], [role="radio"], [role="switch"], [role="tab"], [contenteditable="true"]';
@@ -37,15 +36,19 @@ function identifyFullStoryUser(user) {
     if (user.email)
         setCapturedUserEmail?.(user.email);
 }
-function shouldStopSessionUploads(status) {
-    return status < 200 || status >= 300;
-}
 export function initCapture(options) {
     if (typeof window === "undefined")
         throw new Error("initCapture must run in the browser");
+    const maxNetworkCaptureBytes = options.maxNetworkCaptureBytes ?? DEFAULT_MAX_NETWORK_CAPTURE_BYTES;
+    if (!Number.isSafeInteger(maxNetworkCaptureBytes) ||
+        maxNetworkCaptureBytes <= 0)
+        throw new Error("maxNetworkCaptureBytes must be a positive safe integer");
     if (active) {
         if (options.orgId !== activeOptions?.orgId ||
-            options.endpoint !== activeOptions?.endpoint) {
+            options.endpoint !== activeOptions?.endpoint ||
+            maxNetworkCaptureBytes !==
+                (activeOptions?.maxNetworkCaptureBytes ??
+                    DEFAULT_MAX_NETWORK_CAPTURE_BYTES)) {
             throw new Error("Capture is already initialized with different options");
         }
         return active;
@@ -88,7 +91,8 @@ export function initCapture(options) {
             queuedExchangeCount: 0,
             queuedInteractionCount: -1,
             queuedCapturedInteractionCount: 0,
-            uploadsStopped: false,
+            pendingBatches: [],
+            uploadError: undefined,
             userEmail,
             queuedUserEmail: null,
             uploadTimer: null,
@@ -140,11 +144,9 @@ export function initCapture(options) {
         session.userEmail = email;
         uploadCapture(session);
     };
-    async function boundedBody(value) {
+    async function captureBody(value) {
         const bytes = await value.clone().arrayBuffer();
-        if (bytes.byteLength > MAX_BODY_BYTES)
-            return null;
-        return new TextDecoder().decode(bytes);
+        return decodeCaptureBody(bytes);
     }
     async function sendCaptureBatch(body) {
         for (let attempt = 0;; attempt++) {
@@ -172,8 +174,12 @@ export function initCapture(options) {
         }
     }
     function queueCaptureUpload(session) {
-        if (!session.sessionUrl || session.uploadsStopped)
+        if (!session.sessionUrl)
             return Promise.resolve();
+        session.uploadChain = session.uploadChain.then(() => drainCaptureUpload(session));
+        return session.uploadChain;
+    }
+    async function drainCaptureUpload(session) {
         const exchanges = session.capturedExchanges
             .slice(session.queuedExchangeCount)
             .map(({ startup_body: _body, status_text: _status, ...exchange }) => exchange)
@@ -190,9 +196,9 @@ export function initCapture(options) {
                 namespace: "session",
                 key: "capture-producer",
                 schema_version: 1,
-                payload: { name: "@replayio/self-healing-capture", version: "0.1.0" },
+                payload: { name: "@replayio/self-healing-capture", version: "0.1.3" },
             },
-            ...(context !== session.queuedContext
+            ...(context !== session.queuedContext || session.pendingBatches.length > 0
                 ? [
                     {
                         namespace: "session",
@@ -237,7 +243,8 @@ export function initCapture(options) {
                     },
                 ]
                 : []),
-            ...(session.interactionCount !== session.queuedInteractionCount
+            ...(session.interactionCount !== session.queuedInteractionCount ||
+                session.pendingBatches.length > 0
                 ? [
                     {
                         namespace: "session",
@@ -250,7 +257,9 @@ export function initCapture(options) {
                     },
                 ]
                 : []),
-            ...(session.userEmail && session.userEmail !== session.queuedUserEmail
+            ...(session.userEmail &&
+                (session.userEmail !== session.queuedUserEmail ||
+                    session.pendingBatches.length > 0)
                 ? [
                     {
                         namespace: "session",
@@ -261,39 +270,51 @@ export function initCapture(options) {
                 ]
                 : []),
         ];
-        if (auxiliaryData.length === 1)
-            return session.uploadChain;
+        if (auxiliaryData.length === 1 && !session.pendingBatches.length)
+            return;
+        let batches;
+        try {
+            batches = splitBatches({
+                session_url: session.sessionUrl,
+                auxiliary_data: auxiliaryData,
+            }, maxNetworkCaptureBytes);
+        }
+        catch (error) {
+            reportError(error);
+            return;
+        }
         session.queuedContext = context;
         session.queuedExchangeCount = session.capturedExchanges.length;
         session.queuedInteractionCount = session.interactionCount;
         session.queuedCapturedInteractionCount =
             session.capturedInteractions.length;
         session.queuedUserEmail = session.userEmail;
-        const body = {
-            session_url: session.sessionUrl,
-            auxiliary_data: auxiliaryData,
-        };
-        // Each immutable batch is retried with the same event IDs; the server deduplicates it.
-        session.uploadChain = session.uploadChain
-            .then(async () => {
-            if (session.uploadsStopped)
-                return;
-            for (const batch of splitBatches(body)) {
+        session.pendingBatches.push(...batches);
+        const failed = [];
+        session.uploadError = undefined;
+        // Retry identical event IDs, but a rejected batch must not block later interactions.
+        for (const batch of session.pendingBatches) {
+            try {
                 const response = await sendCaptureBatch(batch);
-                if (shouldStopSessionUploads(response.status)) {
-                    session.uploadsStopped = true;
+                if (!response.ok)
                     throw new Error(`Capture upload failed: ${response.status}`);
+            }
+            catch (error) {
+                failed.push(batch);
+                session.uploadError =
+                    error instanceof Error ? error : new Error(String(error));
+                try {
+                    (options.onError ?? console.error)(session.uploadError);
+                }
+                catch {
+                    /* observers cannot break the app */
                 }
             }
-        })
-            .catch((error) => {
-            session.uploadsStopped = true;
-            reportError(error);
-        });
-        return session.uploadChain;
+        }
+        session.pendingBatches = failed;
     }
     function uploadCapture(session) {
-        if (!session.sessionUrl || session.uploadsStopped)
+        if (!session.sessionUrl)
             return;
         if (session.uploadTimer)
             return;
@@ -431,7 +452,7 @@ export function initCapture(options) {
         const capturedAt = performance.timeOrigin + sourceTimestamp;
         const exchangeId = crypto.randomUUID();
         const requestBody = request.method !== "GET" && request.method !== "HEAD"
-            ? boundedBody(request).catch(() => null)
+            ? captureBody(request).catch(() => null)
             : Promise.resolve(null);
         const responsePromise = nativeFetch(request);
         inFlight.add(responsePromise);
@@ -464,16 +485,20 @@ export function initCapture(options) {
                 request_headers: Object.fromEntries(request.headers.entries()),
                 request_body: capturedRequestBody,
                 response_headers: Object.fromEntries(clone.headers.entries()),
-                response_body: responseBytes && responseBytes.byteLength <= MAX_BODY_BYTES
-                    ? new TextDecoder().decode(responseBytes)
-                    : null,
+                response_body: responseBytes ? decodeCaptureBody(responseBytes) : null,
                 ...(startedBeforeReady &&
                     (request.method === "GET" || request.method === "HEAD") &&
-                    responseBytes &&
-                    responseBytes.byteLength <= MAX_BODY_BYTES
+                    responseBytes
                     ? { startup_body: responseBytes, status_text: clone.statusText }
                     : {}),
             };
+            const { startup_body: _startup, status_text: _status, ...uploadedExchange } = exchange;
+            if (new TextEncoder().encode(JSON.stringify(uploadedExchange)).byteLength >
+                maxNetworkCaptureBytes) {
+                session.droppedNetworkCount++;
+                uploadCapture(session);
+                return;
+            }
             session.capturedBytes += exchangeBytes;
             session.capturedExchanges.push(exchange);
             uploadCapture(session);
@@ -503,6 +528,10 @@ export function initCapture(options) {
             }
             if (lastError)
                 throw lastError;
+            for (const session of sessions) {
+                if (session.uploadError)
+                    throw session.uploadError;
+            }
             if (!currentSession.sessionUrl)
                 throw new Error("FullStory session is not ready");
         },

package/dist/transport.d.ts

@@ -9,6 +9,10 @@ export interface CaptureBatch {
     session_url: string;
     auxiliary_data: Artifact[];
 }
-export declare const MAX_BATCH_BYTES: number;
+export declare const DEFAULT_MAX_NETWORK_CAPTURE_BYTES = 1000000;
+/** Derive the upload budget from the capture limit, allowing one full exchange plus its envelope. */
+export declare function uploadBatchBytes(sessionUrl: string, maxNetworkCaptureBytes?: number): number;
 /** Split between whole events, preserving IDs and payloads for identical retries. */
-export declare function splitBatches(input: CaptureBatch, maxBytes?: number): string[];
+export declare function splitBatches(input: CaptureBatch, maxNetworkCaptureBytes?: number): string[];
+/** Binary or NUL-containing bodies cannot be represented in the text-only capture schema. */
+export declare function decodeCaptureBody(bytes: ArrayBuffer): string | null;

package/dist/transport.js

@@ -1,6 +1,22 @@
-export const MAX_BATCH_BYTES = 256 * 1024;
+export const DEFAULT_MAX_NETWORK_CAPTURE_BYTES = 1_000_000;
+/** Derive the upload budget from the capture limit, allowing one full exchange plus its envelope. */
+export function uploadBatchBytes(sessionUrl, maxNetworkCaptureBytes = DEFAULT_MAX_NETWORK_CAPTURE_BYTES) {
+    return (maxNetworkCaptureBytes +
+        new TextEncoder().encode(JSON.stringify({
+            session_url: sessionUrl,
+            auxiliary_data: [
+                {
+                    namespace: "network",
+                    key: "captured-exchanges",
+                    schema_version: 1,
+                    payload: { version: 1, exchanges: [] },
+                },
+            ],
+        })).byteLength);
+}
 /** Split between whole events, preserving IDs and payloads for identical retries. */
-export function splitBatches(input, maxBytes = MAX_BATCH_BYTES) {
+export function splitBatches(input, maxNetworkCaptureBytes = DEFAULT_MAX_NETWORK_CAPTURE_BYTES) {
+    const maxBytes = uploadBatchBytes(input.session_url, maxNetworkCaptureBytes);
     const encode = (artifacts) => JSON.stringify({
         session_url: input.session_url,
         auxiliary_data: artifacts,
@@ -63,3 +79,13 @@ export function splitBatches(input, maxBytes = MAX_BATCH_BYTES) {
         result.push(encode(pending));
     return result;
 }
+/** Binary or NUL-containing bodies cannot be represented in the text-only capture schema. */
+export function decodeCaptureBody(bytes) {
+    try {
+        const text = new TextDecoder("utf-8", { fatal: true }).decode(bytes);
+        return text.includes("\0") ? null : text;
+    }
+    catch {
+        return null;
+    }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@replayio/self-healing-capture",
-  "version": "0.1.0",
+  "version": "0.1.3",
   "description": "FullStory auxiliary capture for Self Healing and compatible session ingesters",
   "type": "module",
   "main": "./dist/index.js",
@@ -13,7 +13,8 @@
   },
   "files": [
     "dist",
-    "README.md"
+    "README.md",
+    "UPGRADING.md"
   ],
   "sideEffects": false,
   "license": "UNLICENSED",