@scoutline-ai/middleware 0.2.0 → 0.4.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Scoutline
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -85,10 +85,22 @@ The server sees every request, including crawlers that never run JavaScript. The
 
 The script and its events are served from your own domain by this package, so there is no public key and no third-party request. If your proxy `matcher` is limited to certain paths, include `/_scoutline/:path*`. These routes run before your own proxy, so an auth check won't block them.
 
-It stores nothing and sets no cookies. A page links to its server request through a `Server-Timing` header. Each batch of events is one request to your server, sent when the page closes or after 20 events.
+It stores nothing and sets no cookies. A page links to its server request through a `Server-Timing` header. Page views are sent straight away. Other events wait for the next page view, the page closing, or 20 events, and each batch is one request to your server. Returning visitors can keep a cached script for up to an hour after you upgrade this package.
 
 Browser events can be forged, so they are never `verified` and can't confirm an outcome. Reports keep them out of request counts.
 
+## Custom events
+
+Name something that happened on a page, and use it as a funnel step.
+
+```js
+window.scoutline?.track("signup_started", { plan: "pro" });
+```
+
+Names can be up to 64 letters, numbers, spaces and `_ . : -`. Each event keeps up to 8 properties, and each property's value is cut to 128 characters. Keys must start with a letter or number. Never put personal data in a name or property: unlike everything else on this page, what you send here is your choice.
+
+`window.scoutline` exists once the script has loaded, so call this from a click or after the page is ready. A page can forge these, so a custom event can never confirm an outcome, even with the same name.
+
 ## Outcomes
 
 Outcomes are what your server confirms really happened, such as an order or a signup. They are never inferred from a 200.
@@ -120,7 +132,7 @@ await outcomes.record("checkout_attempted", { taskId, errorCategory: "card_decli
 
 - **`id`** makes repeats safe: the same id counts once. It is hashed before sending. Without it, every call counts.
 - **`taskId`** is your id for one piece of work, and puts the event on that Workflow timeline.
-- **`headers`** are the incoming request's headers, which record who acted and the trace id.
+- **`headers`** are the incoming request's headers, which record who acted and the trace id. Without them an outcome can't be tied to a visitor, so it can't be a funnel step.
 
 Nothing here throws, so a Scoutline problem can't fail your action. Each call returns `{ ok: true }` or `{ ok: false, reason }`. Events are sent straight away, at most twice for up to 2 seconds each, so don't await them if your action can't spare that.
 
@@ -140,15 +152,47 @@ withScoutline(yourProxy, {
 
 Requests that share a W3C `traceparent` with an event in the task also appear on its timeline, labelled as linked by trace.
 
+## Recognising visitors
+
+Without any setup, each visitor gets an anonymous id that changes every day and differs per site. It is a hash of their address and browser, made on the collector with a random salt that is thrown away afterwards. There is no cookie, and it can't follow anyone across days or sites. With `sendClientIp: false` there is no id, so those visitors can't appear in funnels or sessions.
+
+People on one address with the same browser share a daily id, and a session ends when the id changes at midnight UTC. A session also ends after 30 minutes with nothing from that visitor. Funnels count a visitor who finishes the steps in order within 24 hours. `sampleUnknown` below 1 leaves gaps in both.
+
+To recognise visitors who agree across days, turn consent on. This is off by default, and without it this package never sets a cookie.
+
+```ts
+withScoutline(yourProxy, {
+  consent: { banner: { privacyUrl: "/privacy" } },
+});
+```
+
+This shows a small banner with equal Accept and Decline buttons. Nothing is stored until someone chooses. Accepting sets a random first-party id in `_scoutline_id` for 13 months, and any choice is remembered in `_scoutline_consent`. Declining removes the id. Browsers sending Global Privacy Control count as declining. You can change the wording with `message`, `accept` and `decline`.
+
+If you already have a consent tool, leave out `banner` and connect it instead:
+
+```ts
+withScoutline(yourProxy, { consent: {} });
+```
+
+```js
+window.scoutline.consent(true); // or false, which also removes the id
+```
+
+To let visitors change their mind, call `window.scoutline.showConsent()` from a link in your footer. Your privacy policy still needs to describe this, and whether you need consent for the daily id depends on where your visitors are.
+
 ## What is sent
 
 From the server: the method, pathname, user agent, whether a few standard headers were present, and request and trace ids. Images, video, fonts and similar files are recorded for crawlers and agents, and skipped for everyone else, so one page load counts once. From the browser: the pathname, event type, element type such as `button`, status, and load time.
 
-Query strings, cookies, auth headers, bodies, page text and field values are never sent.
+**Referrers** are the other site's hostname only, such as `google.com`. **Outbound links** keep their origin and path, such as `https://stripe.com/pay`.
+
+Query strings, auth headers, bodies, page text and field values are never sent, and neither are cookies, apart from Scoutline's own id if you turn on consent and the visitor agrees. The only other thing that leaves your site is what you pass to `track`.
+
+**Device details** come from the browser's Client Hints and, where your page runs the script, its major OS version, device model, whether it has a touch screen, and a screen size group such as `small`. Exact screen sizes and full versions are not sent, and these details are never combined into an identifier. Safari and Firefox share less, so some fields stay empty rather than guessed.
 
 **The visitor's IP address** is sent to the collector, which uses it to find the country and network, then discards it. It is never stored. Country is where the network is registered, so cloud traffic may not show where the machine really is. Turn this off with `sendClientIp: false`, and country and network stay empty.
 
-The collector classifies each request, so new crawlers are recognised without updating this package. A matching user agent is recorded as `declared`, because any client can send one. Other automation is `inferred`, and everything else is `unknown`. A browser claim that doesn't add up is only `inferred` when two different kinds of evidence agree, such as missing browser headers on a hosting network. This package never reports `verified`.
+The collector classifies each request, so new crawlers are recognised without updating this package. A matching user agent is recorded as `declared`, because any client can send one. Other automation is `inferred`, and everything else is `unknown`. A browser claim that doesn't add up is only `inferred` when two different kinds of evidence agree, such as missing browser headers on a hosting network. This package never decides `verified`. The collector does, when a crawler's address is in the list its operator publishes and that address came from something a client can't fake: Vercel, Cloudflare, a direct connection, or your `trustedIpHeader`.
 
 ## Options
 
@@ -159,4 +203,6 @@ The collector classifies each request, so new crawlers are recognised without up
 | `sampleUnknown` | `1` | Share of unclassified traffic to send, from 0 to 1 |
 | `ignore` | none | `(pathname) => boolean` to skip a path |
 | `sendClientIp` | `true` | Send the visitor's IP for the country and network lookup. It is never stored |
+| `trustedIpHeader` | none | The header your own proxy sets to the visitor's IP. Needed to verify crawlers on self-hosted sites |
+| `consent` | off | `{ banner: true }` shows a banner, and `{}` connects your own consent tool |
 | `context` | none | `(request) => ({ taskId, sessionId, resourceId, contentVersion })` adds your own ids |

package/dist/browser-script.d.ts

@@ -1,7 +1,16 @@
-/**
- * Served from the customer's own domain at /_scoutline/s. It stays plain ES5 and
- * dependency free so it can load on any page. It collects no text, field values
- * or query strings, and stores nothing, so no cookie or consent is involved.
- * Everything it sends is an unverified claim and is revalidated in `browser.ts`.
- */
-export declare const BROWSER_SCRIPT = "(function(){\n  var ENDPOINT=\"/_scoutline/e\",MAX=20,queue=[],path=location.pathname;\n  var nav=performance.getEntriesByType?performance.getEntriesByType(\"navigation\")[0]:null;\n  var automated=navigator.webdriver===true;\n  function serverId(){\n    var timing=nav&&nav.serverTiming;\n    for(var i=0;timing&&i<timing.length;i++){\n      if(timing[i].name===\"scoutline\")return timing[i].description;\n    }\n  }\n  function send(){\n    if(!queue.length)return;\n    var now=Date.now(),events=[];\n    for(var i=0;i<queue.length;i++){\n      var e=queue[i];e.ageMs=now-e.at;delete e.at;events.push(e);\n    }\n    queue=[];\n    var body=JSON.stringify({automated:automated,events:events});\n    if(!navigator.sendBeacon||!navigator.sendBeacon(ENDPOINT,body)){\n      fetch(ENDPOINT,{method:\"POST\",body:body,keepalive:true})[\"catch\"](function(){});\n    }\n  }\n  function add(type,extra){\n    var e=extra||{};e.type=type;e.path=location.pathname;e.at=Date.now();\n    queue.push(e);\n    if(queue.length>=MAX)send();\n  }\n  function firstView(){\n    add(\"page_view\",{\n      parent:serverId(),\n      status:nav&&nav.responseStatus?nav.responseStatus:undefined,\n      durationMs:nav&&nav.duration?Math.round(nav.duration):undefined\n    });\n  }\n  function changed(){\n    if(location.pathname===path)return;\n    path=location.pathname;add(\"page_view\");\n  }\n  [\"pushState\",\"replaceState\"].forEach(function(name){\n    var original=history[name];\n    history[name]=function(){var r=original.apply(this,arguments);changed();return r;};\n  });\n  addEventListener(\"popstate\",changed);\n  document.addEventListener(\"click\",function(event){\n    var t=event.target&&event.target.closest&&event.target.closest(\"a,button,[role=button],input[type=submit]\");\n    if(t)add(\"click\",{element:t.tagName.toLowerCase()});\n  },true);\n  document.addEventListener(\"submit\",function(){add(\"form_submit\");},true);\n  addEventListener(\"error\",function(event){if(event.message)add(\"error\");});\n  addEventListener(\"unhandledrejection\",function(){add(\"error\");});\n  addEventListener(\"pagehide\",send);\n  document.addEventListener(\"visibilitychange\",function(){\n    if(document.visibilityState===\"hidden\")send();\n  });\n  if(document.readyState===\"complete\")setTimeout(firstView);\n  else addEventListener(\"load\",function(){setTimeout(firstView);});\n})();";
+export type ConsentBanner = {
+    message?: string;
+    accept?: string;
+    decline?: string;
+    /** Linked from the banner. Must start with `/` or `https://`. */
+    privacyUrl?: string;
+};
+export type ConsentOptions = {
+    /**
+     * Shows Scoutline's banner. Leave it out if you already have a consent tool,
+     * and call `window.scoutline.consent(true)` or `(false)` from that instead.
+     */
+    banner?: boolean | ConsentBanner;
+};
+export declare function browserScript(consent?: ConsentOptions): string;
+export declare const BROWSER_SCRIPT: string;

package/dist/browser-script.js

@@ -1,13 +1,28 @@
 /**
  * Served from the customer's own domain at /_scoutline/s. It stays plain ES5 and
  * dependency free so it can load on any page. It collects no text, field values
- * or query strings, and stores nothing, so no cookie or consent is involved.
- * Everything it sends is an unverified claim and is revalidated in `browser.ts`.
+ * or query strings, and stores nothing. Device details stay coarse, such as a
+ * screen size group, and are never combined into an identifier. Everything it
+ * sends is an unverified claim and is revalidated in `browser.ts`. It only touches
+ * cookies when a site turns consent on, and then only after the visitor agrees.
+ * A page view goes out straight away, because a visitor who stays on a page
+ * shouldn't be missing from sessions until they leave.
  */
-export const BROWSER_SCRIPT = `(function(){
-  var ENDPOINT="/_scoutline/e",MAX=20,queue=[],path=location.pathname;
+const SCRIPT = `/*CONSENT*/(function(){
+  var ENDPOINT="/_scoutline/e",MAX=20,MAX_PROPERTIES=8,MAX_VALUE=128,queue=[],path=location.pathname;
   var nav=performance.getEntriesByType?performance.getEntriesByType("navigation")[0]:null;
   var automated=navigator.webdriver===true;
+  var device={touch:navigator.maxTouchPoints>0};
+  var shortSide=window.screen?Math.min(screen.width,screen.height):0;
+  if(shortSide)device.screen=shortSide<600?"small":shortSide<900?"medium":"large";
+  if(window.matchMedia)device.coarsePointer=matchMedia("(pointer:coarse)").matches;
+  var agent=navigator.userAgentData,details=null;
+  if(agent&&agent.getHighEntropyValues){
+    details=agent.getHighEntropyValues(["platformVersion","model"]).then(function(values){
+      device.platform=agent.platform;device.mobile=agent.mobile;
+      device.platformVersion=values.platformVersion;device.model=values.model;
+    })["catch"](function(){});
+  }
   function serverId(){
     var timing=nav&&nav.serverTiming;
     for(var i=0;timing&&i<timing.length;i++){
@@ -21,7 +36,7 @@ export const BROWSER_SCRIPT = `(function(){
       var e=queue[i];e.ageMs=now-e.at;delete e.at;events.push(e);
     }
     queue=[];
-    var body=JSON.stringify({automated:automated,events:events});
+    var body=JSON.stringify({automated:automated,device:device,events:events});
     if(!navigator.sendBeacon||!navigator.sendBeacon(ENDPOINT,body)){
       fetch(ENDPOINT,{method:"POST",body:body,keepalive:true})["catch"](function(){});
     }
@@ -29,7 +44,23 @@ export const BROWSER_SCRIPT = `(function(){
   function add(type,extra){
     var e=extra||{};e.type=type;e.path=location.pathname;e.at=Date.now();
     queue.push(e);
-    if(queue.length>=MAX)send();
+    if(queue.length>=MAX||type==="page_view")send();
+  }
+  function outboundUrl(link){
+    if(link.tagName!=="A"||!/^https?:$/.test(link.protocol))return;
+    var host=location.hostname.replace(/^www\\./,"");
+    if(link.hostname.replace(/^www\\./,"")===host)return;
+    return link.origin+link.pathname;
+  }
+  function properties(raw){
+    var clean={},count=0;
+    for(var key in raw){
+      var value=raw[key],type=typeof value;
+      if(count<MAX_PROPERTIES&&(type==="string"||type==="number"||type==="boolean")){
+        clean[key]=String(value).slice(0,MAX_VALUE);count++;
+      }
+    }
+    return clean;
   }
   function firstView(){
     add("page_view",{
@@ -49,7 +80,10 @@ export const BROWSER_SCRIPT = `(function(){
   addEventListener("popstate",changed);
   document.addEventListener("click",function(event){
     var t=event.target&&event.target.closest&&event.target.closest("a,button,[role=button],input[type=submit]");
-    if(t)add("click",{element:t.tagName.toLowerCase()});
+    if(!t)return;
+    var url=outboundUrl(t);
+    if(url)add("outbound",{url:url});
+    else add("click",{element:t.tagName.toLowerCase()});
   },true);
   document.addEventListener("submit",function(){add("form_submit");},true);
   addEventListener("error",function(event){if(event.message)add("error");});
@@ -58,6 +92,111 @@ export const BROWSER_SCRIPT = `(function(){
   document.addEventListener("visibilitychange",function(){
     if(document.visibilityState==="hidden")send();
   });
-  if(document.readyState==="complete")setTimeout(firstView);
-  else addEventListener("load",function(){setTimeout(firstView);});
+  window.scoutline=window.scoutline||{};
+  window.scoutline.track=function(name,raw){
+    if(typeof name==="string")add("custom",{name:name,properties:properties(raw)});
+  };
+  function start(){if(details)details.then(firstView);else firstView();}
+  if(document.readyState==="complete")setTimeout(start);
+  else addEventListener("load",function(){setTimeout(start);});
 })();`;
+/**
+ * Added only for sites that turn consent on, so every other site's script has
+ * no cookie code in it at all. Nothing is stored until a visitor agrees. The
+ * choice itself is remembered, which is what a consent cookie is for, and Global
+ * Privacy Control counts as a refusal.
+ */
+const CONSENT = `(function(banner){
+  var ID="_scoutline_id",CHOICE="_scoutline_consent",AGE=34128000,box;
+  function read(name){
+    var match=document.cookie.match("(?:^|; )"+name+"=([^;]*)");
+    return match?match[1]:null;
+  }
+  function write(name,value,age){
+    document.cookie=name+"="+value+"; Max-Age="+age+"; Path=/; SameSite=Lax"+
+      (location.protocol==="https:"?"; Secure":"");
+  }
+  function randomId(){
+    if(crypto.randomUUID)return crypto.randomUUID();
+    var b=crypto.getRandomValues(new Uint8Array(16)),h=[];
+    b[6]=b[6]&15|64;b[8]=b[8]&63|128;
+    for(var i=0;i<16;i++)h.push((b[i]+256).toString(16).slice(1));
+    return h.slice(0,4).join("")+"-"+h.slice(4,6).join("")+"-"+h.slice(6,8).join("")+
+      "-"+h.slice(8,10).join("")+"-"+h.slice(10).join("");
+  }
+  function hide(){if(box&&box.parentNode)box.parentNode.removeChild(box);box=null;}
+  function choose(yes){
+    write(CHOICE,yes?"yes":"no",AGE);
+    if(!yes)write(ID,"",0);
+    else if(!read(ID))write(ID,randomId(),AGE);
+    hide();
+  }
+  function button(label,yes){
+    var b=document.createElement("button");
+    b.type="button";b.textContent=label;
+    b.style.cssText="flex:1;padding:8px 12px;border:1px solid #d4d4d8;border-radius:8px;"+
+      "background:#fff;color:#18181b;font:inherit;cursor:pointer";
+    b.addEventListener("click",function(){choose(yes);});
+    return b;
+  }
+  function show(){
+    if(box||!document.body)return;
+    box=document.createElement("div");
+    box.setAttribute("role","dialog");
+    box.setAttribute("aria-label","Cookie choice");
+    box.style.cssText="position:fixed;left:16px;right:16px;bottom:16px;z-index:2147483647;"+
+      "max-width:380px;box-sizing:border-box;padding:16px;border-radius:12px;background:#fff;"+
+      "color:#18181b;box-shadow:0 8px 30px rgba(0,0,0,.18);"+
+      "font:14px/1.45 system-ui,sans-serif";
+    var text=document.createElement("p");
+    text.style.margin="0 0 12px";
+    text.textContent=banner.message;
+    if(banner.privacyUrl){
+      var link=document.createElement("a");
+      link.href=banner.privacyUrl;link.textContent=" Privacy policy";
+      link.style.color="inherit";
+      text.appendChild(link);
+    }
+    var row=document.createElement("div");
+    row.style.cssText="display:flex;gap:8px";
+    row.appendChild(button(banner.decline,false));
+    row.appendChild(button(banner.accept,true));
+    box.appendChild(text);box.appendChild(row);
+    document.body.appendChild(box);
+  }
+  window.scoutline=window.scoutline||{};
+  window.scoutline.consent=choose;
+  window.scoutline.showConsent=function(){if(banner)show();};
+  if(navigator.globalPrivacyControl===true){
+    if(read(ID))write(ID,"",0);
+    return;
+  }
+  if(!banner||read(CHOICE))return;
+  if(document.body)show();
+  else addEventListener("DOMContentLoaded",show);
+})(/*BANNER*/);`;
+const DEFAULT_BANNER = {
+    message: "We use a cookie to recognise you when you come back and to understand how this site is used.",
+    accept: "Accept",
+    decline: "Decline",
+};
+function bannerConfig(banner) {
+    if (!banner)
+        return null;
+    const custom = banner === true ? {} : banner;
+    return {
+        ...DEFAULT_BANNER,
+        ...custom,
+        privacyUrl: custom.privacyUrl && /^(\/|https:\/\/)/.test(custom.privacyUrl)
+            ? custom.privacyUrl
+            : undefined,
+    };
+}
+export function browserScript(consent) {
+    if (!consent)
+        return BROWSER_SCRIPT;
+    // Escaped so a site's own text can't close the script or add to it.
+    const banner = JSON.stringify(bannerConfig(consent.banner)).replace(/</g, "\\u003c");
+    return SCRIPT.replace("/*CONSENT*/", () => CONSENT.replace("/*BANNER*/", () => banner));
+}
+export const BROWSER_SCRIPT = SCRIPT.replace("/*CONSENT*/", "");

package/dist/browser.d.ts

@@ -14,6 +14,43 @@ export declare function isSameOrigin(headers: Headers): boolean;
  * load also produces a server event, and Googlebot runs this script too.
  */
 export declare function toBrowserEvent(raw: unknown, actor: Actor, now: number, sampleRate: number): {
+    eventName: string;
+    eventProperties: {
+        [k: string]: any;
+    } | undefined;
+    outboundUrl?: undefined;
+    eventId: `${string}-${string}-${string}-${string}-${string}`;
+    occurredAt: number;
+    family: string;
+    actor: Actor;
+    action: string;
+    url: string;
+    interface: string;
+    parentActionId: string | undefined;
+    status: number | undefined;
+    durationMs: number | undefined;
+    sampleRate: number;
+    classifierVersion: string;
+} | {
+    outboundUrl: string;
+    eventName?: undefined;
+    eventProperties?: undefined;
+    eventId: `${string}-${string}-${string}-${string}-${string}`;
+    occurredAt: number;
+    family: string;
+    actor: Actor;
+    action: string;
+    url: string;
+    interface: string;
+    parentActionId: string | undefined;
+    status: number | undefined;
+    durationMs: number | undefined;
+    sampleRate: number;
+    classifierVersion: string;
+} | {
+    eventName?: undefined;
+    eventProperties?: undefined;
+    outboundUrl?: undefined;
     eventId: `${string}-${string}-${string}-${string}-${string}`;
     occurredAt: number;
     family: string;
@@ -29,5 +66,14 @@ export declare function toBrowserEvent(raw: unknown, actor: Actor, now: number,
 } | null;
 export declare function parseBrowserBody(text: string): {
     automated: boolean;
+    device: {
+        touch: boolean | undefined;
+        coarsePointer: boolean | undefined;
+        mobile: boolean | undefined;
+        screen: string | undefined;
+        platform: string | undefined;
+        platformVersion: string | undefined;
+        model: string | undefined;
+    } | undefined;
     events: unknown[];
 } | null;

package/dist/browser.js

@@ -1,14 +1,31 @@
 export const SCRIPT_PATH = "/_scoutline/s";
 export const EVENTS_PATH = "/_scoutline/e";
 export const BROWSER_CLASSIFIER_VERSION = "browser-0.1";
-const MAX_BODY_CHARS = 16 * 1024;
+/** Twenty custom events at their limits, and under the 64 KiB a beacon can carry. */
+const MAX_BODY_CHARS = 48 * 1024;
 const MAX_EVENTS = 20;
 const MAX_AGE_MS = 10 * 60 * 1000;
 const MAX_PATH_CHARS = 2048;
 const MAX_DURATION_MS = 60 * 60 * 1000;
-const TYPES = new Set(["page_view", "click", "form_submit", "error"]);
+const TYPES = new Set([
+    "page_view",
+    "click",
+    "form_submit",
+    "error",
+    "custom",
+    "outbound",
+]);
 const ELEMENTS = new Set(["a", "button", "input", "div", "span"]);
 const SERVER_ID = /^[0-9a-f-]{36}$/;
+const SCREENS = new Set(["small", "medium", "large"]);
+const MAX_MODEL_CHARS = 64;
+const PLATFORM = /^[A-Za-z][A-Za-z0-9 ]{0,31}$/;
+const MODEL = /^[\w .()+-]+$/;
+const EVENT_NAME = /^[\w.:\- ]{1,64}$/;
+const PROPERTY_KEY = /^[A-Za-z0-9][\w.-]{0,39}$/;
+const MAX_PROPERTIES = 8;
+const MAX_PROPERTY_CHARS = 128;
+const MAX_OUTBOUND_CHARS = 512;
 /**
  * Anything in the page can be forged, so only a same-origin browser request is
  * accepted. This keeps out other sites, not a determined client, which is why
@@ -47,6 +64,45 @@ function pathOf(value) {
         return null;
     }
 }
+/** Keeps the origin and path. A query can carry a token, and a link is enough. */
+function outboundOf(value) {
+    if (typeof value !== "string" || value.length > MAX_OUTBOUND_CHARS)
+        return null;
+    try {
+        const url = new URL(value);
+        return /^https?:$/.test(url.protocol) ? `${url.origin}${url.pathname}` : null;
+    }
+    catch {
+        return null;
+    }
+}
+function propertiesOf(value) {
+    if (typeof value !== "object" || value === null)
+        return undefined;
+    const entries = Object.entries(value)
+        .filter(([key, item]) => PROPERTY_KEY.test(key) &&
+        typeof item === "string" &&
+        item.length <= MAX_PROPERTY_CHARS)
+        .slice(0, MAX_PROPERTIES);
+    return entries.length > 0 ? Object.fromEntries(entries) : undefined;
+}
+/** Only the two types that carry something more. A bad one is dropped whole. */
+function detailsOf(event) {
+    if (event.type === "custom") {
+        if (typeof event.name !== "string" || !EVENT_NAME.test(event.name)) {
+            return null;
+        }
+        return {
+            eventName: event.name,
+            eventProperties: propertiesOf(event.properties),
+        };
+    }
+    if (event.type === "outbound") {
+        const outboundUrl = outboundOf(event.url);
+        return outboundUrl === null ? null : { outboundUrl };
+    }
+    return {};
+}
 /**
  * Rebuilds each event from an allowlist, so unexpected fields are dropped. The
  * `browser:` action prefix keeps these out of request counts, because a page
@@ -61,6 +117,9 @@ export function toBrowserEvent(raw, actor, now, sampleRate) {
     const url = pathOf(event.path);
     if (url === null)
         return null;
+    const details = detailsOf(event);
+    if (details === null)
+        return null;
     const element = typeof event.element === "string" && ELEMENTS.has(event.element)
         ? `:${event.element}`
         : "";
@@ -80,6 +139,32 @@ export function toBrowserEvent(raw, actor, now, sampleRate) {
         durationMs: wholeNumber(event.durationMs, MAX_DURATION_MS),
         sampleRate,
         classifierVersion: BROWSER_CLASSIFIER_VERSION,
+        ...details,
+    };
+}
+/** Forged values can only mislabel the page that sent them. */
+function deviceOf(raw) {
+    if (typeof raw !== "object" || raw === null)
+        return undefined;
+    const { coarsePointer, mobile, model, platform, platformVersion, screen, touch } = raw;
+    const major = typeof platformVersion === "string"
+        ? /^\d{1,4}/.exec(platformVersion)?.[0]
+        : undefined;
+    return {
+        touch: typeof touch === "boolean" ? touch : undefined,
+        coarsePointer: typeof coarsePointer === "boolean" ? coarsePointer : undefined,
+        mobile: typeof mobile === "boolean" ? mobile : undefined,
+        screen: typeof screen === "string" && SCREENS.has(screen) ? screen : undefined,
+        platform: typeof platform === "string" && PLATFORM.test(platform)
+            ? platform
+            : undefined,
+        // The major version is enough, and the rest narrows down one machine.
+        platformVersion: major,
+        model: typeof model === "string" &&
+            model.length <= MAX_MODEL_CHARS &&
+            MODEL.test(model)
+            ? model
+            : undefined,
     };
 }
 export function parseBrowserBody(text) {
@@ -92,6 +177,7 @@ export function parseBrowserBody(text) {
         }
         return {
             automated: body.automated === true,
+            device: deviceOf(body.device),
             events: body.events,
         };
     }

package/dist/fetch.js

@@ -1,5 +1,7 @@
 import { SCRIPT_PATH } from "./browser.js";
-import { createScoutline, isDocumentRequest, serverTiming, } from "./index.js";
+import { usableIp } from "./client-ip.js";
+import { createScoutline } from "./index.js";
+import { isDocumentRequest, serverTiming } from "./request.js";
 const SCRIPT_TAG = `<script src="${SCRIPT_PATH}" defer></script>`;
 function forPage(response, eventId) {
     if (!response.headers.get("content-type")?.includes("text/html")) {
@@ -18,6 +20,12 @@ function forPage(response, eventId) {
     })
         .transform(page);
 }
+/** Only a real Worker has `cf`, so this header can't be faked there. */
+function workerIp(request) {
+    return request.cf
+        ? usableIp(request.headers.get("cf-connecting-ip"))
+        : undefined;
+}
 function contentLength(response) {
     const value = Number(response.headers.get("content-length"));
     return response.headers.has("content-length") && Number.isFinite(value)
@@ -39,7 +47,8 @@ export function withScoutline(handler, options) {
     const scoutline = createScoutline(options);
     return async function scoutlineFetch(request, ...args) {
         const context = args.find((arg) => typeof arg?.waitUntil === "function");
-        const own = scoutline.handle(request);
+        const clientIp = workerIp(request);
+        const own = scoutline.handle(request, clientIp);
         if (own) {
             const answer = await own;
             context?.waitUntil(scoutline.flush());
@@ -48,6 +57,7 @@ export function withScoutline(handler, options) {
         const started = Date.now();
         const finish = (status, bytes) => {
             const eventId = scoutline.observe(request, {
+                clientIp,
                 status,
                 bytes,
                 durationMs: Date.now() - started,

package/dist/index.d.ts

@@ -1,4 +1,6 @@
+import { type ConsentOptions } from "./browser-script.js";
 export { classify } from "./classify.js";
+export type { ConsentBanner, ConsentOptions } from "./browser-script.js";
 /**
  * What your app knows about a request that Scoutline can't see. Everything is
  * optional, and only what you return here leaves your server.
@@ -23,10 +25,21 @@ export type ScoutlineOptions = {
     /** Adds your own ids to a request. A failure here only leaves them out. */
     context?: (request: Request) => RequestContext | null | undefined;
     /**
-     * Sends the visitor's IP so the collector can find its country and network.
-     * The collector never stores it. Default true.
+     * Sends the visitor's IP so the collector can find its country and network,
+     * and verify crawlers. The collector never stores it. Default true.
      */
     sendClientIp?: boolean;
+    /**
+     * The header your own proxy sets to the visitor's IP. Set this on self-hosted
+     * sites, or crawlers can't be verified, because any client can fake the rest.
+     * Vercel and Cloudflare are detected automatically.
+     */
+    trustedIpHeader?: string;
+    /**
+     * Lets visitors who agree be recognised across days with a first-party cookie.
+     * Off by default, and without it this package never sets a cookie.
+     */
+    consent?: ConsentOptions;
 };
 export type Observation = {
     /** Set by adapters that must announce the id before the response finishes. */
@@ -53,32 +66,6 @@ export type Scoutline = {
      */
     handle(request: Request, clientIp?: string): Promise<Response> | null;
 };
-/** Only full page loads carry a server id for the browser script to read. */
-export declare function isDocumentRequest(request: Request): boolean;
-export declare function serverTiming(eventId: string): string;
-/** A proxy may list several hosts, nearest client first. */
-export declare function hostFromHeaders(headers: Headers): string | null;
-/** The site's hostname. */
-export declare function hostOf(request: Request): string | undefined;
-/**
- * What the collector needs to classify a request. Only whether these headers
- * exist is sent, never their values, and rules live there so they can change
- * without a new release of this package.
- */
-export declare function requestFacts(headers: Headers, options: {
-    sendClientIp?: boolean;
-    clientIp?: string;
-}): {
-    userAgent: string | undefined;
-    clientIp: string | undefined;
-    requestShape: {
-        clientHints: boolean;
-        fetchMetadata: boolean;
-        acceptLanguage: boolean;
-    };
-};
-/** The W3C trace id from `traceparent`, a claim until something validates it. */
-export declare function traceId(headers: Headers): string | undefined;
 /**
  * Failure here must never reach the customer's site. A missing key turns the
  * client into a no-op, and a full queue drops the oldest events.

package/dist/index.js

@@ -1,8 +1,8 @@
 import { EVENTS_PATH, isSameOrigin, parseBrowserBody, SCRIPT_PATH, toBrowserEvent, } from "./browser.js";
-import { BROWSER_SCRIPT } from "./browser-script.js";
+import { browserScript } from "./browser-script.js";
 import { classify, CLASSIFIER_VERSION } from "./classify.js";
-import { clientIpFromHeaders } from "./client-ip.js";
 import { DEFAULT_ENDPOINT, env } from "./config.js";
+import { hostOf, isPrefetch, limited, referrerHost, requestFacts, traceId, } from "./request.js";
 export { classify } from "./classify.js";
 const MAX_QUEUED = 500;
 const MAX_BATCH = 100;
@@ -11,9 +11,7 @@ const FLUSH_INTERVAL_MS = 5000;
 const HEARTBEAT_INTERVAL_MS = 5 * 60 * 1000;
 const REJECTED_BACKOFF_MS = 60 * 1000;
 const RATE_LIMITED_BACKOFF_MS = 60 * 1000;
-const MAX_FIELD_CHARS = 128;
 const MAX_URL_CHARS = 2048;
-const MAX_USER_AGENT_CHARS = 512;
 /** Caps browser events per server instance, so a page can't flood the queue. */
 const MAX_BROWSER_EVENTS_PER_MINUTE = 2000;
 const SCRIPT_CACHE_SECONDS = 3600;
@@ -24,17 +22,8 @@ const FRAMEWORK_PATH = /^\/(?:_next|_vercel|\.well-known\/vercel)\//;
  * to serve, so they stay for anyone identified. PDFs always stay.
  */
 const ASSET_PATH = /\.(?:css|js|mjs|map|png|jpe?g|gif|webp|avif|svg|ico|woff2?|ttf|otf|mp4|m4v|webm|mov|ogv|mp3|m4a|aac|flac|wav|ogg|opus)$/i;
-/** Only full page loads carry a server id for the browser script to read. */
-export function isDocumentRequest(request) {
-    return (request.method === "GET" &&
-        request.headers.get("accept")?.includes("text/html") === true &&
-        !isPrefetch(request.headers));
-}
-export function serverTiming(eventId) {
-    return `scoutline;desc="${eventId}"`;
-}
-function scriptResponse() {
-    return new Response(BROWSER_SCRIPT, {
+function scriptResponse(consent) {
+    return new Response(browserScript(consent), {
         headers: {
             "Content-Type": "text/javascript; charset=utf-8",
             "Cache-Control": `public, max-age=${SCRIPT_CACHE_SECONDS}`,
@@ -48,9 +37,6 @@ function ownRoute(request) {
     const { pathname } = new URL(request.url);
     return pathname === SCRIPT_PATH || pathname === EVENTS_PATH ? pathname : null;
 }
-function limited(value, max = MAX_FIELD_CHARS) {
-    return value && value.length <= max ? value : undefined;
-}
 function interfaceFor(pathname, headers) {
     if (pathname === "/mcp" || pathname.startsWith("/mcp/"))
         return "tool";
@@ -58,43 +44,6 @@ function interfaceFor(pathname, headers) {
         return "api";
     return headers.get("accept")?.includes("application/json") ? "api" : "web";
 }
-/** A proxy may list several hosts, nearest client first. */
-export function hostFromHeaders(headers) {
-    const forwarded = headers.get("x-forwarded-host")?.split(",")[0];
-    return forwarded?.trim() || headers.get("host") || null;
-}
-/** The site's hostname. */
-export function hostOf(request) {
-    const host = hostFromHeaders(request.headers) ?? new URL(request.url).host;
-    return limited(host.toLowerCase(), 255);
-}
-/**
- * What the collector needs to classify a request. Only whether these headers
- * exist is sent, never their values, and rules live there so they can change
- * without a new release of this package.
- */
-export function requestFacts(headers, options) {
-    return {
-        userAgent: limited(headers.get("user-agent"), MAX_USER_AGENT_CHARS),
-        clientIp: options.sendClientIp === false
-            ? undefined
-            : (options.clientIp ?? clientIpFromHeaders(headers)),
-        requestShape: {
-            clientHints: headers.has("sec-ch-ua"),
-            fetchMetadata: headers.has("sec-fetch-mode") || headers.has("sec-fetch-site"),
-            acceptLanguage: headers.has("accept-language"),
-        },
-    };
-}
-function isPrefetch(headers) {
-    return (headers.has("next-router-prefetch") ||
-        headers.get("purpose") === "prefetch" ||
-        headers.get("sec-purpose")?.includes("prefetch") === true);
-}
-/** The W3C trace id from `traceparent`, a claim until something validates it. */
-export function traceId(headers) {
-    return limited(headers.get("traceparent")?.split("-")[1] ?? null);
-}
 function contextOf(request, read) {
     try {
         return read?.(request) ?? {};
@@ -126,10 +75,8 @@ function toEvent(request, observation, sampleUnknown, options) {
         // The pathname only: queries, cookies, headers and bodies never leave here.
         url: pathname.slice(0, MAX_URL_CHARS),
         host: hostOf(request),
-        ...requestFacts(headers, {
-            sendClientIp: options.sendClientIp,
-            clientIp: observation.clientIp,
-        }),
+        referrer: referrerHost(headers),
+        ...requestFacts(headers, { ...options, clientIp: observation.clientIp }),
         interface: interfaceFor(pathname, headers),
         requestId: limited(headers.get("x-vercel-id") ??
             headers.get("cf-ray") ??
@@ -165,7 +112,7 @@ export function createScoutline(options = {}) {
             handle(request) {
                 const route = ownRoute(request);
                 if (route === SCRIPT_PATH)
-                    return Promise.resolve(scriptResponse());
+                    return Promise.resolve(scriptResponse(options.consent));
                 return route ? Promise.resolve(emptyResponse(204)) : null;
             },
         };
@@ -299,14 +246,18 @@ export function createScoutline(options = {}) {
             if (sampled && Math.random() >= sampleUnknown)
                 return emptyResponse(204);
             const now = Date.now();
-            const facts = requestFacts(request.headers, {
-                sendClientIp: options.sendClientIp,
-                clientIp,
-            });
+            const facts = requestFacts(request.headers, { ...options, clientIp });
             for (const raw of body.events) {
                 const event = toBrowserEvent(raw, actor, now, sampled ? sampleUnknown : 1);
-                if (event)
-                    enqueue({ ...event, ...facts, host: hostOf(request) });
+                if (!event)
+                    continue;
+                const device = event.action === "browser:page_view" ? body.device : null;
+                enqueue({
+                    ...event,
+                    ...facts,
+                    host: hostOf(request),
+                    device: device ?? undefined,
+                });
             }
             return emptyResponse(204);
         }
@@ -317,7 +268,7 @@ export function createScoutline(options = {}) {
     function handle(request, clientIp) {
         const route = ownRoute(request);
         if (route === SCRIPT_PATH)
-            return Promise.resolve(scriptResponse());
+            return Promise.resolve(scriptResponse(options.consent));
         return route ? acceptBrowserEvents(request, clientIp) : null;
     }
     return { observe, flush, handle };

package/dist/next.js

@@ -1,4 +1,5 @@
-import { createScoutline, isDocumentRequest, serverTiming, } from "./index.js";
+import { createScoutline } from "./index.js";
+import { isDocumentRequest, serverTiming } from "./request.js";
 /**
  * Wraps a Next.js `proxy.ts`. It adds no latency, because sending happens in
  * `waitUntil` after the response is on its way.

package/dist/node.js

@@ -1,6 +1,7 @@
 import { EVENTS_PATH, SCRIPT_PATH } from "./browser.js";
 import { usableIp } from "./client-ip.js";
-import { createScoutline, isDocumentRequest, serverTiming, } from "./index.js";
+import { createScoutline, } from "./index.js";
+import { isDocumentRequest, serverTiming } from "./request.js";
 const MAX_BODY_BYTES = 32 * 1024;
 function toRequest(request, body) {
     const headers = new Headers();

package/dist/outcomes.d.ts

@@ -8,6 +8,8 @@ export type OutcomeOptions = {
      * and network. The collector never stores it. Default true.
      */
     sendClientIp?: boolean;
+    /** The header your own proxy sets to the visitor's IP. */
+    trustedIpHeader?: string;
 };
 export type EventDetails = {
     /**

package/dist/outcomes.js

@@ -1,6 +1,6 @@
 import { classify, CLASSIFIER_VERSION } from "./classify.js";
 import { DEFAULT_ENDPOINT, env } from "./config.js";
-import { hostFromHeaders, requestFacts, traceId } from "./index.js";
+import { hostFromHeaders, requestFacts, traceId } from "./request.js";
 const SEND_TIMEOUT_MS = 2000;
 const MAX_ID_CHARS = 128;
 function toHeaders(headers) {
@@ -87,7 +87,8 @@ export function createOutcomes(options = {}) {
                 family,
                 outcomeKey,
                 actor: classify(userAgent),
-                ...requestFacts(headers, { sendClientIp: options.sendClientIp }),
+                // A site with consent on has this cookie, and it must match their pages.
+                ...requestFacts(headers, { ...options, consent: true }),
                 host: hostFromHeaders(headers)?.toLowerCase() ?? undefined,
                 interface: "app",
                 traceId: traceId(headers),

package/dist/request.d.ts

@@ -0,0 +1,55 @@
+export declare function limited(value: string | null, max?: number): string | undefined;
+export declare function isPrefetch(headers: Headers): boolean;
+/** Only full page loads carry a server id for the browser script to read. */
+export declare function isDocumentRequest(request: Request): boolean;
+export declare function serverTiming(eventId: string): string;
+/** A proxy may list several hosts, nearest client first. */
+export declare function hostFromHeaders(headers: Headers): string | null;
+/** The site's hostname. */
+export declare function hostOf(request: Request): string | undefined;
+/** The site a visitor came from. Its path and query can carry anything. */
+export declare function referrerHost(headers: Headers): string | undefined;
+/** The W3C trace id from `traceparent`, a claim until something validates it. */
+export declare function traceId(headers: Headers): string | undefined;
+/**
+ * What the collector needs to classify a request and name its device. The
+ * three Client Hints are the browser's own low-detail summary. For the others,
+ * only whether a header exists is sent. Rules live on the collector, so they can
+ * change without a new release of this package.
+ */
+export declare function requestFacts(headers: Headers, options: {
+    sendClientIp?: boolean;
+    clientIp?: string;
+    trustedIpHeader?: string;
+    consent?: unknown;
+}): {
+    clientHints: {
+        brands: string | undefined;
+        mobile: string | undefined;
+        platform: string | undefined;
+    };
+    requestShape: {
+        clientHints: boolean;
+        fetchMetadata: boolean;
+        acceptLanguage: boolean;
+    };
+    clientIp: string;
+    clientIpSource: string;
+    userAgent: string | undefined;
+    visitorId: string | undefined;
+} | {
+    clientHints: {
+        brands: string | undefined;
+        mobile: string | undefined;
+        platform: string | undefined;
+    };
+    requestShape: {
+        clientHints: boolean;
+        fetchMetadata: boolean;
+        acceptLanguage: boolean;
+    };
+    clientIp?: undefined;
+    clientIpSource?: undefined;
+    userAgent: string | undefined;
+    visitorId: string | undefined;
+};

package/dist/request.js

@@ -0,0 +1,103 @@
+import { clientIpFromHeaders, usableIp } from "./client-ip.js";
+import { env } from "./config.js";
+const MAX_FIELD_CHARS = 128;
+const MAX_USER_AGENT_CHARS = 512;
+const MAX_HINT_CHARS = 256;
+const VISITOR_COOKIE = /(?:^|;\s*)_scoutline_id=([0-9a-f-]{36})(?:;|$)/i;
+const CONSENT_COOKIE = /(?:^|;\s*)_scoutline_consent=yes(?:;|$)/;
+export function limited(value, max = MAX_FIELD_CHARS) {
+    return value && value.length <= max ? value : undefined;
+}
+export function isPrefetch(headers) {
+    return (headers.has("next-router-prefetch") ||
+        headers.get("purpose") === "prefetch" ||
+        headers.get("sec-purpose")?.includes("prefetch") === true);
+}
+/** Only full page loads carry a server id for the browser script to read. */
+export function isDocumentRequest(request) {
+    return (request.method === "GET" &&
+        request.headers.get("accept")?.includes("text/html") === true &&
+        !isPrefetch(request.headers));
+}
+export function serverTiming(eventId) {
+    return `scoutline;desc="${eventId}"`;
+}
+/** A proxy may list several hosts, nearest client first. */
+export function hostFromHeaders(headers) {
+    const forwarded = headers.get("x-forwarded-host")?.split(",")[0];
+    return forwarded?.trim() || headers.get("host") || null;
+}
+/** The site's hostname. */
+export function hostOf(request) {
+    const host = hostFromHeaders(request.headers) ?? new URL(request.url).host;
+    return limited(host.toLowerCase(), 255);
+}
+/** The site a visitor came from. Its path and query can carry anything. */
+export function referrerHost(headers) {
+    try {
+        const referer = headers.get("referer");
+        return referer ? limited(new URL(referer).hostname.toLowerCase(), 255) : undefined;
+    }
+    catch {
+        return undefined;
+    }
+}
+/** The W3C trace id from `traceparent`, a claim until something validates it. */
+export function traceId(headers) {
+    return limited(headers.get("traceparent")?.split("-")[1] ?? null);
+}
+/**
+ * Says how the address was learned, because only the collector decides what that
+ * proves. A forged header must never earn a crawler `verified`, so anything that
+ * isn't set by the connection, the platform or the site's own proxy is `header`.
+ */
+function clientAddress(headers, options) {
+    if (options.clientIp) {
+        return { clientIp: options.clientIp, clientIpSource: "connection" };
+    }
+    const configured = options.trustedIpHeader
+        ? usableIp(headers.get(options.trustedIpHeader)?.split(",")[0])
+        : undefined;
+    if (configured)
+        return { clientIp: configured, clientIpSource: "configured" };
+    const platform = env("VERCEL") ? usableIp(headers.get("x-real-ip")) : undefined;
+    if (platform)
+        return { clientIp: platform, clientIpSource: "platform" };
+    const claimed = clientIpFromHeaders(headers);
+    return claimed ? { clientIp: claimed, clientIpSource: "header" } : {};
+}
+/**
+ * Only for sites that turned consent on, and only Scoutline's own id, and only
+ * while the visitor's choice says yes. No other cookie is ever read.
+ */
+function consentedVisitor(headers, consent) {
+    if (!consent)
+        return undefined;
+    const cookies = headers.get("cookie") ?? "";
+    if (!CONSENT_COOKIE.test(cookies))
+        return undefined;
+    return VISITOR_COOKIE.exec(cookies)?.[1];
+}
+/**
+ * What the collector needs to classify a request and name its device. The
+ * three Client Hints are the browser's own low-detail summary. For the others,
+ * only whether a header exists is sent. Rules live on the collector, so they can
+ * change without a new release of this package.
+ */
+export function requestFacts(headers, options) {
+    return {
+        userAgent: limited(headers.get("user-agent"), MAX_USER_AGENT_CHARS),
+        visitorId: consentedVisitor(headers, options.consent),
+        ...(options.sendClientIp === false ? {} : clientAddress(headers, options)),
+        clientHints: {
+            brands: limited(headers.get("sec-ch-ua"), MAX_HINT_CHARS),
+            mobile: limited(headers.get("sec-ch-ua-mobile"), MAX_HINT_CHARS),
+            platform: limited(headers.get("sec-ch-ua-platform"), MAX_HINT_CHARS),
+        },
+        requestShape: {
+            clientHints: headers.has("sec-ch-ua"),
+            fetchMetadata: headers.has("sec-fetch-mode") || headers.has("sec-fetch-site"),
+            acceptLanguage: headers.has("accept-language"),
+        },
+    };
+}

package/package.json

@@ -1,6 +1,7 @@
 {
   "name": "@scoutline-ai/middleware",
-  "version": "0.2.0",
+  "version": "0.4.0",
+  "license": "MIT",
   "description": "In-path agent observation for Scoutline. Fails open and adds no work to the hot path.",
   "type": "module",
   "files": [