@thg-agentic/hyper-personalisation-web 3.17.1 → 3.19.0

package/README.md

@@ -1,7 +1,7 @@
 # @thg-agentic/hyper-personalisation-web
 
 The browser SDK for THG Hyper-Personalisation, a customer data platform. It implements the event
-specification (spec 1.6, stamped on every event as `context.spec.version`) for any storefront: it
+specification (spec 1.10, stamped on every event as `context.spec.version`) for any storefront: it
 sends events in the Segment wire format, applies consent strictly whichever CMP the site uses, reads
 the GA4 `dataLayer` the site already pushes to, and has no runtime dependencies.
 
@@ -37,8 +37,8 @@ const hp = await bootstrap(
 organisation, the property and test or live, so there is no host, property or switch to set per
 environment, as with Stripe's or LaunchDarkly's keys. `apiHost` is optional, only for a first-party
 proxy (a bare host or an `https://` origin); without it the SDK uses the HP API
-(`DEFAULT_API_HOST`; in 3.17 an interim nonprod host for test keys only, see
-[Default host](#default-host-until-production-is-live)). Everything tenant-level comes from
+(`DEFAULT_API_HOST`, `https://hp.thgingenuity.com` since 3.18, for test and live keys alike; see
+[Default host](#default-host-by-release)). Everything tenant-level comes from
 `GET {apiHost}/v1/projects/{writeKey}/settings`, so switching a site on or off, or changing what it
 collects, is a settings change and never a deploy:
 
@@ -51,7 +51,9 @@ collects, is a settings change and never a deploy:
 | `queryParamAllowlist` | Query parameters kept on page URLs (plus campaign parameters). |
 | `anonymousId` | Where device ids come from, chosen per source in the console (since 3.6): `{ source: 'site_cookie', cookie }` or `{ source: 'hp_cookie' }`. See [Device ids](#device-ids). |
 | `apiHost` | Where events go, when not the host the settings came from. |
-| `plan.track` | Segment's tracking plan: an event with `enabled: false` is dropped at the source. |
+| `plan.track` | Segment's tracking plan: an event with `enabled: false` is dropped at the source. The console's blocked events arrive here. |
+| `sampleRate` | The share of visitors events are sent for, by a hash of the anonymous id (since 3.18). Absent: 1. |
+| `debug` | `{ enabled, tokenSha256 }`: console logging for everyone, or for a session opened with `?hp_debug=<token>` whose SHA-256 is `tokenSha256` (since 3.18). `?hp_debug=` ends it. |
 
 **The page configures only what the page knows** (the second argument):
 
@@ -68,7 +70,7 @@ collects, is a settings change and never a deploy:
 | `propertyId` | Only for a key scoped to several properties, whose settings name none. |
 | `consentCookieDomain` | `Domain` for the consent cookie, e.g. `.acme.com` to share it with checkout. |
 | `storageNames` | Rename what the SDK stores; see [Storage](#storage). |
-| `debug` | Log every event and delivery problem. |
+| `debug` | Log every event and delivery problem. Also on when the settings' `debug` says so. |
 | `settings` | Development and tests only: use this settings response, never fetch or cache one. |
 | `extend` | `createAnalytics` options that are functions, such as `getRecommendations` or `getExtensions`. |
 
@@ -79,14 +81,15 @@ What `bootstrap` does:
   every problem. It never rejects, and a second call on the same page returns the first client.
 - **Settings.** The last good answer is cached in `localStorage`, only once analytics is granted,
   and forgotten on a refusal. A page with a cache starts from it at once and asks the API again for
-  the next page; a first visit waits for the API (which answers `Cache-Control: max-age=60,
-  stale-while-revalidate=600`). A key the API switches off clears the cache, so the next page is
+  the next page; a first visit waits for the API (which answers `Cache-Control: max-age=10`
+  and an `ETag`, so the browser revalidates for a `304`). A key the API switches off clears the cache, so the next page is
   off too.
 - **Consent**, identity, one page view, the dataLayer bridge and a once-per-session `identify`, as
   configured.
 - **Never waited for.** Call `hp` (`@thg-agentic/hyper-personalisation-web/hp`) from anywhere, at
   any time, and never await or flush it: calls made before the client starts wait, each with the
-  messageId and time it was made at, and are sent in order once it has. A page that navigates
+  messageId and time it was made at, and are sent in order once it has. The page view is stamped
+  when the page was navigated to, so none of them precedes it (3.18). A page that navigates
   first (a sign-in that redirects at once) carries them in sessionStorage, and the next page's
   `bootstrap` sends them once, before its own page view (since 3.9). So does a page whose settings
   request never answered: the API down, or the request cancelled by the navigation (3.9.1). The carry is a device write,
@@ -175,6 +178,13 @@ and `productAdded` (your search engine's id for the result set, such as Algolia'
 `newsletterSubscribed`. A search page sends `productsSearched` with its results and never a
 `productListViewed` for the same grid.
 
+Since 3.19 (spec 1.12), a search box's suggestions are their own event:
+
+| Method | Event | Notes |
+|---|---|---|
+| `productsSearched` | Products Searched | A submitted search and the results page it produced: `query`, the results in `products`, and `pagination` with `total_results` |
+| `searchSuggestionsViewed` | Search Suggestions Viewed | The products a search box suggested for a partly typed `query`, sent once per suggestion set with the products seen (`position` in the suggestions). Give each set its own `search_request_id`, and a click on one of its products carries that id on `productClicked`. Never `productsSearched`: suggestions and results are ranked by different models and are measured apart |
+
 ## Options
 
 | Option | Default | |
@@ -187,16 +197,17 @@ and `productAdded` (your search engine's id for the result set, such as Algolia'
 | `shipToCountry`, `getShipToCountry` | `null` | ISO-3166 alpha-2 destination, per event. |
 | `locale`, `getLocale` | `null` | The storefront locale served (BCP-47), per event. |
 | `country` | `null` | Visitor country; the ingest derives it at the edge and overwrites this. |
-| `consent` | none | A consent adapter (see below). Makes the SDK strict. |
-| `consentTimeoutMs` | `5000` | How long events wait for the CMP before they are dropped. |
+| `consent` | none | A consent adapter (see below). Makes the SDK strict. Without one the SDK sends under your lawful basis (`basis: 'host'`) and says so once in the console (3.19). |
+| `consentTimeoutMs` | `5000` | How long the CMP has to answer before the status is `unknown`. Events stay held, in memory, until it answers. |
 | `consentCookieDomain` | host only | `Domain` for the consent cookie. |
 | `onConsentStatus` | none | Called with the gate's status: `granted`, `denied` or `unknown` (timed out). |
 | `storageNames` | `thg_hp_*` | Rename what the SDK stores; see [Storage](#storage). |
 | `plan` | none | A Segment tracking plan, `{ track: { [event]: { enabled } } }`; `__default` covers unlisted events. |
+| `sampleRate` | `1` | The share of visitors to send for, by a hash of the anonymous id salted `hp_sample`; a visitor with no anonymous id is sent for. A consent withdrawal is always sent. Since 3.18. |
 | `queryParamAllowlist` | none | Keep only these query params (plus campaign params) in `context.page`. Patterns are anchored and case-insensitive. |
 | `queryParamBlocklist` | `[]` | Strip these params. Ignored when an allowlist is given. |
 | `sensitiveParamAllowlist` | `[]` | Whatever else is configured, a param is always dropped from every reported URL when its name contains `email`, `token`, `password`, `secret`, `auth`, `session`, `phone`, `address`, `card`, `customerid` and the like (see `SENSITIVE_PARAM_FRAGMENTS` in `src/context.js`), or when its value is an email address or a card-length number — campaign params and the referrer included. This exempts names from the first rule, for a false positive such as an `author` filter; nothing exempts a value. |
-| `getAnonymousId`, `getUserId`, `getSessionId`, `getRenderId` | none | Identity accessors. Without `getSessionId` the SDK keeps its own session (see [Session ids](#session-ids)); without `hp_cookie` it never mints a device id. |
+| `getAnonymousId`, `getUserId`, `getSessionId`, `getRenderId` | none | Identity accessors. Without `getSessionId` the SDK keeps its own session (see [Session ids](#session-ids)); a missing or empty device id gives way to the SDK's own (see [Device ids](#device-ids)). |
 | `getExperiments`, `getAudiences`, `getDecisions`, `getRecommendations`, `getExtensions` | none | Read per event into the matching `context` field. Without `getDecisions`, `context.decisions` is what `recordExposure` last recorded per slot. |
 | `pageType` | `null` | `context.page.type` before the first `page()` call. |
 | `onEvent` | none | Observer for every event that will be sent. |
@@ -207,8 +218,36 @@ and `productAdded` (your search engine's id for the result set, such as Algolia'
 The client has one method per browser event in the spec, plus `track`, `page`, `identify`,
 `reset`, `signedOut`, `flush`, `servingHeaders`, and the read-only `consent` and
 `lastPageMessageId`. Every method takes an optional last argument of context overrides, either as
-the overrides or as `{ context }`. An event method returns the event, or null when the tracking
-plan disables it. Declarations are in `types/`, and event properties are typed from the registry.
+the overrides or as `{ context }`. An event method returns the event, or null when it is not sent
+(the tracking plan disables it, the visitor is outside the sample, or `beforeSend` drops it).
+Declarations are in `types/`, and event properties are typed from the registry.
+
+- **No method throws into your page** (3.19). A call that fails, whatever it was passed, logs a
+  warning and answers what a client that cannot decide answers, failing closed: an event method
+  null, `inHoldout()` true, `personalisationGroup()` `'excluded'`, `servingHeaders()` and
+  `recommendationContext()` `{}`, `flush()` a resolved promise. The one exception is
+  `throwOnInvalid`, which you ask for in development.
+- **One event is at most 55 KB** of JSON (3.19), so that it can always travel in a keepalive request
+  at unload. A bigger one is not sent and its method returns null; with `debug` the SDK logs it.
+  Segment's limit is 32 KB; this one is higher so that a long product list still fits.
+- **A sign-out forgets the visit** (3.19). `signedOut()`, or `reset()` while a user is signed in,
+  also forgets the SDK's session, the decisions in force and the recommendation context, so the
+  next person on a shared device inherits none of them. `reset()` with nobody signed in changes
+  nothing else, so a host that calls it on every page keeps its session. The SDK's own device id
+  starts afresh too, as Segment's `analytics.reset()` does; a device id from the site's own cookie
+  (`getAnonymousId`, or a `site_cookie` source) stays, because the site owns it. On one page,
+  `identify` with a different `userId` than an earlier `identify` named is a sign-out first; the
+  same user again, a first `identify`, a `getUserId` cookie or bootstrap's `identity` option of
+  another form changes nothing (with `debug`, a mismatch with `identity` is logged). Across
+  pages, call `signedOut()` or `reset()` at the sign-out.
+- **A refusal forgets the visit at once** (3.19): the kept recommendation context and the session
+  signals go when consent is refused or personalisation withdrawn, even if `beforeSend` drops the
+  `Consent Updated` event. Before the client starts, `hp.recommendationContext()` reads the kept
+  visit only when the consent cookie grants personalisation.
+- **Retries wait out `Retry-After`** on a 429 or 5xx (3.19), as well as backing off exponentially
+  with jitter, up to 30 seconds. The browser lets the SDK read the header only from a first-party
+  proxy for now: cross-origin it needs `Access-Control-Expose-Headers: Retry-After`, which the HP
+  ingest does not send yet. Without it the backoff alone applies.
 
 ## Calling the serving API
 
@@ -361,6 +400,19 @@ and put nothing in force: the trader is not a shopper, so must neither train a s
 readout. Every other event is sent as usual. `?hp_preview=` with no value ends the preview. The
 SDK never checks the token: serving does, and ignores one that has expired.
 
+### Visit traits
+
+A session audience's rules name fields the SDK keeps for the visit, from the events it sends:
+`productViews`, `listViews`, `cartAdds`, `searches`, `browseViews.<path>`, `browseShare.<path>`,
+`categoryViews.<path>`, `categoryShare.<path>` and, since 3.18, `basketValue`. A field this build
+does not know fails its rule closed.
+
+`basketValue` is the `value` of the last `cartUpdated` (Cart Updated), in that event's currency:
+the basket as your platform priced it. It is never summed from `productAdded` lines, whose prices
+can include tax where `value` does not. It is `null`, so every rule on it fails, until the visit
+sends a Cart Updated, and again after one without a `value`. Send `cartUpdated` after every confirmed basket change, as the spec asks, or
+a basket audience goes stale.
+
 ### Audiences only some integrations may target
 
 Since 3.8, only audiences marked Targetable in the console are membership. Pass the membership
@@ -378,7 +430,8 @@ them, so `context.audiences` and exposures carry targetable audiences only. Befo
 audience matches nobody, and a shipped untargetable visit audience the visitor meets may be reported
 as membership. Every other audience resolves as it did.
 
-Social proof has a client of its own, which batches, caches and never throws:
+Social proof has a client of its own, which batches and caches, and never throws once created
+(creating it without a `writeKey` or with an invalid `apiHost` throws):
 
 ```js
 const proof = createSocialProof({ writeKey, getConsent: () => hp.consent });
@@ -397,8 +450,9 @@ With a `consent` adapter the SDK is strict:
 - **Analytics granted:** held events are sent, each carrying the answer in `context.consent`.
 - **Analytics refused:** held events are dropped, and so is everything after. Nothing goes into
   local or session storage.
-- **No answer by the timeout:** held and later events are dropped until an answer arrives. Strict
-  is the only mode: an unanswered CMP is never a grant, however long it takes.
+- **No answer by the timeout:** the status is `unknown`, and held and later events stay held, in
+  memory, while the SDK keeps listening for the answer. Strict is the only mode: an unanswered CMP
+  is never a grant, however long it takes.
 - **Page closed before an answer:** nothing is sent. Switching to another tab is not closing the
   page: held events wait for the visitor to come back.
 - **A navigation before an answer** (a sign-in that redirects) loses what the page held, unless
@@ -482,6 +536,22 @@ cookie with `storageNames.consent` gets the same name read back by `"forwarded"`
 pass the same `storageNames`, or passes `forwardedConsent({ name })` directly. No cookie, or no analytics answer in it,
 means not answered, and strict mode then drops the events.
 
+A checkout on **another registrable domain** never receives that cookie. Carry it in the checkout
+handoff your servers already make:
+
+1. The storefront's server reads `thg_hp_consent` from the shopper's request and puts it on the
+   checkout session it creates.
+2. On every page it renders, checkout's server sets `thg_hp_consent_handoff` on its own host
+   from the session. The value is `formatConsentCookie(parseConsentCookie(value))` (from
+   `@thg-agentic/hyper-personalisation-web/consent`), so an unreadable value is unknown, never a
+   grant. The cookie is not `HttpOnly`. With no value it deletes the cookie.
+3. The checkout page boots with `storageNames: { consent: 'thg_hp_consent_handoff' }` and
+   `"forwarded"`. Use a name of its own: a checkout host that also serves a storefront on its
+   own domain sees that storefront's `thg_hp_consent` first, and a delete cannot remove it.
+
+Hand the device cookie over the same way if checkout's events should land on the storefront's
+visitor. Never put the value on the checkout URL: it would end up in access logs and referrers.
+
 ## GA4 dataLayer
 
 ```js
@@ -630,8 +700,11 @@ delivery differs.
   removed when the ingest accepts it (or rejects it for good), so an outage costs a delay, not the
   events. The buffer is per tenant and capped at 20 batches and 1 MB, oldest dropped first.
 - On page unload, sends by `sendBeacon` in chunks that fit.
-- Size: the CDN script, which is `bootstrap` with every consent adapter and the dataLayer bridge,
-  is 13.4 KB gzip (38.4 KB minified). A test fails it past 15 KB.
+- Never throws into the page. An event JSON cannot serialise (a BigInt, a circular object, a
+  throwing `toJSON`) is dropped on its own, logged under `debug`, and the queue behind it carries
+  on. A `Date` in properties is sent as its ISO string.
+- Size: the CDN script is `bootstrap` with every consent adapter and the dataLayer bridge. A test
+  fails it past 90 KB gzip, and the inline snippet past 1 KB; the budget catches a runaway bundle.
 
 **analytics-next plugins** (`cdpPlugins` from `./segment`)
 - For sites already using `@segment/analytics-next`: keep it, and add the spec on top of it.
@@ -687,7 +760,7 @@ to Segment.io after a withdrawal.
 
 ## CDN script
 
-For a site without a bundler, `dist/hp.min.js` is `bootstrap` as one script (13.4 KB gzip), and
+For a site without a bundler, `dist/hp.min.js` is `bootstrap` as one script, and
 `dist/snippet.min.js` is the loader to paste inline in `<head>`, Segment-style: it puts a stub on
 `window.hp` that queues every call, loads the script asynchronously, and the script drains the
 queue into the client once the settings have answered.
@@ -732,6 +805,7 @@ SDK per page.
 | `anonymousId` | `thg_hp_anonymous_id` | cookie, one per browser | Hyper-Personalisation's device id, for a source set to `hp_cookie`; only under a grant (see [Device ids](#device-ids)) |
 | `session` | `thg_hp_session` | cookie, one per browser | the session id the SDK keeps when you name none, 30 minutes from the last event; only under a grant (3.15, see [Session ids](#session-ids)) |
 | `preview` | `thg_hp_preview` | session cookie | a trader's preview token from `?hp_preview=` (3.15, see [Preview](#preview)) |
+| `debug` | `thg_hp_debug` | sessionStorage | a developer's debug token from `?hp_debug=` (3.18), checked against the settings on every page |
 | — | `thg_hp_conformance_page_type` | sessionStorage | the previous page's type, only while the development-only conformance checker runs |
 
 Keys are scoped by property because one origin can serve several tenants. The names the SDK used
@@ -742,12 +816,12 @@ moved to the new names once, on the first page that loads 3.x, and deleted. A ba
 ### Device ids
 
 Each web source takes its device ids (`anonymousId`) from one place, chosen in the console and
-served in the settings as `hp.anonymousId`. There is never a fallback from one to the other: one
-browser with ids from two places would split into two visitors, and nothing could say why.
+served in the settings as `hp.anonymousId`. Since 3.18 an event never leaves without one: when the
+site's id is missing, empty or a placeholder, the SDK's own cookie stands in, as analytics.js does.
 
 | `hp.anonymousId` | The device id | |
 |---|---|---|
-| `{ source: 'site_cookie', cookie: 'site_visitor_id' }` | your own first-party cookie, by the name the console gives | read only, never minted; a visitor without it is sent with no `anonymousId` and `identity_trust: 'none'` |
+| `{ source: 'site_cookie', cookie: 'site_visitor_id' }` | your own first-party cookie, by the name the console gives | read only; a visitor without it gets the SDK's `thg_hp_anonymous_id` (3.18; before, no `anonymousId`) |
 | `{ source: 'site_cookie', cookie: null }` | the cookie you name in `identityCookies.anonymousId` (or return from `getAnonymousId`) | as above. Every web source that existed before the setting is set to this |
 | `{ source: 'hp_cookie' }` | Hyper-Personalisation's first-party cookie, `thg_hp_anonymous_id` | minted by the SDK; your own device cookie is ignored. The default for a new source |
 | absent (settings from before 3.6) | as `site_cookie` with `cookie: null` | |
@@ -757,8 +831,11 @@ The `hp_cookie` cookie is one per browser, not per property. It is written with
 `Max-Age` is 400 days, and it is written again on every page under a grant, so it lasts 400 days
 from the last visit. With a consent adapter it is never written before analytics is granted: until
 then the id lives in memory, and in the held store with `holdAcrossPages`, so a redirect keeps it.
-A refusal or withdrawal deletes it. Without a consent adapter you have decided the lawful basis
-yourself, so it is written at once.
+A refusal or withdrawal deletes it, and the id with it: a later grant, on the same page or another,
+starts a new device, as Segment's `analytics.reset()` does in analytics.js (3.18). A sign-out
+(`signedOut()`, or `reset()` with a user signed in) deletes it too, and the next event mints a new
+one (3.19). Without a consent adapter you have decided the lawful basis yourself, so it is written
+at once.
 
 Changing a source's choice gives every returning visitor a new device id. A device id 3.5 minted
 into localStorage (`thg_hp_anonymous_id:<property>`) becomes the cookie's value under a grant, and
@@ -769,9 +846,8 @@ handed the same one.
 ### Session ids
 
 Every event carries `context.session_id`. Name your own session cookie in
-`identityCookies.sessionId`, or pass `getSessionId`, and the SDK only reads it: it is then the
-only source, even for a visitor who lacks the cookie. Without either, since 3.15, the SDK keeps
-its own, as Amplitude and GA4 do: a random UUID in the `thg_hp_session` cookie, ended by 30 minutes
+`identityCookies.sessionId`, or pass `getSessionId`, and the SDK reads it. Without either (since
+3.15), or while it is missing or empty (since 3.18), the SDK keeps its own, as Amplitude and GA4 do: a random UUID in the `thg_hp_session` cookie, ended by 30 minutes
 without an event. Its `Max-Age` is 30 minutes and every event writes it again, so it is shared by
 the site's tabs and ends on its own. It is written with the attributes of the device cookie. With a
 consent adapter it is never written before analytics is granted (until then the session lives in
@@ -781,24 +857,21 @@ memory), and a refusal deletes it. Before 3.15 such a site sent `session_id: nul
 
 The SDK makes requests only to `apiHost` (the settings, and the events unless the settings name
 another host), so allow it in `connect-src`: `resolveApiHost(apiHost)` is that origin, the HP API
-(`DEFAULT_API_HOST`) when you set none. Since 3.17 that is `https://hp.uat.thgingenuity.com`
-(`connect-src https://hp.uat.thgingenuity.com`); 3.16 and earlier used a staging host that is
-being retired. Allow the CDN host in `script-src` too if you use the script. Call the API host directly rather than through a CDN proxy route: behind a proxy the ingest
+(`DEFAULT_API_HOST`) when you set none. Since 3.18 that is `https://hp.thgingenuity.com`
+(`connect-src https://hp.thgingenuity.com`); 3.17 used HP's nonprod host and 3.16 and earlier a
+staging host that is being retired. Allow the CDN host in `script-src` too if you use the script. Call the API host directly rather than through a CDN proxy route: behind a proxy the ingest
 would see the CDN's IP and country instead of the shopper's.
 
-### Default host until production is live
+### Default host by release
 
 | Release | `DEFAULT_API_HOST` | For |
 |---|---|---|
 | 3.12 to 3.16 | A staging host | Being retired. Upgrade. |
-| 3.17 | `https://hp.uat.thgingenuity.com` | Interim: HP's nonprod stack. Test keys, UAT and previews only. |
-| A later release | `https://hp.thgingenuity.com` | Production, for test and live keys alike. |
-
-`hp.uat.thgingenuity.com` is the interim nonprod host, for test keys only, and is not open to
-shoppers on the internet: where it cannot be reached the SDK starts, its settings request fails
-and it sends nothing, which never breaks the page. Use it for UAT and preview builds with a
-`hp_pk_test_…` key. **Do not ship a live key to production on 3.17**: go
-live on the release that moves the default to `hp.thgingenuity.com`, whose notes will say so.
+| 3.17 | `https://hp.uat.thgingenuity.com` | Interim: HP's nonprod stack. Test keys only. |
+| 3.18 | `https://hp.thgingenuity.com` | Production, for test and live keys alike. Go live on this. |
+
+The key, not the host, keeps test traffic out of live results, so UAT and preview builds use the
+same host with a `hp_pk_test_…` key. Upgrade from 3.17 before shipping a live key.
 
 The SDK loads no scripts and sets no cookie except `thg_hp_consent`, once the CMP has answered,
 and `thg_hp_anonymous_id` for a source set to `hp_cookie`, once analytics is granted.