@harshankur/viewcounter 3.2.0 → 3.4.0

package/.env.example

@@ -115,7 +115,9 @@ PORT=3030
 LOG_LEVEL=info
 
 # Rate limiting, applied per client IP across all endpoints. This is the
-# single-abuser backstop.
+# single-abuser backstop. Engagement reports (/engage, two a minute from each
+# page being read) have a separate budget of the same size, here and per app,
+# so they never use up the one page views depend on.
 RATE_LIMIT_WINDOW_MS=60000
 RATE_LIMIT_MAX=100
 

package/README.md

@@ -3,7 +3,7 @@
 [![Documentation](https://img.shields.io/badge/docs-viewcounter.harshankur.com-blueviolet)](https://viewcounter.harshankur.com)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 [![Test Suite](https://github.com/harshankur/viewcounter/actions/workflows/test.yml/badge.svg)](https://github.com/harshankur/viewcounter/actions/workflows/test.yml)
-[![Tests](https://img.shields.io/badge/tests-1098%20passing-success)](TEST_REPORT.md)
+[![Tests](https://img.shields.io/badge/tests-1134%20passing-success)](TEST_REPORT.md)
 [![npm](https://img.shields.io/npm/v/@harshankur/viewcounter?logo=npm)](https://www.npmjs.com/package/@harshankur/viewcounter)
 [![provenance](https://img.shields.io/badge/provenance-signed-brightgreen?logo=github)](https://www.npmjs.com/package/@harshankur/viewcounter#provenance)
 
@@ -12,13 +12,48 @@ A comprehensive Node.js/Express analytics server for tracking website views with
 ## 📖 Documentation
 Visit our [Interactive Documentation](https://viewcounter.harshankur.com) for detailed API specifications, debugging tips, and integration guides.
 
-## 🛡️ GDPR Compliant & Privacy-First
-**100% GDPR Compliant By Design.** This project is built from the ground up to respect user privacy and adhere to modern ethical standards:
+## 🛡️ Built for GDPR Compliance, Privacy-First
+**Built for GDPR compliance, by design.** This project is built from the ground up to respect user privacy and adhere to modern ethical standards:
 - **Nothing on the visitor's device**: no cookie, no localStorage, no sessionStorage, and no identifier sent with a view, so the tracker needs no consent banner under the ePrivacy rules on device storage. (The optional admin UI signs its operator in with a session cookie; tracking never sets one.)
 - **Data Sovereignty**: You own your data. Analytics never leave your private infrastructure.
-- **Minimal Collection**: records what analytics needs, each in a form that does not identify a person. [What Gets Tracked?](#what-gets-tracked) lists every field, where it comes from, and how it is stored; the raw IP address, the user agent, and the query string are never stored.
+- **Minimal Collection**: records what analytics needs, each in a form that does not identify a person directly. [What Gets Tracked?](#what-gets-tracked) lists every field, where it comes from, and how it is stored; the raw IP address, the user agent, and the query string are never stored.
 - **Bots left out**: crawlers, link previewers, and automated browsers are recognised and never stored, only counted per minute by name.
 
+### What is still yours to do
+
+ViewCounter does its part, and that is what "built for GDPR compliance" means.
+Compliance itself belongs to whoever runs it, and a few things no software can
+do for you:
+
+- **Say so.** Mention in your privacy notice that visits are counted, what is
+  recorded ([What Gets Tracked?](#what-gets-tracked)), and for how long.
+- **Have a legal basis.** For plain visit counting this is normally legitimate
+  interest; that judgement is yours to make and record.
+- **Keep personal details out.** ViewCounter stores what you send in custom
+  event data and the optional `sessionId` as given. Do not put names, email
+  addresses, or account IDs there.
+- **Answer requests.** During its window (a day by default) a visitor hash is
+  pseudonymous: whoever holds both the server secret and that window's salt
+  could tie it to a known address and browser. A few minutes after the window
+  ends the salt is deleted and, as long as no copy of it survives (next point),
+  nobody can, you included. What remains is a masked address
+  and coarse details. Treat it as personal data all the same: a visitor may
+  ask what is held or ask for erasure, which the admin UI does
+  ([Deleting, and GDPR](#deleting-and-gdpr)).
+- **Keep no copies of the salts.** The `_visitor_salts` table holds the salts
+  of the windows now running and nothing else. A copy of an old salt, wherever
+  it is, keeps the one thing whose deletion makes old hashes untraceable:
+  - **Backups**: exclude the table
+    (`mysqldump --ignore-table=<database>._visitor_salts`). Restoring without
+    it is harmless: a new salt is made on the next view.
+  - **The binary log**: MySQL 8 has it on by default and keeps 30 days, salts
+    included. If you do not replicate, turn it off (`skip-log-bin`) or leave
+    this database out of it (`binlog-ignore-db`); otherwise shorten
+    `binlog_expire_logs_seconds` to what you can accept. MariaDB has it off
+    unless you turned it on.
+
+This is a description of the software, not legal advice.
+
 ### 🔄 Data Privacy Lifecycle
 ```mermaid
 graph LR
@@ -39,7 +74,8 @@ We believe in total transparency regarding your visitors' data:
 2. **Immediate Masking**: Before being saved, the IP is masked (IPv4 last octet zeroed; IPv6 interface identifier zeroed).
 3. **Keyed, Not Just Hashed**: The visitor identifier is an HMAC-SHA-256 keyed with a 32-byte server secret generated on first run and stored at mode `0600`. This matters: an *unkeyed* hash of an IP is reversible by exhausting the 2^32 IPv4 space, which takes about an hour on one CPU core. Without the secret, that search is infeasible.
 4. **Rotating**: The hash also mixes in a time window (`UNIQUE_VISITOR_WINDOW_HOURS`), so the same visitor hashes differently after each window and their visits cannot be linked over time.
-5. **Automated Guards**: [`tests/privacyFailSafe.test.js`](tests/privacyFailSafe.test.js) asserts that no raw IP or User-Agent reaches either the bound parameters *or* the SQL text of any statement, and that the hash is genuinely keyed. CI runs it on every push, so a change that started storing raw IPs would fail the build.
+5. **Salted, and the salt is thrown away**: each window has its own random salt, kept in the `_visitor_salts` table only while the window lasts and deleted a few minutes after it ends, by a timer, whether or not anyone visits. From then on the hashes of that window cannot be recomputed from a known address and browser by anyone, the secret's holder included. The salt is useless without the server secret, which is never in the database.
+6. **Automated Guards**: [`tests/privacyFailSafe.test.js`](tests/privacyFailSafe.test.js) asserts that no raw IP or User-Agent reaches either the bound parameters *or* the SQL text of any statement, and that the hash is genuinely keyed. CI runs it on every push, so a change that started storing raw IPs would fail the build.
 
 ## ✨ Features
 
@@ -220,8 +256,9 @@ every app together under **All apps**; the choice follows you between them.
     bounce rate, visit duration, pages per visit, time on page, scroll depth),
     each against the period before, with a sparkline; choose one to chart it
     over time, or read every number per period as a table;
-  - **Right now**: visitors in the last few minutes, views per minute over
-    the last half hour, and the pages open, refreshed while you look;
+  - **Right now**: visitors in the last few minutes (by a new view, or by the
+    tracker's report from a page still being read), views per minute over the
+    last half hour, and the pages open, refreshed while you look;
   - where visits come from (channels, referrers, referring pages, and every
     campaign tag), pages (top, entry with bounce rate, exit, titles, sites),
     locations (a world map, countries, regions, cities, languages), devices,
@@ -392,7 +429,13 @@ Content-Type: text/plain   # or application/json
 {"appId": "blog", "id": "<the id /registerView returned>", "ms": 42000, "scroll": 80}
 ```
 How long the page was visible (`ms`, up to 6 hours) and how much of it had been
-on screen (`scroll`, 0 to 100). A later report can only raise either. A report
+on screen (`scroll`, 0 to 100). `scroll` is optional: leave it out for a page
+that fits its window, where there is nothing to scroll, and the view keeps no
+depth and stays out of the scroll averages. The tracker script does this
+itself. A later report can only raise either. Each
+report also marks the view as seen just now, which keeps its visitor in the
+admin's **Right now** for the next few minutes; the tracker script sends one
+every half minute while the page is being read. A report
 for a view that is unknown, trashed, or older than a day changes nothing and is
 counted in the tracking log as refused. `text/plain` is accepted so
 `navigator.sendBeacon` can deliver it as the page closes, without a CORS
@@ -618,7 +661,7 @@ returned by any API.
 |-------|-----------|-----------|-----|
 | **Timestamp** | Server | When the view was recorded | Everything over time |
 | **Masked IP** | Request | IPv4 with the last octet zeroed, IPv6 with the interface identifier zeroed | Abuse investigation at network level, never a person |
-| **Visitor hash** | IP and User-Agent, with a secret | HMAC-SHA-256, keyed with a server secret, rotating every window | Unique views, visitors, and visits; never returned |
+| **Visitor hash** | IP and User-Agent, with a secret | HMAC-SHA-256, keyed with a server secret and the window's salt, rotating every window; the salt is deleted when the window ends | Unique views, visitors, and visits; never returned |
 | **Country** | IP, looked up in memory | Two-letter code | Where visitors are |
 | **Region, City** | IP, with an optional [city database](#location-data) | Names, such as Bavaria and Munich | Where visitors are, more finely |
 | **Language** | `Accept-Language` | Primary subtag only, such as `de` (never `de-CH`, never a list) | Which languages to write in |
@@ -631,7 +674,7 @@ returned by any API.
 | **Device Size** | `deviceSize` | small, medium, large | Layout decisions |
 | **Browser, OS, and versions** | User-Agent, parsed in memory | Names and versions, such as Chrome 140 on macOS 15 | Compatibility |
 | **Device Type** | User-Agent | desktop, mobile, tablet, tv, console, wearable | Compatibility |
-| **Time on page, Scroll depth** | The tracker script's `/engage` report | Milliseconds visible (at most 6 hours); percent of the page seen | Whether pages are read |
+| **Time on page, Scroll depth, Last seen** | The tracker script's `/engage` report | Milliseconds visible (at most 6 hours); percent of the page seen, empty for a page that fits its window; when the page last reported | Whether pages are read |
 | **Event Type, Event Data** | `/event` | Type name; JSON up to 4 kB, as your site sends it | Custom events |
 | **Session ID** | `sessionId` (optional) | As your site sends it | Your own grouping; the tracker never sends one |
 
@@ -662,7 +705,10 @@ This setting prevents counting the same visitor multiple times within a time win
 - If no: it is stored as a unique view
 
 The window is also how often the visitor hash rotates, so it bounds how long
-the same person counts as one visitor.
+the same person counts as one visitor. It is the lifetime of the window's salt
+as well: the longer the window, the longer a hash stays traceable by whoever
+holds the secret and the salt. A day is what privacy-first analytics commonly
+uses; lengthen it knowingly.
 
 **Examples:**
 - `24` (default): the same visitor counts once per day
@@ -688,7 +734,7 @@ because a browser on your site must be able to reach them. Everything that
 - ✅ **Authenticated, scoped read API**: every analytics endpoint requires `x-api-key`, compared in constant time, and each key is authorized against the specific `appId` requested. Fails closed when unconfigured.
 - ✅ **Separate admin tier**: provisioning apps uses its own credential; a read key cannot provision and an admin key cannot read.
 - ✅ **Per-tenant rate limits**: an `appId`-keyed budget alongside the per-IP limit.
-- ✅ **Keyed visitor hashing**: HMAC-SHA-256 with a persisted 32-byte server secret, rotating per window, so stored hashes are not reversible to an IP.
+- ✅ **Keyed visitor hashing**: HMAC-SHA-256 with a persisted 32-byte server secret and a per-window salt that is deleted when the window ends, so stored hashes are not reversible to an IP, and after their window not even recomputable.
 - ✅ **SQL injection prevention**: every value is a bound parameter; the only interpolated identifier is `appId`, gated by the allowlist.
 - ✅ **Explicit CORS allowlist**: no wildcard, and writes can be bound to registered origins per app.
 - ✅ **Proxy-aware IP derivation**: client-supplied forwarding headers are not trusted unless `TRUST_PROXY` says so.
@@ -821,6 +867,15 @@ Two independent limits apply to writes:
   everyone else on the instance depends on. Keyed on `appId` alone, so it cannot
   be bypassed by rotating addresses. Set `0` to disable for single-tenant use.
 
+Each limit is applied twice, as two separate budgets of that size: one for
+engagement reports (`/engage`) and one for everything else. A page being read
+reports every half minute, so each open, active tab costs two reports a minute;
+on a shared budget, the readers behind one office address could have used it up
+and had their page views refused. Apart, reports can only crowd out other
+reports. With the defaults that is room for about 50 readers at once per
+address and 500 per app; raise the limits if you expect more, or a reader's
+time on page is only updated when their page is hidden or left.
+
 ### What is still yours to build
 
 Tenancy here is data isolation and quota, not a billing system. There is no
@@ -943,7 +998,8 @@ One tag, anywhere in the page:
 It records a view of each page, including page changes in single-page apps
 (`history.pushState`, `replaceState`, and the back button, each referred by
 the page it left); how long each page was visible and how far it was
-scrolled; clicks on links to other sites (the other site's hostname only);
+scrolled, reported when the page is hidden or left and every half minute
+while it is being read; clicks on links to other sites (the other site's hostname only);
 clicks on downloads (the file name only); and the landing URL's campaign tags.
 It stores nothing on the device and sends no identifier, and it skips
 automated browsers.
@@ -953,6 +1009,9 @@ automated browsers.
 | `data-app` | required | The app ID the views belong to |
 | `data-hosts` | every host | Only track on these hostnames, comma-separated, so development servers and previews stay out of the data |
 | `data-spa` | `true` | Treat history changes as page views |
+| `data-hash` | none | Fragment prefixes, comma-separated (`#docs/,#spec/`), that count as their own page, for pages that route by fragment. Any other fragment stays part of the same page. A matching fragment is stored whole as part of the page, so list only prefixes whose fragments carry nothing private. As a referrer, such a page is its path alone |
+| `data-heartbeat` | `true` | Report time on page every half minute while the page is visible and in use, not only when it is hidden or left. It keeps the visitor in the admin's **Right now** while they read one page, and saves the time of a tab the browser closes without warning. It stops after half an hour without any input |
+| `data-campaigns` | `true` | Send the landing URL's `utm_*` tags with the view |
 | `data-outbound` | `true` | Record clicks on links to other sites, as `outbound` events |
 | `data-downloads` | `true` | Record clicks on downloads (pdf, zip, dmg, docx, and so on), as `download` events |
 | `data-respect-dnt` | `false` | Send nothing when the browser's Do Not Track is on |
@@ -1030,10 +1089,10 @@ npm run test:watch
 # Run tests and persist database for inspection
 npm run test:persist
 
-# Run tests for CI/CD (no report generation)
+# The fast gate CI runs: lint and Jest with coverage, no browser, no report
 npm run test:ci
 
-# Run only the admin UI tests in a real browser (Playwright)
+# Run only the admin UI and tracker tests in a real browser (Playwright)
 npx playwright install chromium   # once
 npm run test:ui
 ```
@@ -1041,6 +1100,12 @@ npm run test:ui
 `npm test` includes the Playwright suite, so run `npx playwright install
 chromium` once before the first run.
 
+CI runs everything except the browser tests: lint, Jest with its coverage
+floor, the dependency audit, the tarball check, and the end-to-end run against
+a real MySQL. It does not download a browser, to save CI time, so the browser
+tests are a local step: run `npm run test:ui` whenever you change anything
+under `admin/` or `tracker/`, and the full `npm test` before a release.
+
 ### Test Database
 
 **Automatic Management:**

package/constants.js

@@ -97,7 +97,7 @@ const DATABASE = {
     QUERY_TIMEOUT_MS: 5_000,
     CONNECT_TIMEOUT_MS: 10_000,
     DEFAULT_PORT: 3306,
-    SCHEMA_VERSION: 'schema_v5',
+    SCHEMA_VERSION: 'schema_v7',
     /** Rows given a public_id per statement when backfilling an old table. */
     BACKFILL_BATCH_SIZE: 500,
 };
@@ -123,6 +123,14 @@ const PRIVACY = {
     /** Owner-only. The secret is what makes visitor hashes irreversible. */
     SECRET_FILE_MODE: 0o600,
     SECRET_FILENAME: '.visitor-secret',
+    /**
+     * How long after its window ended a visitor salt may still exist, so that
+     * instances whose clocks differ by a little agree on the window's salt
+     * right up to its end.
+     */
+    SALT_GRACE_MS: 5 * 60 * 1000,
+    /** How often ended windows' salts are looked for and deleted. */
+    SALT_PRUNE_INTERVAL_MS: 5 * 60 * 1000,
     /** Rejects a key short enough to be guessable. */
     MIN_API_KEY_LENGTH: 32,
     /**
@@ -371,6 +379,8 @@ const ADMIN_SESSIONS_TABLE = '_admin_sessions';
 const VIEW_LOG_TABLE = '_view_log';
 /** Tracking requests that were not stored, counted per minute. */
 const TRACKING_REJECTIONS_TABLE = '_tracking_rejections';
+/** The salt of the current visitor-hash window, deleted when the window ends. */
+const VISITOR_SALTS_TABLE = '_visitor_salts';
 
 /** The tracking pipeline's own bounds. */
 const TRACKING = {
@@ -540,6 +550,7 @@ module.exports = {
     ADMIN_SESSIONS_TABLE,
     VIEW_LOG_TABLE,
     TRACKING_REJECTIONS_TABLE,
+    VISITOR_SALTS_TABLE,
     TRACKING,
     TRACKING_OUTCOME,
     REJECTION_REASON,

package/db/DatabaseManager.js

@@ -33,6 +33,7 @@ const {
 } = require('./adminSchema');
 const LogRepository = require('./LogRepository');
 const AdminRepository = require('./AdminRepository');
+const { createVisitorSaltStore } = require('./visitorSalt');
 
 /**
  * Columns returned for a session lookup.
@@ -182,6 +183,7 @@ class DatabaseManager {
         this.mode = config.mode || 'connect';
         this.logs = new LogRepository(this);
         this.admin = new AdminRepository(this);
+        this.visitorSalts = createVisitorSaltStore(() => this.pool);
     }
 
     /**
@@ -203,6 +205,10 @@ class DatabaseManager {
                 logger.info(`Assigned public IDs to ${result.backfilled} existing row(s) in '${appId}'`);
             }
         }
+
+        // From here on the salts of ended visitor-hash windows are deleted on
+        // the clock, whether or not anyone visits.
+        this.visitorSalts.start();
     }
 
     /** @throws {Error} when a query is attempted before initialize() */
@@ -457,12 +463,19 @@ class DatabaseManager {
 
         // Privacy boundary. Neither the raw IP nor the raw User-Agent is bound
         // into any statement below; only the masked address and the keyed,
-        // rotating hash derived from them.
+        // rotating hash derived from them. The hash is also salted with a
+        // value that is deleted when its window ends, so that afterwards not
+        // even the secret's holder can recompute it.
+        if (!visitorSecret) throw getError(ErrorType.SECRET_UNAVAILABLE);
+        const now = Date.now();
+        const salt = await this.visitorSalts.current(uniqueWindowHours, now);
         const hashedVisitor = PrivacyUtils.generateVisitorHash(
             ip,
             userAgent,
             visitorSecret,
             uniqueWindowHours,
+            now,
+            salt,
         );
         const maskedIp = PrivacyUtils.maskIP(ip);
 
@@ -548,14 +561,16 @@ class DatabaseManager {
     /**
      * Record how long a view's page was visible and how far it was scrolled.
      *
-     * A page reports this when it is hidden or left, possibly more than once
-     * (hidden, shown again, then left), each time with its running total, so
-     * the larger value always wins. Only a live view from the last
+     * A page reports this when it is hidden or left, and every half minute
+     * while it is being read, each time with its running total, so the larger
+     * value always wins. Each report also marks the view as seen just now,
+     * which is what keeps its visitor in "right now" between page views. Only a live view from the last
      * TRACKING.ENGAGE_WINDOW_HOURS is updated: an old or trashed view keeps
      * what it had.
      *
      * @param {string} appId already validated
-     * @param {{ viewId: string, engagedMs: number, scrollDepth: number }} engagement
+     * @param {{ viewId: string, engagedMs: number, scrollDepth: number|null }} engagement
+     *     `scrollDepth` is null for a page that did not scroll, which leaves the stored depth as it was
      * @returns {Promise<boolean>} whether a view was updated
      */
     async addEngagement(appId, { viewId, engagedMs, scrollDepth }) {
@@ -563,10 +578,12 @@ class DatabaseManager {
         const [result] = await this.pool.query(
             `UPDATE \`${appId}\`
              SET engaged_ms = GREATEST(COALESCE(engaged_ms, 0), ?),
-                 scroll_depth = GREATEST(COALESCE(scroll_depth, 0), ?)
+                 scroll_depth = CASE WHEN ? IS NULL THEN scroll_depth
+                                     ELSE GREATEST(COALESCE(scroll_depth, 0), ?) END,
+                 last_seen_at = NOW()
              WHERE public_id = ? AND ${LIVE_ROW}
                AND timestamp > DATE_SUB(NOW(), INTERVAL ? HOUR)`,
-            [engagedMs, scrollDepth, viewId, TRACKING.ENGAGE_WINDOW_HOURS]
+            [engagedMs, scrollDepth, scrollDepth, viewId, TRACKING.ENGAGE_WINDOW_HOURS]
         );
         return result.affectedRows > 0;
     }
@@ -811,6 +828,7 @@ class DatabaseManager {
      * Gracefully close all connections
      */
     async close() {
+        this.visitorSalts.stop();
         if (this.pool) {
             await this.pool.end();
             this.pool = null;

package/db/adminSchema.js

@@ -14,6 +14,7 @@ const {
     DATABASE,
     FIELD_MAX_LENGTH,
     TRACKING_REJECTIONS_TABLE,
+    VISITOR_SALTS_TABLE,
     VIEW_LOG_TABLE,
 } = require('../constants');
 const { getError, logWarning, ErrorType, WarningType } = require('../utils/errorUtils');
@@ -42,7 +43,9 @@ const ADMIN_COLUMNS = [
  * Columns 3.2 added for richer, still identifier-free analysis: which of the
  * app's sites and which language, the campaign tags of the landing URL, an
  * optional region and city (only with a city database configured), and how
- * long the page was visible and how far it was scrolled.
+ * long the page was visible and how far it was scrolled. 3.3 added
+ * `last_seen_at`: when the page last reported its engagement, which is how
+ * "right now" still counts a visitor who has been reading one page for a while.
  */
 const TRACKING_COLUMNS = [
     { name: 'hostname', ddl: `VARCHAR(${FIELD_MAX_LENGTH.HOSTNAME}) DEFAULT NULL` },
@@ -56,13 +59,29 @@ const TRACKING_COLUMNS = [
     { name: 'city', ddl: `VARCHAR(${FIELD_MAX_LENGTH.CITY}) DEFAULT NULL` },
     { name: 'engaged_ms', ddl: 'INT UNSIGNED DEFAULT NULL' },
     { name: 'scroll_depth', ddl: 'TINYINT UNSIGNED DEFAULT NULL' },
+    { name: 'last_seen_at', ddl: 'DATETIME DEFAULT NULL' },
 ];
 
+/**
+ * The salt of the current visitor-hash window (db/visitorSalt.js). One row at
+ * a time: the rows of windows that have ended are deleted, which is the point.
+ */
+const VISITOR_SALTS_DDL = `
+    CREATE TABLE IF NOT EXISTS \`${VISITOR_SALTS_TABLE}\` (
+        \`rotation_hours\` INT UNSIGNED NOT NULL,
+        \`window_id\` BIGINT UNSIGNED NOT NULL,
+        \`salt\` CHAR(64) NOT NULL,
+        \`created_at\` DATETIME NOT NULL,
+        PRIMARY KEY (\`rotation_hours\`, \`window_id\`)
+    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
+`;
+
 /** Indexes the admin columns need, keyed by index name. */
 const ADMIN_INDEXES = {
     uq_public_id: 'UNIQUE INDEX `uq_public_id` (`public_id`)',
     idx_deleted_at: 'INDEX `idx_deleted_at` (`deleted_at`)',
     idx_admin_modified_at: 'INDEX `idx_admin_modified_at` (`admin_modified_at`)',
+    idx_last_seen_at: 'INDEX `idx_last_seen_at` (`last_seen_at`)',
 };
 
 /**
@@ -303,6 +322,7 @@ async function ensureLogTables(pool) {
     await pool.query(VIEW_LOG_DDL);
     await pool.query(TRACKING_REJECTIONS_DDL);
     await pool.query(ADMIN_SESSIONS_DDL);
+    await pool.query(VISITOR_SALTS_DDL);
 
     const viewLogColumns = await readColumns(pool, VIEW_LOG_TABLE);
     for (const column of VIEW_LOG_ADDED_COLUMNS) {
@@ -321,6 +341,7 @@ module.exports = {
     VIEW_LOG_ADDED_COLUMNS,
     TRACKING_REJECTIONS_DDL,
     ADMIN_SESSIONS_DDL,
+    VISITOR_SALTS_DDL,
     NEW_TABLE_ADMIN_COLUMNS,
     NEW_TABLE_ADMIN_INDEXES,
     backfillPublicIds,

package/db/analysis.js

@@ -436,12 +436,17 @@ function tallyEventProperties(rows) {
  * @param {string[]} appIds
  */
 async function runRealtime(pool, table, appIds) {
-    const branches = appIds.map((appId) => `SELECT ? AS app_id, visitor_hash, page_path, event_type, timestamp
+    const within = (column, minutes) => `${column} >= DATE_SUB(NOW(), INTERVAL ${minutes} MINUTE)`;
+    // A visitor is here now when a view of theirs was recorded, or last
+    // reported its engagement, in the last few minutes: someone reading one
+    // page for a quarter of an hour sends no new view, only those reports.
+    const recent = `(${within('timestamp', ANALYSIS.REALTIME_VISITOR_MINUTES)} OR ${within('last_seen_at', ANALYSIS.REALTIME_VISITOR_MINUTES)})`;
+    const charted = within('timestamp', ANALYSIS.REALTIME_CHART_MINUTES);
+    const branches = appIds.map((appId) => `SELECT ? AS app_id, visitor_hash, page_path, event_type, timestamp, last_seen_at
         FROM ${table(appId)}
-        WHERE deleted_at IS NULL AND timestamp >= DATE_SUB(NOW(), INTERVAL ${ANALYSIS.REALTIME_CHART_MINUTES} MINUTE)`);
+        WHERE deleted_at IS NULL AND (${charted} OR ${within('last_seen_at', ANALYSIS.REALTIME_VISITOR_MINUTES)})`);
     const cte = `WITH v AS (${branches.join(' UNION ALL ')})`;
     const params = [...appIds];
-    const recent = `timestamp >= DATE_SUB(NOW(), INTERVAL ${ANALYSIS.REALTIME_VISITOR_MINUTES} MINUTE)`;
 
     const [[summary = {}]] = await pool.query(`${cte}
         SELECT COUNT(DISTINCT CASE WHEN ${recent} THEN visitor_hash END) AS visitors,
@@ -449,7 +454,7 @@ async function runRealtime(pool, table, appIds) {
         FROM v`, params);
     const [minutes] = await pool.query(`${cte}
         SELECT FLOOR(UNIX_TIMESTAMP(timestamp) / 60) AS minute, COUNT(*) AS views
-        FROM v GROUP BY minute ORDER BY minute`, params);
+        FROM v WHERE ${charted} GROUP BY minute ORDER BY minute`, params);
     const [pages] = await pool.query(`${cte}
         SELECT app_id, page_path AS page, COUNT(DISTINCT visitor_hash) AS visitors
         FROM v WHERE ${recent} AND event_type = ${PAGEVIEW}

package/db/visitorSalt.js

@@ -0,0 +1,137 @@
+/**
+ * The salt of each visitor-hash window.
+ *
+ * A visitor hash is keyed with the server secret and with a random salt that
+ * exists only for the window it belongs to. Once the window is over the salt
+ * is deleted, and from then on nobody, the operator included, can recompute a
+ * hash of that window from a known address and browser: the records it left
+ * can no longer be tied to anyone.
+ *
+ * The salt lives in the database so that a restart, or a second instance,
+ * within the window keeps telling the same visitors apart. It is useless
+ * without the server secret, which is never in the database.
+ *
+ * Deletion goes by the clock, not by traffic: a salt is removed a few minutes
+ * after its window ended (the grace covers instances whose clocks differ),
+ * by a timer and by every read, whether or not another view ever arrives.
+ */
+
+const crypto = require('crypto');
+
+const { PRIVACY, VISITOR_SALTS_TABLE } = require('../constants');
+const PrivacyUtils = require('../utils/privacyUtils');
+const { getError, logWarning, ErrorType, WarningType } = require('../utils/errorUtils');
+
+const TABLE = `\`${VISITOR_SALTS_TABLE}\``;
+const SALT_PATTERN = new RegExp(`^[0-9a-f]{${PRIVACY.SECRET_BYTES * 2}}$`);
+const MS_PER_HOUR = 60 * 60 * 1000;
+
+/** When the window `windowId` of length `hours` is over, in epoch millis. */
+const windowEnd = (hours, windowId) => (windowId + 1) * hours * MS_PER_HOUR;
+
+/**
+ * @param {() => { query: Function }} getPool the manager's pool, read on each use
+ * @returns {{
+ *   current: (rotationHours: number, now?: number) => Promise<string>,
+ *   prune: (now?: number) => Promise<number>,
+ *   start: (intervalMs?: number) => void,
+ *   stop: () => void,
+ * }}
+ */
+function createVisitorSaltStore(getPool) {
+    /**
+     * The salts in use, and the reads under way, by `hours:windowId`. The
+     * service hashes page views and custom events over different window
+     * lengths, so there is one entry per length in use: a handful at most.
+     * @type {Map<string, { salt?: string, promise?: Promise<string>, end: number }>}
+     */
+    const windows = new Map();
+    let timer = null;
+
+    /** Forget, in memory too, every window that is over. */
+    function forgetEnded(now) {
+        for (const [key, entry] of windows) {
+            if (entry.end + PRIVACY.SALT_GRACE_MS <= now) windows.delete(key);
+        }
+    }
+
+    /**
+     * Delete the salt of every window that ended more than the grace ago.
+     * @param {number} [now] epoch millis
+     * @returns {Promise<number>} salts deleted
+     */
+    async function prune(now = Date.now()) {
+        forgetEnded(now);
+        const [result] = await getPool().query(
+            `DELETE FROM ${TABLE} WHERE (window_id + 1) * rotation_hours * ${MS_PER_HOUR} + ? <= ?`,
+            [PRIVACY.SALT_GRACE_MS, now]);
+        return (result && result.affectedRows) || 0;
+    }
+
+    async function read(hours, windowId, now) {
+        const pool = getPool();
+        // Whoever gets there first decides the window's salt; everyone reads that one.
+        await pool.query(
+            `INSERT IGNORE INTO ${TABLE} (rotation_hours, window_id, salt, created_at) VALUES (?, ?, ?, NOW())`,
+            [hours, windowId, crypto.randomBytes(PRIVACY.SECRET_BYTES).toString('hex')]);
+        const [rows] = await pool.query(
+            `SELECT salt FROM ${TABLE} WHERE rotation_hours = ? AND window_id = ?`, [hours, windowId]);
+        const salt = Array.isArray(rows) && rows[0] ? rows[0].salt : null;
+        // Failing closed is deliberate: a hash made without the salt would
+        // stay recomputable for ever while looking like the others.
+        if (typeof salt !== 'string' || !SALT_PATTERN.test(salt)) throw getError(ErrorType.SECRET_UNAVAILABLE);
+        // A new window is the moment the one before it ended.
+        await prune(now);
+        return salt;
+    }
+
+    return {
+        /**
+         * @param {number} rotationHours the unique-visitor window
+         * @param {number} [now] epoch millis; pass the same value to the hash
+         * @returns {Promise<string>} hex salt of the window `now` falls in
+         */
+        async current(rotationHours, now = Date.now()) {
+            const hours = PrivacyUtils.rotationHours(rotationHours);
+            const windowId = PrivacyUtils.currentWindowId(hours, now);
+            const key = `${hours}:${windowId}`;
+            const known = windows.get(key);
+            if (known) return known.salt || known.promise;
+
+            const entry = { end: windowEnd(hours, windowId) };
+            entry.promise = read(hours, windowId, now).then((salt) => {
+                entry.salt = salt;
+                return salt;
+            }, (error) => {
+                // Nothing is remembered of a failed read: the next view tries again.
+                if (windows.get(key) === entry) windows.delete(key);
+                throw error;
+            });
+            windows.set(key, entry);
+            return entry.promise;
+        },
+
+        prune,
+
+        /**
+         * Prune now and then on an interval, so a salt goes when its window
+         * ends even on a site nobody visits for days. The timer is unref'd:
+         * it never keeps the process alive on its own.
+         * @param {number} [intervalMs]
+         */
+        start(intervalMs = PRIVACY.SALT_PRUNE_INTERVAL_MS) {
+            if (timer) return;
+            const run = () => prune().catch((cause) => logWarning(WarningType.SALT_PRUNE_FAILED, { cause: cause.message }));
+            run();
+            timer = setInterval(run, intervalMs);
+            timer.unref();
+        },
+
+        stop() {
+            clearInterval(timer);
+            timer = null;
+        },
+    };
+}
+
+module.exports = { createVisitorSaltStore };

package/index.js

@@ -1,14 +1,13 @@
 const express = require('express');
 const cors = require('cors');
 const helmet = require('helmet');
-const rateLimit = require('express-rate-limit');
 
 const { ADMIN, APP_NAME, PAYLOAD_LIMITS, REJECTION_REASON, SERVER } = require('./constants');
 const config = require('./config');
 const DatabaseManager = require('./db/DatabaseManager');
 const logger = require('./utils/logger');
 const { buildCorsOptions, countRefusedPreflights } = require('./middleware/security');
-const { createAnalyticsRouter, trackingSourceFor } = require('./routes/analytics');
+const { createAnalyticsRouter, buildPerIpLimiters, trackingSourceFor } = require('./routes/analytics');
 const { createAdminRouter } = require('./routes/admin');
 const { startRetention } = require('./db/retention');
 const { createDbSessionStore } = require('./db/adminSessionStore');
@@ -79,20 +78,10 @@ function createApp() {
     // endpoint taking a body and its payload is small.
     app.use(express.json({ limit: PAYLOAD_LIMITS.MAX_BODY_BYTES }));
 
-    app.use(rateLimit({
-        windowMs: config.server.rateLimit.windowMs,
-        limit: config.server.rateLimit.max,
-        message: { message: 'Too many requests, please try again later.' },
-        standardHeaders: true,
-        legacyHeaders: false,
-        // A tracking request turned away here is counted in the tracking log
-        // like any other refusal (in memory, written in batches).
-        handler: (req, res, next, options) => {
-            if (trackingSourceFor(req.path)) {
-                router.countRejection(req, REJECTION_REASON.RATE_LIMITED, { detail: 'ip' });
-            }
-            res.status(options.statusCode).json(options.message);
-        },
+    // A tracking request turned away here is counted in the tracking log like
+    // any other refusal (in memory, written in batches).
+    app.use(buildPerIpLimiters(config.server.rateLimit, (req) => {
+        if (trackingSourceFor(req.path)) router.countRejection(req, REJECTION_REASON.RATE_LIMITED, { detail: 'ip' });
     }));
 
     app.use(router);

package/middleware/validation.js

@@ -103,7 +103,9 @@ const validateEngage = (allowedValues) => [
         .matches(UUID_PATTERN).withMessage('id must be a view ID'),
     body('ms')
         .isInt({ min: 0, max: TRACKING.MAX_ENGAGED_MS }).withMessage(`ms must be an integer between 0 and ${TRACKING.MAX_ENGAGED_MS}`),
+    // Absent for a page that fits its window: there was nothing to scroll.
     body('scroll')
+        .optional({ values: 'null' })
         .isInt({ min: 0, max: 100 }).withMessage('scroll must be an integer between 0 and 100'),
 ];
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@harshankur/viewcounter",
   "description": "A middleware backend server that registers views to my db server when requested to register a view from my other projects.",
-  "version": "3.2.0",
+  "version": "3.4.0",
   "main": "index.js",
   "engines": {
     "node": ">=24"
@@ -49,7 +49,7 @@
     "i18n-iso-countries": "^7.14.0",
     "jest": "^30.4.2",
     "jest-html-reporter": "^4.4.0",
-    "supertest": "^7.2.2",
+    "supertest": "^7.3.1",
     "topojson-client": "^3.1.0",
     "world-atlas": "^2.0.2"
   },

package/routes/analytics.js

@@ -155,6 +155,40 @@ function buildPerAppLimiter(rateLimitConfig, onLimit = () => {}) {
     });
 }
 
+/** Whether a request is an engagement report, the one tracking request a page repeats. */
+const isEngagement = (req) => trackingSourceFor(req.path) === VIEW_LOG_SOURCE.ENGAGE;
+
+/**
+ * The per-IP limiters: one budget for engagement reports, one for everything
+ * else, each of `max` requests per window.
+ *
+ * They are separate because a page being read reports its engagement every
+ * half minute. On one shared budget, a few dozen readers behind one address
+ * (an office, a campus) would use it up with those reports alone, and the page
+ * views of everyone at that address would be refused. Apart, reports can only
+ * ever crowd out other reports.
+ *
+ * @param {{ max: number, windowMs: number }} rateLimitConfig
+ * @param {(req: import('express').Request) => void} [onLimit] called for a refused request
+ * @returns {import('express').RequestHandler[]}
+ */
+function buildPerIpLimiters(rateLimitConfig, onLimit = () => {}) {
+    const { max, windowMs } = rateLimitConfig || {};
+    const limiter = (skip) => rateLimit({
+        windowMs,
+        limit: max,
+        message: { message: 'Too many requests, please try again later.' },
+        standardHeaders: true,
+        legacyHeaders: false,
+        skip,
+        handler: (req, res, next, options) => {
+            onLimit(req);
+            res.status(options.statusCode).json(options.message);
+        },
+    });
+    return [limiter(isEngagement), limiter((req) => !isEngagement(req))];
+}
+
 /**
  * Attach a request id used for correlating a client-visible error with the
  * server-side log line that has the real detail.
@@ -232,6 +266,11 @@ function createAnalyticsRouter({ config, dbManager, isReady = () => true, geo =
     });
     const limitPerApp = buildPerAppLimiter(config.server?.rateLimit,
         (req) => reject(req, REJECTION_REASON.RATE_LIMITED, { detail: 'app' }));
+    // Engagement reports draw on a per-app budget of their own, for the reason
+    // buildPerIpLimiters gives: an app with many readers must not have its
+    // views refused because of the reports those readers' pages send.
+    const limitEngagePerApp = buildPerAppLimiter(config.server?.rateLimit,
+        (req) => reject(req, REJECTION_REASON.RATE_LIMITED, { detail: 'app' }));
     const trackingValidation = handleTrackingValidation(reject);
 
     /** Views from these are counted in the tracking log and never stored. */
@@ -429,7 +468,7 @@ function createAnalyticsRouter({ config, dbManager, isReady = () => true, geo =
     router.post('/engage',
         express.text({ type: () => true, limit: TRACKING.ENGAGE_BODY_BYTES }),
         parseBeaconBody,
-        limitPerApp,
+        limitEngagePerApp,
         requireOrigin,
         validateEngage(config.allowed),
         trackingValidation,
@@ -441,7 +480,7 @@ function createAnalyticsRouter({ config, dbManager, isReady = () => true, geo =
                 const updated = await dbManager.addEngagement(appId, {
                     viewId: id,
                     engagedMs: Number(ms),
-                    scrollDepth: Number(scroll),
+                    scrollDepth: scroll === undefined || scroll === null ? null : Number(scroll),
                 });
                 if (!updated) reject(req, REJECTION_REASON.UNKNOWN_VIEW);
                 return res.status(HTTP_STATUS.NO_CONTENT).end();
@@ -641,4 +680,4 @@ function createAnalyticsRouter({ config, dbManager, isReady = () => true, geo =
     return router;
 }
 
-module.exports = { createAnalyticsRouter, handleRouteError, logContext, withRequestId, trackingSourceFor };
+module.exports = { createAnalyticsRouter, buildPerIpLimiters, handleRouteError, logContext, withRequestId, trackingSourceFor };

package/tracker/tracker.js

@@ -1,16 +1,31 @@
 /*!
- * viewcounter tracker, https://viewcounter.harshankur.com
+ * viewcounter tracker
+ *
+ * If you found this on a site you were visiting: this is viewcounter, a
+ * privacy-first view counter that the site runs on its own server. It is built
+ * for GDPR compliance:
+ *   - it sets no cookie and stores nothing on your device;
+ *   - it sends nothing that identifies you or your device;
+ *   - the server never stores your IP address or your browser's user agent.
+ *     It keeps a masked address, a visitor hash that changes every day by
+ *     default, and coarse details such as your country, browser and system,
+ *     so nothing it keeps identifies you directly.
+ * Check it yourself: the source is at https://github.com/harshankur/viewcounter
+ * and the homepage at https://viewcounter.harshankur.com.
+ *
+ * For site owners:
  *
  *   <script defer src="https://your-server/tracker.js" data-app="blog"></script>
  *
  * Records a view of each page (including page changes in single-page apps),
- * how long it was visible and how far it was scrolled, clicks on links to
+ * how long it was visible and, on a page that scrolls, how far (reported when the page
+ * is hidden or left, and every half minute while it is being read), clicks on links to
  * other sites and on downloads, and the campaign tags of the landing URL.
  * The page before is sent as its origin and path only.
  *
  * It stores nothing on the visitor's device (no cookie, no localStorage, no
  * sessionStorage), so it needs no consent banner, and it sends no identifier:
- * the server tells repeat visits apart with a hash it rotates every day.
+ * the server tells repeat visits apart with a hash it rotates every day by default.
  *
  * Options, as attributes on the script tag:
  *   data-app="blog"                 required: the app ID the views belong to
@@ -18,6 +33,11 @@
  *                                   only track on these hostnames (keeps dev
  *                                   servers and previews out of the data)
  *   data-spa="false"                do not treat history changes as page views
+ *   data-hash="#docs/,#spec/"       count a URL fragment that starts with one of
+ *                                   these as its own page (hash-routed pages)
+ *   data-heartbeat="false"          report time on page only when the page is
+ *                                   hidden or left, not while it is being read
+ *   data-campaigns="false"          do not send the landing URL's utm_* tags
  *   data-outbound="false"           do not record clicks on links to other sites
  *   data-downloads="false"          do not record clicks on downloads
  *   data-respect-dnt="true"         send nothing when Do Not Track is on
@@ -32,6 +52,7 @@
     const option = (name, fallback) => (script.dataset[name] === undefined ? fallback : script.dataset[name] !== 'false');
     const hosts = (script.dataset.hosts || '').split(',').map((host) => host.trim().toLowerCase()).filter(Boolean);
     if (hosts.length && !hosts.includes(location.hostname.toLowerCase())) return;
+    const hashRoutes = (script.dataset.hash || '').split(',').map((prefix) => prefix.trim()).filter(Boolean);
     if (location.protocol !== 'http:' && location.protocol !== 'https:') return;
     // Automated browsers are not visitors.
     if (navigator.webdriver) return;
@@ -41,12 +62,25 @@
     const UTM = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content'];
     const DOWNLOAD = /\.(pdf|zip|gz|tgz|rar|7z|dmg|exe|msi|pkg|deb|rpm|apk|iso|csv|xlsx?|docx?|pptx?|odt|ods|epub|mp3|mp4|mov|avi|wav)$/i;
     const MAX_ENGAGED_MS = 6 * 60 * 60 * 1000;
+    const ENGAGE_WINDOW_MS = 24 * 60 * 60 * 1000;
+    const HEARTBEAT_MS = 30 * 1000;
+    // A tab left open with nobody at it stops reporting after this long without input.
+    const IDLE_MS = 30 * 60 * 1000;
 
     const deviceSize = () => (innerWidth < 768 ? 'small' : innerWidth < 1200 ? 'medium' : 'large');
-    /** How much of the page has been on screen, from 0 to 100. */
+    /**
+     * How much of the page has been on screen, from 0 to 100, or null for a
+     * page that fits the window: there is nothing to scroll, so "all of it"
+     * would say nothing about the reader.
+     */
     const seen = () => {
         const height = Math.max(document.documentElement.scrollHeight, document.body ? document.body.scrollHeight : 0);
-        return height <= 0 ? 100 : Math.min(100, Math.round(((scrollY + innerHeight) / height) * 100));
+        return height <= innerHeight + 1 ? null : Math.min(100, Math.round(((scrollY + innerHeight) / height) * 100));
+    };
+    /** Note how far the page on screen has been scrolled, if it scrolls at all (it may have grown since it loaded). */
+    const measure = () => {
+        const depth = view ? seen() : null;
+        if (depth !== null) view.scroll = Math.max(view.scroll === null ? 0 : view.scroll, depth);
     };
 
     /** A URL's origin and path: its query and fragment can carry tokens or emails. */
@@ -59,18 +93,28 @@
         }
     };
 
+    /** The page on screen: its path, and its fragment when that is one of the site's own routes. */
+    const currentPage = () => location.pathname
+        + (hashRoutes.some((prefix) => location.hash.startsWith(prefix)) ? location.hash : '');
+
     let view = null;
     let referrer = document.referrer ? originAndPath(document.referrer) : '';
-    let path = location.pathname;
-
-    /** Tell the server how long the current page was visible and how far it was scrolled. */
-    function reportEngagement() {
-        if (!view || !view.id) return;
-        const ms = Math.min(MAX_ENGAGED_MS, Math.round(view.visibleMs + (view.visibleSince === null ? 0 : performance.now() - view.visibleSince)));
-        if (ms <= view.sentMs && view.scroll <= view.sentScroll) return;
-        view.sentMs = ms;
-        view.sentScroll = view.scroll;
-        const body = JSON.stringify({ appId: app, id: view.id, ms, scroll: view.scroll });
+    let path = currentPage();
+
+    /**
+     * Tell the server how long a page was visible and how far it was scrolled.
+     * `alive` sends it even when neither has grown since the last report: that
+     * is the heartbeat saying the visitor is still there.
+     */
+    function reportEngagement(of = view, alive = false) {
+        if (!of || !of.id) return;
+        const ms = Math.min(MAX_ENGAGED_MS, Math.round(of.visibleMs + (of.visibleSince === null ? 0 : performance.now() - of.visibleSince)));
+        const scroll = of.scroll === null ? -1 : of.scroll;
+        if (!alive && ms <= of.sentMs && scroll <= of.sentScroll) return;
+        of.sentMs = ms;
+        of.sentScroll = scroll;
+        // A page that never scrolled reports its time alone.
+        const body = JSON.stringify({ appId: app, id: of.id, ms, scroll: of.scroll === null ? undefined : of.scroll });
         // text/plain needs no CORS preflight, so the beacon survives the page closing.
         if (!(navigator.sendBeacon && navigator.sendBeacon(`${base}engage`, new Blob([body], { type: 'text/plain' })))) {
             fetch(`${base}engage`, { method: 'POST', body, keepalive: true, credentials: 'omit', headers: { 'Content-Type': 'text/plain' } })
@@ -79,34 +123,55 @@
     }
 
     function pageview() {
-        reportEngagement();
+        // The page being left stops counting here, and reports what it has.
+        // It is still the one on screen (a single-page app draws the next
+        // only after changing the address), so this is its last measure.
+        if (view) {
+            if (view.visibleSince !== null) view.visibleMs += performance.now() - view.visibleSince;
+            view.visibleSince = null;
+            measure();
+            reportEngagement();
+        }
+        const first = !view;
         const params = new URLSearchParams({
             appId: app,
             deviceSize: deviceSize(),
-            page: location.pathname.slice(0, 500),
+            page: currentPage().slice(0, 500),
             title: document.title.slice(0, 200),
             referrer: referrer.slice(0, 500),
         });
         // Only the campaign tags: the rest of a query string can carry
         // emails, tokens, or IDs, and never leaves the page.
-        const query = new URLSearchParams(location.search);
-        for (const key of UTM) {
-            const value = query.get(key);
-            if (value) params.set(key, value.slice(0, 100));
+        if (option('campaigns', true)) {
+            const query = new URLSearchParams(location.search);
+            for (const key of UTM) {
+                const value = query.get(key);
+                if (value) params.set(key, value.slice(0, 100));
+            }
         }
 
         const current = {
             id: null,
+            startedAt: Date.now(),
             visibleMs: 0,
             visibleSince: document.visibilityState === 'visible' ? performance.now() : null,
-            scroll: seen(),
+            scroll: null,
             sentMs: 0,
-            sentScroll: 0,
+            sentScroll: -1,
         };
         view = current;
+        // A page that has just loaded can be measured. After a navigation the
+        // old page is still on screen, and its depth is not the new one's:
+        // the new page is measured when it is scrolled, hidden, or reports.
+        if (first) measure();
         fetch(`${base}registerView?${params}`, { keepalive: true, credentials: 'omit', referrerPolicy: 'no-referrer' })
             .then((response) => (response.ok ? response.json() : null))
-            .then((result) => { if (result && result.id) current.id = result.id; })
+            .then((result) => {
+                if (!result || !result.id) return;
+                current.id = result.id;
+                // Left before the server answered: its report could not go then, so it goes now.
+                if (view !== current) reportEngagement(current);
+            })
             .catch(() => {});
     }
 
@@ -122,7 +187,7 @@
                 appId: app,
                 eventType: eventType.slice(0, 50),
                 eventData: eventData && typeof eventData === 'object' ? eventData : undefined,
-                page: location.pathname.slice(0, 500),
+                page: currentPage().slice(0, 500),
                 title: document.title.slice(0, 200),
             }),
         }).catch(() => {});
@@ -131,9 +196,10 @@
     // A page change in a single-page app is a new page view, with the page it
     // came from as its referrer (which the server files as internal).
     function navigated() {
-        if (location.pathname === path) return;
-        referrer = `${location.origin}${path}`;
-        path = location.pathname;
+        if (currentPage() === path) return;
+        // Like every referrer, without the fragment: the server keeps an origin and a path.
+        referrer = `${location.origin}${path.split('#')[0]}`;
+        path = currentPage();
         pageview();
     }
     if (option('spa', true)) {
@@ -147,6 +213,8 @@
         }
         addEventListener('popstate', navigated);
     }
+    // Asked for by name, so it does not wait on data-spa.
+    if (hashRoutes.length) addEventListener('hashchange', navigated);
 
     let scrollQueued = false;
     addEventListener('scroll', () => {
@@ -154,7 +222,7 @@
         scrollQueued = true;
         requestAnimationFrame(() => {
             scrollQueued = false;
-            if (view) view.scroll = Math.max(view.scroll, seen());
+            measure();
         });
     }, { passive: true });
 
@@ -163,12 +231,31 @@
         if (document.visibilityState === 'hidden') {
             if (view.visibleSince !== null) view.visibleMs += performance.now() - view.visibleSince;
             view.visibleSince = null;
+            measure();
             reportEngagement();
         } else if (view.visibleSince === null) {
             view.visibleSince = performance.now();
         }
     });
-    addEventListener('pagehide', reportEngagement);
+    addEventListener('pagehide', () => { measure(); reportEngagement(); });
+
+    // While the page is being read, report as it goes: the server then knows the
+    // visitor is still there, and a tab the browser kills without warning (common
+    // on phones) loses half a minute of its time at most, not all of it.
+    if (option('heartbeat', true)) {
+        let lastInput = performance.now();
+        const active = () => { lastInput = performance.now(); };
+        for (const type of ['pointerdown', 'pointermove', 'keydown', 'scroll', 'touchstart']) {
+            addEventListener(type, active, { passive: true, capture: true });
+        }
+        setInterval(() => {
+            if (!view || document.visibilityState !== 'visible' || performance.now() - lastInput > IDLE_MS) return;
+            // The server takes reports for a view for a day. A page open longer says no more.
+            if (Date.now() - view.startedAt > ENGAGE_WINDOW_MS) return;
+            measure();
+            reportEngagement(view, true);
+        }, HEARTBEAT_MS);
+    }
 
     // Links out and downloads. Only the other site's hostname, or the file's
     // name, is recorded: never the whole URL, which can carry personal data.

package/utils/errorUtils.js

@@ -48,6 +48,7 @@ const WarningType = {
     TRACKING_LOG_WRITE_FAILED: 'TRACKING_LOG_WRITE_FAILED',
     ADMIN_LOG_WRITE_FAILED: 'ADMIN_LOG_WRITE_FAILED',
     TRASH_PURGE_FAILED: 'TRASH_PURGE_FAILED',
+    SALT_PRUNE_FAILED: 'SALT_PRUNE_FAILED',
 };
 
 /**
@@ -124,6 +125,8 @@ const WARNING_MESSAGES = {
         `Automatic view log pruning failed: ${info?.cause}`,
     [WarningType.TRASH_PURGE_FAILED]: (info) =>
         `Automatic trash purge failed for '${info?.appId}': ${info?.cause}`,
+    [WarningType.SALT_PRUNE_FAILED]: (info) =>
+        `Could not delete the visitor salts of ended windows (it is tried again shortly): ${info?.cause}`,
 };
 
 /**

package/utils/privacyUtils.js

@@ -61,8 +61,17 @@ class PrivacyUtils {
      * @returns {number}
      */
     static currentWindowId(rotationHours, now = Date.now()) {
-        const hours = Math.max(Number(rotationHours) || 0, MIN_ROTATION_HOURS);
-        return Math.floor(now / (hours * MS_PER_HOUR));
+        return Math.floor(now / (this.rotationHours(rotationHours) * MS_PER_HOUR));
+    }
+
+    /**
+     * The length of a rotation window, in hours: the unique-visitor window,
+     * and never less than MIN_ROTATION_HOURS.
+     * @param {number} rotationHours
+     * @returns {number}
+     */
+    static rotationHours(rotationHours) {
+        return Math.max(Number(rotationHours) || 0, MIN_ROTATION_HOURS);
     }
 
     /**
@@ -80,6 +89,10 @@ class PrivacyUtils {
      * @param {string} secret Server secret from utils/secretStore.js
      * @param {number} [rotationHours] Window length, defaults to the unique-visitor window
      * @param {number} [now] epoch millis, injectable for tests
+     * @param {string} [salt] The window's own salt (db/visitorSalt.js), which
+     *     is deleted when the window ends. With it, a hash cannot be recomputed
+     *     afterwards even by whoever holds the secret. Without it, the window
+     *     id alone separates the windows, and the secret holder still can.
      * @returns {string} HMAC-SHA-256 hex digest
      * @throws {Error} ErrorType.SECRET_UNAVAILABLE when no secret is supplied
      */
@@ -89,6 +102,7 @@ class PrivacyUtils {
         secret,
         rotationHours = SERVER.DEFAULT_UNIQUE_VISITOR_WINDOW_HOURS,
         now = Date.now(),
+        salt = '',
     ) {
         if (!secret) {
             // Failing closed is deliberate: silently hashing without the key
@@ -98,7 +112,8 @@ class PrivacyUtils {
         }
 
         const windowId = this.currentWindowId(rotationHours, now);
-        const input = `${ip}|${userAgent}|${windowId}`;
+        // The unsalted form is kept byte for byte, for callers that use this helper on its own.
+        const input = salt ? `${salt}|${ip}|${userAgent}|${windowId}` : `${ip}|${userAgent}|${windowId}`;
 
         return crypto.createHmac('sha256', secret).update(input).digest('hex');
     }