siteread 0.2.0 → 0.2.2

package/README.md

@@ -3,7 +3,8 @@
 **Was that really ClaudeBot?** `siteread` takes the IP and User-Agent of a request from your access
 log and checks the IP against the list the claimed vendor publishes: `verified`, `spoofed` or `unknown`.
 
-Node ≥ 20, zero dependencies, nothing leaves your machine except GETs of the vendors' public IP lists.
+Node ≥ 20, zero dependencies, nothing leaves your machine except GETs of the vendors' public IP lists
+(and a `share`, if you choose to send one).
 
 ## Install
 
@@ -14,8 +15,10 @@ npx siteread --help         # the CLI, no install
 
 ## CLI
 
-`check` answers one request. The first call fetches the vendor lists into a cache file; later calls
-reuse a copy younger than 24 h.
+`check` answers one request. The first call fetches the crawler lists into a cache file; later calls
+reuse a copy younger than 24 h. The lists come from the siteread feed (one GET instead of nine;
+below); `--no-feed` fetches each from its vendor instead, and any list the feed cannot give comes
+from its vendor anyway.
 
 ```console
 $ npx siteread check 216.73.216.12 "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
@@ -60,7 +63,7 @@ Options: `--no-cache` refetches every list now (a failed fetch still falls back
 ## Digest (for the `seo-read` skill)
 
 `digest` reads a Search Console export (the Performance → Export → Download CSV zip) and your
-access logs (nginx/Apache combined, `.gz` fine) and prints one JSON document: weekly GSC totals with
+access logs (nginx/Apache combined, Caddy's JSON, or an nginx JSON `log_format`; `.gz` fine) and prints one JSON document: weekly GSC totals with
 impression-weighted position, people by first-touch source (an IP that sent a browser UA, loaded
 an asset and never probed a scanner path), goal counts, and every crawler checked against its
 vendor's list.
@@ -71,8 +74,87 @@ $ npx siteread digest --gsc export.zip --log access.log.1.gz --log access.log \
 .siteread/digests/2026-10-18.json
 ```
 
-`npx siteread --help` lists every option. The skill in [`skill/`](../../skill/README.md) turns
-the JSON into a weekly memo.
+A JSON log that holds several sites (one nginx serving an API, an app and a blog) is cut to one
+with `--host`, repeated for each name (`www.` is a different host). Combined lines carry no host,
+so `--host` keeps them and says so. For nginx JSON, the common field names are read
+(`remote_addr`, `request_uri` or `request`, `status`, `time`/`time_iso8601`/`time_local`/`msec`,
+`http_user_agent` or `user_agent`, `http_referer` or `referer`, `host` or `http_host`), with
+`escape=json` or nginx's default escaping.
+
+`npx siteread --help` lists every option. The `seo-read` skill, in siteread's Claude Code plugin
+(`/plugin install siteread@siteread` after `/plugin marketplace add synacl-iot/siteread`), turns the
+JSON into a weekly memo.
+
+## Share (opt-in)
+
+`share` sends one week of a digest's aggregate numbers to the public benchmark and prints where each
+number stands among the quartiles the benchmark publishes. It prints the exact bytes first and sends nothing without a yes (a
+prompt, or `--yes` in a script). `--dry-run` prints and stops.
+
+```console
+$ npx siteread share --dry-run
+siteread: the whole payload from .siteread/digests/2026-10-18.json, byte for byte, for https://api.siteread.dev/v1/share:
+{
+  "schema": 1,
+  "id": "3803e566762f2e56ba381b993ddf466e",
+  "week": "2026-W42",
+  "siteread": "0.2.1",
+  "metrics": {
+    "people": 4,
+    "peopleFromSearch": 1,
+    "peopleFromAi": 0,
+    "peopleFromSocial": 2,
+    "aiAnswerFetches": 0,
+    "aiSearchFetches": 0,
+    "aiTrainingFetches": 2,
+    "searchEngineFetches": 8,
+    "aiSpoofedPct": 25,
+    "gscImpressions": 238,
+    "gscClicks": 3,
+    "gscCtrPct": 1.3,
+    "gscPosition": 17.9
+  },
+  "proof": "2746561"
+}
+siteread: not sent (--dry-run)
+```
+
+The answer places each number in a band among the published 25th, 50th and 75th percentiles, and
+only for numbers the benchmark page already shows a spread for; for the rest it gives the count of
+sites so far. It is read from the published benchmark (rebuilt hourly), not from other sites' rows,
+so it never tells you more than the page does. With 14 anonymous sites, 9 of them with Search
+Console numbers (made up for this example):
+
+```console
+sent 2026-W42. Against the benchmark built 2026-10-21 09:00 UTC: 14 anonymous sites, 2026-W39 to 2026-W42
+
+metric                   yours  where
+people                       4  below the 25th percentile
+peopleFromAi                 0  25th to 50th percentile
+gscImpressions             238  9 sites; shown from 10
+gscPosition               17.9  9 sites; shown from 10
+…
+```
+
+- **What is sent:** the numbers above for the digest's last 7 days (`aiSpoofedPct` is over the whole
+  log), a week label, the version, and a random id. `SHARE_METRICS` defines each one.
+- **What is never sent:** hostnames, URLs or paths, visitor IPs, user-agents, query text, goal
+  labels. The payload is built from an allowlist of numbers, and the endpoint refuses anything else.
+- **The id** is 128 random bits in `.siteread/share-id`, made on first use. It makes the benchmark
+  count sites rather than shares, and a second share of the same week replaces the first. Delete the
+  file to start over as a new site.
+- **The proof** is a proof of work tied to the id: a number that makes SHA-256 of the id start with
+  22 zero bits, a few seconds of CPU once, kept in `.siteread/share-proof`. Only a site's first
+  share needs it. Ids are free to make, so without it one sender could invent a flood of sites; the
+  service never reads an address, so it makes each new site cost time instead, and the price rises
+  by one bit for every 50 new sites in a day. On a busy day the service asks for more; the CLI makes
+  it and tells you to run `share` again.
+- A part with less than a full week (a 4-day log, an export whose last week is cut short) is left
+  out and the CLI says so. `--endpoint` (or `SITEREAD_SHARE_URL`) sends elsewhere: `https://` only,
+  plain `http://` only to localhost.
+
+In the library, `sharePayload(digest, { id })` returns `{ payload, notes }`, and `validateShare(body)`
+is the endpoint's check.
 
 ## Library
 
@@ -98,7 +180,8 @@ verifier.check('216.73.216.12', 'Mozilla/5.0 … ClaudeBot/1.0; +claudebot@anthr
 | `now` | `Date.now` | a clock, for tests |
 | `log` | `console` | `{ warn, error }`; pass `{}` for silence |
 | `isKnownScannerIp` | none | `(ip) => boolean`, the 444 rule (below) |
-| `feedUrl` | none | one merged document instead of the vendor URLs (below) |
+| `feedUrl` | none | one merged document instead of the vendor URLs (below); `FEED_URL` is the hosted one |
+| `fallback` | `false` | with `feedUrl`: fetch from the vendors what the feed could not give |
 
 It returns:
 
@@ -213,21 +296,31 @@ All of them answer `{ "creationTime": …, "prefixes": [{ "ipv4Prefix": … } |
   failure, not as an empty list.
 - **Drop:** a copy nobody could refresh for 8 days is dropped, so a vendor's moved ranges cannot mark
   its new IPs as spoofed forever. With no copy, the verdict is `unknown`, never `spoofed`.
-- **Bounds:** a body over 2,000,000 characters, more than 20,000 prefixes, or a prefix broader than
-  /8 (IPv4) or /16 (IPv6) is rejected. A `/0` would verify every address.
+- **Bounds:** a body over 2,000,000 characters or more than 20,000 prefixes is rejected, and a list
+  holding a prefix broader than /16 (IPv4) or /32 (IPv6) is refused whole, keeping the previous
+  copy. The broadest real prefix in any list is a /16 (IPv4) and a /64 (IPv6); a `/0` would verify
+  every address, and since the CLI reads the hosted feed, the limit is what stops a tampered feed
+  from vouching for a whole provider's range. From the feed, a refused list is fetched from its
+  vendor instead.
 
 ## Privacy
 
-Local use sends nothing anywhere except plain GETs of the URLs above, with the User-Agent
-`siteread/<version>`. No hostnames, URLs or visitor IPs leave your machine. There is no reverse-DNS
-check for the same reason: a PTR lookup sends the visitor's IP to your resolver.
+Local use sends nothing anywhere except plain GETs of public crawler lists, with the User-Agent
+`siteread/<version>`: the siteread feed by default, or the vendors' URLs above with `--no-feed` (or
+`SITEREAD_FEED_URL=off`), and those vendors whenever the feed fails. The feed's service keeps no
+address; it counts fetches per day by siteread version. No hostnames, URLs or visitor IPs leave
+your machine. There is no reverse-DNS
+check for the same reason: a PTR lookup sends the visitor's IP to your resolver. `share` is the one
+exception: opt-in, its exact payload printed first, aggregate numbers only.
 
 `feedUrl` points the verifier at one merged document,
-`{ "lists": { "<agent>": { "prefixes": [...], "fetchedAt": "..." } } }`, instead of the vendors. It
-is for a hosted feed that is planned but **not live yet**, so it is off by default and strictly opt-in.
-Every prefix from a feed is re-validated like a vendor's, and a `fetchedAt` in the future is clamped
-to now.
+`{ "lists": { "<agent>": { "prefixes": [...], "fetchedAt": "..." } } }`, instead of the vendors: the
+hosted feed is `https://api.siteread.dev/v1/lists.json` (`FEED_URL`, this repo's `cloud/`, rebuilt
+from the vendors every 6 h). The CLI uses it by default with `fallback: true`, which fetches from the
+vendors whatever the feed could not give; the library leaves both off unless you pass them. Every
+prefix from a feed is re-validated like a vendor's, and a `fetchedAt` in the future is clamped to
+now.
 
 ## License
 
-MIT. Extracted from the pipeline Synacl Labs runs on synacl.com.
+MIT.

package/bin/siteread.js

@@ -4,15 +4,21 @@
 //   siteread check <ip> "<user-agent>"   exit 0 verified · 2 spoofed · 3 unknown / n/a · 1 usage or error
 //   siteread lists
 //   siteread digest --gsc <export> --log <file> …   JSON for the seo-read skill's weekly memo
+//   siteread share [<digest>]           opt-in: one week's aggregates → the benchmark, where you stand back
 //   siteread --help
 //
 // Lists are cached in one file (os.tmpdir()/siteread/lists.json, or $SITEREAD_CACHE) and
-// refetched only when older than 24 h. The only network traffic is GETs of the vendors' lists.
+// refetched only when older than 24 h, from the siteread feed (one GET of public data; the service
+// keeps no address) or, with --no-feed and whenever the feed fails, from each vendor. The only
+// other traffic is `share`, which prints the exact bytes it would send and sends only on a yes.
 
 import os from 'node:os';
 import path from 'node:path';
-import { readFileSync, writeFileSync, mkdirSync, readdirSync } from 'node:fs';
-import { createVerifier, fileStore, detectAgent, detectCrawler, digest, parseGoal, VERSION } from '../src/index.js';
+import { createHash, randomBytes } from 'node:crypto';
+import { createInterface } from 'node:readline/promises';
+import { readFileSync, writeFileSync, mkdirSync, readdirSync, statSync } from 'node:fs';
+import { createVerifier, fileStore, detectAgent, detectCrawler, digest, parseGoal, sharePayload, SHARE_METRICS, SHARE_URL, FEED_URL, VERSION } from '../src/index.js';
+import { leadingZeroBits, PROOF_BASE_BITS, PROOF_PREFIX } from '../src/share.js';
 
 const EXIT = { verified: 0, spoofed: 2, unknown: 3, 'n/a': 3, usage: 1 };
 const cacheFile = process.env.SITEREAD_CACHE || path.join(os.tmpdir(), 'siteread', 'lists.json');
@@ -24,16 +30,24 @@ Usage:
   siteread lists                       every vendor list: prefixes and when it was fetched
   siteread digest [options]            a Search Console export + access log -> JSON numbers
                                        for a weekly read (the seo-read skill writes the memo)
+  siteread share [<digest.json|dir>]   opt-in: send one week of a digest's aggregate numbers
+                                       to the public benchmark and see where you stand
+                                       (default: the newest digest in .siteread/digests)
   siteread --help | --version
 
 Options:
   --no-cache   refetch every list now (a copy younger than 24 h is otherwise reused;
                a failed fetch still falls back to the cached copy)
+  --no-feed    fetch each list from its vendor instead of the siteread feed
+               (also SITEREAD_FEED_URL=off; SITEREAD_FEED_URL=<url> reads another feed)
   --json       print the full result as JSON
 
 Digest options (at least one of --gsc / --log):
   --gsc <zip|dir>        Search Console Performance export (Export -> Download CSV)
-  --log <file>           access log, nginx/Apache combined, .gz ok (repeat for several)
+  --log <file>           access log: nginx/Apache combined, Caddy JSON or nginx JSON, .gz ok
+                         (repeat for several)
+  --host <name>          keep only requests to this host, for a log of several sites (repeatable;
+                         JSON logs carry the host, combined does not)
   --site <host>          your site's host (guessed from Referers otherwise)
   --sitemap <file>       a local sitemap.xml: count crawler fetches on its URLs only
   --from / --to <day>    keep only log days in this range (YYYY-MM-DD)
@@ -50,6 +64,13 @@ Digest options (at least one of --gsc / --log):
   --limit <n>            rows per table (default 25)
   --offline              use cached vendor lists only; no network at all
 
+Share options (prints the exact payload first, then asks; never hostnames, URLs or IPs):
+  --dry-run              print the payload and stop
+  --yes                  send without asking (scripts; without a terminal it is required)
+  --endpoint <url>       where to send it (default ${SHARE_URL}, or $SITEREAD_SHARE_URL)
+  --id-file <file>       this site's random id (default .siteread/share-id, made on first use;
+                         delete it to start over as a new site)
+
 Exit codes (check):
   0 verified   the IP is in the vendor's published list for that agent
   2 spoofed    the vendor publishes a list and the IP is not in it
@@ -57,11 +78,12 @@ Exit codes (check):
   1            usage error, or not an IP address
 
 Cache: ${cacheFile}  (override with SITEREAD_CACHE)
-Nothing leaves this machine except GETs of the vendors' public IP lists.`;
+Nothing leaves this machine except GETs of public crawler lists (the siteread feed, or with
+--no-feed the vendors' own), and a share you approve.`;
 
 // Options that take a value (digest only). Each keeps every value given, in order.
 const VALUE_OPTS = new Set(['--gsc', '--log', '--out', '--site', '--sitemap', '--from', '--to', '--split', '--goal',
-    '--team-path', '--exclude-ip', '--allow-path', '--tz', '--prev', '--limit']);
+    '--team-path', '--exclude-ip', '--allow-path', '--tz', '--prev', '--limit', '--endpoint', '--id-file', '--host']);
 
 function parseArgs(argv) {
     const flags = new Set();
@@ -137,10 +159,18 @@ function describe(ip, ua, r, list) {
     }
 }
 
+// The feed by default, as the charter has it once the feed exists: one GET instead of nine, of
+// public data, from a service that keeps no address. What it cannot give comes from the vendors.
+function newVerifier(flags) {
+    const env = process.env.SITEREAD_FEED_URL;
+    const feedUrl = flags.has('--no-feed') || env === 'off' ? null : (env || FEED_URL);
+    return createVerifier({ store: fileStore(cacheFile), log: {}, feedUrl, fallback: true });
+}
+
 async function cmdCheck({ flags, pos }) {
     if (pos.length !== 3) return usage('check takes exactly two arguments: <ip> "<user-agent>" (quote the user-agent)');
     const [, ip, ua] = pos;
-    const verifier = createVerifier({ store: fileStore(cacheFile), log: {} });
+    const verifier = newVerifier(flags);
     let r = verifier.check(ip, ua);
     if (r.reason === 'invalid-ip') return usage(`"${ip}" is not an IP address`);
     // Only touch the network when the claimed agent has a list to check against.
@@ -160,7 +190,7 @@ async function cmdCheck({ flags, pos }) {
 
 async function cmdLists({ flags, pos }) {
     if (pos.length !== 1) return usage('lists takes no arguments');
-    const verifier = createVerifier({ store: fileStore(cacheFile), log: {} });
+    const verifier = newVerifier(flags);
     const result = await verifier.refresh({ force: flags.has('--no-cache') });
     reportFailures(result, verifier);
     const rows = verifier.lists();
@@ -203,7 +233,7 @@ async function cmdDigest({ flags, opts, pos }) {
 
     let verifier = null;
     if (logs.length) {
-        verifier = createVerifier({ store: fileStore(cacheFile), log: {} });
+        verifier = newVerifier(flags);
         if (flags.has('--offline')) {
             await verifier.load();
         } else {
@@ -220,6 +250,7 @@ async function cmdDigest({ flags, opts, pos }) {
         site: one('--site'), tz, from: one('--from'), to: one('--to'), goals,
         teamPath: re('--team-path'), allowPath: re('--allow-path'),
         excludeIps: new Set((opts.get('--exclude-ip') ?? []).flatMap(s => s.split(',')).map(s => s.trim()).filter(Boolean)),
+        hosts: opts.get('--host') ?? [],
     });
     if (verifier) out.vendorLists = verifier.lists().map(({ agent, prefixes, fetchedAt }) => ({ agent, prefixes, fetchedAt }));
     for (const w of out.warnings) process.stderr.write(`siteread: ${w}\n`);
@@ -247,13 +278,157 @@ async function cmdDigest({ flags, opts, pos }) {
     return 0;
 }
 
+const USER_AGENT = `siteread/${VERSION} (+https://www.npmjs.com/package/siteread)`;
+const DIGEST_RE = /^\d{4}-\d{2}-\d{2}\.json$/;
+// A server's words go to the terminal: no escape sequences from someone else's endpoint.
+const clean = s => String(s ?? '').replace(/[\u0000-\u001f\u007f-\u009f]/g, '');
+const BAND_TEXT = Object.freeze({
+    'below-p25': 'below the 25th percentile', 'p25-p50': '25th to 50th percentile',
+    'p50-p75': '50th to 75th percentile', 'p75-up': '75th percentile or above',
+});
+
+/** The site's random share id, made (0600) on first use. */
+function shareId(file) {
+    let id;
+    try { id = readFileSync(file, 'utf8').trim(); } catch (err) {
+        if (err.code !== 'ENOENT') throw err;
+        id = randomBytes(16).toString('hex');
+        mkdirSync(path.dirname(file), { recursive: true });
+        writeFileSync(file, `${id}\n`, { mode: 0o600, flag: 'wx' });
+        process.stderr.write(`siteread: made a random id for this site in ${file} (delete it to start over as a new site)\n`);
+    }
+    if (!/^[0-9a-f]{32}$/.test(id)) throw new UsageError(`${file} does not hold a share id; delete it to make a new one`);
+    return id;
+}
+
+// A new site's proof of work (src/share.js, PROOF OF WORK), kept next to the id as "nonce bits id"
+// so it is made once. A file for another id, or one that does not hold up, is made again.
+function readProof(file, id) {
+    let parts;
+    try { parts = readFileSync(file, 'utf8').trim().split(/\s+/); } catch { return null; }
+    const [nonce, bits, forId] = parts;
+    if (forId !== id || !/^\d{1,20}$/.test(nonce ?? '')) return null;
+    const got = leadingZeroBits(createHash('sha256').update(`${PROOF_PREFIX}${id}:${nonce}`).digest());
+    return got >= Number(bits) ? { nonce, bits: got } : null;
+}
+
+function makeProof(file, id, bits) {
+    process.stderr.write(`siteread: making this site's proof of work (${bits} bits; seconds on a laptop, once, so made-up sites cost real time)\n`);
+    const t0 = Date.now();
+    for (let n = 0; ; n++) {
+        const nonce = String(n);
+        const got = leadingZeroBits(createHash('sha256').update(`${PROOF_PREFIX}${id}:${nonce}`).digest());
+        if (got >= bits) {
+            mkdirSync(path.dirname(file), { recursive: true });
+            writeFileSync(file, `${nonce} ${got} ${id}\n`, { mode: 0o600 });
+            process.stderr.write(`siteread: made it in ${((Date.now() - t0) / 1000).toFixed(1)} s, kept in ${file}\n`);
+            return { nonce, bits: got };
+        }
+    }
+}
+
+// Plain http only to this machine: a share crosses the internet over TLS or not at all.
+function endpointOk(raw) {
+    let u;
+    try { u = new URL(raw); } catch { return false; }
+    if (u.protocol === 'https:') return true;
+    return u.protocol === 'http:' && ['localhost', '127.0.0.1', '[::1]'].includes(u.hostname);
+}
+
+async function cmdShare({ flags, opts, pos }) {
+    if (pos.length > 2) return usage('share takes at most one digest file or folder');
+    const one = k => opts.get(k)?.at(-1) ?? null;
+    let file = pos[1] ?? path.join('.siteread', 'digests');
+    if (statSync(file).isDirectory()) {
+        const newest = readdirSync(file).filter(f => DIGEST_RE.test(f)).sort().at(-1);
+        if (!newest) return usage(`no digest (YYYY-MM-DD.json) in ${file}; run "siteread digest --out ${file}" first`);
+        file = path.join(file, newest);
+    }
+    let d;
+    try { d = JSON.parse(readFileSync(file, 'utf8')); } catch (err) { return usage(`${file}: ${err.message}`); }
+    const endpoint = one('--endpoint') ?? process.env.SITEREAD_SHARE_URL ?? SHARE_URL;
+    if (!endpointOk(endpoint)) return usage(`--endpoint must be an https:// URL (plain http only to localhost): ${endpoint}`);
+
+    let built;
+    const idFile = one('--id-file') ?? path.join('.siteread', 'share-id');
+    const proofFile = path.join(path.dirname(idFile), 'share-proof');
+    try {
+        const id = shareId(idFile);
+        // A known site never needs its proof again; a new one needs it at today's price, which the
+        // service names when it asks for more. SITEREAD_PROOF_BITS only sets where the first try starts.
+        const proof = readProof(proofFile, id) ?? makeProof(proofFile, id, Number(process.env.SITEREAD_PROOF_BITS) || PROOF_BASE_BITS);
+        built = sharePayload(d, { id, proof: proof.nonce });
+    } catch (err) {
+        if (err instanceof UsageError) throw err;
+        process.stderr.write(`siteread: share: ${err.message}\n`);
+        return 1;
+    }
+    // These exact bytes are what goes over the wire: print, then send the same string.
+    const body = JSON.stringify(built.payload, null, 2);
+    process.stderr.write(`siteread: the whole payload from ${file}, byte for byte, for ${endpoint}:\n`);
+    process.stdout.write(`${body}\n`);
+    for (const n of built.notes) process.stderr.write(`siteread: ${n}\n`);
+    if (flags.has('--dry-run')) { process.stderr.write('siteread: not sent (--dry-run)\n'); return 0; }
+    if (!flags.has('--yes')) {
+        if (!process.stdin.isTTY) { process.stderr.write('siteread: not sent: no terminal to ask on; pass --yes to send it\n'); return 1; }
+        const rl = createInterface({ input: process.stdin, output: process.stderr });
+        const answer = await rl.question('Send it? [y/N] ');
+        rl.close();
+        if (!/^y(es)?$/i.test(answer.trim())) { process.stderr.write('siteread: not sent\n'); return 0; }
+    }
+
+    let res, r = null;
+    try {
+        res = await fetch(endpoint, {
+            method: 'POST', body, signal: AbortSignal.timeout(15_000),
+            headers: { 'content-type': 'application/json', accept: 'application/json', 'user-agent': USER_AGENT },
+        });
+        try { r = JSON.parse(await res.text()); } catch { r = null; }
+    } catch (err) {
+        process.stderr.write(`siteread: could not reach ${endpoint}: ${err.cause?.code || err.message}\n`);
+        return 1;
+    }
+    if (!res.ok || r?.ok !== true) {
+        process.stderr.write(`siteread: ${endpoint} did not take it: HTTP ${res.status}${r?.error ? `: ${clean(r.error)}` : ''}\n`);
+        // A busy day for new sites: make the stronger proof now, and let the user send again (the
+        // payload changes, so it is printed again before anything goes out).
+        const need = Number(r?.need);
+        if (Number.isInteger(need) && need > 0 && need <= 40) {
+            makeProof(proofFile, built.payload.id, need);
+            process.stderr.write('siteread: run share again to send it with the new proof\n');
+        }
+        return 1;
+    }
+    // Where each number falls among the quartiles the benchmark publishes, and only where it
+    // publishes them: the service never answers finer than the public page (cloud/src/benchmark.js).
+    const a = r.against;
+    const min = Number(a?.minSites) || 10;
+    const rows = Object.entries(r.metrics ?? {}).filter(([, m]) => m && typeof m === 'object');
+    const where = m => BAND_TEXT[m.band] ?? `${Number(m.sites) || 0} site${Number(m.sites) === 1 ? '' : 's'}; shown from ${min}`;
+    const w = Math.max(6, ...rows.map(([k]) => k.length));
+    const ww = Math.max(5, ...rows.map(([, m]) => where(m).length));
+    const out = [
+        a
+            ? `sent ${clean(r.week)}. Against the benchmark built ${clean(a.generatedAt).slice(0, 16).replace('T', ' ')} UTC: ${Number(a.sites) || 0} anonymous site${Number(a.sites) === 1 ? '' : 's'}, ${clean(a.window?.from)} to ${clean(a.window?.to)}`
+            : `sent ${clean(r.week)}. The benchmark has not been built yet; it is rebuilt every hour.`,
+        '',
+        `${'metric'.padEnd(w)}  ${'yours'.padStart(9)}  ${'where'.padEnd(ww)}`,
+        ...rows.map(([k, m]) => `${clean(k).padEnd(w)}  ${clean(m.value).padStart(9)}  ${where(m).padEnd(ww)}`.trimEnd()
+            + (SHARE_METRICS[k]?.lowerIsBetter && m.band ? '  (lower is better)' : '')),
+    ];
+    if (!rows.some(([, m]) => m.band)) out.push('', `No number has ${min} anonymous sites yet, so none shows where you stand; your share counts toward them.`);
+    if (r.benchmark) out.push('', `benchmark: ${clean(r.benchmark)}`);
+    process.stdout.write(`${out.join('\n')}\n`);
+    return 0;
+}
+
 async function main(argv) {
     let args;
     try { args = parseArgs(argv); } catch (err) { if (err instanceof UsageError) return usage(err.message); throw err; }
     const { flags, pos } = args;
     if (flags.has('--help') || flags.has('-h') || pos[0] === 'help') { process.stdout.write(`${HELP}\n`); return 0; }
     if (flags.has('--version') || flags.has('-v')) { process.stdout.write(`${VERSION}\n`); return 0; }
-    const known = new Set(['--no-cache', '--json', '--offline']);
+    const known = new Set(['--no-cache', '--no-feed', '--json', '--offline', '--dry-run', '--yes']);
     const bad = [...flags].find(f => !known.has(f));
     if (bad) return usage(`unknown option ${bad}`);
     if (pos[0] === 'check') return cmdCheck(args);
@@ -267,6 +442,14 @@ async function main(argv) {
             return 1;
         }
     }
+    if (pos[0] === 'share') {
+        try { return await cmdShare(args); } catch (err) {
+            if (err instanceof UsageError) return usage(err.message);
+            if (err?.code === 'ENOENT') return usage(`no such file: ${err.path}`);
+            process.stderr.write(`siteread: share: ${err?.message || err}\n`);
+            return 1;
+        }
+    }
     if (!pos.length) { process.stdout.write(`${HELP}\n`); return EXIT.usage; }
     return usage(`unknown command "${pos[0]}"`);
 }

package/package.json

@@ -1,7 +1,16 @@
 {
   "name": "siteread",
-  "version": "0.2.0",
+  "version": "0.2.2",
   "description": "Is that crawler who it says it is? Checks an IP + User-Agent against the AI and search vendors' published IP lists. Zero dependencies.",
+  "homepage": "https://siteread.dev",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/synacl-iot/siteread.git",
+    "directory": "packages/verify"
+  },
+  "bugs": {
+    "url": "https://github.com/synacl-iot/siteread/issues"
+  },
   "type": "module",
   "engines": {
     "node": ">=20"
@@ -19,7 +28,7 @@
     "README.md"
   ],
   "scripts": {
-    "test": "node --test"
+    "test": "node --test test/*.test.js"
   },
   "keywords": [
     "crawler",

package/src/agents.js

@@ -1,9 +1,6 @@
 // Who a request CLAIMS to be, from its User-Agent. A claim only: anyone can send any UA, and
-// on the site this was extracted from most AI user-agents outside the vendors' IP lists were a
-// scanner cycling through them (ip-ranges.js has the numbers). verifier.js turns a claim into a
-// verdict.
-//
-// Ported from the AI-traffic classifier Synacl Labs runs on synacl.com (2026-10).
+// on one site's log most AI user-agents outside the vendors' IP lists were a scanner cycling
+// through them (ip-ranges.js has the numbers). verifier.js turns a claim into a verdict.
 //
 // purpose: `answer`   fetched live while answering someone (ChatGPT-User, Perplexity-User)
 //          `search`   building the assistant's search index (OAI-SearchBot, PerplexityBot)

package/src/digest/gsc.js

@@ -1,7 +1,7 @@
 // A Google Search Console Performance export (the "Export → Download CSV" zip, or the folder it
 // unzips to) → the numbers a weekly read needs.
 //
-// The method is the one that held up over six weekly reads of synacl.com (2026-09 → 10):
+// The method is the one that held up over six weekly reads of one site (2026-09 → 10):
 //   - Totals come from the daily series (Dates.csv; older exports call it Chart.csv), never by
 //     summing Pages.csv: a page table double-counts an impression that showed two URLs.
 //   - Position is impression-weighted (Σ position·impressions / Σ impressions). A plain mean

package/src/digest/index.js

@@ -10,7 +10,7 @@ import { VERSION } from '../version.js';
 
 // GSC clicks vs people the log saw arrive from Google, over the days both cover. They are
 // different counts (a click without a Referer reads as direct; one person can click twice), so
-// this is a sanity check, never a sum. On synacl.com they tracked within 10 % day by day.
+// this is a sanity check, never a sum. On one site they tracked within 10 % day by day.
 function crossCheck(gsc, log) {
     const from = gsc.range.from > log.range.from ? gsc.range.from : log.range.from;
     const to = gsc.range.to < log.range.to ? gsc.range.to : log.range.to;
@@ -61,8 +61,13 @@ export async function digest({ gsc = null, logs = [], verifier = null, prev = nu
         for (const f of logs) await readLog(f, d);
         out.log = d.finish(verifier);
         if (out.log.lines.unparsed > out.log.lines.total / 2) {
-            out.warnings.push(`log: ${out.log.lines.unparsed} of ${out.log.lines.total} lines are not in combined format; counts are from the rest`);
+            out.warnings.push(`log: ${out.log.lines.unparsed} of ${out.log.lines.total} lines are in no format siteread reads (combined, Caddy JSON, nginx JSON); counts are from the rest`);
         }
+        const { total, unparsed, otherHost, noHost } = out.log.lines;
+        if (logOpts.hosts?.length && otherHost && otherHost === total - unparsed) {
+            out.warnings.push(`log: no request was to ${logOpts.hosts.join(' or ')}; every line was for another host`);
+        }
+        if (noHost) out.warnings.push(`log: ${noHost} lines carry no host, so --host could not filter them; they are counted`);
         if (out.log.site.guessed) out.warnings.push(`log: site host guessed as ${out.log.site.host ?? 'unknown'} from Referers; pass --site to set it`);
         if (out.gsc && out.log.range.from) {
             const a = out.gsc.range, b = out.log.range;

package/src/digest/log.js

@@ -1,6 +1,7 @@
-// An access log (nginx / Apache "combined", plain or .gz) → people, sources and crawlers.
+// An access log (nginx / Apache "combined", Caddy's JSON or an nginx JSON log_format; plain or .gz)
+// → people, sources and crawlers. A file that holds several sites is cut to one with `hosts`.
 //
-// Who counts as a person is the rule synacl.com settled on after its first counts were wrong
+// Who counts as a person is the rule one site settled on after its first counts were wrong
 // (2026-09-28, stricter than the read a week earlier, so compare only reads made the same way):
 //   a person = an IP that sent no bot User-Agent, never asked for a scanner path or drew a 444,
 //   and loaded at least one asset (CSS/JS/font/image). Link checkers, previewers and most bots
@@ -11,7 +12,7 @@
 // Referer from a cloud provider's address space is set aside and counted on its own line.
 //
 // Crawlers are counted on page fetches (and on sitemap URLs when a sitemap is given), each IP
-// checked against its vendor's list. The raw UA count lies: one week synacl.com's log showed
+// checked against its vendor's list. The raw UA count lies: one week that site's log showed
 // "Googlebot 6.9k", and it was scanners wearing Googlebot's UA against the bare IP. Googlebot
 // also crawls from 192.178.x, not only 66.249.x: a first-octet guess undercounted it by a third.
 
@@ -39,19 +40,108 @@ const GENERIC_BOT_RE = /bot\b|bot\/|crawl|spider|slurp|fetch|scan|curl\/|wget\/|
 export const CLOUD_FIRST_OCTETS = Object.freeze([3, 13, 15, 16, 18, 34, 35, 44, 52, 54]);
 const CLOUD = new Set(CLOUD_FIRST_OCTETS);
 
-/** One combined-format line → request, or null. */
+/** "05/Oct/2026:20:00:00 +0200" (nginx $time_local, Apache %t) → ms, or NaN. */
+function clfTime(s) {
+    const t = /^(\d{2})\/(\w{3})\/(\d{4}):(\d{2}):(\d{2}):(\d{2}) ([+-])(\d{2})(\d{2})$/.exec(s);
+    if (!t || !(t[2] in MONTHS)) return NaN;
+    const off = (t[7] === '-' ? -1 : 1) * (Number(t[8]) * 60 + Number(t[9]));
+    return Date.UTC(+t[3], MONTHS[t[2]], +t[1], +t[4], +t[5], +t[6]) - off * 60_000;
+}
+
+/**
+ * One access-log line → { ip, ts, method, target, status, referer, ua, host }, or null. Combined
+ * format, or a JSON object per line: Caddy's (it has a `request` object) or an nginx log_format.
+ * `host` is null when the format does not carry one (combined).
+ */
 export function parseLine(line) {
+    if (line.charCodeAt(0) === 123) return parseJsonLine(line);
     const m = LINE_RE.exec(line);
     if (!m) return null;
     const [, ip, time, method, target, status, referer = '-', ua = '-'] = m;
-    const t = /^(\d{2})\/(\w{3})\/(\d{4}):(\d{2}):(\d{2}):(\d{2}) ([+-])(\d{2})(\d{2})$/.exec(time);
-    if (!t || !(t[2] in MONTHS)) return null;
-    const off = (t[7] === '-' ? -1 : 1) * (Number(t[8]) * 60 + Number(t[9]));
-    const ts = Date.UTC(+t[3], MONTHS[t[2]], +t[1], +t[4], +t[5], +t[6]) - off * 60_000;
+    const ts = clfTime(time);
+    if (!Number.isFinite(ts)) return null;
     const unesc = s => s.replace(/\\x22/g, '"').replace(/\\"/g, '"');
-    return { ip, ts, method, target, status: Number(status), referer: unesc(referer), ua: unesc(ua) };
+    return { ip, ts, method, target, status: Number(status), referer: unesc(referer), ua: unesc(ua), host: null };
+}
+
+// nginx writes JSON by hand from a log_format. With `escape=json` it is valid JSON; with the
+// default escaping a quote or a non-ASCII byte comes out as \xHH, which JSON does not allow, so a
+// line that fails is read again with those as \u00HH.
+function parseJsonObject(line) {
+    try { return JSON.parse(line); } catch { /* try the nginx default escaping */ }
+    if (!line.includes('\\x')) return null;
+    try { return JSON.parse(line.replace(/\\x([0-9a-fA-F]{2})/g, '\\u00$1')); } catch { return null; }
+}
+
+function parseJsonLine(line) {
+    const e = parseJsonObject(line);
+    if (!e || typeof e !== 'object' || Array.isArray(e)) return null;
+    return e.request && typeof e.request === 'object' ? caddyEntry(e) : nginxEntry(e);
 }
 
+// Caddy writes its access log as JSON (the `log` directive's default) and cannot write "combined"
+// without a plugin a distribution's package does not carry; siteread.dev's own log is one. Caddy
+// has no 444, so a scanner there shows by its path alone, which the 444 rule reads anyway.
+// `client_ip` (Caddy ≥ 2.7, after trusted proxies) wins over `remote_ip`; `ts` is Unix seconds
+// unless the log was configured with another time format.
+function caddyEntry(e) {
+    const r = e.request;
+    const ip = r.client_ip || r.remote_ip;
+    const ts = typeof e.ts === 'number' ? Math.round(e.ts * 1000) : Date.parse(e.ts);
+    if (typeof ip !== 'string' || !ip || !Number.isFinite(ts) || typeof r.method !== 'string' || typeof r.uri !== 'string'
+        || !Number.isInteger(e.status)) return null;
+    const first = k => (Array.isArray(r.headers?.[k]) && typeof r.headers[k][0] === 'string' ? r.headers[k][0] : '-');
+    return { ip, ts, method: r.method, target: r.uri, status: e.status, referer: first('Referer'), ua: first('User-Agent'), host: str(r.host) };
+}
+
+// An nginx JSON log_format names its fields as its author chose. These are the names in common use
+// for each, first match wins: the variable's own name (remote_addr, http_user_agent) or a short one
+// (ip, user_agent). The target is the raw $request_uri where there is one, since a scanner path
+// is read undecoded; $uri is decoded, so it is the fallback, with $args put back.
+const NGINX = {
+    ip: ['remote_addr', 'client_ip', 'remote_ip', 'ip', 'client', 'clientip'],
+    time: ['time', 'time_iso8601', 'timestamp', '@timestamp', 'time_local', 'msec', 'ts'],
+    method: ['method', 'request_method'],
+    target: ['request_uri'],
+    uri: ['uri', 'path'],
+    args: ['args', 'query_string'],
+    status: ['status', 'status_code', 'response'],
+    referer: ['referer', 'http_referer', 'referrer', 'http_referrer'],
+    ua: ['user_agent', 'http_user_agent', 'agent', 'useragent', 'ua'],
+    host: ['host', 'http_host', 'server_name', 'vhost', 'hostname'],
+};
+const str = v => (typeof v === 'string' && v !== '' ? v : null);
+const pick = (e, keys) => { for (const k of keys) { const v = e[k]; if (v != null && v !== '') return v; } return null; };
+
+function nginxEntry(e) {
+    const ip = str(pick(e, NGINX.ip));
+    if (!ip) return null;
+    let method = str(pick(e, NGINX.method));
+    let target = str(pick(e, NGINX.target));
+    if (!target) {
+        const uri = str(pick(e, NGINX.uri));
+        const args = str(pick(e, NGINX.args));
+        if (uri) target = args && args !== '-' ? `${uri}?${args}` : uri;
+    }
+    // Or the request line itself, "GET /path?q HTTP/1.1" ($request).
+    const line = typeof e.request === 'string' ? /^(\S+) (\S+)/.exec(e.request) : null;
+    if (line) { method ??= line[1]; target ??= line[2]; }
+    const t = pick(e, NGINX.time);
+    // $msec is seconds with a fraction; a number that large is already milliseconds.
+    const ts = typeof t === 'number' ? Math.round(t < 1e11 ? t * 1000 : t)
+        : typeof t === 'string' ? (/^\d+(\.\d+)?$/.test(t) ? Math.round(Number(t) * 1000) : Date.parse(t) || clfTime(t)) : NaN;
+    const rawStatus = pick(e, NGINX.status);
+    const status = rawStatus == null ? NaN : Number(rawStatus);   // Number(null) would be a status 0
+    if (!method || !target || !Number.isFinite(ts) || !Number.isInteger(status)) return null;
+    return {
+        ip, ts, method, target, status,
+        referer: str(pick(e, NGINX.referer)) ?? '-', ua: str(pick(e, NGINX.ua)) ?? '-', host: str(pick(e, NGINX.host)),
+    };
+}
+
+/** A host as a filter compares it: lower case, no port, no trailing dot. */
+export const normHost = h => String(h).trim().toLowerCase().replace(/:\d+$/, '').replace(/\.$/, '');
+
 /** Day (YYYY-MM-DD) of a timestamp in an IANA zone. */
 export function dayIn(tz) {
     const f = new Intl.DateTimeFormat('en-CA', { timeZone: tz, year: 'numeric', month: '2-digit', day: '2-digit' });
@@ -76,18 +166,21 @@ function top(map, limit) {
  * @param {RegExp} [o.teamPath]     an IP that got a 2xx on such a path is the site's own team
  * @param {Set<string>} [o.excludeIps]
  * @param {RegExp} [o.allowPath]    a site's own paths that look like scanner probes
+ * @param {string[]} [o.hosts]      keep only requests to these hosts (a log of several sites);
+ *                                  a line whose format carries no host is kept and counted
  */
 export function createLogDigest({
     site = null, tz = 'America/Los_Angeles', from = null, to = null, sitemap = null,
-    goals = [], teamPath = null, excludeIps = new Set(), allowPath = null, limit = 15,
+    goals = [], teamPath = null, excludeIps = new Set(), allowPath = null, limit = 15, hosts = [],
 } = {}) {
+    const keepHosts = hosts.length ? new Set(hosts.map(normHost)) : null;
     const day = dayIn(tz);
     const ips = new Map();       // ip → per-IP facts
     const crawl = new Map();     // `${agent}\t${ip}` → { agent, ip, ua, requests, pages: Map(day → [path]) }
     const generic = { requests: 0, ips: new Set(), uas: new Map() };
     const scannerPaths = new Map();
     const refHosts = new Map();
-    let lines = 0, unparsed = 0, outOfRange = 0, firstDay = null, lastDay = null;
+    let lines = 0, unparsed = 0, outOfRange = 0, otherHost = 0, noHost = 0, firstDay = null, lastDay = null;
 
     function ipRec(ip) {
         let r = ips.get(ip);
@@ -102,6 +195,10 @@ export function createLogDigest({
         lines++;
         const q = parseLine(line);
         if (!q) { unparsed++; return; }
+        if (keepHosts) {
+            if (q.host == null) noHost++;
+            else if (!keepHosts.has(normHost(q.host))) { otherHost++; return; }
+        }
         const d = day(q.ts);
         if ((from && d < from) || (to && d > to)) { outOfRange++; return; }
         if (!firstDay || d < firstDay) firstDay = d;
@@ -150,8 +247,9 @@ export function createLogDigest({
     }
 
     function finish(verifier) {
-        // The site's own host: given, or the Referer host browsers' asset loads carry most.
-        const siteHost = site ? site.replace(/^www\./, '') : (top(refHosts, 1)[0]?.[0] ?? null);
+        // The site's own host: given, the first --host, or the Referer host asset loads carry most.
+        const given = site ?? hosts[0] ?? null;
+        const siteHost = given ? normHost(given).replace(/^www\./, '') : (top(refHosts, 1)[0]?.[0] ?? null);
 
         // People.
         const people = [];
@@ -252,7 +350,7 @@ export function createLogDigest({
                 a = {
                     agent: c.agent, kind: detectAgent(c.ua) ? 'ai' : 'crawler', purpose: e?.purpose ?? null,
                     requests: 0, ips: 0,
-                    verified: { ips: 0, requests: 0, pageFetches: 0, sitemapFetches: 0, pages: new Set(), perDay: new Map() },
+                    verified: { ips: 0, requests: 0, pageFetches: 0, sitemapFetches: 0, pages: new Set(), perDay: new Map(), perDayAll: new Map() },
                     spoofed: { ips: 0, requests: 0 },
                     unknown: { ips: 0, requests: 0, pageFetches: 0 },
                 };
@@ -267,6 +365,7 @@ export function createLogDigest({
                 for (const [d, ps] of c.pages) {
                     const counted = sitemap ? ps.filter(p => sitemap.has(p)) : ps;
                     countBy(b.perDay, d, counted.length);
+                    countBy(b.perDayAll, d, ps.length);
                     b.sitemapFetches += sitemap ? counted.length : 0;
                     for (const p of ps) b.pages.add(p);
                 }
@@ -289,7 +388,7 @@ export function createLogDigest({
                         sitemapFetches: sitemap ? v.sitemapFetches : null,
                         uniquePages: v.pages.size, sitemapCovered: covered,
                         offSitemapPages: offSitemap ? offSitemap.slice(0, 10) : null,
-                        perWeek: weekly(v.perDay, lastDay, firstDay),
+                        perWeek: weekly(v.perDay, lastDay, firstDay, v.perDayAll),
                     },
                     spoofed: a.spoofed,
                     unknown: a.unknown,
@@ -301,8 +400,8 @@ export function createLogDigest({
 
         return {
             range: { from: firstDay, to: lastDay, tz },
-            site: { host: siteHost, guessed: !site },
-            lines: { total: lines, unparsed, outOfRange },
+            site: { host: siteHost, guessed: !given },
+            lines: { total: lines, unparsed, outOfRange, otherHost, noHost },
             people: {
                 total: people.length,
                 perDay: dayList,
@@ -323,7 +422,7 @@ export function createLogDigest({
     return { add, finish };
 }
 
-// Sitemap URLs no verified search crawler fetched in this log. On synacl.com one clean guide was
+// Sitemap URLs no verified search crawler fetched in this log. On one site a clean guide was
 // never fetched by Googlebot in 21 days and had 0 impressions; it was linked from too few pages.
 function sitemapCoverage(sitemap, crawlers, crawl, limit) {
     const fetched = new Set();
@@ -335,21 +434,25 @@ function sitemapCoverage(sitemap, crawlers, crawl, limit) {
     return { urls: sitemap.size, fetchedBySearch: sitemap.size - never.length, neverFetched: never.slice(0, limit), neverFetchedCount: never.length };
 }
 
-// Verified fetches per 7-day bucket ending on the last day: sitemap URLs only when a sitemap was given.
-function weekly(perDay, lastDay, firstDay) {
+// Verified fetches per 7-day bucket ending on the last day: `fetches` counts sitemap URLs only when
+// a sitemap was given, `pageFetches` every page. `share` sends the second, so a benchmark row
+// means the same thing with or without a sitemap.
+function weekly(perDay, lastDay, firstDay, perDayAll) {
     if (!lastDay) return [];
     const end = Date.parse(`${lastDay}T00:00:00Z`);
-    const b = new Map();
-    for (const [d, n] of perDay) {
-        const w = Math.floor((end - Date.parse(`${d}T00:00:00Z`)) / (7 * 86_400_000));
-        countBy(b, w, n);
-    }
+    const bucket = (byDay) => {
+        const b = new Map();
+        for (const [d, n] of byDay) countBy(b, Math.floor((end - Date.parse(`${d}T00:00:00Z`)) / (7 * 86_400_000)), n);
+        return b;
+    };
+    const b = bucket(perDay);
+    const all = bucket(perDayAll);
     const out = [];
     const nWeeks = Math.floor((end - Date.parse(`${firstDay}T00:00:00Z`)) / (7 * 86_400_000));
     for (let w = nWeeks; w >= 0; w--) {
         const iso = ms => new Date(ms).toISOString().slice(0, 10);
         const lo = iso(end - (7 * w + 6) * 86_400_000);
-        out.push({ from: lo < firstDay ? firstDay : lo, to: iso(end - 7 * w * 86_400_000), fetches: b.get(w) ?? 0 });
+        out.push({ from: lo < firstDay ? firstDay : lo, to: iso(end - 7 * w * 86_400_000), fetches: b.get(w) ?? 0, pageFetches: all.get(w) ?? 0 });
     }
     return out;
 }

package/src/digest/referrer.js

@@ -1,7 +1,7 @@
 // Where a visit came from: the Referer of its first page view, or the landing URL's utm_source.
 //
 // utm_source wins because the AI assistants mostly send no Referer: ChatGPT tags its links
-// `?utm_source=chatgpt.com`, and on synacl.com that tag was the only trace of what became the
+// `?utm_source=chatgpt.com`, and on one site that tag was the only trace of what became the
 // channel with the most tool use per visitor (2026-09). ChatGPT traffic never appears in GSC.
 
 // [source, kind, host test]. First match wins, so gemini.google.com sits before google.

package/src/index.js

@@ -4,7 +4,7 @@
 export { AGENTS, CRAWLERS, detectAgent, detectCrawler, isSearchCrawler } from './agents.js';
 export { SCANNER_PATH_RE, isScannerPath } from './scanner.js';
 export {
-    SOURCES, parseCidr, parseList, normIp,
+    SOURCES, FEED_URL, parseCidr, parseList, normIp,
     REFRESH_AFTER_MS, DROP_AFTER_MS, MIN_PREFIX, MAX_PREFIXES, MAX_BODY_CHARS,
 } from './ip-ranges.js';
 export { createVerifier, memoryStore } from './verifier.js';
@@ -14,3 +14,4 @@ export { digest, parseSitemap, parseGoal } from './digest/index.js';
 export { parseCsv } from './digest/csv.js';
 export { classifySource } from './digest/referrer.js';
 export { parseLine } from './digest/log.js';
+export { sharePayload, validateShare, isoWeek, weekStart, addWeeks, SHARE_METRICS, SHARE_SCHEMA, SHARE_URL, SHARE_REFUSALS } from './share.js';

package/src/ip-ranges.js

@@ -1,10 +1,7 @@
 // The vendors' published crawler IP lists: what tells a real ChatGPT-User from a scanner
-// wearing its User-Agent.
+// wearing its User-Agent. The store is pluggable (verifier.js).
 //
-// Ported from the crawler-verification code Synacl Labs runs on synacl.com (2026-10). There it
-// lived behind a Redis key shared by a cluster; here the store is pluggable (verifier.js).
-//
-// WHY (measured on that site's nginx log, 2026-09-17 → 10-16). A scanner cycles AI user-agents:
+// WHY (measured on one site's nginx log, 2026-09-17 → 10-16). A scanner cycles AI user-agents:
 // 217 of 221 ChatGPT-User requests from outside OpenAI's list came from IPs that also got a 444
 // (nginx's "close the connection, send nothing" answer to a scanner path), and DeepSeekBot /
 // CCBot / Grok / Perplexity-User / Claude-SearchBot were 100 % spoofed. Where a vendor publishes
@@ -35,9 +32,15 @@ export const FETCH_TIMEOUT_MS = 8_000;
 // A list is a few hundred prefixes; these bound what a broken (or hostile) response can cost.
 export const MAX_BODY_CHARS = 2_000_000;
 export const MAX_PREFIXES = 20_000;
-// No vendor publishes anything near this broad. A /0 in a list would verify EVERY address,
-// so a prefix shorter than these is a broken list, not a big one.
-export const MIN_PREFIX = Object.freeze({ ipv4: 8, ipv6: 16 });
+// The broadest prefix a list may hold. A /0 would verify EVERY address, and since the CLI reads
+// the hosted feed by default, the feed's server vouches for every client's "verified": if it were
+// ever taken over, it could add a broad range to any agent and every spoofer in it would pass.
+// Measured 2026-10-22, the broadest real prefix in any list is OpenAI's /16 (chatgpt-user) for
+// IPv4 and Google's /64 for IPv6, so these limits cost nothing real and cut the worst case from
+// a /8 to a /16. A list holding a broader prefix is refused whole (its previous copy kept, the
+// error reported), never trimmed: trimming would silently turn a vendor's real new range into
+// "spoofed", and a refusal says out loud that a limit or a list needs a look.
+export const MIN_PREFIX = Object.freeze({ ipv4: 16, ipv6: 32 });
 // Two processes sharing a store can disagree on the time by a little. A fetchedAt further in
 // the future than this is not skew: it would keep a list "fresh" forever and defeat the drop.
 const CLOCK_SKEW_MS = 5 * 60_000;
@@ -46,6 +49,11 @@ const CLAUDE_BOTS = 'https://claude.com/crawling/bots.json';
 
 // Agent key (agents.js AGENTS / CRAWLERS) → its vendor's published list. Every URL answers
 // { creationTime, prefixes: [{ ipv4Prefix } | { ipv6Prefix }] } (all checked 2026-10-21).
+// The hosted feed (cloud/): every list above in one document, refetched from the vendors every
+// 6 h. The CLI reads it by default (the charter: the feed is opt-out); the library does not, and
+// cloud's own feed job must not, since it is what builds the feed.
+export const FEED_URL = 'https://api.siteread.dev/v1/lists.json';
+
 export const SOURCES = Object.freeze([
     { agent: 'chatgpt-user',     url: 'https://openai.com/chatgpt-user.json' },
     { agent: 'oai-searchbot',    url: 'https://openai.com/searchbot.json' },
@@ -62,8 +70,14 @@ export const SOURCES = Object.freeze([
     { agent: 'bingbot',          url: 'https://www.bing.com/toolbox/bingbot.json' },
 ].map(Object.freeze));
 
-/** 'a.b.c.d/n' or 'x::/n' → { type, net, prefix, cidr } (normalised), or null. */
+/** 'a.b.c.d/n' or 'x::/n' → { type, net, prefix, cidr } (normalised), or null; MIN_PREFIX applies. */
 export function parseCidr(raw) {
+    const p = cidrOf(raw);
+    return p && p.prefix >= MIN_PREFIX[p.type] ? p : null;
+}
+
+/** parseCidr without the breadth limit, so a list parser can tell "too broad" from "not a prefix". */
+function cidrOf(raw) {
     if (typeof raw !== 'string') return null;
     const s = raw.trim();
     const slash = s.indexOf('/');
@@ -74,30 +88,35 @@ export function parseCidr(raw) {
     const prefix = Number(bits);
     const type = net.isIPv4(addr) ? 'ipv4' : net.isIPv6(addr) ? 'ipv6' : null;
     if (!type) return null;
-    if (prefix < MIN_PREFIX[type] || prefix > (type === 'ipv4' ? 32 : 128)) return null;
+    if (prefix > (type === 'ipv4' ? 32 : 128)) return null;
     return { type, net: addr.toLowerCase(), prefix, cidr: `${addr.toLowerCase()}/${prefix}` };
 }
 
-/** The vendors' shared format: { prefixes: [{ ipv4Prefix } | { ipv6Prefix }] } → [cidr]. */
+// One prefix of a list: skipped when it is not a prefix at all, the whole list refused when it
+// is one broader than MIN_PREFIX (see there).
+function addPrefix(out, raw) {
+    const p = cidrOf(raw);
+    if (!p) return;
+    if (p.prefix < MIN_PREFIX[p.type]) throw new Error(`${p.cidr} is broader than /${MIN_PREFIX[p.type]}; list refused`);
+    out.add(p.cidr);
+}
+
+/** The vendors' shared format: { prefixes: [{ ipv4Prefix } | { ipv6Prefix }] } → [cidr]. Throws on a too-broad prefix. */
 export function parseList(body) {
     const items = Array.isArray(body?.prefixes) ? body.prefixes : [];
     const out = new Set();
     for (const it of items) {
         if (out.size >= MAX_PREFIXES) break;
-        const p = parseCidr(it?.ipv4Prefix ?? it?.ipv6Prefix);
-        if (p) out.add(p.cidr);
+        addPrefix(out, it?.ipv4Prefix ?? it?.ipv6Prefix);
     }
     return [...out];
 }
 
-/** A stored copy's prefixes ([cidr]) → [cidr], re-validated and bounded. */
+/** A feed's or a stored copy's prefixes ([cidr]) → [cidr], re-validated and bounded. Throws on a too-broad prefix. */
 export function parsePrefixes(list) {
     if (!Array.isArray(list)) return [];
     const out = new Set();
-    for (const c of list.slice(0, MAX_PREFIXES)) {
-        const p = parseCidr(c);
-        if (p) out.add(p.cidr);
-    }
+    for (const c of list.slice(0, MAX_PREFIXES)) addPrefix(out, c);
     return [...out];
 }
 

package/src/scanner.js

@@ -3,8 +3,7 @@
 // "444 rule" (nginx answers these with 444, "close, send nothing") the verifier falls back on
 // when a vendor publishes no IP list (createVerifier's `isKnownScannerIp`).
 //
-// Ported from the edge classifier Synacl Labs runs on synacl.com (2026-10), keeping only the
-// host-blind, product-agnostic rules; that site's own carve-outs are gone. A site with a real
+// Only host-blind, product-agnostic rules live here, no one site's carve-outs. A site with a real
 // path that matches (a /install.sh, a Decap CMS /admin/config.yml) passes `allow`.
 //
 // Classify on the REQUEST PATH ONLY, never the User-Agent: a scanner whose UA was literally

package/src/share.js

@@ -0,0 +1,221 @@
+// `siteread share`: one week of a digest → the numbers a site can choose to send to the public
+// benchmark, and the validator the endpoint runs on whatever arrives. Opt-in: the CLI prints the
+// exact bytes it will send and sends nothing without a yes.
+//
+// WHAT NEVER LEAVES: hostnames, URLs or paths, visitor IPs, user-agents, query text, goal labels.
+// The payload is built from an allowlist of numeric metrics, so a field added to the digest later
+// cannot leak by accident, and the endpoint rejects any key or string it does not know for the
+// same reason. The only strings are the schema's own: a random id, an ISO week and the version.
+//
+// The id is 128 random bits the CLI keeps in .siteread/share-id. It is what makes the benchmark's
+// `n` count sites rather than shares, and lets a site that shares the same week twice replace its
+// row instead of counting twice. Delete the file to start over as a new site.
+
+import { VERSION } from './version.js';
+
+export const SHARE_SCHEMA = 1;
+// Where `share` sends by default: the service's address is written here and nowhere else.
+export const SHARE_URL = 'https://api.siteread.dev/v1/share';
+
+const DAY = 86_400_000;
+const BIG = 1e9;
+
+// Every metric describes the last 7 days the digest covers (the digest's newest 7-day bucket),
+// except aiSpoofedPct, a ratio over the whole log. Bounds reject garbage, not big sites.
+export const SHARE_METRICS = Object.freeze({
+    people:              { int: true, min: 0, max: BIG, about: 'people who read a page (an IP with a browser UA that loaded an asset and never probed a scanner path)' },
+    peopleFromSearch:    { int: true, min: 0, max: BIG, about: 'people whose first visit in the week came from a search engine' },
+    peopleFromAi:        { int: true, min: 0, max: BIG, about: 'people whose first visit in the week came from an AI assistant (Referer or utm_source)' },
+    peopleFromSocial:    { int: true, min: 0, max: BIG, about: 'people whose first visit in the week came from a social site or forum' },
+    aiAnswerFetches:     { int: true, min: 0, max: BIG, about: 'page fetches by AI agents reading for a person right now (ChatGPT-User, Perplexity-User, Claude-User), verified against vendor lists' },
+    aiSearchFetches:     { int: true, min: 0, max: BIG, about: 'page fetches by AI search indexers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot), verified' },
+    aiTrainingFetches:   { int: true, min: 0, max: BIG, about: 'page fetches by AI training crawlers (GPTBot, ClaudeBot), verified' },
+    searchEngineFetches: { int: true, min: 0, max: BIG, about: 'page fetches by Googlebot and bingbot, verified' },
+    aiSpoofedPct:        { int: false, min: 0, max: 100, about: 'share of requests under an AI user-agent that were spoofed, over the whole log (%)' },
+    gscImpressions:      { int: true, min: 0, max: BIG, about: 'Search Console impressions' },
+    gscClicks:           { int: true, min: 0, max: BIG, about: 'Search Console clicks' },
+    gscCtrPct:           { int: false, min: 0, max: 100, about: 'Search Console click-through rate (%)' },
+    gscPosition:         { int: false, min: 1, max: 1000, lowerIsBetter: true, about: 'Search Console position, impression-weighted (lower is better)' },
+});
+
+// The `code` a refused share carries (validateShare's own, then the endpoint's).
+export const SHARE_REFUSALS = Object.freeze([
+    'body', 'unknown-field', 'schema', 'id', 'week', 'version', 'unknown-metric', 'metric-value',
+    'content-type', 'too-large', 'not-json', 'future-week', 'old-week', 'proof',
+]);
+
+export const ID_RE = /^[0-9a-f]{32}$/;
+const WEEK_RE = /^(\d{4})-W(\d{2})$/;
+const VERSION_RE = /^\d{1,3}\.\d{1,3}\.\d{1,3}(-[0-9a-z.]{1,20})?$/;
+const KEYS = new Set(['schema', 'id', 'week', 'siteread', 'metrics', 'proof']);
+const PROOF_RE = /^\d{1,20}$/;
+
+// PROOF OF WORK. A share id is free to make, so a fixed cap on new sites a day was one sender's
+// to use up: a hundred made-up ids in a second, and every real new site refused until midnight.
+// The service never reads an address, so it cannot tell one sender from many; what it can do is
+// make each NEW id cost work. A site's first share carries `proof`, a nonce such that
+// SHA-256(PROOF_PREFIX + id + ":" + nonce) starts with enough zero bits, made once and kept next
+// to the id. The bits a new id needs rise by one for every PROOF_STEP new sites that day (UTC),
+// so on a quiet day it is ~2 s on a laptop, and a flood doubles its own price every 50 ids: 500
+// fakes in a day is ~30 CPU-hours, and a real site still gets in by paying the day's price. Known
+// ids never pay again. Checking is one hash.
+export const PROOF_PREFIX = 'siteread-site-v1:';
+export const PROOF_BASE_BITS = 22;
+export const PROOF_STEP = 50;
+
+/** The zero bits a new site's proof needs when `newToday` new sites came in today. */
+export const proofBitsFor = (newToday, base = PROOF_BASE_BITS) => base + Math.floor(newToday / PROOF_STEP);
+
+/** Leading zero bits of a digest. */
+export function leadingZeroBits(bytes) {
+    let n = 0;
+    for (const b of bytes) {
+        if (b === 0) { n += 8; continue; }
+        return n + Math.clz32(b) - 24;
+    }
+    return n;
+}
+
+const dayMs = d => Date.parse(`${d}T00:00:00Z`);
+const isoDay = ms => new Date(ms).toISOString().slice(0, 10);
+
+/** The ISO 8601 week ('2026-W42') a day (YYYY-MM-DD or ms) falls in. */
+export function isoWeek(day) {
+    const t = typeof day === 'number' ? Math.floor(day / DAY) * DAY : dayMs(day);
+    const thursday = t + (3 - ((new Date(t).getUTCDay() + 6) % 7)) * DAY;
+    const year = new Date(thursday).getUTCFullYear();
+    const week = 1 + Math.floor((thursday - Date.UTC(year, 0, 1)) / (7 * DAY));
+    return `${year}-W${String(week).padStart(2, '0')}`;
+}
+
+/** ms of the Monday that starts an ISO week, or NaN when it is not a real week (2026-W54). */
+export function weekStart(week) {
+    const m = WEEK_RE.exec(String(week));
+    if (!m) return NaN;
+    const jan4 = Date.UTC(Number(m[1]), 0, 4);
+    const t = jan4 - ((new Date(jan4).getUTCDay() + 6) % 7) * DAY + (Number(m[2]) - 1) * 7 * DAY;
+    return isoWeek(t) === week ? t : NaN;
+}
+
+/** An ISO week moved by n weeks. */
+export function addWeeks(week, n) {
+    return isoWeek(weekStart(week) + n * 7 * DAY);
+}
+
+// The digest's newest 7-day bucket is labelled with the ISO week holding most of its days.
+const weekOfWindow = to => isoWeek(dayMs(to) - 3 * DAY);
+
+const pct1 = n => Math.round(n * 10) / 10;
+
+/**
+ * A digest (its JSON) → { payload, notes }. `notes` says, in plain words, each metric left out
+ * and why. Throws when there is no full week to share.
+ */
+export function sharePayload(digest, { id, proof = null, version = VERSION } = {}) {
+    if (!ID_RE.test(String(id))) throw new Error('share needs a 32-hex-digit id');
+    const metrics = {};
+    const notes = [];
+    let week = null;
+
+    const log = digest?.log;
+    if (log?.range?.to) {
+        const to = log.range.to;
+        const from = isoDay(dayMs(to) - 6 * DAY);
+        if (log.range.from > from) {
+            notes.push(`log: the log covers ${log.range.from} → ${to}, less than a full week; log numbers left out`);
+        } else {
+            week = weekOfWindow(to);
+            // people.weeks lists only weeks someone came in: none ending on the last day is a 0.
+            metrics.people = log.people?.weeks?.find(w => w.to === to)?.people ?? 0;
+            const arrived = { search: 0, ai: 0, social: 0 };
+            for (const s of log.sources ?? []) {
+                if (!(s.kind in arrived)) continue;
+                for (const [d, n] of Object.entries(s.landingDays ?? {})) if (d >= from && d <= to) arrived[s.kind] += n;
+            }
+            metrics.peopleFromSearch = arrived.search;
+            metrics.peopleFromAi = arrived.ai;
+            metrics.peopleFromSocial = arrived.social;
+
+            const fetches = { aiAnswerFetches: 0, aiSearchFetches: 0, aiTrainingFetches: 0, searchEngineFetches: 0 };
+            let complete = true;
+            let aiRequests = 0, aiSpoofed = 0;
+            for (const c of log.crawlers ?? []) {
+                if (c.kind === 'ai') { aiRequests += c.requests ?? 0; aiSpoofed += c.spoofed?.requests ?? 0; }
+                const w = c.verified?.perWeek?.at(-1);
+                if (!w || w.to !== to) continue;
+                // A digest before 0.2.1 has only `fetches`, which counts sitemap URLs alone when it
+                // had a sitemap: that would not mean the same as everyone else's row.
+                const n = w.pageFetches ?? (c.verified.sitemapFetches == null ? w.fetches : null);
+                if (n == null) { complete = false; continue; }
+                const key = c.kind === 'crawler'
+                    ? (c.purpose === 'search' ? 'searchEngineFetches' : null)
+                    : { answer: 'aiAnswerFetches', search: 'aiSearchFetches', training: 'aiTrainingFetches' }[c.purpose];
+                if (key) fetches[key] += n;
+            }
+            if (complete) Object.assign(metrics, fetches);
+            else notes.push('log: this digest predates per-week page fetches (siteread < 0.2.1 with a sitemap); crawler numbers left out, run the digest again');
+            if (aiRequests) metrics.aiSpoofedPct = pct1((100 * aiSpoofed) / aiRequests);
+            else notes.push('log: no requests under AI user-agents; aiSpoofedPct left out');
+        }
+    } else {
+        notes.push(log ? 'log: no parseable lines in the log; log numbers left out' : 'log: no access log in this digest; log numbers left out');
+    }
+
+    const gscWeek = digest?.gsc?.weeks?.at(-1);
+    if (gscWeek) {
+        const label = weekOfWindow(gscWeek.to);
+        if (gscWeek.days !== 7) {
+            notes.push(`gsc: the export's last week ${gscWeek.from} → ${gscWeek.to} has ${gscWeek.days} days; Search Console numbers left out`);
+        } else if (week && label !== week) {
+            notes.push(`gsc: the export's last full week is ${label}, the log's is ${week}; Search Console numbers left out (export again, or pass --from/--to to the digest)`);
+        } else {
+            week ??= label;
+            metrics.gscImpressions = gscWeek.impressions;
+            metrics.gscClicks = gscWeek.clicks;
+            if (gscWeek.ctr != null) metrics.gscCtrPct = gscWeek.ctr;
+            if (gscWeek.position != null) metrics.gscPosition = gscWeek.position;
+        }
+    } else if (!digest?.gsc) {
+        notes.push('gsc: no Search Console export in this digest; Search Console numbers left out');
+    }
+
+    if (!week || !Object.keys(metrics).length) throw new Error(`nothing to share: the digest has no full week of numbers${notes.length ? ` (${notes.join('; ')})` : ''}`);
+    const payload = { schema: SHARE_SCHEMA, id, week, siteread: version, metrics };
+    if (proof != null) payload.proof = String(proof);
+    const check = validateShare(payload);
+    if (!check.ok) throw new Error(`the digest gave a number the endpoint would refuse: ${check.error}`);
+    return { payload: check.share, notes };
+}
+
+/**
+ * What the endpoint accepts, nothing more: { ok: true, share } with the metrics in schema order,
+ * or { ok: false, error, code }. `code` is one of SHARE_REFUSALS, for counting refusals without
+ * keeping the message (which can echo a field name the sender chose). It does not know the
+ * date: the endpoint checks the week's age itself.
+ */
+export function validateShare(body) {
+    const bad = (code, error) => ({ ok: false, error, code });
+    if (!body || typeof body !== 'object' || Array.isArray(body)) return bad('body', 'the body must be a JSON object');
+    for (const k of Object.keys(body)) if (!KEYS.has(k)) return bad('unknown-field', `unknown field "${k}"`);
+    if (body.schema !== SHARE_SCHEMA) return bad('schema', `schema must be ${SHARE_SCHEMA}`);
+    if (typeof body.id !== 'string' || !ID_RE.test(body.id)) return bad('id', 'id must be 32 lowercase hex digits');
+    if (typeof body.week !== 'string' || !Number.isFinite(weekStart(body.week))) return bad('week', 'week must be an ISO week like 2026-W42');
+    if (typeof body.siteread !== 'string' || !VERSION_RE.test(body.siteread)) return bad('version', 'siteread must be a version like 0.3.0');
+    if (body.proof !== undefined && (typeof body.proof !== 'string' || !PROOF_RE.test(body.proof))) return bad('proof', 'proof must be a string of digits');
+    const m = body.metrics;
+    if (!m || typeof m !== 'object' || Array.isArray(m)) return bad('metric-value', 'metrics must be an object');
+    const metrics = {};
+    for (const k of Object.keys(m)) if (!Object.hasOwn(SHARE_METRICS, k)) return bad('unknown-metric', `unknown metric "${k}"`);
+    for (const [k, spec] of Object.entries(SHARE_METRICS)) {
+        if (!Object.hasOwn(m, k)) continue;
+        const v = m[k];
+        if (typeof v !== 'number' || !Number.isFinite(v)) return bad('metric-value', `${k} must be a number`);
+        if (spec.int && !Number.isInteger(v)) return bad('metric-value', `${k} must be a whole number`);
+        if (v < spec.min || v > spec.max) return bad('metric-value', `${k} must be between ${spec.min} and ${spec.max}`);
+        metrics[k] = v;
+    }
+    if (!Object.keys(metrics).length) return bad('metric-value', 'metrics is empty');
+    const share = { schema: body.schema, id: body.id, week: body.week, siteread: body.siteread, metrics };
+    if (body.proof !== undefined) share.proof = body.proof;
+    return { ok: true, share };
+}

package/src/verifier.js

@@ -9,7 +9,7 @@
 // A vendor list always wins over the 444 rule: ClaudeBot once followed a broken README link
 // into a scanner path, and the 444 rule called a real crawler spoofed (ip-ranges.js).
 //
-// Refresh rules are the ported ones (ip-ranges.js): refetch a list when the stored copy is
+// Refresh rules (ip-ranges.js): refetch a list when the stored copy is
 // > 24 h old, keep the old copy when a fetch fails, drop a copy nobody could refresh for 8 days.
 // Nothing here throws.
 
@@ -38,7 +38,8 @@ export function memoryStore(initial = null) {
  * @param {Array<{agent: string, url: string}>} [opts.sources]
  * @param {() => number} [opts.now]
  * @param {{warn?: Function, error?: Function}} [opts.log]
- * @param {string|null} [opts.feedUrl]  one merged document instead of the vendor URLs (opt-in)
+ * @param {string|null} [opts.feedUrl]  one merged document instead of the vendor URLs
+ * @param {boolean} [opts.fallback]     with feedUrl: fetch from the vendors what the feed could not give
  * @param {(ip: string) => boolean} [opts.isKnownScannerIp]  the 444 rule, for agents with no list
  */
 export function createVerifier({
@@ -48,6 +49,7 @@ export function createVerifier({
     now = Date.now,
     log = console,
     feedUrl = null,
+    fallback = false,
     isKnownScannerIp = null,
 } = {}) {
     // One entry per agent (the first wins), in the order given.
@@ -93,7 +95,8 @@ export function createVerifier({
         const lists = body && typeof body.lists === 'object' && body.lists ? body.lists : {};
         for (const { agent } of due) {
             const l = Object.hasOwn(lists, agent) ? lists[agent] : null;
-            const prefixes = parsePrefixes(l?.prefixes);
+            let prefixes;
+            try { prefixes = parsePrefixes(l?.prefixes); } catch (err) { fail(out, agent, err, 'feed'); continue; }
             // A future fetchedAt is clamped to now: it would otherwise keep the copy fresh forever.
             const at = Math.min(fetchedAtMs(l), t);
             const stored = fetchedAtMs(doc.lists[agent]);
@@ -140,6 +143,14 @@ export function createVerifier({
                     for (const { agent } of due) fail(out, agent, new Error('no fetch available'), agent);
                 } else if (feedUrl) {
                     await fetchFromFeed(due, t, doc, out);
+                    // The feed is a shortcut to the vendors, never the only way to them: what it
+                    // could not give (down, stale, a list missing) comes from the vendor itself.
+                    const missed = fallback ? due.filter(s => out.failed.includes(s.agent)) : [];
+                    if (missed.length) {
+                        for (const { agent } of missed) delete out.errors[agent];
+                        out.failed = out.failed.filter(a => !missed.some(s => s.agent === a));
+                        await fetchFromVendors(missed, t, doc, out);
+                    }
                 } else {
                     await fetchFromVendors(due, t, doc, out);
                 }

package/src/version.js

@@ -1,2 +1,2 @@
 // Kept in step with package.json by test/package.test.js (no JSON import: Node 20 needs no flag for this).
-export const VERSION = '0.2.0';
+export const VERSION = '0.2.2';