siteread 0.0.0-stage → 0.2.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Synacl Labs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,233 @@
-# Temporary Holding Version
+# siteread
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+**Was that really ClaudeBot?** `siteread` takes the IP and User-Agent of a request from your access
+log and checks the IP against the list the claimed vendor publishes: `verified`, `spoofed` or `unknown`.
+
+Node ≥ 20, zero dependencies, nothing leaves your machine except GETs of the vendors' public IP lists.
+
+## Install
+
+```sh
+npm install siteread        # the library
+npx siteread --help         # the CLI, no install
+```
+
+## CLI
+
+`check` answers one request. The first call fetches the vendor lists into a cache file; later calls
+reuse a copy younger than 24 h.
+
+```console
+$ npx siteread check 216.73.216.12 "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
+verified  216.73.216.12 is claudebot (claude, training): in claude.com/crawling/bots.json (38 prefixes, fetched 2026-10-20T20:04:53Z)
+
+$ npx siteread check 203.0.113.9 "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
+spoofed   203.0.113.9 claims chatgpt-user (chatgpt, answer) but is not in openai.com/chatgpt-user.json (234 prefixes, fetched 2026-10-20T20:04:53Z)
+
+$ npx siteread check 203.0.113.9 "Claude-User (claude-code/2.1.292; +https://support.anthropic.com/)"
+unknown   203.0.113.9 claims claude-code (claude, answer): Claude Code's WebFetch runs from the user's own machine, so no vendor list can cover it. If this IP also requested scanner paths in your log, treat it as spoofed
+```
+
+| Exit | Meaning |
+|---|---|
+| `0` | verified: the IP is in the vendor's list for that agent |
+| `2` | spoofed: the vendor publishes a list and the IP is not in it |
+| `3` | unknown: no list to check against, or the UA is not a known AI agent or search crawler |
+| `1` | usage error, or the IP is not an IP address |
+
+`lists` shows what the cache holds:
+
+```console
+$ npx siteread lists
+agent             prefixes  fetched (UTC)         source
+chatgpt-user           234  2026-10-20T20:04:53Z  https://openai.com/chatgpt-user.json
+oai-searchbot           39  2026-10-20T20:04:53Z  https://openai.com/searchbot.json
+gptbot                  18  2026-10-20T20:04:53Z  https://openai.com/gptbot.json
+perplexity-user          4  2026-10-20T20:04:53Z  https://www.perplexity.com/perplexity-user.json
+perplexitybot            8  2026-10-20T20:04:53Z  https://www.perplexity.com/perplexitybot.json
+applebot                24  2026-10-20T20:04:53Z  https://search.developer.apple.com/applebot.json
+claude-user             38  2026-10-20T20:04:53Z  https://claude.com/crawling/bots.json
+claude-searchbot        38  2026-10-20T20:04:53Z  https://claude.com/crawling/bots.json
+claudebot               38  2026-10-20T20:04:53Z  https://claude.com/crawling/bots.json
+googlebot              317  2026-10-20T20:04:53Z  https://developers.google.com/static/search/apis/ipranges/googlebot.json
+bingbot                 28  2026-10-20T20:04:53Z  https://www.bing.com/toolbox/bingbot.json
+```
+
+Options: `--no-cache` refetches every list now (a failed fetch still falls back to the cached copy),
+`--json` prints the whole result. The cache is `<os.tmpdir()>/siteread/lists.json`; set
+`SITEREAD_CACHE` to put it elsewhere. `siteread --help` prints all of this.
+
+## Digest (for the `seo-read` skill)
+
+`digest` reads a Search Console export (the Performance → Export → Download CSV zip) and your
+access logs (nginx/Apache combined, `.gz` fine) and prints one JSON document: weekly GSC totals with
+impression-weighted position, people by first-touch source (an IP that sent a browser UA, loaded
+an asset and never probed a scanner path), goal counts, and every crawler checked against its
+vendor's list.
+
+```console
+$ npx siteread digest --gsc export.zip --log access.log.1.gz --log access.log \
+    --site example.org --sitemap sitemap.xml --goal 'signup=^POST /signup 2' --out .siteread/digests
+.siteread/digests/2026-10-18.json
+```
+
+`npx siteread --help` lists every option. The skill in [`skill/`](../../skill/README.md) turns
+the JSON into a weekly memo.
+
+## Library
+
+```js
+import { createVerifier, isScannerPath } from 'siteread';
+
+const verifier = createVerifier();
+await verifier.refresh();                 // fetch the lists; call again every few hours
+verifier.check('216.73.216.12', 'Mozilla/5.0 … ClaudeBot/1.0; +claudebot@anthropic.com)');
+// {
+//   agent: 'claudebot', kind: 'ai', assistant: 'claude', purpose: 'training',
+//   source: 'https://claude.com/crawling/bots.json', verdict: 'verified', reason: 'in-vendor-list'
+// }
+```
+
+### `createVerifier(options)`
+
+| Option | Default | |
+|---|---|---|
+| `fetch` | `globalThis.fetch` | any WHATWG fetch |
+| `store` | in memory | `{ get(): Promise<doc \| null>, set(doc): Promise<void> }`; `fileStore(path)` keeps it in a JSON file |
+| `sources` | `SOURCES` | `[{ agent, url }]`, which lists to fetch |
+| `now` | `Date.now` | a clock, for tests |
+| `log` | `console` | `{ warn, error }`; pass `{}` for silence |
+| `isKnownScannerIp` | none | `(ip) => boolean`, the 444 rule (below) |
+| `feedUrl` | none | one merged document instead of the vendor URLs (below) |
+
+It returns:
+
+- **`refresh({ force })`** reads the store, fetches every list that is due, drops what is too old and
+  writes back. It never throws and returns `{ fetched, failed, fresh, dropped, errors }` (agent keys;
+  `errors` maps a failed agent to its reason). `force` refetches lists that are still fresh.
+- **`check(ip, ua)`** is synchronous and uses the lists the last `refresh()` or `load()` saw. It
+  returns `{ agent, kind, assistant, purpose, source, verdict, reason }`:
+
+  | verdict | reason | when |
+  |---|---|---|
+  | `verified` | `in-vendor-list` | the IP is in the list for the agent the UA claims |
+  | `spoofed` | `not-in-vendor-list` | that agent has a list and the IP is not in it |
+  | `spoofed` | `robots-token-not-a-user-agent` | the UA says Google-Extended (a robots.txt token, never sent as a UA) |
+  | `spoofed` | `known-scanner-ip` | no list, and your `isKnownScannerIp(ip)` said yes |
+  | `unknown` | `no-vendor-list` | the operator publishes no list (CCBot, YandexBot, Claude Code, …) |
+  | `unknown` | `list-unavailable` | it has one, but it could not be loaded and there is no copy younger than 8 days |
+  | `unknown` | `invalid-ip` | the IP did not parse |
+  | `n/a` | `not-a-known-agent` | the UA claims to be no known AI agent or search crawler (`agent: null`) |
+
+  `kind` is `'ai'` (an `AGENTS` entry) or `'crawler'` (a `CRAWLERS` entry). An IPv4-mapped IPv6
+  address (`::ffff:1.2.3.4`, as a dual-stack listener logs it) is checked as the IPv4 it is.
+- **`load()`** rebuilds the matcher from the store without fetching: for workers that share a store
+  with one process that refreshes.
+- **`lists()`** returns `[{ agent, url, prefixes, fetchedAt, fresh }]` in `sources` order.
+
+### The 444 rule, for agents without a list
+
+Most operators publish no IP list. For those, `siteread` says `unknown` rather than guess. If you
+know which IPs probed your site for scanner paths, pass that and an `unknown` becomes `spoofed`:
+
+```js
+import { readFileSync } from 'node:fs';
+import { createVerifier, fileStore, isScannerPath } from 'siteread';
+
+const lines = readFileSync('access.log', 'utf8').trim().split('\n').map(JSON.parse);   // your log format here
+const scanners = new Set(lines.filter(l => isScannerPath(l.request_uri)).map(l => l.remote_addr));
+
+const verifier = createVerifier({ store: fileStore('.siteread.json'), isKnownScannerIp: ip => scanners.has(ip) });
+await verifier.refresh();
+for (const l of lines) {
+  const r = verifier.check(l.remote_addr, l.http_user_agent);
+  if (r.agent) console.log(r.verdict, r.agent, l.request_uri);
+}
+```
+
+The hook only ever turns `unknown` into `spoofed`. When the agent has a vendor list, the list wins,
+even for an IP that hit a scanner path. The ClaudeBot story below is why.
+
+### UA tables and scanner paths
+
+- **`AGENTS`** lists 23 AI agents as `{ key, re, assistant, purpose }`, where `purpose` is `answer`
+  (fetched live for a person), `search` (an assistant's index) or `training`. The FIRST match wins.
+  **`detectAgent(ua)`** returns the entry or `null`.
+- **`CRAWLERS`** lists classic search and SEO crawlers (Googlebot, bingbot, YandexBot, DuckDuckBot,
+  Slurp, PetalBot, AhrefsBot, SemrushBot, MJ12bot, facebookexternalhit). **`detectCrawler(ua)`**
+  returns the entry, and **`isSearchCrawler(ua)`** returns a boolean. AI agents are never search
+  crawlers here: Applebot, GPTBot and the rest are in `AGENTS`.
+- **`isScannerPath(path, { allow })`** is true for a vulnerability-scanner request target: dotfiles
+  (but not `/.well-known/`), `.php`/`.asp`/`.jsp`/`.env`/`.sql`/`.bak`/`.sh`…, `wp-*`, phpMyAdmin,
+  `cgi-bin`, router exploits, percent-encoded dots (`/%2eenv`), secret and config file names,
+  `Dockerfile`. Pass the raw request target (path + query, not URL-decoded: nginx's
+  `$request_uri`). If a real path of yours matches, exempt it:
+  `isScannerPath(p, { allow: /^\/(install\.sh|admin\/config\.yml)(\?|$)/ })`.
+- **`parseCidr`**, **`parseList`**, **`normIp`** and **`SOURCES`** are exported for tests and extensions.
+
+## Why
+
+This code was built to count AI traffic on one small site's nginx log, and every simple approach
+gave the wrong answer.
+
+**Scanners wear AI user-agents.** One scanner cycles through them. In 30 days of the log
+(to 2026-10-16), 217 of the 221 ChatGPT-User requests from outside OpenAI's list came from IPs that
+also drew a 444 (nginx's "close, send nothing" reply to a scanner path). Every request claiming to be
+DeepSeekBot, CCBot, Grok, Perplexity-User or Claude-SearchBot was fake. A UA count is a count of
+what scanners chose to say.
+
+**The 444 rule cannot clear a real crawler that follows a bad link.** Before Anthropic's list was
+used, the test was "this IP never hit a scanner path." Then ClaudeBot resolved a GitHub README's
+relative links against the site and asked for `/protocol//blob/master/examples/docker-compose.yml`.
+It drew one 444, and all 50 of that IP's reads that day were counted as spoofed. Anthropic publishes
+one list for its bots (`claude.com/crawling/bots.json`). In those 30 days every ClaudeBot IP and
+every page-reading Claude-User IP was in it, and the four IPs probing `/.env`-style paths under
+Claude names were not. Hence the rule: where a list exists, it decides.
+
+**Claude Code is not Anthropic's crawler.** Claude Code's WebFetch sends
+`Claude-User (claude-code/…)` from the user's own machine and IP, so no list can cover it. It is
+matched before `Claude-User` and always comes back `unknown`.
+
+**Google-Extended is not a User-Agent.** It is a robots.txt token that Google's crawlers obey while
+fetching as Googlebot. A request that sends it as a UA is not from Google.
+
+## Vendor lists
+
+| Agent(s) | List |
+|---|---|
+| chatgpt-user | https://openai.com/chatgpt-user.json |
+| oai-searchbot | https://openai.com/searchbot.json |
+| gptbot | https://openai.com/gptbot.json |
+| perplexity-user | https://www.perplexity.com/perplexity-user.json |
+| perplexitybot | https://www.perplexity.com/perplexitybot.json |
+| applebot | https://search.developer.apple.com/applebot.json |
+| claude-user, claude-searchbot, claudebot | https://claude.com/crawling/bots.json (fetched once for all three) |
+| googlebot | https://developers.google.com/static/search/apis/ipranges/googlebot.json |
+| bingbot | https://www.bing.com/toolbox/bingbot.json |
+
+All of them answer `{ "creationTime": …, "prefixes": [{ "ipv4Prefix": … } | { "ipv6Prefix": … }] }`.
+
+- **Refresh:** a list is refetched only when the stored copy is more than 24 h old.
+- **Failure:** a failed fetch keeps the previous copy, so a vendor's CDN hiccup does not turn its real
+  crawlers into `spoofed`. A 200 that is not JSON (an HTML challenge page, a moved list) counts as a
+  failure, not as an empty list.
+- **Drop:** a copy nobody could refresh for 8 days is dropped, so a vendor's moved ranges cannot mark
+  its new IPs as spoofed forever. With no copy, the verdict is `unknown`, never `spoofed`.
+- **Bounds:** a body over 2,000,000 characters, more than 20,000 prefixes, or a prefix broader than
+  /8 (IPv4) or /16 (IPv6) is rejected. A `/0` would verify every address.
+
+## Privacy
+
+Local use sends nothing anywhere except plain GETs of the URLs above, with the User-Agent
+`siteread/<version>`. No hostnames, URLs or visitor IPs leave your machine. There is no reverse-DNS
+check for the same reason: a PTR lookup sends the visitor's IP to your resolver.
+
+`feedUrl` points the verifier at one merged document,
+`{ "lists": { "<agent>": { "prefixes": [...], "fetchedAt": "..." } } }`, instead of the vendors. It
+is for a hosted feed that is planned but **not live yet**, so it is off by default and strictly opt-in.
+Every prefix from a feed is re-validated like a vendor's, and a `fetchedAt` in the future is clamped
+to now.
+
+## License
+
+MIT. Extracted from the pipeline Synacl Labs runs on synacl.com.

package/bin/siteread.js

@@ -0,0 +1,277 @@
+#!/usr/bin/env node
+// siteread CLI. No dependencies; arguments parsed by hand.
+//
+//   siteread check <ip> "<user-agent>"   exit 0 verified · 2 spoofed · 3 unknown / n/a · 1 usage or error
+//   siteread lists
+//   siteread digest --gsc <export> --log <file> …   JSON for the seo-read skill's weekly memo
+//   siteread --help
+//
+// Lists are cached in one file (os.tmpdir()/siteread/lists.json, or $SITEREAD_CACHE) and
+// refetched only when older than 24 h. The only network traffic is GETs of the vendors' lists.
+
+import os from 'node:os';
+import path from 'node:path';
+import { readFileSync, writeFileSync, mkdirSync, readdirSync } from 'node:fs';
+import { createVerifier, fileStore, detectAgent, detectCrawler, digest, parseGoal, VERSION } from '../src/index.js';
+
+const EXIT = { verified: 0, spoofed: 2, unknown: 3, 'n/a': 3, usage: 1 };
+const cacheFile = process.env.SITEREAD_CACHE || path.join(os.tmpdir(), 'siteread', 'lists.json');
+
+const HELP = `siteread ${VERSION}: is that crawler who it says it is?
+
+Usage:
+  siteread check <ip> "<user-agent>"   verdict for one request
+  siteread lists                       every vendor list: prefixes and when it was fetched
+  siteread digest [options]            a Search Console export + access log -> JSON numbers
+                                       for a weekly read (the seo-read skill writes the memo)
+  siteread --help | --version
+
+Options:
+  --no-cache   refetch every list now (a copy younger than 24 h is otherwise reused;
+               a failed fetch still falls back to the cached copy)
+  --json       print the full result as JSON
+
+Digest options (at least one of --gsc / --log):
+  --gsc <zip|dir>        Search Console Performance export (Export -> Download CSV)
+  --log <file>           access log, nginx/Apache combined, .gz ok (repeat for several)
+  --site <host>          your site's host (guessed from Referers otherwise)
+  --sitemap <file>       a local sitemap.xml: count crawler fetches on its URLs only
+  --from / --to <day>    keep only log days in this range (YYYY-MM-DD)
+  --split <day>          GSC before/after comparison around a change made that day
+  --goal <label=regex>   count people whose "METHOD /path status" matches (repeatable),
+                         e.g. --goal 'signup=^POST /signup 2'
+  --team-path <regex>    an IP that got a 2xx on such a path is your team, not a reader
+  --exclude-ip <ip,...>  more IPs that are you (repeatable)
+  --allow-path <regex>   your real paths that look like scanner probes (/install.sh)
+  --tz <zone>            zone for log days (default America/Los_Angeles, GSC's)
+  --prev <digest.json>   an earlier digest: GSC rows gain last read's numbers
+  --out <dir>            write <dir>/<last day>.json instead of stdout, and use the newest
+                         earlier digest in <dir> as --prev unless --prev is given
+  --limit <n>            rows per table (default 25)
+  --offline              use cached vendor lists only; no network at all
+
+Exit codes (check):
+  0 verified   the IP is in the vendor's published list for that agent
+  2 spoofed    the vendor publishes a list and the IP is not in it
+  3 unknown    no list to check against, or the UA is not a known AI agent or crawler
+  1            usage error, or not an IP address
+
+Cache: ${cacheFile}  (override with SITEREAD_CACHE)
+Nothing leaves this machine except GETs of the vendors' public IP lists.`;
+
+// Options that take a value (digest only). Each keeps every value given, in order.
+const VALUE_OPTS = new Set(['--gsc', '--log', '--out', '--site', '--sitemap', '--from', '--to', '--split', '--goal',
+    '--team-path', '--exclude-ip', '--allow-path', '--tz', '--prev', '--limit']);
+
+function parseArgs(argv) {
+    const flags = new Set();
+    const opts = new Map();
+    const pos = [];
+    for (let i = 0; i < argv.length; i++) {
+        let a = argv[i];
+        if (a === '--') { pos.push(...argv.slice(i + 1)); break; }
+        let val;
+        const eq = a.indexOf('=');
+        if (a.startsWith('--') && eq > 0 && VALUE_OPTS.has(a.slice(0, eq))) { val = a.slice(eq + 1); a = a.slice(0, eq); }
+        if (VALUE_OPTS.has(a)) {
+            if (val === undefined) {
+                if (i + 1 >= argv.length) throw new UsageError(`${a} needs a value`);
+                val = argv[++i];
+            }
+            if (!opts.has(a)) opts.set(a, []);
+            opts.get(a).push(val);
+        } else if (a.startsWith('--') || a === '-h' || a === '-v') flags.add(a);
+        else pos.push(a);
+    }
+    return { flags, opts, pos };
+}
+
+class UsageError extends Error {}
+
+function usage(msg) {
+    process.stderr.write(`siteread: ${msg}\nRun "siteread --help" for usage.\n`);
+    return EXIT.usage;
+}
+
+function when(iso) {
+    return iso ? iso.replace(/\.\d{3}Z$/, 'Z') : 'never';
+}
+
+// Short host+path for one-line output: https://claude.com/crawling/bots.json → claude.com/crawling/bots.json
+function shortUrl(u) {
+    return String(u || '').replace(/^https?:\/\//, '');
+}
+
+function reportFailures(result, verifier) {
+    const failed = [...new Set(result.failed)];
+    if (!failed.length && !result.error) return;
+    const held = new Map(verifier.lists().map(l => [l.agent, l]));
+    for (const agent of failed) {
+        const l = held.get(agent);
+        const kept = l?.prefixes ? `kept the copy from ${when(l.fetchedAt)}` : 'no copy to fall back on';
+        process.stderr.write(`siteread: could not refresh ${agent}: ${result.errors?.[agent] || 'failed'} (${kept})\n`);
+    }
+    if (result.error) process.stderr.write(`siteread: cache ${cacheFile}: ${result.error}\n`);
+}
+
+function describe(ip, ua, r, list) {
+    const who = r.assistant ? `${r.agent} (${r.assistant}, ${r.purpose})` : `${r.agent} (${r.purpose})`;
+    switch (r.reason) {
+        case 'in-vendor-list':
+            return `verified  ${ip} is ${who}: in ${shortUrl(r.source)} (${list.prefixes} prefixes, fetched ${when(list.fetchedAt)})`;
+        case 'not-in-vendor-list':
+            return `spoofed   ${ip} claims ${who} but is not in ${shortUrl(r.source)} (${list.prefixes} prefixes, fetched ${when(list.fetchedAt)})`;
+        case 'robots-token-not-a-user-agent':
+            return `spoofed   ${ip} claims ${who}: Google-Extended is a robots.txt token, never a real user-agent`;
+        case 'known-scanner-ip':
+            return `spoofed   ${ip} claims ${who} and has probed scanner paths`;
+        case 'list-unavailable':
+            return `unknown   ${ip} claims ${who}: could not load ${shortUrl(r.source)} and no cached copy`;
+        case 'no-vendor-list': {
+            const e = detectAgent(ua) || detectCrawler(ua);
+            const why = e?.note ? `${e.note}` : 'its operator publishes no IP list';
+            return `unknown   ${ip} claims ${who}: ${why}. If this IP also requested scanner paths in your log, treat it as spoofed`;
+        }
+        default:
+            return `n/a       "${ua}" is not a known AI agent or search crawler`;
+    }
+}
+
+async function cmdCheck({ flags, pos }) {
+    if (pos.length !== 3) return usage('check takes exactly two arguments: <ip> "<user-agent>" (quote the user-agent)');
+    const [, ip, ua] = pos;
+    const verifier = createVerifier({ store: fileStore(cacheFile), log: {} });
+    let r = verifier.check(ip, ua);
+    if (r.reason === 'invalid-ip') return usage(`"${ip}" is not an IP address`);
+    // Only touch the network when the claimed agent has a list to check against.
+    if (r.source) {
+        const result = await verifier.refresh({ force: flags.has('--no-cache') });
+        reportFailures(result, verifier);
+        r = verifier.check(ip, ua);
+    }
+    const list = verifier.lists().find(l => l.agent === r.agent) || {};
+    if (flags.has('--json')) {
+        process.stdout.write(`${JSON.stringify({ ip, userAgent: ua, ...r, list: r.source ? list : null }, null, 2)}\n`);
+    } else {
+        process.stdout.write(`${describe(ip, ua, r, list)}\n`);
+    }
+    return EXIT[r.verdict] ?? EXIT.usage;
+}
+
+async function cmdLists({ flags, pos }) {
+    if (pos.length !== 1) return usage('lists takes no arguments');
+    const verifier = createVerifier({ store: fileStore(cacheFile), log: {} });
+    const result = await verifier.refresh({ force: flags.has('--no-cache') });
+    reportFailures(result, verifier);
+    const rows = verifier.lists();
+    if (flags.has('--json')) {
+        process.stdout.write(`${JSON.stringify({ cache: cacheFile, lists: rows }, null, 2)}\n`);
+    } else {
+        const w = Math.max(...rows.map(l => l.agent.length), 5);
+        const out = [`${'agent'.padEnd(w)}  prefixes  fetched (UTC)         source`];
+        for (const l of rows) {
+            out.push(`${l.agent.padEnd(w)}  ${String(l.prefixes).padStart(8)}  ${when(l.fetchedAt).padEnd(20)}  ${l.url}`);
+        }
+        process.stdout.write(`${out.join('\n')}\n`);
+    }
+    return rows.some(l => l.prefixes) ? 0 : 1;
+}
+
+const DAY_RE = /^\d{4}-\d{2}-\d{2}$/;
+
+async function cmdDigest({ flags, opts, pos }) {
+    if (pos.length !== 1) return usage('digest takes options only, e.g. digest --gsc export.zip --log access.log');
+    const one = k => opts.get(k)?.at(-1) ?? null;
+    const logs = opts.get('--log') ?? [];
+    const gsc = one('--gsc');
+    if (!gsc && !logs.length) return usage('digest needs --gsc <export>, --log <file>, or both');
+    for (const k of ['--from', '--to', '--split']) if (one(k) && !DAY_RE.test(one(k))) return usage(`${k} wants YYYY-MM-DD`);
+    const re = (k) => {
+        if (!one(k)) return null;
+        try { return new RegExp(one(k), 'i'); } catch (err) { throw new UsageError(`${k}: ${err.message}`); }
+    };
+    let goals;
+    try { goals = (opts.get('--goal') ?? []).map(parseGoal); } catch (err) { return usage(err.message); }
+    const tz = one('--tz') ?? 'America/Los_Angeles';
+    try { new Intl.DateTimeFormat('en', { timeZone: tz }); } catch { return usage(`--tz: unknown time zone "${tz}"`); }
+    const limit = one('--limit') ? Number(one('--limit')) : 25;
+    if (!Number.isInteger(limit) || limit < 1) return usage('--limit wants a positive whole number');
+    let prev = null;
+    if (one('--prev')) {
+        try { prev = JSON.parse(readFileSync(one('--prev'), 'utf8')); } catch (err) { return usage(`--prev: ${err.message}`); }
+    }
+
+    let verifier = null;
+    if (logs.length) {
+        verifier = createVerifier({ store: fileStore(cacheFile), log: {} });
+        if (flags.has('--offline')) {
+            await verifier.load();
+        } else {
+            reportFailures(await verifier.refresh({ force: flags.has('--no-cache') }), verifier);
+        }
+        if (!verifier.lists().some(l => l.prefixes)) {
+            process.stderr.write('siteread: no vendor lists available; every crawler will read as unknown\n');
+        }
+    }
+
+    const out = await digest({
+        gsc, logs, verifier, prev, limit,
+        split: one('--split'), sitemapFile: one('--sitemap'),
+        site: one('--site'), tz, from: one('--from'), to: one('--to'), goals,
+        teamPath: re('--team-path'), allowPath: re('--allow-path'),
+        excludeIps: new Set((opts.get('--exclude-ip') ?? []).flatMap(s => s.split(',')).map(s => s.trim()).filter(Boolean)),
+    });
+    if (verifier) out.vendorLists = verifier.lists().map(({ agent, prefixes, fetchedAt }) => ({ agent, prefixes, fetchedAt }));
+    for (const w of out.warnings) process.stderr.write(`siteread: ${w}\n`);
+    const dir = one('--out');
+    if (!dir) {
+        process.stdout.write(`${JSON.stringify(out, null, 2)}\n`);
+        return 0;
+    }
+    // Named for the last day the read covers, which is only known now (GSC's lag is cut first).
+    const name = `${out.gsc?.range.to ?? out.log?.range.to ?? 'empty'}.json`;
+    mkdirSync(dir, { recursive: true });
+    if (!prev && gsc) {
+        const earlier = readdirSync(dir).filter(f => /^\d{4}-\d{2}-\d{2}\.json$/.test(f) && f < name).sort().at(-1);
+        if (earlier) {
+            const file = path.join(dir, earlier);
+            const again = await digest({ gsc, prev: JSON.parse(readFileSync(file, 'utf8')), split: one('--split'), limit });
+            out.gsc = again.gsc;
+            out.prevFile = earlier;
+        }
+    }
+    const file = path.join(dir, name);
+    writeFileSync(file, `${JSON.stringify(out, null, 2)}\n`);
+    process.stderr.write(`siteread: wrote ${file}${out.prevFile ? ` (compared with ${out.prevFile})` : ' (no earlier digest to compare with)'}\n`);
+    process.stdout.write(`${file}\n`);
+    return 0;
+}
+
+async function main(argv) {
+    let args;
+    try { args = parseArgs(argv); } catch (err) { if (err instanceof UsageError) return usage(err.message); throw err; }
+    const { flags, pos } = args;
+    if (flags.has('--help') || flags.has('-h') || pos[0] === 'help') { process.stdout.write(`${HELP}\n`); return 0; }
+    if (flags.has('--version') || flags.has('-v')) { process.stdout.write(`${VERSION}\n`); return 0; }
+    const known = new Set(['--no-cache', '--json', '--offline']);
+    const bad = [...flags].find(f => !known.has(f));
+    if (bad) return usage(`unknown option ${bad}`);
+    if (pos[0] === 'check') return cmdCheck(args);
+    if (pos[0] === 'lists') return cmdLists(args);
+    if (pos[0] === 'digest') {
+        try { return await cmdDigest(args); } catch (err) {
+            if (err instanceof UsageError) return usage(err.message);
+            if (err?.code === 'ENOENT') return usage(`no such file: ${err.path}`);
+            // Bad input (an export with no Dates.csv, a broken zip): say what, no stack trace.
+            process.stderr.write(`siteread: digest: ${err?.message || err}\n`);
+            return 1;
+        }
+    }
+    if (!pos.length) { process.stdout.write(`${HELP}\n`); return EXIT.usage; }
+    return usage(`unknown command "${pos[0]}"`);
+}
+
+main(process.argv.slice(2)).then(
+    (code) => { process.exitCode = code; },
+    (err) => { process.stderr.write(`siteread: ${err?.stack || err}\n`); process.exitCode = 1; },
+);

package/package.json

@@ -1,6 +1,37 @@
 {
   "name": "siteread",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.2.0",
+  "description": "Is that crawler who it says it is? Checks an IP + User-Agent against the AI and search vendors' published IP lists. Zero dependencies.",
+  "type": "module",
+  "engines": {
+    "node": ">=20"
+  },
+  "bin": {
+    "siteread": "./bin/siteread.js"
+  },
+  "exports": {
+    ".": "./src/index.js",
+    "./package.json": "./package.json"
+  },
+  "files": [
+    "src",
+    "bin",
+    "README.md"
+  ],
+  "scripts": {
+    "test": "node --test"
+  },
+  "keywords": [
+    "crawler",
+    "bot",
+    "verification",
+    "gptbot",
+    "claudebot",
+    "chatgpt-user",
+    "googlebot",
+    "user-agent",
+    "ip-ranges",
+    "access-log"
+  ],
+  "license": "MIT"
+}