link-cache 0.4.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -4,38 +4,41 @@ Zero-dependency helper CLIs for **cached** link checking with [Lychee][], for
4
4
  any static site that builds to a `public/` directory (Docsy, Hugo, and others).
5
5
  Two tools:
6
6
 
7
- - **`lychee-norm-cache`** run lychee over your built `public/` output, keeping
7
+ - **`lychee-norm-cache`**: run lychee over your built `public/` output, keeping
8
8
  the committed `link-cache.jsonc` cache and lychee's derived `.lycheecache` in
9
9
  sync.
10
- - **`link-cache`** inspect and prune the cache: list the oldest entries, prune
11
- a count or percentage (optionally scoped by URL regex), or print a summary
12
- (status, provenance, ages). (`refcache` is a deprecated alias.)
10
+ - **`link-cache`**: inspect and prune the cache. List the oldest entries, prune
11
+ a count or percentage (optionally scoped by URL regex; `manual` entries are
12
+ exempt, and those with an `expires` date retire then), print a summary
13
+ (result, provenance, ages), or run the staleness guard (`--max-age`).
14
+ (`refcache` is a deprecated alias.)
13
15
 
14
16
  With a committed `lychee.toml` and `link-cache.jsonc`, these give a site a
15
17
  self-contained, cached link-checking setup: fast reruns, and diffs that reflect
16
- real changes link statuses, and check recency for freshly re-verified entries.
18
+ real changes (link statuses, and check recency for freshly re-verified entries).
17
19
 
18
20
  ## The owned cache: `link-cache.jsonc`
19
21
 
20
- The committed source of truth is `link-cache.jsonc` a JSONC file,
22
+ The committed source of truth is `link-cache.jsonc`: a JSONC file,
21
23
  pretty-printed by construction in Prettier's style (so `prettier --check` passes
22
24
  it untouched), one multi-line object per URL, sorted, with `//` comments allowed
23
25
  on their own lines. Each comment attaches to the entry below it and survives
24
- rewrites for as long as the entry's status holds; a status change replaces the
25
- entry and retires its comments (a stale rationale is worse than none). The
26
- multi-line shape is deliberate: field-per-line entries keep concurrent updates
27
- merging cleanly under git's normal 3-way merge.
26
+ re-confirmations and recorded failures (the rationale still explains the URL);
27
+ an entry replaced by a different live HTTP result drops its comments (a stale
28
+ rationale is worse than none). The multi-line shape is deliberate:
29
+ field-per-line entries keep concurrent updates merging cleanly under git's
30
+ normal 3-way merge.
28
31
 
29
32
  ```jsonc
30
33
  {
31
34
  "https://example.com/": {
32
- "status": 200,
35
+ "result": 200,
33
36
  "when": "2026-08-29T20:06:38Z",
34
37
  "via": "lychee",
35
38
  },
36
39
  // Seeded pending my-org/repo#123; the target lands with that merge.
37
40
  "https://example.com/future-page/": {
38
- "status": 200,
41
+ "result": 200,
39
42
  "when": "2026-08-29T20:06:38Z",
40
43
  "via": "manual",
41
44
  "expires": "2026-09-30",
@@ -45,51 +48,109 @@ merging cleanly under git's normal 3-way merge.
45
48
 
46
49
  Each entry records:
47
50
 
48
- - **`status`** generalized: positive values are HTTP statuses; `0` is
49
- unchecked; negative values are tool-specific errors (`-10` timeout, `-20`
50
- network/DNS, `-30` certificate, `-40` generic client error).
51
- - **`when`** the moment the status was established, as RFC3339 UTC at whole
51
+ - **`result`**: an HTTP status int (`200`, `206`, …), or a failure word from
52
+ lychee's own tag vocabulary (`"error"`, `"timeout"`). Only 2xx results serve
53
+ as lychee cache hits; failure words and non-2xx results live in the owned file
54
+ only. (0.4.x files spelled this field `status` with negative error codes;
55
+ they're read compatibly and rewritten to `result` on the first run.)
56
+ - **`when`**: the moment the result was established, as RFC3339 UTC at whole
52
57
  seconds (`YYYY-MM-DDTHH:MM:SSZ`), converting exactly to and from lychee's
53
- epoch-seconds cache timestamps. The form is strict no fractional seconds, no
54
- offsets so timestamps are byte-comparable and lexicographically
58
+ epoch-seconds cache timestamps. The form is strict (no fractional seconds, no
59
+ offsets), so timestamps are byte-comparable and lexicographically
55
60
  chronological.
56
- - **`via`** the resolver that set the status: `lychee`, `manual`
61
+ - **`via`**: the resolver that set the result, one of `lychee`, `manual`
57
62
  (hand-seeded), or a named specialized resolver (e.g. a browser-grade probe).
58
- - **`expires`** (optional, `manual` entries) `YYYY-MM-DD`. Until then the
59
- entry is trusted (never re-checked, overriding lychee's `max_cache_age`);
60
- after that, it's re-checked live and replaced by the verified result.
63
+ Key hand-seeded entries by the URL exactly as lychee prints it: lowercase
64
+ host, no default port, resolved dot segments, and a `/` path on bare hosts
65
+ (`https://example.com/`, never `https://example.com`). Results merge back by
66
+ byte-for-byte key comparison, so a non-canonical spelling never matches its
67
+ re-check.
68
+ - **`expires`** (optional, `manual` entries): `YYYY-MM-DD`. Until then the entry
69
+ is trusted; after that, a `--check-stale` run re-checks it live and replaces
70
+ it with the verified result. Seed **2xx results only**: a seed's job is to
71
+ vouch that a URL is good so it isn't re-checked, and only 2xx results serve as
72
+ cache hits, so a non-2xx seed is re-checked every run, fails it, and is
73
+ replaced by the live result. For URLs whose expected status is non-2xx, use
74
+ `exclude` or `accept` instead.
61
75
 
62
76
  Lychee's own CSV cache, `.lycheecache`, is **derived**: `lychee-norm-cache`
63
77
  projects the owned cache into it before each run and folds lychee's results back
64
78
  afterwards. Gitignore `.lycheecache`; commit `link-cache.jsonc`. A re-check that
65
- changes an entry's status replaces the entry (provenance moves to `lychee`); a
79
+ changes an entry's result replaces the entry (provenance moves to `lychee`); a
66
80
  re-confirmation leaves provenance-bearing entries (`manual`, named resolvers)
67
- untouched, while `lychee`-owned entries refresh their `when` to record recency.
68
- A URL the run itself reports as failing is recorded as a negative tool-error
69
- status; an entry that merely goes missing from lychee's CSV is left untouched
81
+ untouched, while a live re-check of a `lychee`-owned entry refreshes its `when`
82
+ to record recency (a default-mode cache hit is not a re-check and leaves `when`
83
+ untouched). A URL the run itself reports as failing is recorded with its failure
84
+ word; an entry that merely goes missing from lychee's CSV is left untouched
70
85
  (cache-status excludes, cache aging, and site changes all remove entries from
71
- healthy runs).
86
+ healthy runs). Failure evidence counts only on a dead-links exit, and new
87
+ failure entries mint for http(s) URLs only; for the rationale, see `mergeBack`'s
88
+ contract in `lib/cache.mjs`.
89
+
90
+ ## Two modes: PR checks vs. cache refresh
91
+
92
+ `lychee-norm-cache` applies staleness checks only on request:
93
+
94
+ - **Default (PR checks)**: every cached 2xx result is projected into lychee's
95
+ CSV with a fresh timestamp, so lychee's `max_cache_age` never triggers and
96
+ expired manual seeds aren't re-checked. A run verifies only URLs without a
97
+ cached 2xx result, that is, URLs new to the cache plus recorded failures and
98
+ non-2xx results (lychee's cache loader accepts success codes only, so those
99
+ never serve as hits and re-check on every run). Entries still age for real
100
+ (their `when` timestamps are untouched in the owned file); the default just
101
+ doesn't act on the age.
102
+ - **`--check-stale` (cache refresh)**: real timestamps are projected and
103
+ lychee's `max_cache_age` and manual `expires` dates apply, so stale and
104
+ expired entries are re-verified. Use this mode in a scheduled cache-refresh
105
+ job. In steady state, `link-cache --prune` and `--check-stale` runs are what
106
+ drive re-checks.
107
+
108
+ Because a default run never re-checks cached 2xx results, a stopped refresh job
109
+ lets the cache age silently. Guard against that with the staleness guard:
110
+
111
+ ```sh
112
+ link-cache --max-age 60 # exit 3 when the oldest refreshable entry is >60 days old
113
+ ```
114
+
115
+ Under a healthy `--check-stale` schedule the oldest entries legitimately age up
116
+ to `max_cache_age` plus one refresh interval before their re-check lands, so set
117
+ the threshold **at or above that sum** (a lower threshold fails on healthy
118
+ caches); a prune-driven rotation sizes the threshold to its full prune cycle
119
+ instead. Run the guard where its failure gets seen (the refresh job itself, or
120
+ another scheduled check). The guard ages lychee-owned entries by their check
121
+ time, and expired manual seeds by their `expires` date; unexpired seeds and
122
+ named-resolver entries are exempt (their lifecycle is owned elsewhere). It also
123
+ fails on evidence it can't trust: malformed cache lines, or a future-dated
124
+ timestamp anywhere among the aged entries.
72
125
 
73
126
  Without a `link-cache.jsonc`, `lychee-norm-cache` falls back to the legacy mode:
74
- normalize the committed `.lycheecache` in place. To migrate:
127
+ normalize the committed `.lycheecache` in place. To import an existing CSV
128
+ cache:
75
129
 
76
130
  ```sh
77
- npm run check:links -- --migrate # .lycheecache -> link-cache.jsonc
131
+ npm run check:links -- --import # .lycheecache -> link-cache.jsonc
78
132
  ```
79
133
 
80
134
  then commit `link-cache.jsonc` and gitignore `.lycheecache`. If the CSV cache
81
135
  carried a `merge=union` gitattribute, drop it rather than moving it over: union
82
136
  merging proved ineffective in practice, and on a multi-line file it can
83
137
  interleave entries into invalid JSON. The owned cache merges with git's normal
84
- 3-way merge; on a conflict, resolve either way and rerun the check the next
85
- run re-normalizes the file.
138
+ 3-way merge; on a conflict, resolve either way and rerun the check; the next run
139
+ re-normalizes the file.
140
+
141
+ In your `lychee.toml`, prefer URL-scoped mechanisms (`exclude` patterns, or
142
+ manual seeds in the owned cache) for URL-specific problems, and reserve lychee's
143
+ `accept` list for statuses that are acceptable **site-wide**: an accepted status
144
+ is recorded in the committed cache for every URL that returns it. Note that
145
+ `accept` buys no caching: only 2xx results serve as cache hits, so an accepted
146
+ non-2xx URL is re-checked on every run.
86
147
 
87
148
  ## Exit codes (`lychee-norm-cache`)
88
149
 
89
- - `0` success.
90
- - `1` dead links: the check ran and found failures.
91
- - `2` preflight or sanity failure: lychee or `public/` missing, lychee config
92
- error, or **zero links checked** (an empty or fully-excluded `public/` is a
150
+ - `0`: success.
151
+ - `1`: dead links (the check ran and found failures).
152
+ - `2`: preflight or sanity failure (lychee or `public/` missing, lychee config
153
+ error, or **zero links checked**; an empty or fully-excluded `public/` is a
93
154
  false-clean, not a pass).
94
155
 
95
156
  Warn-style wrappers can soften exit 1 (advisory link rot) while still failing
@@ -101,8 +162,9 @@ hard on exit 2 (the check didn't actually run).
101
162
  - A `lychee.toml` at your site root (lychee's config and ignore rules).
102
163
  - A built site under `public/` (run your site build first).
103
164
  - [Node.js][] ≥ 24.
104
- - Optional: the [`gh`][gh] CLI `lychee-norm-cache` bridges its token to lychee
105
- to raise the github.com rate limit when `GITHUB_TOKEN` isn't already set.
165
+ - Optional: the [`gh`][gh] CLI, whose token `lychee-norm-cache` bridges to
166
+ lychee to raise the github.com rate limit when `GITHUB_TOKEN` isn't already
167
+ set.
106
168
 
107
169
  ## Install
108
170
 
@@ -113,14 +175,14 @@ npm install --save-dev link-cache
113
175
  Or, to install from GitHub rather than the npm registry:
114
176
 
115
177
  ```sh
116
- npm install --save-dev github:chalin/link-cache#semver:^0.4.0
178
+ npm install --save-dev github:chalin/link-cache#semver:^0.5.0
117
179
  ```
118
180
 
119
181
  This puts both bins on your project's `PATH`.
120
182
 
121
183
  ## Usage
122
184
 
123
- Wire the bins into your `package.json` scripts (bare names `npm run` puts
185
+ Wire the bins into your `package.json` scripts, using bare names (`npm run` puts
124
186
  `node_modules/.bin` on the `PATH`):
125
187
 
126
188
  ```json
@@ -131,9 +193,11 @@ Wire the bins into your `package.json` scripts (bare names — `npm run` puts
131
193
  ```
132
194
 
133
195
  ```sh
134
- npm run check:links # sync caches, run lychee, fold results back
135
- npm run link-cache -- --summary # cache stats (count, oldest, status, via, ages)
196
+ npm run check:links # verify URLs not yet in the cache
197
+ npm run check:links -- --check-stale # also re-verify stale/expired entries
198
+ npm run link-cache -- --summary # cache stats (count, oldest, result, via, ages)
136
199
  npm run link-cache -- --match 'github\.com' --prune 10 # trim 10 oldest matching
200
+ npm run link-cache -- --max-age 60 # staleness guard (exit 3 when breached)
137
201
  ```
138
202
 
139
203
  > [!WARNING]
@@ -141,13 +205,15 @@ npm run link-cache -- --match 'github\.com' --prune 10 # trim 10 oldest matchin
141
205
  > Don't invoke these bins via `npx`: on a stale or missing `node_modules`, `npx`
142
206
  > falls back to the public registry and runs **whatever package holds the bin's
143
207
  > name there** (the `lychee-norm-cache` name is squatted). Bare bin names in
144
- > `npm run` scripts resolve locally or fail loudly they never touch the
208
+ > `npm run` scripts resolve locally or fail loudly; they never touch the
145
209
  > registry.
146
210
 
147
211
  `lychee-norm-cache` runs in the current directory (your site root) and forwards
148
212
  any extra arguments to lychee. Run either tool with `--help` for its full
149
213
  options, and `lychee --help` for the link-checking flags `lychee-norm-cache`
150
- forwards (e.g. `--offline`, `--max-cache-age 0`).
214
+ forwards (e.g. `--offline`). To force re-checks, use `--check-stale`; the
215
+ default mode rejects a forwarded `--max-cache-age`, whose file-age check would
216
+ silently defeat the fresh-timestamp projection.
151
217
 
152
218
  ## Development
153
219
 
package/check/index.mjs CHANGED
@@ -11,13 +11,21 @@
11
11
  // or public/ missing, zero links checked, lychee config error).
12
12
 
13
13
  import { spawnSync } from 'node:child_process';
14
- import { existsSync, readFileSync, realpathSync, statSync } from 'node:fs';
14
+ import {
15
+ existsSync,
16
+ readFileSync,
17
+ realpathSync,
18
+ rmSync,
19
+ statSync,
20
+ } from 'node:fs';
15
21
  import path from 'node:path';
16
22
  import { fileURLToPath } from 'node:url';
17
23
 
18
24
  import {
19
25
  CSV_FILE,
20
26
  OWNED_FILE,
27
+ RESULT_ERROR,
28
+ RESULT_TIMEOUT,
21
29
  mergeBack,
22
30
  migrateCsvText,
23
31
  parseCsv,
@@ -34,7 +42,7 @@ export const EXIT_OK = 0;
34
42
  export const EXIT_DEAD_LINKS = 1;
35
43
  export const EXIT_PREFLIGHT = 2;
36
44
 
37
- const USAGE = `Usage: lychee-norm-cache [--migrate] [lychee args...]
45
+ const USAGE = `Usage: lychee-norm-cache [--check-stale] [--import] [lychee args...]
38
46
 
39
47
  Run lychee over this site's built ./public output. With a committed
40
48
  ${OWNED_FILE}, the ${CSV_FILE} handed to lychee is derived from it before the
@@ -42,8 +50,16 @@ run and folded back into it afterwards; otherwise ${CSV_FILE} is normalized in
42
50
  place (legacy mode). Bridges a GitHub token from the gh CLI when GITHUB_TOKEN
43
51
  isn't set; extra arguments pass through to lychee.
44
52
 
45
- --migrate convert an existing ${CSV_FILE} to ${OWNED_FILE} and exit
46
- -h, --help show this help
53
+ By default staleness checks are not applied: cached 2xx results project with
54
+ fresh timestamps, so a run verifies only URLs without a cached 2xx result
55
+ (failure words and non-2xx results never serve as cache hits and re-check on
56
+ every run). With --check-stale, real timestamps project, so lychee's
57
+ max_cache_age and manual expires dates apply and stale entries are re-verified
58
+ (the scheduled cache-refresh job's mode).
59
+
60
+ --check-stale apply staleness checks (re-verify stale and expired entries)
61
+ --import convert an existing ${CSV_FILE} to ${OWNED_FILE} and exit
62
+ -h, --help show this help
47
63
 
48
64
  Exit codes: 0 success; 1 dead links; 2 preflight/sanity failure (missing
49
65
  lychee or public/, lychee config or usage errors, zero links verified).
@@ -92,10 +108,19 @@ export function publicDirOf(cwd) {
92
108
  return publicDir;
93
109
  }
94
110
 
111
+ // lychee colors its output when CLICOLOR_FORCE is set (even onto the piped
112
+ // stdout we capture, since the spawn env passes process.env through), and SGR
113
+ // codes around the tags defeat the line parsers (captured live from 0.24.2),
114
+ // so failures would vanish silently. Strip before parsing.
115
+ function stripAnsi(s) {
116
+ return s.replace(/\x1b\[[0-9;]*m/g, '');
117
+ }
118
+
95
119
  // Check counts from lychee's stdout summary — the human line ("🔍 2 Total …
96
120
  // ✅ 1 OK 🚫 0 Errors …") or --format json fields; null when neither is
97
121
  // present. A parsed summary is the proof that a check actually completed.
98
122
  export function parseSummary(stdout) {
123
+ stdout = stripAnsi(stdout);
99
124
  const m = /(\d+)\s+Total\b[\s\S]*?(\d+)\s+OK\b[\s\S]*?(\d+)\s+Errors?\b/.exec(
100
125
  stdout,
101
126
  );
@@ -120,20 +145,47 @@ export function mapLycheeExit(code, summary) {
120
145
  return EXIT_PREFLIGHT;
121
146
  }
122
147
 
123
- // URLs the run itself reported as failing the human "[ERROR] URL …" lines,
124
- // or --format json's fail_map. Positive per-URL evidence for the merge-back's
125
- // failure recording; CSV absence alone proves nothing (cache_exclude_status,
126
- // max_cache_age, and site changes all remove entries from healthy runs).
148
+ // URLs the run itself reported as failing, mapped to a failure word from
149
+ // lychee's own tag vocabulary ("error" or "timeout"): the human per-URL
150
+ // lines, or --format json's failure maps. Positive per-URL evidence for the
151
+ // merge-back's failure recording (CSV absence alone proves nothing:
152
+ // merge-back rules, lib/cache.mjs).
153
+ //
154
+ // Line shapes (lychee 0.24, verified against live output and Status::
155
+ // code_as_string in its source): failures carry the word tags ERROR or
156
+ // TIMEOUT, or a numeric status tag with a failure remark ("[403] URL … |
157
+ // Rejected status code: 403 Forbidden", "[404] URL | Error (cached)").
158
+ // Numeric tags alone are not failures: -vv prints accepted URLs the same way
159
+ // ("[200] URL (at 1:1)", "[200] URL | OK (cached)"). Word tags are
160
+ // whitelisted, not blacklisted: the others (EXCLUDED; IGNORED, unsupported
161
+ // URLs printed on green runs; UNKNOWN, mail outside lychee's is_error) are
162
+ // non-failures, as are lychee's log-level lines ([INFO], [WARN], …), and a
163
+ // recorded failure becomes a committed cache entry (merge-back rules again),
164
+ // making a false positive costlier than a false negative. For the same reason
165
+ // the URL token must look like a URL: absolute with scheme:// or mailto:
166
+ // (the one scheme://-less form lychee checks).
167
+ const FAILURE_TAGS = new Set(['ERROR', 'TIMEOUT']);
168
+ const URL_SHAPE = /^(\w[\w+.-]*:\/\/|mailto:)/;
127
169
  export function parseFailedUrls(stdout) {
128
- const failed = new Set();
170
+ stdout = stripAnsi(stdout);
171
+ const failed = new Map();
129
172
  const trimmed = stdout.trim();
130
173
  if (trimmed.startsWith('{')) {
174
+ // --format json, possibly with a trailing human "Hint:" line after the
175
+ // document. Failures live in error_map/timeout_map (0.24's shape);
176
+ // fail_map covers older lychee.
131
177
  try {
132
- const failMap = JSON.parse(trimmed).fail_map ?? {};
133
- for (const failures of Object.values(failMap)) {
134
- for (const f of failures) {
135
- const url = typeof f.url === 'string' ? f.url : f.url?.url;
136
- if (url) failed.add(url);
178
+ const json = JSON.parse(trimmed.slice(0, trimmed.lastIndexOf('}') + 1));
179
+ for (const [map, word] of [
180
+ [json.fail_map, RESULT_ERROR],
181
+ [json.error_map, RESULT_ERROR],
182
+ [json.timeout_map, RESULT_TIMEOUT],
183
+ ]) {
184
+ for (const failures of Object.values(map ?? {})) {
185
+ for (const f of failures) {
186
+ const url = typeof f.url === 'string' ? f.url : f.url?.url;
187
+ if (url) failed.set(url, word);
188
+ }
137
189
  }
138
190
  }
139
191
  return failed;
@@ -141,8 +193,24 @@ export function parseFailedUrls(stdout) {
141
193
  // fall through to the line scan
142
194
  }
143
195
  }
144
- for (const m of stdout.matchAll(/^\s*\[ERROR\]\s+(\S+)/gm)) {
145
- failed.add(m[1]);
196
+ for (const m of stdout.matchAll(/^\s*\[(\w+)\]\s+(\S+)(.*)$/gm)) {
197
+ const [, tag, url, rest] = m;
198
+ if (!URL_SHAPE.test(url)) continue;
199
+ if (/^\d+$/.test(tag)) {
200
+ if (
201
+ /\|\s*(Rejected|Failed|Error \(cached\)|Request timed out)/.test(rest)
202
+ ) {
203
+ failed.set(
204
+ url,
205
+ /Request timed out/.test(rest) ? RESULT_TIMEOUT : RESULT_ERROR,
206
+ );
207
+ }
208
+ } else if (FAILURE_TAGS.has(tag.toUpperCase())) {
209
+ failed.set(
210
+ url,
211
+ tag.toUpperCase() === 'TIMEOUT' ? RESULT_TIMEOUT : RESULT_ERROR,
212
+ );
213
+ }
146
214
  }
147
215
  return failed;
148
216
  }
@@ -163,12 +231,12 @@ function fail(message) {
163
231
  return EXIT_PREFLIGHT;
164
232
  }
165
233
 
166
- function migrate(cwd) {
234
+ function importCsv(cwd) {
167
235
  const csvPath = path.join(cwd, CSV_FILE);
168
236
  const ownedPath = path.join(cwd, OWNED_FILE);
169
237
  if (existsSync(ownedPath)) return fail(`${OWNED_FILE} already exists.`);
170
238
  if (!existsSync(csvPath)) return fail(`${CSV_FILE} not found.`);
171
- const { text, count, malformed } = migrateCsvText(
239
+ const { text, count, malformed, conflicting, unmappable } = migrateCsvText(
172
240
  readFileSync(csvPath, 'utf8'),
173
241
  );
174
242
  // Migration is specified lossless: refuse rather than silently drop.
@@ -177,9 +245,19 @@ function migrate(cwd) {
177
245
  `${CSV_FILE} has ${malformed} malformed line(s); fix or remove them, then rerun.`,
178
246
  );
179
247
  }
248
+ if (conflicting) {
249
+ return fail(
250
+ `${CSV_FILE} has ${conflicting} URL(s) with conflicting duplicate rows; fix or remove them, then rerun.`,
251
+ );
252
+ }
253
+ if (unmappable) {
254
+ return fail(
255
+ `${CSV_FILE} has ${unmappable} entr${unmappable === 1 ? 'y' : 'ies'} with an unmappable status or timestamp; fix or remove them, then rerun.`,
256
+ );
257
+ }
180
258
  writeFileAtomic(ownedPath, text);
181
259
  console.log(
182
- `Migrated ${count} entries to ${OWNED_FILE}. Commit it and gitignore ${CSV_FILE}.`,
260
+ `Imported ${count} ${count === 1 ? 'entry' : 'entries'} to ${OWNED_FILE}. Commit it and gitignore ${CSV_FILE}.`,
183
261
  );
184
262
  return EXIT_OK;
185
263
  }
@@ -191,7 +269,11 @@ function main(argv) {
191
269
  }
192
270
 
193
271
  const cwd = process.cwd();
194
- if (argv.includes('--migrate')) return migrate(cwd);
272
+ if (argv.includes('--import')) return importCsv(cwd);
273
+
274
+ // Wrapper-owned flag: consumed here, never forwarded to lychee.
275
+ const checkStale = argv.includes('--check-stale');
276
+ argv = argv.filter((a) => a !== '--check-stale');
195
277
 
196
278
  if (!hasLychee()) {
197
279
  return fail(`lychee not found. Install: ${INSTALL_HINT}`);
@@ -217,14 +299,28 @@ function main(argv) {
217
299
  if (owned && argv.includes('--cache=false')) {
218
300
  return fail(`--cache=false is incompatible with ${OWNED_FILE}.`);
219
301
  }
302
+ // A forwarded cache-age flag is either inert (the fresh-timestamp
303
+ // projection defeats any realistic value) or, near zero, discards the
304
+ // just-written cache wholesale via lychee's file-age check; either way it
305
+ // contradicts the flag's intent, and --check-stale is the sanctioned
306
+ // re-check path.
307
+ const hasCacheAgeFlag = argv.some(
308
+ (a) => a === '--max-cache-age' || a.startsWith('--max-cache-age='),
309
+ );
310
+ if (owned && !checkStale && hasCacheAgeFlag) {
311
+ return fail(
312
+ '--max-cache-age has no effect on cached entries in the default mode; use --check-stale to re-verify stale entries.',
313
+ );
314
+ }
220
315
  const cacheArgs = hasCacheFlag ? [] : ['--cache'];
221
316
 
222
- // Derive the CSV lychee will read from the owned cache.
317
+ // Derive the CSV lychee will read from the owned cache; keep the projected
318
+ // timestamps so merge-back can tell echoed cache hits from real re-checks.
319
+ let projectedTs = new Map();
223
320
  if (owned) {
224
- writeFileAtomic(
225
- cachePath,
226
- serializeCsv(projectToCsv(owned.entries, { now })),
227
- );
321
+ const projected = projectToCsv(owned.entries, { now, checkStale });
322
+ projectedTs = new Map(projected.map((e) => [e.url, e.ts]));
323
+ writeFileAtomic(cachePath, serializeCsv(projected));
228
324
  }
229
325
 
230
326
  const token = resolveToken();
@@ -250,16 +346,26 @@ function main(argv) {
250
346
  },
251
347
  );
252
348
  if (run.stdout) process.stdout.write(run.stdout);
349
+
350
+ // A failed or verdict-free run never folds; remove the derived CSV rather
351
+ // than leave it behind (the projection's fresh timestamps are a lie about
352
+ // recency once no completed run consumed them).
353
+ const bail = (message) => {
354
+ if (owned) rmSync(cachePath, { force: true });
355
+ return fail(message);
356
+ };
357
+
253
358
  if (run.error) {
254
- return fail(`lychee failed to run: ${run.error.message}`);
359
+ return bail(`lychee failed to run: ${run.error.message}`);
255
360
  }
256
361
  const summary = parseSummary(run.stdout ?? '');
257
362
  const status = mapLycheeExit(run.status ?? 1, summary);
258
363
 
259
- // On a preflight failure lychee produced no trustworthy results: leave both
260
- // caches untouched (folding would mislabel unprojected entries as failed).
364
+ // On a preflight failure lychee produced no trustworthy results: leave the
365
+ // owned cache untouched (folding would mislabel unprojected entries as
366
+ // failed).
261
367
  if (status === EXIT_PREFLIGHT) {
262
- return fail('lychee did not complete a check; caches left untouched.');
368
+ return bail('lychee did not complete a check; owned cache left untouched.');
263
369
  }
264
370
 
265
371
  // A clean run in which nothing got a verdict is a false-clean (empty or
@@ -267,7 +373,7 @@ function main(argv) {
267
373
  // unattempted entries as failed, so the check precedes the fold. Cache hits
268
374
  // count toward OK in lychee's summary, so a fully-cached run passes.
269
375
  if (status === EXIT_OK && summary && summary.ok + summary.errors === 0) {
270
- return fail('lychee verified 0 links: empty or fully-excluded public/?');
376
+ return bail('lychee verified 0 links: empty or fully-excluded public/?');
271
377
  }
272
378
 
273
379
  // Fold the post-run CSV back into the owned cache (also on dead links, so a
@@ -277,10 +383,19 @@ function main(argv) {
277
383
  const csvEntries = existsSync(cachePath)
278
384
  ? parseCsv(readFileSync(cachePath, 'utf8')).entries
279
385
  : [];
280
- const failedUrls = parseFailedUrls(run.stdout ?? '');
386
+ // Failure evidence counts only on a dead-links exit: a green run means
387
+ // lychee accepted everything it printed (--accept-timeouts runs print
388
+ // "[TIMEOUT] URL …" lines while exiting 0), so recording those would mint
389
+ // and churn failure entries on every clean run.
390
+ const failedUrls =
391
+ status === EXIT_DEAD_LINKS
392
+ ? parseFailedUrls(run.stdout ?? '')
393
+ : new Map();
281
394
  writeFileAtomic(
282
395
  ownedPath,
283
- serializeOwned(mergeBack(owned, csvEntries, { now, failedUrls })),
396
+ serializeOwned(
397
+ mergeBack(owned, csvEntries, { now, failedUrls, projectedTs }),
398
+ ),
284
399
  );
285
400
  writeFileAtomic(cachePath, sortCacheText(readFileSync(cachePath, 'utf8')));
286
401
  } else if (existsSync(cachePath)) {
package/lib/cache.mjs CHANGED
@@ -10,10 +10,14 @@
10
10
  // raw URL byte order (matching LC_ALL=C). Stripping comments and trailing
11
11
  // commas yields valid JSON.
12
12
  //
13
- // Entry schema: { "status": INT, "when": RFC3339_UTC, "via": "RESOLVER" }
14
- // with an optional "expires": "YYYY-MM-DD" (manual entries). Status
15
- // generalizes HTTP: positive = HTTP status, 0 = unchecked, negative =
16
- // tool-specific error. `when` is the canonical whole-second UTC form
13
+ // Entry schema: { "result": INT|WORD, "when": RFC3339_UTC, "via": "RESOLVER" }
14
+ // with an optional "expires": "YYYY-MM-DD" (manual entries). `result` is
15
+ // heterogeneous: an HTTP status int (100-999, the only values that project
16
+ // into Lychee's CSV), or a failure word from lychee's own tag vocabulary
17
+ // ("error", "timeout"). Legacy 0.4.x files spell the field "status" with
18
+ // negative tool-error codes; parseOwned accepts that on read and maps it, so
19
+ // caches self-migrate on the first rewrite (read-old-write-new, one version).
20
+ // `when` is the canonical whole-second UTC form
17
21
  // (YYYY-MM-DDTHH:MM:SSZ) — no fractional seconds, no offsets — validated
18
22
  // strictly: this tool is the file's only writer, so a non-canonical value is
19
23
  // a bug to surface, not smooth over (lesson from the htmltest-era
@@ -22,11 +26,18 @@
22
26
  export const OWNED_FILE = 'link-cache.jsonc';
23
27
  export const CSV_FILE = '.lycheecache';
24
28
 
25
- // Tool-specific error statuses (the htmltest-fork convention).
26
- export const STATUS_TIMEOUT = -10;
27
- export const STATUS_NETWORK_ERROR = -20;
28
- export const STATUS_CERT_ERROR = -30;
29
- export const STATUS_CLIENT_ERROR = -40;
29
+ // Failure words (lychee's tag vocabulary, lowercased).
30
+ export const RESULT_ERROR = 'error';
31
+ export const RESULT_TIMEOUT = 'timeout';
32
+ const FAILURE_WORDS = new Set([RESULT_ERROR, RESULT_TIMEOUT]);
33
+
34
+ // Legacy 0.4.x numeric tool-error codes -> failure words (read-side only).
35
+ const LEGACY_STATUS_WORDS = new Map([
36
+ [-10, RESULT_TIMEOUT],
37
+ [-20, RESULT_ERROR],
38
+ [-30, RESULT_ERROR],
39
+ [-40, RESULT_ERROR],
40
+ ]);
30
41
 
31
42
  const WHEN_RE = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/;
32
43
 
@@ -49,11 +60,16 @@ const byUrlBytes = (a, b) =>
49
60
 
50
61
  // --- CSV (.lycheecache) ------------------------------------------------------
51
62
 
52
- // Unquote a CSV field: lychee quotes URLs that contain a comma.
63
+ // Unquote a CSV field: lychee quotes URLs that contain a comma or a quote.
64
+ // Null when the quote grammar is violated: an unquoted field may not contain
65
+ // quotes, and a quoted field's interior quotes must be doubled.
53
66
  export function csvUnquote(field) {
54
- return field.startsWith('"') && field.endsWith('"')
55
- ? field.slice(1, -1).replace(/""/g, '"')
56
- : field;
67
+ if (field.startsWith('"') && field.endsWith('"') && field.length >= 2) {
68
+ const inner = field.slice(1, -1);
69
+ if (inner.replace(/""/g, '').includes('"')) return null;
70
+ return inner.replace(/""/g, '"');
71
+ }
72
+ return field.includes('"') ? null : field;
57
73
  }
58
74
 
59
75
  export function csvQuote(url) {
@@ -62,7 +78,8 @@ export function csvQuote(url) {
62
78
 
63
79
  // Parse one URL,STATUS,TIMESTAMP line; null when malformed. Lexically strict:
64
80
  // STATUS and TIMESTAMP must be non-empty digit runs (`Number('')` is 0, which
65
- // would silently coin status-0/epoch-0 entries from truncated lines).
81
+ // would silently coin status-0/epoch-0 entries from truncated lines), and the
82
+ // URL field must satisfy the CSV quote grammar (csvUnquote).
66
83
  export function parseCsvLine(raw) {
67
84
  const lastComma = raw.lastIndexOf(',');
68
85
  if (lastComma < 0) return null;
@@ -73,12 +90,15 @@ export function parseCsvLine(raw) {
73
90
  const statusField = head.slice(statusComma + 1).trim();
74
91
  if (!/^\d+$/.test(tsField) || !/^-?\d+$/.test(statusField)) return null;
75
92
  const url = csvUnquote(head.slice(0, statusComma));
76
- if (url === '') return null;
93
+ if (url === null || url === '') return null;
77
94
  return { url, status: Number(statusField), ts: Number(tsField) };
78
95
  }
79
96
 
80
97
  // Parse a whole CSV cache; malformed lines are counted, duplicates keep the
81
- // newest timestamp.
98
+ // newest timestamp. Ambiguity is order-independent: a newer row resolves an
99
+ // older equal-ts status conflict, and only ties among the final (newest)
100
+ // winners count as conflicting: both such rows are dropped so a one-shot
101
+ // consumer (import) can refuse.
82
102
  export function parseCsv(text) {
83
103
  const byUrl = new Map();
84
104
  let malformed = 0;
@@ -90,9 +110,22 @@ export function parseCsv(text) {
90
110
  continue;
91
111
  }
92
112
  const prior = byUrl.get(entry.url);
93
- if (!prior || entry.ts >= prior.ts) byUrl.set(entry.url, entry);
113
+ if (!prior || entry.ts > prior.entry.ts) {
114
+ byUrl.set(entry.url, { entry, conflicted: false });
115
+ } else if (
116
+ entry.ts === prior.entry.ts &&
117
+ entry.status !== prior.entry.status
118
+ ) {
119
+ prior.conflicted = true;
120
+ }
121
+ }
122
+ const entries = [];
123
+ let conflicting = 0;
124
+ for (const { entry, conflicted } of byUrl.values()) {
125
+ if (conflicted) conflicting += 1;
126
+ else entries.push(entry);
94
127
  }
95
- return { entries: [...byUrl.values()].sort(byUrlBytes), malformed };
128
+ return { entries: entries.sort(byUrlBytes), malformed, conflicting };
96
129
  }
97
130
 
98
131
  export function serializeCsv(entries) {
@@ -128,7 +161,7 @@ function stripTrailingCommas(lines) {
128
161
  return out;
129
162
  }
130
163
 
131
- const ENTRY_FIELDS = new Set(['status', 'when', 'via', 'expires']);
164
+ const ENTRY_FIELDS = new Set(['result', 'when', 'via', 'expires']);
132
165
 
133
166
  // A calendar-real YYYY-MM-DD (round-trip check rejects e.g. Feb 30).
134
167
  function isCanonicalDate(s) {
@@ -137,26 +170,43 @@ function isCanonicalDate(s) {
137
170
  return Number.isFinite(t) && new Date(t).toISOString().slice(0, 10) === s;
138
171
  }
139
172
 
140
- // Rejects unknown fields (a rewrite would silently drop them) and restricts
141
- // `expires` to manual entries resolvers set statuses, owners set trust
142
- // windows. Relax the via restriction if a named resolver ever needs expiry.
143
- // Statuses live in one of three domains: HTTP (100-999, the only ones that
144
- // project into Lychee's CSV — Lychee rejects codes outside that range at
145
- // load, which would wedge the run), 0 (unchecked), negative (tool errors).
146
- function isValidStatus(s) {
147
- return Number.isInteger(s) && (s < 0 || s === 0 || (s >= 100 && s <= 999));
173
+ // A result is an HTTP status (100-999, the only values that project into
174
+ // Lychee's CSV, which rejects codes outside that range at load) or a known
175
+ // failure word.
176
+ function isValidResult(r) {
177
+ return typeof r === 'string'
178
+ ? FAILURE_WORDS.has(r)
179
+ : Number.isInteger(r) && r >= 100 && r <= 999;
148
180
  }
149
181
 
182
+ // A legacy `status` field is accepted on read and mapped to `result` (never
183
+ // both): the 0.4.x negative codes become failure words; status 0 (unchecked,
184
+ // never tool-written) has no successor by design and fails validation as
185
+ // malformed.
186
+ function migrateLegacyStatus(v) {
187
+ if (!('status' in v) || 'result' in v) return v;
188
+ const { status, ...rest } = v;
189
+ const result =
190
+ Number.isInteger(status) && status < 0
191
+ ? LEGACY_STATUS_WORDS.get(status)
192
+ : status;
193
+ return isValidResult(result) ? { ...rest, result } : v;
194
+ }
195
+
196
+ // Rejects unknown fields (a rewrite would silently drop them) and restricts
197
+ // `expires` to manual entries: resolvers set results, owners set trust
198
+ // windows. Relax the via restriction if a named resolver ever needs expiry.
150
199
  function validateEntry(url, v) {
151
200
  return (
152
201
  v !== null &&
153
202
  typeof v === 'object' &&
154
203
  Object.keys(v).every((k) => ENTRY_FIELDS.has(k)) &&
155
- isValidStatus(v.status) &&
204
+ isValidResult(v.result) &&
156
205
  typeof v.when === 'string' &&
157
206
  Number.isFinite(whenToTs(v.when)) &&
158
207
  whenToTs(v.when) >= 0 && // pre-epoch would project a negative CSV ts
159
208
  typeof v.via === 'string' &&
209
+ v.via !== '' &&
160
210
  (v.expires === undefined ||
161
211
  (v.via === 'manual' && isCanonicalDate(v.expires)))
162
212
  );
@@ -217,13 +267,14 @@ export function parseOwned(text) {
217
267
  }
218
268
 
219
269
  const entries = [];
220
- for (const [url, v] of Object.entries(obj)) {
270
+ for (const [url, raw] of Object.entries(obj)) {
271
+ const v = migrateLegacyStatus(raw);
221
272
  if (!validateEntry(url, v)) {
222
273
  throw new Error(`${OWNED_FILE}: malformed entry for ${url}`);
223
274
  }
224
275
  entries.push({
225
276
  url,
226
- status: v.status,
277
+ result: v.result,
227
278
  ts: whenToTs(v.when), // epoch internally; `when` re-derives on write
228
279
  via: v.via,
229
280
  expires: v.expires,
@@ -236,9 +287,9 @@ export function parseOwned(text) {
236
287
  // Canonical serialization = Prettier's jsonc style (2-space indent, one field
237
288
  // per line, trailing commas, always multi-line), so the output is
238
289
  // prettier-idempotent by construction.
239
- export function serializeEntry({ url, status, ts, via, expires }) {
290
+ export function serializeEntry({ url, result, ts, via, expires }) {
240
291
  const fields = [
241
- ` "status": ${status},`,
292
+ ` "result": ${JSON.stringify(result)},`,
242
293
  ` "when": ${JSON.stringify(tsToWhen(ts))},`,
243
294
  ` "via": ${JSON.stringify(via)},`,
244
295
  ];
@@ -263,40 +314,68 @@ export function serializeOwned({ entries, trailing = [] }) {
263
314
 
264
315
  const dateToTs = (isoDate) => Date.parse(`${isoDate}T23:59:59Z`) / 1000;
265
316
 
266
- // Project the owned entries into Lychee's CSV. Only positive (HTTP) statuses
267
- // project Lychee never persists errors, and would re-drop the rest. An
268
- // unexpired entry with `expires` projects with a fresh timestamp so it
269
- // outlives Lychee's max_cache_age until its own expiry; once expired it is
270
- // omitted, forcing a live re-check.
271
- export function projectToCsv(entries, { now = Date.now() / 1000 } = {}) {
317
+ // Project the owned entries into Lychee's CSV. Only 2xx results project:
318
+ // Lychee's cache loader accepts success codes only (verified against 0.24.2,
319
+ // which drops a non-2xx row at load and re-checks the URL live), and it never
320
+ // persists errors, so recorded failures and non-2xx results re-check on every
321
+ // run. By default every projected entry carries a fresh timestamp (staleness
322
+ // checks not applied): lychee's max_cache_age never triggers and expired
323
+ // seeds are not re-checked, so a run verifies only URLs without a cached 2xx
324
+ // result. With `checkStale`, real timestamps project so max_cache_age
325
+ // applies, and manual `expires` governs its entry: unexpired seeds still
326
+ // project fresh (the expiry date, not max_cache_age, owns a seed's lifecycle)
327
+ // while expired ones are omitted, forcing a live re-check.
328
+ export function projectToCsv(
329
+ entries,
330
+ { now = Date.now() / 1000, checkStale = false } = {},
331
+ ) {
332
+ const fresh = Math.floor(now);
272
333
  const projected = [];
273
334
  for (const e of entries) {
274
- if (e.status <= 0) continue;
335
+ if (typeof e.result !== 'number' || e.result < 200 || e.result > 299) {
336
+ continue;
337
+ }
338
+ if (!checkStale) {
339
+ projected.push({ url: e.url, status: e.result, ts: fresh });
340
+ continue;
341
+ }
275
342
  if (e.expires !== undefined) {
276
343
  const cutoff = dateToTs(e.expires);
277
344
  if (!Number.isFinite(cutoff) || now > cutoff) continue;
278
- projected.push({ url: e.url, status: e.status, ts: Math.floor(now) });
345
+ projected.push({ url: e.url, status: e.result, ts: fresh });
279
346
  continue;
280
347
  }
281
- projected.push({ url: e.url, status: e.status, ts: e.ts });
348
+ projected.push({ url: e.url, status: e.result, ts: e.ts });
282
349
  }
283
350
  return projected;
284
351
  }
285
352
 
286
353
  // --- lens: CSV -> owned merge-back ------------------------------------------
287
354
 
288
- // Fold a post-run CSV back into the owned entries. Rules (owner, 08-31;
289
- // absence-inference dropped 08-31 adversarial round):
355
+ // Fold a post-run CSV back into the owned entries. Rules:
290
356
  // - new URL: added with via "lychee";
291
- // - status changed: the whole entry is replaced (via "lychee", fresh fields,
357
+ // - result changed: the whole entry is replaced (via "lychee", fresh fields,
292
358
  // comments dropped — a stale rationale is worse than none);
293
- // - status equal: provenance-bearing entries (via != "lychee") stay
359
+ // - result equal: provenance-bearing entries (via != "lychee") stay
294
360
  // untouched — a re-confirmation; "lychee" entries adopt the fresh ts;
295
- // - expired `expires` entry present in the CSV: replaced wholesale (the seed
361
+ // - CSV row that merely echoes the projection (same result, the exact ts we
362
+ // projected, per `projectedTs`): the entry is untouched. Under the default
363
+ // fresh-ts projection a cache hit echoes the projected row byte-exactly
364
+ // (verified against lychee 0.24.2); adopting it would freshen `when` on
365
+ // every run, erasing the real ages the staleness guard reads, and would
366
+ // let the `expired` branch overwrite seeds no live check ever touched;
367
+ // - expired `expires` entry re-checked live: replaced wholesale (the seed
296
368
  // served its purpose);
297
- // - URL in `failedUrls` (positive evidence from the run's own error report):
298
- // recorded as STATUS_CLIENT_ERROR under via "lychee", keeping the entry's
299
- // comments (the rationale still explains the URL);
369
+ // - URL in `failedUrls` (positive evidence from the run's own error report,
370
+ // a Map of URL -> failure word): recorded under via "lychee" with that
371
+ // word as its result: for existing entries (keeping their comments: the
372
+ // rationale still explains the URL) whatever their scheme, and for URLs
373
+ // new to the cache when http(s), beating a residual CSV row for the same
374
+ // URL. Minting is http(s)-only because a failure word never projects, so
375
+ // it heals only via a later CSV success row, which never comes for
376
+ // file:// or mailto: (lychee persists only http(s) rows); file:// keys
377
+ // also embed machine-specific absolute paths. Those failures already fail
378
+ // the run via the exit code;
300
379
  // - entry missing from the CSV without failure evidence: untouched. CSV
301
380
  // absence is ambiguous — cache_exclude_status, max_cache_age expiry, and
302
381
  // URLs no longer in the site all remove entries from healthy runs — so it
@@ -304,11 +383,30 @@ export function projectToCsv(entries, { now = Date.now() / 1000 } = {}) {
304
383
  export function mergeBack(
305
384
  owned,
306
385
  csvEntries,
307
- { now = Date.now() / 1000, failedUrls = new Set() } = {},
386
+ {
387
+ now = Date.now() / 1000,
388
+ failedUrls = new Map(),
389
+ projectedTs = new Map(),
390
+ } = {},
308
391
  ) {
309
392
  const csvByUrl = new Map(csvEntries.map((e) => [e.url, e]));
393
+ const failed = new Map(failedUrls);
310
394
  const merged = [];
311
395
 
396
+ const failureEntry = (url, word, comments = []) => ({
397
+ url,
398
+ result: word,
399
+ ts: Math.floor(now),
400
+ via: 'lychee',
401
+ comments,
402
+ });
403
+
404
+ const takeFailure = (url) => {
405
+ const word = failed.get(url);
406
+ failed.delete(url);
407
+ return word;
408
+ };
409
+
312
410
  for (const entry of owned.entries) {
313
411
  const csv = csvByUrl.get(entry.url);
314
412
  csvByUrl.delete(entry.url);
@@ -318,14 +416,10 @@ export function mergeBack(
318
416
  (!Number.isFinite(dateToTs(entry.expires)) ||
319
417
  now > dateToTs(entry.expires));
320
418
 
321
- if (failedUrls.has(entry.url)) {
322
- merged.push({
323
- url: entry.url,
324
- status: STATUS_CLIENT_ERROR,
325
- ts: Math.floor(now),
326
- via: 'lychee',
327
- comments: entry.comments ?? [],
328
- });
419
+ if (failed.has(entry.url)) {
420
+ merged.push(
421
+ failureEntry(entry.url, takeFailure(entry.url), entry.comments ?? []),
422
+ );
329
423
  continue;
330
424
  }
331
425
 
@@ -334,10 +428,14 @@ export function mergeBack(
334
428
  continue;
335
429
  }
336
430
 
337
- if (expired || csv.status !== entry.status) {
431
+ const echoed =
432
+ csv.status === entry.result && csv.ts === projectedTs.get(entry.url);
433
+ if (echoed) {
434
+ merged.push(entry); // a cache hit of our own projection: no new fact
435
+ } else if (expired || csv.status !== entry.result) {
338
436
  merged.push({
339
437
  url: entry.url,
340
- status: csv.status,
438
+ result: csv.status,
341
439
  ts: csv.ts,
342
440
  via: 'lychee',
343
441
  comments: [],
@@ -350,7 +448,22 @@ export function mergeBack(
350
448
  }
351
449
 
352
450
  for (const csv of csvByUrl.values()) {
353
- merged.push({ ...csv, via: 'lychee', comments: [] });
451
+ if (failed.has(csv.url)) {
452
+ merged.push(failureEntry(csv.url, takeFailure(csv.url)));
453
+ continue;
454
+ }
455
+ merged.push({
456
+ url: csv.url,
457
+ result: csv.status,
458
+ ts: csv.ts,
459
+ via: 'lychee',
460
+ comments: [],
461
+ });
462
+ }
463
+
464
+ // Unmatched failures: http(s)-only minting, per the rules above.
465
+ for (const [url, word] of failed) {
466
+ if (/^https?:\/\//.test(url)) merged.push(failureEntry(url, word));
354
467
  }
355
468
 
356
469
  return { entries: merged.sort(byUrlBytes), trailing: owned.trailing };
@@ -359,11 +472,28 @@ export function mergeBack(
359
472
  // --- migration ---------------------------------------------------------------
360
473
 
361
474
  // One-shot .lycheecache -> link-cache.jsonc: every entry becomes via "lychee".
475
+ // Legacy negative codes (htmltest-era refcache CSVs) map to failure words;
476
+ // statuses with no result equivalent and timestamps outside the canonical
477
+ // `when` range are counted as unmappable, never written (the parser would
478
+ // reject the output: a false-clean import). Ambiguous duplicate rows surface
479
+ // via `conflicting` (parseCsv).
362
480
  export function migrateCsvText(csvText) {
363
- const { entries, malformed } = parseCsv(csvText);
364
- const owned = {
365
- entries: entries.map((e) => ({ ...e, via: 'lychee', comments: [] })),
366
- trailing: [],
481
+ const { entries, malformed, conflicting } = parseCsv(csvText);
482
+ const migrated = [];
483
+ let unmappable = 0;
484
+ for (const { url, status, ts } of entries) {
485
+ const result = status < 0 ? LEGACY_STATUS_WORDS.get(status) : status;
486
+ if (!isValidResult(result) || Number.isNaN(whenToTs(tsToWhen(ts)))) {
487
+ unmappable += 1;
488
+ continue;
489
+ }
490
+ migrated.push({ url, result, ts, via: 'lychee', comments: [] });
491
+ }
492
+ return {
493
+ text: serializeOwned({ entries: migrated, trailing: [] }),
494
+ count: migrated.length,
495
+ malformed,
496
+ conflicting,
497
+ unmappable,
367
498
  };
368
- return { text: serializeOwned(owned), count: entries.length, malformed };
369
499
  }
@@ -13,6 +13,7 @@ import { fileURLToPath } from 'node:url';
13
13
  import {
14
14
  CSV_FILE,
15
15
  OWNED_FILE,
16
+ csvUnquote,
16
17
  parseOwned,
17
18
  serializeOwned,
18
19
  } from '../lib/cache.mjs';
@@ -23,17 +24,24 @@ const BUCKET_COUNT = 5;
23
24
 
24
25
  // --- parsing ---------------------------------------------------------------
25
26
 
27
+ // Parse one URL,STATUS,TIMESTAMP line; null when malformed. Lexically strict:
28
+ // junk rows must count as malformed (the staleness guard fails on them)
29
+ // rather than pass as entries. Quote-grammar validation delegates to lib's
30
+ // csvUnquote; the URL keeps its raw (possibly quoted) spelling because prune
31
+ // rewrites lines byte-for-byte.
26
32
  function parseLine(raw) {
27
33
  const lastComma = raw.lastIndexOf(',');
28
34
  if (lastComma < 0) return null;
29
- const ts = Number(raw.slice(lastComma + 1));
35
+ const tsField = raw.slice(lastComma + 1);
30
36
  const head = raw.slice(0, lastComma);
31
37
  const statusComma = head.lastIndexOf(',');
32
38
  if (statusComma < 0) return null;
33
39
  const status = head.slice(statusComma + 1).trim();
34
- const url = head.slice(0, statusComma);
35
- if (!Number.isInteger(ts) || status === '') return null;
36
- return { url, status, ts };
40
+ const urlField = head.slice(0, statusComma);
41
+ if (!/^\d+$/.test(tsField) || !/^-?\d+$/.test(status)) return null;
42
+ const unquoted = csvUnquote(urlField);
43
+ if (unquoted === null || unquoted === '') return null;
44
+ return { url: urlField, status, ts: Number(tsField) };
37
45
  }
38
46
 
39
47
  // Parse a whole CSV cache, keeping the original lines (so a prune can rewrite
@@ -60,7 +68,7 @@ export function parseOwnedCache(text) {
60
68
  const owned = parseOwned(text);
61
69
  const entries = owned.entries.map((e, index) => ({
62
70
  url: e.url,
63
- status: String(e.status),
71
+ status: String(e.result),
64
72
  ts: e.ts,
65
73
  via: e.via,
66
74
  index,
@@ -197,7 +205,7 @@ export function formatStats(stats, { now = Date.now() / 1000, path } = {}) {
197
205
  );
198
206
  }
199
207
 
200
- lines.push(' Status:');
208
+ lines.push(' Result:');
201
209
  for (const [status, n] of stats.byStatus) {
202
210
  lines.push(` ${status.padEnd(5)} ${n}`);
203
211
  }
@@ -224,8 +232,8 @@ export function formatStats(stats, { now = Date.now() / 1000, path } = {}) {
224
232
  // --- ordered execution -----------------------------------------------------
225
233
 
226
234
  // Run the parsed ops in order over the evolving cache, optionally scoped to
227
- // URLs matching `match`. Pure: returns the text to print and the text to write
228
- // (null when nothing was pruned); no I/O.
235
+ // URLs matching `match`. Pure: returns the text to print and the text to
236
+ // write (null when nothing was pruned); no I/O.
229
237
  export function runOps(
230
238
  parsed,
231
239
  ops,
@@ -233,6 +241,7 @@ export function runOps(
233
241
  ) {
234
242
  const removed = new Set();
235
243
  let pruned = 0;
244
+ let guardFailed = false;
236
245
  const out = [];
237
246
  // Match against the unquoted URL: legacy-CSV entries carry the raw
238
247
  // (possibly CSV-quoted) field, which would defeat anchored regexes.
@@ -244,13 +253,74 @@ export function runOps(
244
253
  if (op.kind === 'list') {
245
254
  out.push(formatList(current(), Number(op.value), { now }));
246
255
  } else if (op.kind === 'prune') {
247
- const cur = current();
256
+ // Manual entries are exempt: pruning schedules re-checks for
257
+ // lychee-verified entries, while a manual entry's lifecycle is owned by
258
+ // its author and its `expires` date — and under prune-refresh rotation
259
+ // re-confirmed seeds keep their original timestamp, so they'd otherwise
260
+ // age to the front and get evicted mid-lifecycle.
261
+ const cur = current().filter((e) => e.via !== 'manual');
248
262
  const k = resolvePruneCount(op.value, cur.length);
249
263
  for (const victim of selectOldest(cur, k)) removed.add(victim.index);
250
264
  pruned += k;
251
265
  out.push(
252
266
  `Pruned ${k} oldest ${k === 1 ? 'entry' : 'entries'} (${cur.length} → ${cur.length - k}).`,
253
267
  );
268
+ } else if (op.kind === 'max-age') {
269
+ // Staleness guard: with staleness checks off by default in
270
+ // lychee-norm-cache, a dead refresh job rots the cache silently; an
271
+ // entry aging past the threshold is the signal. Refreshable entries are
272
+ // the lychee-owned ones (via-less legacy CSV entries included), aged by
273
+ // their check time, plus expired manual seeds, aged from their expiry
274
+ // date (a live refresh job replaces them within a rotation). Unexpired
275
+ // and no-expires seeds and named-resolver entries are exempt: their
276
+ // timestamps never refresh on re-confirmation, so their age says
277
+ // nothing about the refresh job.
278
+ const limit = Number(op.value);
279
+ if (parsed.malformed > 0) {
280
+ guardFailed = true;
281
+ out.push(
282
+ `Staleness guard: ${parsed.malformed} malformed line(s); the cache's age cannot be trusted.`,
283
+ );
284
+ } else {
285
+ const candidates = [];
286
+ for (const e of current()) {
287
+ if (e.via === undefined || e.via === 'lychee') {
288
+ candidates.push({ url: e.url, ts: e.ts });
289
+ } else if (e.via === 'manual' && e.src?.expires !== undefined) {
290
+ const cutoff = Date.parse(`${e.src.expires}T23:59:59Z`) / 1000;
291
+ if (now > cutoff) candidates.push({ url: e.url, ts: cutoff });
292
+ }
293
+ }
294
+ // Corrupt evidence anywhere in the candidate set poisons the verdict:
295
+ // a single oldest-entry probe would let any past entry mask a
296
+ // future-dated one.
297
+ const future = candidates.find((c) => c.ts > now);
298
+ const oldest = candidates.sort((a, b) => a.ts - b.ts)[0];
299
+ if (future) {
300
+ guardFailed = true;
301
+ out.push(
302
+ `Staleness guard: future-dated timestamp on ${displayUrl(future.url)}; the cache's age cannot be trusted.`,
303
+ );
304
+ } else if (!oldest) {
305
+ out.push('Staleness guard: no refreshable entries; nothing to age.');
306
+ } else {
307
+ const ageDays = Math.floor((now - oldest.ts) / DAY);
308
+ if (now - oldest.ts > limit * DAY) {
309
+ guardFailed = true;
310
+ out.push(
311
+ `Staleness guard: oldest refreshable entry is ${ageDays}d old, ` +
312
+ `exceeds --max-age ${limit}d:\n ${displayUrl(oldest.url)}\n` +
313
+ 'Is the cache-refresh job running? If the URL has left the ' +
314
+ 'site, its entry never re-checks: prune it (--match URL --prune 1).',
315
+ );
316
+ } else {
317
+ out.push(
318
+ `Staleness guard: oldest refreshable entry is ${ageDays}d old ` +
319
+ `(within --max-age ${limit}d).`,
320
+ );
321
+ }
322
+ }
323
+ }
254
324
  } else if (op.kind === 'summary') {
255
325
  const stats = computeStats(current(), {
256
326
  now,
@@ -278,7 +348,7 @@ export function runOps(
278
348
  writeText = parsed.lines.filter((_, i) => !removed.has(i)).join('\n');
279
349
  }
280
350
  }
281
- return { output: out.join('\n\n'), writeText, pruned };
351
+ return { output: out.join('\n\n'), writeText, pruned, guardFailed };
282
352
  }
283
353
 
284
354
  // --- CLI -------------------------------------------------------------------
@@ -290,6 +360,7 @@ const FLAGS = new Map([
290
360
  ['--match', 'match'],
291
361
  ['-p', 'prune'],
292
362
  ['--prune', 'prune'],
363
+ ['--max-age', 'max-age'],
293
364
  ['-s', 'summary'],
294
365
  ['--summary', 'summary'],
295
366
  ]);
@@ -331,6 +402,9 @@ export function parseArgs(argv) {
331
402
  if (kind === 'prune' && !/^\d+%?$/.test(value)) {
332
403
  throw new Error(`--prune needs NUM or NUM%, got: ${value}`);
333
404
  }
405
+ if (kind === 'max-age' && !/^\d+$/.test(value)) {
406
+ throw new Error(`--max-age needs a day count, got: ${value}`);
407
+ }
334
408
  ops.push({ kind, value });
335
409
  continue;
336
410
  }
@@ -339,6 +413,12 @@ export function parseArgs(argv) {
339
413
  path = a;
340
414
  }
341
415
 
416
+ // Scoping the staleness backstop to a URL subset would silently blind it
417
+ // to unmatched rot; refuse rather than pick a winner.
418
+ if (match && ops.some((op) => op.kind === 'max-age')) {
419
+ throw new Error('--max-age cannot be combined with --match');
420
+ }
421
+
342
422
  return { ops, path, match, help };
343
423
  }
344
424
 
@@ -350,12 +430,21 @@ the evolving cache, so \`-l 5 -p 5\` lists the 5 oldest about to be pruned,
350
430
  while \`-p 5 -l 5\` lists the next 5 after pruning.
351
431
 
352
432
  -l, --list NUM list the NUM oldest entries
353
- -m, --match REGEX scope all operations to URLs matching REGEX
354
- -p, --prune NUM[%] drop the NUM (or NUM%) oldest entries, then rewrite
355
- -s, --summary print a summary (counts, ages, status, via, histogram)
433
+ -m, --match REGEX scope all operations except --max-age to URLs matching
434
+ REGEX (the guard refuses the combination)
435
+ --max-age DAYS staleness guard: fail (exit 3) when the oldest
436
+ refreshable entry is older than DAYS days
437
+ -p, --prune NUM[%] drop the NUM (or NUM%) oldest non-manual entries, then
438
+ rewrite (manual entries retire via their expires date)
439
+ -s, --summary print a summary (counts, ages, result, via, histogram)
356
440
  -h, --help show this help
357
441
 
358
- With no options, prints the summary. A flag may not be repeated.`;
442
+ With no options, prints the summary. A flag may not be repeated. A --match
443
+ scope shields out-of-scope entries from the operations, never from a prune's
444
+ rewrite: they always survive intact.
445
+
446
+ Exit codes: 0 success; 1 unreadable or malformed cache file; 2 usage error;
447
+ 3 staleness-guard breach.`;
359
448
 
360
449
  export function main(argv) {
361
450
  let args;
@@ -392,7 +481,7 @@ export function main(argv) {
392
481
 
393
482
  const now = Date.now() / 1000;
394
483
  const ops = args.ops.length ? args.ops : [{ kind: 'summary' }];
395
- const { output, writeText } = runOps(parsed, ops, {
484
+ const { output, writeText, guardFailed } = runOps(parsed, ops, {
396
485
  now,
397
486
  path,
398
487
  match: args.match,
@@ -400,6 +489,7 @@ export function main(argv) {
400
489
 
401
490
  if (output) console.log(output);
402
491
  if (writeText !== null) writeFileAtomic(path, writeText);
492
+ if (guardFailed) process.exit(3);
403
493
  }
404
494
 
405
495
  // Real-path compare, not `file://${argv[1]}`: npm links bins as symlinks, so
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "link-cache",
3
- "version": "0.4.0",
3
+ "version": "0.5.0",
4
4
  "description": "Zero-dependency helper CLIs for cached Lychee link-checking around an owned JSONC cache (link-cache.jsonc) with provenance: lychee-norm-cache (run + sync caches) and link-cache (inspect/prune).",
5
5
  "keywords": [
6
6
  "lychee",