link-cache 0.3.0 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +104 -18
- package/check/index.mjs +216 -34
- package/lib/cache.mjs +369 -0
- package/lib/write.mjs +12 -0
- package/{refcache → link-cache}/index.mjs +128 -37
- package/link-cache/refcache.mjs +11 -0
- package/package.json +9 -4
package/README.md
CHANGED
|
@@ -4,14 +4,97 @@ Zero-dependency helper CLIs for **cached** link checking with [Lychee][], for
|
|
|
4
4
|
any static site that builds to a `public/` directory (Docsy, Hugo, and others).
|
|
5
5
|
Two tools:
|
|
6
6
|
|
|
7
|
-
- **`lychee-norm-cache`** — run lychee over your built `public/` output,
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
7
|
+
- **`lychee-norm-cache`** — run lychee over your built `public/` output, keeping
|
|
8
|
+
the committed `link-cache.jsonc` cache and lychee's derived `.lycheecache` in
|
|
9
|
+
sync.
|
|
10
|
+
- **`link-cache`** — inspect and prune the cache: list the oldest entries, prune
|
|
11
|
+
a count or percentage (optionally scoped by URL regex; `manual` entries are
|
|
12
|
+
exempt — they retire via their `expires` date), or print a summary (status,
|
|
13
|
+
provenance, ages). (`refcache` is a deprecated alias.)
|
|
14
|
+
|
|
15
|
+
With a committed `lychee.toml` and `link-cache.jsonc`, these give a site a
|
|
16
|
+
self-contained, cached link-checking setup: fast reruns, and diffs that reflect
|
|
17
|
+
real changes — link statuses, and check recency for freshly re-verified entries.
|
|
18
|
+
|
|
19
|
+
## The owned cache: `link-cache.jsonc`
|
|
20
|
+
|
|
21
|
+
The committed source of truth is `link-cache.jsonc` — a JSONC file,
|
|
22
|
+
pretty-printed by construction in Prettier's style (so `prettier --check` passes
|
|
23
|
+
it untouched), one multi-line object per URL, sorted, with `//` comments allowed
|
|
24
|
+
on their own lines. Each comment attaches to the entry below it and survives
|
|
25
|
+
rewrites for as long as the entry's status holds; a status change replaces the
|
|
26
|
+
entry and retires its comments (a stale rationale is worse than none). The
|
|
27
|
+
multi-line shape is deliberate: field-per-line entries keep concurrent updates
|
|
28
|
+
merging cleanly under git's normal 3-way merge.
|
|
29
|
+
|
|
30
|
+
```jsonc
|
|
31
|
+
{
|
|
32
|
+
"https://example.com/": {
|
|
33
|
+
"status": 200,
|
|
34
|
+
"when": "2026-08-29T20:06:38Z",
|
|
35
|
+
"via": "lychee",
|
|
36
|
+
},
|
|
37
|
+
// Seeded pending my-org/repo#123; the target lands with that merge.
|
|
38
|
+
"https://example.com/future-page/": {
|
|
39
|
+
"status": 200,
|
|
40
|
+
"when": "2026-08-29T20:06:38Z",
|
|
41
|
+
"via": "manual",
|
|
42
|
+
"expires": "2026-09-30",
|
|
43
|
+
},
|
|
44
|
+
}
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Each entry records:
|
|
48
|
+
|
|
49
|
+
- **`status`** — generalized: positive values are HTTP statuses; `0` is
|
|
50
|
+
unchecked; negative values are tool-specific errors (`-10` timeout, `-20`
|
|
51
|
+
network/DNS, `-30` certificate, `-40` generic client error).
|
|
52
|
+
- **`when`** — the moment the status was established, as RFC3339 UTC at whole
|
|
53
|
+
seconds (`YYYY-MM-DDTHH:MM:SSZ`), converting exactly to and from lychee's
|
|
54
|
+
epoch-seconds cache timestamps. The form is strict — no fractional seconds, no
|
|
55
|
+
offsets — so timestamps are byte-comparable and lexicographically
|
|
56
|
+
chronological.
|
|
57
|
+
- **`via`** — the resolver that set the status: `lychee`, `manual`
|
|
58
|
+
(hand-seeded), or a named specialized resolver (e.g. a browser-grade probe).
|
|
59
|
+
- **`expires`** (optional, `manual` entries) — `YYYY-MM-DD`. Until then the
|
|
60
|
+
entry is trusted (never re-checked, overriding lychee's `max_cache_age`);
|
|
61
|
+
after that, it's re-checked live and replaced by the verified result.
|
|
62
|
+
|
|
63
|
+
Lychee's own CSV cache, `.lycheecache`, is **derived**: `lychee-norm-cache`
|
|
64
|
+
projects the owned cache into it before each run and folds lychee's results back
|
|
65
|
+
afterwards. Gitignore `.lycheecache`; commit `link-cache.jsonc`. A re-check that
|
|
66
|
+
changes an entry's status replaces the entry (provenance moves to `lychee`); a
|
|
67
|
+
re-confirmation leaves provenance-bearing entries (`manual`, named resolvers)
|
|
68
|
+
untouched, while `lychee`-owned entries refresh their `when` to record recency.
|
|
69
|
+
A URL the run itself reports as failing is recorded as a negative tool-error
|
|
70
|
+
status; an entry that merely goes missing from lychee's CSV is left untouched
|
|
71
|
+
(cache-status excludes, cache aging, and site changes all remove entries from
|
|
72
|
+
healthy runs).
|
|
73
|
+
|
|
74
|
+
Without a `link-cache.jsonc`, `lychee-norm-cache` falls back to the legacy mode:
|
|
75
|
+
normalize the committed `.lycheecache` in place. To migrate:
|
|
76
|
+
|
|
77
|
+
```sh
|
|
78
|
+
npm run check:links -- --migrate # .lycheecache -> link-cache.jsonc
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
then commit `link-cache.jsonc` and gitignore `.lycheecache`. If the CSV cache
|
|
82
|
+
carried a `merge=union` gitattribute, drop it rather than moving it over: union
|
|
83
|
+
merging proved ineffective in practice, and on a multi-line file it can
|
|
84
|
+
interleave entries into invalid JSON. The owned cache merges with git's normal
|
|
85
|
+
3-way merge; on a conflict, resolve either way and rerun the check — the next
|
|
86
|
+
run re-normalizes the file.
|
|
87
|
+
|
|
88
|
+
## Exit codes (`lychee-norm-cache`)
|
|
89
|
+
|
|
90
|
+
- `0` — success.
|
|
91
|
+
- `1` — dead links: the check ran and found failures.
|
|
92
|
+
- `2` — preflight or sanity failure: lychee or `public/` missing, lychee config
|
|
93
|
+
error, or **zero links checked** (an empty or fully-excluded `public/` is a
|
|
94
|
+
false-clean, not a pass).
|
|
11
95
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
change when links actually change.
|
|
96
|
+
Warn-style wrappers can soften exit 1 (advisory link rot) while still failing
|
|
97
|
+
hard on exit 2 (the check didn't actually run).
|
|
15
98
|
|
|
16
99
|
## Requirements
|
|
17
100
|
|
|
@@ -31,7 +114,7 @@ npm install --save-dev link-cache
|
|
|
31
114
|
Or, to install from GitHub rather than the npm registry:
|
|
32
115
|
|
|
33
116
|
```sh
|
|
34
|
-
npm install --save-dev github:chalin/link-cache#semver:^0.
|
|
117
|
+
npm install --save-dev github:chalin/link-cache#semver:^0.4.0
|
|
35
118
|
```
|
|
36
119
|
|
|
37
120
|
This puts both bins on your project's `PATH`.
|
|
@@ -44,22 +127,23 @@ Wire the bins into your `package.json` scripts (bare names — `npm run` puts
|
|
|
44
127
|
```json
|
|
45
128
|
"scripts": {
|
|
46
129
|
"check:links": "lychee-norm-cache",
|
|
47
|
-
"
|
|
130
|
+
"link-cache": "link-cache"
|
|
48
131
|
}
|
|
49
132
|
```
|
|
50
133
|
|
|
51
134
|
```sh
|
|
52
|
-
npm run check:links
|
|
53
|
-
npm run
|
|
135
|
+
npm run check:links # sync caches, run lychee, fold results back
|
|
136
|
+
npm run link-cache -- --summary # cache stats (count, oldest, status, via, ages)
|
|
137
|
+
npm run link-cache -- --match 'github\.com' --prune 10 # trim 10 oldest matching
|
|
54
138
|
```
|
|
55
139
|
|
|
56
140
|
> [!WARNING]
|
|
57
141
|
>
|
|
58
|
-
> Don't invoke these bins via `npx`:
|
|
59
|
-
>
|
|
60
|
-
>
|
|
61
|
-
>
|
|
62
|
-
>
|
|
142
|
+
> Don't invoke these bins via `npx`: on a stale or missing `node_modules`, `npx`
|
|
143
|
+
> falls back to the public registry and runs **whatever package holds the bin's
|
|
144
|
+
> name there** (the `lychee-norm-cache` name is squatted). Bare bin names in
|
|
145
|
+
> `npm run` scripts resolve locally or fail loudly — they never touch the
|
|
146
|
+
> registry.
|
|
63
147
|
|
|
64
148
|
`lychee-norm-cache` runs in the current directory (your site root) and forwards
|
|
65
149
|
any extra arguments to lychee. Run either tool with `--help` for its full
|
|
@@ -69,10 +153,12 @@ forwards (e.g. `--offline`, `--max-cache-age 0`).
|
|
|
69
153
|
## Development
|
|
70
154
|
|
|
71
155
|
The published CLIs have **zero runtime dependencies**; Prettier is the only dev
|
|
72
|
-
dependency.
|
|
156
|
+
dependency. The committed `.npmrc` applies the usual supply-chain controls
|
|
157
|
+
(lock-exact installs, script execution default-deny, release cooldown). Run the
|
|
158
|
+
checks (format + tests) with:
|
|
73
159
|
|
|
74
160
|
```sh
|
|
75
|
-
npm install
|
|
161
|
+
npm run install:safe
|
|
76
162
|
npm run check
|
|
77
163
|
```
|
|
78
164
|
|
package/check/index.mjs
CHANGED
|
@@ -1,32 +1,58 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
// The `lychee-norm-cache` bin: run lychee over a built site
|
|
3
|
-
//
|
|
4
|
-
//
|
|
2
|
+
// The `lychee-norm-cache` bin: run lychee over a built site, maintained around
|
|
3
|
+
// an owned JSONC cache (link-cache.jsonc, the committed source of truth) with
|
|
4
|
+
// lychee's CSV .lycheecache derived from it. Falls back to legacy CSV-only
|
|
5
|
+
// normalization when no owned cache exists. Run with `--help` for usage.
|
|
5
6
|
//
|
|
6
7
|
// When GITHUB_TOKEN is unset, a token is bridged from the gh CLI — lychee reads
|
|
7
8
|
// GITHUB_TOKEN, which also lifts the github.com rate limit; CI sets it directly.
|
|
9
|
+
//
|
|
10
|
+
// Exit codes: 0 success; 1 dead links; 2 preflight or sanity failure (lychee
|
|
11
|
+
// or public/ missing, zero links checked, lychee config error).
|
|
8
12
|
|
|
9
13
|
import { spawnSync } from 'node:child_process';
|
|
10
|
-
import {
|
|
11
|
-
existsSync,
|
|
12
|
-
readFileSync,
|
|
13
|
-
realpathSync,
|
|
14
|
-
statSync,
|
|
15
|
-
writeFileSync,
|
|
16
|
-
} from 'node:fs';
|
|
14
|
+
import { existsSync, readFileSync, realpathSync, statSync } from 'node:fs';
|
|
17
15
|
import path from 'node:path';
|
|
18
16
|
import { fileURLToPath } from 'node:url';
|
|
19
17
|
|
|
18
|
+
import {
|
|
19
|
+
CSV_FILE,
|
|
20
|
+
OWNED_FILE,
|
|
21
|
+
mergeBack,
|
|
22
|
+
migrateCsvText,
|
|
23
|
+
parseCsv,
|
|
24
|
+
parseOwned,
|
|
25
|
+
projectToCsv,
|
|
26
|
+
serializeCsv,
|
|
27
|
+
serializeOwned,
|
|
28
|
+
} from '../lib/cache.mjs';
|
|
29
|
+
import { writeFileAtomic } from '../lib/write.mjs';
|
|
30
|
+
|
|
20
31
|
const INSTALL_HINT = 'https://github.com/lycheeverse/lychee#installation';
|
|
21
32
|
|
|
22
|
-
const
|
|
33
|
+
export const EXIT_OK = 0;
|
|
34
|
+
export const EXIT_DEAD_LINKS = 1;
|
|
35
|
+
export const EXIT_PREFLIGHT = 2;
|
|
36
|
+
|
|
37
|
+
const USAGE = `Usage: lychee-norm-cache [--migrate] [lychee args...]
|
|
23
38
|
|
|
24
|
-
Run lychee over this site's built ./public output
|
|
25
|
-
|
|
26
|
-
|
|
39
|
+
Run lychee over this site's built ./public output. With a committed
|
|
40
|
+
${OWNED_FILE}, the ${CSV_FILE} handed to lychee is derived from it before the
|
|
41
|
+
run and folded back into it afterwards; otherwise ${CSV_FILE} is normalized in
|
|
42
|
+
place (legacy mode). Bridges a GitHub token from the gh CLI when GITHUB_TOKEN
|
|
43
|
+
isn't set; extra arguments pass through to lychee.
|
|
27
44
|
|
|
45
|
+
--migrate convert an existing ${CSV_FILE} to ${OWNED_FILE} and exit
|
|
28
46
|
-h, --help show this help
|
|
29
47
|
|
|
48
|
+
Exit codes: 0 success; 1 dead links; 2 preflight/sanity failure (missing
|
|
49
|
+
lychee or public/, lychee config or usage errors, zero links verified).
|
|
50
|
+
|
|
51
|
+
Lychee's summary must reach stdout in its default or JSON format — it is the
|
|
52
|
+
wrapper's proof that a check completed — so flags that divert or reshape
|
|
53
|
+
stdout (--output, --format junit, --dump-inputs, ...) are unsupported here;
|
|
54
|
+
run lychee directly for those.
|
|
55
|
+
|
|
30
56
|
Requires the lychee binary on your PATH and a lychee.toml at the site root.
|
|
31
57
|
Run \`lychee --help\` for all link-checking options.`;
|
|
32
58
|
|
|
@@ -66,6 +92,61 @@ export function publicDirOf(cwd) {
|
|
|
66
92
|
return publicDir;
|
|
67
93
|
}
|
|
68
94
|
|
|
95
|
+
// Check counts from lychee's stdout summary — the human line ("🔍 2 Total …
|
|
96
|
+
// ✅ 1 OK 🚫 0 Errors …") or --format json fields; null when neither is
|
|
97
|
+
// present. A parsed summary is the proof that a check actually completed.
|
|
98
|
+
export function parseSummary(stdout) {
|
|
99
|
+
const m = /(\d+)\s+Total\b[\s\S]*?(\d+)\s+OK\b[\s\S]*?(\d+)\s+Errors?\b/.exec(
|
|
100
|
+
stdout,
|
|
101
|
+
);
|
|
102
|
+
if (m) return { total: +m[1], ok: +m[2], errors: +m[3] };
|
|
103
|
+
const total = /"total"\s*:\s*(\d+)/.exec(stdout);
|
|
104
|
+
const ok = /"successful"\s*:\s*(\d+)/.exec(stdout);
|
|
105
|
+
const errors = /"errors"\s*:\s*(\d+)/.exec(stdout);
|
|
106
|
+
return total && ok && errors
|
|
107
|
+
? { total: +total[1], ok: +ok[1], errors: +errors[1] }
|
|
108
|
+
: null;
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
// Map lychee's exit code to ours (0 ok, 1 dead links, 2 preflight). A parsed
|
|
112
|
+
// summary is required in every case: lychee exits 2 both for broken links and
|
|
113
|
+
// for argument errors, and exits 0 from non-check modes (--dump-inputs) and
|
|
114
|
+
// diverted-output runs (--output FILE, --format junit/detailed) — without the
|
|
115
|
+
// summary there is no proof a check completed.
|
|
116
|
+
export function mapLycheeExit(code, summary) {
|
|
117
|
+
if (!summary) return EXIT_PREFLIGHT;
|
|
118
|
+
if (code === 0) return EXIT_OK;
|
|
119
|
+
if (code === 2) return EXIT_DEAD_LINKS;
|
|
120
|
+
return EXIT_PREFLIGHT;
|
|
121
|
+
}
|
|
122
|
+
|
|
123
|
+
// URLs the run itself reported as failing — the human "[ERROR] URL …" lines,
|
|
124
|
+
// or --format json's fail_map. Positive per-URL evidence for the merge-back's
|
|
125
|
+
// failure recording; CSV absence alone proves nothing (cache_exclude_status,
|
|
126
|
+
// max_cache_age, and site changes all remove entries from healthy runs).
|
|
127
|
+
export function parseFailedUrls(stdout) {
|
|
128
|
+
const failed = new Set();
|
|
129
|
+
const trimmed = stdout.trim();
|
|
130
|
+
if (trimmed.startsWith('{')) {
|
|
131
|
+
try {
|
|
132
|
+
const failMap = JSON.parse(trimmed).fail_map ?? {};
|
|
133
|
+
for (const failures of Object.values(failMap)) {
|
|
134
|
+
for (const f of failures) {
|
|
135
|
+
const url = typeof f.url === 'string' ? f.url : f.url?.url;
|
|
136
|
+
if (url) failed.add(url);
|
|
137
|
+
}
|
|
138
|
+
}
|
|
139
|
+
return failed;
|
|
140
|
+
} catch {
|
|
141
|
+
// fall through to the line scan
|
|
142
|
+
}
|
|
143
|
+
}
|
|
144
|
+
for (const m of stdout.matchAll(/^\s*\[ERROR\]\s+(\S+)/gm)) {
|
|
145
|
+
failed.add(m[1]);
|
|
146
|
+
}
|
|
147
|
+
return failed;
|
|
148
|
+
}
|
|
149
|
+
|
|
69
150
|
// --- lychee invocation -----------------------------------------------------
|
|
70
151
|
|
|
71
152
|
function ghAuthToken() {
|
|
@@ -77,39 +158,133 @@ function hasLychee() {
|
|
|
77
158
|
return !spawnSync('lychee', ['--version'], { stdio: 'ignore' }).error;
|
|
78
159
|
}
|
|
79
160
|
|
|
161
|
+
function fail(message) {
|
|
162
|
+
process.stderr.write(`[help] ${message}\n`);
|
|
163
|
+
return EXIT_PREFLIGHT;
|
|
164
|
+
}
|
|
165
|
+
|
|
166
|
+
function migrate(cwd) {
|
|
167
|
+
const csvPath = path.join(cwd, CSV_FILE);
|
|
168
|
+
const ownedPath = path.join(cwd, OWNED_FILE);
|
|
169
|
+
if (existsSync(ownedPath)) return fail(`${OWNED_FILE} already exists.`);
|
|
170
|
+
if (!existsSync(csvPath)) return fail(`${CSV_FILE} not found.`);
|
|
171
|
+
const { text, count, malformed } = migrateCsvText(
|
|
172
|
+
readFileSync(csvPath, 'utf8'),
|
|
173
|
+
);
|
|
174
|
+
// Migration is specified lossless: refuse rather than silently drop.
|
|
175
|
+
if (malformed) {
|
|
176
|
+
return fail(
|
|
177
|
+
`${CSV_FILE} has ${malformed} malformed line(s); fix or remove them, then rerun.`,
|
|
178
|
+
);
|
|
179
|
+
}
|
|
180
|
+
writeFileAtomic(ownedPath, text);
|
|
181
|
+
console.log(
|
|
182
|
+
`Migrated ${count} entries to ${OWNED_FILE}. Commit it and gitignore ${CSV_FILE}.`,
|
|
183
|
+
);
|
|
184
|
+
return EXIT_OK;
|
|
185
|
+
}
|
|
186
|
+
|
|
80
187
|
function main(argv) {
|
|
81
188
|
if (argv.includes('-h') || argv.includes('--help')) {
|
|
82
189
|
console.log(USAGE);
|
|
83
|
-
return
|
|
190
|
+
return EXIT_OK;
|
|
84
191
|
}
|
|
85
192
|
|
|
193
|
+
const cwd = process.cwd();
|
|
194
|
+
if (argv.includes('--migrate')) return migrate(cwd);
|
|
195
|
+
|
|
86
196
|
if (!hasLychee()) {
|
|
87
|
-
|
|
88
|
-
return 1;
|
|
197
|
+
return fail(`lychee not found. Install: ${INSTALL_HINT}`);
|
|
89
198
|
}
|
|
90
199
|
|
|
91
|
-
const cwd = process.cwd();
|
|
92
200
|
const publicDir = publicDirOf(cwd);
|
|
93
201
|
if (!publicDir) {
|
|
94
|
-
|
|
95
|
-
|
|
202
|
+
return fail(`${path.join(cwd, 'public')} not found. Build the site first.`);
|
|
203
|
+
}
|
|
204
|
+
|
|
205
|
+
const ownedPath = path.join(cwd, OWNED_FILE);
|
|
206
|
+
const cachePath = path.join(cwd, CSV_FILE);
|
|
207
|
+
const owned = existsSync(ownedPath)
|
|
208
|
+
? parseOwned(readFileSync(ownedPath, 'utf8'))
|
|
209
|
+
: null;
|
|
210
|
+
const now = Date.now() / 1000;
|
|
211
|
+
|
|
212
|
+
// The owned-cache lens requires lychee's cache; enforce rather than let a
|
|
213
|
+
// cacheless run erase every projected entry on merge-back.
|
|
214
|
+
const hasCacheFlag = argv.some(
|
|
215
|
+
(a) => a === '--cache' || a.startsWith('--cache='),
|
|
216
|
+
);
|
|
217
|
+
if (owned && argv.includes('--cache=false')) {
|
|
218
|
+
return fail(`--cache=false is incompatible with ${OWNED_FILE}.`);
|
|
219
|
+
}
|
|
220
|
+
const cacheArgs = hasCacheFlag ? [] : ['--cache'];
|
|
221
|
+
|
|
222
|
+
// Derive the CSV lychee will read from the owned cache.
|
|
223
|
+
if (owned) {
|
|
224
|
+
writeFileAtomic(
|
|
225
|
+
cachePath,
|
|
226
|
+
serializeCsv(projectToCsv(owned.entries, { now })),
|
|
96
227
|
);
|
|
97
|
-
return 1;
|
|
98
228
|
}
|
|
99
229
|
|
|
100
230
|
const token = resolveToken();
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
231
|
+
// stdout is captured for the completion/zero-check sanity checks and echoed
|
|
232
|
+
// afterwards; stderr (progress) still streams. The generous buffer bound
|
|
233
|
+
// keeps spawnSync simple; overruns surface via run.error below.
|
|
234
|
+
const run = spawnSync(
|
|
235
|
+
'lychee',
|
|
236
|
+
[
|
|
237
|
+
'--config',
|
|
238
|
+
'lychee.toml',
|
|
239
|
+
...cacheArgs,
|
|
240
|
+
'--root-dir',
|
|
241
|
+
publicDir,
|
|
242
|
+
...argv,
|
|
243
|
+
publicDir,
|
|
244
|
+
],
|
|
245
|
+
{
|
|
246
|
+
stdio: ['inherit', 'pipe', 'inherit'],
|
|
247
|
+
encoding: 'utf8',
|
|
248
|
+
maxBuffer: 64 * 1024 * 1024,
|
|
249
|
+
env: { ...process.env, GITHUB_TOKEN: token },
|
|
250
|
+
},
|
|
251
|
+
);
|
|
252
|
+
if (run.stdout) process.stdout.write(run.stdout);
|
|
253
|
+
if (run.error) {
|
|
254
|
+
return fail(`lychee failed to run: ${run.error.message}`);
|
|
255
|
+
}
|
|
256
|
+
const summary = parseSummary(run.stdout ?? '');
|
|
257
|
+
const status = mapLycheeExit(run.status ?? 1, summary);
|
|
258
|
+
|
|
259
|
+
// On a preflight failure lychee produced no trustworthy results: leave both
|
|
260
|
+
// caches untouched (folding would mislabel unprojected entries as failed).
|
|
261
|
+
if (status === EXIT_PREFLIGHT) {
|
|
262
|
+
return fail('lychee did not complete a check; caches left untouched.');
|
|
263
|
+
}
|
|
264
|
+
|
|
265
|
+
// A clean run in which nothing got a verdict is a false-clean (empty or
|
|
266
|
+
// fully-excluded public/), not a success — and folding it would mislabel
|
|
267
|
+
// unattempted entries as failed, so the check precedes the fold. Cache hits
|
|
268
|
+
// count toward OK in lychee's summary, so a fully-cached run passes.
|
|
269
|
+
if (status === EXIT_OK && summary && summary.ok + summary.errors === 0) {
|
|
270
|
+
return fail('lychee verified 0 links: empty or fully-excluded public/?');
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
// Fold the post-run CSV back into the owned cache (also on dead links, so a
|
|
274
|
+
// completed run with failures still leaves a stable, truthful cache), or
|
|
275
|
+
// normalize the CSV in place in legacy mode.
|
|
276
|
+
if (owned) {
|
|
277
|
+
const csvEntries = existsSync(cachePath)
|
|
278
|
+
? parseCsv(readFileSync(cachePath, 'utf8')).entries
|
|
279
|
+
: [];
|
|
280
|
+
const failedUrls = parseFailedUrls(run.stdout ?? '');
|
|
281
|
+
writeFileAtomic(
|
|
282
|
+
ownedPath,
|
|
283
|
+
serializeOwned(mergeBack(owned, csvEntries, { now, failedUrls })),
|
|
284
|
+
);
|
|
285
|
+
writeFileAtomic(cachePath, sortCacheText(readFileSync(cachePath, 'utf8')));
|
|
286
|
+
} else if (existsSync(cachePath)) {
|
|
287
|
+
writeFileAtomic(cachePath, sortCacheText(readFileSync(cachePath, 'utf8')));
|
|
113
288
|
}
|
|
114
289
|
|
|
115
290
|
return status;
|
|
@@ -126,5 +301,12 @@ function isEntryPoint() {
|
|
|
126
301
|
}
|
|
127
302
|
|
|
128
303
|
if (isEntryPoint()) {
|
|
129
|
-
|
|
304
|
+
// Wrapper exceptions (e.g. a malformed owned cache) are preflight failures,
|
|
305
|
+
// not dead links: warn wrappers must not swallow them.
|
|
306
|
+
try {
|
|
307
|
+
process.exit(main(process.argv.slice(2)));
|
|
308
|
+
} catch (err) {
|
|
309
|
+
process.stderr.write(`[error] ${err.message}\n`);
|
|
310
|
+
process.exit(EXIT_PREFLIGHT);
|
|
311
|
+
}
|
|
130
312
|
}
|
package/lib/cache.mjs
ADDED
|
@@ -0,0 +1,369 @@
|
|
|
1
|
+
// Shared cache-file model: the owned JSONC cache (link-cache.jsonc, source of
|
|
2
|
+
// truth) and its derived Lychee CSV (.lycheecache). Pure functions only — no
|
|
3
|
+
// I/O — so both bins and the tests share one implementation.
|
|
4
|
+
//
|
|
5
|
+
// Owned-file format: JSONC, pretty-printed by construction in Prettier's jsonc
|
|
6
|
+
// style (2-space indent, one field per line, trailing commas), so the file
|
|
7
|
+
// passes `prettier --check` untouched. Every entry is multi-line — the field
|
|
8
|
+
// separation is what keeps concurrent edits merging cleanly — and whole-line
|
|
9
|
+
// `//` comments attach to the entry that follows them. Entries are sorted by
|
|
10
|
+
// raw URL byte order (matching LC_ALL=C). Stripping comments and trailing
|
|
11
|
+
// commas yields valid JSON.
|
|
12
|
+
//
|
|
13
|
+
// Entry schema: { "status": INT, "when": RFC3339_UTC, "via": "RESOLVER" }
|
|
14
|
+
// with an optional "expires": "YYYY-MM-DD" (manual entries). Status
|
|
15
|
+
// generalizes HTTP: positive = HTTP status, 0 = unchecked, negative =
|
|
16
|
+
// tool-specific error. `when` is the canonical whole-second UTC form
|
|
17
|
+
// (YYYY-MM-DDTHH:MM:SSZ) — no fractional seconds, no offsets — validated
|
|
18
|
+
// strictly: this tool is the file's only writer, so a non-canonical value is
|
|
19
|
+
// a bug to surface, not smooth over (lesson from the htmltest-era
|
|
20
|
+
// RFC3339Nano/offset churn).
|
|
21
|
+
|
|
22
|
+
export const OWNED_FILE = 'link-cache.jsonc';
|
|
23
|
+
export const CSV_FILE = '.lycheecache';
|
|
24
|
+
|
|
25
|
+
// Tool-specific error statuses (the htmltest-fork convention).
|
|
26
|
+
export const STATUS_TIMEOUT = -10;
|
|
27
|
+
export const STATUS_NETWORK_ERROR = -20;
|
|
28
|
+
export const STATUS_CERT_ERROR = -30;
|
|
29
|
+
export const STATUS_CLIENT_ERROR = -40;
|
|
30
|
+
|
|
31
|
+
const WHEN_RE = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/;
|
|
32
|
+
|
|
33
|
+
// Epoch seconds -> canonical `when`.
|
|
34
|
+
export function tsToWhen(ts) {
|
|
35
|
+
return new Date(ts * 1000).toISOString().replace(/\.\d{3}Z$/, 'Z');
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
// Canonical `when` -> epoch seconds; NaN when non-canonical (strict: the
|
|
39
|
+
// regex gates shape, and the round-trip equality check rejects impossible
|
|
40
|
+
// dates that Date.parse would silently normalize, e.g. Feb 30 -> Mar 2).
|
|
41
|
+
export function whenToTs(when) {
|
|
42
|
+
if (!WHEN_RE.test(when)) return NaN;
|
|
43
|
+
const ts = Date.parse(when) / 1000;
|
|
44
|
+
return tsToWhen(ts) === when ? ts : NaN;
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
const byUrlBytes = (a, b) =>
|
|
48
|
+
Buffer.compare(Buffer.from(a.url), Buffer.from(b.url));
|
|
49
|
+
|
|
50
|
+
// --- CSV (.lycheecache) ------------------------------------------------------
|
|
51
|
+
|
|
52
|
+
// Unquote a CSV field: lychee quotes URLs that contain a comma.
|
|
53
|
+
export function csvUnquote(field) {
|
|
54
|
+
return field.startsWith('"') && field.endsWith('"')
|
|
55
|
+
? field.slice(1, -1).replace(/""/g, '"')
|
|
56
|
+
: field;
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
export function csvQuote(url) {
|
|
60
|
+
return /[",]/.test(url) ? `"${url.replace(/"/g, '""')}"` : url;
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
// Parse one URL,STATUS,TIMESTAMP line; null when malformed. Lexically strict:
|
|
64
|
+
// STATUS and TIMESTAMP must be non-empty digit runs (`Number('')` is 0, which
|
|
65
|
+
// would silently coin status-0/epoch-0 entries from truncated lines).
|
|
66
|
+
export function parseCsvLine(raw) {
|
|
67
|
+
const lastComma = raw.lastIndexOf(',');
|
|
68
|
+
if (lastComma < 0) return null;
|
|
69
|
+
const tsField = raw.slice(lastComma + 1);
|
|
70
|
+
const head = raw.slice(0, lastComma);
|
|
71
|
+
const statusComma = head.lastIndexOf(',');
|
|
72
|
+
if (statusComma < 0) return null;
|
|
73
|
+
const statusField = head.slice(statusComma + 1).trim();
|
|
74
|
+
if (!/^\d+$/.test(tsField) || !/^-?\d+$/.test(statusField)) return null;
|
|
75
|
+
const url = csvUnquote(head.slice(0, statusComma));
|
|
76
|
+
if (url === '') return null;
|
|
77
|
+
return { url, status: Number(statusField), ts: Number(tsField) };
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
// Parse a whole CSV cache; malformed lines are counted, duplicates keep the
|
|
81
|
+
// newest timestamp.
|
|
82
|
+
export function parseCsv(text) {
|
|
83
|
+
const byUrl = new Map();
|
|
84
|
+
let malformed = 0;
|
|
85
|
+
for (const raw of text.split('\n')) {
|
|
86
|
+
if (raw.trim() === '') continue;
|
|
87
|
+
const entry = parseCsvLine(raw);
|
|
88
|
+
if (!entry) {
|
|
89
|
+
malformed += 1;
|
|
90
|
+
continue;
|
|
91
|
+
}
|
|
92
|
+
const prior = byUrl.get(entry.url);
|
|
93
|
+
if (!prior || entry.ts >= prior.ts) byUrl.set(entry.url, entry);
|
|
94
|
+
}
|
|
95
|
+
return { entries: [...byUrl.values()].sort(byUrlBytes), malformed };
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
export function serializeCsv(entries) {
|
|
99
|
+
if (entries.length === 0) return '';
|
|
100
|
+
return (
|
|
101
|
+
[...entries]
|
|
102
|
+
.sort(byUrlBytes)
|
|
103
|
+
.map((e) => `${csvQuote(e.url)},${e.status},${e.ts}`)
|
|
104
|
+
.join('\n') + '\n'
|
|
105
|
+
);
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
// --- owned JSONC (link-cache.jsonc) -----------------------------------------
|
|
109
|
+
|
|
110
|
+
const COMMENT_RE = /^\s*\/\//;
|
|
111
|
+
// The line that opens an entry: `"URL": {` (the key is a JSON string, which
|
|
112
|
+
// cannot span lines).
|
|
113
|
+
const ENTRY_OPEN_RE = /^\s*("(?:[^"\\]|\\.)*")\s*:\s*\{/;
|
|
114
|
+
|
|
115
|
+
// Strip a line-ending comma when the next non-blank line closes an object —
|
|
116
|
+
// per-line, so commas inside URL strings are never touched (JSON strings
|
|
117
|
+
// cannot span lines).
|
|
118
|
+
function stripTrailingCommas(lines) {
|
|
119
|
+
const out = [...lines];
|
|
120
|
+
for (let i = 0; i < out.length; i++) {
|
|
121
|
+
if (!out[i].trimEnd().endsWith(',')) continue;
|
|
122
|
+
let j = i + 1;
|
|
123
|
+
while (j < out.length && out[j].trim() === '') j++;
|
|
124
|
+
if (j < out.length && out[j].trim().startsWith('}')) {
|
|
125
|
+
out[i] = out[i].trimEnd().slice(0, -1);
|
|
126
|
+
}
|
|
127
|
+
}
|
|
128
|
+
return out;
|
|
129
|
+
}
|
|
130
|
+
|
|
131
|
+
const ENTRY_FIELDS = new Set(['status', 'when', 'via', 'expires']);
|
|
132
|
+
|
|
133
|
+
// A calendar-real YYYY-MM-DD (round-trip check rejects e.g. Feb 30).
|
|
134
|
+
function isCanonicalDate(s) {
|
|
135
|
+
if (!/^\d{4}-\d{2}-\d{2}$/.test(s)) return false;
|
|
136
|
+
const t = Date.parse(`${s}T00:00:00Z`);
|
|
137
|
+
return Number.isFinite(t) && new Date(t).toISOString().slice(0, 10) === s;
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
// Rejects unknown fields (a rewrite would silently drop them) and restricts
|
|
141
|
+
// `expires` to manual entries — resolvers set statuses, owners set trust
|
|
142
|
+
// windows. Relax the via restriction if a named resolver ever needs expiry.
|
|
143
|
+
// Statuses live in one of three domains: HTTP (100-999, the only ones that
|
|
144
|
+
// project into Lychee's CSV — Lychee rejects codes outside that range at
|
|
145
|
+
// load, which would wedge the run), 0 (unchecked), negative (tool errors).
|
|
146
|
+
function isValidStatus(s) {
|
|
147
|
+
return Number.isInteger(s) && (s < 0 || s === 0 || (s >= 100 && s <= 999));
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
function validateEntry(url, v) {
|
|
151
|
+
return (
|
|
152
|
+
v !== null &&
|
|
153
|
+
typeof v === 'object' &&
|
|
154
|
+
Object.keys(v).every((k) => ENTRY_FIELDS.has(k)) &&
|
|
155
|
+
isValidStatus(v.status) &&
|
|
156
|
+
typeof v.when === 'string' &&
|
|
157
|
+
Number.isFinite(whenToTs(v.when)) &&
|
|
158
|
+
whenToTs(v.when) >= 0 && // pre-epoch would project a negative CSV ts
|
|
159
|
+
typeof v.via === 'string' &&
|
|
160
|
+
(v.expires === undefined ||
|
|
161
|
+
(v.via === 'manual' && isCanonicalDate(v.expires)))
|
|
162
|
+
);
|
|
163
|
+
}
|
|
164
|
+
|
|
165
|
+
// Parse the owned cache. Throws on malformed content (the file is committed
|
|
166
|
+
// and tool-written; silent drops would lose data on the next rewrite),
|
|
167
|
+
// including blank input — a zero-byte file is a truncated write, not an empty
|
|
168
|
+
// cache (that one is `{}`). Comments are collected line-wise and attached to
|
|
169
|
+
// the entry whose opening `"URL": {` line follows them; comments after the
|
|
170
|
+
// last entry are kept as `trailing`. Duplicate URL keys are rejected: with
|
|
171
|
+
// merge=union gone, a duplicate is a bad merge resolution to surface, not a
|
|
172
|
+
// leftover to paper over.
|
|
173
|
+
export function parseOwned(text) {
|
|
174
|
+
if (text.trim() === '') {
|
|
175
|
+
throw new Error(`${OWNED_FILE}: blank file (truncated write?)`);
|
|
176
|
+
}
|
|
177
|
+
const lines = text.split('\n');
|
|
178
|
+
|
|
179
|
+
// Comment attachment pass; also detects duplicate keys line-wise (the JSON
|
|
180
|
+
// pass below cannot: Object.entries already collapsed them).
|
|
181
|
+
const commentsByUrl = new Map();
|
|
182
|
+
const seenUrls = new Set();
|
|
183
|
+
const trailing = [];
|
|
184
|
+
let pending = [];
|
|
185
|
+
for (const raw of lines) {
|
|
186
|
+
if (COMMENT_RE.test(raw)) {
|
|
187
|
+
pending.push(` ${raw.trim()}`);
|
|
188
|
+
continue;
|
|
189
|
+
}
|
|
190
|
+
const open = ENTRY_OPEN_RE.exec(raw);
|
|
191
|
+
if (open) {
|
|
192
|
+
const url = JSON.parse(open[1]);
|
|
193
|
+
if (seenUrls.has(url)) {
|
|
194
|
+
throw new Error(`${OWNED_FILE}: duplicate entry for ${url}`);
|
|
195
|
+
}
|
|
196
|
+
seenUrls.add(url);
|
|
197
|
+
if (pending.length) {
|
|
198
|
+
commentsByUrl.set(url, pending);
|
|
199
|
+
pending = [];
|
|
200
|
+
}
|
|
201
|
+
}
|
|
202
|
+
}
|
|
203
|
+
trailing.push(...pending);
|
|
204
|
+
|
|
205
|
+
// Data pass: strip comments and trailing commas, then parse as JSON.
|
|
206
|
+
const dataLines = stripTrailingCommas(
|
|
207
|
+
lines.filter((l) => !COMMENT_RE.test(l)),
|
|
208
|
+
);
|
|
209
|
+
let obj;
|
|
210
|
+
try {
|
|
211
|
+
obj = JSON.parse(dataLines.join('\n') || '{}');
|
|
212
|
+
} catch (err) {
|
|
213
|
+
throw new Error(`${OWNED_FILE}: invalid JSONC: ${err.message}`);
|
|
214
|
+
}
|
|
215
|
+
if (obj === null || typeof obj !== 'object' || Array.isArray(obj)) {
|
|
216
|
+
throw new Error(`${OWNED_FILE}: top level must be an object`);
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
const entries = [];
|
|
220
|
+
for (const [url, v] of Object.entries(obj)) {
|
|
221
|
+
if (!validateEntry(url, v)) {
|
|
222
|
+
throw new Error(`${OWNED_FILE}: malformed entry for ${url}`);
|
|
223
|
+
}
|
|
224
|
+
entries.push({
|
|
225
|
+
url,
|
|
226
|
+
status: v.status,
|
|
227
|
+
ts: whenToTs(v.when), // epoch internally; `when` re-derives on write
|
|
228
|
+
via: v.via,
|
|
229
|
+
expires: v.expires,
|
|
230
|
+
comments: commentsByUrl.get(url) ?? [],
|
|
231
|
+
});
|
|
232
|
+
}
|
|
233
|
+
return { entries: entries.sort(byUrlBytes), trailing };
|
|
234
|
+
}
|
|
235
|
+
|
|
236
|
+
// Canonical serialization = Prettier's jsonc style (2-space indent, one field
|
|
237
|
+
// per line, trailing commas, always multi-line), so the output is
|
|
238
|
+
// prettier-idempotent by construction.
|
|
239
|
+
export function serializeEntry({ url, status, ts, via, expires }) {
|
|
240
|
+
const fields = [
|
|
241
|
+
` "status": ${status},`,
|
|
242
|
+
` "when": ${JSON.stringify(tsToWhen(ts))},`,
|
|
243
|
+
` "via": ${JSON.stringify(via)},`,
|
|
244
|
+
];
|
|
245
|
+
if (expires !== undefined) {
|
|
246
|
+
fields.push(` "expires": ${JSON.stringify(expires)},`);
|
|
247
|
+
}
|
|
248
|
+
return [` ${JSON.stringify(url)}: {`, ...fields, ' },'].join('\n');
|
|
249
|
+
}
|
|
250
|
+
|
|
251
|
+
export function serializeOwned({ entries, trailing = [] }) {
|
|
252
|
+
if (entries.length === 0 && trailing.length === 0) return '{}\n';
|
|
253
|
+
const lines = ['{'];
|
|
254
|
+
for (const entry of [...entries].sort(byUrlBytes)) {
|
|
255
|
+
lines.push(...(entry.comments ?? []).map((c) => ` ${c.trim()}`));
|
|
256
|
+
lines.push(serializeEntry(entry));
|
|
257
|
+
}
|
|
258
|
+
lines.push(...trailing.map((c) => ` ${c.trim()}`), '}');
|
|
259
|
+
return lines.join('\n') + '\n';
|
|
260
|
+
}
|
|
261
|
+
|
|
262
|
+
// --- lens: owned -> CSV projection ------------------------------------------
|
|
263
|
+
|
|
264
|
+
const dateToTs = (isoDate) => Date.parse(`${isoDate}T23:59:59Z`) / 1000;
|
|
265
|
+
|
|
266
|
+
// Project the owned entries into Lychee's CSV. Only positive (HTTP) statuses
|
|
267
|
+
// project — Lychee never persists errors, and would re-drop the rest. An
|
|
268
|
+
// unexpired entry with `expires` projects with a fresh timestamp so it
|
|
269
|
+
// outlives Lychee's max_cache_age until its own expiry; once expired it is
|
|
270
|
+
// omitted, forcing a live re-check.
|
|
271
|
+
export function projectToCsv(entries, { now = Date.now() / 1000 } = {}) {
|
|
272
|
+
const projected = [];
|
|
273
|
+
for (const e of entries) {
|
|
274
|
+
if (e.status <= 0) continue;
|
|
275
|
+
if (e.expires !== undefined) {
|
|
276
|
+
const cutoff = dateToTs(e.expires);
|
|
277
|
+
if (!Number.isFinite(cutoff) || now > cutoff) continue;
|
|
278
|
+
projected.push({ url: e.url, status: e.status, ts: Math.floor(now) });
|
|
279
|
+
continue;
|
|
280
|
+
}
|
|
281
|
+
projected.push({ url: e.url, status: e.status, ts: e.ts });
|
|
282
|
+
}
|
|
283
|
+
return projected;
|
|
284
|
+
}
|
|
285
|
+
|
|
286
|
+
// --- lens: CSV -> owned merge-back ------------------------------------------
|
|
287
|
+
|
|
288
|
+
// Fold a post-run CSV back into the owned entries. Rules (owner, 08-31;
|
|
289
|
+
// absence-inference dropped 08-31 adversarial round):
|
|
290
|
+
// - new URL: added with via "lychee";
|
|
291
|
+
// - status changed: the whole entry is replaced (via "lychee", fresh fields,
|
|
292
|
+
// comments dropped — a stale rationale is worse than none);
|
|
293
|
+
// - status equal: provenance-bearing entries (via != "lychee") stay
|
|
294
|
+
// untouched — a re-confirmation; "lychee" entries adopt the fresh ts;
|
|
295
|
+
// - expired `expires` entry present in the CSV: replaced wholesale (the seed
|
|
296
|
+
// served its purpose);
|
|
297
|
+
// - URL in `failedUrls` (positive evidence from the run's own error report):
|
|
298
|
+
// recorded as STATUS_CLIENT_ERROR under via "lychee", keeping the entry's
|
|
299
|
+
// comments (the rationale still explains the URL);
|
|
300
|
+
// - entry missing from the CSV without failure evidence: untouched. CSV
|
|
301
|
+
// absence is ambiguous — cache_exclude_status, max_cache_age expiry, and
|
|
302
|
+
// URLs no longer in the site all remove entries from healthy runs — so it
|
|
303
|
+
// never justifies a failure verdict on its own.
|
|
304
|
+
export function mergeBack(
|
|
305
|
+
owned,
|
|
306
|
+
csvEntries,
|
|
307
|
+
{ now = Date.now() / 1000, failedUrls = new Set() } = {},
|
|
308
|
+
) {
|
|
309
|
+
const csvByUrl = new Map(csvEntries.map((e) => [e.url, e]));
|
|
310
|
+
const merged = [];
|
|
311
|
+
|
|
312
|
+
for (const entry of owned.entries) {
|
|
313
|
+
const csv = csvByUrl.get(entry.url);
|
|
314
|
+
csvByUrl.delete(entry.url);
|
|
315
|
+
|
|
316
|
+
const expired =
|
|
317
|
+
entry.expires !== undefined &&
|
|
318
|
+
(!Number.isFinite(dateToTs(entry.expires)) ||
|
|
319
|
+
now > dateToTs(entry.expires));
|
|
320
|
+
|
|
321
|
+
if (failedUrls.has(entry.url)) {
|
|
322
|
+
merged.push({
|
|
323
|
+
url: entry.url,
|
|
324
|
+
status: STATUS_CLIENT_ERROR,
|
|
325
|
+
ts: Math.floor(now),
|
|
326
|
+
via: 'lychee',
|
|
327
|
+
comments: entry.comments ?? [],
|
|
328
|
+
});
|
|
329
|
+
continue;
|
|
330
|
+
}
|
|
331
|
+
|
|
332
|
+
if (!csv) {
|
|
333
|
+
merged.push(entry); // absent without evidence: untouched
|
|
334
|
+
continue;
|
|
335
|
+
}
|
|
336
|
+
|
|
337
|
+
if (expired || csv.status !== entry.status) {
|
|
338
|
+
merged.push({
|
|
339
|
+
url: entry.url,
|
|
340
|
+
status: csv.status,
|
|
341
|
+
ts: csv.ts,
|
|
342
|
+
via: 'lychee',
|
|
343
|
+
comments: [],
|
|
344
|
+
});
|
|
345
|
+
} else if (entry.via === 'lychee') {
|
|
346
|
+
merged.push({ ...entry, ts: csv.ts });
|
|
347
|
+
} else {
|
|
348
|
+
merged.push(entry); // re-confirmation: provenance untouched
|
|
349
|
+
}
|
|
350
|
+
}
|
|
351
|
+
|
|
352
|
+
for (const csv of csvByUrl.values()) {
|
|
353
|
+
merged.push({ ...csv, via: 'lychee', comments: [] });
|
|
354
|
+
}
|
|
355
|
+
|
|
356
|
+
return { entries: merged.sort(byUrlBytes), trailing: owned.trailing };
|
|
357
|
+
}
|
|
358
|
+
|
|
359
|
+
// --- migration ---------------------------------------------------------------
|
|
360
|
+
|
|
361
|
+
// One-shot .lycheecache -> link-cache.jsonc: every entry becomes via "lychee".
|
|
362
|
+
export function migrateCsvText(csvText) {
|
|
363
|
+
const { entries, malformed } = parseCsv(csvText);
|
|
364
|
+
const owned = {
|
|
365
|
+
entries: entries.map((e) => ({ ...e, via: 'lychee', comments: [] })),
|
|
366
|
+
trailing: [],
|
|
367
|
+
};
|
|
368
|
+
return { text: serializeOwned(owned), count: entries.length, malformed };
|
|
369
|
+
}
|
package/lib/write.mjs
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
// Atomic file write shared by the bins: write to a same-directory temp file,
|
|
2
|
+
// then rename over the target. A crash mid-write leaves the original intact
|
|
3
|
+
// instead of a truncated cache (which the strict parsers would then reject,
|
|
4
|
+
// wedging every subsequent run).
|
|
5
|
+
|
|
6
|
+
import { renameSync, writeFileSync } from 'node:fs';
|
|
7
|
+
|
|
8
|
+
export function writeFileAtomic(path, text) {
|
|
9
|
+
const tmp = `${path}.${process.pid}.tmp`;
|
|
10
|
+
writeFileSync(tmp, text);
|
|
11
|
+
renameSync(tmp, path);
|
|
12
|
+
}
|
|
@@ -1,17 +1,25 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
// The `
|
|
3
|
-
//
|
|
2
|
+
// The `link-cache` bin (deprecated alias: `refcache`): inspect and prune a
|
|
3
|
+
// link cache — the owned JSONC cache (link-cache.jsonc) or a legacy Lychee CSV
|
|
4
|
+
// (.lycheecache). Read-only unless `--prune` is given, and never hits the
|
|
5
|
+
// network. Run with `--help` for usage.
|
|
4
6
|
//
|
|
5
|
-
//
|
|
6
|
-
//
|
|
7
|
-
// fields.
|
|
7
|
+
// CSV line format: URL,STATUS,UNIX_TIMESTAMP — the URL is CSV-quoted when it
|
|
8
|
+
// contains a comma, so STATUS and TIMESTAMP are read from the final two fields.
|
|
8
9
|
|
|
9
|
-
import { readFileSync, realpathSync
|
|
10
|
+
import { existsSync, readFileSync, realpathSync } from 'node:fs';
|
|
10
11
|
import { fileURLToPath } from 'node:url';
|
|
11
12
|
|
|
13
|
+
import {
|
|
14
|
+
CSV_FILE,
|
|
15
|
+
OWNED_FILE,
|
|
16
|
+
parseOwned,
|
|
17
|
+
serializeOwned,
|
|
18
|
+
} from '../lib/cache.mjs';
|
|
19
|
+
import { writeFileAtomic } from '../lib/write.mjs';
|
|
20
|
+
|
|
12
21
|
const DAY = 86400;
|
|
13
22
|
const BUCKET_COUNT = 5;
|
|
14
|
-
const DEFAULT_PATH = '.lycheecache';
|
|
15
23
|
|
|
16
24
|
// --- parsing ---------------------------------------------------------------
|
|
17
25
|
|
|
@@ -28,9 +36,9 @@ function parseLine(raw) {
|
|
|
28
36
|
return { url, status, ts };
|
|
29
37
|
}
|
|
30
38
|
|
|
31
|
-
// Parse
|
|
32
|
-
// file with a minimal, still-sorted diff) and tagging each entry with its
|
|
33
|
-
// index.
|
|
39
|
+
// Parse a whole CSV cache, keeping the original lines (so a prune can rewrite
|
|
40
|
+
// the file with a minimal, still-sorted diff) and tagging each entry with its
|
|
41
|
+
// line index.
|
|
34
42
|
export function parseCache(text) {
|
|
35
43
|
const lines = text.split('\n');
|
|
36
44
|
const entries = [];
|
|
@@ -44,7 +52,21 @@ export function parseCache(text) {
|
|
|
44
52
|
}
|
|
45
53
|
entries.push({ ...parsed, index });
|
|
46
54
|
});
|
|
47
|
-
return { lines, entries, malformed };
|
|
55
|
+
return { kind: 'csv', lines, entries, malformed };
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
// Adapt the owned JSONC cache to the same entry shape (string status, index).
|
|
59
|
+
export function parseOwnedCache(text) {
|
|
60
|
+
const owned = parseOwned(text);
|
|
61
|
+
const entries = owned.entries.map((e, index) => ({
|
|
62
|
+
url: e.url,
|
|
63
|
+
status: String(e.status),
|
|
64
|
+
ts: e.ts,
|
|
65
|
+
via: e.via,
|
|
66
|
+
index,
|
|
67
|
+
src: e,
|
|
68
|
+
}));
|
|
69
|
+
return { kind: 'owned', owned, entries, malformed: 0 };
|
|
48
70
|
}
|
|
49
71
|
|
|
50
72
|
// --- selection / pruning ---------------------------------------------------
|
|
@@ -88,20 +110,24 @@ export function computeStats(
|
|
|
88
110
|
{ now = Date.now() / 1000, malformed = 0, pruned = 0 } = {},
|
|
89
111
|
) {
|
|
90
112
|
const statusCounts = new Map();
|
|
113
|
+
const viaCounts = new Map();
|
|
91
114
|
const ages = [];
|
|
92
115
|
let oldestTs = null;
|
|
93
116
|
let newestTs = null;
|
|
94
117
|
|
|
95
118
|
for (const entry of entries) {
|
|
96
119
|
statusCounts.set(entry.status, (statusCounts.get(entry.status) ?? 0) + 1);
|
|
120
|
+
if (entry.via !== undefined) {
|
|
121
|
+
viaCounts.set(entry.via, (viaCounts.get(entry.via) ?? 0) + 1);
|
|
122
|
+
}
|
|
97
123
|
if (oldestTs === null || entry.ts < oldestTs) oldestTs = entry.ts;
|
|
98
124
|
if (newestTs === null || entry.ts > newestTs) newestTs = entry.ts;
|
|
99
125
|
ages.push(Math.floor((now - entry.ts) / DAY));
|
|
100
126
|
}
|
|
101
127
|
|
|
102
|
-
const
|
|
103
|
-
|
|
104
|
-
);
|
|
128
|
+
const desc = (a, b) => b[1] - a[1] || a[0].localeCompare(b[0]);
|
|
129
|
+
const byStatus = [...statusCounts.entries()].sort(desc);
|
|
130
|
+
const byVia = [...viaCounts.entries()].sort(desc);
|
|
105
131
|
|
|
106
132
|
const at = (ts) =>
|
|
107
133
|
ts === null ? null : { ts, ageDays: Math.floor((now - ts) / DAY) };
|
|
@@ -112,6 +138,7 @@ export function computeStats(
|
|
|
112
138
|
malformed,
|
|
113
139
|
pruned,
|
|
114
140
|
byStatus,
|
|
141
|
+
byVia,
|
|
115
142
|
oldest,
|
|
116
143
|
newest: at(newestTs),
|
|
117
144
|
ageBuckets: ageHistogram(ages, oldest ? oldest.ageDays : 0),
|
|
@@ -145,14 +172,15 @@ export function formatList(entries, n, { now = Date.now() / 1000 } = {}) {
|
|
|
145
172
|
const head = `${chosen.length} oldest ${chosen.length === 1 ? 'entry' : 'entries'}:`;
|
|
146
173
|
const rows = chosen.map((e) => {
|
|
147
174
|
const ageDays = Math.floor((now - e.ts) / DAY);
|
|
148
|
-
|
|
175
|
+
const via = e.via !== undefined ? ` ${e.via}` : '';
|
|
176
|
+
return ` ${localStamp(e.ts)} ${String(ageDays).padStart(4)}d ${e.status}${via} ${displayUrl(e.url)}`;
|
|
149
177
|
});
|
|
150
178
|
return [head, ...rows].join('\n');
|
|
151
179
|
}
|
|
152
180
|
|
|
153
181
|
export function formatStats(stats, { now = Date.now() / 1000, path } = {}) {
|
|
154
182
|
const lines = [];
|
|
155
|
-
lines.push(`
|
|
183
|
+
lines.push(`Link cache${path ? `: ${path}` : ''}`);
|
|
156
184
|
lines.push(` Entries: ${stats.count}`);
|
|
157
185
|
if (stats.pruned) {
|
|
158
186
|
const before = stats.count + stats.pruned;
|
|
@@ -174,6 +202,14 @@ export function formatStats(stats, { now = Date.now() / 1000, path } = {}) {
|
|
|
174
202
|
lines.push(` ${status.padEnd(5)} ${n}`);
|
|
175
203
|
}
|
|
176
204
|
|
|
205
|
+
if (stats.byVia?.length) {
|
|
206
|
+
lines.push(' Via:');
|
|
207
|
+
const w = Math.max(...stats.byVia.map(([via]) => via.length));
|
|
208
|
+
for (const [via, n] of stats.byVia) {
|
|
209
|
+
lines.push(` ${via.padEnd(w)} ${n}`);
|
|
210
|
+
}
|
|
211
|
+
}
|
|
212
|
+
|
|
177
213
|
if (stats.ageBuckets.length) {
|
|
178
214
|
lines.push(' Age:');
|
|
179
215
|
const w = Math.max(...stats.ageBuckets.map((b) => b.label.length));
|
|
@@ -187,19 +223,33 @@ export function formatStats(stats, { now = Date.now() / 1000, path } = {}) {
|
|
|
187
223
|
|
|
188
224
|
// --- ordered execution -----------------------------------------------------
|
|
189
225
|
|
|
190
|
-
// Run the parsed ops in order over the evolving cache
|
|
191
|
-
//
|
|
192
|
-
|
|
226
|
+
// Run the parsed ops in order over the evolving cache, optionally scoped to
|
|
227
|
+
// URLs matching `match`. Pure: returns the text to print and the text to write
|
|
228
|
+
// (null when nothing was pruned); no I/O.
|
|
229
|
+
export function runOps(
|
|
230
|
+
parsed,
|
|
231
|
+
ops,
|
|
232
|
+
{ now = Date.now() / 1000, path, match = null } = {},
|
|
233
|
+
) {
|
|
193
234
|
const removed = new Set();
|
|
194
235
|
let pruned = 0;
|
|
195
236
|
const out = [];
|
|
196
|
-
|
|
237
|
+
// Match against the unquoted URL: legacy-CSV entries carry the raw
|
|
238
|
+
// (possibly CSV-quoted) field, which would defeat anchored regexes.
|
|
239
|
+
const inScope = (e) => (match ? match.test(displayUrl(e.url)) : true);
|
|
240
|
+
const current = () =>
|
|
241
|
+
parsed.entries.filter((e) => !removed.has(e.index) && inScope(e));
|
|
197
242
|
|
|
198
243
|
for (const op of ops) {
|
|
199
244
|
if (op.kind === 'list') {
|
|
200
245
|
out.push(formatList(current(), Number(op.value), { now }));
|
|
201
246
|
} else if (op.kind === 'prune') {
|
|
202
|
-
|
|
247
|
+
// Manual entries are exempt: pruning schedules re-checks for
|
|
248
|
+
// lychee-verified entries, while a manual entry's lifecycle is owned by
|
|
249
|
+
// its author and its `expires` date — and under prune-refresh rotation
|
|
250
|
+
// re-confirmed seeds keep their original timestamp, so they'd otherwise
|
|
251
|
+
// age to the front and get evicted mid-lifecycle.
|
|
252
|
+
const cur = current().filter((e) => e.via !== 'manual');
|
|
203
253
|
const k = resolvePruneCount(op.value, cur.length);
|
|
204
254
|
for (const victim of selectOldest(cur, k)) removed.add(victim.index);
|
|
205
255
|
pruned += k;
|
|
@@ -216,9 +266,24 @@ export function runOps(parsed, ops, { now = Date.now() / 1000, path } = {}) {
|
|
|
216
266
|
}
|
|
217
267
|
}
|
|
218
268
|
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
269
|
+
let writeText = null;
|
|
270
|
+
if (pruned > 0) {
|
|
271
|
+
// Survivors are all unpruned entries — including those outside the
|
|
272
|
+
// --match scope, which the scope only shields from ops, never from the
|
|
273
|
+
// rewrite.
|
|
274
|
+
const survivors = parsed.entries.filter((e) => !removed.has(e.index));
|
|
275
|
+
if (parsed.kind === 'owned') {
|
|
276
|
+
// Comments of surviving entries travel with them; pruned entries take
|
|
277
|
+
// their comment lines along.
|
|
278
|
+
writeText = serializeOwned({
|
|
279
|
+
entries: survivors.map((e) => e.src),
|
|
280
|
+
trailing: parsed.owned.trailing,
|
|
281
|
+
});
|
|
282
|
+
} else {
|
|
283
|
+
writeText = parsed.lines.filter((_, i) => !removed.has(i)).join('\n');
|
|
284
|
+
}
|
|
285
|
+
}
|
|
286
|
+
return { output: out.join('\n\n'), writeText, pruned };
|
|
222
287
|
}
|
|
223
288
|
|
|
224
289
|
// --- CLI -------------------------------------------------------------------
|
|
@@ -226,6 +291,8 @@ export function runOps(parsed, ops, { now = Date.now() / 1000, path } = {}) {
|
|
|
226
291
|
const FLAGS = new Map([
|
|
227
292
|
['-l', 'list'],
|
|
228
293
|
['--list', 'list'],
|
|
294
|
+
['-m', 'match'],
|
|
295
|
+
['--match', 'match'],
|
|
229
296
|
['-p', 'prune'],
|
|
230
297
|
['--prune', 'prune'],
|
|
231
298
|
['-s', 'summary'],
|
|
@@ -236,6 +303,7 @@ export function parseArgs(argv) {
|
|
|
236
303
|
const ops = [];
|
|
237
304
|
const seen = new Set();
|
|
238
305
|
let path = null;
|
|
306
|
+
let match = null;
|
|
239
307
|
let help = false;
|
|
240
308
|
|
|
241
309
|
for (let i = 0; i < argv.length; i++) {
|
|
@@ -254,6 +322,14 @@ export function parseArgs(argv) {
|
|
|
254
322
|
}
|
|
255
323
|
const value = argv[++i];
|
|
256
324
|
if (value === undefined) throw new Error(`${a} requires a value`);
|
|
325
|
+
if (kind === 'match') {
|
|
326
|
+
try {
|
|
327
|
+
match = new RegExp(value);
|
|
328
|
+
} catch (err) {
|
|
329
|
+
throw new Error(`--match needs a valid regex: ${err.message}`);
|
|
330
|
+
}
|
|
331
|
+
continue;
|
|
332
|
+
}
|
|
257
333
|
if (kind === 'list' && !/^\d+$/.test(value)) {
|
|
258
334
|
throw new Error(`--list needs a count, got: ${value}`);
|
|
259
335
|
}
|
|
@@ -268,23 +344,26 @@ export function parseArgs(argv) {
|
|
|
268
344
|
path = a;
|
|
269
345
|
}
|
|
270
346
|
|
|
271
|
-
return { ops, path
|
|
347
|
+
return { ops, path, match, help };
|
|
272
348
|
}
|
|
273
349
|
|
|
274
|
-
const USAGE = `Usage:
|
|
350
|
+
const USAGE = `Usage: link-cache [CACHE_FILE] [options]
|
|
275
351
|
|
|
276
|
-
Inspect and prune a
|
|
277
|
-
|
|
278
|
-
|
|
352
|
+
Inspect and prune a link cache: the owned ${OWNED_FILE} (default when present)
|
|
353
|
+
or a legacy Lychee CSV like ${CSV_FILE}. Options run in the order given, over
|
|
354
|
+
the evolving cache, so \`-l 5 -p 5\` lists the 5 oldest about to be pruned,
|
|
355
|
+
while \`-p 5 -l 5\` lists the next 5 after pruning.
|
|
279
356
|
|
|
280
357
|
-l, --list NUM list the NUM oldest entries
|
|
281
|
-
-
|
|
282
|
-
-
|
|
358
|
+
-m, --match REGEX scope all operations to URLs matching REGEX
|
|
359
|
+
-p, --prune NUM[%] drop the NUM (or NUM%) oldest non-manual entries, then
|
|
360
|
+
rewrite (manual entries retire via their expires date)
|
|
361
|
+
-s, --summary print a summary (counts, ages, status, via, histogram)
|
|
283
362
|
-h, --help show this help
|
|
284
363
|
|
|
285
364
|
With no options, prints the summary. A flag may not be repeated.`;
|
|
286
365
|
|
|
287
|
-
function main(argv) {
|
|
366
|
+
export function main(argv) {
|
|
288
367
|
let args;
|
|
289
368
|
try {
|
|
290
369
|
args = parseArgs(argv);
|
|
@@ -298,23 +377,35 @@ function main(argv) {
|
|
|
298
377
|
return;
|
|
299
378
|
}
|
|
300
379
|
|
|
380
|
+
const path = args.path ?? (existsSync(OWNED_FILE) ? OWNED_FILE : CSV_FILE);
|
|
381
|
+
|
|
301
382
|
let text;
|
|
302
383
|
try {
|
|
303
|
-
text = readFileSync(
|
|
384
|
+
text = readFileSync(path, 'utf8');
|
|
304
385
|
} catch {
|
|
305
|
-
console.error(`[error] cannot read
|
|
386
|
+
console.error(`[error] cannot read cache file: ${path}`);
|
|
387
|
+
process.exit(1);
|
|
388
|
+
}
|
|
389
|
+
|
|
390
|
+
const isOwned = path.endsWith('.jsonc') || text.trimStart().startsWith('{');
|
|
391
|
+
let parsed;
|
|
392
|
+
try {
|
|
393
|
+
parsed = isOwned ? parseOwnedCache(text) : parseCache(text);
|
|
394
|
+
} catch (err) {
|
|
395
|
+
console.error(`[error] ${err.message}`);
|
|
306
396
|
process.exit(1);
|
|
307
397
|
}
|
|
308
398
|
|
|
309
399
|
const now = Date.now() / 1000;
|
|
310
400
|
const ops = args.ops.length ? args.ops : [{ kind: 'summary' }];
|
|
311
|
-
const { output,
|
|
401
|
+
const { output, writeText } = runOps(parsed, ops, {
|
|
312
402
|
now,
|
|
313
|
-
path
|
|
403
|
+
path,
|
|
404
|
+
match: args.match,
|
|
314
405
|
});
|
|
315
406
|
|
|
316
407
|
if (output) console.log(output);
|
|
317
|
-
if (
|
|
408
|
+
if (writeText !== null) writeFileAtomic(path, writeText);
|
|
318
409
|
}
|
|
319
410
|
|
|
320
411
|
// Real-path compare, not `file://${argv[1]}`: npm links bins as symlinks, so
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// The deprecated `refcache` alias bin: warn, then delegate to link-cache. A
|
|
3
|
+
// dedicated wrapper (not an argv[1] sniff) so the notice also fires under
|
|
4
|
+
// npm's Windows shims, which invoke the target file directly.
|
|
5
|
+
|
|
6
|
+
import { main } from './index.mjs';
|
|
7
|
+
|
|
8
|
+
process.stderr.write(
|
|
9
|
+
'[warn] the refcache bin is deprecated; use link-cache instead.\n',
|
|
10
|
+
);
|
|
11
|
+
main(process.argv.slice(2));
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "link-cache",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "Zero-dependency helper CLIs for cached Lychee link-checking: lychee-norm-cache (run +
|
|
3
|
+
"version": "0.4.1",
|
|
4
|
+
"description": "Zero-dependency helper CLIs for cached Lychee link-checking around an owned JSONC cache (link-cache.jsonc) with provenance: lychee-norm-cache (run + sync caches) and link-cache (inspect/prune).",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"lychee",
|
|
7
7
|
"link-checker",
|
|
@@ -17,16 +17,21 @@
|
|
|
17
17
|
"type": "module",
|
|
18
18
|
"bin": {
|
|
19
19
|
"lychee-norm-cache": "check/index.mjs",
|
|
20
|
-
"
|
|
20
|
+
"link-cache": "link-cache/index.mjs",
|
|
21
|
+
"refcache": "link-cache/refcache.mjs"
|
|
21
22
|
},
|
|
22
23
|
"files": [
|
|
23
24
|
"check/index.mjs",
|
|
24
|
-
"
|
|
25
|
+
"lib/cache.mjs",
|
|
26
|
+
"lib/write.mjs",
|
|
27
|
+
"link-cache/index.mjs",
|
|
28
|
+
"link-cache/refcache.mjs"
|
|
25
29
|
],
|
|
26
30
|
"scripts": {
|
|
27
31
|
"check": "npm run check:format && npm test",
|
|
28
32
|
"check:format": "prettier --check .",
|
|
29
33
|
"fix:format": "prettier --write .",
|
|
34
|
+
"install:safe": "npm ci --ignore-scripts --no-audit --no-fund",
|
|
30
35
|
"test": "node --test"
|
|
31
36
|
},
|
|
32
37
|
"prettier": {
|