domma-cms 1.8.26 → 1.8.27
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/docs/cache.md +21 -0
- package/package.json +1 -1
- package/server/routes/api/cache.js +2 -1
- package/server/routes/docs-public.js +1 -1
- package/server/routes/public.js +14 -52
- package/server/services/cache/index.js +54 -6
- package/server/services/content.js +61 -0
- package/server/services/htmlResponse.js +138 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [1.8.27] - 2026-10-19
|
|
4
|
+
|
|
5
|
+
- **Pages come out of the cache about twice as fast.** Every public request decided whether you may see the page before it looked in the response cache, and that decision parsed the whole page - shortcodes, collections and all - so a cached page still cost a full parse on every visit. The parsed page is now kept per file (keyed on the file's time and size, emptied whenever anything leaves the response cache, and only while the cache is on), so a cached page costs a file stat and a lookup. On a 40 KB home page: 230 to 540 pages a second, the typical response from 41 ms to 18 ms under load.
|
|
6
|
+
- **A cached page is compressed once, not on every visit.** The response cache held the page as HTML and brotli ran on it for every visitor; the compressed body is now kept as well, per encoding, addressed by the page's own content hash (so it can never be stale and needs no invalidation). After the change above this was most of what a cached page still cost. The ETag now names the encoding (`"<hash>-br"`); a browser holding the other one's validator still gets its 304.
|
|
7
|
+
- **`x-dm-cache` on every page response** says which way it went: `hit`, `miss`, `off` (the cache is disabled) or `bypass` (a draft, an expiring page or a per-viewer page, never cached). `GET /api/cache/status` gains `pageMemo` (size, hits, misses).
|
|
8
|
+
|
|
3
9
|
## [1.8.26] - 2026-10-18
|
|
4
10
|
|
|
5
11
|
- **Updates clear out their old backups.** Every `npx domma-cms update` copies the engine and plugins into `.domma-backups/` first, and nothing ever removed those copies - some sites held 250+ of up to 490 MB each. After a successful update it now keeps the newest 7, plus one about a week, a month, three, six and twelve months old (at most 16), and nothing older than a year. Backups you put in that folder yourself, under any other name, are never touched.
|
package/docs/cache.md
CHANGED
|
@@ -73,6 +73,27 @@ All endpoints require an admin Bearer token. The dashboard has a cache widget sh
|
|
|
73
73
|
|
|
74
74
|
When you toggle the cache **off**, the in-memory data is preserved - toggle back on and your warm entries are still there. When you turn it **off in `config/cache.json`** and restart, the next boot starts with a `NoneDriver` and no data.
|
|
75
75
|
|
|
76
|
+
## Seeing whether a page was cached
|
|
77
|
+
|
|
78
|
+
Every page response carries `x-dm-cache`:
|
|
79
|
+
|
|
80
|
+
| Value | Meaning |
|
|
81
|
+
|---|---|
|
|
82
|
+
| `hit` | served from the response cache |
|
|
83
|
+
| `miss` | rendered on this request and stored |
|
|
84
|
+
| `off` | the cache is disabled; rendered |
|
|
85
|
+
| `bypass` | never cached: a draft preview, a page with expiring shortcodes, or a per-viewer page for a signed-in visitor |
|
|
86
|
+
|
|
87
|
+
Open the browser's network panel, or `curl -sI https://example.com/ | grep x-dm-cache`.
|
|
88
|
+
|
|
89
|
+
## The parsed-page memo
|
|
90
|
+
|
|
91
|
+
Before the response cache is asked, the route decides whether the viewer may see the page at all (`resolvePublishedPage()`), and that decision reads the page with `getPage()`, which parses the body to learn whether it expires, renders per viewer, and which collections and blocks it uses. Without help a cache hit still paid for that parse. `content.js` keeps the anonymous parse per file, keyed on the file's mtime and size, emptied through `cache.onInvalidate()` whenever anything leaves the response cache, and only while the cache is enabled. A viewer's read (`{user}`) or an expiry read is never memoised. `GET /api/cache/status` reports it as `pageMemo`.
|
|
92
|
+
|
|
93
|
+
## The compressed body
|
|
94
|
+
|
|
95
|
+
`services/htmlResponse.js` `sendHtml()`: a page that came from the response cache (`{cached: true}` - the page route and the docs route) is also kept compressed, one entry per encoding under `enc:<br|gzip>:<sha1 of the HTML>`. Content-addressed, so never stale and never invalidated; bounded by the cache's LRU and TTL. The negotiation and parameters match @fastify/compress (brotli quality 4), which steps aside when `content-encoding` is already set. A 404, a draft or a per-viewer page is not kept and goes out through @fastify/compress as before.
|
|
96
|
+
|
|
76
97
|
## Thundering-herd safety
|
|
77
98
|
|
|
78
99
|
If three concurrent requests hit a cold key, the producer runs exactly once - the other two await the first's result via an in-flight promise map. This matters for popular pages on cache cold-start: without coalescing, N parallel requests trigger N parallel renders, each writing the same value back to the cache.
|
package/package.json
CHANGED
|
@@ -11,6 +11,7 @@ import fs from 'fs/promises';
|
|
|
11
11
|
import path from 'path';
|
|
12
12
|
import {fileURLToPath} from 'url';
|
|
13
13
|
import * as cache from '../../services/cache/index.js';
|
|
14
|
+
import {pageMemoStats} from '../../services/content.js';
|
|
14
15
|
import {authenticate, requireAdmin} from '../../middleware/auth.js';
|
|
15
16
|
|
|
16
17
|
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
|
@@ -37,7 +38,7 @@ async function writeCacheConfig(cfg) {
|
|
|
37
38
|
export async function cacheRoutes(fastify) {
|
|
38
39
|
const adminOnly = {preHandler: [authenticate, requireAdmin]};
|
|
39
40
|
|
|
40
|
-
fastify.get('/cache/status', adminOnly, async () => cache.getStats());
|
|
41
|
+
fastify.get('/cache/status', adminOnly, async () => ({...cache.getStats(), pageMemo: pageMemoStats()}));
|
|
41
42
|
|
|
42
43
|
fastify.get('/cache/keys', adminOnly, async () => ({entries: cache.listEntries()}));
|
|
43
44
|
|
|
@@ -41,7 +41,7 @@ export async function docsPublicRoutes(fastify) {
|
|
|
41
41
|
return reply.type('text/html; charset=utf-8').send(await render404(`/domma-docs${docPath ? '/' + docPath : ''}`));
|
|
42
42
|
}
|
|
43
43
|
// private + ETag: the response is role-dependent, so no shared proxy may reuse it.
|
|
44
|
-
return sendHtml(request, reply, html);
|
|
44
|
+
return sendHtml(request, reply, html, {cached: true}); // renderDocPage() caches per role
|
|
45
45
|
};
|
|
46
46
|
|
|
47
47
|
fastify.get('/domma-docs', handle); // exact - params['*'] === '' (home landing)
|
package/server/routes/public.js
CHANGED
|
@@ -18,7 +18,6 @@ import {viewerRoleKey} from '../services/userRoles.js';
|
|
|
18
18
|
import {hooks} from '../services/hooks.js';
|
|
19
19
|
import {getConfig} from '../config.js';
|
|
20
20
|
import * as cache from '../services/cache/index.js';
|
|
21
|
-
import {createHash} from 'crypto';
|
|
22
21
|
|
|
23
22
|
/**
|
|
24
23
|
* Derive the absolute origin for the current request.
|
|
@@ -40,55 +39,10 @@ export function getBaseUrl(request) {
|
|
|
40
39
|
return `${request.protocol}://${host}`;
|
|
41
40
|
}
|
|
42
41
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
* @param {string} str
|
|
48
|
-
* @returns {string}
|
|
49
|
-
*/
|
|
50
|
-
function escapeHtml(str) {
|
|
51
|
-
return String(str)
|
|
52
|
-
.replace(/&/g, '&')
|
|
53
|
-
.replace(/</g, '<')
|
|
54
|
-
.replace(/>/g, '>')
|
|
55
|
-
.replace(/"/g, '"')
|
|
56
|
-
.replace(/'/g, ''');
|
|
57
|
-
}
|
|
58
|
-
|
|
59
|
-
/*
|
|
60
|
-
* Send an HTML page with headers that make a browser CHECK before reusing it.
|
|
61
|
-
*
|
|
62
|
-
* Public pages went out with no Cache-Control, no ETag and no Last-Modified at
|
|
63
|
-
* all. With nothing to go on a browser applies heuristic caching, so an edited
|
|
64
|
-
* page could keep showing the old content on a normal refresh - the server was
|
|
65
|
-
* serving the new one the whole time. That is indistinguishable from a CMS bug
|
|
66
|
-
* and was reported as one.
|
|
67
|
-
*
|
|
68
|
-
* `no-cache` does not mean "do not store": it means store it, then revalidate
|
|
69
|
-
* before every reuse. Paired with an ETag that makes the common case a 304 with
|
|
70
|
-
* no body rather than a full re-download.
|
|
71
|
-
*
|
|
72
|
-
* `private` because these responses are role-dependent - the response cache
|
|
73
|
-
* keys on role for exactly that reason - and a shared proxy must never hand one
|
|
74
|
-
* visitor a page rendered for another's permissions.
|
|
75
|
-
*/
|
|
76
|
-
export function sendHtml(request, reply, html) {
|
|
77
|
-
const etag = `"${createHash('sha1').update(html).digest('base64')}"`;
|
|
78
|
-
reply.header('cache-control', 'private, no-cache');
|
|
79
|
-
reply.header('etag', etag);
|
|
80
|
-
|
|
81
|
-
// If-None-Match may carry a list, and a weak validator's W/ prefix.
|
|
82
|
-
const seen = String(request.headers['if-none-match'] || '')
|
|
83
|
-
.split(',')
|
|
84
|
-
.map((t) => t.trim().replace(/^W\//, ''));
|
|
85
|
-
if (seen.includes(etag)) return reply.code(304).send();
|
|
86
|
-
|
|
87
|
-
// Deliberately no reply.code() here: the 404 handlers have already called
|
|
88
|
-
// reply.status(404), and forcing 200 would turn every not-found into a
|
|
89
|
-
// soft 404 that search engines index as a real page.
|
|
90
|
-
return reply.type('text/html; charset=utf-8').send(html);
|
|
91
|
-
}
|
|
42
|
+
// Validators, the 304 and the compressed body kept per encoding live in
|
|
43
|
+
// services/htmlResponse.js; re-exported because docs-public.js imports it from here.
|
|
44
|
+
import {sendHtml} from '../services/htmlResponse.js';
|
|
45
|
+
export {sendHtml};
|
|
92
46
|
|
|
93
47
|
export async function publicRoutes(fastify) {
|
|
94
48
|
// Admin panel: serve index.html for all /admin/* paths (SPA fallback)
|
|
@@ -249,6 +203,7 @@ export async function publicRoutes(fastify) {
|
|
|
249
203
|
if (isDraft) {
|
|
250
204
|
reply.header('cache-control', 'no-store, private');
|
|
251
205
|
reply.header('x-robots-tag', 'noindex, nofollow');
|
|
206
|
+
reply.header('x-dm-cache', 'bypass');
|
|
252
207
|
return reply.type('text/html; charset=utf-8').send(await render());
|
|
253
208
|
}
|
|
254
209
|
|
|
@@ -256,6 +211,7 @@ export async function publicRoutes(fastify) {
|
|
|
256
211
|
// so these pages must render against the expiry ledger on every request.
|
|
257
212
|
if (page.hasExpiry) {
|
|
258
213
|
reply.header('cache-control', 'no-store, private');
|
|
214
|
+
reply.header('x-dm-cache', 'bypass');
|
|
259
215
|
hooks.emit('content:pageViewed', {urlPath, title: page.title || ''});
|
|
260
216
|
return reply.type('text/html; charset=utf-8').send(await render());
|
|
261
217
|
}
|
|
@@ -265,6 +221,7 @@ export async function publicRoutes(fastify) {
|
|
|
265
221
|
// must not share an entry. Anonymous viewers of such a view get no rows
|
|
266
222
|
// at all (views.js), which is one output, so they still share the cache.
|
|
267
223
|
if (page.perViewer && userObj) {
|
|
224
|
+
reply.header('x-dm-cache', 'bypass');
|
|
268
225
|
hooks.emit('content:pageViewed', {urlPath, title: page.title || ''});
|
|
269
226
|
return sendHtml(request, reply, await render());
|
|
270
227
|
}
|
|
@@ -276,9 +233,14 @@ export async function publicRoutes(fastify) {
|
|
|
276
233
|
const cacheKey = `page:${urlPath}:m${pageMtime}:role${viewerRoleKey(userObj)}:o${baseUrl}`;
|
|
277
234
|
// The footer's blocks (site.json footer.blocks) are on every page: editing one must reach the cached pages.
|
|
278
235
|
const cacheTags = [`page:${urlPath}`, ...(page.cacheTags || []), 'nav', 'site', ...footerBlockCacheTags(getConfig('site'))];
|
|
279
|
-
|
|
236
|
+
// x-dm-cache says which way this response went - hit, miss, off (the
|
|
237
|
+
// cache is disabled) or bypass (a draft, an expiring or a per-viewer
|
|
238
|
+
// page, never cached) - so a slow page can be read from the browser.
|
|
239
|
+
const {value: html, status} = await cache.wrapStatus(cacheKey, render, {tags: cacheTags});
|
|
240
|
+
reply.header('x-dm-cache', status);
|
|
280
241
|
hooks.emit('content:pageViewed', {urlPath, title: page.title || ''});
|
|
281
|
-
|
|
242
|
+
// cached: this HTML is shared, so its compressed body is kept too (htmlResponse.js).
|
|
243
|
+
return sendHtml(request, reply, html, {cached: true});
|
|
282
244
|
});
|
|
283
245
|
}
|
|
284
246
|
|
|
@@ -9,6 +9,9 @@ let _lastConfig = null;
|
|
|
9
9
|
/** @type {Map<string, Promise<*>>} */
|
|
10
10
|
const _inflight = new Map();
|
|
11
11
|
|
|
12
|
+
/** Called with a reason whenever anything leaves the cache - see onInvalidate(). */
|
|
13
|
+
const _invalidateListeners = new Set();
|
|
14
|
+
|
|
12
15
|
const _stats = {
|
|
13
16
|
hits: 0,
|
|
14
17
|
misses: 0,
|
|
@@ -28,6 +31,7 @@ const _stats = {
|
|
|
28
31
|
*/
|
|
29
32
|
export async function initCache(config = {}) {
|
|
30
33
|
_lastConfig = {...config};
|
|
34
|
+
_notifyInvalidate('init');
|
|
31
35
|
_stats.initialisedAt = new Date().toISOString();
|
|
32
36
|
if (config.enabled === false) {
|
|
33
37
|
_driver = new NoneDriver();
|
|
@@ -72,6 +76,7 @@ export async function setEnabled(enabled) {
|
|
|
72
76
|
} else {
|
|
73
77
|
_enabled = false;
|
|
74
78
|
_inflight.clear();
|
|
79
|
+
_notifyInvalidate('disabled');
|
|
75
80
|
}
|
|
76
81
|
}
|
|
77
82
|
|
|
@@ -79,6 +84,28 @@ export function isEnabled() {
|
|
|
79
84
|
return _enabled;
|
|
80
85
|
}
|
|
81
86
|
|
|
87
|
+
/**
|
|
88
|
+
* Hear about every invalidation: a tag flush, a clear, a re-init or the cache
|
|
89
|
+
* being switched off. Anything that keeps its own derived copy of what the
|
|
90
|
+
* response cache holds (content.js's parsed-page memo) drops it here, so it
|
|
91
|
+
* can never be fresher or staler than the cache itself. The listener gets
|
|
92
|
+
* `(reason, tags)` and its errors are swallowed - an invalidation must never
|
|
93
|
+
* fail because an onlooker threw.
|
|
94
|
+
*
|
|
95
|
+
* @param {(reason: 'tags'|'clear'|'init'|'disabled', tags?: string[]) => void} fn
|
|
96
|
+
* @returns {() => void} unsubscribe
|
|
97
|
+
*/
|
|
98
|
+
export function onInvalidate(fn) {
|
|
99
|
+
_invalidateListeners.add(fn);
|
|
100
|
+
return () => _invalidateListeners.delete(fn);
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
function _notifyInvalidate(reason, tags) {
|
|
104
|
+
for (const fn of _invalidateListeners) {
|
|
105
|
+
try { fn(reason, tags); } catch { /* an onlooker's problem, not the cache's */ }
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
|
|
82
109
|
export async function get(key) {
|
|
83
110
|
return _driver.get(_vk(key));
|
|
84
111
|
}
|
|
@@ -91,13 +118,17 @@ export async function invalidateTags(tags) {
|
|
|
91
118
|
if (!Array.isArray(tags) || tags.length === 0) return;
|
|
92
119
|
_stats.invalidations++;
|
|
93
120
|
_stats.lastInvalidationAt = new Date().toISOString();
|
|
94
|
-
|
|
121
|
+
const out = await _driver.invalidateTags(tags);
|
|
122
|
+
_notifyInvalidate('tags', tags);
|
|
123
|
+
return out;
|
|
95
124
|
}
|
|
96
125
|
|
|
97
126
|
export async function clear() {
|
|
98
127
|
_inflight.clear();
|
|
99
128
|
_stats.lastClearedAt = new Date().toISOString();
|
|
100
|
-
|
|
129
|
+
const out = await _driver.clear();
|
|
130
|
+
_notifyInvalidate('clear');
|
|
131
|
+
return out;
|
|
101
132
|
}
|
|
102
133
|
|
|
103
134
|
/**
|
|
@@ -151,19 +182,35 @@ export function getStats() {
|
|
|
151
182
|
* @param {{ttlSeconds?: number, tags?: string[]}} [opts]
|
|
152
183
|
*/
|
|
153
184
|
export async function wrap(key, producer, opts = {}) {
|
|
185
|
+
return (await wrapStatus(key, producer, opts)).value;
|
|
186
|
+
}
|
|
187
|
+
|
|
188
|
+
/**
|
|
189
|
+
* wrap(), and which way it went: `{value, status}` with status `hit`, `miss`
|
|
190
|
+
* (this call produced it, or waited on the in-flight production) or `off`
|
|
191
|
+
* (the cache is disabled and the producer simply ran). The public route puts
|
|
192
|
+
* the status on the response as `x-dm-cache`, so a slow page can be told
|
|
193
|
+
* apart from a page that is never cached without reading the server's logs.
|
|
194
|
+
*
|
|
195
|
+
* @param {string} key
|
|
196
|
+
* @param {() => Promise<*>} producer
|
|
197
|
+
* @param {{ttlSeconds?: number, tags?: string[]}} [opts]
|
|
198
|
+
* @returns {Promise<{value: *, status: 'hit'|'miss'|'off'}>}
|
|
199
|
+
*/
|
|
200
|
+
export async function wrapStatus(key, producer, opts = {}) {
|
|
154
201
|
if (!_enabled) {
|
|
155
202
|
const raw = await producer();
|
|
156
|
-
return _stripCacheTags(raw);
|
|
203
|
+
return {value: _stripCacheTags(raw), status: 'off'};
|
|
157
204
|
}
|
|
158
205
|
|
|
159
206
|
const vkey = _vk(key);
|
|
160
207
|
const hit = await _driver.get(vkey);
|
|
161
208
|
if (hit !== undefined) {
|
|
162
209
|
_stats.hits++;
|
|
163
|
-
return hit;
|
|
210
|
+
return {value: hit, status: 'hit'};
|
|
164
211
|
}
|
|
165
212
|
|
|
166
|
-
if (_inflight.has(vkey)) return _inflight.get(vkey);
|
|
213
|
+
if (_inflight.has(vkey)) return {value: await _inflight.get(vkey), status: 'miss'};
|
|
167
214
|
|
|
168
215
|
_stats.misses++;
|
|
169
216
|
const promise = (async () => {
|
|
@@ -183,7 +230,7 @@ export async function wrap(key, producer, opts = {}) {
|
|
|
183
230
|
})();
|
|
184
231
|
|
|
185
232
|
_inflight.set(vkey, promise);
|
|
186
|
-
return promise;
|
|
233
|
+
return {value: await promise, status: 'miss'};
|
|
187
234
|
}
|
|
188
235
|
|
|
189
236
|
function _vk(key) {
|
|
@@ -207,6 +254,7 @@ function _stripCacheTags(raw) {
|
|
|
207
254
|
}
|
|
208
255
|
|
|
209
256
|
export function _resetForTests() {
|
|
257
|
+
_notifyInvalidate('init');
|
|
210
258
|
_driver = new NoneDriver();
|
|
211
259
|
_enabled = false;
|
|
212
260
|
_lastConfig = null;
|
|
@@ -16,6 +16,63 @@ const CONTENT_DIR = config.content.contentDir;
|
|
|
16
16
|
const PAGES_DIR = path.join(CONTENT_DIR, 'pages');
|
|
17
17
|
const MEDIA_DIR = config.content.mediaDir;
|
|
18
18
|
|
|
19
|
+
// ---------------------------------------------------------------------------
|
|
20
|
+
// Parsed-page memo
|
|
21
|
+
// ---------------------------------------------------------------------------
|
|
22
|
+
//
|
|
23
|
+
// Every public request decides whether the viewer may see the page BEFORE the
|
|
24
|
+
// response cache is consulted, and that decision (resolvePublishedPage) reads
|
|
25
|
+
// the page with getPage(), which parses the whole body - shortcodes,
|
|
26
|
+
// collections, sanitising - to learn `hasExpiry`, `perViewer` and the cache
|
|
27
|
+
// tags. So a response-cache "hit" still cost a full anonymous parse, 3-4 ms on
|
|
28
|
+
// an ordinary page and the whole of the per-request CPU; the cache only ever
|
|
29
|
+
// saved the second parse. This memo keeps the anonymous parse (no user, no
|
|
30
|
+
// expiry ledger) per file, keyed on the file's mtime and size, and is emptied
|
|
31
|
+
// whenever ANYTHING leaves the response cache (cache.onInvalidate), so it is
|
|
32
|
+
// exactly as fresh as the cache: an edit changes the mtime, a collection or
|
|
33
|
+
// block save flushes both. It is on only while the response cache is on - in
|
|
34
|
+
// development every read parses, as before - and a short TTL bounds anything
|
|
35
|
+
// the hooks do not know about. Entries are cloned in and out, so a caller
|
|
36
|
+
// that edits the object it was handed edits its own copy.
|
|
37
|
+
const PAGE_MEMO_MAX = 500;
|
|
38
|
+
const PAGE_MEMO_TTL_MS = 5 * 60 * 1000;
|
|
39
|
+
/** @type {Map<string, {mtimeMs: number, size: number, at: number, page: object}>} */
|
|
40
|
+
const _pageMemo = new Map();
|
|
41
|
+
const _pageMemoStats = {hits: 0, misses: 0};
|
|
42
|
+
cache.onInvalidate(() => _pageMemo.clear());
|
|
43
|
+
|
|
44
|
+
/** For the cache status route and tests: how the memo is doing. */
|
|
45
|
+
export function pageMemoStats() {
|
|
46
|
+
return {size: _pageMemo.size, max: PAGE_MEMO_MAX, hits: _pageMemoStats.hits, misses: _pageMemoStats.misses};
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
export function _resetPageMemoForTests() {
|
|
50
|
+
_pageMemo.clear();
|
|
51
|
+
_pageMemoStats.hits = 0;
|
|
52
|
+
_pageMemoStats.misses = 0;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
async function readPageMemo(filePath) {
|
|
56
|
+
const st = await fs.stat(filePath);
|
|
57
|
+
const now = Date.now();
|
|
58
|
+
const kept = _pageMemo.get(filePath);
|
|
59
|
+
if (kept && kept.mtimeMs === st.mtimeMs && kept.size === st.size && now - kept.at < PAGE_MEMO_TTL_MS) {
|
|
60
|
+
_pageMemo.delete(filePath);
|
|
61
|
+
_pageMemo.set(filePath, kept); // most recently used last
|
|
62
|
+
_pageMemoStats.hits++;
|
|
63
|
+
return structuredClone(kept.page);
|
|
64
|
+
}
|
|
65
|
+
_pageMemoStats.misses++;
|
|
66
|
+
const page = await readPageFile(filePath);
|
|
67
|
+
try {
|
|
68
|
+
_pageMemo.set(filePath, {mtimeMs: st.mtimeMs, size: st.size, at: now, page: structuredClone(page)});
|
|
69
|
+
} catch {
|
|
70
|
+
return page; // something in it cannot be cloned: hand it over unmemoised
|
|
71
|
+
}
|
|
72
|
+
while (_pageMemo.size > PAGE_MEMO_MAX) _pageMemo.delete(_pageMemo.keys().next().value);
|
|
73
|
+
return page;
|
|
74
|
+
}
|
|
75
|
+
|
|
19
76
|
// ---------------------------------------------------------------------------
|
|
20
77
|
// Pages
|
|
21
78
|
// ---------------------------------------------------------------------------
|
|
@@ -71,6 +128,10 @@ export async function getPageMtime(urlPath) {
|
|
|
71
128
|
export async function getPage(urlPath, opts = {}) {
|
|
72
129
|
const filePath = await resolveExistingFilePath(urlPath);
|
|
73
130
|
try {
|
|
131
|
+
// Only the anonymous, ledger-free read is memoised: a viewer's read
|
|
132
|
+
// differs per role ([menu], row-scoped [view]) and an expiry read
|
|
133
|
+
// writes to the ledger.
|
|
134
|
+
if (!opts.user && !opts.expiry && cache.isEnabled()) return await readPageMemo(filePath);
|
|
74
135
|
return await readPageFile(filePath, {user: opts.user || null, expiry:opts.expiry});
|
|
75
136
|
} catch {
|
|
76
137
|
return null;
|
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Sending a rendered page: validators, and the compressed body kept per
|
|
3
|
+
* encoding.
|
|
4
|
+
*
|
|
5
|
+
* @fastify/compress compresses every response as it leaves, so a page that
|
|
6
|
+
* came out of the response cache was still brotli'd on every request - after
|
|
7
|
+
* the parsed-page memo that was most of what a cache hit cost (a 40 KB page
|
|
8
|
+
* is ~2 ms of brotli at quality 4, in the thread pool but paid for all the
|
|
9
|
+
* same). Here the compressed body is cached too, addressed by the content:
|
|
10
|
+
* the key is the encoding plus the sha1 of the HTML, the same hash the ETag
|
|
11
|
+
* is made from. Content-addressed means it can never be stale - a different
|
|
12
|
+
* page is a different hash - so it needs no tags and no invalidation, and two
|
|
13
|
+
* cache keys that render the same HTML (two base URLs, say) share one body.
|
|
14
|
+
* Its memory is bounded by the cache's own LRU and TTL.
|
|
15
|
+
*
|
|
16
|
+
* Only a response that came from the response cache asks for it (`cached`):
|
|
17
|
+
* a 404 carries the requested path, a per-viewer page is one visitor's, and
|
|
18
|
+
* caching those would fill the cache with bodies nobody asks for twice.
|
|
19
|
+
* Everything else goes out as HTML and @fastify/compress does what it did.
|
|
20
|
+
*
|
|
21
|
+
* The negotiation and the parameters match @fastify/compress (br before
|
|
22
|
+
* gzip, brotli quality 4, zlib's default gzip level, the 1 KB threshold), so
|
|
23
|
+
* a visitor gets the same bytes either way; compress sees `content-encoding`
|
|
24
|
+
* already set and steps aside.
|
|
25
|
+
*/
|
|
26
|
+
import {createHash} from 'node:crypto';
|
|
27
|
+
import zlib from 'node:zlib';
|
|
28
|
+
import {promisify} from 'node:util';
|
|
29
|
+
import * as cache from './cache/index.js';
|
|
30
|
+
|
|
31
|
+
const brotli = promisify(zlib.brotliCompress);
|
|
32
|
+
const gzip = promisify(zlib.gzip);
|
|
33
|
+
|
|
34
|
+
/** Responses under this many bytes are not worth compressing - @fastify/compress's threshold. */
|
|
35
|
+
export const COMPRESS_THRESHOLD = 1024;
|
|
36
|
+
/** Offered in order of preference: the tie-break when the client rates them equally. */
|
|
37
|
+
export const ENCODINGS = ['br', 'gzip'];
|
|
38
|
+
|
|
39
|
+
/**
|
|
40
|
+
* Which of our encodings the client prefers, or null for none (no header, or
|
|
41
|
+
* nothing acceptable). `*` and `x-gzip` count as gzip, as @fastify/compress
|
|
42
|
+
* reads them; a `q=0` rules an encoding out.
|
|
43
|
+
*
|
|
44
|
+
* @param {string|undefined} header - the Accept-Encoding value
|
|
45
|
+
* @param {string[]} [offered]
|
|
46
|
+
* @returns {string|null}
|
|
47
|
+
*/
|
|
48
|
+
export function pickEncoding(header, offered = ENCODINGS) {
|
|
49
|
+
if (!header) return null;
|
|
50
|
+
const q = new Map();
|
|
51
|
+
for (const part of String(header).toLowerCase().split(',')) {
|
|
52
|
+
const [rawName, ...params] = part.trim().split(';');
|
|
53
|
+
let name = rawName.trim();
|
|
54
|
+
if (!name) continue;
|
|
55
|
+
if (name === '*' || name === 'x-gzip') name = 'gzip';
|
|
56
|
+
let weight = 1;
|
|
57
|
+
for (const p of params) {
|
|
58
|
+
const m = p.trim().match(/^q=([0-9.]+)$/);
|
|
59
|
+
if (m) weight = Number(m[1]);
|
|
60
|
+
}
|
|
61
|
+
if (!q.has(name) || q.get(name) < weight) q.set(name, weight);
|
|
62
|
+
}
|
|
63
|
+
let best = null;
|
|
64
|
+
let bestQ = 0;
|
|
65
|
+
for (const enc of offered) {
|
|
66
|
+
const weight = q.get(enc) ?? 0;
|
|
67
|
+
if (weight > bestQ) { best = enc; bestQ = weight; }
|
|
68
|
+
}
|
|
69
|
+
return best;
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
/**
|
|
73
|
+
* The compressed body, the way @fastify/compress would have made it.
|
|
74
|
+
*
|
|
75
|
+
* @param {string} html
|
|
76
|
+
* @param {string} encoding - 'br' | 'gzip'
|
|
77
|
+
* @returns {Promise<Buffer>}
|
|
78
|
+
*/
|
|
79
|
+
export function encodeHtml(html, encoding) {
|
|
80
|
+
const input = Buffer.from(html, 'utf8');
|
|
81
|
+
if (encoding === 'br') {
|
|
82
|
+
return brotli(input, {params: {
|
|
83
|
+
[zlib.constants.BROTLI_PARAM_QUALITY]: 4,
|
|
84
|
+
[zlib.constants.BROTLI_PARAM_SIZE_HINT]: input.length,
|
|
85
|
+
}});
|
|
86
|
+
}
|
|
87
|
+
if (encoding === 'gzip') return gzip(input);
|
|
88
|
+
throw new Error(`Unknown encoding: ${encoding}`);
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
/** The validator's content part: the same hash whatever the encoding. */
|
|
92
|
+
export function htmlHash(html) {
|
|
93
|
+
return createHash('sha1').update(html).digest('base64');
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
/**
|
|
97
|
+
* Send a rendered page: cache-control, an ETag, a 304 when the browser
|
|
98
|
+
* already has it, and - for a response that came from the response cache -
|
|
99
|
+
* the compressed body from the cache, made once.
|
|
100
|
+
*
|
|
101
|
+
* The ETag is the content hash, suffixed with the encoding when the body is
|
|
102
|
+
* compressed (`"<hash>-br"`): a strong validator names a representation, and
|
|
103
|
+
* the bytes differ per encoding. A browser's If-None-Match is matched on the
|
|
104
|
+
* hash whatever suffix it carries, so a visitor who first got gzip and now
|
|
105
|
+
* accepts brotli still gets their 304.
|
|
106
|
+
*
|
|
107
|
+
* Deliberately no reply.code() on the success path: the 404 handlers call
|
|
108
|
+
* reply.status(404) before sending, and forcing 200 would turn every
|
|
109
|
+
* not-found into a soft 404 that search engines index as a real page.
|
|
110
|
+
*
|
|
111
|
+
* @param {import('fastify').FastifyRequest} request
|
|
112
|
+
* @param {import('fastify').FastifyReply} reply
|
|
113
|
+
* @param {string} html
|
|
114
|
+
* @param {{cached?: boolean}} [opts] - cached: the HTML came from the response cache, so its
|
|
115
|
+
* compressed form may be kept too
|
|
116
|
+
*/
|
|
117
|
+
export async function sendHtml(request, reply, html, {cached = false} = {}) {
|
|
118
|
+
const hash = htmlHash(html);
|
|
119
|
+
const encoding = cached && Buffer.byteLength(html) >= COMPRESS_THRESHOLD
|
|
120
|
+
? pickEncoding(request.headers['accept-encoding'])
|
|
121
|
+
: null;
|
|
122
|
+
const etag = `"${hash}${encoding ? `-${encoding}` : ''}"`;
|
|
123
|
+
reply.header('cache-control', 'private, no-cache');
|
|
124
|
+
reply.header('etag', etag);
|
|
125
|
+
reply.header('vary', 'accept-encoding');
|
|
126
|
+
|
|
127
|
+
// If-None-Match may carry a list, and a weak validator's W/ prefix.
|
|
128
|
+
const seen = String(request.headers['if-none-match'] || '')
|
|
129
|
+
.split(',')
|
|
130
|
+
.map((t) => t.trim().replace(/^W\//, ''));
|
|
131
|
+
if (seen.some((t) => t === `"${hash}"` || t.startsWith(`"${hash}-`))) return reply.code(304).send();
|
|
132
|
+
|
|
133
|
+
if (!encoding) return reply.type('text/html; charset=utf-8').send(html);
|
|
134
|
+
|
|
135
|
+
const body = await cache.wrap(`enc:${encoding}:${hash}`, () => encodeHtml(html, encoding), {tags: ['encoded']});
|
|
136
|
+
reply.header('content-encoding', encoding);
|
|
137
|
+
return reply.type('text/html; charset=utf-8').send(body);
|
|
138
|
+
}
|