@drupflare/spool 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +499 -43
- package/dist/cdn/client.js +1 -1
- package/dist/cdn/manifest.json +7 -7
- package/dist/cdn/spool-sw.js +1 -1
- package/dist/client/activity.d.ts +37 -0
- package/dist/client/activity.js +114 -0
- package/dist/client/entry.js +34 -1
- package/dist/client/index.d.ts +1 -0
- package/dist/client/index.js +1 -0
- package/dist/client/register.d.ts +23 -1
- package/dist/client/register.js +53 -7
- package/dist/media/embeds.d.ts +3 -2
- package/dist/media/embeds.js +16 -1
- package/dist/protocol/config.d.ts +16 -1
- package/dist/protocol/config.js +3 -1
- package/dist/protocol/dataset.d.ts +183 -0
- package/dist/protocol/dataset.js +206 -0
- package/dist/protocol/envelope.d.ts +3 -0
- package/dist/protocol/envelope.js +14 -3
- package/dist/protocol/errors.d.ts +1 -1
- package/dist/protocol/graph.d.ts +23 -4
- package/dist/protocol/graph.js +34 -3
- package/dist/protocol/hash.d.ts +2 -0
- package/dist/protocol/hash.js +8 -0
- package/dist/protocol/index.d.ts +7 -0
- package/dist/protocol/index.js +7 -0
- package/dist/protocol/inline.d.ts +14 -0
- package/dist/protocol/inline.js +43 -0
- package/dist/protocol/navigation.d.ts +81 -0
- package/dist/protocol/navigation.js +249 -0
- package/dist/protocol/normalise.d.ts +4 -3
- package/dist/protocol/normalise.js +6 -4
- package/dist/protocol/plan.d.ts +100 -0
- package/dist/protocol/plan.js +661 -0
- package/dist/protocol/provision.d.ts +52 -1
- package/dist/protocol/provision.js +53 -5
- package/dist/protocol/query.d.ts +141 -0
- package/dist/protocol/query.js +252 -0
- package/dist/protocol/route.d.ts +85 -5
- package/dist/protocol/route.js +91 -0
- package/dist/protocol/write.d.ts +146 -0
- package/dist/protocol/write.js +101 -0
- package/dist/proxy/bases.d.ts +53 -0
- package/dist/proxy/bases.js +76 -0
- package/dist/proxy/index.d.ts +1 -0
- package/dist/proxy/index.js +1 -0
- package/dist/proxy/proxy.js +8 -2
- package/dist/report/capacity.d.ts +59 -0
- package/dist/report/capacity.js +76 -0
- package/dist/report/channels.d.ts +60 -0
- package/dist/report/channels.js +65 -0
- package/dist/report/dataset.d.ts +48 -0
- package/dist/report/dataset.js +93 -0
- package/dist/report/derive.d.ts +285 -0
- package/dist/report/derive.js +676 -1
- package/dist/report/economics.d.ts +11 -1
- package/dist/report/economics.js +19 -1
- package/dist/report/index.d.ts +5 -0
- package/dist/report/index.js +5 -0
- package/dist/report/inputs.d.ts +97 -0
- package/dist/report/inputs.js +133 -0
- package/dist/report/institution.d.ts +284 -0
- package/dist/report/institution.js +225 -0
- package/dist/report/ledger.d.ts +28 -0
- package/dist/report/ledger.js +30 -0
- package/dist/report/loops.d.ts +73 -0
- package/dist/report/loops.js +101 -0
- package/dist/report/measurement.d.ts +1 -1
- package/dist/report/measurement.js +6 -1
- package/dist/report/model.d.ts +2 -0
- package/dist/report/model.js +4 -0
- package/dist/report/thirdparty.d.ts +57 -0
- package/dist/report/thirdparty.js +111 -0
- package/dist/report/traffic.d.ts +11 -2
- package/dist/report/traffic.js +13 -3
- package/dist/report/trim.d.ts +56 -0
- package/dist/report/trim.js +83 -1
- package/dist/server/dataset.d.ts +48 -0
- package/dist/server/dataset.js +308 -0
- package/dist/server/embedded.d.ts +3 -3
- package/dist/server/embedded.js +3 -3
- package/dist/server/index.d.ts +3 -0
- package/dist/server/index.js +3 -0
- package/dist/server/spool.d.ts +40 -0
- package/dist/server/spool.js +184 -23
- package/dist/server/write.d.ts +27 -0
- package/dist/server/write.js +127 -0
- package/dist/sw/analytics.d.ts +6 -0
- package/dist/sw/analytics.js +14 -0
- package/dist/sw/dataset-sync.d.ts +76 -0
- package/dist/sw/dataset-sync.js +149 -0
- package/dist/sw/dataset.d.ts +91 -0
- package/dist/sw/dataset.js +318 -0
- package/dist/sw/graph/index.d.ts +31 -1
- package/dist/sw/graph/index.js +57 -5
- package/dist/sw/graph/invalidate.d.ts +15 -3
- package/dist/sw/graph/invalidate.js +42 -4
- package/dist/sw/idb.js +78 -9
- package/dist/sw/partition.d.ts +6 -1
- package/dist/sw/partition.js +21 -2
- package/dist/sw/principal-dataset.d.ts +64 -0
- package/dist/sw/principal-dataset.js +254 -0
- package/dist/sw/render.d.ts +8 -1
- package/dist/sw/render.js +28 -1
- package/dist/sw/replica.d.ts +14 -3
- package/dist/sw/replica.js +46 -5
- package/dist/sw/router.js +2 -2
- package/dist/sw/runtime.d.ts +43 -2
- package/dist/sw/runtime.js +440 -29
- package/dist/sw/seal.d.ts +13 -0
- package/dist/sw/seal.js +13 -4
- package/dist/sw/static.d.ts +18 -0
- package/dist/sw/static.js +84 -1
- package/dist/sw/storage.d.ts +51 -1
- package/dist/sw/storage.js +33 -1
- package/dist/sw/write.d.ts +107 -0
- package/dist/sw/write.js +442 -0
- package/package.json +5 -2
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# 🧵 spool
|
|
2
2
|
|
|
3
|
-
>
|
|
3
|
+
> A browser replica that removes repeated demand from your server
|
|
4
4
|
|
|
5
5
|
[](https://github.com/drupflare/spool/actions/workflows/build.yml)
|
|
6
6
|
[](https://github.com/drupflare/spool/actions/workflows/browser.yml)
|
|
@@ -8,9 +8,16 @@
|
|
|
8
8
|
[](https://codecov.io/gh/drupflare/spool)
|
|
9
9
|
[](LICENSE)
|
|
10
10
|
|
|
11
|
-
spool keeps a replica of what a page needs
|
|
12
|
-
|
|
13
|
-
|
|
11
|
+
spool removes repeated execution demand from your server. It keeps a replica of what a page needs
|
|
12
|
+
on the visitor's device. The server sends templates and authorised, versioned data, and a service
|
|
13
|
+
worker computes eligible pages from them. A page answered there is a request that never reaches
|
|
14
|
+
your Worker or origin, and server work that never runs. The contract is a count: a test asserts
|
|
15
|
+
the server hits of a local view, and in Chromium and Firefox the count is zero.
|
|
16
|
+
|
|
17
|
+
spool has three uses. In a drupflare deployment it takes eligible public views off the Worker.
|
|
18
|
+
For a team on a free or constrained plan it lowers the requests the plan has to carry, by an
|
|
19
|
+
amount that depends on the site's traffic ([Request Arithmetic](#request-arithmetic)). It is also
|
|
20
|
+
a research system: every figure it reports carries a layer and a method.
|
|
14
21
|
|
|
15
22
|
It runs on any runtime that speaks `Request` and `Response`, bridges `node:http` for the rest, and
|
|
16
23
|
has no runtime dependencies. A site with no server code can load it from two files and a manifest.
|
|
@@ -28,7 +35,9 @@ has no runtime dependencies. A site with no server code can load it from two fil
|
|
|
28
35
|
- [Static Sites](#-static-sites)
|
|
29
36
|
- [Strategies](#-strategies)
|
|
30
37
|
- [Freshness Contracts](#-freshness-contracts)
|
|
38
|
+
- [Datasets](#-datasets)
|
|
31
39
|
- [Authenticated Data](#-authenticated-data)
|
|
40
|
+
- [Declared Writes](#-declared-writes)
|
|
32
41
|
- [Catalogue](#-catalogue)
|
|
33
42
|
- [Analytics Batching](#-analytics-batching)
|
|
34
43
|
- [Third-Party Files](#-third-party-files)
|
|
@@ -37,10 +46,15 @@ has no runtime dependencies. A site with no server code can load it from two fil
|
|
|
37
46
|
- [Proxy Mode](#-proxy-mode)
|
|
38
47
|
- [Trimmed Stylesheets](#-trimmed-stylesheets)
|
|
39
48
|
- [Images and Video](#-images-and-video)
|
|
49
|
+
- [Capacity Reports](#-capacity-reports)
|
|
50
|
+
- [Security](#-security)
|
|
51
|
+
- [Energy Reporting](#-energy-reporting)
|
|
52
|
+
- [Prior Art](#-prior-art)
|
|
40
53
|
- [Out of Scope](#-out-of-scope)
|
|
41
54
|
- [Browser Support](#-browser-support)
|
|
42
55
|
- [Subpath Exports](#-subpath-exports)
|
|
43
56
|
- [Testing](#-testing)
|
|
57
|
+
- [Documentation](#-documentation)
|
|
44
58
|
- [Related Repositories](#-related-repositories)
|
|
45
59
|
- [License](#-license)
|
|
46
60
|
|
|
@@ -48,16 +62,25 @@ has no runtime dependencies. A site with no server code can load it from two fil
|
|
|
48
62
|
|
|
49
63
|
## 🎯 Why Spool
|
|
50
64
|
|
|
65
|
+
A local view removes, in order, a request that would have reached the Worker, the server work
|
|
66
|
+
behind it (rendering, queries, Durable Object and database reads), and the bytes of a document the
|
|
67
|
+
device already holds. Energy follows from those and is one objective an administrator may weigh.
|
|
68
|
+
|
|
51
69
|
A response is a deterministic transformation of a set of authorised, versioned dependencies. The
|
|
52
70
|
server half names those dependencies and sends the snapshot the visitor may hold. The worker keeps
|
|
53
71
|
a graph of records, projections and responses, with their versions, and syncs only what the next
|
|
54
72
|
page needs.
|
|
55
73
|
|
|
56
|
-
Your application keeps every
|
|
57
|
-
|
|
58
|
-
changes no page that was not
|
|
74
|
+
Your application keeps every access decision and every page it cannot reproduce safely on the
|
|
75
|
+
device. It keeps every write too, except the last-value writes you declare, which ship off. A route
|
|
76
|
+
the worker cannot answer goes to the network, so adopting spool changes no page that was not
|
|
77
|
+
declared.
|
|
59
78
|
|
|
60
|
-
|
|
79
|
+
A collection of thousands of rows, such as a directory or a course catalogue, can be held as a
|
|
80
|
+
[dataset](#-datasets). The device seeds it once and then receives only the rows that changed.
|
|
81
|
+
|
|
82
|
+
[docs/impact.md](docs/impact.md) holds the request, cost and energy figures and the measurements
|
|
83
|
+
behind them. See [Energy Reporting](#-energy-reporting).
|
|
61
84
|
|
|
62
85
|
---
|
|
63
86
|
|
|
@@ -192,8 +215,44 @@ it against the hash, so a changed file never runs. Release files are never overw
|
|
|
192
215
|
script fails to load, the page works as it did before.
|
|
193
216
|
|
|
194
217
|
The script registers `/spool-sw.js` and exposes `window.spool.ready`, which resolves to the client
|
|
195
|
-
handle or to `null` where service workers do not exist.
|
|
196
|
-
|
|
218
|
+
handle or to `null` where service workers do not exist. It also exposes `window.spool.activity`,
|
|
219
|
+
described below, and the write calls of [Declared Writes](#-declared-writes). To bundle the runtime
|
|
220
|
+
yourself, import `register` from `@drupflare/spool/client`.
|
|
221
|
+
|
|
222
|
+
### Activity Hooks
|
|
223
|
+
|
|
224
|
+
A loop that keeps running while nobody can see its output costs the visitor's device power. The
|
|
225
|
+
page runtime exposes `window.spool.activity.register`, which lets a page hand spool a `pause` and
|
|
226
|
+
a `resume` for one of its own loops. Spool calls `pause` while the loop's `target` is off screen
|
|
227
|
+
and after the page has been hidden for `hiddenAfterMs`, and calls `resume` when neither holds.
|
|
228
|
+
|
|
229
|
+
```ts
|
|
230
|
+
import { createActivity } from '@drupflare/spool/client';
|
|
231
|
+
|
|
232
|
+
const canvas = document.querySelector('canvas') as HTMLCanvasElement;
|
|
233
|
+
let frame = 0;
|
|
234
|
+
|
|
235
|
+
const draw = () => {
|
|
236
|
+
canvas.style.transform = `translateX(${(performance.now() / 10) % 200}px)`;
|
|
237
|
+
frame = requestAnimationFrame(draw);
|
|
238
|
+
};
|
|
239
|
+
|
|
240
|
+
draw();
|
|
241
|
+
|
|
242
|
+
export const stop = createActivity().register(
|
|
243
|
+
{ pause: () => cancelAnimationFrame(frame), resume: draw },
|
|
244
|
+
{ target: canvas, hiddenAfterMs: 10_000 }
|
|
245
|
+
);
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
`target` is optional, and without it only the hidden-page rule applies. `hiddenAfterMs` defaults to
|
|
249
|
+
10000, and 0 pauses at once. `register` returns a function that stops watching and resumes the loop
|
|
250
|
+
if it is paused. Spool does not touch `setInterval`, `requestAnimationFrame`, `fetch`, audio,
|
|
251
|
+
WebRTC or any element the page did not name, and a `pause` or `resume` that throws is logged once.
|
|
252
|
+
Where `IntersectionObserver` is missing the target counts as on screen. On the one desktop
|
|
253
|
+
measured, the off-screen pause lowered power for a page whose frame loop kept running, and the
|
|
254
|
+
hidden-page pause showed no saving for a once-a-second poll. Both readings are in
|
|
255
|
+
[docs/impact.md](docs/impact.md#metered-figures-beside-the-ledger).
|
|
197
256
|
|
|
198
257
|
---
|
|
199
258
|
|
|
@@ -708,7 +767,7 @@ from one directory:
|
|
|
708
767
|
- `client.js`, the page script, from the same place.
|
|
709
768
|
- `spool.json`, the list of files the worker may keep.
|
|
710
769
|
|
|
711
|
-
Each page loads the script with the tag from [CDN Script](
|
|
770
|
+
Each page loads the script with the tag from [CDN Script](#cdn-script), or from the site's own copy.
|
|
712
771
|
The worker has no server to talk to, so it reads `spool.json` from the directory its own script
|
|
713
772
|
sits in.
|
|
714
773
|
|
|
@@ -862,6 +921,13 @@ export const news = defineRoute('/news/:slug', {
|
|
|
862
921
|
|
|
863
922
|
A template that names a field it does not trust for markup fails when the route is declared.
|
|
864
923
|
|
|
924
|
+
A public `template` route can carry its own data in the page. `inlineMaxBytes` on `createSpool`
|
|
925
|
+
sets the largest artifact, in bytes, that `handle` and `decorate` write into the HTML as an
|
|
926
|
+
escaped `<script type="application/spool+json">` element. The worker reads it from the response
|
|
927
|
+
and skips the `/.spool/artifact` request. It is 0, which is off, because every network view of the
|
|
928
|
+
page carries the element, used or not. The page the visitor sees and the shadow comparison ignore
|
|
929
|
+
it.
|
|
930
|
+
|
|
865
931
|
### Static Files
|
|
866
932
|
|
|
867
933
|
A `static` route has no `load` and no template. The application serves the file as it always did,
|
|
@@ -930,9 +996,79 @@ records included, against what a local view saves: reuse times future views time
|
|
|
930
996
|
avoids, less the cost of the request. When that is not positive it answers `204` with
|
|
931
997
|
`Spool-Provision: declined`, and the worker leaves that path to the network until it restarts.
|
|
932
998
|
The priors are scenario assumptions: set them from your own telemetry in `provision`, or per route
|
|
933
|
-
as above. A `data` route weighs against its own size, so
|
|
999
|
+
as above. A `data` route weighs against its own size, so its reuse times future views must exceed
|
|
1000
|
+
one. The defaults of 0.5 and 2 give exactly one, which breaks even, and the route is left out.
|
|
1001
|
+
|
|
1002
|
+
### Query Routes
|
|
934
1003
|
|
|
935
|
-
|
|
1004
|
+
A URL with a query string goes to the network. A public `template`, `projection` or `data` route
|
|
1005
|
+
can declare the query strings it answers, and the device then answers a new filter, sort or page of
|
|
1006
|
+
a collection it already holds with no request.
|
|
1007
|
+
|
|
1008
|
+
```ts
|
|
1009
|
+
import { datasetProof, defineRoute } from '@drupflare/spool/server';
|
|
1010
|
+
|
|
1011
|
+
const people = [
|
|
1012
|
+
{ name: 'Ada', department: 'math', age: 36 },
|
|
1013
|
+
{ name: 'Bo', department: 'cs', age: 25 }
|
|
1014
|
+
];
|
|
1015
|
+
|
|
1016
|
+
export const directory = defineRoute('/people', {
|
|
1017
|
+
strategy: 'template',
|
|
1018
|
+
template: {
|
|
1019
|
+
source:
|
|
1020
|
+
'<h1>{{#if query}}Page {{query.page}} of {{query.pages}}{{else}}Everyone{{/if}}</h1><ul>{{#each people}}<li>{{name}}</li>{{/each}}</ul>'
|
|
1021
|
+
},
|
|
1022
|
+
depends: () => ['people'],
|
|
1023
|
+
load: async () => ({ people, proof: await datasetProof(people) }),
|
|
1024
|
+
query: {
|
|
1025
|
+
list: 'people',
|
|
1026
|
+
filter: { department: 'string', age: 'number' },
|
|
1027
|
+
sort: ['name', 'age'],
|
|
1028
|
+
pageSize: 20
|
|
1029
|
+
},
|
|
1030
|
+
provision: { reuse: 0.8, futureViews: 6 }
|
|
1031
|
+
});
|
|
1032
|
+
```
|
|
1033
|
+
|
|
1034
|
+
`list` names the key of the loaded data that holds the whole collection, an array of flat objects.
|
|
1035
|
+
Each key of `filter` is a query parameter that keeps the items whose field of the same name equals
|
|
1036
|
+
it, read as a `string` or a `number`. A repeated parameter matches any of its values, and different
|
|
1037
|
+
parameters combine with and. `sort` lists the fields the `sort` parameter may name, and `-name`
|
|
1038
|
+
sorts descending. Ties keep the stored order. Strings compare by UTF-16 code unit, with no locale
|
|
1039
|
+
and no case folding, so the application's own sort has to agree. A database that compares bytes,
|
|
1040
|
+
such as SQLite by default, agrees except for characters above U+FFFF, and a case-insensitive or
|
|
1041
|
+
locale collation does not. `pageSize` sets the items per page and `page` counts from 1; without it
|
|
1042
|
+
the route is one page of everything. A route with `static` or `assembly` strategy cannot declare
|
|
1043
|
+
`query`. A route with `principal` scope can declare it only with `query.dataset`, which
|
|
1044
|
+
[Datasets](#-datasets) describes.
|
|
1045
|
+
|
|
1046
|
+
`load` returns the array at `list` and a `proof` made by `datasetProof`. The server refuses to
|
|
1047
|
+
build the artifact when the proof is missing or does not match the array. The worker computes the
|
|
1048
|
+
proof again from the stored copy before it answers a query. The proof shows that the device holds
|
|
1049
|
+
the array the server sent. That the array is the whole collection is the route's declaration:
|
|
1050
|
+
`list` has to hold every item the application's own answer could draw from, or a filter returns
|
|
1051
|
+
fewer rows than the server would.
|
|
1052
|
+
|
|
1053
|
+
The template, or the renderer of a `projection` route, receives the stored data with the array
|
|
1054
|
+
replaced by the page's items and one added key, `query`, holding `{ total, page, pages }`. `total`
|
|
1055
|
+
counts the items after filtering. A `data` route answers with that same object as JSON. The URL
|
|
1056
|
+
without a query string renders the stored data as it is and has no `query` key, which is what
|
|
1057
|
+
`{{#if query}}` tests above.
|
|
1058
|
+
|
|
1059
|
+
The worker answers locally only when the artifact for the path is held and fresh under the route's
|
|
1060
|
+
freshness contract, the proof matches and the query is one the route declares. Everything else goes
|
|
1061
|
+
to the network as before: a parameter the route does not name, a number filter that is not a
|
|
1062
|
+
finite number, a `sort` given twice or not listed, a `page` that is not a positive integer or
|
|
1063
|
+
lies past the last page, and an item that is not an object. A copy that fails its proof is dropped
|
|
1064
|
+
and fetched whole again. The worker never stores anything under a URL with a query string, and in
|
|
1065
|
+
shadow mode it compares its render with the network answer for the same URL.
|
|
1066
|
+
|
|
1067
|
+
A query route's artifact is the whole collection, so it is large. The provisioning rule weighs it
|
|
1068
|
+
like any other artifact, and `provision` on the route sets the priors. A collection repays its
|
|
1069
|
+
transfer after `entryBreakEven(bytes, documentBytes)` local answers per device: a 40 KB collection
|
|
1070
|
+
that replaces 17 KB documents needs 2.3 (L2). Pages of one visit come from one held copy until a
|
|
1071
|
+
sync replaces it, so paging never mixes two versions on a device.
|
|
936
1072
|
|
|
937
1073
|
---
|
|
938
1074
|
|
|
@@ -978,6 +1114,72 @@ invalidations, so a sync does not always need a request of its own.
|
|
|
978
1114
|
|
|
979
1115
|
---
|
|
980
1116
|
|
|
1117
|
+
## 💾 Datasets
|
|
1118
|
+
|
|
1119
|
+
A route that lists thousands of rows, such as a staff directory, makes a device fetch the whole
|
|
1120
|
+
collection again after each change. A dataset lets the device hold the rows and receive only the
|
|
1121
|
+
rows that changed. The device seeds a dataset once, then asks for a patch from the cursor it holds.
|
|
1122
|
+
A patch carries the rows to put, the keys to delete and a proof the device checks before it stores
|
|
1123
|
+
anything.
|
|
1124
|
+
|
|
1125
|
+
`defineDataset(name, declaration)` from `@drupflare/spool/server` declares a dataset, and
|
|
1126
|
+
`createSpool({ datasets })` serves it at `GET /.spool/dataset`. The declaration gives the
|
|
1127
|
+
functions `key`, `seed`, `read`, `changes`, `horizon` and `head`, a `schema` string, and the
|
|
1128
|
+
`maxRows` and `maxBytes` of a seed. A dataset has its own integer cursor, which your backend holds.
|
|
1129
|
+
It does not use the site's version vector. `changes(since)` returns the net change after a cursor
|
|
1130
|
+
and decides which rows are in scope. `read(ids)` is a keyed read that never runs a route's `load`.
|
|
1131
|
+
A device whose cursor is below `horizon`, or whose `schema` differs, seeds again.
|
|
1132
|
+
|
|
1133
|
+
The proof is the number of rows and the sum of every row's hash. A sum does not depend on order, so
|
|
1134
|
+
a patch moves it by the rows it touches. The device applies a seed or a patch to a staging copy,
|
|
1135
|
+
compares the result with the proof, and writes the rows and their metadata in one transaction. A
|
|
1136
|
+
failed proof clears that dataset and seeds it once more. The proof catches corruption and a
|
|
1137
|
+
half-applied patch. It does not defend against a server that sends rows and a digest that agree.
|
|
1138
|
+
|
|
1139
|
+
A route reads a dataset in two ways. `query.dataset` names it in place of `list`: the worker takes
|
|
1140
|
+
the held rows in key order and applies the route's filter, sort and page, and goes to the network
|
|
1141
|
+
while the dataset is not held. `ctx.row(dataset, key)` puts a reference to one row in the route's
|
|
1142
|
+
data. The reference resolves when the page renders, so a changed row needs no new artifact.
|
|
1143
|
+
|
|
1144
|
+
A route with `query.dataset` may also declare `query.plan`, a read-only plan written as JSON with
|
|
1145
|
+
joins, conditions, groups and aggregates over held datasets. It replaces `filter`, `sort` and
|
|
1146
|
+
`pageSize`. The worker compares values the way SQLite compares a column with no declared type, and
|
|
1147
|
+
sends the query to the network when a plan could give another answer than SQLite would. The server
|
|
1148
|
+
never evaluates a plan.
|
|
1149
|
+
|
|
1150
|
+
A dataset declared with `provision: 'auto'` reaches a device only while it beats two controls: the
|
|
1151
|
+
held page under v0.3, and one plain resource that a browser cache revalidates with an `ETag`.
|
|
1152
|
+
Otherwise the seed request gets `409` with `Spool-Provision: declined`, and the worker holds
|
|
1153
|
+
nothing and does not ask again in that session. The default, `'always'`, answers every seed. The worker also
|
|
1154
|
+
refuses a seed whose rows would not fit the headroom that `navigator.storage.estimate()` reports,
|
|
1155
|
+
and the page runtime asks the browser for persistent storage once the first dataset is held.
|
|
1156
|
+
|
|
1157
|
+
On synthetic rows (L1, no real site measured), one changed row of 500 costs 228 bytes as a patch and
|
|
1158
|
+
19,981 bytes as a v0.3 list refetch, and the first load costs 867 bytes more than under v0.3
|
|
1159
|
+
([docs/impact.md](docs/impact.md#dataset-patches)). [docs/protocol.md](docs/protocol.md#datasets)
|
|
1160
|
+
has the declaration rules, the wire format, query plans and the device store.
|
|
1161
|
+
|
|
1162
|
+
### Principal Datasets
|
|
1163
|
+
|
|
1164
|
+
Principal datasets are off by default and unreviewed: no external review of this feature has
|
|
1165
|
+
happened. They hold rows that differ for each signed-in principal. Two switches turn them on, and
|
|
1166
|
+
both are needed. `defineDataset(name, declaration, { principalDatasets: true })` accepts
|
|
1167
|
+
`scope: 'principal'`, and `createSpool({ principalDatasets: true, identity })` serves the dataset.
|
|
1168
|
+
A principal dataset declares `maxStaleMs`, which has no default, and cannot use
|
|
1169
|
+
`provision: 'auto'`.
|
|
1170
|
+
|
|
1171
|
+
The server takes the identity from the request and passes it to every dataset function as
|
|
1172
|
+
`ctx.principal`. Which rows a principal may see is your code, and spool does not check what the
|
|
1173
|
+
functions return. The device keeps the rows sealed in the principal's partition, as
|
|
1174
|
+
[Authenticated Data](#-authenticated-data) describes, and wipes them with it. Held rows answer a
|
|
1175
|
+
query for `maxStaleMs` after the last verified sync, so a principal removed on the server can read
|
|
1176
|
+
held rows for up to that long. A principal route can declare `query` only with `query.dataset` and
|
|
1177
|
+
no plan, and a route and its dataset need the same scope.
|
|
1178
|
+
[docs/security.md](docs/security.md#principal-datasets) lists the threat modes and the test that
|
|
1179
|
+
holds each one.
|
|
1180
|
+
|
|
1181
|
+
---
|
|
1182
|
+
|
|
981
1183
|
## 🔐 Authenticated Data
|
|
982
1184
|
|
|
983
1185
|
Set `scope: 'principal'` on a route that depends on who is signed in, and give `createSpool` an
|
|
@@ -1055,6 +1257,38 @@ that page reads.
|
|
|
1055
1257
|
|
|
1056
1258
|
---
|
|
1057
1259
|
|
|
1260
|
+
## 📝 Declared Writes
|
|
1261
|
+
|
|
1262
|
+
Declared writes are off by default and unreviewed: no external review of this feature has
|
|
1263
|
+
happened. A declared write queues one kind of edit on the device and sends it at a network moment.
|
|
1264
|
+
The last value for a key wins. `createSpool({ identity, writes: { enabled: true, classes } })`
|
|
1265
|
+
turns them on. With the flag off the worker queues nothing, the page calls answer `writes-off`, and
|
|
1266
|
+
`PUT /.spool/write` answers `403`.
|
|
1267
|
+
|
|
1268
|
+
`defineWrite` from `@drupflare/spool/server` declares a class. Its `scope` has to be `principal`,
|
|
1269
|
+
and its `path` a declared principal route. `key` says which key this principal may write, and
|
|
1270
|
+
`read` gives that key's current version. `apply` writes the value only when the key is still at the
|
|
1271
|
+
version the device edited from, in one atomic step, and returns the new version or the conflict.
|
|
1272
|
+
`maxBytes` and `maxPending` bound one value and the queue. The only `kind` is `last-value`, so a
|
|
1273
|
+
declaration cannot queue a comment, a cart, a payment, a counter, an inventory change or an audit
|
|
1274
|
+
record.
|
|
1275
|
+
|
|
1276
|
+
The page calls `window.spool.write(name, key, value, version)`, `window.spool.flush()`,
|
|
1277
|
+
`window.spool.writes()` and `window.spool.resolve(name, key, 'local' | 'server')`. Each answers
|
|
1278
|
+
`{ ok: true }` with its result or `{ ok: false, code, message }`, and none throws. Edits of one key
|
|
1279
|
+
replace each other, so several edits cost one request. A flush sends `PUT /.spool/write` with the
|
|
1280
|
+
version in `If-Match` and a `Spool-Write-Id` that stays the same across retries. The page runtime
|
|
1281
|
+
flushes when the page is hidden. A stale version gets `412`, and the record waits as `conflicted`
|
|
1282
|
+
until `resolve` settles it.
|
|
1283
|
+
|
|
1284
|
+
The queue is sealed in the principal's partition and deleted with it. A value that no flush has
|
|
1285
|
+
sent is lost at logout, so call `window.spool.flush()` before the logout navigation. The server
|
|
1286
|
+
remembers recent write ids in memory, 10,000 by default, so a site with several instances stores
|
|
1287
|
+
`ctx.writeId` with the value in `apply`. [docs/protocol.md](docs/protocol.md#declared-writes) has
|
|
1288
|
+
the request and its answers, and [docs/security.md](docs/security.md#declared-writes) the limits.
|
|
1289
|
+
|
|
1290
|
+
---
|
|
1291
|
+
|
|
1058
1292
|
## 📚 Catalogue
|
|
1059
1293
|
|
|
1060
1294
|
A catalogue sends the data of pages a visitor may open next, so that opening one makes no
|
|
@@ -1072,9 +1306,20 @@ is larger. An entry is data for a `template`, `projection` or `data` route and n
|
|
|
1072
1306
|
The site supplies the candidates. A route's `catalogue` returns the pages a visitor on it may open
|
|
1073
1307
|
next, each with the share expected to. Take the share from your own `nextShare` over the sink
|
|
1074
1308
|
(each `spool.view` names the route the visitor came from), from link position, or from a fixed
|
|
1075
|
-
prior. `
|
|
1309
|
+
prior. Pass `{ first: true }` to `nextShare` to count only first views, which is what an entry
|
|
1310
|
+
saves: the share over all views ran 1.5 to 6.6 times high in the bins that decide an entry in a replay
|
|
1311
|
+
of public visitors ([docs/impact.md](docs/impact.md#next-page-prediction)). `documentBytes` on the entry's route says what a hit saves, and the provisioning prior is
|
|
1076
1312
|
used when it is absent.
|
|
1077
1313
|
|
|
1314
|
+
A share taken from counts can carry `n`, the observations behind it. `catalogueConfidence` on
|
|
1315
|
+
`createSpool` is a number of standard deviations, and with it above 0 an entry whose guess has an
|
|
1316
|
+
`n` is judged on the lower confidence bound of its `p`, so a share seen on 5 visits stops counting
|
|
1317
|
+
as one seen on 5,000. It defaults to 0, which judges every guess on its `p`. `NavModel` in
|
|
1318
|
+
`@drupflare/spool/protocol` keeps first-order counts with decay and a prior and returns a `p` with
|
|
1319
|
+
the weight to pass as `n`. In a replay of 989,818 public visitors, a model like that earned more
|
|
1320
|
+
per provisioned kilobyte than the most popular page
|
|
1321
|
+
([docs/impact.md](docs/impact.md#next-page-prediction)).
|
|
1322
|
+
|
|
1078
1323
|
```ts
|
|
1079
1324
|
import { createSpool, defineRoute } from '@drupflare/spool/server';
|
|
1080
1325
|
import { memoryStore } from '@drupflare/spool/stores';
|
|
@@ -1260,13 +1505,10 @@ that. `passthrough` leaves assets to the browser's cache and never touches them.
|
|
|
1260
1505
|
asks the browser through `InstallEvent.addRoutes` to send only navigations, beacons and routed
|
|
1261
1506
|
paths to the worker and the rest of the origin to the network, and acts as `passthrough` where the
|
|
1262
1507
|
API does not exist: Chromium has it, Playwright's WebKit has it on macOS and not on Linux, and
|
|
1263
|
-
Firefox does not.
|
|
1264
|
-
|
|
1265
|
-
|
|
1266
|
-
|
|
1267
|
-
and Static Routing (8 rounds each), and the ranges overlap. Under Static Routing a same-origin write no
|
|
1268
|
-
longer reaches the worker, so it is not a network moment and its response cannot carry
|
|
1269
|
-
invalidations; navigations still are.
|
|
1508
|
+
Firefox does not. Static Routing keeps the requests it does not route out of the worker's fetch
|
|
1509
|
+
handler, and neither option changes server-side Worker invocations. Neither is measured to save
|
|
1510
|
+
energy, and [docs/impact.md](docs/impact.md) holds no reading for them. Under Static Routing a same-origin write no longer reaches the worker,
|
|
1511
|
+
so it is not a network moment and its response cannot carry invalidations; navigations still are.
|
|
1270
1512
|
|
|
1271
1513
|
```ts
|
|
1272
1514
|
import { createSpool } from '@drupflare/spool/server';
|
|
@@ -1312,7 +1554,11 @@ requests keep the browser's own path.
|
|
|
1312
1554
|
or anything else that speaks HTTP. It returns a Fetch handler. The proxy answers `/.spool/*` and
|
|
1313
1555
|
`/spool-sw.js`. Every other request goes to the origin with its method, headers and body, and a
|
|
1314
1556
|
redirect is passed on, not followed. A page the proxy may keep is decorated like a `createSpool`
|
|
1315
|
-
response, so the worker stores it and answers later views with no request.
|
|
1557
|
+
response, so the worker stores it and answers later views with no request. A cold visit costs one
|
|
1558
|
+
origin fetch: the proxy names a hash of the page in `Spool-Artifact`, and the worker keeps the
|
|
1559
|
+
page it already received as the artifact when its own hash matches. On any mismatch it asks
|
|
1560
|
+
`/.spool/artifact` as before. In the first-payload reading (L1, a synthetic 11,145-byte page) a
|
|
1561
|
+
cold visit made 1 request and 1 origin fetch where it made 2 and 2 before.
|
|
1316
1562
|
|
|
1317
1563
|
`spool` takes the options `createSpool` takes, minus `routes`. The proxy adds one public
|
|
1318
1564
|
`assembly` route that matches every path (`/**`) and uses the origin's finished HTML as the
|
|
@@ -1320,7 +1566,8 @@ artifact. `routes` lists overrides, such as a shorter freshness bound for one pa
|
|
|
1320
1566
|
match first and take the same fields as `defineRoute`, without `load`.
|
|
1321
1567
|
|
|
1322
1568
|
On Node the handler goes through `toNodeListener`. `origin` is the address of the real server
|
|
1323
|
-
and has no path.
|
|
1569
|
+
and has no path. `fetch` replaces the global `fetch` the proxy calls the origin with, such as a
|
|
1570
|
+
Workers service binding.
|
|
1324
1571
|
|
|
1325
1572
|
```ts
|
|
1326
1573
|
import { createServer } from 'node:http';
|
|
@@ -1420,7 +1667,8 @@ in the instance: a restart forgets them, and each instance of a scaled-out proxy
|
|
|
1420
1667
|
A whole page is a large artifact, and the provisioning rule declines one that costs more than the
|
|
1421
1668
|
priors say it saves, so set `provision` on the spool for large pages. A `Content-Security-Policy`
|
|
1422
1669
|
that allows only nonce or hash scripts blocks the injected client tag. A URL with a query string
|
|
1423
|
-
always goes to the origin
|
|
1670
|
+
always goes to the origin, because the proxy holds finished HTML and has no collection to filter;
|
|
1671
|
+
[Query Routes](#query-routes) apply to `createSpool` only. [Trimmed Stylesheets](#-trimmed-stylesheets) covers the `trim` option
|
|
1424
1672
|
and [Images and Video](#-images-and-video) the `images` and `video` options.
|
|
1425
1673
|
|
|
1426
1674
|
---
|
|
@@ -1600,8 +1848,11 @@ and any tag inside `<picture>`, stays as it is.
|
|
|
1600
1848
|
`embeds` replaces a server-rendered YouTube or Vimeo `<iframe>` with one whose `srcdoc` holds a
|
|
1601
1849
|
link to the player. YouTube shows its thumbnail (`thumbnail: false` shows a plain frame) and the
|
|
1602
1850
|
player loads on click. A page that sends `Content-Security-Policy` is left alone unless
|
|
1603
|
-
`ignoreCsp` is true, and so is an embed with `enablejsapi=1`.
|
|
1604
|
-
`
|
|
1851
|
+
`ignoreCsp` is true, and so is an embed with `enablejsapi=1`. An embed that plays on its own by
|
|
1852
|
+
design stays too: a URL with `background=1`, or with `autoplay=1` beside `mute`, `muted` or `loop`,
|
|
1853
|
+
and an iframe whose `class` or `id` names a hero, ambient, background or banner. In a crawl of the
|
|
1854
|
+
server-rendered pages of 52 sites (L1, `measurements/embed-markup`), 2 of 4 Vimeo iframes were
|
|
1855
|
+
background loops. `proxy.embeds` counts `pages`, `rewritten` and `skipped` by reason. [docs/security.md](docs/security.md#embed-facades) covers the rules.
|
|
1605
1856
|
|
|
1606
1857
|
### Dictionary Transport
|
|
1607
1858
|
|
|
@@ -1641,7 +1892,62 @@ expression or a list of them. Without it the browser holds the latest file of ea
|
|
|
1641
1892
|
fetched, and the sizing in [docs/impact.md](docs/impact.md#savings-today-and-at-maximum) shows that
|
|
1642
1893
|
choice recovers little. With it the browser holds only the files you name, one per directory, and
|
|
1643
1894
|
every other file of the directory is encoded against the base the request names.
|
|
1644
|
-
|
|
1895
|
+
|
|
1896
|
+
`chooseBases` picks the bases from a crawl. It takes the site's stylesheets and scripts with their
|
|
1897
|
+
compressed sizes and a `delta` function that returns the size of one file compressed against
|
|
1898
|
+
another. For each directory and file kind it returns the file whose deltas against the others, each
|
|
1899
|
+
capped at the other file's plain size, add up smallest. That `held` policy scores a base for a
|
|
1900
|
+
visitor who already holds it. Pass `{ policy: 'first-visit' }` to add the base's own plain size
|
|
1901
|
+
to the sum, for visitors who load the whole group before they hold anything. Neither policy
|
|
1902
|
+
weighs how often a file is requested or how long browsers keep the base, so neither is a proven
|
|
1903
|
+
minimum of total transfer. A directory with one file has no base, and a tie goes to the URL that
|
|
1904
|
+
sorts first. `baseMatcher` turns the choices into a `base` function.
|
|
1905
|
+
|
|
1906
|
+
```ts
|
|
1907
|
+
import { promisify } from 'node:util';
|
|
1908
|
+
import { brotliCompressSync, zstdCompress } from 'node:zlib';
|
|
1909
|
+
import { baseMatcher, chooseBases, spoolProxy } from '@drupflare/spool/proxy';
|
|
1910
|
+
import { memoryStore } from '@drupflare/spool/stores';
|
|
1911
|
+
|
|
1912
|
+
const compress = promisify(zstdCompress);
|
|
1913
|
+
|
|
1914
|
+
const crawl = [
|
|
1915
|
+
{ url: 'https://origin.example.edu/css/a.css', kind: 'style', text: '.a{color:red}.b{margin:0}' },
|
|
1916
|
+
{
|
|
1917
|
+
url: 'https://origin.example.edu/css/b.css',
|
|
1918
|
+
kind: 'style',
|
|
1919
|
+
text: '.a{color:red}.c{margin:1px}'
|
|
1920
|
+
},
|
|
1921
|
+
{
|
|
1922
|
+
url: 'https://origin.example.edu/css/c.css',
|
|
1923
|
+
kind: 'style',
|
|
1924
|
+
text: '.a{color:red}.d{margin:2px}'
|
|
1925
|
+
}
|
|
1926
|
+
] as const;
|
|
1927
|
+
|
|
1928
|
+
const bytes = (index: number) => Buffer.from(crawl[index]?.text ?? '');
|
|
1929
|
+
|
|
1930
|
+
const choices = await chooseBases(
|
|
1931
|
+
crawl.map(({ url, kind, text }) => ({ url, kind, plain: brotliCompressSync(text).length })),
|
|
1932
|
+
async (file, dictionary) =>
|
|
1933
|
+
(await compress(bytes(file), { dictionary: bytes(dictionary) })).length
|
|
1934
|
+
);
|
|
1935
|
+
|
|
1936
|
+
export const proxy = spoolProxy({
|
|
1937
|
+
origin: 'https://origin.example.edu',
|
|
1938
|
+
spool: { store: memoryStore() },
|
|
1939
|
+
dictionary: {
|
|
1940
|
+
encoder: async (dictionary, input) => new Uint8Array(await compress(input, { dictionary })),
|
|
1941
|
+
base: baseMatcher(choices)
|
|
1942
|
+
}
|
|
1943
|
+
});
|
|
1944
|
+
```
|
|
1945
|
+
|
|
1946
|
+
On the 10 Drupal sites of the reading, a base chosen this way and held from an earlier visit saves
|
|
1947
|
+
29.9% of the brotli 11 bytes of new stylesheets and scripts at the median site with `dcz`, and a
|
|
1948
|
+
visit that has to fetch the base first saves 3.9%
|
|
1949
|
+
(L1, [docs/impact.md](docs/impact.md#savings-today-and-at-maximum)).
|
|
1950
|
+
`scripts/measure/dictionary-size.ts` records the same choice for a site's crawl.
|
|
1645
1951
|
|
|
1646
1952
|
### Defaults
|
|
1647
1953
|
|
|
@@ -1666,12 +1972,149 @@ set and how to change it.
|
|
|
1666
1972
|
|
|
1667
1973
|
---
|
|
1668
1974
|
|
|
1975
|
+
## 📈 Capacity Reports
|
|
1976
|
+
|
|
1977
|
+
`@drupflare/spool/report` turns peak work and an institution's own limits into three separate
|
|
1978
|
+
results: potential, financial and physical. None is added to another, and any can be empty.
|
|
1979
|
+
|
|
1980
|
+
`capacityPlan` takes intervals of demand per resource with the part spool avoided, and what one
|
|
1981
|
+
node serves per second within the latency target. It returns the nodes needed before and after,
|
|
1982
|
+
as whole numbers, and how many could retire. `perNode.busyMs` is the milliseconds of handler work
|
|
1983
|
+
a node does per second, such as 1000 times its cores times a target utilisation. `spares`, `floor`,
|
|
1984
|
+
`regions` and `minPerRegion` raise the answer for redundancy, and a resource with no `avoided`, such
|
|
1985
|
+
as memory, does not shrink. `intervalsFromWork` builds the intervals from the `WorkSample`s the
|
|
1986
|
+
work series records.
|
|
1987
|
+
|
|
1988
|
+
`institutionReport` adds the plan, an `observed` before and after of the same workload from
|
|
1989
|
+
production telemetry, an invoice total before and after, and a ledger of retired equipment. The
|
|
1990
|
+
plan says what could change, `observed` what changed, the invoices what was paid, and the ledger
|
|
1991
|
+
what was removed. The physical result reads only the ledger, and an entry needs the evidence for
|
|
1992
|
+
its removal. With no ledger the statement reads `No physical capacity saving demonstrated.` The
|
|
1993
|
+
report also returns `economics`, `storage`, `trust` and `research` blocks. Each is null without its
|
|
1994
|
+
input: an `institution` profile for `economics` and `research`, `storage` for the held bytes, and
|
|
1995
|
+
`views` for `trust`.
|
|
1996
|
+
[docs/impact.md](docs/impact.md#an-institutions-figures) prints them for three example
|
|
1997
|
+
institutions.
|
|
1998
|
+
|
|
1999
|
+
```ts
|
|
2000
|
+
import { capacityPlan, institutionReport } from '@drupflare/spool/report';
|
|
2001
|
+
|
|
2002
|
+
const plan = capacityPlan(
|
|
2003
|
+
[
|
|
2004
|
+
{
|
|
2005
|
+
start: Date.UTC(2026, 8, 1, 14),
|
|
2006
|
+
seconds: 300,
|
|
2007
|
+
demand: { busyMs: 9000 },
|
|
2008
|
+
avoided: { busyMs: 6300 }
|
|
2009
|
+
},
|
|
2010
|
+
{
|
|
2011
|
+
start: Date.UTC(2026, 8, 1, 14, 5),
|
|
2012
|
+
seconds: 300,
|
|
2013
|
+
demand: { busyMs: 7000 },
|
|
2014
|
+
avoided: { busyMs: 4900 }
|
|
2015
|
+
}
|
|
2016
|
+
],
|
|
2017
|
+
{ perNode: { busyMs: 700 }, spares: 1, floor: 2 }
|
|
2018
|
+
);
|
|
2019
|
+
|
|
2020
|
+
export const report = institutionReport({ plan, planLayer: 'L3' });
|
|
2021
|
+
```
|
|
2022
|
+
|
|
2023
|
+
The example needs 14 nodes without spool and 5 with it, a potential of 9, at layer L3 because the
|
|
2024
|
+
numbers are made up. `channelRows` lists the metered, attributional, capacity and avoided-demand
|
|
2025
|
+
channels as separate rows, and nothing in the module sums them. [docs/impact.md](docs/impact.md#channels)
|
|
2026
|
+
says what each one counts.
|
|
2027
|
+
|
|
2028
|
+
### Request Arithmetic
|
|
2029
|
+
|
|
2030
|
+
On a plan that limits requests, the question is how many views the plan carries. Name three
|
|
2031
|
+
inputs, each a scenario (L3) until a site's traffic supplies it:
|
|
2032
|
+
|
|
2033
|
+
- `h`, the share of views answered on the device
|
|
2034
|
+
- `r`, the requests a network view makes to the platform, and `rho`, the part of them a local view
|
|
2035
|
+
removes (one for the document when the page's API calls still leave)
|
|
2036
|
+
- `s`, the requests spool sends itself per local view: syncs, beacon flushes, catalogue asks and
|
|
2037
|
+
worker script checks
|
|
2038
|
+
|
|
2039
|
+
The platform sees `r - h * rho + h * s` requests per view, so a limit `L` carries
|
|
2040
|
+
`L / (r - h * rho + h * s)` views. With `r = rho = 1` this is `freeTierCeiling` in
|
|
2041
|
+
`@drupflare/spool/report`. Three things bound `h` below 1:
|
|
2042
|
+
|
|
2043
|
+
- A device's first view of a site, a view after the browser evicted the replica, and a view past a
|
|
2044
|
+
freshness bound all reach the network. A device that makes `v` views in a freshness window
|
|
2045
|
+
answers at most `v - 1` of them locally, so `h <= 1 - 1/v` for that device.
|
|
2046
|
+
- A client that does not register the worker, such as a crawler, answers none.
|
|
2047
|
+
- A request outside every route family always leaves.
|
|
2048
|
+
|
|
2049
|
+
A limit on CPU per request, such as the 10 ms of the Workers Free plan, does not change with `h`.
|
|
2050
|
+
The docs' free-tier table uses one sync per 20 local views as an illustration. No multiplier
|
|
2051
|
+
follows from the formula until a site's `v`, its share of clients that cannot answer locally and
|
|
2052
|
+
`s` are measured.
|
|
2053
|
+
|
|
2054
|
+
---
|
|
2055
|
+
|
|
2056
|
+
## 🔒 Security
|
|
2057
|
+
|
|
2058
|
+
Authorisation stays on the server. The device holds only what the server already sent that
|
|
2059
|
+
principal, and the worker has no access rule. Principal data sits in a per-partition IndexedDB
|
|
2060
|
+
database sealed with AES-GCM, and deleting the key at logout is crypto-shredding of data at rest.
|
|
2061
|
+
That does not protect against script running on your origin. A revoked permission can still show
|
|
2062
|
+
for up to a route's `maxStale`, because nothing pushes a revocation to a device. Tag names are
|
|
2063
|
+
readable through `/.spool/sync`, so use opaque ids for anything sensitive. Principal datasets and
|
|
2064
|
+
declared writes ship off and have had no external review.
|
|
2065
|
+
|
|
2066
|
+
[docs/security.md](docs/security.md) is the threat model. It names the file and the test behind
|
|
2067
|
+
each claim and lists the open limitations in one table. Report a vulnerability as
|
|
2068
|
+
[SECURITY.md](https://github.com/drupflare/spool/blob/master/SECURITY.md) describes.
|
|
2069
|
+
|
|
2070
|
+
---
|
|
2071
|
+
|
|
2072
|
+
## 🔋 Energy Reporting
|
|
2073
|
+
|
|
2074
|
+
Energy is one objective among those spool reports, and the administrator decides how much weight
|
|
2075
|
+
it gets. The energy of one deployment is small against an institution's total, and small against
|
|
2076
|
+
the idle and hosting energy drupflare removes per site. The figures below are research figures.
|
|
2077
|
+
|
|
2078
|
+
[docs/impact.md](docs/impact.md) is generated from the readings in `measurements/` by
|
|
2079
|
+
`bun run impact`, and `bun run impact:check` fails when the page differs from them. Energy is
|
|
2080
|
+
reported in kWh per million views, and nothing sets a target intensity. Every figure carries a
|
|
2081
|
+
layer: L0 published, L1 measured here, L2 derived, L3 scenario. A per-view figure that spreads a
|
|
2082
|
+
published intensity over bytes is an allocation of that figure, and the page prints the metered
|
|
2083
|
+
reading beside it. The savings from trimming, images and dictionaries apply to network views and
|
|
2084
|
+
never add to the local-view figure.
|
|
2085
|
+
|
|
2086
|
+
[An Institution's Figures](docs/impact.md#an-institutions-figures) leads with economics, storage
|
|
2087
|
+
and trust and treats energy as a research figure. [What This Does Not
|
|
2088
|
+
Claim](docs/impact.md#what-this-does-not-claim) lists what the page leaves unsaid.
|
|
2089
|
+
|
|
2090
|
+
---
|
|
2091
|
+
|
|
2092
|
+
## 🧬 Prior Art
|
|
2093
|
+
|
|
2094
|
+
spool is a server-authoritative replica whose freshness policy lets a provisioned view render with
|
|
2095
|
+
no per-view validation request, with shadow mode and a fall back to the server for anything it
|
|
2096
|
+
cannot answer. HTTP caching, service-worker precaching and earlier replication systems cover parts
|
|
2097
|
+
of it. What is shared and what differs:
|
|
2098
|
+
|
|
2099
|
+
| System | What it does | Shared with spool | Difference |
|
|
2100
|
+
| ------------------ | ------------------------------------------------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------- |
|
|
2101
|
+
| HTTP caching | reuses a response within `max-age` with no validation | a repeat view with no request | one response at a time; no per-dependency invalidation and no held collections |
|
|
2102
|
+
| Workbox precaching | answers a precached navigation from the cache with no request | a worker answering a navigation with no request | the list and its revisions are fixed at build time and refresh when a new worker installs |
|
|
2103
|
+
| Fawkes (NSDI 2020) | caches static page templates and sends a patch of DOM updates | a cached template, the server authoritative | the patch is requested on every load, so a cached template never answers a view alone |
|
|
2104
|
+
| Speculation Rules | prefetches or prerenders pages the visitor may open next | acting on a predicted next view | the origin answers every speculated request, used or not |
|
|
2105
|
+
|
|
2106
|
+
Sync Kit (WWW 2010) also keeps templates and data in the browser to reduce server load. The
|
|
2107
|
+
[technical report](TECHNICAL_REPORT.md#executive-summary) states the narrow claim.
|
|
2108
|
+
|
|
2109
|
+
---
|
|
2110
|
+
|
|
1669
2111
|
## 🚫 Out of Scope
|
|
1670
2112
|
|
|
1671
2113
|
- spool never authorises on the client. The device holds only what the server already sent that
|
|
1672
2114
|
principal.
|
|
1673
2115
|
- It never shows a hard route past its bound without validating it first.
|
|
1674
|
-
- It never writes. Your application owns every write, and `invalidate` only announces
|
|
2116
|
+
- It never writes on its own. Your application owns every write, and `invalidate` only announces
|
|
2117
|
+
one. The one exception is a declared write class, which is off until you enable it.
|
|
1675
2118
|
- It never infers dependencies from opaque HTML. A route names its tags, or the records it
|
|
1676
2119
|
reads, and in proxy mode the origin names them in its cache tag headers.
|
|
1677
2120
|
- It never replaces a published release file on the CDN.
|
|
@@ -1698,20 +2141,20 @@ loads per view and reports the view as such.
|
|
|
1698
2141
|
|
|
1699
2142
|
## 🧭 Subpath Exports
|
|
1700
2143
|
|
|
1701
|
-
| Import | What is in it
|
|
1702
|
-
| ---------------------------------------- |
|
|
1703
|
-
| `@drupflare/spool` | the protocol and report modules
|
|
1704
|
-
| `@drupflare/spool/protocol` | envelopes, artifacts, patterns, templates, the manifest, view events
|
|
1705
|
-
| `@drupflare/spool/server` | `createSpool`, `defineRoute`, sinks, identity and work series
|
|
1706
|
-
| `@drupflare/spool/node` | `toNodeListener`, `toRequest`, `writeResponse`
|
|
1707
|
-
| `@drupflare/spool/proxy` | `spoolProxy`, the tag readers and the eligibility verdict
|
|
1708
|
-
| `@drupflare/spool/stores` | `memoryStore` and the `VersionStore` type
|
|
1709
|
-
| `@drupflare/spool/stores/durable-object` | `durableObjectStore` and the `SpoolVersions` class
|
|
1710
|
-
| `@drupflare/spool/sw` | `serve`, for a site that bundles its own worker
|
|
1711
|
-
| `@drupflare/spool/client` | `register
|
|
1712
|
-
| `@drupflare/spool/report` | the energy model
|
|
1713
|
-
| `@drupflare/spool/trim` | `buildTrimManifest`, `trimCss` and `guardPage`
|
|
1714
|
-
| `@drupflare/spool/media` | `tinyimgEncoder`, `imageVariant` and `
|
|
2144
|
+
| Import | What is in it |
|
|
2145
|
+
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
|
|
2146
|
+
| `@drupflare/spool` | the protocol and report modules |
|
|
2147
|
+
| `@drupflare/spool/protocol` | envelopes, artifacts, patterns, templates, the manifest, view events |
|
|
2148
|
+
| `@drupflare/spool/server` | `createSpool`, `defineRoute`, `defineDataset`, `defineWrite`, `datasetProof`, sinks, identity and work series |
|
|
2149
|
+
| `@drupflare/spool/node` | `toNodeListener`, `toRequest`, `writeResponse` |
|
|
2150
|
+
| `@drupflare/spool/proxy` | `spoolProxy`, `chooseBases`, the tag readers and the eligibility verdict |
|
|
2151
|
+
| `@drupflare/spool/stores` | `memoryStore` and the `VersionStore` type |
|
|
2152
|
+
| `@drupflare/spool/stores/durable-object` | `durableObjectStore` and the `SpoolVersions` class |
|
|
2153
|
+
| `@drupflare/spool/sw` | `serve`, for a site that bundles its own worker |
|
|
2154
|
+
| `@drupflare/spool/client` | `register`, the page handle and `createActivity` |
|
|
2155
|
+
| `@drupflare/spool/report` | the energy model, capacity plans and institution reports |
|
|
2156
|
+
| `@drupflare/spool/trim` | `buildTrimManifest`, `trimCss` and `guardPage` |
|
|
2157
|
+
| `@drupflare/spool/media` | `tinyimgEncoder`, `imageVariant`, `rewriteVideo` and `rewriteEmbeds` |
|
|
1715
2158
|
|
|
1716
2159
|
---
|
|
1717
2160
|
|
|
@@ -1723,17 +2166,30 @@ bun run test
|
|
|
1723
2166
|
bun run test:coverage
|
|
1724
2167
|
bun run test:browser
|
|
1725
2168
|
bun run prettier:check
|
|
2169
|
+
bun run impact:check
|
|
1726
2170
|
bun run docs:build
|
|
1727
2171
|
bun run build
|
|
1728
2172
|
```
|
|
1729
2173
|
|
|
1730
|
-
`
|
|
2174
|
+
`impact:check` fails when `docs/impact.md` differs from the readings in `measurements/`. A node
|
|
2175
|
+
test fails when `src/server/embedded.ts` differs from the source under `src/protocol`, `src/sw`
|
|
2176
|
+
and `src/client`, and `bun run build:sw` regenerates it. `test` runs the unit project in workerd and the node project, which includes a test that packs the
|
|
1731
2177
|
package and imports every subpath in plain Node. `test:browser` runs Chromium, Firefox and WebKit
|
|
1732
2178
|
against real servers and counts the requests that reach them. `bunx playwright install chromium
|
|
1733
2179
|
firefox webkit` fetches the browsers.
|
|
1734
2180
|
|
|
1735
2181
|
---
|
|
1736
2182
|
|
|
2183
|
+
## 📖 Documentation
|
|
2184
|
+
|
|
2185
|
+
- [docs/protocol.md](docs/protocol.md) holds the paths, headers, JSON bodies and limits for anyone
|
|
2186
|
+
who writes an adapter or a second server. A test pins what `Spool-Protocol: 1` freezes, and the
|
|
2187
|
+
page lists what may change without a new number.
|
|
2188
|
+
- [docs/security.md](docs/security.md) holds the threat model and the open limitations.
|
|
2189
|
+
- [docs/impact.md](docs/impact.md) holds the energy model, the readings and what they leave unsaid.
|
|
2190
|
+
|
|
2191
|
+
---
|
|
2192
|
+
|
|
1737
2193
|
## 🔗 Related Repositories
|
|
1738
2194
|
|
|
1739
2195
|
| Repository | What it is |
|