pmtiles-swarm 0.90.0 → 0.91.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,62 @@
7
7
  ### 🐞 Bug fixes
8
8
  - _...Add new stuff here..._
9
9
 
10
+ ## 0.91.0
11
+ ### ✨ Features and improvements
12
+ - **A source at a URL can be added by hand, not only imported.** **Add source → an address you
13
+ type…** in the stack editor, writing the same source the importer does. The card asks for the
14
+ address and for the zoom range the archive holds — a catalog archive states its range in its own
15
+ header and this has no header anybody has read, so leaving it empty means every tile asks it.
16
+ Anything published as a plain HTTPS download works, an S3 object or presigned URL included, and so
17
+ does `s3://bucket/key.pmtiles` for a private bucket — see below.
18
+
19
+ - **A stack's source list folds up past five sources.** An imported list is several hundred rows,
20
+ which buried every other stack on the page under one of them. The fold says how many there are and
21
+ names the base; a stack of a base and a layer or two stays open, since folding that hides nothing
22
+ worth a click.
23
+
24
+
25
+ - **A stack source may be an object in a private S3 bucket.**
26
+ `s3://bucket/terrain.pmtiles`, read with a signed request per byte range. Only for a bucket that is
27
+ not public — a public object or a presigned URL is an ordinary HTTPS address and always worked.
28
+ `endpoint` is what makes it S3-compatible rather than S3: MinIO, Ceph, Garage, R2, Wasabi and B2
29
+ all answer the same protocol at their own address, and path-style addressing is the default because
30
+ it is what they speak. Credentials go under **Settings → Feeds → S3 buckets**, per bucket or once
31
+ for an account; with none configured the standard `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
32
+ `AWS_REGION` and `AWS_S3_ENDPOINT` variables are read, so a machine already set up to reach a
33
+ bucket needs nothing typed in.
34
+
35
+ Signed here rather than by an SDK: SigV4 for a GET is a hash, four HMACs and a string, all of which
36
+ `node:crypto` has. An AWS client would be a large dependency for one signature, and an optional one
37
+ would make reading a bucket work on some installs and not on others for no reason the operator
38
+ could see — so it is neither optional nor a dependency. The signer is checked against AWS's own
39
+ published test vectors rather than against itself.
40
+
41
+
42
+ ### 🐞 Bug fixes
43
+ - **A stack of remote sources could be served but not exported.** The bake refused any stack whose
44
+ sources resolved to no archive on this node, which is every stack read from URLs — refusing exactly
45
+ the export worth having, since baking is how terrain that lives somewhere else becomes something
46
+ this node holds and can seed. It also had no way to walk a remote archive's directories, which is
47
+ how a bake finds which tiles exist at all.
48
+
49
+ - **An archive whose deepest tiles repeat said it stopped a zoom short.** Identical tiles collapse
50
+ into one directory entry covering a range of ids, so the last entry's own id can sit at a shallower
51
+ zoom than the tiles it addresses — and the header was read off that id. A client asks for nothing
52
+ past a header's `maxZoom`, so the deepest zoom of such an archive was unreachable. Found baking a
53
+ stack over flat terrain, where every tile of the last zoom was the same tile.
54
+
55
+ - **A stack used as a source of another stack was reported as unresolved.** It resolves to a recipe
56
+ rather than to bytes, and the console's reader looked only for a catalog entry or an address — so a
57
+ working nested stack showed "does not resolve", with no zoom range and no box, and put a "sources
58
+ missing" badge on the stack above it. It now says what it stands for and how deep it reaches,
59
+ worked out from the sources that stack would serve.
60
+
61
+ - **A stack's advertised box ignored any source that stated none.** The union was taken over
62
+ whichever sources had a box, so a stack of a global base plus regional patches — an imported list,
63
+ exactly — advertised the patches as its extent and left the base out of it. A source with no box
64
+ is not one covering nothing, so it is now every box or none, and none means the world.
65
+
10
66
  ## 0.90.0
11
67
  ### ✨ Features and improvements
12
68
  - **A stack source may name a URL instead of a category, archive or stack.** `{ "url": "https://…" }`,
@@ -20,6 +20,7 @@ Some settings only take effect on restart. Those are marked **restart**.
20
20
  - [Scheduled sources](#scheduled-sources)
21
21
  - [Subscriptions](#subscriptions)
22
22
  - [Serving tiles](#serving-tiles)
23
+ - [S3 buckets](#s3-buckets)
23
24
  - [Mutable publishing](#mutable-publishing)
24
25
  - [Authentication](#authentication)
25
26
  - [Statistics](#statistics)
@@ -776,6 +777,96 @@ connection, a piece already in flight. Ten attempts later it is one piece at the
776
777
  far end of an archive nobody has finished, and asking every fifteen seconds
777
778
  achieves nothing but log lines.
778
779
 
780
+ ## S3 buckets
781
+
782
+ A stack source may be an address rather than an archive this node holds, and
783
+ that address may name an object in a bucket:
784
+
785
+ ```json
786
+ {
787
+ "sources": [{ "url": "s3://terrain/planet.pmtiles", "encoding": "terrarium" }]
788
+ }
789
+ ```
790
+
791
+ Only for a bucket that is **not public**. A public object, or a presigned URL,
792
+ is an ordinary HTTPS address: write it as one and none of this applies.
793
+
794
+ ```json
795
+ {
796
+ "s3": [
797
+ {
798
+ "bucket": "terrain",
799
+ "endpoint": "https://minio.lan:9000",
800
+ "region": "us-east-1",
801
+ "accessKeyId": "…",
802
+ "secretAccessKey": "…"
803
+ }
804
+ ]
805
+ }
806
+ ```
807
+
808
+ | field | default | |
809
+ | ----------------- | ------------------ | ---------------------------------------------------- |
810
+ | `bucket` | any | which bucket this row is for; unset matches all |
811
+ | `endpoint` | AWS for the region | what makes it S3-compatible rather than S3 |
812
+ | `region` | `us-east-1` | signed into every request, whether or not it is used |
813
+ | `accessKeyId` | — | |
814
+ | `secretAccessKey` | — | |
815
+ | `sessionToken` | unset | for temporary credentials |
816
+ | `pathStyle` | path, unless AWS | `false` puts the bucket in the hostname |
817
+
818
+ A source is read through the row naming its bucket, or through whichever row
819
+ names none: one set of credentials for a whole account is the ordinary case,
820
+ and writing the same keys once per bucket is busywork with a copy-paste
821
+ mistake in it.
822
+
823
+ ### The endpoint is the whole point
824
+
825
+ MinIO, Ceph, Garage, Cloudflare R2, Wasabi and Backblaze B2 all answer the
826
+ same protocol at their own address, so naming that address is the difference
827
+ between "S3" and "S3-compatible". Path-style addressing is what they speak and
828
+ is the default here; AWS withdrew it for buckets made after September 2020, so
829
+ an `amazonaws.com` endpoint puts the bucket in the hostname instead. Set
830
+ `pathStyle` only for a server that insists on the other one.
831
+
832
+ ### Credentials from the environment
833
+
834
+ With no rows configured at all, the standard variables are read — the same
835
+ ones the AWS CLI, rclone, go-pmtiles and tileserver-gl use, so a machine
836
+ already set up to reach a bucket needs nothing written here:
837
+
838
+ | variable | |
839
+ | -------------------------------------------- | ------------------------------- |
840
+ | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | required for any of it to apply |
841
+ | `AWS_SESSION_TOKEN` | temporary credentials |
842
+ | `AWS_REGION`, `AWS_DEFAULT_REGION` | region |
843
+ | `AWS_S3_ENDPOINT`, `AWS_ENDPOINT_URL_S3` | endpoint |
844
+ | `AWS_S3_FORCE_PATH_STYLE` | addressing |
845
+
846
+ ### Signed here, not by an SDK
847
+
848
+ SigV4 for a GET is a hash, four HMACs and a string, all of which `node:crypto`
849
+ already has. An AWS client would be a large dependency for one signature, and
850
+ an optional one would make reading a bucket work on some installs and not on
851
+ others for no reason the operator could see — so it is neither optional nor a
852
+ dependency. The signer is checked against AWS's own published test vectors
853
+ rather than against itself.
854
+
855
+ Every request is signed as it goes out, including the range: the header, each
856
+ directory and each tile. Nothing is presigned and nothing is cached but the
857
+ bytes, so revoking a key stops reads at the next request rather than at the
858
+ next restart. Changing the settings drops the open readers for the same
859
+ reason.
860
+
861
+ ### What a bucket source is, and is not
862
+
863
+ It is a URL source in every other respect — see
864
+ [tile-stacks.md](tile-stacks.md), "A source read straight from a URL". It
865
+ carries its own `minzoom`, `maxzoom` and `bounds`, it is never seeded or
866
+ retired, and it can be baked: a stack over private buckets exports to an
867
+ ordinary archive with an infohash, which is the path from "terrain we hold in
868
+ a bucket" to "terrain we publish".
869
+
779
870
  ## Mutable publishing
780
871
 
781
872
  Announcing the current build of each category over the DHT (BEP 46).
@@ -1681,6 +1681,22 @@ archive — Mapterhorn's own base stops at z12 — is upscaled for a deeper
1681
1681
  request exactly as GEBCO is, through the same code, because reading one is now
1682
1682
  a question of which store method answers and nothing else.
1683
1683
 
1684
+ ### Adding one by hand
1685
+
1686
+ **Add source → an address you type…** in the stack editor, which is the same
1687
+ source the importer writes and edits the same way. The card asks for two
1688
+ things nothing else can know: the address, and the zoom range the archive
1689
+ holds. A catalog archive states its range in its own header and a URL source
1690
+ has no header this node has read, so an unstated range means every tile asks
1691
+ it — correct, and the thing worth avoiding once there are several.
1692
+
1693
+ Anything published as a plain HTTPS download works, which includes an S3
1694
+ bucket that serves range requests: an object URL, or a presigned one, is an
1695
+ address like any other, and this asks for byte ranges the way any PMTiles
1696
+ reader does. What it cannot do is a private bucket named `s3://…`, which is
1697
+ not a URL a browser or a fetch can follow — that needs a signed request and
1698
+ somewhere to keep the credentials, and neither exists here yet.
1699
+
1684
1700
  ### Cheap to have hundreds of
1685
1701
 
1686
1702
  The one thing that makes hundreds of these practical rather than merely
@@ -1830,6 +1846,14 @@ that is a map changing at 4am with nothing to say why. A hand-written source
1830
1846
  carries no `importedFrom`, is never replaced, and is the place for anything the
1831
1847
  index could not have told us.
1832
1848
 
1849
+ ### Seeing them in the console
1850
+
1851
+ A stack's source list collapses behind a summary once there are more than
1852
+ five, with the count and the base named on the fold. A stack imported from a
1853
+ provider's list is several hundred rows and drawing them flat buries every
1854
+ other stack on the page under one of them; a stack of a base and a layer or
1855
+ two stays open, since folding that hides nothing worth a click.
1856
+
1833
1857
  ## Finding a stack
1834
1858
 
1835
1859
  A stack has no infohash and appears in no feed, so nothing about it is
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pmtiles-swarm",
3
- "version": "0.90.0",
3
+ "version": "0.91.0",
4
4
  "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
5
5
  "type": "module",
6
6
  "main": "src/index.js",
package/src/api.js CHANGED
@@ -3400,45 +3400,64 @@ export function createApp({
3400
3400
  // The order the recipe lists them: lowest priority first, last
3401
3401
  // wins. The console shows this order as it stands rather than
3402
3402
  // inverting it, so the screen and the file never disagree.
3403
- sources: resolved.sources.map((source) => ({
3404
- index: source.index,
3405
- name: source.name,
3406
- pinned: source.pinned,
3407
- required: source.required,
3408
- // What kind of source this is, said outright rather than left to
3409
- // be inferred from which fields happen to be null. A URL source
3410
- // has no infohash and no catalog entry, so a reader guessing from
3411
- // those calls it an unresolved category -- which is two wrong
3412
- // things about a source that is working perfectly well.
3413
- kind: source.remote
3414
- ? 'url'
3415
- : source.nested
3416
- ? 'stack'
3417
- : source.pinned
3418
- ? 'archive'
3419
- : 'category',
3420
- // A URL source resolves by definition: there is nothing to look
3421
- // up, the address is the answer. Whether it can actually be read
3422
- // is a question for the first tile, the same as for an archive
3423
- // this node holds but cannot open.
3424
- resolved: Boolean(source.entry) || Boolean(source.remote),
3425
- url: source.remote ?? null,
3426
- infohash: source.entry?.infoHash ?? null,
3427
- archiveName: source.entry?.name ?? null,
3428
- // The recipe's own, where it states them. For a URL source that is
3429
- // the only place they are known without opening the file, and it
3430
- // is what the merge itself uses to decide whether to.
3431
- minzoom:
3432
- source.entry?.pmtiles?.minZoom ?? source.source?.minzoom ?? null,
3433
- maxzoom:
3434
- source.entry?.pmtiles?.maxZoom ?? source.source?.maxzoom ?? null,
3435
- bounds:
3436
- source.entry?.pmtiles?.bounds ??
3437
- (Array.isArray(source.source?.bounds)
3438
- ? source.source.bounds
3439
- : null),
3440
- format: source.entry?.pmtiles?.format ?? null,
3441
- })),
3403
+ sources: resolved.sources.map((source) => {
3404
+ // A nested stack answers for the ground its own sources cover,
3405
+ // and knows it without opening anything -- so the row can say
3406
+ // what it reaches rather than a dash.
3407
+ const inner = source.nested ? stackCoverage(source.nested) : null;
3408
+ return {
3409
+ index: source.index,
3410
+ name: source.name,
3411
+ pinned: source.pinned,
3412
+ required: source.required,
3413
+ // What kind of source this is, said outright rather than left to
3414
+ // be inferred from which fields happen to be null. A URL source
3415
+ // has no infohash and no catalog entry, so a reader guessing from
3416
+ // those calls it an unresolved category -- which is two wrong
3417
+ // things about a source that is working perfectly well.
3418
+ kind: source.remote
3419
+ ? 'url'
3420
+ : source.nested
3421
+ ? 'stack'
3422
+ : source.pinned
3423
+ ? 'archive'
3424
+ : 'category',
3425
+ // A URL source resolves by definition: there is nothing to look
3426
+ // up, the address is the answer. Whether it can actually be read
3427
+ // is a question for the first tile, the same as for an archive
3428
+ // this node holds but cannot open.
3429
+ resolved:
3430
+ Boolean(source.entry) ||
3431
+ Boolean(source.remote) ||
3432
+ Boolean(source.nested),
3433
+ // How many sources the nested recipe has, which is the only
3434
+ // useful thing to say in a column that names an archive file for
3435
+ // every other kind: a stack resolves to a recipe, not to bytes.
3436
+ nested: source.nested ? source.nested.sources.length : null,
3437
+ url: source.remote ?? null,
3438
+ infohash: source.entry?.infoHash ?? null,
3439
+ archiveName: source.entry?.name ?? null,
3440
+ // The recipe's own, where it states them. For a URL source that is
3441
+ // the only place they are known without opening the file, and it
3442
+ // is what the merge itself uses to decide whether to.
3443
+ minzoom:
3444
+ source.entry?.pmtiles?.minZoom ??
3445
+ source.source?.minzoom ??
3446
+ inner?.minzoom ??
3447
+ null,
3448
+ maxzoom:
3449
+ source.entry?.pmtiles?.maxZoom ??
3450
+ source.source?.maxzoom ??
3451
+ inner?.maxzoom ??
3452
+ null,
3453
+ bounds:
3454
+ source.entry?.pmtiles?.bounds ??
3455
+ (Array.isArray(source.source?.bounds)
3456
+ ? source.source.bounds
3457
+ : (inner?.bounds ?? null)),
3458
+ format: source.entry?.pmtiles?.format ?? null,
3459
+ };
3460
+ }),
3442
3461
  };
3443
3462
  });
3444
3463
  // Reported so the console can say "install sharp" beside the stacks
package/src/bake-jobs.js CHANGED
@@ -411,12 +411,18 @@ export class BakeManager {
411
411
  // presses the button rather than an hour in.
412
412
  assertBakeable(resolved, codec);
413
413
 
414
+ // A source this node can read: an archive it holds, a recipe it can
415
+ // evaluate, or an address it can fetch. The last was missing, which
416
+ // refused exactly the bake worth having -- a stack of remote sources
417
+ // exported to an archive is how terrain that lives somewhere else, in a
418
+ // provider's file list or a private bucket, becomes something this node
419
+ // holds and can seed.
414
420
  const unresolved = resolved.sources.filter(
415
- (source) => !source.entry && !source.nested,
421
+ (source) => !source.entry && !source.nested && !source.remote,
416
422
  );
417
423
  if (unresolved.length === resolved.sources.length) {
418
424
  const error = new Error(
419
- "none of this stack's sources resolved to an archive on this node",
425
+ "none of this stack's sources resolved to anything this node can read",
420
426
  );
421
427
  error.status = 409;
422
428
  throw error;
@@ -594,15 +600,22 @@ export class BakeManager {
594
600
  // source is scanned the same way it is served: its directories come out of
595
601
  // the swarm, and the store holds them to its own byte budget.
596
602
  const sources = resolved.sources
597
- .filter((source) => source.entry)
603
+ .filter((source) => source.entry || source.remote)
598
604
  .map((source) => ({
599
605
  getBytes: async (offset, length) => {
600
- const bytes = await this.#tiles.readRange(
601
- source.entry.infoHash,
602
- offset,
603
- length,
604
- { signal: job.controller.signal },
605
- );
606
+ // The one difference between a source this node holds and one it
607
+ // reads over the network, here as everywhere else: which method of
608
+ // the store answers. What is done with the bytes is identical.
609
+ const bytes = source.remote
610
+ ? await this.#tiles.readRemoteRange(source.remote, offset, length, {
611
+ signal: job.controller.signal,
612
+ })
613
+ : await this.#tiles.readRange(
614
+ source.entry.infoHash,
615
+ offset,
616
+ length,
617
+ { signal: job.controller.signal },
618
+ );
606
619
  // Sliced by offset and length rather than handed `.buffer` outright.
607
620
  // A Buffer is a view, and where it is a view *into* something larger
608
621
  // -- which is what Buffer.allocUnsafe hands back -- `.buffer` is the
package/src/config.js CHANGED
@@ -459,6 +459,25 @@ const DEFAULTS = {
459
459
  * See docs/tile-stacks.md -- "Syncing a stack to another node".
460
460
  */
461
461
  stackFeeds: [],
462
+ /**
463
+ * S3-compatible buckets a stack source may be read from.
464
+ *
465
+ * `[{ bucket, endpoint, region, accessKeyId, secretAccessKey, sessionToken,
466
+ * pathStyle }]`. A source whose URL is `s3://bucket/key.pmtiles` is read
467
+ * through the row naming that bucket, or through whichever row names none.
468
+ * A row is needed only for a bucket that is not public: a presigned or
469
+ * public HTTPS address is read as any other URL and knows nothing of this.
470
+ *
471
+ * `endpoint` is what makes it S3-compatible rather than S3: MinIO, Ceph,
472
+ * Garage, R2, Wasabi and B2 all answer the same protocol at their own
473
+ * address. Left unset it is AWS's own for the region.
474
+ *
475
+ * With nothing here at all the standard `AWS_ACCESS_KEY_ID`,
476
+ * `AWS_SECRET_ACCESS_KEY`, `AWS_REGION` and `AWS_S3_ENDPOINT` variables are
477
+ * read, so a machine already set up to reach a bucket needs no settings.
478
+ * See docs/configuration.md -- "S3 buckets".
479
+ */
480
+ s3: [],
462
481
  /**
463
482
  * How often to re-check whether the sources archives were built from have
464
483
  * changed, in seconds. Zero disables it.
@@ -861,6 +880,10 @@ export const RELOADABLE = new Map([
861
880
  ['sourceCheckIntervalHours', 'sources'],
862
881
  ['subscriptions', 'subscriptions'],
863
882
  ['stackFeeds', 'stackFeeds'],
883
+ // Read when a bucket-backed source is first opened, and the open handles are
884
+ // keyed by address rather than by credentials -- so a corrected key applies
885
+ // to the next archive opened rather than needing a restart.
886
+ ['s3', 's3'],
864
887
  ['subscriptionIntervalSeconds', 'subscriptions'],
865
888
  ['subscriptionsEnabled', 'subscriptions'],
866
889
  ['seeding', 'seeding'],
package/src/index.js CHANGED
@@ -586,6 +586,10 @@ PMTILES_SWARM_PUBLIC_URL
586
586
  stackFeeds.stop();
587
587
  stackFeeds.start();
588
588
  },
589
+ // Nothing to restart: the next archive opened from a bucket reads the new
590
+ // settings. What has to go is the readers already open, which are holding
591
+ // the keys they were opened with.
592
+ s3: () => tiles.forgetRemote(),
589
593
  seeding: () => {
590
594
  seeding.stop();
591
595
  seeding.start();
@@ -459,9 +459,14 @@ export class PMTilesWriter {
459
459
  // client asks for, and a header disagreeing with the directory sends it
460
460
  // looking for tiles that are not there.
461
461
  minZoom: header.minZoom ?? zoomOf(this.#entries[0].tileId),
462
- maxZoom:
463
- header.maxZoom ??
464
- zoomOf(this.#entries[this.#entries.length - 1].tileId),
462
+ // The last id written, not the last entry's own. Identical tiles
463
+ // collapse into one entry covering a range, so an archive whose deepest
464
+ // tiles repeat -- open ocean, a masked-out region, anything flat --
465
+ // ends with an entry whose id sits at a shallower zoom than the tiles
466
+ // it addresses. Read as the entry's id, the header said the archive
467
+ // stopped short of the zoom it reaches, and a client asks for nothing
468
+ // past a header's maxZoom.
469
+ maxZoom: header.maxZoom ?? zoomOf(this.#lastId),
465
470
  };
466
471
 
467
472
  const out = await fs.open(destination, 'w');
@@ -0,0 +1,302 @@
1
+ import crypto from 'node:crypto';
2
+
3
+ /**
4
+ * Reading a PMTiles archive out of an S3-compatible bucket.
5
+ *
6
+ * A public or presigned URL needs nothing from this file: it is an address,
7
+ * `FetchSource` asks it for byte ranges and the bucket answers. What needs
8
+ * signing is the ordinary private object, and that is the case worth having --
9
+ * it is the difference between terrain somebody published and terrain this
10
+ * node holds, and it is what lets a stack read a bucket and bake the result
11
+ * into an archive without the bucket ever being public.
12
+ *
13
+ * Signed here rather than through an SDK. SigV4 for a GET is a hash, four
14
+ * HMACs and a string, all of which `node:crypto` already has; the AWS client
15
+ * that would do it instead is a large dependency for one signature, and an
16
+ * optional one would make reading a bucket work on some installs and not on
17
+ * others for no reason the operator could see.
18
+ *
19
+ * See docs/configuration.md -- "S3 buckets".
20
+ */
21
+
22
+ /** What SigV4 calls an empty body, which every GET has. */
23
+ const EMPTY_SHA256 = crypto.createHash('sha256').update('').digest('hex');
24
+
25
+ /**
26
+ * HMAC-SHA256, the only primitive the signature needs.
27
+ * @param {Buffer|string} key - The key.
28
+ * @param {string} data - What to sign.
29
+ * @returns {Buffer} - The digest.
30
+ */
31
+ const hmac = (key, data) =>
32
+ crypto.createHmac('sha256', key).update(data, 'utf8').digest();
33
+
34
+ /**
35
+ * Query parameters in the order and encoding SigV4 asks for.
36
+ * @param {URL} url - The address being signed.
37
+ * @returns {string} - The canonical query string.
38
+ */
39
+ const canonicalQuery = (url) => {
40
+ const pairs = [...url.searchParams.entries()].map(([name, value]) => [
41
+ encodeURIComponent(name),
42
+ encodeURIComponent(value),
43
+ ]);
44
+ pairs.sort((one, two) =>
45
+ one[0] === two[0] ? one[1].localeCompare(two[1]) : one[0] < two[0] ? -1 : 1,
46
+ );
47
+ return pairs.map(([name, value]) => `${name}=${value}`).join('&');
48
+ };
49
+
50
+ /**
51
+ * The `Authorization` header for one request.
52
+ *
53
+ * Signs exactly the headers it is given, which is what makes it checkable
54
+ * against AWS's own published test vectors rather than only against itself.
55
+ * @param {object} request - `method`, `url`, `headers`, `payloadHash`.
56
+ * @param {object} credentials - `accessKeyId`, `secretAccessKey`, `region`,
57
+ * `service` and `amzDate`, which is `YYYYMMDDTHHMMSSZ`.
58
+ * @returns {string} - The header value.
59
+ */
60
+ export function authorization(request, credentials) {
61
+ const url = new URL(request.url);
62
+ const { amzDate } = credentials;
63
+ const day = amzDate.slice(0, 8);
64
+ const service = credentials.service ?? 's3';
65
+ const scope = `${day}/${credentials.region}/${service}/aws4_request`;
66
+
67
+ const named = Object.entries(request.headers ?? {})
68
+ .map(([name, value]) => [name.toLowerCase(), String(value).trim()])
69
+ .sort((one, two) => (one[0] < two[0] ? -1 : 1));
70
+ const signed = named.map(([name]) => name).join(';');
71
+ const canonicalHeaders = named
72
+ .map(([name, value]) => `${name}:${value}\n`)
73
+ .join('');
74
+
75
+ // The path as it appears on the wire rather than a normalised form of it:
76
+ // S3 signs what it is sent, so a key holding a literal per-cent sign signs
77
+ // with that sign encoded, and re-encoding here would sign something else.
78
+ const canonical = [
79
+ request.method ?? 'GET',
80
+ url.pathname || '/',
81
+ canonicalQuery(url),
82
+ canonicalHeaders,
83
+ signed,
84
+ request.payloadHash ?? EMPTY_SHA256,
85
+ ].join('\n');
86
+
87
+ const toSign = [
88
+ 'AWS4-HMAC-SHA256',
89
+ amzDate,
90
+ scope,
91
+ crypto.createHash('sha256').update(canonical).digest('hex'),
92
+ ].join('\n');
93
+
94
+ const key = ['aws4_request', service, credentials.region, day].reduceRight(
95
+ (acc, part) => hmac(acc, part),
96
+ `AWS4${credentials.secretAccessKey}`,
97
+ );
98
+ const signature = hmac(key, toSign).toString('hex');
99
+
100
+ return (
101
+ `AWS4-HMAC-SHA256 Credential=${credentials.accessKeyId}/${scope}, ` +
102
+ `SignedHeaders=${signed}, Signature=${signature}`
103
+ );
104
+ }
105
+
106
+ /**
107
+ * Whether an address names an object in a bucket rather than a web server.
108
+ * @param {string} value - The address.
109
+ * @returns {boolean} - True for `s3://bucket/key`.
110
+ */
111
+ export const isS3Url = (value) =>
112
+ typeof value === 'string' && /^s3:\/\/[^/]+\/.+/i.test(value);
113
+
114
+ /**
115
+ * The bucket and key an `s3://` address names.
116
+ * @param {string} value - The address.
117
+ * @returns {object} - `{bucket, key}`.
118
+ */
119
+ export function splitS3Url(value) {
120
+ const rest = String(value).slice('s3://'.length);
121
+ const at = rest.indexOf('/');
122
+ return { bucket: rest.slice(0, at), key: rest.slice(at + 1) };
123
+ }
124
+
125
+ /**
126
+ * Credentials from the environment, in the names every S3 tool already uses.
127
+ *
128
+ * The same variables the AWS CLI, rclone, go-pmtiles and tileserver-gl read,
129
+ * so a machine already set up to reach a bucket needs nothing written into
130
+ * this node's settings -- and a container gets its keys the way a container
131
+ * gets its keys, rather than through a config file baked into an image.
132
+ *
133
+ * `AWS_S3_ENDPOINT` and `AWS_ENDPOINT_URL_S3` are both read: the first is
134
+ * what tileserver-gl documents, the second what the current AWS SDKs do.
135
+ * @param {object} [env] - The environment.
136
+ * @returns {object|null} - A row like `config.s3` holds, or null.
137
+ */
138
+ export function bucketFromEnv(env = process.env) {
139
+ if (!env.AWS_ACCESS_KEY_ID || !env.AWS_SECRET_ACCESS_KEY) return null;
140
+ const forced = String(
141
+ env.AWS_S3_FORCE_PATH_STYLE ?? env.AWS_ENDPOINT_FORCE_PATH_STYLE ?? '',
142
+ ).toLowerCase();
143
+ return {
144
+ from: 'the environment',
145
+ endpoint: env.AWS_S3_ENDPOINT || env.AWS_ENDPOINT_URL_S3 || undefined,
146
+ region: env.AWS_REGION || env.AWS_DEFAULT_REGION || undefined,
147
+ accessKeyId: env.AWS_ACCESS_KEY_ID,
148
+ secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
149
+ sessionToken: env.AWS_SESSION_TOKEN || undefined,
150
+ pathStyle: forced === '' ? undefined : forced !== 'false' && forced !== '0',
151
+ };
152
+ }
153
+
154
+ /**
155
+ * Which configured bucket a source's address should be read through.
156
+ *
157
+ * By name first, then whichever row names no bucket, then the environment --
158
+ * one set of credentials for a whole account is the ordinary case, and making
159
+ * the operator write the same keys once per bucket would be busywork with a
160
+ * copy-paste mistake in it.
161
+ * @param {string} url - An `s3://` address.
162
+ * @param {object[]} rows - `config.s3`.
163
+ * @param {object} [env] - The environment, for the last resort.
164
+ * @returns {object|null} - The row, or null if nothing matches.
165
+ */
166
+ export function bucketFor(url, rows = [], env = process.env) {
167
+ if (!isS3Url(url)) return null;
168
+ const { bucket } = splitS3Url(url);
169
+ const named = (rows ?? []).find(
170
+ (row) => row?.bucket && row.bucket === bucket,
171
+ );
172
+ return (
173
+ named ??
174
+ (rows ?? []).find((row) => row && !row.bucket) ??
175
+ bucketFromEnv(env)
176
+ );
177
+ }
178
+
179
+ /**
180
+ * The HTTPS address an `s3://` one is actually fetched from.
181
+ *
182
+ * Path style unless the endpoint is AWS's own, where it was withdrawn for
183
+ * buckets made after September 2020. Every other S3-compatible server this is
184
+ * likely to meet -- MinIO, Ceph, Garage, R2 -- speaks path style, and it is
185
+ * the form that works with an endpoint naming a port or a bare host.
186
+ * @param {string} url - The `s3://` address.
187
+ * @param {object} row - The matching row of `config.s3`.
188
+ * @returns {string} - An https address.
189
+ */
190
+ export function httpsFor(url, row) {
191
+ const { bucket, key } = splitS3Url(url);
192
+ const region = row?.region || 'us-east-1';
193
+ const endpoint = row?.endpoint || `https://s3.${region}.amazonaws.com`;
194
+ const base = new URL(
195
+ endpoint.includes('://') ? endpoint : `https://${endpoint}`,
196
+ );
197
+ const path = key
198
+ .split('/')
199
+ .map((part) => encodeURIComponent(part))
200
+ .join('/');
201
+
202
+ const virtual =
203
+ row?.pathStyle === false ||
204
+ (row?.pathStyle === undefined &&
205
+ /(^|\.)amazonaws\.com$/i.test(base.hostname));
206
+ if (virtual) {
207
+ base.hostname = `${bucket}.${base.hostname}`;
208
+ base.pathname = `/${path}`;
209
+ } else {
210
+ base.pathname = `${base.pathname.replace(/\/$/, '')}/${bucket}/${path}`;
211
+ }
212
+ return base.toString();
213
+ }
214
+
215
+ /**
216
+ * A PMTiles source reading a private bucket, signing every request.
217
+ *
218
+ * The same shape as `NodeFileSource` and the swarm's own source, because that
219
+ * is all PMTiles asks for: a key to cache under, and a way to get a range of
220
+ * bytes. Nothing about the archive is different -- only where the bytes come
221
+ * from, and that the request carries a signature.
222
+ */
223
+ export class S3Source {
224
+ #url;
225
+ #row;
226
+ #now;
227
+
228
+ /**
229
+ * @param {string} url - An `s3://bucket/key` address.
230
+ * @param {object} row - The matching row of `config.s3`.
231
+ * @param {Function} [now] - The clock, for tests.
232
+ */
233
+ constructor(url, row, now = () => new Date()) {
234
+ this.#url = url;
235
+ this.#row = row ?? {};
236
+ this.#now = now;
237
+ }
238
+
239
+ /**
240
+ * A stable key for PMTiles' internal caching.
241
+ *
242
+ * The `s3://` address rather than the signed one: the signature changes
243
+ * every request, and a cache keyed on it would never hit.
244
+ * @returns {string} - The address.
245
+ */
246
+ getKey() {
247
+ return this.#url;
248
+ }
249
+
250
+ /**
251
+ * Reads a byte range out of the object.
252
+ * @param {number} offset - Byte offset.
253
+ * @param {number} length - How many bytes.
254
+ * @param {AbortSignal} [signal] - To give up early.
255
+ * @returns {Promise<object>} - `{data, etag, cacheControl, expires}`.
256
+ */
257
+ async getBytes(offset, length, signal) {
258
+ const href = httpsFor(this.#url, this.#row);
259
+ const amzDate = `${this.#now()
260
+ .toISOString()
261
+ .replace(/[-:]/g, '')
262
+ .slice(0, 15)}Z`;
263
+ const headers = {
264
+ host: new URL(href).host,
265
+ range: `bytes=${offset}-${offset + length - 1}`,
266
+ 'x-amz-content-sha256': EMPTY_SHA256,
267
+ 'x-amz-date': amzDate,
268
+ };
269
+ if (this.#row.sessionToken) {
270
+ headers['x-amz-security-token'] = this.#row.sessionToken;
271
+ }
272
+
273
+ headers.authorization = authorization(
274
+ { method: 'GET', url: href, headers },
275
+ {
276
+ accessKeyId: this.#row.accessKeyId,
277
+ secretAccessKey: this.#row.secretAccessKey,
278
+ region: this.#row.region || 'us-east-1',
279
+ amzDate,
280
+ },
281
+ );
282
+
283
+ // `host` is set by fetch itself and refused as a header, but it has to be
284
+ // in the signature: it is what stops a signed request being replayed
285
+ // against a different endpoint.
286
+ const sent = { ...headers };
287
+ delete sent.host;
288
+
289
+ const response = await fetch(href, { headers: sent, signal });
290
+ if (!response.ok) {
291
+ throw new Error(
292
+ `${this.#url} answered ${response.status} ${response.statusText}`,
293
+ );
294
+ }
295
+ return {
296
+ data: await response.arrayBuffer(),
297
+ etag: response.headers.get('etag') ?? undefined,
298
+ cacheControl: response.headers.get('cache-control') ?? undefined,
299
+ expires: response.headers.get('expires') ?? undefined,
300
+ };
301
+ }
302
+ }
package/src/stacks.js CHANGED
@@ -1,6 +1,7 @@
1
1
  import crypto from 'node:crypto';
2
2
  import fs from 'node:fs/promises';
3
3
  import path from 'node:path';
4
+ import { isS3Url } from './s3-source.js';
4
5
  import {
5
6
  MAX_BLUR_SIGMA,
6
7
  RESAMPLING,
@@ -91,6 +92,20 @@ export function isHttpUrl(value) {
91
92
  }
92
93
  }
93
94
 
95
+ /**
96
+ * Whether an address is one a source may be read from.
97
+ *
98
+ * Wider than `isHttpUrl` by one scheme: `s3://bucket/key` is an object in a
99
+ * bucket, fetched over HTTPS like anything else once a configured endpoint
100
+ * and a signature have been put on it. A feed address is not this -- that is
101
+ * something the node fetches plainly and has no credentials for.
102
+ * @param {string} value - What the recipe said.
103
+ * @returns {boolean} - True for http, https or s3.
104
+ */
105
+ export function isSourceUrl(value) {
106
+ return isHttpUrl(value) || isS3Url(value);
107
+ }
108
+
94
109
  /**
95
110
  * How many stacks deep a recipe may reach.
96
111
  *
@@ -159,8 +174,10 @@ export function validateStack(stack) {
159
174
  );
160
175
  }
161
176
  if (source?.url !== undefined) {
162
- if (typeof source.url !== 'string' || !isHttpUrl(source.url)) {
163
- problems.push(`sources[${index}].url must be an http(s) address`);
177
+ if (typeof source.url !== 'string' || !isSourceUrl(source.url)) {
178
+ problems.push(
179
+ `sources[${index}].url must be an http(s) or s3:// address`,
180
+ );
164
181
  }
165
182
  }
166
183
  for (const key of ['minzoom', 'maxzoom']) {
@@ -653,8 +670,13 @@ export function stackCoverage(resolved) {
653
670
  (at?.nested ? stackCoverage(at.nested).bounds : undefined);
654
671
  }
655
672
  if (!bounds && summaries.length) {
656
- const boxes = summaries.map((s) => s.bounds).filter(Array.isArray);
657
- if (boxes.length) {
673
+ const boxes = summaries.map((s) => s.bounds);
674
+ // Every source, or none of them. A source that states no box is not one
675
+ // covering nothing: an archive's box comes off its own header, and the
676
+ // source without one is the imported global base, which covers the world.
677
+ // Unioning only the boxes that exist advertised a planet-wide stack as
678
+ // covering whichever patches happened to state theirs.
679
+ if (boxes.length && boxes.every(Array.isArray)) {
658
680
  bounds = [
659
681
  Math.min(...boxes.map((b) => b[0])),
660
682
  Math.min(...boxes.map((b) => b[1])),
package/src/tiles.js CHANGED
@@ -2,6 +2,7 @@ import fs from 'node:fs/promises';
2
2
  import path from 'node:path';
3
3
  import zlib from 'node:zlib';
4
4
  import { FetchSource, PMTiles, SharedPromiseCache } from 'pmtiles';
5
+ import { S3Source, bucketFor, isS3Url, splitS3Url } from './s3-source.js';
5
6
  import { TorrentSource } from 'pmtiles-torrent';
6
7
  import { NodeFileSource } from './file-source.js';
7
8
  import { onDiskPath } from './incomplete.js';
@@ -199,6 +200,18 @@ export class TileStore {
199
200
  * @param {string} url - Where the archive is.
200
201
  * @returns {Promise<object>} - The open handle.
201
202
  */
203
+ /**
204
+ * Drops every open reader of an archive read over the network.
205
+ *
206
+ * Called when the bucket settings change. A handle holds the credentials it
207
+ * was opened with, so without this a corrected key would apply to archives
208
+ * opened afterwards and not to the one being looked at.
209
+ * @returns {void}
210
+ */
211
+ forgetRemote() {
212
+ this.#remote.clear();
213
+ }
214
+
202
215
  async #acquireRemote(url) {
203
216
  const existing = this.#remote.get(url);
204
217
  if (existing) {
@@ -207,7 +220,18 @@ export class TileStore {
207
220
  return existing;
208
221
  }
209
222
 
210
- const source = new FetchSource(url);
223
+ // A bucket rather than a web server. The difference is the signature on
224
+ // every request and where the address is really fetched from; everything
225
+ // after this line -- the archive, the directory cache, the LRU -- is the
226
+ // same, because a source is only a key and a way to get bytes.
227
+ const row = bucketFor(url, this.#config.s3);
228
+ if (isS3Url(url) && !row) {
229
+ throw new TileReadError(
230
+ `no credentials configured for ${splitS3Url(url).bucket}`,
231
+ 502,
232
+ );
233
+ }
234
+ const source = row ? new S3Source(url, row) : new FetchSource(url);
211
235
  const handle = {
212
236
  mode: 'remote',
213
237
  source,
@@ -261,6 +285,36 @@ export class TileStore {
261
285
  const answer = await handle.source.getBytes(offset, length, options.signal);
262
286
  return Buffer.from(answer.data);
263
287
  }
288
+
289
+ /**
290
+ * Reads a byte range out of an archive at a URL.
291
+ *
292
+ * The same question `readRange` answers, for the sources that have no
293
+ * infohash to ask it about. A bake needs this rather than tile-by-tile
294
+ * reads: it walks the directories of every source to find which tiles exist
295
+ * at all, and a stack of remote sources with no way to do that could be
296
+ * served and not exported -- which is backwards, since exporting is how
297
+ * something read from somewhere else becomes something this node holds.
298
+ * @param {string} url - The archive's address.
299
+ * @param {number} offset - Byte offset into the archive file.
300
+ * @param {number} length - How many bytes.
301
+ * @param {object} [options] - Abort signal.
302
+ * @returns {Promise<Buffer>} - The bytes.
303
+ */
304
+ async readRemoteRange(url, offset, length, options = {}) {
305
+ const handle = await this.#acquireRemote(url);
306
+ try {
307
+ const answer = await handle.source.getBytes(
308
+ offset,
309
+ length,
310
+ options.signal,
311
+ );
312
+ return Buffer.from(answer.data);
313
+ } catch (error) {
314
+ if (error instanceof TileReadError) throw error;
315
+ throw new TileReadError(`${url}: ${error.message}`, 502);
316
+ }
317
+ }
264
318
  /**
265
319
  * Reads an archive's header and metadata, through whatever source applies.
266
320
  *
@@ -6726,6 +6726,56 @@ Every piece is hashed against the ` +
6726
6726
  'anybody remembering to press it.',
6727
6727
  });
6728
6728
 
6729
+ renderRowEditor({
6730
+ into: paneFor('Feeds'),
6731
+ key: 's3',
6732
+ title: 'S3 buckets',
6733
+ blurb:
6734
+ 'Credentials for reading a stack source written as ' +
6735
+ '<code>s3://bucket/key.pmtiles</code>. Only for a bucket that is ' +
6736
+ 'not public — a public or presigned address is read as any other ' +
6737
+ 'URL and needs nothing here. <b>Endpoint</b> is what makes it ' +
6738
+ 'S3-compatible rather than S3: MinIO, Ceph, Garage, R2, Wasabi ' +
6739
+ 'and B2 all answer the same protocol at their own address.',
6740
+ columns: [
6741
+ {
6742
+ field: 'bucket',
6743
+ label: 'Bucket',
6744
+ placeholder: 'any bucket',
6745
+ },
6746
+ {
6747
+ field: 'endpoint',
6748
+ label: 'Endpoint',
6749
+ placeholder: 'AWS for the region',
6750
+ wide: true,
6751
+ },
6752
+ { field: 'region', label: 'Region', placeholder: 'us-east-1' },
6753
+ { field: 'accessKeyId', label: 'Access key' },
6754
+ { field: 'secretAccessKey', label: 'Secret key', secret: true },
6755
+ {
6756
+ field: 'pathStyle',
6757
+ label: 'Addressing',
6758
+ options: [
6759
+ ['', 'path style, unless AWS'],
6760
+ ['true', 'path style'],
6761
+ ['false', 'bucket as a subdomain'],
6762
+ ],
6763
+ },
6764
+ ],
6765
+ rows: config.s3 ?? [],
6766
+ footnote:
6767
+ 'A source is read through the row naming its bucket, or through ' +
6768
+ 'whichever row names none — one set of credentials for a whole ' +
6769
+ 'account is the ordinary case. With no rows at all the standard ' +
6770
+ '<code>AWS_ACCESS_KEY_ID</code>, <code>AWS_SECRET_ACCESS_KEY</code>, ' +
6771
+ '<code>AWS_REGION</code> and <code>AWS_S3_ENDPOINT</code> ' +
6772
+ 'variables are read, so a machine already set up to reach a ' +
6773
+ 'bucket needs nothing typed here. Addressing is only worth ' +
6774
+ 'setting for a server that insists: path style is what every ' +
6775
+ 'S3-compatible server speaks, and AWS itself withdrew it for ' +
6776
+ 'buckets made after September 2020.',
6777
+ });
6778
+
6729
6779
  renderRowEditor({
6730
6780
  // With the monitored folders and the scheduled sources, because it
6731
6781
  // is the same kind of thing: something that puts an archive into
@@ -7655,21 +7705,64 @@ Every piece is hashed against the ` +
7655
7705
  and this node has no codec. <code>npm install sharp</code>.</p>`
7656
7706
  : ''
7657
7707
  }
7658
- <table class="tight">
7659
- <thead>
7660
- <tr><th>#</th><th>Source</th><th>Resolves to</th><th>Zooms</th><th></th></tr>
7661
- </thead>
7662
- <tbody>
7663
- ${stack.sources.map(renderStackSource).join('')}
7664
- </tbody>
7665
- </table>
7666
- <p class="muted">
7667
- Listed in the file's order: the first is the base, each one covers
7668
- the one before it wherever it has data, and the last wins.
7669
- </p>
7708
+ <details${stack.sources.length > SOURCES_SHOWN ? '' : ' open'}>
7709
+ <summary>${stack.sources.length} source${
7710
+ stack.sources.length === 1 ? '' : 's'
7711
+ }${
7712
+ stack.sources.length > SOURCES_SHOWN
7713
+ ? ` <span class="muted">— ${escapeHtml(
7714
+ summarizeSources(stack.sources),
7715
+ )}</span>`
7716
+ : ''
7717
+ }</summary>
7718
+ <table class="tight">
7719
+ <thead>
7720
+ <tr><th>#</th><th>Source</th><th>Resolves to</th><th>Zooms</th><th></th></tr>
7721
+ </thead>
7722
+ <tbody>
7723
+ ${stack.sources.map(renderStackSource).join('')}
7724
+ </tbody>
7725
+ </table>
7726
+ <p class="muted">
7727
+ Listed in the file's order: the first is the base, each one covers
7728
+ the one before it wherever it has data, and the last wins.
7729
+ </p>
7730
+ </details>
7670
7731
  </div>`;
7671
7732
  };
7672
7733
 
7734
+ /**
7735
+ * How many sources a stack shows without being asked.
7736
+ *
7737
+ * A stack built from a provider's file list runs to several hundred
7738
+ * rows, which buries every other stack on the page under one of them.
7739
+ * Small enough that the ordinary stack -- a base and a layer or two --
7740
+ * is still open on arrival, since collapsing that hides nothing worth
7741
+ * the click.
7742
+ */
7743
+ const SOURCES_SHOWN = 5;
7744
+
7745
+ /**
7746
+ * What a collapsed source list says about itself.
7747
+ * @param {object[]} sources - The stack's sources.
7748
+ * @returns {string} - A line naming the base and the shape of the rest.
7749
+ */
7750
+ const summarizeSources = (sources) => {
7751
+ const first = sources[0];
7752
+ const base =
7753
+ first?.kind === 'url'
7754
+ ? (first.url ?? '').replace(/^https?:\/\//, '')
7755
+ : (first?.name ?? '');
7756
+ const urls = sources.filter((s) => s.kind === 'url').length;
7757
+ const rest =
7758
+ urls === sources.length
7759
+ ? 'all read over HTTP'
7760
+ : urls
7761
+ ? `${urls} read over HTTP`
7762
+ : '';
7763
+ return [base, rest].filter(Boolean).join(', ');
7764
+ };
7765
+
7673
7766
  /**
7674
7767
  * One source row.
7675
7768
  * @param {object} source - A source from /api/stacks.
@@ -7713,7 +7806,11 @@ Every piece is hashed against the ` +
7713
7806
  source.resolved
7714
7807
  ? source.kind === 'url'
7715
7808
  ? `<span class="muted" title="Read straight from this address over HTTP. Nothing is downloaded or seeded, and whether it answers is settled at the first tile.">read over HTTP</span>`
7716
- : `<code class="muted">${escapeHtml(source.archiveName ?? '')}</code>`
7809
+ : source.kind === 'stack'
7810
+ ? `<span class="muted" title="Another recipe, evaluated for this tile and merged as one layer. It follows every later change to that stack.">recipe of ${
7811
+ source.nested ?? 0
7812
+ } source${source.nested === 1 ? '' : 's'}</span>`
7813
+ : `<code class="muted">${escapeHtml(source.archiveName ?? '')}</code>`
7717
7814
  : '<span class="bad">does not resolve</span>'
7718
7815
  }</td>
7719
7816
  <td>${zooms}${
@@ -7754,6 +7851,18 @@ Every piece is hashed against the ` +
7754
7851
  return candidate;
7755
7852
  };
7756
7853
 
7854
+ /**
7855
+ * Whether an address is one a source can be read from.
7856
+ *
7857
+ * `s3://bucket/key` as well as the two web schemes: a bucket is fetched
7858
+ * over HTTPS like anything else, once the endpoint and the signature
7859
+ * configured under Settings have been put on it.
7860
+ * @param {string} value - What was typed.
7861
+ * @returns {boolean} - True if a source could use it.
7862
+ */
7863
+ const isSourceAddress = (value) =>
7864
+ /^https?:\/\//i.test(value) || /^s3:\/\/[^/]+\/.+/i.test(value);
7865
+
7757
7866
  /** Forty characters of hex is not a label. */
7758
7867
  const shortHash = (hash) => (hash ? `${hash.slice(0, 12)}…` : hash);
7759
7868
 
@@ -7972,11 +8081,14 @@ Every piece is hashed against the ` +
7972
8081
  );
7973
8082
  }
7974
8083
 
7975
- select.innerHTML = groups.length
7976
- ? groups.join('')
7977
- : '<option value="">nothing left to add</option>';
7978
- select.disabled = groups.length === 0;
7979
- $('stack-add-source-go').disabled = groups.length === 0;
8084
+ // Always offered, and last: it names nothing this node holds, so
8085
+ // there is no list to be empty and no reason to prefer it over an
8086
+ // archive that is already here.
8087
+ groups.push(
8088
+ `<optgroup label="A URL — read over HTTP, never downloaded"><option value="url:">an address you type…</option></optgroup>`,
8089
+ );
8090
+
8091
+ select.innerHTML = groups.join('');
7980
8092
  };
7981
8093
 
7982
8094
  /** Redraws the source list and the fields that depend on the space. */
@@ -8124,6 +8236,10 @@ Every piece is hashed against the ` +
8124
8236
  // against. Those fields are left out rather than shown and refused on
8125
8237
  // save; everything that acts on heights stays.
8126
8238
  const nested = Boolean(source.stack);
8239
+ // An archive somewhere else. It has no catalog entry to read a zoom
8240
+ // range or an extent from, so the recipe is the only place either can
8241
+ // come from -- which is why the card asks for them.
8242
+ const remote = source.url !== undefined;
8127
8243
  const perSpace = rgba
8128
8244
  ? `<label class="choice">Opacity
8129
8245
  <input type="number" min="0" max="1" step="0.05" style="width:5rem"
@@ -8240,16 +8356,29 @@ Every piece is hashed against the ` +
8240
8356
  <div class="row">
8241
8357
  <span class="muted">${index}</span>
8242
8358
  <code>${escapeHtml(
8243
- source.stack ?? source.category ?? shortHash(source.archive) ?? '',
8359
+ remote
8360
+ ? (source.url || 'an address').replace(/^https?:\/\//, '')
8361
+ : (source.stack ??
8362
+ source.category ??
8363
+ shortHash(source.archive) ??
8364
+ ''),
8244
8365
  )}</code>
8245
8366
  <span class="sub" title="${
8246
- source.archive
8247
- ? 'Pinned to this build. It will not follow a rebuild, and breaks if the archive is removed.'
8248
- : source.stack
8249
- ? 'Another stack, merged as one layer. It follows every later change to that recipe, and to whatever it resolves to.'
8250
- : 'Follows whichever build in this category is newest.'
8367
+ remote
8368
+ ? 'Read straight from this address over HTTP, with byte ranges. Nothing is downloaded, nothing is seeded, and this node never rebuilds or retires it.'
8369
+ : source.archive
8370
+ ? 'Pinned to this build. It will not follow a rebuild, and breaks if the archive is removed.'
8371
+ : source.stack
8372
+ ? 'Another stack, merged as one layer. It follows every later change to that recipe, and to whatever it resolves to.'
8373
+ : 'Follows whichever build in this category is newest.'
8251
8374
  }">${
8252
- source.archive ? 'pinned' : source.stack ? 'stack' : 'category'
8375
+ remote
8376
+ ? 'url'
8377
+ : source.archive
8378
+ ? 'pinned'
8379
+ : source.stack
8380
+ ? 'stack'
8381
+ : 'category'
8253
8382
  }</span>
8254
8383
  ${role}
8255
8384
  <span class="bar-gap"></span>
@@ -8259,6 +8388,38 @@ Every piece is hashed against the ` +
8259
8388
  ${last ? 'disabled' : ''} title="Later: covers more">↓</button>
8260
8389
  <button type="button" data-stack-remove="${index}">Remove</button>
8261
8390
  </div>
8391
+ ${
8392
+ remote
8393
+ ? `<div class="bar">
8394
+ <label class="choice" style="flex:1">Address
8395
+ <input style="flex:1;min-width:22rem" placeholder="https://example.org/terrain.pmtiles or s3://bucket/terrain.pmtiles"
8396
+ data-stack-field="url" data-stack-index="${index}"
8397
+ value="${escapeHtml(source.url ?? '')}" />
8398
+ </label>
8399
+ ${
8400
+ source.url && !isSourceAddress(source.url)
8401
+ ? '<span class="bad">http://, https:// or s3://bucket/key</span>'
8402
+ : ''
8403
+ }
8404
+ </div>
8405
+ <div class="bar">
8406
+ <label class="choice"
8407
+ title="The zooms this archive actually holds. Checked before anything is opened, so a tile outside the range skips it without a request leaving this node — which is what makes a stack of many of these cheap. Leave empty and every tile asks it.">
8408
+ Only zooms
8409
+ <input type="number" min="0" max="24" step="1" style="width:4.5rem"
8410
+ placeholder="any"
8411
+ data-stack-field="minzoom" data-stack-index="${index}"
8412
+ value="${source.minzoom ?? ''}" />
8413
+ to
8414
+ <input type="number" min="0" max="24" step="1" style="width:4.5rem"
8415
+ placeholder="any"
8416
+ data-stack-field="maxzoom" data-stack-index="${index}"
8417
+ value="${source.maxzoom ?? ''}" />
8418
+ </label>
8419
+ <span class="sub">Clip it to a box below to skip it by area too.</span>
8420
+ </div>`
8421
+ : ''
8422
+ }
8262
8423
  <div class="bar">
8263
8424
  ${perSpace}
8264
8425
  ${
@@ -8402,6 +8563,12 @@ Every piece is hashed against the ` +
8402
8563
  source.featherMetres = width;
8403
8564
  }
8404
8565
  }
8566
+ } else if (field === 'minzoom' || field === 'maxzoom') {
8567
+ // Blank means "every zoom", so an empty box removes the field
8568
+ // rather than writing 0 -- which for minzoom is the same thing and
8569
+ // for maxzoom would silence the source everywhere above z0.
8570
+ if (event.target.value === '') delete source[field];
8571
+ else source[field] = Math.round(Number(event.target.value));
8405
8572
  } else if (field === 'baseVal' || field === 'interval') {
8406
8573
  // Blank means "the default", not zero -- an interval of 0 would make
8407
8574
  // every height the base value.
@@ -8499,7 +8666,12 @@ Every piece is hashed against the ` +
8499
8666
  ? { archive: name }
8500
8667
  : kind === 'stk'
8501
8668
  ? { stack: name }
8502
- : { category: name },
8669
+ : kind === 'url'
8670
+ ? // Empty, because the address is typed into the card rather
8671
+ // than chosen from a menu. It is refused on save until it
8672
+ // is filled in.
8673
+ { url: '' }
8674
+ : { category: name },
8503
8675
  );
8504
8676
  renderStackDraft();
8505
8677
  fillStackSourceChoices();
@@ -8546,6 +8718,23 @@ Every piece is hashed against the ` +
8546
8718
  'is made of, so saving would replace it rather than add this one.';
8547
8719
  return;
8548
8720
  }
8721
+ // A URL source is the one kind whose name is typed rather than
8722
+ // chosen, so it is the one kind that can be left half-made. Caught
8723
+ // here because the reply to a recipe the node refuses says which
8724
+ // source is wrong by index, which is not where the operator is
8725
+ // looking.
8726
+ const blank = stackDraft.sources.findIndex(
8727
+ (source) =>
8728
+ source.url !== undefined && !isSourceAddress(source.url ?? ''),
8729
+ );
8730
+ if (blank >= 0) {
8731
+ $('stack-error').textContent =
8732
+ `Source ${blank} needs an address: http://, https://, or ` +
8733
+ 's3://bucket/key for a bucket set up under Settings → Feeds. It ' +
8734
+ 'is read straight from wherever it is, so there is nothing else ' +
8735
+ 'to find it by.';
8736
+ return;
8737
+ }
8549
8738
  const body = {
8550
8739
  title: $('stack-title').value.trim() || undefined,
8551
8740
  space: stackDraft.space,