pmtiles-swarm 0.90.0 → 0.91.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +56 -0
- package/docs/configuration.md +91 -0
- package/docs/tile-stacks.md +24 -0
- package/package.json +1 -1
- package/src/api.js +58 -39
- package/src/bake-jobs.js +22 -9
- package/src/config.js +23 -0
- package/src/index.js +4 -0
- package/src/pmtiles-write.js +8 -3
- package/src/s3-source.js +302 -0
- package/src/stacks.js +26 -4
- package/src/tiles.js +55 -1
- package/src/web/index.html +215 -26
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,62 @@
|
|
|
7
7
|
### 🐞 Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.91.0
|
|
11
|
+
### ✨ Features and improvements
|
|
12
|
+
- **A source at a URL can be added by hand, not only imported.** **Add source → an address you
|
|
13
|
+
type…** in the stack editor, writing the same source the importer does. The card asks for the
|
|
14
|
+
address and for the zoom range the archive holds — a catalog archive states its range in its own
|
|
15
|
+
header and this has no header anybody has read, so leaving it empty means every tile asks it.
|
|
16
|
+
Anything published as a plain HTTPS download works, an S3 object or presigned URL included, and so
|
|
17
|
+
does `s3://bucket/key.pmtiles` for a private bucket — see below.
|
|
18
|
+
|
|
19
|
+
- **A stack's source list folds up past five sources.** An imported list is several hundred rows,
|
|
20
|
+
which buried every other stack on the page under one of them. The fold says how many there are and
|
|
21
|
+
names the base; a stack of a base and a layer or two stays open, since folding that hides nothing
|
|
22
|
+
worth a click.
|
|
23
|
+
|
|
24
|
+
|
|
25
|
+
- **A stack source may be an object in a private S3 bucket.**
|
|
26
|
+
`s3://bucket/terrain.pmtiles`, read with a signed request per byte range. Only for a bucket that is
|
|
27
|
+
not public — a public object or a presigned URL is an ordinary HTTPS address and always worked.
|
|
28
|
+
`endpoint` is what makes it S3-compatible rather than S3: MinIO, Ceph, Garage, R2, Wasabi and B2
|
|
29
|
+
all answer the same protocol at their own address, and path-style addressing is the default because
|
|
30
|
+
it is what they speak. Credentials go under **Settings → Feeds → S3 buckets**, per bucket or once
|
|
31
|
+
for an account; with none configured the standard `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
|
|
32
|
+
`AWS_REGION` and `AWS_S3_ENDPOINT` variables are read, so a machine already set up to reach a
|
|
33
|
+
bucket needs nothing typed in.
|
|
34
|
+
|
|
35
|
+
Signed here rather than by an SDK: SigV4 for a GET is a hash, four HMACs and a string, all of which
|
|
36
|
+
`node:crypto` has. An AWS client would be a large dependency for one signature, and an optional one
|
|
37
|
+
would make reading a bucket work on some installs and not on others for no reason the operator
|
|
38
|
+
could see — so it is neither optional nor a dependency. The signer is checked against AWS's own
|
|
39
|
+
published test vectors rather than against itself.
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
### 🐞 Bug fixes
|
|
43
|
+
- **A stack of remote sources could be served but not exported.** The bake refused any stack whose
|
|
44
|
+
sources resolved to no archive on this node, which is every stack read from URLs — refusing exactly
|
|
45
|
+
the export worth having, since baking is how terrain that lives somewhere else becomes something
|
|
46
|
+
this node holds and can seed. It also had no way to walk a remote archive's directories, which is
|
|
47
|
+
how a bake finds which tiles exist at all.
|
|
48
|
+
|
|
49
|
+
- **An archive whose deepest tiles repeat said it stopped a zoom short.** Identical tiles collapse
|
|
50
|
+
into one directory entry covering a range of ids, so the last entry's own id can sit at a shallower
|
|
51
|
+
zoom than the tiles it addresses — and the header was read off that id. A client asks for nothing
|
|
52
|
+
past a header's `maxZoom`, so the deepest zoom of such an archive was unreachable. Found baking a
|
|
53
|
+
stack over flat terrain, where every tile of the last zoom was the same tile.
|
|
54
|
+
|
|
55
|
+
- **A stack used as a source of another stack was reported as unresolved.** It resolves to a recipe
|
|
56
|
+
rather than to bytes, and the console's reader looked only for a catalog entry or an address — so a
|
|
57
|
+
working nested stack showed "does not resolve", with no zoom range and no box, and put a "sources
|
|
58
|
+
missing" badge on the stack above it. It now says what it stands for and how deep it reaches,
|
|
59
|
+
worked out from the sources that stack would serve.
|
|
60
|
+
|
|
61
|
+
- **A stack's advertised box ignored any source that stated none.** The union was taken over
|
|
62
|
+
whichever sources had a box, so a stack of a global base plus regional patches — an imported list,
|
|
63
|
+
exactly — advertised the patches as its extent and left the base out of it. A source with no box
|
|
64
|
+
is not one covering nothing, so it is now every box or none, and none means the world.
|
|
65
|
+
|
|
10
66
|
## 0.90.0
|
|
11
67
|
### ✨ Features and improvements
|
|
12
68
|
- **A stack source may name a URL instead of a category, archive or stack.** `{ "url": "https://…" }`,
|
package/docs/configuration.md
CHANGED
|
@@ -20,6 +20,7 @@ Some settings only take effect on restart. Those are marked **restart**.
|
|
|
20
20
|
- [Scheduled sources](#scheduled-sources)
|
|
21
21
|
- [Subscriptions](#subscriptions)
|
|
22
22
|
- [Serving tiles](#serving-tiles)
|
|
23
|
+
- [S3 buckets](#s3-buckets)
|
|
23
24
|
- [Mutable publishing](#mutable-publishing)
|
|
24
25
|
- [Authentication](#authentication)
|
|
25
26
|
- [Statistics](#statistics)
|
|
@@ -776,6 +777,96 @@ connection, a piece already in flight. Ten attempts later it is one piece at the
|
|
|
776
777
|
far end of an archive nobody has finished, and asking every fifteen seconds
|
|
777
778
|
achieves nothing but log lines.
|
|
778
779
|
|
|
780
|
+
## S3 buckets
|
|
781
|
+
|
|
782
|
+
A stack source may be an address rather than an archive this node holds, and
|
|
783
|
+
that address may name an object in a bucket:
|
|
784
|
+
|
|
785
|
+
```json
|
|
786
|
+
{
|
|
787
|
+
"sources": [{ "url": "s3://terrain/planet.pmtiles", "encoding": "terrarium" }]
|
|
788
|
+
}
|
|
789
|
+
```
|
|
790
|
+
|
|
791
|
+
Only for a bucket that is **not public**. A public object, or a presigned URL,
|
|
792
|
+
is an ordinary HTTPS address: write it as one and none of this applies.
|
|
793
|
+
|
|
794
|
+
```json
|
|
795
|
+
{
|
|
796
|
+
"s3": [
|
|
797
|
+
{
|
|
798
|
+
"bucket": "terrain",
|
|
799
|
+
"endpoint": "https://minio.lan:9000",
|
|
800
|
+
"region": "us-east-1",
|
|
801
|
+
"accessKeyId": "…",
|
|
802
|
+
"secretAccessKey": "…"
|
|
803
|
+
}
|
|
804
|
+
]
|
|
805
|
+
}
|
|
806
|
+
```
|
|
807
|
+
|
|
808
|
+
| field | default | |
|
|
809
|
+
| ----------------- | ------------------ | ---------------------------------------------------- |
|
|
810
|
+
| `bucket` | any | which bucket this row is for; unset matches all |
|
|
811
|
+
| `endpoint` | AWS for the region | what makes it S3-compatible rather than S3 |
|
|
812
|
+
| `region` | `us-east-1` | signed into every request, whether or not it is used |
|
|
813
|
+
| `accessKeyId` | — | |
|
|
814
|
+
| `secretAccessKey` | — | |
|
|
815
|
+
| `sessionToken` | unset | for temporary credentials |
|
|
816
|
+
| `pathStyle` | path, unless AWS | `false` puts the bucket in the hostname |
|
|
817
|
+
|
|
818
|
+
A source is read through the row naming its bucket, or through whichever row
|
|
819
|
+
names none: one set of credentials for a whole account is the ordinary case,
|
|
820
|
+
and writing the same keys once per bucket is busywork with a copy-paste
|
|
821
|
+
mistake in it.
|
|
822
|
+
|
|
823
|
+
### The endpoint is the whole point
|
|
824
|
+
|
|
825
|
+
MinIO, Ceph, Garage, Cloudflare R2, Wasabi and Backblaze B2 all answer the
|
|
826
|
+
same protocol at their own address, so naming that address is the difference
|
|
827
|
+
between "S3" and "S3-compatible". Path-style addressing is what they speak and
|
|
828
|
+
is the default here; AWS withdrew it for buckets made after September 2020, so
|
|
829
|
+
an `amazonaws.com` endpoint puts the bucket in the hostname instead. Set
|
|
830
|
+
`pathStyle` only for a server that insists on the other one.
|
|
831
|
+
|
|
832
|
+
### Credentials from the environment
|
|
833
|
+
|
|
834
|
+
With no rows configured at all, the standard variables are read — the same
|
|
835
|
+
ones the AWS CLI, rclone, go-pmtiles and tileserver-gl use, so a machine
|
|
836
|
+
already set up to reach a bucket needs nothing written here:
|
|
837
|
+
|
|
838
|
+
| variable | |
|
|
839
|
+
| -------------------------------------------- | ------------------------------- |
|
|
840
|
+
| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | required for any of it to apply |
|
|
841
|
+
| `AWS_SESSION_TOKEN` | temporary credentials |
|
|
842
|
+
| `AWS_REGION`, `AWS_DEFAULT_REGION` | region |
|
|
843
|
+
| `AWS_S3_ENDPOINT`, `AWS_ENDPOINT_URL_S3` | endpoint |
|
|
844
|
+
| `AWS_S3_FORCE_PATH_STYLE` | addressing |
|
|
845
|
+
|
|
846
|
+
### Signed here, not by an SDK
|
|
847
|
+
|
|
848
|
+
SigV4 for a GET is a hash, four HMACs and a string, all of which `node:crypto`
|
|
849
|
+
already has. An AWS client would be a large dependency for one signature, and
|
|
850
|
+
an optional one would make reading a bucket work on some installs and not on
|
|
851
|
+
others for no reason the operator could see — so it is neither optional nor a
|
|
852
|
+
dependency. The signer is checked against AWS's own published test vectors
|
|
853
|
+
rather than against itself.
|
|
854
|
+
|
|
855
|
+
Every request is signed as it goes out, including the range: the header, each
|
|
856
|
+
directory and each tile. Nothing is presigned and nothing is cached but the
|
|
857
|
+
bytes, so revoking a key stops reads at the next request rather than at the
|
|
858
|
+
next restart. Changing the settings drops the open readers for the same
|
|
859
|
+
reason.
|
|
860
|
+
|
|
861
|
+
### What a bucket source is, and is not
|
|
862
|
+
|
|
863
|
+
It is a URL source in every other respect — see
|
|
864
|
+
[tile-stacks.md](tile-stacks.md), "A source read straight from a URL". It
|
|
865
|
+
carries its own `minzoom`, `maxzoom` and `bounds`, it is never seeded or
|
|
866
|
+
retired, and it can be baked: a stack over private buckets exports to an
|
|
867
|
+
ordinary archive with an infohash, which is the path from "terrain we hold in
|
|
868
|
+
a bucket" to "terrain we publish".
|
|
869
|
+
|
|
779
870
|
## Mutable publishing
|
|
780
871
|
|
|
781
872
|
Announcing the current build of each category over the DHT (BEP 46).
|
package/docs/tile-stacks.md
CHANGED
|
@@ -1681,6 +1681,22 @@ archive — Mapterhorn's own base stops at z12 — is upscaled for a deeper
|
|
|
1681
1681
|
request exactly as GEBCO is, through the same code, because reading one is now
|
|
1682
1682
|
a question of which store method answers and nothing else.
|
|
1683
1683
|
|
|
1684
|
+
### Adding one by hand
|
|
1685
|
+
|
|
1686
|
+
**Add source → an address you type…** in the stack editor, which is the same
|
|
1687
|
+
source the importer writes and edits the same way. The card asks for two
|
|
1688
|
+
things nothing else can know: the address, and the zoom range the archive
|
|
1689
|
+
holds. A catalog archive states its range in its own header and a URL source
|
|
1690
|
+
has no header this node has read, so an unstated range means every tile asks
|
|
1691
|
+
it — correct, and the thing worth avoiding once there are several.
|
|
1692
|
+
|
|
1693
|
+
Anything published as a plain HTTPS download works, which includes an S3
|
|
1694
|
+
bucket that serves range requests: an object URL, or a presigned one, is an
|
|
1695
|
+
address like any other, and this asks for byte ranges the way any PMTiles
|
|
1696
|
+
reader does. What it cannot do is a private bucket named `s3://…`, which is
|
|
1697
|
+
not a URL a browser or a fetch can follow — that needs a signed request and
|
|
1698
|
+
somewhere to keep the credentials, and neither exists here yet.
|
|
1699
|
+
|
|
1684
1700
|
### Cheap to have hundreds of
|
|
1685
1701
|
|
|
1686
1702
|
The one thing that makes hundreds of these practical rather than merely
|
|
@@ -1830,6 +1846,14 @@ that is a map changing at 4am with nothing to say why. A hand-written source
|
|
|
1830
1846
|
carries no `importedFrom`, is never replaced, and is the place for anything the
|
|
1831
1847
|
index could not have told us.
|
|
1832
1848
|
|
|
1849
|
+
### Seeing them in the console
|
|
1850
|
+
|
|
1851
|
+
A stack's source list collapses behind a summary once there are more than
|
|
1852
|
+
five, with the count and the base named on the fold. A stack imported from a
|
|
1853
|
+
provider's list is several hundred rows and drawing them flat buries every
|
|
1854
|
+
other stack on the page under one of them; a stack of a base and a layer or
|
|
1855
|
+
two stays open, since folding that hides nothing worth a click.
|
|
1856
|
+
|
|
1833
1857
|
## Finding a stack
|
|
1834
1858
|
|
|
1835
1859
|
A stack has no infohash and appears in no feed, so nothing about it is
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.91.0",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
package/src/api.js
CHANGED
|
@@ -3400,45 +3400,64 @@ export function createApp({
|
|
|
3400
3400
|
// The order the recipe lists them: lowest priority first, last
|
|
3401
3401
|
// wins. The console shows this order as it stands rather than
|
|
3402
3402
|
// inverting it, so the screen and the file never disagree.
|
|
3403
|
-
sources: resolved.sources.map((source) =>
|
|
3404
|
-
|
|
3405
|
-
|
|
3406
|
-
|
|
3407
|
-
|
|
3408
|
-
|
|
3409
|
-
|
|
3410
|
-
|
|
3411
|
-
|
|
3412
|
-
|
|
3413
|
-
|
|
3414
|
-
|
|
3415
|
-
|
|
3416
|
-
|
|
3417
|
-
|
|
3418
|
-
|
|
3419
|
-
|
|
3420
|
-
|
|
3421
|
-
|
|
3422
|
-
|
|
3423
|
-
|
|
3424
|
-
|
|
3425
|
-
|
|
3426
|
-
|
|
3427
|
-
|
|
3428
|
-
|
|
3429
|
-
|
|
3430
|
-
|
|
3431
|
-
|
|
3432
|
-
|
|
3433
|
-
|
|
3434
|
-
|
|
3435
|
-
|
|
3436
|
-
source.
|
|
3437
|
-
|
|
3438
|
-
|
|
3439
|
-
|
|
3440
|
-
|
|
3441
|
-
|
|
3403
|
+
sources: resolved.sources.map((source) => {
|
|
3404
|
+
// A nested stack answers for the ground its own sources cover,
|
|
3405
|
+
// and knows it without opening anything -- so the row can say
|
|
3406
|
+
// what it reaches rather than a dash.
|
|
3407
|
+
const inner = source.nested ? stackCoverage(source.nested) : null;
|
|
3408
|
+
return {
|
|
3409
|
+
index: source.index,
|
|
3410
|
+
name: source.name,
|
|
3411
|
+
pinned: source.pinned,
|
|
3412
|
+
required: source.required,
|
|
3413
|
+
// What kind of source this is, said outright rather than left to
|
|
3414
|
+
// be inferred from which fields happen to be null. A URL source
|
|
3415
|
+
// has no infohash and no catalog entry, so a reader guessing from
|
|
3416
|
+
// those calls it an unresolved category -- which is two wrong
|
|
3417
|
+
// things about a source that is working perfectly well.
|
|
3418
|
+
kind: source.remote
|
|
3419
|
+
? 'url'
|
|
3420
|
+
: source.nested
|
|
3421
|
+
? 'stack'
|
|
3422
|
+
: source.pinned
|
|
3423
|
+
? 'archive'
|
|
3424
|
+
: 'category',
|
|
3425
|
+
// A URL source resolves by definition: there is nothing to look
|
|
3426
|
+
// up, the address is the answer. Whether it can actually be read
|
|
3427
|
+
// is a question for the first tile, the same as for an archive
|
|
3428
|
+
// this node holds but cannot open.
|
|
3429
|
+
resolved:
|
|
3430
|
+
Boolean(source.entry) ||
|
|
3431
|
+
Boolean(source.remote) ||
|
|
3432
|
+
Boolean(source.nested),
|
|
3433
|
+
// How many sources the nested recipe has, which is the only
|
|
3434
|
+
// useful thing to say in a column that names an archive file for
|
|
3435
|
+
// every other kind: a stack resolves to a recipe, not to bytes.
|
|
3436
|
+
nested: source.nested ? source.nested.sources.length : null,
|
|
3437
|
+
url: source.remote ?? null,
|
|
3438
|
+
infohash: source.entry?.infoHash ?? null,
|
|
3439
|
+
archiveName: source.entry?.name ?? null,
|
|
3440
|
+
// The recipe's own, where it states them. For a URL source that is
|
|
3441
|
+
// the only place they are known without opening the file, and it
|
|
3442
|
+
// is what the merge itself uses to decide whether to.
|
|
3443
|
+
minzoom:
|
|
3444
|
+
source.entry?.pmtiles?.minZoom ??
|
|
3445
|
+
source.source?.minzoom ??
|
|
3446
|
+
inner?.minzoom ??
|
|
3447
|
+
null,
|
|
3448
|
+
maxzoom:
|
|
3449
|
+
source.entry?.pmtiles?.maxZoom ??
|
|
3450
|
+
source.source?.maxzoom ??
|
|
3451
|
+
inner?.maxzoom ??
|
|
3452
|
+
null,
|
|
3453
|
+
bounds:
|
|
3454
|
+
source.entry?.pmtiles?.bounds ??
|
|
3455
|
+
(Array.isArray(source.source?.bounds)
|
|
3456
|
+
? source.source.bounds
|
|
3457
|
+
: (inner?.bounds ?? null)),
|
|
3458
|
+
format: source.entry?.pmtiles?.format ?? null,
|
|
3459
|
+
};
|
|
3460
|
+
}),
|
|
3442
3461
|
};
|
|
3443
3462
|
});
|
|
3444
3463
|
// Reported so the console can say "install sharp" beside the stacks
|
package/src/bake-jobs.js
CHANGED
|
@@ -411,12 +411,18 @@ export class BakeManager {
|
|
|
411
411
|
// presses the button rather than an hour in.
|
|
412
412
|
assertBakeable(resolved, codec);
|
|
413
413
|
|
|
414
|
+
// A source this node can read: an archive it holds, a recipe it can
|
|
415
|
+
// evaluate, or an address it can fetch. The last was missing, which
|
|
416
|
+
// refused exactly the bake worth having -- a stack of remote sources
|
|
417
|
+
// exported to an archive is how terrain that lives somewhere else, in a
|
|
418
|
+
// provider's file list or a private bucket, becomes something this node
|
|
419
|
+
// holds and can seed.
|
|
414
420
|
const unresolved = resolved.sources.filter(
|
|
415
|
-
(source) => !source.entry && !source.nested,
|
|
421
|
+
(source) => !source.entry && !source.nested && !source.remote,
|
|
416
422
|
);
|
|
417
423
|
if (unresolved.length === resolved.sources.length) {
|
|
418
424
|
const error = new Error(
|
|
419
|
-
"none of this stack's sources resolved to
|
|
425
|
+
"none of this stack's sources resolved to anything this node can read",
|
|
420
426
|
);
|
|
421
427
|
error.status = 409;
|
|
422
428
|
throw error;
|
|
@@ -594,15 +600,22 @@ export class BakeManager {
|
|
|
594
600
|
// source is scanned the same way it is served: its directories come out of
|
|
595
601
|
// the swarm, and the store holds them to its own byte budget.
|
|
596
602
|
const sources = resolved.sources
|
|
597
|
-
.filter((source) => source.entry)
|
|
603
|
+
.filter((source) => source.entry || source.remote)
|
|
598
604
|
.map((source) => ({
|
|
599
605
|
getBytes: async (offset, length) => {
|
|
600
|
-
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
+
// The one difference between a source this node holds and one it
|
|
607
|
+
// reads over the network, here as everywhere else: which method of
|
|
608
|
+
// the store answers. What is done with the bytes is identical.
|
|
609
|
+
const bytes = source.remote
|
|
610
|
+
? await this.#tiles.readRemoteRange(source.remote, offset, length, {
|
|
611
|
+
signal: job.controller.signal,
|
|
612
|
+
})
|
|
613
|
+
: await this.#tiles.readRange(
|
|
614
|
+
source.entry.infoHash,
|
|
615
|
+
offset,
|
|
616
|
+
length,
|
|
617
|
+
{ signal: job.controller.signal },
|
|
618
|
+
);
|
|
606
619
|
// Sliced by offset and length rather than handed `.buffer` outright.
|
|
607
620
|
// A Buffer is a view, and where it is a view *into* something larger
|
|
608
621
|
// -- which is what Buffer.allocUnsafe hands back -- `.buffer` is the
|
package/src/config.js
CHANGED
|
@@ -459,6 +459,25 @@ const DEFAULTS = {
|
|
|
459
459
|
* See docs/tile-stacks.md -- "Syncing a stack to another node".
|
|
460
460
|
*/
|
|
461
461
|
stackFeeds: [],
|
|
462
|
+
/**
|
|
463
|
+
* S3-compatible buckets a stack source may be read from.
|
|
464
|
+
*
|
|
465
|
+
* `[{ bucket, endpoint, region, accessKeyId, secretAccessKey, sessionToken,
|
|
466
|
+
* pathStyle }]`. A source whose URL is `s3://bucket/key.pmtiles` is read
|
|
467
|
+
* through the row naming that bucket, or through whichever row names none.
|
|
468
|
+
* A row is needed only for a bucket that is not public: a presigned or
|
|
469
|
+
* public HTTPS address is read as any other URL and knows nothing of this.
|
|
470
|
+
*
|
|
471
|
+
* `endpoint` is what makes it S3-compatible rather than S3: MinIO, Ceph,
|
|
472
|
+
* Garage, R2, Wasabi and B2 all answer the same protocol at their own
|
|
473
|
+
* address. Left unset it is AWS's own for the region.
|
|
474
|
+
*
|
|
475
|
+
* With nothing here at all the standard `AWS_ACCESS_KEY_ID`,
|
|
476
|
+
* `AWS_SECRET_ACCESS_KEY`, `AWS_REGION` and `AWS_S3_ENDPOINT` variables are
|
|
477
|
+
* read, so a machine already set up to reach a bucket needs no settings.
|
|
478
|
+
* See docs/configuration.md -- "S3 buckets".
|
|
479
|
+
*/
|
|
480
|
+
s3: [],
|
|
462
481
|
/**
|
|
463
482
|
* How often to re-check whether the sources archives were built from have
|
|
464
483
|
* changed, in seconds. Zero disables it.
|
|
@@ -861,6 +880,10 @@ export const RELOADABLE = new Map([
|
|
|
861
880
|
['sourceCheckIntervalHours', 'sources'],
|
|
862
881
|
['subscriptions', 'subscriptions'],
|
|
863
882
|
['stackFeeds', 'stackFeeds'],
|
|
883
|
+
// Read when a bucket-backed source is first opened, and the open handles are
|
|
884
|
+
// keyed by address rather than by credentials -- so a corrected key applies
|
|
885
|
+
// to the next archive opened rather than needing a restart.
|
|
886
|
+
['s3', 's3'],
|
|
864
887
|
['subscriptionIntervalSeconds', 'subscriptions'],
|
|
865
888
|
['subscriptionsEnabled', 'subscriptions'],
|
|
866
889
|
['seeding', 'seeding'],
|
package/src/index.js
CHANGED
|
@@ -586,6 +586,10 @@ PMTILES_SWARM_PUBLIC_URL
|
|
|
586
586
|
stackFeeds.stop();
|
|
587
587
|
stackFeeds.start();
|
|
588
588
|
},
|
|
589
|
+
// Nothing to restart: the next archive opened from a bucket reads the new
|
|
590
|
+
// settings. What has to go is the readers already open, which are holding
|
|
591
|
+
// the keys they were opened with.
|
|
592
|
+
s3: () => tiles.forgetRemote(),
|
|
589
593
|
seeding: () => {
|
|
590
594
|
seeding.stop();
|
|
591
595
|
seeding.start();
|
package/src/pmtiles-write.js
CHANGED
|
@@ -459,9 +459,14 @@ export class PMTilesWriter {
|
|
|
459
459
|
// client asks for, and a header disagreeing with the directory sends it
|
|
460
460
|
// looking for tiles that are not there.
|
|
461
461
|
minZoom: header.minZoom ?? zoomOf(this.#entries[0].tileId),
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
462
|
+
// The last id written, not the last entry's own. Identical tiles
|
|
463
|
+
// collapse into one entry covering a range, so an archive whose deepest
|
|
464
|
+
// tiles repeat -- open ocean, a masked-out region, anything flat --
|
|
465
|
+
// ends with an entry whose id sits at a shallower zoom than the tiles
|
|
466
|
+
// it addresses. Read as the entry's id, the header said the archive
|
|
467
|
+
// stopped short of the zoom it reaches, and a client asks for nothing
|
|
468
|
+
// past a header's maxZoom.
|
|
469
|
+
maxZoom: header.maxZoom ?? zoomOf(this.#lastId),
|
|
465
470
|
};
|
|
466
471
|
|
|
467
472
|
const out = await fs.open(destination, 'w');
|
package/src/s3-source.js
ADDED
|
@@ -0,0 +1,302 @@
|
|
|
1
|
+
import crypto from 'node:crypto';
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* Reading a PMTiles archive out of an S3-compatible bucket.
|
|
5
|
+
*
|
|
6
|
+
* A public or presigned URL needs nothing from this file: it is an address,
|
|
7
|
+
* `FetchSource` asks it for byte ranges and the bucket answers. What needs
|
|
8
|
+
* signing is the ordinary private object, and that is the case worth having --
|
|
9
|
+
* it is the difference between terrain somebody published and terrain this
|
|
10
|
+
* node holds, and it is what lets a stack read a bucket and bake the result
|
|
11
|
+
* into an archive without the bucket ever being public.
|
|
12
|
+
*
|
|
13
|
+
* Signed here rather than through an SDK. SigV4 for a GET is a hash, four
|
|
14
|
+
* HMACs and a string, all of which `node:crypto` already has; the AWS client
|
|
15
|
+
* that would do it instead is a large dependency for one signature, and an
|
|
16
|
+
* optional one would make reading a bucket work on some installs and not on
|
|
17
|
+
* others for no reason the operator could see.
|
|
18
|
+
*
|
|
19
|
+
* See docs/configuration.md -- "S3 buckets".
|
|
20
|
+
*/
|
|
21
|
+
|
|
22
|
+
/** What SigV4 calls an empty body, which every GET has. */
|
|
23
|
+
const EMPTY_SHA256 = crypto.createHash('sha256').update('').digest('hex');
|
|
24
|
+
|
|
25
|
+
/**
|
|
26
|
+
* HMAC-SHA256, the only primitive the signature needs.
|
|
27
|
+
* @param {Buffer|string} key - The key.
|
|
28
|
+
* @param {string} data - What to sign.
|
|
29
|
+
* @returns {Buffer} - The digest.
|
|
30
|
+
*/
|
|
31
|
+
const hmac = (key, data) =>
|
|
32
|
+
crypto.createHmac('sha256', key).update(data, 'utf8').digest();
|
|
33
|
+
|
|
34
|
+
/**
|
|
35
|
+
* Query parameters in the order and encoding SigV4 asks for.
|
|
36
|
+
* @param {URL} url - The address being signed.
|
|
37
|
+
* @returns {string} - The canonical query string.
|
|
38
|
+
*/
|
|
39
|
+
const canonicalQuery = (url) => {
|
|
40
|
+
const pairs = [...url.searchParams.entries()].map(([name, value]) => [
|
|
41
|
+
encodeURIComponent(name),
|
|
42
|
+
encodeURIComponent(value),
|
|
43
|
+
]);
|
|
44
|
+
pairs.sort((one, two) =>
|
|
45
|
+
one[0] === two[0] ? one[1].localeCompare(two[1]) : one[0] < two[0] ? -1 : 1,
|
|
46
|
+
);
|
|
47
|
+
return pairs.map(([name, value]) => `${name}=${value}`).join('&');
|
|
48
|
+
};
|
|
49
|
+
|
|
50
|
+
/**
|
|
51
|
+
* The `Authorization` header for one request.
|
|
52
|
+
*
|
|
53
|
+
* Signs exactly the headers it is given, which is what makes it checkable
|
|
54
|
+
* against AWS's own published test vectors rather than only against itself.
|
|
55
|
+
* @param {object} request - `method`, `url`, `headers`, `payloadHash`.
|
|
56
|
+
* @param {object} credentials - `accessKeyId`, `secretAccessKey`, `region`,
|
|
57
|
+
* `service` and `amzDate`, which is `YYYYMMDDTHHMMSSZ`.
|
|
58
|
+
* @returns {string} - The header value.
|
|
59
|
+
*/
|
|
60
|
+
export function authorization(request, credentials) {
|
|
61
|
+
const url = new URL(request.url);
|
|
62
|
+
const { amzDate } = credentials;
|
|
63
|
+
const day = amzDate.slice(0, 8);
|
|
64
|
+
const service = credentials.service ?? 's3';
|
|
65
|
+
const scope = `${day}/${credentials.region}/${service}/aws4_request`;
|
|
66
|
+
|
|
67
|
+
const named = Object.entries(request.headers ?? {})
|
|
68
|
+
.map(([name, value]) => [name.toLowerCase(), String(value).trim()])
|
|
69
|
+
.sort((one, two) => (one[0] < two[0] ? -1 : 1));
|
|
70
|
+
const signed = named.map(([name]) => name).join(';');
|
|
71
|
+
const canonicalHeaders = named
|
|
72
|
+
.map(([name, value]) => `${name}:${value}\n`)
|
|
73
|
+
.join('');
|
|
74
|
+
|
|
75
|
+
// The path as it appears on the wire rather than a normalised form of it:
|
|
76
|
+
// S3 signs what it is sent, so a key holding a literal per-cent sign signs
|
|
77
|
+
// with that sign encoded, and re-encoding here would sign something else.
|
|
78
|
+
const canonical = [
|
|
79
|
+
request.method ?? 'GET',
|
|
80
|
+
url.pathname || '/',
|
|
81
|
+
canonicalQuery(url),
|
|
82
|
+
canonicalHeaders,
|
|
83
|
+
signed,
|
|
84
|
+
request.payloadHash ?? EMPTY_SHA256,
|
|
85
|
+
].join('\n');
|
|
86
|
+
|
|
87
|
+
const toSign = [
|
|
88
|
+
'AWS4-HMAC-SHA256',
|
|
89
|
+
amzDate,
|
|
90
|
+
scope,
|
|
91
|
+
crypto.createHash('sha256').update(canonical).digest('hex'),
|
|
92
|
+
].join('\n');
|
|
93
|
+
|
|
94
|
+
const key = ['aws4_request', service, credentials.region, day].reduceRight(
|
|
95
|
+
(acc, part) => hmac(acc, part),
|
|
96
|
+
`AWS4${credentials.secretAccessKey}`,
|
|
97
|
+
);
|
|
98
|
+
const signature = hmac(key, toSign).toString('hex');
|
|
99
|
+
|
|
100
|
+
return (
|
|
101
|
+
`AWS4-HMAC-SHA256 Credential=${credentials.accessKeyId}/${scope}, ` +
|
|
102
|
+
`SignedHeaders=${signed}, Signature=${signature}`
|
|
103
|
+
);
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
/**
|
|
107
|
+
* Whether an address names an object in a bucket rather than a web server.
|
|
108
|
+
* @param {string} value - The address.
|
|
109
|
+
* @returns {boolean} - True for `s3://bucket/key`.
|
|
110
|
+
*/
|
|
111
|
+
export const isS3Url = (value) =>
|
|
112
|
+
typeof value === 'string' && /^s3:\/\/[^/]+\/.+/i.test(value);
|
|
113
|
+
|
|
114
|
+
/**
|
|
115
|
+
* The bucket and key an `s3://` address names.
|
|
116
|
+
* @param {string} value - The address.
|
|
117
|
+
* @returns {object} - `{bucket, key}`.
|
|
118
|
+
*/
|
|
119
|
+
export function splitS3Url(value) {
|
|
120
|
+
const rest = String(value).slice('s3://'.length);
|
|
121
|
+
const at = rest.indexOf('/');
|
|
122
|
+
return { bucket: rest.slice(0, at), key: rest.slice(at + 1) };
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
/**
|
|
126
|
+
* Credentials from the environment, in the names every S3 tool already uses.
|
|
127
|
+
*
|
|
128
|
+
* The same variables the AWS CLI, rclone, go-pmtiles and tileserver-gl read,
|
|
129
|
+
* so a machine already set up to reach a bucket needs nothing written into
|
|
130
|
+
* this node's settings -- and a container gets its keys the way a container
|
|
131
|
+
* gets its keys, rather than through a config file baked into an image.
|
|
132
|
+
*
|
|
133
|
+
* `AWS_S3_ENDPOINT` and `AWS_ENDPOINT_URL_S3` are both read: the first is
|
|
134
|
+
* what tileserver-gl documents, the second what the current AWS SDKs do.
|
|
135
|
+
* @param {object} [env] - The environment.
|
|
136
|
+
* @returns {object|null} - A row like `config.s3` holds, or null.
|
|
137
|
+
*/
|
|
138
|
+
export function bucketFromEnv(env = process.env) {
|
|
139
|
+
if (!env.AWS_ACCESS_KEY_ID || !env.AWS_SECRET_ACCESS_KEY) return null;
|
|
140
|
+
const forced = String(
|
|
141
|
+
env.AWS_S3_FORCE_PATH_STYLE ?? env.AWS_ENDPOINT_FORCE_PATH_STYLE ?? '',
|
|
142
|
+
).toLowerCase();
|
|
143
|
+
return {
|
|
144
|
+
from: 'the environment',
|
|
145
|
+
endpoint: env.AWS_S3_ENDPOINT || env.AWS_ENDPOINT_URL_S3 || undefined,
|
|
146
|
+
region: env.AWS_REGION || env.AWS_DEFAULT_REGION || undefined,
|
|
147
|
+
accessKeyId: env.AWS_ACCESS_KEY_ID,
|
|
148
|
+
secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
|
|
149
|
+
sessionToken: env.AWS_SESSION_TOKEN || undefined,
|
|
150
|
+
pathStyle: forced === '' ? undefined : forced !== 'false' && forced !== '0',
|
|
151
|
+
};
|
|
152
|
+
}
|
|
153
|
+
|
|
154
|
+
/**
|
|
155
|
+
* Which configured bucket a source's address should be read through.
|
|
156
|
+
*
|
|
157
|
+
* By name first, then whichever row names no bucket, then the environment --
|
|
158
|
+
* one set of credentials for a whole account is the ordinary case, and making
|
|
159
|
+
* the operator write the same keys once per bucket would be busywork with a
|
|
160
|
+
* copy-paste mistake in it.
|
|
161
|
+
* @param {string} url - An `s3://` address.
|
|
162
|
+
* @param {object[]} rows - `config.s3`.
|
|
163
|
+
* @param {object} [env] - The environment, for the last resort.
|
|
164
|
+
* @returns {object|null} - The row, or null if nothing matches.
|
|
165
|
+
*/
|
|
166
|
+
export function bucketFor(url, rows = [], env = process.env) {
|
|
167
|
+
if (!isS3Url(url)) return null;
|
|
168
|
+
const { bucket } = splitS3Url(url);
|
|
169
|
+
const named = (rows ?? []).find(
|
|
170
|
+
(row) => row?.bucket && row.bucket === bucket,
|
|
171
|
+
);
|
|
172
|
+
return (
|
|
173
|
+
named ??
|
|
174
|
+
(rows ?? []).find((row) => row && !row.bucket) ??
|
|
175
|
+
bucketFromEnv(env)
|
|
176
|
+
);
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
/**
|
|
180
|
+
* The HTTPS address an `s3://` one is actually fetched from.
|
|
181
|
+
*
|
|
182
|
+
* Path style unless the endpoint is AWS's own, where it was withdrawn for
|
|
183
|
+
* buckets made after September 2020. Every other S3-compatible server this is
|
|
184
|
+
* likely to meet -- MinIO, Ceph, Garage, R2 -- speaks path style, and it is
|
|
185
|
+
* the form that works with an endpoint naming a port or a bare host.
|
|
186
|
+
* @param {string} url - The `s3://` address.
|
|
187
|
+
* @param {object} row - The matching row of `config.s3`.
|
|
188
|
+
* @returns {string} - An https address.
|
|
189
|
+
*/
|
|
190
|
+
export function httpsFor(url, row) {
|
|
191
|
+
const { bucket, key } = splitS3Url(url);
|
|
192
|
+
const region = row?.region || 'us-east-1';
|
|
193
|
+
const endpoint = row?.endpoint || `https://s3.${region}.amazonaws.com`;
|
|
194
|
+
const base = new URL(
|
|
195
|
+
endpoint.includes('://') ? endpoint : `https://${endpoint}`,
|
|
196
|
+
);
|
|
197
|
+
const path = key
|
|
198
|
+
.split('/')
|
|
199
|
+
.map((part) => encodeURIComponent(part))
|
|
200
|
+
.join('/');
|
|
201
|
+
|
|
202
|
+
const virtual =
|
|
203
|
+
row?.pathStyle === false ||
|
|
204
|
+
(row?.pathStyle === undefined &&
|
|
205
|
+
/(^|\.)amazonaws\.com$/i.test(base.hostname));
|
|
206
|
+
if (virtual) {
|
|
207
|
+
base.hostname = `${bucket}.${base.hostname}`;
|
|
208
|
+
base.pathname = `/${path}`;
|
|
209
|
+
} else {
|
|
210
|
+
base.pathname = `${base.pathname.replace(/\/$/, '')}/${bucket}/${path}`;
|
|
211
|
+
}
|
|
212
|
+
return base.toString();
|
|
213
|
+
}
|
|
214
|
+
|
|
215
|
+
/**
|
|
216
|
+
* A PMTiles source reading a private bucket, signing every request.
|
|
217
|
+
*
|
|
218
|
+
* The same shape as `NodeFileSource` and the swarm's own source, because that
|
|
219
|
+
* is all PMTiles asks for: a key to cache under, and a way to get a range of
|
|
220
|
+
* bytes. Nothing about the archive is different -- only where the bytes come
|
|
221
|
+
* from, and that the request carries a signature.
|
|
222
|
+
*/
|
|
223
|
+
export class S3Source {
|
|
224
|
+
#url;
|
|
225
|
+
#row;
|
|
226
|
+
#now;
|
|
227
|
+
|
|
228
|
+
/**
|
|
229
|
+
* @param {string} url - An `s3://bucket/key` address.
|
|
230
|
+
* @param {object} row - The matching row of `config.s3`.
|
|
231
|
+
* @param {Function} [now] - The clock, for tests.
|
|
232
|
+
*/
|
|
233
|
+
constructor(url, row, now = () => new Date()) {
|
|
234
|
+
this.#url = url;
|
|
235
|
+
this.#row = row ?? {};
|
|
236
|
+
this.#now = now;
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
/**
|
|
240
|
+
* A stable key for PMTiles' internal caching.
|
|
241
|
+
*
|
|
242
|
+
* The `s3://` address rather than the signed one: the signature changes
|
|
243
|
+
* every request, and a cache keyed on it would never hit.
|
|
244
|
+
* @returns {string} - The address.
|
|
245
|
+
*/
|
|
246
|
+
getKey() {
|
|
247
|
+
return this.#url;
|
|
248
|
+
}
|
|
249
|
+
|
|
250
|
+
/**
|
|
251
|
+
* Reads a byte range out of the object.
|
|
252
|
+
* @param {number} offset - Byte offset.
|
|
253
|
+
* @param {number} length - How many bytes.
|
|
254
|
+
* @param {AbortSignal} [signal] - To give up early.
|
|
255
|
+
* @returns {Promise<object>} - `{data, etag, cacheControl, expires}`.
|
|
256
|
+
*/
|
|
257
|
+
async getBytes(offset, length, signal) {
|
|
258
|
+
const href = httpsFor(this.#url, this.#row);
|
|
259
|
+
const amzDate = `${this.#now()
|
|
260
|
+
.toISOString()
|
|
261
|
+
.replace(/[-:]/g, '')
|
|
262
|
+
.slice(0, 15)}Z`;
|
|
263
|
+
const headers = {
|
|
264
|
+
host: new URL(href).host,
|
|
265
|
+
range: `bytes=${offset}-${offset + length - 1}`,
|
|
266
|
+
'x-amz-content-sha256': EMPTY_SHA256,
|
|
267
|
+
'x-amz-date': amzDate,
|
|
268
|
+
};
|
|
269
|
+
if (this.#row.sessionToken) {
|
|
270
|
+
headers['x-amz-security-token'] = this.#row.sessionToken;
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
headers.authorization = authorization(
|
|
274
|
+
{ method: 'GET', url: href, headers },
|
|
275
|
+
{
|
|
276
|
+
accessKeyId: this.#row.accessKeyId,
|
|
277
|
+
secretAccessKey: this.#row.secretAccessKey,
|
|
278
|
+
region: this.#row.region || 'us-east-1',
|
|
279
|
+
amzDate,
|
|
280
|
+
},
|
|
281
|
+
);
|
|
282
|
+
|
|
283
|
+
// `host` is set by fetch itself and refused as a header, but it has to be
|
|
284
|
+
// in the signature: it is what stops a signed request being replayed
|
|
285
|
+
// against a different endpoint.
|
|
286
|
+
const sent = { ...headers };
|
|
287
|
+
delete sent.host;
|
|
288
|
+
|
|
289
|
+
const response = await fetch(href, { headers: sent, signal });
|
|
290
|
+
if (!response.ok) {
|
|
291
|
+
throw new Error(
|
|
292
|
+
`${this.#url} answered ${response.status} ${response.statusText}`,
|
|
293
|
+
);
|
|
294
|
+
}
|
|
295
|
+
return {
|
|
296
|
+
data: await response.arrayBuffer(),
|
|
297
|
+
etag: response.headers.get('etag') ?? undefined,
|
|
298
|
+
cacheControl: response.headers.get('cache-control') ?? undefined,
|
|
299
|
+
expires: response.headers.get('expires') ?? undefined,
|
|
300
|
+
};
|
|
301
|
+
}
|
|
302
|
+
}
|
package/src/stacks.js
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import crypto from 'node:crypto';
|
|
2
2
|
import fs from 'node:fs/promises';
|
|
3
3
|
import path from 'node:path';
|
|
4
|
+
import { isS3Url } from './s3-source.js';
|
|
4
5
|
import {
|
|
5
6
|
MAX_BLUR_SIGMA,
|
|
6
7
|
RESAMPLING,
|
|
@@ -91,6 +92,20 @@ export function isHttpUrl(value) {
|
|
|
91
92
|
}
|
|
92
93
|
}
|
|
93
94
|
|
|
95
|
+
/**
|
|
96
|
+
* Whether an address is one a source may be read from.
|
|
97
|
+
*
|
|
98
|
+
* Wider than `isHttpUrl` by one scheme: `s3://bucket/key` is an object in a
|
|
99
|
+
* bucket, fetched over HTTPS like anything else once a configured endpoint
|
|
100
|
+
* and a signature have been put on it. A feed address is not this -- that is
|
|
101
|
+
* something the node fetches plainly and has no credentials for.
|
|
102
|
+
* @param {string} value - What the recipe said.
|
|
103
|
+
* @returns {boolean} - True for http, https or s3.
|
|
104
|
+
*/
|
|
105
|
+
export function isSourceUrl(value) {
|
|
106
|
+
return isHttpUrl(value) || isS3Url(value);
|
|
107
|
+
}
|
|
108
|
+
|
|
94
109
|
/**
|
|
95
110
|
* How many stacks deep a recipe may reach.
|
|
96
111
|
*
|
|
@@ -159,8 +174,10 @@ export function validateStack(stack) {
|
|
|
159
174
|
);
|
|
160
175
|
}
|
|
161
176
|
if (source?.url !== undefined) {
|
|
162
|
-
if (typeof source.url !== 'string' || !
|
|
163
|
-
problems.push(
|
|
177
|
+
if (typeof source.url !== 'string' || !isSourceUrl(source.url)) {
|
|
178
|
+
problems.push(
|
|
179
|
+
`sources[${index}].url must be an http(s) or s3:// address`,
|
|
180
|
+
);
|
|
164
181
|
}
|
|
165
182
|
}
|
|
166
183
|
for (const key of ['minzoom', 'maxzoom']) {
|
|
@@ -653,8 +670,13 @@ export function stackCoverage(resolved) {
|
|
|
653
670
|
(at?.nested ? stackCoverage(at.nested).bounds : undefined);
|
|
654
671
|
}
|
|
655
672
|
if (!bounds && summaries.length) {
|
|
656
|
-
const boxes = summaries.map((s) => s.bounds)
|
|
657
|
-
|
|
673
|
+
const boxes = summaries.map((s) => s.bounds);
|
|
674
|
+
// Every source, or none of them. A source that states no box is not one
|
|
675
|
+
// covering nothing: an archive's box comes off its own header, and the
|
|
676
|
+
// source without one is the imported global base, which covers the world.
|
|
677
|
+
// Unioning only the boxes that exist advertised a planet-wide stack as
|
|
678
|
+
// covering whichever patches happened to state theirs.
|
|
679
|
+
if (boxes.length && boxes.every(Array.isArray)) {
|
|
658
680
|
bounds = [
|
|
659
681
|
Math.min(...boxes.map((b) => b[0])),
|
|
660
682
|
Math.min(...boxes.map((b) => b[1])),
|
package/src/tiles.js
CHANGED
|
@@ -2,6 +2,7 @@ import fs from 'node:fs/promises';
|
|
|
2
2
|
import path from 'node:path';
|
|
3
3
|
import zlib from 'node:zlib';
|
|
4
4
|
import { FetchSource, PMTiles, SharedPromiseCache } from 'pmtiles';
|
|
5
|
+
import { S3Source, bucketFor, isS3Url, splitS3Url } from './s3-source.js';
|
|
5
6
|
import { TorrentSource } from 'pmtiles-torrent';
|
|
6
7
|
import { NodeFileSource } from './file-source.js';
|
|
7
8
|
import { onDiskPath } from './incomplete.js';
|
|
@@ -199,6 +200,18 @@ export class TileStore {
|
|
|
199
200
|
* @param {string} url - Where the archive is.
|
|
200
201
|
* @returns {Promise<object>} - The open handle.
|
|
201
202
|
*/
|
|
203
|
+
/**
|
|
204
|
+
* Drops every open reader of an archive read over the network.
|
|
205
|
+
*
|
|
206
|
+
* Called when the bucket settings change. A handle holds the credentials it
|
|
207
|
+
* was opened with, so without this a corrected key would apply to archives
|
|
208
|
+
* opened afterwards and not to the one being looked at.
|
|
209
|
+
* @returns {void}
|
|
210
|
+
*/
|
|
211
|
+
forgetRemote() {
|
|
212
|
+
this.#remote.clear();
|
|
213
|
+
}
|
|
214
|
+
|
|
202
215
|
async #acquireRemote(url) {
|
|
203
216
|
const existing = this.#remote.get(url);
|
|
204
217
|
if (existing) {
|
|
@@ -207,7 +220,18 @@ export class TileStore {
|
|
|
207
220
|
return existing;
|
|
208
221
|
}
|
|
209
222
|
|
|
210
|
-
|
|
223
|
+
// A bucket rather than a web server. The difference is the signature on
|
|
224
|
+
// every request and where the address is really fetched from; everything
|
|
225
|
+
// after this line -- the archive, the directory cache, the LRU -- is the
|
|
226
|
+
// same, because a source is only a key and a way to get bytes.
|
|
227
|
+
const row = bucketFor(url, this.#config.s3);
|
|
228
|
+
if (isS3Url(url) && !row) {
|
|
229
|
+
throw new TileReadError(
|
|
230
|
+
`no credentials configured for ${splitS3Url(url).bucket}`,
|
|
231
|
+
502,
|
|
232
|
+
);
|
|
233
|
+
}
|
|
234
|
+
const source = row ? new S3Source(url, row) : new FetchSource(url);
|
|
211
235
|
const handle = {
|
|
212
236
|
mode: 'remote',
|
|
213
237
|
source,
|
|
@@ -261,6 +285,36 @@ export class TileStore {
|
|
|
261
285
|
const answer = await handle.source.getBytes(offset, length, options.signal);
|
|
262
286
|
return Buffer.from(answer.data);
|
|
263
287
|
}
|
|
288
|
+
|
|
289
|
+
/**
|
|
290
|
+
* Reads a byte range out of an archive at a URL.
|
|
291
|
+
*
|
|
292
|
+
* The same question `readRange` answers, for the sources that have no
|
|
293
|
+
* infohash to ask it about. A bake needs this rather than tile-by-tile
|
|
294
|
+
* reads: it walks the directories of every source to find which tiles exist
|
|
295
|
+
* at all, and a stack of remote sources with no way to do that could be
|
|
296
|
+
* served and not exported -- which is backwards, since exporting is how
|
|
297
|
+
* something read from somewhere else becomes something this node holds.
|
|
298
|
+
* @param {string} url - The archive's address.
|
|
299
|
+
* @param {number} offset - Byte offset into the archive file.
|
|
300
|
+
* @param {number} length - How many bytes.
|
|
301
|
+
* @param {object} [options] - Abort signal.
|
|
302
|
+
* @returns {Promise<Buffer>} - The bytes.
|
|
303
|
+
*/
|
|
304
|
+
async readRemoteRange(url, offset, length, options = {}) {
|
|
305
|
+
const handle = await this.#acquireRemote(url);
|
|
306
|
+
try {
|
|
307
|
+
const answer = await handle.source.getBytes(
|
|
308
|
+
offset,
|
|
309
|
+
length,
|
|
310
|
+
options.signal,
|
|
311
|
+
);
|
|
312
|
+
return Buffer.from(answer.data);
|
|
313
|
+
} catch (error) {
|
|
314
|
+
if (error instanceof TileReadError) throw error;
|
|
315
|
+
throw new TileReadError(`${url}: ${error.message}`, 502);
|
|
316
|
+
}
|
|
317
|
+
}
|
|
264
318
|
/**
|
|
265
319
|
* Reads an archive's header and metadata, through whatever source applies.
|
|
266
320
|
*
|
package/src/web/index.html
CHANGED
|
@@ -6726,6 +6726,56 @@ Every piece is hashed against the ` +
|
|
|
6726
6726
|
'anybody remembering to press it.',
|
|
6727
6727
|
});
|
|
6728
6728
|
|
|
6729
|
+
renderRowEditor({
|
|
6730
|
+
into: paneFor('Feeds'),
|
|
6731
|
+
key: 's3',
|
|
6732
|
+
title: 'S3 buckets',
|
|
6733
|
+
blurb:
|
|
6734
|
+
'Credentials for reading a stack source written as ' +
|
|
6735
|
+
'<code>s3://bucket/key.pmtiles</code>. Only for a bucket that is ' +
|
|
6736
|
+
'not public — a public or presigned address is read as any other ' +
|
|
6737
|
+
'URL and needs nothing here. <b>Endpoint</b> is what makes it ' +
|
|
6738
|
+
'S3-compatible rather than S3: MinIO, Ceph, Garage, R2, Wasabi ' +
|
|
6739
|
+
'and B2 all answer the same protocol at their own address.',
|
|
6740
|
+
columns: [
|
|
6741
|
+
{
|
|
6742
|
+
field: 'bucket',
|
|
6743
|
+
label: 'Bucket',
|
|
6744
|
+
placeholder: 'any bucket',
|
|
6745
|
+
},
|
|
6746
|
+
{
|
|
6747
|
+
field: 'endpoint',
|
|
6748
|
+
label: 'Endpoint',
|
|
6749
|
+
placeholder: 'AWS for the region',
|
|
6750
|
+
wide: true,
|
|
6751
|
+
},
|
|
6752
|
+
{ field: 'region', label: 'Region', placeholder: 'us-east-1' },
|
|
6753
|
+
{ field: 'accessKeyId', label: 'Access key' },
|
|
6754
|
+
{ field: 'secretAccessKey', label: 'Secret key', secret: true },
|
|
6755
|
+
{
|
|
6756
|
+
field: 'pathStyle',
|
|
6757
|
+
label: 'Addressing',
|
|
6758
|
+
options: [
|
|
6759
|
+
['', 'path style, unless AWS'],
|
|
6760
|
+
['true', 'path style'],
|
|
6761
|
+
['false', 'bucket as a subdomain'],
|
|
6762
|
+
],
|
|
6763
|
+
},
|
|
6764
|
+
],
|
|
6765
|
+
rows: config.s3 ?? [],
|
|
6766
|
+
footnote:
|
|
6767
|
+
'A source is read through the row naming its bucket, or through ' +
|
|
6768
|
+
'whichever row names none — one set of credentials for a whole ' +
|
|
6769
|
+
'account is the ordinary case. With no rows at all the standard ' +
|
|
6770
|
+
'<code>AWS_ACCESS_KEY_ID</code>, <code>AWS_SECRET_ACCESS_KEY</code>, ' +
|
|
6771
|
+
'<code>AWS_REGION</code> and <code>AWS_S3_ENDPOINT</code> ' +
|
|
6772
|
+
'variables are read, so a machine already set up to reach a ' +
|
|
6773
|
+
'bucket needs nothing typed here. Addressing is only worth ' +
|
|
6774
|
+
'setting for a server that insists: path style is what every ' +
|
|
6775
|
+
'S3-compatible server speaks, and AWS itself withdrew it for ' +
|
|
6776
|
+
'buckets made after September 2020.',
|
|
6777
|
+
});
|
|
6778
|
+
|
|
6729
6779
|
renderRowEditor({
|
|
6730
6780
|
// With the monitored folders and the scheduled sources, because it
|
|
6731
6781
|
// is the same kind of thing: something that puts an archive into
|
|
@@ -7655,21 +7705,64 @@ Every piece is hashed against the ` +
|
|
|
7655
7705
|
and this node has no codec. <code>npm install sharp</code>.</p>`
|
|
7656
7706
|
: ''
|
|
7657
7707
|
}
|
|
7658
|
-
<
|
|
7659
|
-
<
|
|
7660
|
-
|
|
7661
|
-
|
|
7662
|
-
|
|
7663
|
-
|
|
7664
|
-
|
|
7665
|
-
|
|
7666
|
-
|
|
7667
|
-
|
|
7668
|
-
|
|
7669
|
-
|
|
7708
|
+
<details${stack.sources.length > SOURCES_SHOWN ? '' : ' open'}>
|
|
7709
|
+
<summary>${stack.sources.length} source${
|
|
7710
|
+
stack.sources.length === 1 ? '' : 's'
|
|
7711
|
+
}${
|
|
7712
|
+
stack.sources.length > SOURCES_SHOWN
|
|
7713
|
+
? ` <span class="muted">— ${escapeHtml(
|
|
7714
|
+
summarizeSources(stack.sources),
|
|
7715
|
+
)}</span>`
|
|
7716
|
+
: ''
|
|
7717
|
+
}</summary>
|
|
7718
|
+
<table class="tight">
|
|
7719
|
+
<thead>
|
|
7720
|
+
<tr><th>#</th><th>Source</th><th>Resolves to</th><th>Zooms</th><th></th></tr>
|
|
7721
|
+
</thead>
|
|
7722
|
+
<tbody>
|
|
7723
|
+
${stack.sources.map(renderStackSource).join('')}
|
|
7724
|
+
</tbody>
|
|
7725
|
+
</table>
|
|
7726
|
+
<p class="muted">
|
|
7727
|
+
Listed in the file's order: the first is the base, each one covers
|
|
7728
|
+
the one before it wherever it has data, and the last wins.
|
|
7729
|
+
</p>
|
|
7730
|
+
</details>
|
|
7670
7731
|
</div>`;
|
|
7671
7732
|
};
|
|
7672
7733
|
|
|
7734
|
+
/**
|
|
7735
|
+
* How many sources a stack shows without being asked.
|
|
7736
|
+
*
|
|
7737
|
+
* A stack built from a provider's file list runs to several hundred
|
|
7738
|
+
* rows, which buries every other stack on the page under one of them.
|
|
7739
|
+
* Small enough that the ordinary stack -- a base and a layer or two --
|
|
7740
|
+
* is still open on arrival, since collapsing that hides nothing worth
|
|
7741
|
+
* the click.
|
|
7742
|
+
*/
|
|
7743
|
+
const SOURCES_SHOWN = 5;
|
|
7744
|
+
|
|
7745
|
+
/**
|
|
7746
|
+
* What a collapsed source list says about itself.
|
|
7747
|
+
* @param {object[]} sources - The stack's sources.
|
|
7748
|
+
* @returns {string} - A line naming the base and the shape of the rest.
|
|
7749
|
+
*/
|
|
7750
|
+
const summarizeSources = (sources) => {
|
|
7751
|
+
const first = sources[0];
|
|
7752
|
+
const base =
|
|
7753
|
+
first?.kind === 'url'
|
|
7754
|
+
? (first.url ?? '').replace(/^https?:\/\//, '')
|
|
7755
|
+
: (first?.name ?? '');
|
|
7756
|
+
const urls = sources.filter((s) => s.kind === 'url').length;
|
|
7757
|
+
const rest =
|
|
7758
|
+
urls === sources.length
|
|
7759
|
+
? 'all read over HTTP'
|
|
7760
|
+
: urls
|
|
7761
|
+
? `${urls} read over HTTP`
|
|
7762
|
+
: '';
|
|
7763
|
+
return [base, rest].filter(Boolean).join(', ');
|
|
7764
|
+
};
|
|
7765
|
+
|
|
7673
7766
|
/**
|
|
7674
7767
|
* One source row.
|
|
7675
7768
|
* @param {object} source - A source from /api/stacks.
|
|
@@ -7713,7 +7806,11 @@ Every piece is hashed against the ` +
|
|
|
7713
7806
|
source.resolved
|
|
7714
7807
|
? source.kind === 'url'
|
|
7715
7808
|
? `<span class="muted" title="Read straight from this address over HTTP. Nothing is downloaded or seeded, and whether it answers is settled at the first tile.">read over HTTP</span>`
|
|
7716
|
-
:
|
|
7809
|
+
: source.kind === 'stack'
|
|
7810
|
+
? `<span class="muted" title="Another recipe, evaluated for this tile and merged as one layer. It follows every later change to that stack.">recipe of ${
|
|
7811
|
+
source.nested ?? 0
|
|
7812
|
+
} source${source.nested === 1 ? '' : 's'}</span>`
|
|
7813
|
+
: `<code class="muted">${escapeHtml(source.archiveName ?? '')}</code>`
|
|
7717
7814
|
: '<span class="bad">does not resolve</span>'
|
|
7718
7815
|
}</td>
|
|
7719
7816
|
<td>${zooms}${
|
|
@@ -7754,6 +7851,18 @@ Every piece is hashed against the ` +
|
|
|
7754
7851
|
return candidate;
|
|
7755
7852
|
};
|
|
7756
7853
|
|
|
7854
|
+
/**
|
|
7855
|
+
* Whether an address is one a source can be read from.
|
|
7856
|
+
*
|
|
7857
|
+
* `s3://bucket/key` as well as the two web schemes: a bucket is fetched
|
|
7858
|
+
* over HTTPS like anything else, once the endpoint and the signature
|
|
7859
|
+
* configured under Settings have been put on it.
|
|
7860
|
+
* @param {string} value - What was typed.
|
|
7861
|
+
* @returns {boolean} - True if a source could use it.
|
|
7862
|
+
*/
|
|
7863
|
+
const isSourceAddress = (value) =>
|
|
7864
|
+
/^https?:\/\//i.test(value) || /^s3:\/\/[^/]+\/.+/i.test(value);
|
|
7865
|
+
|
|
7757
7866
|
/** Forty characters of hex is not a label. */
|
|
7758
7867
|
const shortHash = (hash) => (hash ? `${hash.slice(0, 12)}…` : hash);
|
|
7759
7868
|
|
|
@@ -7972,11 +8081,14 @@ Every piece is hashed against the ` +
|
|
|
7972
8081
|
);
|
|
7973
8082
|
}
|
|
7974
8083
|
|
|
7975
|
-
|
|
7976
|
-
|
|
7977
|
-
|
|
7978
|
-
|
|
7979
|
-
|
|
8084
|
+
// Always offered, and last: it names nothing this node holds, so
|
|
8085
|
+
// there is no list to be empty and no reason to prefer it over an
|
|
8086
|
+
// archive that is already here.
|
|
8087
|
+
groups.push(
|
|
8088
|
+
`<optgroup label="A URL — read over HTTP, never downloaded"><option value="url:">an address you type…</option></optgroup>`,
|
|
8089
|
+
);
|
|
8090
|
+
|
|
8091
|
+
select.innerHTML = groups.join('');
|
|
7980
8092
|
};
|
|
7981
8093
|
|
|
7982
8094
|
/** Redraws the source list and the fields that depend on the space. */
|
|
@@ -8124,6 +8236,10 @@ Every piece is hashed against the ` +
|
|
|
8124
8236
|
// against. Those fields are left out rather than shown and refused on
|
|
8125
8237
|
// save; everything that acts on heights stays.
|
|
8126
8238
|
const nested = Boolean(source.stack);
|
|
8239
|
+
// An archive somewhere else. It has no catalog entry to read a zoom
|
|
8240
|
+
// range or an extent from, so the recipe is the only place either can
|
|
8241
|
+
// come from -- which is why the card asks for them.
|
|
8242
|
+
const remote = source.url !== undefined;
|
|
8127
8243
|
const perSpace = rgba
|
|
8128
8244
|
? `<label class="choice">Opacity
|
|
8129
8245
|
<input type="number" min="0" max="1" step="0.05" style="width:5rem"
|
|
@@ -8240,16 +8356,29 @@ Every piece is hashed against the ` +
|
|
|
8240
8356
|
<div class="row">
|
|
8241
8357
|
<span class="muted">${index}</span>
|
|
8242
8358
|
<code>${escapeHtml(
|
|
8243
|
-
|
|
8359
|
+
remote
|
|
8360
|
+
? (source.url || 'an address').replace(/^https?:\/\//, '')
|
|
8361
|
+
: (source.stack ??
|
|
8362
|
+
source.category ??
|
|
8363
|
+
shortHash(source.archive) ??
|
|
8364
|
+
''),
|
|
8244
8365
|
)}</code>
|
|
8245
8366
|
<span class="sub" title="${
|
|
8246
|
-
|
|
8247
|
-
? '
|
|
8248
|
-
: source.
|
|
8249
|
-
? '
|
|
8250
|
-
:
|
|
8367
|
+
remote
|
|
8368
|
+
? 'Read straight from this address over HTTP, with byte ranges. Nothing is downloaded, nothing is seeded, and this node never rebuilds or retires it.'
|
|
8369
|
+
: source.archive
|
|
8370
|
+
? 'Pinned to this build. It will not follow a rebuild, and breaks if the archive is removed.'
|
|
8371
|
+
: source.stack
|
|
8372
|
+
? 'Another stack, merged as one layer. It follows every later change to that recipe, and to whatever it resolves to.'
|
|
8373
|
+
: 'Follows whichever build in this category is newest.'
|
|
8251
8374
|
}">${
|
|
8252
|
-
|
|
8375
|
+
remote
|
|
8376
|
+
? 'url'
|
|
8377
|
+
: source.archive
|
|
8378
|
+
? 'pinned'
|
|
8379
|
+
: source.stack
|
|
8380
|
+
? 'stack'
|
|
8381
|
+
: 'category'
|
|
8253
8382
|
}</span>
|
|
8254
8383
|
${role}
|
|
8255
8384
|
<span class="bar-gap"></span>
|
|
@@ -8259,6 +8388,38 @@ Every piece is hashed against the ` +
|
|
|
8259
8388
|
${last ? 'disabled' : ''} title="Later: covers more">↓</button>
|
|
8260
8389
|
<button type="button" data-stack-remove="${index}">Remove</button>
|
|
8261
8390
|
</div>
|
|
8391
|
+
${
|
|
8392
|
+
remote
|
|
8393
|
+
? `<div class="bar">
|
|
8394
|
+
<label class="choice" style="flex:1">Address
|
|
8395
|
+
<input style="flex:1;min-width:22rem" placeholder="https://example.org/terrain.pmtiles or s3://bucket/terrain.pmtiles"
|
|
8396
|
+
data-stack-field="url" data-stack-index="${index}"
|
|
8397
|
+
value="${escapeHtml(source.url ?? '')}" />
|
|
8398
|
+
</label>
|
|
8399
|
+
${
|
|
8400
|
+
source.url && !isSourceAddress(source.url)
|
|
8401
|
+
? '<span class="bad">http://, https:// or s3://bucket/key</span>'
|
|
8402
|
+
: ''
|
|
8403
|
+
}
|
|
8404
|
+
</div>
|
|
8405
|
+
<div class="bar">
|
|
8406
|
+
<label class="choice"
|
|
8407
|
+
title="The zooms this archive actually holds. Checked before anything is opened, so a tile outside the range skips it without a request leaving this node — which is what makes a stack of many of these cheap. Leave empty and every tile asks it.">
|
|
8408
|
+
Only zooms
|
|
8409
|
+
<input type="number" min="0" max="24" step="1" style="width:4.5rem"
|
|
8410
|
+
placeholder="any"
|
|
8411
|
+
data-stack-field="minzoom" data-stack-index="${index}"
|
|
8412
|
+
value="${source.minzoom ?? ''}" />
|
|
8413
|
+
to
|
|
8414
|
+
<input type="number" min="0" max="24" step="1" style="width:4.5rem"
|
|
8415
|
+
placeholder="any"
|
|
8416
|
+
data-stack-field="maxzoom" data-stack-index="${index}"
|
|
8417
|
+
value="${source.maxzoom ?? ''}" />
|
|
8418
|
+
</label>
|
|
8419
|
+
<span class="sub">Clip it to a box below to skip it by area too.</span>
|
|
8420
|
+
</div>`
|
|
8421
|
+
: ''
|
|
8422
|
+
}
|
|
8262
8423
|
<div class="bar">
|
|
8263
8424
|
${perSpace}
|
|
8264
8425
|
${
|
|
@@ -8402,6 +8563,12 @@ Every piece is hashed against the ` +
|
|
|
8402
8563
|
source.featherMetres = width;
|
|
8403
8564
|
}
|
|
8404
8565
|
}
|
|
8566
|
+
} else if (field === 'minzoom' || field === 'maxzoom') {
|
|
8567
|
+
// Blank means "every zoom", so an empty box removes the field
|
|
8568
|
+
// rather than writing 0 -- which for minzoom is the same thing and
|
|
8569
|
+
// for maxzoom would silence the source everywhere above z0.
|
|
8570
|
+
if (event.target.value === '') delete source[field];
|
|
8571
|
+
else source[field] = Math.round(Number(event.target.value));
|
|
8405
8572
|
} else if (field === 'baseVal' || field === 'interval') {
|
|
8406
8573
|
// Blank means "the default", not zero -- an interval of 0 would make
|
|
8407
8574
|
// every height the base value.
|
|
@@ -8499,7 +8666,12 @@ Every piece is hashed against the ` +
|
|
|
8499
8666
|
? { archive: name }
|
|
8500
8667
|
: kind === 'stk'
|
|
8501
8668
|
? { stack: name }
|
|
8502
|
-
:
|
|
8669
|
+
: kind === 'url'
|
|
8670
|
+
? // Empty, because the address is typed into the card rather
|
|
8671
|
+
// than chosen from a menu. It is refused on save until it
|
|
8672
|
+
// is filled in.
|
|
8673
|
+
{ url: '' }
|
|
8674
|
+
: { category: name },
|
|
8503
8675
|
);
|
|
8504
8676
|
renderStackDraft();
|
|
8505
8677
|
fillStackSourceChoices();
|
|
@@ -8546,6 +8718,23 @@ Every piece is hashed against the ` +
|
|
|
8546
8718
|
'is made of, so saving would replace it rather than add this one.';
|
|
8547
8719
|
return;
|
|
8548
8720
|
}
|
|
8721
|
+
// A URL source is the one kind whose name is typed rather than
|
|
8722
|
+
// chosen, so it is the one kind that can be left half-made. Caught
|
|
8723
|
+
// here because the reply to a recipe the node refuses says which
|
|
8724
|
+
// source is wrong by index, which is not where the operator is
|
|
8725
|
+
// looking.
|
|
8726
|
+
const blank = stackDraft.sources.findIndex(
|
|
8727
|
+
(source) =>
|
|
8728
|
+
source.url !== undefined && !isSourceAddress(source.url ?? ''),
|
|
8729
|
+
);
|
|
8730
|
+
if (blank >= 0) {
|
|
8731
|
+
$('stack-error').textContent =
|
|
8732
|
+
`Source ${blank} needs an address: http://, https://, or ` +
|
|
8733
|
+
's3://bucket/key for a bucket set up under Settings → Feeds. It ' +
|
|
8734
|
+
'is read straight from wherever it is, so there is nothing else ' +
|
|
8735
|
+
'to find it by.';
|
|
8736
|
+
return;
|
|
8737
|
+
}
|
|
8549
8738
|
const body = {
|
|
8550
8739
|
title: $('stack-title').value.trim() || undefined,
|
|
8551
8740
|
space: stackDraft.space,
|