pmtiles-swarm 0.53.0 → 0.54.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -0
- package/README.md +52 -52
- package/docs/configuration.md +39 -6
- package/package.json +1 -1
- package/src/api.js +124 -14
- package/src/config.js +39 -0
- package/src/library.js +13 -0
- package/src/tiles.js +37 -0
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,32 @@
|
|
|
7
7
|
### 🐞 Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.54.0
|
|
11
|
+
### ✨ Features and improvements
|
|
12
|
+
- **`serveArchiveFromSwarm` — a byte range for an archive this node does not hold.** Experimental,
|
|
13
|
+
off by default, and the loop cache mode was built for: the node holds no bytes, a reader asks for
|
|
14
|
+
some, and the pieces covering them arrive from peers. It reuses the machinery the tile endpoint has
|
|
15
|
+
always used internally — `TorrentSource.getBytes` already fetches every covering piece
|
|
16
|
+
concurrently, and this shares its piece cache, its directory prefetch and its open handle. Point an
|
|
17
|
+
ordinary PMTiles reader at a node with no copy of the archive and it works.
|
|
18
|
+
|
|
19
|
+
Bounded on purpose. A `Range` header is required, since the only other answer is "pull 700 GiB
|
|
20
|
+
through BitTorrent and stream it out" — no header is `411`, a range over `swarmRangeLimitBytes`
|
|
21
|
+
(8 MiB) is `416`, and a swarm that does not answer within `swarmRangeTimeoutMs` (30s) is `504`
|
|
22
|
+
rather than a socket held open for two minutes. The response carries the same `ETag` and the same
|
|
23
|
+
year-long `immutable` caching a complete copy would, because it is the same content: the infohash
|
|
24
|
+
names those bytes wherever they were read from.
|
|
25
|
+
|
|
26
|
+
Not recommended for anything public, for reasons that are properties of the arrangement rather than
|
|
27
|
+
of the code — every byte is somebody else's upload, and a cache-mode node behind a URL that looks
|
|
28
|
+
like an origin is not one.
|
|
29
|
+
|
|
30
|
+
### 🐞 Bug fixes
|
|
31
|
+
- **A node could be made a web seed for an archive it did not hold.** `selfWebSeed` is now refused on
|
|
32
|
+
an incomplete archive and offered again once the download finishes. Answering from the swarm and
|
|
33
|
+
then advertising that to the swarm is a loop with an amplifier in it: every peer that takes the
|
|
34
|
+
seed makes this node fetch the piece again in order to serve it.
|
|
35
|
+
|
|
10
36
|
## 0.53.0
|
|
11
37
|
### ✨ Features and improvements
|
|
12
38
|
- **The three switches are now settable on every import, from the console.** **Serve file**, **Web
|
package/README.md
CHANGED
|
@@ -739,58 +739,58 @@ which the endpoint answers 501.
|
|
|
739
739
|
|
|
740
740
|
## API
|
|
741
741
|
|
|
742
|
-
| Method | Path | Purpose
|
|
743
|
-
| --------------------- | ----------------------------------------------------------- |
|
|
744
|
-
| `GET` | `/api/status` | Engine health, counts, watched folders, save locations, free space, and whether the swarm can reach this node
|
|
745
|
-
| `GET` | `/api/torrents` | Catalog joined with live swarm state and what is left of each seeding limit
|
|
746
|
-
| `POST` | `/api/torrents` | Add via `{path}`, `{url}`, `{magnet}`, `{torrentUrl}`, or a raw `.torrent` body
|
|
747
|
-
| `DELETE` | `/api/torrents/:infoHash` | Remove (`?deleteData=true` to delete data too)
|
|
748
|
-
| `GET` | `/api/torrents/:infoHash` | One archive, with disk usage and how it is being read
|
|
749
|
-
| `GET` | `/api/torrents/:infoHash/file`, `/magnet` | Download the `.torrent`, or its magnet URI
|
|
750
|
-
| `GET` | `/api/torrents/:infoHash/peers`, `/trackers`, `/content` | Per-peer, per-tracker and per-file detail
|
|
751
|
-
| `GET` | `/api/torrents/:infoHash/pieces` | Which pieces are held, how rare each is, and what peers hold
|
|
752
|
-
| `PATCH` | `/api/torrents/:infoHash/mode` | Switch between mirror and cache
|
|
753
|
-
| `PATCH` | `/api/torrents/:infoHash/categories` | Set, add or remove categories
|
|
754
|
-
| `PATCH` | `/api/torrents/:infoHash/seeding` | Per-archive seeding limit, or "use the global one"
|
|
755
|
-
| `PATCH` `GET` | `/api/torrents/:infoHash/location` | Move the data; poll the move
|
|
756
|
-
| `POST` | `/api/torrents/:infoHash/recheck` | Hash what is on disk again and believe the result — for an archive whose progress and whose files disagree
|
|
757
|
-
| `POST` | `/api/torrents/:infoHash/pause`, `/resume` | Stop offering it, without forgetting it
|
|
758
|
-
| `POST` | `/api/torrents/:infoHash/webseeds` | Add web seeds — does not change the infohash
|
|
759
|
-
| `DELETE` | `/api/torrents/:infoHash/webseeds` | Drop web seeds — likewise
|
|
760
|
-
| `POST` | `/api/torrents/:infoHash/publish` | What this node offers of the archive itself: `serveArchive`, `selfWebSeed`, `publicDownload`
|
|
761
|
-
| `POST` | `/api/torrents/:infoHash/warm` | Pre-fetch a region (`GET` for progress, `DELETE` to cancel)
|
|
762
|
-
| `DELETE` | `/api/torrents/:infoHash/cache` | Reclaim cached pieces, keep the archive
|
|
763
|
-
| `POST` | `/api/torrents/:infoHash/check` | Has the source changed since the torrent was made?
|
|
764
|
-
| `POST` | `/api/torrents/:infoHash/rebuild` | Rebuild from the current source (mints a new infohash)
|
|
765
|
-
| `POST` | `/api/check-origins` | Check every archive with a watchable source
|
|
766
|
-
| `GET` `DELETE` | `/api/adds` | Adds still in flight — downloads, and local files being hashed — and cancelling a download by URL. A hash cannot be stopped part-way, so it is listed but not cancellable
|
|
767
|
-
| `GET` `POST` | `/api/speed` | Which speed limits are in force, and the manual switch between the two sets
|
|
768
|
-
| `GET` | `/api/categories` | Every category, with the endpoints resolving to its newest build
|
|
769
|
-
| `POST` | `/api/adopt`, `/api/adopt/candidates` | Take over what an engine or another node holds
|
|
770
|
-
| `POST` | `/api/sources/preview` | What a watched web location would take, without taking it
|
|
771
|
-
| `POST` | `/api/subscriptions/preview` | Whether a peer is reachable and what it offers
|
|
772
|
-
| `POST` | `/api/subscriptions/refresh` | Poll subscribed feeds now
|
|
773
|
-
| `POST` | `/api/sources/check` | Check scheduled sources now, rather than at the next due time
|
|
774
|
-
| `POST` | `/api/torrents/:infoHash/hooks/complete` | Run the completion hook again for one archive
|
|
775
|
-
| `GET` `POST` `DELETE` | `/api/tokens`, `/api/tokens/:id` | Mint, list and revoke access tokens
|
|
776
|
-
| `GET` `POST` | `/api/restart` | What a restart would do, and doing it
|
|
777
|
-
| `GET` `DELETE` | `/api/stats` | What this node has served — per-archive counters and the last N requests; `DELETE` clears them
|
|
778
|
-
| `GET` | `/api/traffic` | Upload and download speed per archive over time, from `stats.db`; `infoHash`, `hours` and `buckets` narrow it, and `totals` ranks archives by bytes moved
|
|
779
|
-
| `GET` `PATCH` | `/api/config` | Read and change settings
|
|
780
|
-
| `POST` | `/api/login`, `/api/logout` | Console sign-in
|
|
781
|
-
| `GET` | `/api/session` | Who this request is, and whether a credential is needed at all
|
|
782
|
-
| `GET` | `/api/catalog` | The whole catalogue, for a peer keeping itself in step
|
|
783
|
-
| `GET` | `/archives/:infoHash/tiles.json` | TileJSON — **public**
|
|
784
|
-
| `GET` | `/archives/:infoHash/:z/:x/:y.:ext` | One tile — **public**
|
|
785
|
-
| `GET` | `/health` | 200 when this node can serve, 503 when its engine cannot — **public**, no credential, for a load balancer
|
|
786
|
-
| `GET` | `/archives/:infoHash/ready` | Whether this node can serve _this_ archive yet: 200 ready, 503 not yet, 415 never — **public**
|
|
787
|
-
| `GET` | `/archives/:infoHash/archive.torrent` | The `.torrent` a torrent-aware client joins with — **public**
|
|
788
|
-
| `GET` | `/archives/:infoHash/archive.pmtiles` | The archive itself, by byte range — what any PMTiles reader wants, and a valid web seed.
|
|
789
|
-
| `GET` | `/archives/:infoHash/preview` | Map preview for one archive — admin, not public
|
|
790
|
-
| `GET` | `/latest/` | Every category and the endpoints that resolve to its newest build — **public**, the index of everything below
|
|
791
|
-
| `GET` | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is
|
|
792
|
-
| `GET` | `/latest/:category/archive.pmtiles` | The newest build in a category as a file, by byte range — **public**, and gated by [`serveArchive`](docs/configuration.md#servearchive). What to point a PMTiles reader at when you want "whichever is current" rather than one build
|
|
793
|
-
| `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public**
|
|
742
|
+
| Method | Path | Purpose |
|
|
743
|
+
| --------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
744
|
+
| `GET` | `/api/status` | Engine health, counts, watched folders, save locations, free space, and whether the swarm can reach this node |
|
|
745
|
+
| `GET` | `/api/torrents` | Catalog joined with live swarm state and what is left of each seeding limit |
|
|
746
|
+
| `POST` | `/api/torrents` | Add via `{path}`, `{url}`, `{magnet}`, `{torrentUrl}`, or a raw `.torrent` body |
|
|
747
|
+
| `DELETE` | `/api/torrents/:infoHash` | Remove (`?deleteData=true` to delete data too) |
|
|
748
|
+
| `GET` | `/api/torrents/:infoHash` | One archive, with disk usage and how it is being read |
|
|
749
|
+
| `GET` | `/api/torrents/:infoHash/file`, `/magnet` | Download the `.torrent`, or its magnet URI |
|
|
750
|
+
| `GET` | `/api/torrents/:infoHash/peers`, `/trackers`, `/content` | Per-peer, per-tracker and per-file detail |
|
|
751
|
+
| `GET` | `/api/torrents/:infoHash/pieces` | Which pieces are held, how rare each is, and what peers hold |
|
|
752
|
+
| `PATCH` | `/api/torrents/:infoHash/mode` | Switch between mirror and cache |
|
|
753
|
+
| `PATCH` | `/api/torrents/:infoHash/categories` | Set, add or remove categories |
|
|
754
|
+
| `PATCH` | `/api/torrents/:infoHash/seeding` | Per-archive seeding limit, or "use the global one" |
|
|
755
|
+
| `PATCH` `GET` | `/api/torrents/:infoHash/location` | Move the data; poll the move |
|
|
756
|
+
| `POST` | `/api/torrents/:infoHash/recheck` | Hash what is on disk again and believe the result — for an archive whose progress and whose files disagree |
|
|
757
|
+
| `POST` | `/api/torrents/:infoHash/pause`, `/resume` | Stop offering it, without forgetting it |
|
|
758
|
+
| `POST` | `/api/torrents/:infoHash/webseeds` | Add web seeds — does not change the infohash |
|
|
759
|
+
| `DELETE` | `/api/torrents/:infoHash/webseeds` | Drop web seeds — likewise |
|
|
760
|
+
| `POST` | `/api/torrents/:infoHash/publish` | What this node offers of the archive itself: `serveArchive`, `selfWebSeed`, `publicDownload` |
|
|
761
|
+
| `POST` | `/api/torrents/:infoHash/warm` | Pre-fetch a region (`GET` for progress, `DELETE` to cancel) |
|
|
762
|
+
| `DELETE` | `/api/torrents/:infoHash/cache` | Reclaim cached pieces, keep the archive |
|
|
763
|
+
| `POST` | `/api/torrents/:infoHash/check` | Has the source changed since the torrent was made? |
|
|
764
|
+
| `POST` | `/api/torrents/:infoHash/rebuild` | Rebuild from the current source (mints a new infohash) |
|
|
765
|
+
| `POST` | `/api/check-origins` | Check every archive with a watchable source |
|
|
766
|
+
| `GET` `DELETE` | `/api/adds` | Adds still in flight — downloads, and local files being hashed — and cancelling a download by URL. A hash cannot be stopped part-way, so it is listed but not cancellable |
|
|
767
|
+
| `GET` `POST` | `/api/speed` | Which speed limits are in force, and the manual switch between the two sets |
|
|
768
|
+
| `GET` | `/api/categories` | Every category, with the endpoints resolving to its newest build |
|
|
769
|
+
| `POST` | `/api/adopt`, `/api/adopt/candidates` | Take over what an engine or another node holds |
|
|
770
|
+
| `POST` | `/api/sources/preview` | What a watched web location would take, without taking it |
|
|
771
|
+
| `POST` | `/api/subscriptions/preview` | Whether a peer is reachable and what it offers |
|
|
772
|
+
| `POST` | `/api/subscriptions/refresh` | Poll subscribed feeds now |
|
|
773
|
+
| `POST` | `/api/sources/check` | Check scheduled sources now, rather than at the next due time |
|
|
774
|
+
| `POST` | `/api/torrents/:infoHash/hooks/complete` | Run the completion hook again for one archive |
|
|
775
|
+
| `GET` `POST` `DELETE` | `/api/tokens`, `/api/tokens/:id` | Mint, list and revoke access tokens |
|
|
776
|
+
| `GET` `POST` | `/api/restart` | What a restart would do, and doing it |
|
|
777
|
+
| `GET` `DELETE` | `/api/stats` | What this node has served — per-archive counters and the last N requests; `DELETE` clears them |
|
|
778
|
+
| `GET` | `/api/traffic` | Upload and download speed per archive over time, from `stats.db`; `infoHash`, `hours` and `buckets` narrow it, and `totals` ranks archives by bytes moved |
|
|
779
|
+
| `GET` `PATCH` | `/api/config` | Read and change settings |
|
|
780
|
+
| `POST` | `/api/login`, `/api/logout` | Console sign-in |
|
|
781
|
+
| `GET` | `/api/session` | Who this request is, and whether a credential is needed at all |
|
|
782
|
+
| `GET` | `/api/catalog` | The whole catalogue, for a peer keeping itself in step |
|
|
783
|
+
| `GET` | `/archives/:infoHash/tiles.json` | TileJSON — **public** |
|
|
784
|
+
| `GET` | `/archives/:infoHash/:z/:x/:y.:ext` | One tile — **public** |
|
|
785
|
+
| `GET` | `/health` | 200 when this node can serve, 503 when its engine cannot — **public**, no credential, for a load balancer |
|
|
786
|
+
| `GET` | `/archives/:infoHash/ready` | Whether this node can serve _this_ archive yet: 200 ready, 503 not yet, 415 never — **public** |
|
|
787
|
+
| `GET` | `/archives/:infoHash/archive.torrent` | The `.torrent` a torrent-aware client joins with — **public** |
|
|
788
|
+
| `GET` | `/archives/:infoHash/archive.pmtiles` | The archive itself, by byte range — what any PMTiles reader wants, and a valid web seed. Only where [`serveArchive`](docs/configuration.md#servearchive) is on — **public**. Complete archives, unless [`serveArchiveFromSwarm`](docs/configuration.md#servearchivefromswarm) is set, which answers a bounded range from the swarm instead |
|
|
789
|
+
| `GET` | `/archives/:infoHash/preview` | Map preview for one archive — admin, not public |
|
|
790
|
+
| `GET` | `/latest/` | Every category and the endpoints that resolve to its newest build — **public**, the index of everything below |
|
|
791
|
+
| `GET` | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is |
|
|
792
|
+
| `GET` | `/latest/:category/archive.pmtiles` | The newest build in a category as a file, by byte range — **public**, and gated by [`serveArchive`](docs/configuration.md#servearchive). What to point a PMTiles reader at when you want "whichever is current" rather than one build |
|
|
793
|
+
| `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public** |
|
|
794
794
|
|
|
795
795
|
Everything under `/api/` is guarded once a credential is configured; tiles, TileJSON and the feeds
|
|
796
796
|
never are. A `peer` token may read but not change, and may be narrowed to some categories. See
|
package/docs/configuration.md
CHANGED
|
@@ -315,12 +315,13 @@ rotation stays out across the restart you were probably about to do.
|
|
|
315
315
|
|
|
316
316
|
## Offering the archive file itself
|
|
317
317
|
|
|
318
|
-
| setting
|
|
319
|
-
|
|
|
320
|
-
| `serveArchive`
|
|
321
|
-
| `publishingUrl`
|
|
322
|
-
| `
|
|
323
|
-
| `
|
|
318
|
+
| setting | default | |
|
|
319
|
+
| ----------------------- | ------- | ------------------------------------------------------------------------ |
|
|
320
|
+
| `serveArchive` | `false` | answer `/archives/<infohash>/archive.pmtiles` |
|
|
321
|
+
| `publishingUrl` | unset | the address to use where a URL must outlive the request |
|
|
322
|
+
| `serveArchiveFromSwarm` | `false` | answer a range for an archive this node does not hold — **experimental** |
|
|
323
|
+
| `selfWebSeed` | `false` | publish this node as a web seed for archives it holds |
|
|
324
|
+
| `publicDownload` | `false` | offer them as downloads on the public catalogue page |
|
|
324
325
|
|
|
325
326
|
Three switches rather than one, because they are three different exposures and a
|
|
326
327
|
node can reasonably want any of them without the others.
|
|
@@ -346,6 +347,38 @@ GET /latest/<category>/archive.pmtiles whichever is current, with an ETag
|
|
|
346
347
|
|
|
347
348
|
With it off, both answer `403`.
|
|
348
349
|
|
|
350
|
+
### `serveArchiveFromSwarm`
|
|
351
|
+
|
|
352
|
+
**Experimental, and not recommended for anything public.** Off by default.
|
|
353
|
+
|
|
354
|
+
With it on, a byte range for an archive this node does _not_ hold is answered by
|
|
355
|
+
pulling the covering pieces out of the swarm on demand — the same path the tile
|
|
356
|
+
endpoint has always taken internally, one HTTP layer further out, sharing its
|
|
357
|
+
piece cache and its open handle. It is the loop cache mode was built for: point
|
|
358
|
+
an ordinary PMTiles reader at a node holding none of the file, and it works.
|
|
359
|
+
|
|
360
|
+
The reservations are properties of the arrangement rather than of the code:
|
|
361
|
+
|
|
362
|
+
- **Every byte is somebody else's upload.** A cache-mode node is not an origin,
|
|
363
|
+
and putting one behind a URL that looks like one turns each request into swarm
|
|
364
|
+
traffic it neither paid for nor holds.
|
|
365
|
+
- **A piece read takes as long as the swarm takes.** Acceptable for a tile, which
|
|
366
|
+
a reader asked for and will wait on; poor for an HTTP client with its own
|
|
367
|
+
timeout. `swarmRangeTimeoutMs` (30s) bounds it and answers `504`.
|
|
368
|
+
- **There is no honest answer to a request for the whole file.** A `Range` header
|
|
369
|
+
is required — without one the request is refused with `411` — and a range
|
|
370
|
+
larger than `swarmRangeLimitBytes` (8 MiB) with `416`.
|
|
371
|
+
|
|
372
|
+
The response carries the same `ETag` and the same year-long `immutable` caching a
|
|
373
|
+
complete copy would give, because it is the same content: the infohash names
|
|
374
|
+
those bytes wherever they were read from.
|
|
375
|
+
|
|
376
|
+
**A node cannot be a web seed for an archive it does not hold**, whatever this is
|
|
377
|
+
set to. [`selfWebSeed`](#selfwebseed) is refused on an incomplete archive and
|
|
378
|
+
offered again when the download finishes — answering from the swarm and then
|
|
379
|
+
advertising that to the swarm is a loop with an amplifier in it, where every peer
|
|
380
|
+
that takes the seed makes this node fetch the piece again to serve it.
|
|
381
|
+
|
|
349
382
|
### `publishingUrl`
|
|
350
383
|
|
|
351
384
|
The address to use for URLs that outlive the request that made them.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.54.0",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
package/src/api.js
CHANGED
|
@@ -6,6 +6,7 @@ import os from 'node:os';
|
|
|
6
6
|
import path from 'node:path';
|
|
7
7
|
import { fileURLToPath } from 'node:url';
|
|
8
8
|
import express from 'express';
|
|
9
|
+
import parseRange from 'range-parser';
|
|
9
10
|
import {
|
|
10
11
|
ROLES,
|
|
11
12
|
createAuth,
|
|
@@ -2332,6 +2333,121 @@ export function createApp({
|
|
|
2332
2333
|
'ETag, Content-Range, Content-Length, Accept-Ranges',
|
|
2333
2334
|
};
|
|
2334
2335
|
|
|
2336
|
+
/**
|
|
2337
|
+
* Serves a byte range for an archive this node does not hold whole.
|
|
2338
|
+
*
|
|
2339
|
+
* **Experimental.** It closes the loop the cache mode was built for: the
|
|
2340
|
+
* node holds no bytes, a reader asks for some, and the pieces covering them
|
|
2341
|
+
* are pulled out of the swarm on demand — the same path the tile endpoint
|
|
2342
|
+
* has always taken internally, one HTTP layer further out and sharing the
|
|
2343
|
+
* same piece cache and open handle.
|
|
2344
|
+
*
|
|
2345
|
+
* It is off by default and should stay off for anything public, for reasons
|
|
2346
|
+
* that are properties of the arrangement rather than of this code:
|
|
2347
|
+
*
|
|
2348
|
+
* - Every byte is somebody else's upload. A node in cache mode is not an
|
|
2349
|
+
* origin; putting one behind a URL that looks like an origin turns each
|
|
2350
|
+
* request into swarm traffic that this node neither paid for nor holds.
|
|
2351
|
+
* - A piece read takes as long as the swarm takes. That is fine for a tile,
|
|
2352
|
+
* which the reader asked for and will wait on, and poor for an HTTP client
|
|
2353
|
+
* with its own patience.
|
|
2354
|
+
* - There is no way to answer a request for the whole file that is not
|
|
2355
|
+
* "download 700 GiB through BitTorrent and stream it out". So a range is
|
|
2356
|
+
* required, and a large one is refused.
|
|
2357
|
+
* @param {import('express').Request} req - The request, for its Range.
|
|
2358
|
+
* @param {import('express').Response} res - The response.
|
|
2359
|
+
* @param {object} entry - The archive.
|
|
2360
|
+
* @returns {Promise<void>} - Once answered.
|
|
2361
|
+
*/
|
|
2362
|
+
const serveFromSwarm = async (req, res, entry) => {
|
|
2363
|
+
if (!config.serveArchiveFromSwarm || !tiles?.readRange) {
|
|
2364
|
+
return res.status(409).json({
|
|
2365
|
+
error:
|
|
2366
|
+
'this archive is not complete here, so a byte range would answer ' +
|
|
2367
|
+
'with unwritten space rather than data',
|
|
2368
|
+
});
|
|
2369
|
+
}
|
|
2370
|
+
|
|
2371
|
+
const size = Number(entry.size);
|
|
2372
|
+
if (!Number.isFinite(size) || size <= 0) {
|
|
2373
|
+
return res.status(409).json({
|
|
2374
|
+
error: 'this archive has no known length here yet',
|
|
2375
|
+
});
|
|
2376
|
+
}
|
|
2377
|
+
|
|
2378
|
+
// A range, and only a range. Without one the answer is the whole archive,
|
|
2379
|
+
// and the whole archive is not on this disk — serving it would mean
|
|
2380
|
+
// pulling every piece through the swarm to stream it back out, which is
|
|
2381
|
+
// both enormous and somebody else's bandwidth.
|
|
2382
|
+
const asked = req.headers.range;
|
|
2383
|
+
if (!asked) {
|
|
2384
|
+
res.setHeader('accept-ranges', 'bytes');
|
|
2385
|
+
return res.status(411).json({
|
|
2386
|
+
error:
|
|
2387
|
+
'this node does not hold this archive, so it can only answer a ' +
|
|
2388
|
+
'byte range. Send a Range header.',
|
|
2389
|
+
});
|
|
2390
|
+
}
|
|
2391
|
+
|
|
2392
|
+
const ranges = parseRange(size, asked, { combine: true });
|
|
2393
|
+
// -1 is unsatisfiable, -2 is malformed, and more than one range would mean
|
|
2394
|
+
// a multipart response that nothing reading PMTiles has ever asked for.
|
|
2395
|
+
if (ranges === -1) {
|
|
2396
|
+
res.setHeader('content-range', `bytes */${size}`);
|
|
2397
|
+
return res.status(416).json({ error: 'range not satisfiable' });
|
|
2398
|
+
}
|
|
2399
|
+
if (ranges === -2 || ranges.length !== 1 || ranges.type !== 'bytes') {
|
|
2400
|
+
return res.status(400).json({ error: 'unreadable Range header' });
|
|
2401
|
+
}
|
|
2402
|
+
|
|
2403
|
+
const { start, end } = ranges[0];
|
|
2404
|
+
const length = end - start + 1;
|
|
2405
|
+
const limit = config.swarmRangeLimitBytes ?? 8 * 1024 * 1024;
|
|
2406
|
+
if (length > limit) {
|
|
2407
|
+
return res.status(416).json({
|
|
2408
|
+
error:
|
|
2409
|
+
`this node does not hold this archive, so a range is fetched from ` +
|
|
2410
|
+
`the swarm a piece at a time; ${length} bytes is more than the ` +
|
|
2411
|
+
`${limit} it will do that for at once`,
|
|
2412
|
+
});
|
|
2413
|
+
}
|
|
2414
|
+
|
|
2415
|
+
// A deadline of its own, shorter than the piece timeout underneath. An
|
|
2416
|
+
// HTTP client gives up long before libtorrent does, and a request left
|
|
2417
|
+
// hanging on a piece nobody is seeding holds a connection open for nothing.
|
|
2418
|
+
const controller = new AbortController();
|
|
2419
|
+
const deadline = setTimeout(
|
|
2420
|
+
() => controller.abort(),
|
|
2421
|
+
config.swarmRangeTimeoutMs ?? 30000,
|
|
2422
|
+
);
|
|
2423
|
+
try {
|
|
2424
|
+
const body = await tiles.readRange(entry.infoHash, start, length, {
|
|
2425
|
+
signal: controller.signal,
|
|
2426
|
+
});
|
|
2427
|
+
res.status(206);
|
|
2428
|
+
res.setHeader('accept-ranges', 'bytes');
|
|
2429
|
+
res.setHeader('content-range', `bytes ${start}-${end}/${size}`);
|
|
2430
|
+
res.setHeader('content-type', 'application/octet-stream');
|
|
2431
|
+
for (const [name, value] of Object.entries(RANGE_CORS_HEADERS)) {
|
|
2432
|
+
res.setHeader(name, value);
|
|
2433
|
+
}
|
|
2434
|
+
// The same tag a complete copy would answer with, because it is the same
|
|
2435
|
+
// content: the infohash names these bytes wherever they were read from.
|
|
2436
|
+
res.setHeader('etag', `"${entry.infoHash}"`);
|
|
2437
|
+
res.setHeader('cache-control', 'public, max-age=31536000, immutable');
|
|
2438
|
+
res.send(body);
|
|
2439
|
+
} catch (error) {
|
|
2440
|
+
if (controller.signal.aborted) {
|
|
2441
|
+
return res.status(504).json({
|
|
2442
|
+
error: 'the swarm did not answer for those pieces in time',
|
|
2443
|
+
});
|
|
2444
|
+
}
|
|
2445
|
+
res.status(error.status ?? 502).json({ error: error.message });
|
|
2446
|
+
} finally {
|
|
2447
|
+
clearTimeout(deadline);
|
|
2448
|
+
}
|
|
2449
|
+
};
|
|
2450
|
+
|
|
2335
2451
|
/**
|
|
2336
2452
|
* The current build of a category, as a file, by byte range.
|
|
2337
2453
|
*
|
|
@@ -2367,13 +2483,11 @@ export function createApp({
|
|
|
2367
2483
|
});
|
|
2368
2484
|
}
|
|
2369
2485
|
|
|
2370
|
-
|
|
2371
|
-
|
|
2372
|
-
|
|
2373
|
-
|
|
2374
|
-
|
|
2375
|
-
});
|
|
2376
|
-
}
|
|
2486
|
+
// Not on this disk. Either the node can read it out of the swarm, which is
|
|
2487
|
+
// what cache mode is for, or it says so — but it never answers a range
|
|
2488
|
+
// from a sparse file, where the unwritten space reads as zeroes and looks
|
|
2489
|
+
// exactly like data.
|
|
2490
|
+
if (entry.complete === false) return serveFromSwarm(req, res, entry);
|
|
2377
2491
|
const file = entry.savePath ? path.join(entry.savePath, entry.name) : null;
|
|
2378
2492
|
if (!file) return res.status(404).json({ error: 'no file for it here' });
|
|
2379
2493
|
|
|
@@ -2712,13 +2826,9 @@ export function createApp({
|
|
|
2712
2826
|
});
|
|
2713
2827
|
}
|
|
2714
2828
|
|
|
2715
|
-
|
|
2716
|
-
|
|
2717
|
-
|
|
2718
|
-
'this archive is not complete here, so a byte range would answer ' +
|
|
2719
|
-
'with unwritten space rather than data',
|
|
2720
|
-
});
|
|
2721
|
-
}
|
|
2829
|
+
// See the /latest/ route above: read from the swarm where the node is
|
|
2830
|
+
// configured to, and refuse rather than answer with unwritten space.
|
|
2831
|
+
if (entry.complete === false) return serveFromSwarm(req, res, entry);
|
|
2722
2832
|
const file = entry.savePath ? path.join(entry.savePath, entry.name) : null;
|
|
2723
2833
|
if (!file) return res.status(404).json({ error: 'no file for it here' });
|
|
2724
2834
|
|
package/src/config.js
CHANGED
|
@@ -204,6 +204,45 @@ const DEFAULTS = {
|
|
|
204
204
|
* a link that answers 403 is worse than no link.
|
|
205
205
|
*/
|
|
206
206
|
publicDownload: false,
|
|
207
|
+
/**
|
|
208
|
+
* Answer a byte range for an archive this node does not hold, by pulling the
|
|
209
|
+
* covering pieces out of the swarm on demand.
|
|
210
|
+
*
|
|
211
|
+
* **Experimental, and not recommended for anything public.** It closes the
|
|
212
|
+
* loop cache mode was built for — the node holds no bytes, a reader asks for
|
|
213
|
+
* some, and the pieces arrive from peers — over the same path the tile
|
|
214
|
+
* endpoint has always used internally, sharing its piece cache and its open
|
|
215
|
+
* handle. As a way to point an ordinary PMTiles reader at an archive nothing
|
|
216
|
+
* here has a copy of, it works.
|
|
217
|
+
*
|
|
218
|
+
* The reservations are properties of the arrangement rather than of the
|
|
219
|
+
* implementation:
|
|
220
|
+
*
|
|
221
|
+
* - Every byte is somebody else's upload. A cache-mode node is not an origin,
|
|
222
|
+
* and putting one behind a URL that looks like one turns each request into
|
|
223
|
+
* swarm traffic it neither paid for nor holds.
|
|
224
|
+
* - A piece read takes as long as the swarm takes. Acceptable for a tile,
|
|
225
|
+
* which a reader asked for and will wait on; poor for an HTTP client with
|
|
226
|
+
* its own patience and its own timeout.
|
|
227
|
+
* - There is no honest answer to a request for the whole file, so a Range
|
|
228
|
+
* header is required and a large one is refused.
|
|
229
|
+
*/
|
|
230
|
+
serveArchiveFromSwarm: false,
|
|
231
|
+
/**
|
|
232
|
+
* The largest range `serveArchiveFromSwarm` will fetch at once, in bytes.
|
|
233
|
+
*
|
|
234
|
+
* Sized for what a PMTiles reader actually asks for — a 16 KiB header, a
|
|
235
|
+
* directory, a tile — rather than for bulk transfer, which is the thing this
|
|
236
|
+
* must not quietly become.
|
|
237
|
+
*/
|
|
238
|
+
swarmRangeLimitBytes: 8 * 1024 * 1024,
|
|
239
|
+
/**
|
|
240
|
+
* How long to wait for the swarm before giving up on a range, in
|
|
241
|
+
* milliseconds. Shorter than the piece timeout underneath on purpose: an
|
|
242
|
+
* HTTP client gives up long before libtorrent does, and a request left
|
|
243
|
+
* hanging on a piece nobody is seeding holds a connection open for nothing.
|
|
244
|
+
*/
|
|
245
|
+
swarmRangeTimeoutMs: 30000,
|
|
207
246
|
/**
|
|
208
247
|
* How long an unfinished download is kept before startup treats it as
|
|
209
248
|
* abandoned. Until then, re-adding the same URL resumes it.
|
package/src/library.js
CHANGED
|
@@ -2909,6 +2909,19 @@ export class Library {
|
|
|
2909
2909
|
let webSeed = published;
|
|
2910
2910
|
let warning = null;
|
|
2911
2911
|
|
|
2912
|
+
// A node cannot be a web seed for bytes it does not have. It might be able
|
|
2913
|
+
// to *answer* for them, where serveArchiveFromSwarm is on -- but answering
|
|
2914
|
+
// by fetching from the swarm and then advertising that to the swarm is a
|
|
2915
|
+
// loop with an amplifier in it: every peer that takes the seed makes this
|
|
2916
|
+
// node download the piece again to serve it.
|
|
2917
|
+
if (after.selfWebSeed && !published && entry.complete === false) {
|
|
2918
|
+
throw new Error(
|
|
2919
|
+
'this node does not hold a complete copy of this archive, so it ' +
|
|
2920
|
+
'cannot be a web seed for it. It will be offered again once the ' +
|
|
2921
|
+
'download finishes.',
|
|
2922
|
+
);
|
|
2923
|
+
}
|
|
2924
|
+
|
|
2912
2925
|
if (after.selfWebSeed && !published) {
|
|
2913
2926
|
const base = publishingBase({
|
|
2914
2927
|
// Given outright beats the node's own answer, which beats the request
|
package/src/tiles.js
CHANGED
|
@@ -119,6 +119,43 @@ export class TileStore {
|
|
|
119
119
|
return { data: gzipped, encoding: 'gzip' };
|
|
120
120
|
}
|
|
121
121
|
|
|
122
|
+
/**
|
|
123
|
+
* Reads a byte range out of an archive, through whatever source applies.
|
|
124
|
+
*
|
|
125
|
+
* The same acquisition every tile read goes through, so a cache-mode archive
|
|
126
|
+
* answers from the swarm and a complete one answers from its file — and both
|
|
127
|
+
* share the piece cache, the directory prefetch and the open handle that the
|
|
128
|
+
* tile path has already warmed. That sharing is the point: a reader fetching
|
|
129
|
+
* the header, then the root directory, then a tile is doing exactly what the
|
|
130
|
+
* tile endpoint does internally, one HTTP layer further out.
|
|
131
|
+
*
|
|
132
|
+
* The range is not split here. `TorrentSource.getBytes` already fetches every
|
|
133
|
+
* covering piece concurrently, which is what stops a three-piece range paying
|
|
134
|
+
* three sequential swarm round-trips.
|
|
135
|
+
* @param {string} infoHash - Which archive.
|
|
136
|
+
* @param {number} offset - Byte offset into the archive file.
|
|
137
|
+
* @param {number} length - How many bytes.
|
|
138
|
+
* @param {object} [options] - Abort signal.
|
|
139
|
+
* @returns {Promise<Buffer>} - The bytes, clamped to the end of the file.
|
|
140
|
+
*/
|
|
141
|
+
async readRange(infoHash, offset, length, options = {}) {
|
|
142
|
+
const entry = this.#catalog.get(infoHash);
|
|
143
|
+
if (!entry) throw new TileReadError('unknown archive', 404);
|
|
144
|
+
|
|
145
|
+
const handle = await this.#acquire(entry);
|
|
146
|
+
if (!handle.source) {
|
|
147
|
+
// MBTiles opens as a database rather than a byte source, and a byte
|
|
148
|
+
// range into one would be a range into a SQLite file — technically
|
|
149
|
+
// answerable and useless to everybody.
|
|
150
|
+
throw new TileReadError(
|
|
151
|
+
'this archive is not readable as a stream of bytes',
|
|
152
|
+
415,
|
|
153
|
+
);
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
const answer = await handle.source.getBytes(offset, length, options.signal);
|
|
157
|
+
return Buffer.from(answer.data);
|
|
158
|
+
}
|
|
122
159
|
/**
|
|
123
160
|
* Reads an archive's header and metadata, through whatever source applies.
|
|
124
161
|
*
|