pmtiles-swarm 0.52.0 → 0.54.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (10) hide show
  1. package/CHANGELOG.md +45 -0
  2. package/README.md +52 -52
  3. package/docs/configuration.md +56 -12
  4. package/package.json +1 -1
  5. package/src/api.js +124 -14
  6. package/src/config.js +39 -0
  7. package/src/library.js +51 -0
  8. package/src/subscriptions.js +8 -0
  9. package/src/tiles.js +37 -0
  10. package/src/web/index.html +32 -0
package/CHANGELOG.md CHANGED
@@ -7,6 +7,51 @@
7
7
  ### 🐞 Bug fixes
8
8
  - _...Add new stuff here..._
9
9
 
10
+ ## 0.54.0
11
+ ### ✨ Features and improvements
12
+ - **`serveArchiveFromSwarm` — a byte range for an archive this node does not hold.** Experimental,
13
+ off by default, and the loop cache mode was built for: the node holds no bytes, a reader asks for
14
+ some, and the pieces covering them arrive from peers. It reuses the machinery the tile endpoint has
15
+ always used internally — `TorrentSource.getBytes` already fetches every covering piece
16
+ concurrently, and this shares its piece cache, its directory prefetch and its open handle. Point an
17
+ ordinary PMTiles reader at a node with no copy of the archive and it works.
18
+
19
+ Bounded on purpose. A `Range` header is required, since the only other answer is "pull 700 GiB
20
+ through BitTorrent and stream it out" — no header is `411`, a range over `swarmRangeLimitBytes`
21
+ (8 MiB) is `416`, and a swarm that does not answer within `swarmRangeTimeoutMs` (30s) is `504`
22
+ rather than a socket held open for two minutes. The response carries the same `ETag` and the same
23
+ year-long `immutable` caching a complete copy would, because it is the same content: the infohash
24
+ names those bytes wherever they were read from.
25
+
26
+ Not recommended for anything public, for reasons that are properties of the arrangement rather than
27
+ of the code — every byte is somebody else's upload, and a cache-mode node behind a URL that looks
28
+ like an origin is not one.
29
+
30
+ ### 🐞 Bug fixes
31
+ - **A node could be made a web seed for an archive it did not hold.** `selfWebSeed` is now refused on
32
+ an incomplete archive and offered again once the download finishes. Answering from the swarm and
33
+ then advertising that to the swarm is a loop with an amplifier in it: every peer that takes the
34
+ seed makes this node fetch the piece again in order to serve it.
35
+
36
+ ## 0.53.0
37
+ ### ✨ Features and improvements
38
+ - **The three switches are now settable on every import, from the console.** **Serve file**, **Web
39
+ seed** and **Listed** columns on monitored folders, watched web locations, RSS feeds and remote
40
+ nodes, each offering `node` as well as yes and no — because unset is a real answer here, meaning
41
+ the archive follows the node rather than being switched off.
42
+
43
+ ### 🐞 Bug fixes
44
+ - **An RSS feed could not ask for any of them.** 0.50.0 wired these through watched folders and
45
+ scheduled sources and stopped there, and an archive adopted from a feed takes a different path
46
+ through the library — which is the path a mirror node actually uses. A subscription now carries all
47
+ three, and there was no way to set them from the console on any import at all.
48
+
49
+ - **`selfWebSeed` on an import waits for the download to finish.** Publishing a web seed URL for an
50
+ archive that is still arriving would advertise an address answering `409`, and a peer handed a URL
51
+ that refuses spends its retries on it — worse than no web seed, and unfixable afterwards because by
52
+ then the URL is in every copy of the `.torrent`. The intention is recorded when the archive joins
53
+ and acted on at the first moment this node holds the whole file.
54
+
10
55
  ## 0.52.0
11
56
  ### ✨ Features and improvements
12
57
  - **`publishingUrl`, for the URLs that have to be permanent.** Almost every URL this node emits is
package/README.md CHANGED
@@ -739,58 +739,58 @@ which the endpoint answers 501.
739
739
 
740
740
  ## API
741
741
 
742
- | Method | Path | Purpose |
743
- | --------------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
744
- | `GET` | `/api/status` | Engine health, counts, watched folders, save locations, free space, and whether the swarm can reach this node |
745
- | `GET` | `/api/torrents` | Catalog joined with live swarm state and what is left of each seeding limit |
746
- | `POST` | `/api/torrents` | Add via `{path}`, `{url}`, `{magnet}`, `{torrentUrl}`, or a raw `.torrent` body |
747
- | `DELETE` | `/api/torrents/:infoHash` | Remove (`?deleteData=true` to delete data too) |
748
- | `GET` | `/api/torrents/:infoHash` | One archive, with disk usage and how it is being read |
749
- | `GET` | `/api/torrents/:infoHash/file`, `/magnet` | Download the `.torrent`, or its magnet URI |
750
- | `GET` | `/api/torrents/:infoHash/peers`, `/trackers`, `/content` | Per-peer, per-tracker and per-file detail |
751
- | `GET` | `/api/torrents/:infoHash/pieces` | Which pieces are held, how rare each is, and what peers hold |
752
- | `PATCH` | `/api/torrents/:infoHash/mode` | Switch between mirror and cache |
753
- | `PATCH` | `/api/torrents/:infoHash/categories` | Set, add or remove categories |
754
- | `PATCH` | `/api/torrents/:infoHash/seeding` | Per-archive seeding limit, or "use the global one" |
755
- | `PATCH` `GET` | `/api/torrents/:infoHash/location` | Move the data; poll the move |
756
- | `POST` | `/api/torrents/:infoHash/recheck` | Hash what is on disk again and believe the result — for an archive whose progress and whose files disagree |
757
- | `POST` | `/api/torrents/:infoHash/pause`, `/resume` | Stop offering it, without forgetting it |
758
- | `POST` | `/api/torrents/:infoHash/webseeds` | Add web seeds — does not change the infohash |
759
- | `DELETE` | `/api/torrents/:infoHash/webseeds` | Drop web seeds — likewise |
760
- | `POST` | `/api/torrents/:infoHash/publish` | What this node offers of the archive itself: `serveArchive`, `selfWebSeed`, `publicDownload` |
761
- | `POST` | `/api/torrents/:infoHash/warm` | Pre-fetch a region (`GET` for progress, `DELETE` to cancel) |
762
- | `DELETE` | `/api/torrents/:infoHash/cache` | Reclaim cached pieces, keep the archive |
763
- | `POST` | `/api/torrents/:infoHash/check` | Has the source changed since the torrent was made? |
764
- | `POST` | `/api/torrents/:infoHash/rebuild` | Rebuild from the current source (mints a new infohash) |
765
- | `POST` | `/api/check-origins` | Check every archive with a watchable source |
766
- | `GET` `DELETE` | `/api/adds` | Adds still in flight — downloads, and local files being hashed — and cancelling a download by URL. A hash cannot be stopped part-way, so it is listed but not cancellable |
767
- | `GET` `POST` | `/api/speed` | Which speed limits are in force, and the manual switch between the two sets |
768
- | `GET` | `/api/categories` | Every category, with the endpoints resolving to its newest build |
769
- | `POST` | `/api/adopt`, `/api/adopt/candidates` | Take over what an engine or another node holds |
770
- | `POST` | `/api/sources/preview` | What a watched web location would take, without taking it |
771
- | `POST` | `/api/subscriptions/preview` | Whether a peer is reachable and what it offers |
772
- | `POST` | `/api/subscriptions/refresh` | Poll subscribed feeds now |
773
- | `POST` | `/api/sources/check` | Check scheduled sources now, rather than at the next due time |
774
- | `POST` | `/api/torrents/:infoHash/hooks/complete` | Run the completion hook again for one archive |
775
- | `GET` `POST` `DELETE` | `/api/tokens`, `/api/tokens/:id` | Mint, list and revoke access tokens |
776
- | `GET` `POST` | `/api/restart` | What a restart would do, and doing it |
777
- | `GET` `DELETE` | `/api/stats` | What this node has served — per-archive counters and the last N requests; `DELETE` clears them |
778
- | `GET` | `/api/traffic` | Upload and download speed per archive over time, from `stats.db`; `infoHash`, `hours` and `buckets` narrow it, and `totals` ranks archives by bytes moved |
779
- | `GET` `PATCH` | `/api/config` | Read and change settings |
780
- | `POST` | `/api/login`, `/api/logout` | Console sign-in |
781
- | `GET` | `/api/session` | Who this request is, and whether a credential is needed at all |
782
- | `GET` | `/api/catalog` | The whole catalogue, for a peer keeping itself in step |
783
- | `GET` | `/archives/:infoHash/tiles.json` | TileJSON — **public** |
784
- | `GET` | `/archives/:infoHash/:z/:x/:y.:ext` | One tile — **public** |
785
- | `GET` | `/health` | 200 when this node can serve, 503 when its engine cannot — **public**, no credential, for a load balancer |
786
- | `GET` | `/archives/:infoHash/ready` | Whether this node can serve _this_ archive yet: 200 ready, 503 not yet, 415 never — **public** |
787
- | `GET` | `/archives/:infoHash/archive.torrent` | The `.torrent` a torrent-aware client joins with — **public** |
788
- | `GET` | `/archives/:infoHash/archive.pmtiles` | The archive itself, by byte range — what any PMTiles reader wants, and a valid web seed. Complete archives only, and only where [`serveArchive`](docs/configuration.md#servearchive) is on — **public** |
789
- | `GET` | `/archives/:infoHash/preview` | Map preview for one archive — admin, not public |
790
- | `GET` | `/latest/` | Every category and the endpoints that resolve to its newest build — **public**, the index of everything below |
791
- | `GET` | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is |
792
- | `GET` | `/latest/:category/archive.pmtiles` | The newest build in a category as a file, by byte range — **public**, and gated by [`serveArchive`](docs/configuration.md#servearchive). What to point a PMTiles reader at when you want "whichever is current" rather than one build |
793
- | `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public** |
742
+ | Method | Path | Purpose |
743
+ | --------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
744
+ | `GET` | `/api/status` | Engine health, counts, watched folders, save locations, free space, and whether the swarm can reach this node |
745
+ | `GET` | `/api/torrents` | Catalog joined with live swarm state and what is left of each seeding limit |
746
+ | `POST` | `/api/torrents` | Add via `{path}`, `{url}`, `{magnet}`, `{torrentUrl}`, or a raw `.torrent` body |
747
+ | `DELETE` | `/api/torrents/:infoHash` | Remove (`?deleteData=true` to delete data too) |
748
+ | `GET` | `/api/torrents/:infoHash` | One archive, with disk usage and how it is being read |
749
+ | `GET` | `/api/torrents/:infoHash/file`, `/magnet` | Download the `.torrent`, or its magnet URI |
750
+ | `GET` | `/api/torrents/:infoHash/peers`, `/trackers`, `/content` | Per-peer, per-tracker and per-file detail |
751
+ | `GET` | `/api/torrents/:infoHash/pieces` | Which pieces are held, how rare each is, and what peers hold |
752
+ | `PATCH` | `/api/torrents/:infoHash/mode` | Switch between mirror and cache |
753
+ | `PATCH` | `/api/torrents/:infoHash/categories` | Set, add or remove categories |
754
+ | `PATCH` | `/api/torrents/:infoHash/seeding` | Per-archive seeding limit, or "use the global one" |
755
+ | `PATCH` `GET` | `/api/torrents/:infoHash/location` | Move the data; poll the move |
756
+ | `POST` | `/api/torrents/:infoHash/recheck` | Hash what is on disk again and believe the result — for an archive whose progress and whose files disagree |
757
+ | `POST` | `/api/torrents/:infoHash/pause`, `/resume` | Stop offering it, without forgetting it |
758
+ | `POST` | `/api/torrents/:infoHash/webseeds` | Add web seeds — does not change the infohash |
759
+ | `DELETE` | `/api/torrents/:infoHash/webseeds` | Drop web seeds — likewise |
760
+ | `POST` | `/api/torrents/:infoHash/publish` | What this node offers of the archive itself: `serveArchive`, `selfWebSeed`, `publicDownload` |
761
+ | `POST` | `/api/torrents/:infoHash/warm` | Pre-fetch a region (`GET` for progress, `DELETE` to cancel) |
762
+ | `DELETE` | `/api/torrents/:infoHash/cache` | Reclaim cached pieces, keep the archive |
763
+ | `POST` | `/api/torrents/:infoHash/check` | Has the source changed since the torrent was made? |
764
+ | `POST` | `/api/torrents/:infoHash/rebuild` | Rebuild from the current source (mints a new infohash) |
765
+ | `POST` | `/api/check-origins` | Check every archive with a watchable source |
766
+ | `GET` `DELETE` | `/api/adds` | Adds still in flight — downloads, and local files being hashed — and cancelling a download by URL. A hash cannot be stopped part-way, so it is listed but not cancellable |
767
+ | `GET` `POST` | `/api/speed` | Which speed limits are in force, and the manual switch between the two sets |
768
+ | `GET` | `/api/categories` | Every category, with the endpoints resolving to its newest build |
769
+ | `POST` | `/api/adopt`, `/api/adopt/candidates` | Take over what an engine or another node holds |
770
+ | `POST` | `/api/sources/preview` | What a watched web location would take, without taking it |
771
+ | `POST` | `/api/subscriptions/preview` | Whether a peer is reachable and what it offers |
772
+ | `POST` | `/api/subscriptions/refresh` | Poll subscribed feeds now |
773
+ | `POST` | `/api/sources/check` | Check scheduled sources now, rather than at the next due time |
774
+ | `POST` | `/api/torrents/:infoHash/hooks/complete` | Run the completion hook again for one archive |
775
+ | `GET` `POST` `DELETE` | `/api/tokens`, `/api/tokens/:id` | Mint, list and revoke access tokens |
776
+ | `GET` `POST` | `/api/restart` | What a restart would do, and doing it |
777
+ | `GET` `DELETE` | `/api/stats` | What this node has served — per-archive counters and the last N requests; `DELETE` clears them |
778
+ | `GET` | `/api/traffic` | Upload and download speed per archive over time, from `stats.db`; `infoHash`, `hours` and `buckets` narrow it, and `totals` ranks archives by bytes moved |
779
+ | `GET` `PATCH` | `/api/config` | Read and change settings |
780
+ | `POST` | `/api/login`, `/api/logout` | Console sign-in |
781
+ | `GET` | `/api/session` | Who this request is, and whether a credential is needed at all |
782
+ | `GET` | `/api/catalog` | The whole catalogue, for a peer keeping itself in step |
783
+ | `GET` | `/archives/:infoHash/tiles.json` | TileJSON — **public** |
784
+ | `GET` | `/archives/:infoHash/:z/:x/:y.:ext` | One tile — **public** |
785
+ | `GET` | `/health` | 200 when this node can serve, 503 when its engine cannot — **public**, no credential, for a load balancer |
786
+ | `GET` | `/archives/:infoHash/ready` | Whether this node can serve _this_ archive yet: 200 ready, 503 not yet, 415 never — **public** |
787
+ | `GET` | `/archives/:infoHash/archive.torrent` | The `.torrent` a torrent-aware client joins with — **public** |
788
+ | `GET` | `/archives/:infoHash/archive.pmtiles` | The archive itself, by byte range — what any PMTiles reader wants, and a valid web seed. Only where [`serveArchive`](docs/configuration.md#servearchive) is on — **public**. Complete archives, unless [`serveArchiveFromSwarm`](docs/configuration.md#servearchivefromswarm) is set, which answers a bounded range from the swarm instead |
789
+ | `GET` | `/archives/:infoHash/preview` | Map preview for one archive — admin, not public |
790
+ | `GET` | `/latest/` | Every category and the endpoints that resolve to its newest build — **public**, the index of everything below |
791
+ | `GET` | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is |
792
+ | `GET` | `/latest/:category/archive.pmtiles` | The newest build in a category as a file, by byte range — **public**, and gated by [`serveArchive`](docs/configuration.md#servearchive). What to point a PMTiles reader at when you want "whichever is current" rather than one build |
793
+ | `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public** |
794
794
 
795
795
  Everything under `/api/` is guarded once a credential is configured; tiles, TileJSON and the feeds
796
796
  never are. A `peer` token may read but not change, and may be narrowed to some categories. See
package/docs/configuration.md CHANGED
@@ -315,12 +315,13 @@ rotation stays out across the restart you were probably about to do.
315
315
 
316
316
  ## Offering the archive file itself
317
317
 
318
- | setting | default | |
319
- | ---------------- | ------- | ------------------------------------------------------- |
320
- | `serveArchive` | `false` | answer `/archives/<infohash>/archive.pmtiles` |
321
- | `publishingUrl` | unset | the address to use where a URL must outlive the request |
322
- | `selfWebSeed` | `false` | publish this node as a web seed for archives it holds |
323
- | `publicDownload` | `false` | offer them as downloads on the public catalogue page |
318
+ | setting | default | |
319
+ | ----------------------- | ------- | ------------------------------------------------------------------------ |
320
+ | `serveArchive` | `false` | answer `/archives/<infohash>/archive.pmtiles` |
321
+ | `publishingUrl` | unset | the address to use where a URL must outlive the request |
322
+ | `serveArchiveFromSwarm` | `false` | answer a range for an archive this node does not hold — **experimental** |
323
+ | `selfWebSeed` | `false` | publish this node as a web seed for archives it holds |
324
+ | `publicDownload` | `false` | offer them as downloads on the public catalogue page |
324
325
 
325
326
  Three switches rather than one, because they are three different exposures and a
326
327
  node can reasonably want any of them without the others.
@@ -346,6 +347,38 @@ GET /latest/<category>/archive.pmtiles whichever is current, with an ETag
346
347
 
347
348
  With it off, both answer `403`.
348
349
 
350
+ ### `serveArchiveFromSwarm`
351
+
352
+ **Experimental, and not recommended for anything public.** Off by default.
353
+
354
+ With it on, a byte range for an archive this node does _not_ hold is answered by
355
+ pulling the covering pieces out of the swarm on demand — the same path the tile
356
+ endpoint has always taken internally, one HTTP layer further out, sharing its
357
+ piece cache and its open handle. It is the loop cache mode was built for: point
358
+ an ordinary PMTiles reader at a node holding none of the file, and it works.
359
+
360
+ The reservations are properties of the arrangement rather than of the code:
361
+
362
+ - **Every byte is somebody else's upload.** A cache-mode node is not an origin,
363
+ and putting one behind a URL that looks like one turns each request into swarm
364
+ traffic it neither paid for nor holds.
365
+ - **A piece read takes as long as the swarm takes.** Acceptable for a tile, which
366
+ a reader asked for and will wait on; poor for an HTTP client with its own
367
+ timeout. `swarmRangeTimeoutMs` (30s) bounds it and answers `504`.
368
+ - **There is no honest answer to a request for the whole file.** A `Range` header
369
+ is required — without one the request is refused with `411` — and a range
370
+ larger than `swarmRangeLimitBytes` (8 MiB) with `416`.
371
+
372
+ The response carries the same `ETag` and the same year-long `immutable` caching a
373
+ complete copy would give, because it is the same content: the infohash names
374
+ those bytes wherever they were read from.
375
+
376
+ **A node cannot be a web seed for an archive it does not hold**, whatever this is
377
+ set to. [`selfWebSeed`](#selfwebseed) is refused on an incomplete archive and
378
+ offered again when the download finishes — answering from the swarm and then
379
+ advertising that to the swarm is a loop with an amplifier in it, where every peer
380
+ that takes the seed makes this node fetch the piece again to serve it.
381
+
349
382
  ### `publishingUrl`
350
383
 
351
384
  The address to use for URLs that outlive the request that made them.
@@ -433,12 +466,23 @@ web seed, because a client spends its retries on it, and a download link that
433
466
 
434
467
  ### Per archive, per folder, per source
435
468
 
436
- The same three-level rule as [`md5`](#md5). The node's setting is the default; a
437
- [watched folder](#watched-folders) or a [scheduled source](#scheduled-sources)
438
- may carry its own; and any individual archive can be switched in the console,
439
- under **HTTP sources** in its details. An archive that says nothing goes on
440
- following the node, so changing the node's answer reaches every archive that
441
- never had one of its own.
469
+ The node's setting is the default. A [watched folder](#watched-folders), a
470
+ [scheduled source](#scheduled-sources), an [RSS feed](#subscriptions) and a
471
+ remote node may each carry their own — the **Serve file**, **Web seed** and
472
+ **Listed** columns on those tables, where `node` means "no opinion" rather than
473
+ "off". And any individual archive can be switched in the console, under **HTTP
474
+ sources** in its details.
475
+
476
+ An archive that says nothing goes on following the node, so changing the node's
477
+ answer reaches every archive that never had one of its own.
478
+
479
+ Unlike `md5`, these do apply to a subscription. `md5` is a hashing pass that only
480
+ happens where a torrent is built, and a subscription adopts one somebody else
481
+ built; this is about what happens to the archive afterwards, which is this node's
482
+ business whoever made it. **`selfWebSeed` waits for the download to finish**: a
483
+ web seed URL for an archive still arriving answers `409`, and a peer handed a URL
484
+ that refuses spends its retries on it — worse than no web seed, and unfixable
485
+ afterwards, because by then the URL is in every copy of the `.torrent`.
442
486
 
443
487
  ## Trackers
444
488
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pmtiles-swarm",
3
- "version": "0.52.0",
3
+ "version": "0.54.0",
4
4
  "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
5
5
  "type": "module",
6
6
  "main": "src/index.js",
package/src/api.js CHANGED
@@ -6,6 +6,7 @@ import os from 'node:os';
6
6
  import path from 'node:path';
7
7
  import { fileURLToPath } from 'node:url';
8
8
  import express from 'express';
9
+ import parseRange from 'range-parser';
9
10
  import {
10
11
  ROLES,
11
12
  createAuth,
@@ -2332,6 +2333,121 @@ export function createApp({
2332
2333
  'ETag, Content-Range, Content-Length, Accept-Ranges',
2333
2334
  };
2334
2335
 
2336
+ /**
2337
+ * Serves a byte range for an archive this node does not hold whole.
2338
+ *
2339
+ * **Experimental.** It closes the loop the cache mode was built for: the
2340
+ * node holds no bytes, a reader asks for some, and the pieces covering them
2341
+ * are pulled out of the swarm on demand — the same path the tile endpoint
2342
+ * has always taken internally, one HTTP layer further out and sharing the
2343
+ * same piece cache and open handle.
2344
+ *
2345
+ * It is off by default and should stay off for anything public, for reasons
2346
+ * that are properties of the arrangement rather than of this code:
2347
+ *
2348
+ * - Every byte is somebody else's upload. A node in cache mode is not an
2349
+ * origin; putting one behind a URL that looks like an origin turns each
2350
+ * request into swarm traffic that this node neither paid for nor holds.
2351
+ * - A piece read takes as long as the swarm takes. That is fine for a tile,
2352
+ * which the reader asked for and will wait on, and poor for an HTTP client
2353
+ * with its own patience.
2354
+ * - There is no way to answer a request for the whole file that is not
2355
+ * "download 700 GiB through BitTorrent and stream it out". So a range is
2356
+ * required, and a large one is refused.
2357
+ * @param {import('express').Request} req - The request, for its Range.
2358
+ * @param {import('express').Response} res - The response.
2359
+ * @param {object} entry - The archive.
2360
+ * @returns {Promise<void>} - Once answered.
2361
+ */
2362
+ const serveFromSwarm = async (req, res, entry) => {
2363
+ if (!config.serveArchiveFromSwarm || !tiles?.readRange) {
2364
+ return res.status(409).json({
2365
+ error:
2366
+ 'this archive is not complete here, so a byte range would answer ' +
2367
+ 'with unwritten space rather than data',
2368
+ });
2369
+ }
2370
+
2371
+ const size = Number(entry.size);
2372
+ if (!Number.isFinite(size) || size <= 0) {
2373
+ return res.status(409).json({
2374
+ error: 'this archive has no known length here yet',
2375
+ });
2376
+ }
2377
+
2378
+ // A range, and only a range. Without one the answer is the whole archive,
2379
+ // and the whole archive is not on this disk — serving it would mean
2380
+ // pulling every piece through the swarm to stream it back out, which is
2381
+ // both enormous and somebody else's bandwidth.
2382
+ const asked = req.headers.range;
2383
+ if (!asked) {
2384
+ res.setHeader('accept-ranges', 'bytes');
2385
+ return res.status(411).json({
2386
+ error:
2387
+ 'this node does not hold this archive, so it can only answer a ' +
2388
+ 'byte range. Send a Range header.',
2389
+ });
2390
+ }
2391
+
2392
+ const ranges = parseRange(size, asked, { combine: true });
2393
+ // -1 is unsatisfiable, -2 is malformed, and more than one range would mean
2394
+ // a multipart response that nothing reading PMTiles has ever asked for.
2395
+ if (ranges === -1) {
2396
+ res.setHeader('content-range', `bytes */${size}`);
2397
+ return res.status(416).json({ error: 'range not satisfiable' });
2398
+ }
2399
+ if (ranges === -2 || ranges.length !== 1 || ranges.type !== 'bytes') {
2400
+ return res.status(400).json({ error: 'unreadable Range header' });
2401
+ }
2402
+
2403
+ const { start, end } = ranges[0];
2404
+ const length = end - start + 1;
2405
+ const limit = config.swarmRangeLimitBytes ?? 8 * 1024 * 1024;
2406
+ if (length > limit) {
2407
+ return res.status(416).json({
2408
+ error:
2409
+ `this node does not hold this archive, so a range is fetched from ` +
2410
+ `the swarm a piece at a time; ${length} bytes is more than the ` +
2411
+ `${limit} it will do that for at once`,
2412
+ });
2413
+ }
2414
+
2415
+ // A deadline of its own, shorter than the piece timeout underneath. An
2416
+ // HTTP client gives up long before libtorrent does, and a request left
2417
+ // hanging on a piece nobody is seeding holds a connection open for nothing.
2418
+ const controller = new AbortController();
2419
+ const deadline = setTimeout(
2420
+ () => controller.abort(),
2421
+ config.swarmRangeTimeoutMs ?? 30000,
2422
+ );
2423
+ try {
2424
+ const body = await tiles.readRange(entry.infoHash, start, length, {
2425
+ signal: controller.signal,
2426
+ });
2427
+ res.status(206);
2428
+ res.setHeader('accept-ranges', 'bytes');
2429
+ res.setHeader('content-range', `bytes ${start}-${end}/${size}`);
2430
+ res.setHeader('content-type', 'application/octet-stream');
2431
+ for (const [name, value] of Object.entries(RANGE_CORS_HEADERS)) {
2432
+ res.setHeader(name, value);
2433
+ }
2434
+ // The same tag a complete copy would answer with, because it is the same
2435
+ // content: the infohash names these bytes wherever they were read from.
2436
+ res.setHeader('etag', `"${entry.infoHash}"`);
2437
+ res.setHeader('cache-control', 'public, max-age=31536000, immutable');
2438
+ res.send(body);
2439
+ } catch (error) {
2440
+ if (controller.signal.aborted) {
2441
+ return res.status(504).json({
2442
+ error: 'the swarm did not answer for those pieces in time',
2443
+ });
2444
+ }
2445
+ res.status(error.status ?? 502).json({ error: error.message });
2446
+ } finally {
2447
+ clearTimeout(deadline);
2448
+ }
2449
+ };
2450
+
2335
2451
  /**
2336
2452
  * The current build of a category, as a file, by byte range.
2337
2453
  *
@@ -2367,13 +2483,11 @@ export function createApp({
2367
2483
  });
2368
2484
  }
2369
2485
 
2370
- if (entry.complete === false) {
2371
- return res.status(409).json({
2372
- error:
2373
- 'the newest archive in this category is not complete here, so a ' +
2374
- 'byte range would answer with unwritten space rather than data',
2375
- });
2376
- }
2486
+ // Not on this disk. Either the node can read it out of the swarm, which is
2487
+ // what cache mode is for, or it says so — but it never answers a range
2488
+ // from a sparse file, where the unwritten space reads as zeroes and looks
2489
+ // exactly like data.
2490
+ if (entry.complete === false) return serveFromSwarm(req, res, entry);
2377
2491
  const file = entry.savePath ? path.join(entry.savePath, entry.name) : null;
2378
2492
  if (!file) return res.status(404).json({ error: 'no file for it here' });
2379
2493
 
@@ -2712,13 +2826,9 @@ export function createApp({
2712
2826
  });
2713
2827
  }
2714
2828
 
2715
- if (entry.complete === false) {
2716
- return res.status(409).json({
2717
- error:
2718
- 'this archive is not complete here, so a byte range would answer ' +
2719
- 'with unwritten space rather than data',
2720
- });
2721
- }
2829
+ // See the /latest/ route above: read from the swarm where the node is
2830
+ // configured to, and refuse rather than answer with unwritten space.
2831
+ if (entry.complete === false) return serveFromSwarm(req, res, entry);
2722
2832
  const file = entry.savePath ? path.join(entry.savePath, entry.name) : null;
2723
2833
  if (!file) return res.status(404).json({ error: 'no file for it here' });
2724
2834
 
package/src/config.js CHANGED
@@ -204,6 +204,45 @@ const DEFAULTS = {
204
204
  * a link that answers 403 is worse than no link.
205
205
  */
206
206
  publicDownload: false,
207
+ /**
208
+ * Answer a byte range for an archive this node does not hold, by pulling the
209
+ * covering pieces out of the swarm on demand.
210
+ *
211
+ * **Experimental, and not recommended for anything public.** It closes the
212
+ * loop cache mode was built for — the node holds no bytes, a reader asks for
213
+ * some, and the pieces arrive from peers — over the same path the tile
214
+ * endpoint has always used internally, sharing its piece cache and its open
215
+ * handle. As a way to point an ordinary PMTiles reader at an archive nothing
216
+ * here has a copy of, it works.
217
+ *
218
+ * The reservations are properties of the arrangement rather than of the
219
+ * implementation:
220
+ *
221
+ * - Every byte is somebody else's upload. A cache-mode node is not an origin,
222
+ * and putting one behind a URL that looks like one turns each request into
223
+ * swarm traffic it neither paid for nor holds.
224
+ * - A piece read takes as long as the swarm takes. Acceptable for a tile,
225
+ * which a reader asked for and will wait on; poor for an HTTP client with
226
+ * its own patience and its own timeout.
227
+ * - There is no honest answer to a request for the whole file, so a Range
228
+ * header is required and a large one is refused.
229
+ */
230
+ serveArchiveFromSwarm: false,
231
+ /**
232
+ * The largest range `serveArchiveFromSwarm` will fetch at once, in bytes.
233
+ *
234
+ * Sized for what a PMTiles reader actually asks for — a 16 KiB header, a
235
+ * directory, a tile — rather than for bulk transfer, which is the thing this
236
+ * must not quietly become.
237
+ */
238
+ swarmRangeLimitBytes: 8 * 1024 * 1024,
239
+ /**
240
+ * How long to wait for the swarm before giving up on a range, in
241
+ * milliseconds. Shorter than the piece timeout underneath on purpose: an
242
+ * HTTP client gives up long before libtorrent does, and a request left
243
+ * hanging on a piece nobody is seeding holds a connection open for nothing.
244
+ */
245
+ swarmRangeTimeoutMs: 30000,
207
246
  /**
208
247
  * How long an unfinished download is kept before startup treats it as
209
248
  * abandoned. Until then, re-adding the same URL resumes it.
package/src/library.js CHANGED
@@ -526,6 +526,39 @@ export class Library {
526
526
  * @returns {Promise<object>} - The updated catalog entry.
527
527
  */
528
528
  async finalize(infoHash) {
529
+ const settled = await this.#finalizeOnce(infoHash);
530
+
531
+ // Only now, and never at import. A web seed URL for an archive that is
532
+ // still arriving answers 409, and a peer handed a URL that refuses spends
533
+ // its retries on it — worse than no web seed at all, and unfixable
534
+ // afterwards, because the URL is in the .torrent every peer holds. So a
535
+ // subscription records the intention when it joins and it is acted on
536
+ // here, at the first moment this node actually holds the whole file.
537
+ //
538
+ // Safe to reach on an archive that was already complete: setPublishing
539
+ // works from what is on record rather than from a transition, so this does
540
+ // nothing the second time — and does the right thing the first time for an
541
+ // archive that finished before the setting existed.
542
+ if (settled && publishingFor(settled, this.#config).selfWebSeed) {
543
+ try {
544
+ await this.setPublishing(infoHash, {});
545
+ return this.#catalog.get(infoHash) ?? settled;
546
+ } catch (error) {
547
+ console.warn(
548
+ `[web seed] ${settled.name} is not published as a web seed by this ` +
549
+ `node: ${error.message}`,
550
+ );
551
+ }
552
+ }
553
+ return settled;
554
+ }
555
+
556
+ /**
557
+ * The rename and the bookkeeping, without the publishing that follows it.
558
+ * @param {string} infoHash - The archive that finished.
559
+ * @returns {Promise<object>} - The updated catalog entry.
560
+ */
561
+ async #finalizeOnce(infoHash) {
529
562
  const entry = this.#catalog.get(infoHash);
530
563
  if (!entry) throw new Error('unknown archive');
531
564
  if (entry.complete) return entry;
@@ -1145,6 +1178,11 @@ export class Library {
1145
1178
  complete,
1146
1179
  // Held until the download finishes, which may be hours away.
1147
1180
  originMtime: options.originMtime,
1181
+ // Recorded now, applied when the download finishes. Unset leaves the
1182
+ // archive following the node, which is the rule everywhere else.
1183
+ serveArchive: options.serveArchive,
1184
+ selfWebSeed: options.selfWebSeed,
1185
+ publicDownload: options.publicDownload,
1148
1186
  // What the peer that offered this says it holds, where it said anything.
1149
1187
  // The head warmer replaces it with what the archive's own header says as
1150
1188
  // soon as it can read one; until then this is what makes the archive
@@ -2871,6 +2909,19 @@ export class Library {
2871
2909
  let webSeed = published;
2872
2910
  let warning = null;
2873
2911
 
2912
+ // A node cannot be a web seed for bytes it does not have. It might be able
2913
+ // to *answer* for them, where serveArchiveFromSwarm is on -- but answering
2914
+ // by fetching from the swarm and then advertising that to the swarm is a
2915
+ // loop with an amplifier in it: every peer that takes the seed makes this
2916
+ // node download the piece again to serve it.
2917
+ if (after.selfWebSeed && !published && entry.complete === false) {
2918
+ throw new Error(
2919
+ 'this node does not hold a complete copy of this archive, so it ' +
2920
+ 'cannot be a web seed for it. It will be offered again once the ' +
2921
+ 'download finishes.',
2922
+ );
2923
+ }
2924
+
2874
2925
  if (after.selfWebSeed && !published) {
2875
2926
  const base = publishingBase({
2876
2927
  // Given outright beats the node's own answer, which beats the request
package/src/subscriptions.js CHANGED
@@ -423,6 +423,14 @@ export class SubscriptionManager {
423
423
  // 698 GiB download before starting one, which means it is populated
424
424
  // before a single byte exists here.
425
425
  summarySource: 'feed',
426
+ // What this node will offer of the archive once it holds it. Unset here
427
+ // means the node's own answer applies, the same rule as everywhere else.
428
+ // Nothing takes effect until the download finishes: a web seed URL for
429
+ // an archive that is still arriving answers 409, which is worse than no
430
+ // web seed because peers spend their retries on it.
431
+ serveArchive: subscription.serveArchive,
432
+ selfWebSeed: subscription.selfWebSeed,
433
+ publicDownload: subscription.publicDownload,
426
434
  };
427
435
 
428
436
  // The .torrent is preferred where there is one: it carries the trackers
package/src/tiles.js CHANGED
@@ -119,6 +119,43 @@ export class TileStore {
119
119
  return { data: gzipped, encoding: 'gzip' };
120
120
  }
121
121
 
122
+ /**
123
+ * Reads a byte range out of an archive, through whatever source applies.
124
+ *
125
+ * The same acquisition every tile read goes through, so a cache-mode archive
126
+ * answers from the swarm and a complete one answers from its file — and both
127
+ * share the piece cache, the directory prefetch and the open handle that the
128
+ * tile path has already warmed. That sharing is the point: a reader fetching
129
+ * the header, then the root directory, then a tile is doing exactly what the
130
+ * tile endpoint does internally, one HTTP layer further out.
131
+ *
132
+ * The range is not split here. `TorrentSource.getBytes` already fetches every
133
+ * covering piece concurrently, which is what stops a three-piece range paying
134
+ * three sequential swarm round-trips.
135
+ * @param {string} infoHash - Which archive.
136
+ * @param {number} offset - Byte offset into the archive file.
137
+ * @param {number} length - How many bytes.
138
+ * @param {object} [options] - Abort signal.
139
+ * @returns {Promise<Buffer>} - The bytes, clamped to the end of the file.
140
+ */
141
+ async readRange(infoHash, offset, length, options = {}) {
142
+ const entry = this.#catalog.get(infoHash);
143
+ if (!entry) throw new TileReadError('unknown archive', 404);
144
+
145
+ const handle = await this.#acquire(entry);
146
+ if (!handle.source) {
147
+ // MBTiles opens as a database rather than a byte source, and a byte
148
+ // range into one would be a range into a SQLite file — technically
149
+ // answerable and useless to everybody.
150
+ throw new TileReadError(
151
+ 'this archive is not readable as a stream of bytes',
152
+ 415,
153
+ );
154
+ }
155
+
156
+ const answer = await handle.source.getBytes(offset, length, options.signal);
157
+ return Buffer.from(answer.data);
158
+ }
122
159
  /**
123
160
  * Reads an archive's header and metadata, through whatever source applies.
124
161
  *
package/src/web/index.html CHANGED
@@ -4654,6 +4654,34 @@ Every piece is hashed against the ` +
4654
4654
  await renderTokenEditor(body);
4655
4655
  renderHookEditor(body, config, restartKeys);
4656
4656
 
4657
+ // The three switches the details panel offers, as import defaults.
4658
+ // Each has a blank option as well as yes and no, because unset is a
4659
+ // real answer here: it means the archive follows the node, and a row
4660
+ // that has never said anything must keep not saying it. See
4661
+ // publishingFor() -- the node's setting reaches everything that never
4662
+ // had one of its own, and that only works if "no opinion" is storable.
4663
+ const publishing = ['', 'node'];
4664
+ const publishingColumns = [
4665
+ {
4666
+ field: 'serveArchive',
4667
+ label: 'Serve file',
4668
+ boolean: true,
4669
+ options: [publishing, ['true', 'yes'], ['false', 'no']],
4670
+ },
4671
+ {
4672
+ field: 'selfWebSeed',
4673
+ label: 'Web seed',
4674
+ boolean: true,
4675
+ options: [publishing, ['true', 'yes'], ['false', 'no']],
4676
+ },
4677
+ {
4678
+ field: 'publicDownload',
4679
+ label: 'Listed',
4680
+ boolean: true,
4681
+ options: [publishing, ['true', 'yes'], ['false', 'no']],
4682
+ },
4683
+ ];
4684
+
4657
4685
  renderRowEditor({
4658
4686
  into: body,
4659
4687
  key: 'locations',
@@ -4747,6 +4775,7 @@ Every piece is hashed against the ` +
4747
4775
  placeholder: 'for ever',
4748
4776
  number: true,
4749
4777
  },
4778
+ ...publishingColumns,
4750
4779
  ],
4751
4780
  rows: config.watch ?? [],
4752
4781
  footnote:
@@ -4855,6 +4884,7 @@ Every piece is hashed against the ` +
4855
4884
  placeholder: 'for ever',
4856
4885
  number: true,
4857
4886
  },
4887
+ ...publishingColumns,
4858
4888
  ],
4859
4889
  rows: config.sources ?? [],
4860
4890
  checkNow: '/api/sources/check',
@@ -4979,6 +5009,7 @@ Every piece is hashed against the ` +
4979
5009
  ['false', 'no'],
4980
5010
  ],
4981
5011
  },
5012
+ ...publishingColumns,
4982
5013
  ],
4983
5014
  rows: following.filter((row) => !isPeer(row)),
4984
5015
  checkNow: '/api/subscriptions/refresh',
@@ -5053,6 +5084,7 @@ Every piece is hashed against the ` +
5053
5084
  ['false', 'no'],
5054
5085
  ],
5055
5086
  },
5087
+ ...publishingColumns,
5056
5088
  ],
5057
5089
  rows: following.filter(isPeer),
5058
5090
  checkNow: '/api/subscriptions/refresh',