pmtiles-swarm 0.33.0 → 0.35.0

package/CHANGELOG.md

@@ -7,6 +7,89 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.35.0
+### ✨ Features and improvements
+- **Requires pmtiles-torrent 0.7.0, which stops the libtorrent sidecar going deaf while it works.**
+  Its request loop ran each call to completion before reading the next, so a long one starved
+  everything behind it. Two of them are long. `create` hashes a whole archive, which for a 698 GiB
+  local add is hours — reported here as `libtorrent list timed out after 60000ms` every minute, a
+  console header stuck at "connecting…", and archive details that never loaded. `read_piece` waits
+  up to 60s for a piece to arrive from the swarm, and every tile served from a cache-mode archive
+  goes through one, so a serving node spent most of its life unable to answer anything else.
+
+  Both now run off that loop. The second needed alert delivery reworked to a single pump with
+  subscribers first, because every consumer used to drain the session's one alert queue — so two
+  concurrent reads would have swallowed each other's `read_piece_alert` and both timed out. A read
+  also matches its alert on the torrent now, not just the piece number, and a storage fault is
+  reported once by the pump rather than only when a read happened to be waiting to notice it.
+
+  **Upgrade both together.** This release does not itself require the new behaviour, but it is the
+  version that asks for it, and a node running a 0.6.x sidecar keeps the stalls.
+
+### 🐞 Bug fixes
+
+## 0.34.0
+### ✨ Features and improvements
+- **`md5` is a declared setting.** It was already honoured wherever a torrent is created, but it
+  appeared in no defaults list and no document, so it could only be written into the config file by
+  hand — `PATCH /api/config` refused it as an unknown setting, and nothing in the console showed
+  whether it was on. It now defaults to `false`, is documented, and can be changed without a
+  restart.
+- **`incomingRetentionDays`** sets how long an unfinished download stays resumable. Defaults to 14.
+
+### 🐞 Bug fixes
+- **A large download survives a restart instead of starting again from zero.** A scheduled web
+  source fetching a multi-hour archive lost the whole transfer every time the node restarted, and
+  began again from nothing on the next poll. Three things had to hold and only one did. The bytes
+  were always kept — the staging directory is named for a hash of its URL so the next add finds
+  it — but **shutdown deleted them**, because it stopped in-flight adds through `cancelAdd()`, and
+  cancelling discards the partial on purpose: somebody said stop. A restart is not that decision,
+  so shutdown now uses `stopAdds()`, which the fetch can tell apart. **Startup then swept whatever
+  survived**, on the reasoning that a killed process leaves a partial "nothing will ever look in
+  again" — true when staging names were random, false since they became a hash of the URL. And
+  **the validator did not outlive the process**: the `ETag` a resume is checked against lived in a
+  local, so a new process had nothing to compare and refused the resume as "the server offers no
+  ETag or Last-Modified", deleting the partial by the very attempt meant to continue it. It is now
+  written beside the bytes and removed when the download completes. A restart during a 700 GiB
+  transfer now costs the seconds since the last write.
+- **`.incoming` is swept by age rather than emptied.** Only a staging directory nothing has
+  written to for `incomingRetentionDays` (default 14) is cleared, so an unfinished download stays
+  resumable. The sweep also looks under `cacheSavePath`, which it never did — staging lands there
+  for cache-mode adds and under a source's own `savePath`, so the one configured `savePath` was
+  never the whole of where it could be.
+- **Adding a local archive answers when the file has been checked, not when it has been hashed.**
+  `POST /api/torrents` with a `path` held the response open for the whole hash — every byte of the
+  archive, twice with `md5` on — so the console's add dialog sat there for minutes with no sign
+  that anything was happening. Worse than the URL case it mirrors, because nothing was downloading
+  either: the file was already on the disk and visibly not moving, which reads as a submit button
+  that did nothing. It now answers `202` once the path has been identified and accepted, and the
+  hash reports itself through `/api/adds` like a download does. A path that is not there or is not
+  an archive still fails in the response. An archive already held answers `200` with its entry,
+  which the URL branch now does too rather than promising work that was already done. **Scripts
+  reading the created entry straight back from a `path` add need `/api/adds` or `/api/torrents`
+  instead** — magnets and `.torrent` URLs are unchanged and still answer `201`.
+- **A second add of a file already being hashed joins the first rather than starting its own.**
+  Only reachable now that the dialog closes quickly enough to submit twice, and two passes over the
+  same planet archive is an hour of disk for one result.
+- **The console's MD5 checkbox is now the decision it looks like.** It was only sent when ticked,
+  and the server reads an absent `md5` as "unspecified" and falls back to the node's configured
+  default — so on a node with `md5` on, an unticked box still hashed one, and the log said so while
+  the dialog appeared to have turned it off. The value is sent either way, and the box now starts
+  from the node's own setting rather than always unticked — otherwise making it authoritative would
+  have turned a configured default off for every add made from the console, the same disagreement
+  the other way round. Omitting `md5` from an API or CLI call still inherits the config default,
+  which is what that fallback is for.
+- **The save-location picker is hidden when adding a local file.** It did nothing there: a local
+  add registers the file's own directory as the save path whatever was chosen, which is exactly
+  what "hashed in place, nothing is copied" says — but the picker sat next to that sentence
+  implying otherwise, and offered no way to say "leave it where it is" because that is the only
+  thing it does.
+- **"What a torrent-aware client does" describes what they now do.** The section predated the swarm
+  handles moving into the TileJSON URL's fragment and still had a client learning where to join
+  from a TileJSON response — the one thing the fragment exists to avoid, since the swarm is the
+  part that depends on no server. It contradicted "bootstrapping without the server" two sections
+  below it.
+
 ## 0.33.0
 ### ✨ Features and improvements
 - **Requires pmtiles-torrent 0.6.1, which is what finally makes a downloading archive servable.**
@@ -21,7 +104,6 @@
 
 ## 0.32.3
 ### ✨ Features and improvements
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
 - **Requires pmtiles-torrent 0.5.2, which deletes a torrent's resume file with its data.** Resume
@@ -33,11 +115,9 @@
   the data goes too: a removal that keeps the files is how a pause is expressed for an engine with
   no pause of its own, and discarding resume data there would turn every pause into a full re-hash.
 
-- _...Add new stuff here..._
 
 ## 0.32.2
 ### ✨ Features and improvements
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
 - **The head warmer no longer skips every archive it was built for.** It decided an archive was done
@@ -56,11 +136,9 @@
   header off local disk for nothing, while assuming the opposite would leave every existing
   subscription stuck exactly as it is.
 
-- _...Add new stuff here..._
 
 ## 0.32.1
 ### ✨ Features and improvements
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
 - **Requires pmtiles-torrent 0.5.1, so the connection indicator and Recheck files actually work.**
@@ -70,11 +148,9 @@
   cannot answer is deliberately not reported as unreachable, so there was nothing to see and
   nothing to explain why.
 
-- _...Add new stuff here..._
 
 ## 0.32.0
 ### ✨ Features and improvements
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
 - **A mutable magnet no longer carries a web seed.** A `ws=` URL names one build; a BEP 46 magnet
@@ -100,7 +176,6 @@
   **Restyle anything holding one.** A style carrying an older mutable magnet keeps working, but
   carries the stale web seed until it is regenerated.
 
-- _...Add new stuff here..._
 
 ## 0.31.0
 ### ✨ Features and improvements
@@ -127,10 +202,8 @@
   With two engines both are asked, since each keeps its own belief about the same file and a stale
   one on the secondary is why a browser peer would find nothing while the primary seeds happily.
 
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
-- _...Add new stuff here..._
 
 ## 0.30.0
 ### ✨ Features and improvements
@@ -160,10 +233,8 @@
   makes the eye stop to work out which it is looking at. Seconds are dropped rather than the date,
   since nothing here is sorted finely enough for them to matter; hovering still gives them.
 
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
-- _...Add new stuff here..._
 
 ## 0.29.0
 ### ✨ Features and improvements

package/README.md

@@ -761,7 +761,7 @@ which the endpoint answers 501.
 | `POST`                | `/api/torrents/:infoHash/check`                             | Has the source changed since the torrent was made?                                                                                                                                                                                        |
 | `POST`                | `/api/torrents/:infoHash/rebuild`                           | Rebuild from the current source (mints a new infohash)                                                                                                                                                                                    |
 | `POST`                | `/api/check-origins`                                        | Check every archive with a watchable source                                                                                                                                                                                               |
-| `GET` `DELETE`        | `/api/adds`                                                 | Downloads still in flight, and cancelling one by URL                                                                                                                                                                                      |
+| `GET` `DELETE`        | `/api/adds`                                                 | Adds still in flight — downloads, and local files being hashed — and cancelling a download by URL. A hash cannot be stopped part-way, so it is listed but not cancellable                                                                 |
 | `GET` `POST`          | `/api/speed`                                                | Which speed limits are in force, and the manual switch between the two sets                                                                                                                                                               |
 | `GET`                 | `/api/categories`                                           | Every category, with the endpoints resolving to its newest build                                                                                                                                                                          |
 | `POST`                | `/api/adopt`, `/api/adopt/candidates`                       | Take over what an engine or another node holds                                                                                                                                                                                            |

package/docs/architecture-diagram.md

@@ -154,6 +154,15 @@ stops needing the tile endpoint at all.
 The important part is that it does both at once — HTTP for the first paint, swarm
 in the background — so there is never a blank map waiting for metadata.
 
+Discovery comes first, and it comes from the style rather than from a response.
+The source URL carries its handles in a fragment
+(`…/tiles.json#torrent=…&magnet=…`, described under
+[bootstrapping](#bootstrapping-without-the-server)), so a client knows there is a
+swarm behind a source before it makes a single request — which is the point, as
+that is what still works when the server is down. The TileJSON is fetched anyway,
+because it carries the tile endpoint for the first paint and a fuller `torrent`
+block than a fragment can, but it is no longer how the swarm is _found_.
+
 ```mermaid
 sequenceDiagram
     autonumber
@@ -162,24 +171,28 @@ sequenceDiagram
     participant HTTP as pmtiles-swarm<br/>(via CDN)
     participant BT as BitTorrent swarm
 
-    App->>P: load style → tiles.json
-    P->>HTTP: GET /archives/{hash}/tiles.json
-    HTTP-->>P: TileJSON + torrent block
-
-    Note over P: claims the /archives/{hash}/ prefix,<br/>so only these URLs come to it
+    App->>P: load style
+    Note over P: reads the fragment on each source URL:<br/>torrent= and magnet=. No request yet.
 
     par Map is usable immediately
+        P->>HTTP: GET /latest/{category}/tiles.json
+        Note over P,HTTP: the fragment is never sent
+        HTTP-->>P: TileJSON + torrent block
         App->>P: tile 12/2145/1436
         P->>HTTP: GET …/12/2145/1436.pbf
         HTTP-->>App: tile bytes
     and Swarm warms up in the background
-        P->>BT: join (.torrent — metadata already in hand)
+        P->>HTTP: GET the .torrent
+        HTTP-->>P: metainfo (piece hashes, trackers, web seeds)
+        P->>BT: join — metadata already in hand
         BT-->>P: connected
         P->>BT: fetch PMTiles header + root directory
         BT-->>P: those pieces
         Note over P: now able to resolve any tile<br/>to a byte range locally
     end
 
+    Note over P: claims the TileJSON URL's prefix (without<br/>the fragment), so only these URLs come to it
+
     App->>P: tile 12/2146/1436
     Note over P: tile → byte range (PMTiles directory)<br/>→ piece index
     P->>BT: fetch that piece
@@ -201,18 +214,43 @@ is why the `torrent` block carries the archive's `.torrent` rather than per-tile
 URLs: **there is nothing tile-specific in the swarm.** The swarm holds one file,
 and both ends know how to read tiles out of it.
 
-Three consequences worth being clear about:
+Consequences worth being clear about:
 
 - **HTTP is never fully abandoned.** It is the fallback for anything the swarm
   cannot answer quickly, and the only path until the swarm is connected.
 - **The client becomes a seeder.** Every piece it pulls, it serves — so a popular
   region gets _faster_ as more clients view it, which is the opposite of how a
   tile server behaves under load.
-- **Prefer the `.torrent` over the magnet.** A magnet carries only an infohash, so
-  the client must find peers and complete a metadata exchange before it knows
-  anything about the archive — measured at 90 to 240 seconds against a 72 GiB
-  archive. The `.torrent` served alongside the TileJSON already contains the
-  metadata and is ready immediately.
+- **The document is preferred, the fragment is the fallback.** Where the TileJSON
+  is reachable and carries a `torrent` block, that block wins: it is the archive's
+  own account of itself, and it has web seeds, size and piece length that two
+  handles in a URL do not. The fragment is what answers when the document does not
+  — unreachable, not JSON, or carrying no block — which is exactly the case the
+  handles were put in the URL for. A client that used only one of the two would be
+  either poorly informed or dependent on the server it is meant to survive.
+- **Prefer the `.torrent` over the magnet, and for a browser this is not a
+  preference.** A magnet carries only an infohash, so the client must find peers
+  and complete a metadata exchange before it knows anything about the archive —
+  measured at 90 to 240 seconds against a 72 GiB archive. That is the cost for a
+  client with a DHT. A browser has neither DHT nor UDP, and piece hashes reach a
+  BitTorrent client only from a **peer**, over BEP 9 — a web seed serves file
+  payload and never metainfo. So a page that cannot reach a peer cannot use the
+  web seed either, however reachable that web seed is: it has bytes it is not
+  allowed to trust. Fetching the `.torrent` over the same HTTPS the TileJSON came
+  from takes the peer off the critical path entirely, which is the difference
+  between working on a restricted network and not working at all.
+- **A client with a DHT can join on a magnet alone**, and one without cannot. That
+  is the one place the two kinds of client genuinely diverge, and it is why the
+  convention names `torrent=` first: a browser treats a source carrying only a
+  magnet as no candidate at all, while a native client is happy with it.
+- **Holding the metainfo is not evidence that anything will serve it.** It makes
+  the engine ready with no peer involved, which removes the very thing that used
+  to prove an archive was worth binding a source to — waiting for metadata was
+  never only a wait. A client that registers on metainfo alone can bind a source
+  to a swarm with nothing behind it, and that does not fall back, it _stalls_, and
+  the tiles never draw. One `Range: bytes=0-0` against the web seed restores the
+  evidence: a 206 proves the host is reachable, serves ranges and permits the
+  origin, which is all the engine needs from it.
 
 ---
 

package/docs/configuration.md

@@ -215,11 +215,13 @@ seeding.
 
 ## Creating torrents
 
-| setting                | default    |                                              |
-| ---------------------- | ---------- | -------------------------------------------- |
-| `pieceLength`          | `4194304`  | 4 MiB                                        |
-| `torrentFormat`        | `'hybrid'` | `'hybrid'`, `'v1'` or `'v2'`                 |
-| `allowUnknownArchives` | `false`    | publish files not recognised as map archives |
+| setting                 | default    |                                                 |
+| ----------------------- | ---------- | ----------------------------------------------- |
+| `pieceLength`           | `4194304`  | 4 MiB                                           |
+| `torrentFormat`         | `'hybrid'` | `'hybrid'`, `'v1'` or `'v2'`                    |
+| `allowUnknownArchives`  | `false`    | publish files not recognised as map archives    |
+| `md5`                   | `false`    | also compute an MD5 of each archive created     |
+| `incomingRetentionDays` | `14`       | how long an unfinished download stays resumable |
 
 ### `pieceLength`
 
@@ -260,6 +262,31 @@ reading over a swarm does not work the way it does for a flat, Hilbert-ordered
 file; a finished local copy has no such problem. See
 [serving-tiles.md](serving-tiles.md#what-can-be-served).
 
+### `incomingRetentionDays`
+
+An unfinished download is kept in `.incoming` for this long, and adding the same
+URL again resumes it — which is what makes a restart during a multi-hour
+transfer cost minutes rather than the whole download. A scheduled source picks
+its own back up on the next poll without being asked.
+
+Only what nothing has written to for this many days is cleared at startup, since
+whether a URL is still wanted is a question about configuration that the sweep
+cannot see. Set it lower on a small disk, or higher if an upstream can be
+unreachable for weeks.
+
+### `md5`
+
+Off by default, because it costs a second full read of the archive: with it on,
+adding a 700 GiB file takes twice as long and produces one convenience digest
+that nothing in BitTorrent uses. The torrent already verifies the content, and
+per piece rather than as a whole — this is for the manual check somebody wants to
+run against a published checksum, and it is carried in the feed for them.
+
+The console's **Also compute an MD5** box starts from this setting and is sent
+with the add either way, so unticking it applies to that add alone. An API or CLI
+call that omits `md5` inherits this; one that passes `true` or `false` decides
+for itself.
+
 ## Trackers
 
 `trackers` is baked into every torrent this node creates. It defaults to the

package/docs/internals.md

@@ -556,6 +556,35 @@ assumed:
 
 Any of those failing restarts the download rather than guessing.
 
+### Across a restart, not only across a stall
+
+All of that worked within one process and none of it survived leaving one, which
+made a restart during a long download cost the whole download. Three separate
+things had to be true, and only the first was:
+
+1. **The bytes are kept.** They always were — the staging directory is named for
+   `sha256(url)`, so the next add of the same URL finds it.
+2. **Nothing deletes them on the way past.** Two things did. Shutdown expressed
+   itself through `cancelAdd()`, and cancelling deletes the partial on purpose —
+   somebody said stop. A restart is not that decision, so shutdown now calls
+   `stopAdds()`, which aborts with a reason the fetch can tell apart. Startup
+   then swept `.incoming` unconditionally, on the reasoning that a killed
+   process leaves a partial "nothing will ever look in again" — true when
+   staging names were random, false since they became a hash of the URL. It now
+   reaps only what nothing has written to for `incomingRetentionDays`.
+3. **The validator outlives the process.** `stillTheSameFile` compares the
+   `ETag` seen when the download began against the one offered now, and that
+   header lived in a local. A new process had nothing to compare, so the resume
+   was refused for the one reason that cannot be recovered from — "the server
+   offers no ETag or Last-Modified" — and the partial was deleted by the very
+   attempt meant to continue it. It is now written to `<partial>.resume` beside
+   the bytes, carrying the URL it belongs to, and removed when the download
+   finishes so it cannot keep the staging directory from being cleared.
+
+The URL is recorded in the sidecar as well as implied by the directory name, so
+a staging directory that has been reused for something else is refused rather
+than spliced.
+
 ## Retiring and pruning a subscription
 
 Two different questions, which is why an RSS feed can have the first and not the

package/docs/publishing.md

@@ -54,6 +54,31 @@ curl -X POST localhost:8090/api/torrents \
 curl -X POST localhost:8090/api/adopt
 ```
 
+The first two of those answer differently, and a script should know which it is
+reading. A local path and a URL both take as long as it takes to read every byte
+of the archive — minutes for a local file, hours for a planet download, doubled
+again if `md5` is on — so they answer **`202 Accepted`** the moment the source
+has been checked, and the work carries on behind it:
+
+```json
+{
+  "accepted": true,
+  "path": "/mnt/maps/planet.pmtiles",
+  "message": "hashing; progress is reported by /api/adds"
+}
+```
+
+There is no infohash in that, because there is not one yet. `GET /api/adds`
+lists what is still running, and the archive appears in `GET /api/torrents` once
+it finishes. A source that fails its checks — a path that is not there, a URL
+that does not answer, a file that is not an archive — fails in the response
+instead, since that is what somebody can do something about. An archive already
+in the catalog answers `200` with the existing entry.
+
+A magnet, a `.torrent` URL and an uploaded `.torrent` are metadata rather than
+data, so there is nothing slow to wait for: those still answer `201` with the
+entry.
+
 ### When the source URL is not published
 
 Adding from a URL registers that URL as a web seed by default, because it is by

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.33.0",
+  "version": "0.35.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -46,7 +46,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.6.1",
+    "pmtiles-torrent": "^0.7.0",
     "webtorrent": "^3.0.21"
   },
   "engines": {

package/src/api.js

@@ -53,6 +53,55 @@ function route(handler) {
   return (req, res, next) => Promise.resolve(handler(req, res)).catch(next);
 }
 
+/**
+ * Starts an add that takes minutes to hours, and answers as soon as it is safe
+ * to say it has been accepted.
+ *
+ * Everything a caller can do something about is known long before the work is
+ * finished: the source answers, and it is an archive of a kind this will
+ * publish. What remains is transfer and hashing. Awaiting all of it held the
+ * response open for the whole thing, so the console's add dialog stayed on
+ * screen for the duration — over an archive that was visibly appearing behind
+ * it in the URL case, and over a file that had never moved in the local one,
+ * where it read as a submit button that had done nothing at all.
+ *
+ * Progress has its own route already: `runningAdds()` feeds `/api/adds`, the
+ * console polls it, and `DELETE /api/adds` cancels the ones that can be.
+ * @param {object} res - The response to answer.
+ * @param {Function} start - Called with `{onValidated}`; returns the add's promise.
+ * @param {object} accepted - Fields describing the source, for the 202 body.
+ * @param {string} message - What the 202 tells the caller is now happening.
+ * @param {string} what - Prefixed log tag and source, for a failure nobody is waiting on.
+ * @returns {Promise<void>} - Resolves once the response has been sent.
+ */
+async function acceptAdd(res, { start, accepted, message, what }) {
+  const validated = Promise.withResolvers();
+  const running = start({ onValidated: validated.resolve })
+    // A failure after validation has nowhere to be reported: the response has
+    // gone. It is logged where the rest of the work is, and swallowed here so
+    // it cannot take the process down as an unhandled rejection.
+    .catch((error) => {
+      validated.reject(error);
+      console.error(`${what}: ${error.message}`);
+    });
+
+  // Whichever comes first: the checks passing, or the whole attempt failing. A
+  // source that does not answer, or is not an archive, still reports itself in
+  // the dialog where somebody can correct it.
+  const outcome = await validated.promise;
+
+  // Already held, so there is no work to wait on and the entry itself is the
+  // better answer. Deliberately not extended to `joined`, where the promise
+  // belongs to somebody else's transfer and awaiting it would hold the
+  // response open for exactly as long as this exists to avoid.
+  if (outcome?.held) {
+    return void res.status(200).json(await running);
+  }
+
+  void running;
+  res.status(202).json({ accepted: true, ...accepted, message });
+}
+
 /**
  * Builds the HTTP application: JSON API, RSS feeds and the web UI.
  * @param {object} deps - Collaborators.
@@ -1481,41 +1530,24 @@ export function createApp({
 
       let entry;
       if (body.path) {
-        entry = await library.addLocalArchive(body.path, options);
+        // Nothing is copied and nothing is downloaded, but every byte is still
+        // read to compute the piece hashes — twice with md5 on. That is the
+        // whole wait for a local add, and it is no shorter for the file being
+        // on this disk already. See acceptAdd().
+        return await acceptAdd(res, {
+          start: (hooks) =>
+            library.addLocalArchive(body.path, { ...options, ...hooks }),
+          accepted: { path: body.path },
+          message: 'hashing; progress is reported by /api/adds',
+          what: `[hash] ${body.path}`,
+        });
       } else if (body.url) {
-        // Answered as soon as the URL has been checked, not when the transfer
-        // finishes. Awaiting the whole thing held the response open for the
-        // length of the download -- hours for a planet archive -- so the
-        // console's add dialog stayed on screen throughout, over an archive
-        // that was visibly appearing behind it.
-        //
-        // Progress has its own route already: runningAdds() feeds /api/adds,
-        // the console polls it, and DELETE /api/adds cancels one. This is the
-        // piece that was missing rather than a new mechanism.
-        const validated = Promise.withResolvers();
-        const running = library
-          .addRemoteArchive(body.url, {
-            ...options,
-            onValidated: validated.resolve,
-          })
-          // A failure after validation has nowhere to be reported: the
-          // response has gone. It is logged where the rest of the fetch is,
-          // and swallowed here so it cannot take the process down as an
-          // unhandled rejection.
-          .catch((error) => {
-            validated.reject(error);
-            console.error(`[fetch] ${body.url}: ${error.message}`);
-          });
-
-        // Whichever comes first: the checks passing, or the whole attempt
-        // failing. A URL that does not answer, or is not an archive, still
-        // reports itself in the dialog where somebody can correct it.
-        await validated.promise;
-        void running;
-        return res.status(202).json({
-          accepted: true,
-          url: body.url,
+        return await acceptAdd(res, {
+          start: (hooks) =>
+            library.addRemoteArchive(body.url, { ...options, ...hooks }),
+          accepted: { url: body.url },
           message: 'fetching; progress is reported by /api/adds',
+          what: `[fetch] ${body.url}`,
         });
       } else if (body.magnet) {
         entry = await library.addExistingTorrent(

package/src/config.js

@@ -155,6 +155,17 @@ const DEFAULTS = {
    * publish any readable file to a public swarm.
    */
   allowUnknownArchives: false,
+  /**
+   * Also compute an MD5 of each archive created here. Costs a second full read
+   * of the file. Already honoured wherever a torrent is created; declared so it
+   * can be seen and set rather than only written into the file by hand.
+   */
+  md5: false,
+  /**
+   * How long an unfinished download is kept before startup treats it as
+   * abandoned. Until then, re-adding the same URL resumes it.
+   */
+  incomingRetentionDays: 14,
   /**
    * Who may administer this node. Tiles, TileJSON and the feed are always
    * public; everything under /api/ is gated whenever anything here is set. See

package/src/index.js

@@ -237,9 +237,17 @@ PMTILES_SWARM_PUBLIC_URL
   stoppers.unshift({
     label: 'downloads in progress',
     stop: () => {
-      const cancelled = library.cancelAdd();
-      if (cancelled.length > 0) {
-        console.log(`[shutdown] cancelled ${cancelled.length} download(s)`);
+      // stopAdds, not cancelAdd: a restart is not a decision to stop wanting
+      // the archive, and cancelling deletes the partial download. Through
+      // cancelAdd every restart threw away whatever was in flight, and the
+      // scheduled source that asked for it began again from zero on the next
+      // poll — which for a planet build is hours of transfer per restart.
+      const stopped = library.stopAdds();
+      if (stopped.length > 0) {
+        console.log(
+          `[shutdown] stopped ${stopped.length} download(s); their bytes are ` +
+            'kept and resume when the source is next polled',
+        );
       }
     },
     ms: 1000,