pmtiles-swarm 0.32.3 → 0.34.0

package/CHANGELOG.md

@@ -7,9 +7,82 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.34.0
+### ✨ Features and improvements
+- **`md5` is a declared setting.** It was already honoured wherever a torrent is created, but it
+  appeared in no defaults list and no document, so it could only be written into the config file by
+  hand — `PATCH /api/config` refused it as an unknown setting, and nothing in the console showed
+  whether it was on. It now defaults to `false`, is documented, and can be changed without a
+  restart.
+- **`incomingRetentionDays`** sets how long an unfinished download stays resumable. Defaults to 14.
+
+### 🐞 Bug fixes
+- **A large download survives a restart instead of starting again from zero.** A scheduled web
+  source fetching a multi-hour archive lost the whole transfer every time the node restarted, and
+  began again from nothing on the next poll. Three things had to hold and only one did. The bytes
+  were always kept — the staging directory is named for a hash of its URL so the next add finds
+  it — but **shutdown deleted them**, because it stopped in-flight adds through `cancelAdd()`, and
+  cancelling discards the partial on purpose: somebody said stop. A restart is not that decision,
+  so shutdown now uses `stopAdds()`, which the fetch can tell apart. **Startup then swept whatever
+  survived**, on the reasoning that a killed process leaves a partial "nothing will ever look in
+  again" — true when staging names were random, false since they became a hash of the URL. And
+  **the validator did not outlive the process**: the `ETag` a resume is checked against lived in a
+  local, so a new process had nothing to compare and refused the resume as "the server offers no
+  ETag or Last-Modified", deleting the partial by the very attempt meant to continue it. It is now
+  written beside the bytes and removed when the download completes. A restart during a 700 GiB
+  transfer now costs the seconds since the last write.
+- **`.incoming` is swept by age rather than emptied.** Only a staging directory nothing has
+  written to for `incomingRetentionDays` (default 14) is cleared, so an unfinished download stays
+  resumable. The sweep also looks under `cacheSavePath`, which it never did — staging lands there
+  for cache-mode adds and under a source's own `savePath`, so the one configured `savePath` was
+  never the whole of where it could be.
+- **Adding a local archive answers when the file has been checked, not when it has been hashed.**
+  `POST /api/torrents` with a `path` held the response open for the whole hash — every byte of the
+  archive, twice with `md5` on — so the console's add dialog sat there for minutes with no sign
+  that anything was happening. Worse than the URL case it mirrors, because nothing was downloading
+  either: the file was already on the disk and visibly not moving, which reads as a submit button
+  that did nothing. It now answers `202` once the path has been identified and accepted, and the
+  hash reports itself through `/api/adds` like a download does. A path that is not there or is not
+  an archive still fails in the response. An archive already held answers `200` with its entry,
+  which the URL branch now does too rather than promising work that was already done. **Scripts
+  reading the created entry straight back from a `path` add need `/api/adds` or `/api/torrents`
+  instead** — magnets and `.torrent` URLs are unchanged and still answer `201`.
+- **A second add of a file already being hashed joins the first rather than starting its own.**
+  Only reachable now that the dialog closes quickly enough to submit twice, and two passes over the
+  same planet archive is an hour of disk for one result.
+- **The console's MD5 checkbox is now the decision it looks like.** It was only sent when ticked,
+  and the server reads an absent `md5` as "unspecified" and falls back to the node's configured
+  default — so on a node with `md5` on, an unticked box still hashed one, and the log said so while
+  the dialog appeared to have turned it off. The value is sent either way, and the box now starts
+  from the node's own setting rather than always unticked — otherwise making it authoritative would
+  have turned a configured default off for every add made from the console, the same disagreement
+  the other way round. Omitting `md5` from an API or CLI call still inherits the config default,
+  which is what that fallback is for.
+- **The save-location picker is hidden when adding a local file.** It did nothing there: a local
+  add registers the file's own directory as the save path whatever was chosen, which is exactly
+  what "hashed in place, nothing is copied" says — but the picker sat next to that sentence
+  implying otherwise, and offered no way to say "leave it where it is" because that is the only
+  thing it does.
+- **"What a torrent-aware client does" describes what they now do.** The section predated the swarm
+  handles moving into the TileJSON URL's fragment and still had a client learning where to join
+  from a TileJSON response — the one thing the fragment exists to avoid, since the swarm is the
+  part that depends on no server. It contradicted "bootstrapping without the server" two sections
+  below it.
+
+## 0.33.0
+### ✨ Features and improvements
+- **Requires pmtiles-torrent 0.6.1, which is what finally makes a downloading archive servable.**
+  Two fixes there, both about the few kilobytes at the front of a PMTiles archive that say where
+  every other section begins. A read used to ask for its piece with a deadline and then raise on
+  libtorrent's immediate "I do not have that yet" — abandoning the very fetch it had just
+  requested, so each attempt gave up within milliseconds and left nothing behind to hurry the
+  piece. And the head was only ever asked for by a reader, so an archive nothing happened to read
+  was never prioritised at all. Reads now wait out their own timeout, and the head is prioritised
+  when the archive is added. On a 698 GiB mirror with two complete seeds connected this was the
+  difference between 200 GiB downloaded with no tile servable and a head that arrives in seconds.
+
 ## 0.32.3
 ### ✨ Features and improvements
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
 - **Requires pmtiles-torrent 0.5.2, which deletes a torrent's resume file with its data.** Resume
@@ -21,11 +94,9 @@
   the data goes too: a removal that keeps the files is how a pause is expressed for an engine with
   no pause of its own, and discarding resume data there would turn every pause into a full re-hash.
 
-- _...Add new stuff here..._
 
 ## 0.32.2
 ### ✨ Features and improvements
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
 - **The head warmer no longer skips every archive it was built for.** It decided an archive was done
@@ -44,11 +115,9 @@
   header off local disk for nothing, while assuming the opposite would leave every existing
   subscription stuck exactly as it is.
 
-- _...Add new stuff here..._
 
 ## 0.32.1
 ### ✨ Features and improvements
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
 - **Requires pmtiles-torrent 0.5.1, so the connection indicator and Recheck files actually work.**
@@ -58,11 +127,9 @@
   cannot answer is deliberately not reported as unreachable, so there was nothing to see and
   nothing to explain why.
 
-- _...Add new stuff here..._
 
 ## 0.32.0
 ### ✨ Features and improvements
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
 - **A mutable magnet no longer carries a web seed.** A `ws=` URL names one build; a BEP 46 magnet
@@ -88,7 +155,6 @@
   **Restyle anything holding one.** A style carrying an older mutable magnet keeps working, but
   carries the stale web seed until it is regenerated.
 
-- _...Add new stuff here..._
 
 ## 0.31.0
 ### ✨ Features and improvements
@@ -115,10 +181,8 @@
   With two engines both are asked, since each keeps its own belief about the same file and a stale
   one on the secondary is why a browser peer would find nothing while the primary seeds happily.
 
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
-- _...Add new stuff here..._
 
 ## 0.30.0
 ### ✨ Features and improvements
@@ -148,10 +212,8 @@
   makes the eye stop to work out which it is looking at. Seconds are dropped rather than the date,
   since nothing here is sorted finely enough for them to matter; hovering still gives them.
 
-- _...Add new stuff here..._
 
 ### 🐞 Bug fixes
-- _...Add new stuff here..._
 
 ## 0.29.0
 ### ✨ Features and improvements

package/README.md

@@ -761,7 +761,7 @@ which the endpoint answers 501.
 | `POST`                | `/api/torrents/:infoHash/check`                             | Has the source changed since the torrent was made?                                                                                                                                                                                        |
 | `POST`                | `/api/torrents/:infoHash/rebuild`                           | Rebuild from the current source (mints a new infohash)                                                                                                                                                                                    |
 | `POST`                | `/api/check-origins`                                        | Check every archive with a watchable source                                                                                                                                                                                               |
-| `GET` `DELETE`        | `/api/adds`                                                 | Downloads still in flight, and cancelling one by URL                                                                                                                                                                                      |
+| `GET` `DELETE`        | `/api/adds`                                                 | Adds still in flight — downloads, and local files being hashed — and cancelling a download by URL. A hash cannot be stopped part-way, so it is listed but not cancellable                                                                 |
 | `GET` `POST`          | `/api/speed`                                                | Which speed limits are in force, and the manual switch between the two sets                                                                                                                                                               |
 | `GET`                 | `/api/categories`                                           | Every category, with the endpoints resolving to its newest build                                                                                                                                                                          |
 | `POST`                | `/api/adopt`, `/api/adopt/candidates`                       | Take over what an engine or another node holds                                                                                                                                                                                            |

package/docs/architecture-diagram.md

@@ -154,6 +154,15 @@ stops needing the tile endpoint at all.
 The important part is that it does both at once — HTTP for the first paint, swarm
 in the background — so there is never a blank map waiting for metadata.
 
+Discovery comes first, and it comes from the style rather than from a response.
+The source URL carries its handles in a fragment
+(`…/tiles.json#torrent=…&magnet=…`, described under
+[bootstrapping](#bootstrapping-without-the-server)), so a client knows there is a
+swarm behind a source before it makes a single request — which is the point, as
+that is what still works when the server is down. The TileJSON is fetched anyway,
+because it carries the tile endpoint for the first paint and a fuller `torrent`
+block than a fragment can, but it is no longer how the swarm is _found_.
+
 ```mermaid
 sequenceDiagram
     autonumber
@@ -162,24 +171,28 @@ sequenceDiagram
     participant HTTP as pmtiles-swarm<br/>(via CDN)
     participant BT as BitTorrent swarm
 
-    App->>P: load style → tiles.json
-    P->>HTTP: GET /archives/{hash}/tiles.json
-    HTTP-->>P: TileJSON + torrent block
-
-    Note over P: claims the /archives/{hash}/ prefix,<br/>so only these URLs come to it
+    App->>P: load style
+    Note over P: reads the fragment on each source URL:<br/>torrent= and magnet=. No request yet.
 
     par Map is usable immediately
+        P->>HTTP: GET /latest/{category}/tiles.json
+        Note over P,HTTP: the fragment is never sent
+        HTTP-->>P: TileJSON + torrent block
         App->>P: tile 12/2145/1436
         P->>HTTP: GET …/12/2145/1436.pbf
         HTTP-->>App: tile bytes
     and Swarm warms up in the background
-        P->>BT: join (.torrent — metadata already in hand)
+        P->>HTTP: GET the .torrent
+        HTTP-->>P: metainfo (piece hashes, trackers, web seeds)
+        P->>BT: join — metadata already in hand
         BT-->>P: connected
         P->>BT: fetch PMTiles header + root directory
         BT-->>P: those pieces
         Note over P: now able to resolve any tile<br/>to a byte range locally
     end
 
+    Note over P: claims the TileJSON URL's prefix (without<br/>the fragment), so only these URLs come to it
+
     App->>P: tile 12/2146/1436
     Note over P: tile → byte range (PMTiles directory)<br/>→ piece index
     P->>BT: fetch that piece
@@ -201,18 +214,43 @@ is why the `torrent` block carries the archive's `.torrent` rather than per-tile
 URLs: **there is nothing tile-specific in the swarm.** The swarm holds one file,
 and both ends know how to read tiles out of it.
 
-Three consequences worth being clear about:
+Consequences worth being clear about:
 
 - **HTTP is never fully abandoned.** It is the fallback for anything the swarm
   cannot answer quickly, and the only path until the swarm is connected.
 - **The client becomes a seeder.** Every piece it pulls, it serves — so a popular
   region gets _faster_ as more clients view it, which is the opposite of how a
   tile server behaves under load.
-- **Prefer the `.torrent` over the magnet.** A magnet carries only an infohash, so
-  the client must find peers and complete a metadata exchange before it knows
-  anything about the archive — measured at 90 to 240 seconds against a 72 GiB
-  archive. The `.torrent` served alongside the TileJSON already contains the
-  metadata and is ready immediately.
+- **The document is preferred, the fragment is the fallback.** Where the TileJSON
+  is reachable and carries a `torrent` block, that block wins: it is the archive's
+  own account of itself, and it has web seeds, size and piece length that two
+  handles in a URL do not. The fragment is what answers when the document does not
+  — unreachable, not JSON, or carrying no block — which is exactly the case the
+  handles were put in the URL for. A client that used only one of the two would be
+  either poorly informed or dependent on the server it is meant to survive.
+- **Prefer the `.torrent` over the magnet, and for a browser this is not a
+  preference.** A magnet carries only an infohash, so the client must find peers
+  and complete a metadata exchange before it knows anything about the archive —
+  measured at 90 to 240 seconds against a 72 GiB archive. That is the cost for a
+  client with a DHT. A browser has neither DHT nor UDP, and piece hashes reach a
+  BitTorrent client only from a **peer**, over BEP 9 — a web seed serves file
+  payload and never metainfo. So a page that cannot reach a peer cannot use the
+  web seed either, however reachable that web seed is: it has bytes it is not
+  allowed to trust. Fetching the `.torrent` over the same HTTPS the TileJSON came
+  from takes the peer off the critical path entirely, which is the difference
+  between working on a restricted network and not working at all.
+- **A client with a DHT can join on a magnet alone**, and one without cannot. That
+  is the one place the two kinds of client genuinely diverge, and it is why the
+  convention names `torrent=` first: a browser treats a source carrying only a
+  magnet as no candidate at all, while a native client is happy with it.
+- **Holding the metainfo is not evidence that anything will serve it.** It makes
+  the engine ready with no peer involved, which removes the very thing that used
+  to prove an archive was worth binding a source to — waiting for metadata was
+  never only a wait. A client that registers on metainfo alone can bind a source
+  to a swarm with nothing behind it, and that does not fall back, it _stalls_, and
+  the tiles never draw. One `Range: bytes=0-0` against the web seed restores the
+  evidence: a 206 proves the host is reachable, serves ranges and permits the
+  origin, which is all the engine needs from it.
 
 ---
 

package/docs/configuration.md

@@ -215,11 +215,13 @@ seeding.
 
 ## Creating torrents
 
-| setting                | default    |                                              |
-| ---------------------- | ---------- | -------------------------------------------- |
-| `pieceLength`          | `4194304`  | 4 MiB                                        |
-| `torrentFormat`        | `'hybrid'` | `'hybrid'`, `'v1'` or `'v2'`                 |
-| `allowUnknownArchives` | `false`    | publish files not recognised as map archives |
+| setting                 | default    |                                                 |
+| ----------------------- | ---------- | ----------------------------------------------- |
+| `pieceLength`           | `4194304`  | 4 MiB                                           |
+| `torrentFormat`         | `'hybrid'` | `'hybrid'`, `'v1'` or `'v2'`                    |
+| `allowUnknownArchives`  | `false`    | publish files not recognised as map archives    |
+| `md5`                   | `false`    | also compute an MD5 of each archive created     |
+| `incomingRetentionDays` | `14`       | how long an unfinished download stays resumable |
 
 ### `pieceLength`
 
@@ -260,6 +262,31 @@ reading over a swarm does not work the way it does for a flat, Hilbert-ordered
 file; a finished local copy has no such problem. See
 [serving-tiles.md](serving-tiles.md#what-can-be-served).
 
+### `incomingRetentionDays`
+
+An unfinished download is kept in `.incoming` for this long, and adding the same
+URL again resumes it — which is what makes a restart during a multi-hour
+transfer cost minutes rather than the whole download. A scheduled source picks
+its own back up on the next poll without being asked.
+
+Only what nothing has written to for this many days is cleared at startup, since
+whether a URL is still wanted is a question about configuration that the sweep
+cannot see. Set it lower on a small disk, or higher if an upstream can be
+unreachable for weeks.
+
+### `md5`
+
+Off by default, because it costs a second full read of the archive: with it on,
+adding a 700 GiB file takes twice as long and produces one convenience digest
+that nothing in BitTorrent uses. The torrent already verifies the content, and
+per piece rather than as a whole — this is for the manual check somebody wants to
+run against a published checksum, and it is carried in the feed for them.
+
+The console's **Also compute an MD5** box starts from this setting and is sent
+with the add either way, so unticking it applies to that add alone. An API or CLI
+call that omits `md5` inherits this; one that passes `true` or `false` decides
+for itself.
+
 ## Trackers
 
 `trackers` is baked into every torrent this node creates. It defaults to the

package/docs/internals.md

@@ -556,6 +556,35 @@ assumed:
 
 Any of those failing restarts the download rather than guessing.
 
+### Across a restart, not only across a stall
+
+All of that worked within one process and none of it survived leaving one, which
+made a restart during a long download cost the whole download. Three separate
+things had to be true, and only the first was:
+
+1. **The bytes are kept.** They always were — the staging directory is named for
+   `sha256(url)`, so the next add of the same URL finds it.
+2. **Nothing deletes them on the way past.** Two things did. Shutdown expressed
+   itself through `cancelAdd()`, and cancelling deletes the partial on purpose —
+   somebody said stop. A restart is not that decision, so shutdown now calls
+   `stopAdds()`, which aborts with a reason the fetch can tell apart. Startup
+   then swept `.incoming` unconditionally, on the reasoning that a killed
+   process leaves a partial "nothing will ever look in again" — true when
+   staging names were random, false since they became a hash of the URL. It now
+   reaps only what nothing has written to for `incomingRetentionDays`.
+3. **The validator outlives the process.** `stillTheSameFile` compares the
+   `ETag` seen when the download began against the one offered now, and that
+   header lived in a local. A new process had nothing to compare, so the resume
+   was refused for the one reason that cannot be recovered from — "the server
+   offers no ETag or Last-Modified" — and the partial was deleted by the very
+   attempt meant to continue it. It is now written to `<partial>.resume` beside
+   the bytes, carrying the URL it belongs to, and removed when the download
+   finishes so it cannot keep the staging directory from being cleared.
+
+The URL is recorded in the sidecar as well as implied by the directory name, so
+a staging directory that has been reused for something else is refused rather
+than spliced.
+
 ## Retiring and pruning a subscription
 
 Two different questions, which is why an RSS feed can have the first and not the

package/docs/publishing.md

@@ -54,6 +54,31 @@ curl -X POST localhost:8090/api/torrents \
 curl -X POST localhost:8090/api/adopt
 ```
 
+The first two of those answer differently, and a script should know which it is
+reading. A local path and a URL both take as long as it takes to read every byte
+of the archive — minutes for a local file, hours for a planet download, doubled
+again if `md5` is on — so they answer **`202 Accepted`** the moment the source
+has been checked, and the work carries on behind it:
+
+```json
+{
+  "accepted": true,
+  "path": "/mnt/maps/planet.pmtiles",
+  "message": "hashing; progress is reported by /api/adds"
+}
+```
+
+There is no infohash in that, because there is not one yet. `GET /api/adds`
+lists what is still running, and the archive appears in `GET /api/torrents` once
+it finishes. A source that fails its checks — a path that is not there, a URL
+that does not answer, a file that is not an archive — fails in the response
+instead, since that is what somebody can do something about. An archive already
+in the catalog answers `200` with the existing entry.
+
+A magnet, a `.torrent` URL and an uploaded `.torrent` are metadata rather than
+data, so there is nothing slow to wait for: those still answer `201` with the
+entry.
+
 ### When the source URL is not published
 
 Adding from a URL registers that URL as a web seed by default, because it is by

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.32.3",
+  "version": "0.34.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -46,7 +46,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.5.2",
+    "pmtiles-torrent": "^0.6.1",
     "webtorrent": "^3.0.21"
   },
   "engines": {

package/src/api.js

@@ -53,6 +53,55 @@ function route(handler) {
   return (req, res, next) => Promise.resolve(handler(req, res)).catch(next);
 }
 
+/**
+ * Starts an add that takes minutes to hours, and answers as soon as it is safe
+ * to say it has been accepted.
+ *
+ * Everything a caller can do something about is known long before the work is
+ * finished: the source answers, and it is an archive of a kind this will
+ * publish. What remains is transfer and hashing. Awaiting all of it held the
+ * response open for the whole thing, so the console's add dialog stayed on
+ * screen for the duration — over an archive that was visibly appearing behind
+ * it in the URL case, and over a file that had never moved in the local one,
+ * where it read as a submit button that had done nothing at all.
+ *
+ * Progress has its own route already: `runningAdds()` feeds `/api/adds`, the
+ * console polls it, and `DELETE /api/adds` cancels the ones that can be.
+ * @param {object} res - The response to answer.
+ * @param {Function} start - Called with `{onValidated}`; returns the add's promise.
+ * @param {object} accepted - Fields describing the source, for the 202 body.
+ * @param {string} message - What the 202 tells the caller is now happening.
+ * @param {string} what - Prefixed log tag and source, for a failure nobody is waiting on.
+ * @returns {Promise<void>} - Resolves once the response has been sent.
+ */
+async function acceptAdd(res, { start, accepted, message, what }) {
+  const validated = Promise.withResolvers();
+  const running = start({ onValidated: validated.resolve })
+    // A failure after validation has nowhere to be reported: the response has
+    // gone. It is logged where the rest of the work is, and swallowed here so
+    // it cannot take the process down as an unhandled rejection.
+    .catch((error) => {
+      validated.reject(error);
+      console.error(`${what}: ${error.message}`);
+    });
+
+  // Whichever comes first: the checks passing, or the whole attempt failing. A
+  // source that does not answer, or is not an archive, still reports itself in
+  // the dialog where somebody can correct it.
+  const outcome = await validated.promise;
+
+  // Already held, so there is no work to wait on and the entry itself is the
+  // better answer. Deliberately not extended to `joined`, where the promise
+  // belongs to somebody else's transfer and awaiting it would hold the
+  // response open for exactly as long as this exists to avoid.
+  if (outcome?.held) {
+    return void res.status(200).json(await running);
+  }
+
+  void running;
+  res.status(202).json({ accepted: true, ...accepted, message });
+}
+
 /**
  * Builds the HTTP application: JSON API, RSS feeds and the web UI.
  * @param {object} deps - Collaborators.
@@ -1481,41 +1530,24 @@ export function createApp({
 
       let entry;
       if (body.path) {
-        entry = await library.addLocalArchive(body.path, options);
+        // Nothing is copied and nothing is downloaded, but every byte is still
+        // read to compute the piece hashes — twice with md5 on. That is the
+        // whole wait for a local add, and it is no shorter for the file being
+        // on this disk already. See acceptAdd().
+        return await acceptAdd(res, {
+          start: (hooks) =>
+            library.addLocalArchive(body.path, { ...options, ...hooks }),
+          accepted: { path: body.path },
+          message: 'hashing; progress is reported by /api/adds',
+          what: `[hash] ${body.path}`,
+        });
       } else if (body.url) {
-        // Answered as soon as the URL has been checked, not when the transfer
-        // finishes. Awaiting the whole thing held the response open for the
-        // length of the download -- hours for a planet archive -- so the
-        // console's add dialog stayed on screen throughout, over an archive
-        // that was visibly appearing behind it.
-        //
-        // Progress has its own route already: runningAdds() feeds /api/adds,
-        // the console polls it, and DELETE /api/adds cancels one. This is the
-        // piece that was missing rather than a new mechanism.
-        const validated = Promise.withResolvers();
-        const running = library
-          .addRemoteArchive(body.url, {
-            ...options,
-            onValidated: validated.resolve,
-          })
-          // A failure after validation has nowhere to be reported: the
-          // response has gone. It is logged where the rest of the fetch is,
-          // and swallowed here so it cannot take the process down as an
-          // unhandled rejection.
-          .catch((error) => {
-            validated.reject(error);
-            console.error(`[fetch] ${body.url}: ${error.message}`);
-          });
-
-        // Whichever comes first: the checks passing, or the whole attempt
-        // failing. A URL that does not answer, or is not an archive, still
-        // reports itself in the dialog where somebody can correct it.
-        await validated.promise;
-        void running;
-        return res.status(202).json({
-          accepted: true,
-          url: body.url,
+        return await acceptAdd(res, {
+          start: (hooks) =>
+            library.addRemoteArchive(body.url, { ...options, ...hooks }),
+          accepted: { url: body.url },
           message: 'fetching; progress is reported by /api/adds',
+          what: `[fetch] ${body.url}`,
         });
       } else if (body.magnet) {
         entry = await library.addExistingTorrent(

package/src/config.js

@@ -155,6 +155,17 @@ const DEFAULTS = {
    * publish any readable file to a public swarm.
    */
   allowUnknownArchives: false,
+  /**
+   * Also compute an MD5 of each archive created here. Costs a second full read
+   * of the file. Already honoured wherever a torrent is created; declared so it
+   * can be seen and set rather than only written into the file by hand.
+   */
+  md5: false,
+  /**
+   * How long an unfinished download is kept before startup treats it as
+   * abandoned. Until then, re-adding the same URL resumes it.
+   */
+  incomingRetentionDays: 14,
   /**
    * Who may administer this node. Tiles, TileJSON and the feed are always
    * public; everything under /api/ is gated whenever anything here is set. See

package/src/index.js

@@ -237,9 +237,17 @@ PMTILES_SWARM_PUBLIC_URL
   stoppers.unshift({
     label: 'downloads in progress',
     stop: () => {
-      const cancelled = library.cancelAdd();
-      if (cancelled.length > 0) {
-        console.log(`[shutdown] cancelled ${cancelled.length} download(s)`);
+      // stopAdds, not cancelAdd: a restart is not a decision to stop wanting
+      // the archive, and cancelling deletes the partial download. Through
+      // cancelAdd every restart threw away whatever was in flight, and the
+      // scheduled source that asked for it began again from zero on the next
+      // poll — which for a planet build is hours of transfer per restart.
+      const stopped = library.stopAdds();
+      if (stopped.length > 0) {
+        console.log(
+          `[shutdown] stopped ${stopped.length} download(s); their bytes are ` +
+            'kept and resume when the source is next polled',
+        );
       }
     },
     ms: 1000,

package/src/library.js

@@ -40,6 +40,16 @@ import {
  */
 export const INCOMING = '.incoming';
 
+/**
+ * Why an in-flight download was aborted, when the answer is "we are stopping".
+ *
+ * Passed as the abort reason rather than tracked alongside, so it arrives with
+ * the signal at the one place that has to tell the difference — and cannot be
+ * left behind by a path that clears its bookkeeping before it handles the
+ * error, which is exactly what the fetch does.
+ */
+export const STOPPING = { stopping: true };
+
 /**
  * Moves a finished archive out of staging and into its final directory.
  *
@@ -63,6 +73,32 @@ export async function settleFromStaging({ staging, savePath, name }) {
   return to;
 }
 
+/**
+ * When anything inside a directory was last written, as a timestamp.
+ *
+ * One level down is enough: a staging directory holds the archive being
+ * written and the sidecar describing it, and nothing nests below that.
+ * @param {string} dir - The directory to look in.
+ * @returns {Promise<number | null>} - Milliseconds, or null if it cannot be read.
+ */
+async function newestMtime(dir) {
+  const entries = await fs
+    .readdir(dir, { withFileTypes: true })
+    .catch(() => null);
+  if (!entries) return null;
+
+  let newest = await fs
+    .stat(dir)
+    .then((stat) => stat.mtimeMs)
+    .catch(() => null);
+  for (const entry of entries) {
+    const stat = await fs.stat(path.join(dir, entry.name)).catch(() => null);
+    if (stat && (newest === null || stat.mtimeMs > newest))
+      newest = stat.mtimeMs;
+  }
+  return newest;
+}
+
 export class Library {
   #catalog;
   #engine;
@@ -71,9 +107,9 @@ export class Library {
   #rebuildQueue = Promise.resolve();
   /** Moves in flight, and the last outcome for each archive. */
   #moves = new Map();
-  /** In-flight remote adds, by URL, so they can be cancelled. */
+  /** Adds in flight, by URL or absolute path, so they can be watched and stopped. */
   #running = new Map();
-  /** Downloads in progress, by URL, so a second request joins the first. */
+  /** The same adds as promises, so a second request for one joins the first. */
   #inFlight = new Map();
   /** The tile reader, told to forget an archive whose source may have changed. */
   #tiles;
@@ -144,16 +180,60 @@ export class Library {
    * Adds a local PMTiles archive, creating a torrent for it.
    *
    * The data is left where it is and the torrent points at it, so publishing a
-   * 700 GiB archive copies nothing.
+   * 700 GiB archive copies nothing. Reading it is another matter: every byte
+   * goes past the hasher, which is why this reports progress and why a caller
+   * can stop waiting on it. See `onValidated`.
    * @param {string} filePath - Path to the .pmtiles file.
    * @param {object} [options] - Category, trackers, web seeds, piece length.
    * @returns {Promise<object>} - The catalog entry.
    */
   async addLocalArchive(filePath, options = {}) {
     const requested = path.resolve(filePath);
+
+    // Both shortcuts here have to fire onValidated before returning, and for
+    // the same reason as the remote ones: a caller waiting on it to answer a
+    // request would otherwise wait for something that has already happened.
     const existing = this.#catalog.findBySource(requested);
-    if (existing) return existing;
+    if (existing) {
+      options.onValidated?.({
+        path: requested,
+        kind: existing.kind,
+        held: true,
+      });
+      return existing;
+    }
+
+    // A second request for a file already being hashed joins the first. The
+    // catalog cannot answer this, since the entry exists only once hashing has
+    // finished — and until this returned early, nothing was quick enough to
+    // double-submit. Now that the dialog closes on validation it is, and two
+    // passes over the same planet archive is an hour of disk for one result.
+    const inFlight = this.#inFlight.get(requested);
+    if (inFlight) {
+      console.log(
+        `[hash] ${requested} is already being hashed; joining that one`,
+      );
+      options.onValidated?.({ path: requested, joined: true });
+      return inFlight;
+    }
+
+    const attempt = this.#hashLocalArchive(requested, options).finally(() =>
+      this.#inFlight.delete(requested),
+    );
+    this.#inFlight.set(requested, attempt);
+    return attempt;
+  }
 
+  /**
+   * Identifies, hashes and registers one local archive.
+   *
+   * Separate from `addLocalArchive` so that the deduplication above wraps a
+   * single call and cannot be bypassed by a second entry point later.
+   * @param {string} requested - Absolute path to the .pmtiles file.
+   * @param {object} [options] - Category, trackers, web seeds, piece length.
+   * @returns {Promise<object>} - The catalog entry.
+   */
+  async #hashLocalArchive(requested, options = {}) {
     // Move before hashing, not after. A rename on one filesystem is metadata
     // and costs nothing, while hashing the archive is minutes — so the cheap
     // irreversible step goes first, and the torrent is built from where the
@@ -179,41 +259,74 @@ export class Library {
       allowUnknown: options.allowUnknown ?? this.#config.allowUnknownArchives,
     });
 
-    // Only PMTiles can have its tiles served, so only PMTiles gets probed.
-    const summary =
-      identified.kind === 'pmtiles'
-        ? await probePMTiles(absolute).catch(() => undefined)
-        : undefined;
-
-    const created = await createTorrentFromFile(absolute, {
-      creator: this.#creator(),
-      pieceLength: options.pieceLength ?? this.#config.pieceLength,
-      trackers: this.#trackersFor(options),
-      webSeeds: [...new Set(webSeeds)],
-      comment: options.comment,
-      md5: options.md5 ?? this.#config.md5,
+    // Everything a caller can do something about has now been checked: the
+    // path exists, it is an archive of a kind this will publish, and it is not
+    // something else that happened to be readable. What remains is reading it
+    // end to end, which for a planet archive is minutes at best — twice that
+    // with md5 on — and no amount of waiting changes the outcome.
+    //
+    // A caller that wants to stop waiting there says so with onValidated,
+    // exactly as the remote add does. Without it the console's add dialog sat
+    // open for the whole hash, over a file that was already on the disk and
+    // going nowhere, which read as a submit button that had done nothing.
+    options.onValidated?.({ path: absolute, kind: identified.kind });
+
+    // Tracked from here so the console has something to show while the hash
+    // runs. Registered without an AbortController, unlike a remote add: there
+    // is no cancelling a hash in progress, since neither libtorrent's creator
+    // nor create-torrent takes a signal. cancelAdd() skips entries without a
+    // controller, so this is inert there rather than a button that lies.
+    const { size } = await fs.stat(absolute).catch(() => ({ size: undefined }));
+    this.#running.set(requested, {
+      name: path.basename(absolute),
+      startedAt: new Date().toISOString(),
+      // No byte count: hashing reports nothing until it is done, so a progress
+      // bar here would sit at zero and then vanish. `total` is the size it is
+      // working through, which with the start time is enough to show it moving.
+      received: undefined,
+      total: size,
+      phase: 'hashing',
     });
 
-    return this.#register(created, {
-      categories: options.categories ?? options.category,
-      // `watch` names the folder that imported this, where one did. Not the
-      // same thing as the directory it sits in: with publishDir it has already
-      // moved somewhere else, and an archive dropped into a watched folder by
-      // hand is still that folder's to retire.
-      source: { type: 'file', location: absolute, watch: options.watch },
-      // The torrent names the file, so the save path is its parent directory.
-      savePath: path.dirname(absolute),
-      pmtiles: summary,
-      // Read out of the archive, not taken from anybody's word for it. The
-      // prewarmer needs the difference: a summary that arrived in a feed says
-      // nothing about whether the header is on this disk. See prewarm.due().
-      summarySource: 'header',
-      kind: identified.kind,
-      sparse: options.sparse,
-      md5: created.md5,
-      webSeeds: [...new Set(webSeeds)],
-      seedOnly: true,
-    });
+    try {
+      // Only PMTiles can have its tiles served, so only PMTiles gets probed.
+      const summary =
+        identified.kind === 'pmtiles'
+          ? await probePMTiles(absolute).catch(() => undefined)
+          : undefined;
+
+      const created = await createTorrentFromFile(absolute, {
+        creator: this.#creator(),
+        pieceLength: options.pieceLength ?? this.#config.pieceLength,
+        trackers: this.#trackersFor(options),
+        webSeeds: [...new Set(webSeeds)],
+        comment: options.comment,
+        md5: options.md5 ?? this.#config.md5,
+      });
+
+      return await this.#register(created, {
+        categories: options.categories ?? options.category,
+        // `watch` names the folder that imported this, where one did. Not the
+        // same thing as the directory it sits in: with publishDir it has
+        // already moved somewhere else, and an archive dropped into a watched
+        // folder by hand is still that folder's to retire.
+        source: { type: 'file', location: absolute, watch: options.watch },
+        // The torrent names the file, so the save path is its parent directory.
+        savePath: path.dirname(absolute),
+        pmtiles: summary,
+        // Read out of the archive, not taken from anybody's word for it. The
+        // prewarmer needs the difference: a summary that arrived in a feed says
+        // nothing about whether the header is on this disk. See prewarm.due().
+        summarySource: 'header',
+        kind: identified.kind,
+        sparse: options.sparse,
+        md5: created.md5,
+        webSeeds: [...new Set(webSeeds)],
+        seedOnly: true,
+      });
+    } finally {
+      this.#running.delete(requested);
+    }
   }
 
   /**
@@ -503,38 +616,77 @@ export class Library {
   }
 
   /**
-   * Stops an in-flight remote add.
+   * Stops an in-flight remote add, and discards what it had downloaded.
+   *
+   * Only a remote one. A local add is listed alongside them but holds no
+   * controller, because a hash in progress cannot be interrupted — neither
+   * libtorrent's creator nor create-torrent takes a signal. It is skipped
+   * below rather than special-cased, and reports as nothing cancelled.
    * @param {string} [url] - The source URL, or all of them when omitted.
    * @returns {string[]} - The URLs cancelled.
    */
   cancelAdd(url) {
+    return this.#abortAdds(url, undefined);
+  }
+
+  /**
+   * Stops in-flight adds because the process is going down, keeping the bytes.
+   *
+   * The distinction matters more than it looks. Cancelling deletes the partial
+   * download, deliberately: somebody said stop, and leaving a few hundred
+   * gigabytes behind after that is the invisible waste the cleanup exists to
+   * avoid. A restart is not that decision — nobody stopped wanting the
+   * archive — but shutdown used to express itself through the same call, so
+   * every restart deleted whatever was in flight, and the scheduled source
+   * that had asked for it started again from zero on the next poll. The
+   * staging directory is named for its URL precisely so the next attempt finds
+   * it; this is what lets it.
+   * @returns {string[]} - The URLs stopped.
+   */
+  stopAdds() {
+    return this.#abortAdds(undefined, STOPPING);
+  }
+
+  /**
+   * Aborts adds, saying why so the fetch can tell the two apart.
+   * @param {string} [url] - One source URL, or all of them when omitted.
+   * @param {object} [reason] - Passed to abort(); STOPPING keeps the partial.
+   * @returns {string[]} - The URLs aborted.
+   */
+  #abortAdds(url, reason) {
     const targets = url ? [url] : [...this.#running.keys()];
-    const cancelled = [];
+    const aborted = [];
     for (const target of targets) {
       const { controller } = this.#running.get(target) ?? {};
       if (!controller) continue;
-      controller.abort();
+      controller.abort(reason);
       this.#running.delete(target);
-      cancelled.push(target);
+      aborted.push(target);
     }
-    return cancelled;
+    return aborted;
   }
 
   /**
-   * Remote adds currently running.
-   * @returns {string[]} - Their source URLs.
+   * Adds currently running, remote and local alike.
+   * @returns {object[]} - One record per add, with whatever progress there is.
    */
   runningAdds() {
     // Reported with progress, not just named. An archive added from a URL has
     // to be downloaded whole before there is anything to hash a torrent out
     // of, so for hours there is no catalog entry and nothing in the list —
-    // which looks exactly like a source that silently did nothing.
+    // which looks exactly like a source that silently did nothing. A local add
+    // has the same gap for the length of the hash, without the download.
     return [...this.#running.entries()].map(([url, state]) => ({
       url,
       name: state.name,
-      received: state.received ?? 0,
+      // Left absent rather than zeroed for a hash, which reports nothing until
+      // it finishes. A zero here is indistinguishable from a download that has
+      // not moved, and the console draws the two differently.
+      received: state.phase === 'hashing' ? undefined : (state.received ?? 0),
       total: state.total,
       startedAt: state.startedAt,
+      phase: state.phase ?? 'fetching',
+      cancellable: Boolean(state.controller),
     }));
   }
 
@@ -696,6 +848,7 @@ export class Library {
         },
       });
     } catch (error) {
+      const reached = this.#running.get(url)?.received ?? 0;
       this.#running.delete(url);
       // Cancelling is a decision to stop wanting this; running out of attempts
       // is not. The two used to be cleaned up identically, so a download that
@@ -706,8 +859,19 @@ export class Library {
       // A cancelled fetch is still removed. Somebody said stop, and leaving
       // gigabytes behind after that is the invisible waste this was written to
       // avoid in the first place.
-      const cancelled = controller.signal.aborted;
-      if (staging && cancelled) {
+      //
+      // Stopping is neither. The process going down is not a decision about
+      // this archive, so it keeps its bytes and says where they are; the next
+      // poll of whatever asked for it resumes from there. Read off the abort
+      // reason because #running has already been cleared by then.
+      const stopping = controller.signal.reason === STOPPING;
+      const cancelled = controller.signal.aborted && !stopping;
+      if (staging && stopping) {
+        console.log(
+          `[fetch] ${url} stopped at ${reached} bytes for shutdown; kept in ` +
+            `${staging}, and the next add of this URL resumes from there`,
+        );
+      } else if (staging && cancelled) {
         await fs.rm(staging, { recursive: true, force: true }).catch(() => {});
       } else if (staging) {
         console.warn(
@@ -2210,11 +2374,22 @@ export class Library {
   }
 
   /**
-   * Clears out staging directories left by downloads that never finished.
+   * Clears out staging directories nothing is going to come back for.
+   *
+   * This used to remove everything it found, on the reasoning that a partial
+   * archive left by a killed process sits "in a directory nothing will ever
+   * look in again". That stopped being true when the staging directory was
+   * named for a hash of its URL: the next add of the same URL looks in exactly
+   * that directory and continues from what is in it. Sweeping unconditionally
+   * therefore deleted the one thing the naming scheme exists to preserve, and
+   * every restart cost a scheduled source its whole download — the worst case
+   * being the one this is supposed to help with, since a process killed
+   * outright is precisely when hours of transfer are worth keeping.
    *
-   * The failure path removes its own, but a process killed outright cannot.
-   * What it leaves is a partial archive in a directory nothing will ever look
-   * in again — invisible, and for a planet build measured in gigabytes.
+   * What is genuinely abandoned is what nothing has touched for a while: a
+   * source that was removed from the config, a URL that will never be asked
+   * for again. Age is the only honest test available here, because whether a
+   * URL is still wanted is a question about configuration this cannot see.
    *
    * Safe to run at startup precisely because nothing else may be writing here:
    * the data directory lock means one node owns it, and this node has not
@@ -2222,17 +2397,48 @@ export class Library {
    * @returns {Promise<number>} - How many were removed.
    */
   async sweepIncoming() {
-    const root = path.join(this.#config.savePath ?? '', INCOMING);
-    const stale = await fs.readdir(root).catch(() => []);
+    const days = this.#config.incomingRetentionDays ?? 14;
+    const cutoff = Date.now() - days * 24 * 60 * 60 * 1000;
+
+    // Both roots: a source can name its own save path, and cache-mode adds go
+    // to cacheSavePath, so the one configured savePath was never the whole of
+    // where staging lands.
+    const roots = [
+      ...new Set(
+        [this.#config.savePath, this.#config.cacheSavePath]
+          .filter(Boolean)
+          .map((root) => path.join(root, INCOMING)),
+      ),
+    ];
 
     let removed = 0;
-    for (const name of stale) {
-      await fs.rm(path.join(root, name), { recursive: true, force: true });
-      removed += 1;
+    let kept = 0;
+    for (const root of roots) {
+      for (const name of await fs.readdir(root).catch(() => [])) {
+        const at = path.join(root, name);
+        // The newest thing in it, not the directory's own timestamp: on some
+        // filesystems a directory's mtime does not move when a file inside it
+        // is appended to, which for a download in progress is every write.
+        const touched = await newestMtime(at);
+        if (touched !== null && touched >= cutoff) {
+          kept += 1;
+          continue;
+        }
+        await fs.rm(at, { recursive: true, force: true });
+        removed += 1;
+      }
     }
+
     if (removed > 0) {
       console.log(
-        `[library] cleared ${removed} unfinished download(s) from ${INCOMING}`,
+        `[library] cleared ${removed} abandoned download(s) from ${INCOMING} ` +
+          `(nothing written to them for ${days} days)`,
+      );
+    }
+    if (kept > 0) {
+      console.log(
+        `[library] keeping ${kept} unfinished download(s) in ${INCOMING}; ` +
+          'adding the same URL again continues from where each stopped',
       );
     }
     return removed;

package/src/torrent-create.js

@@ -293,6 +293,85 @@ function rangeStart(contentRange) {
   return match ? Number(match[1]) : null;
 }
 
+/**
+ * Where the validator for a partial download is remembered.
+ * @param {string} target - The file being written.
+ * @returns {string} - The sidecar path.
+ */
+function resumePathFor(target) {
+  return `${target}.resume`;
+}
+
+/**
+ * Remembers what the source said about the file, beside the partial itself.
+ *
+ * `stillTheSameFile` compares the validator seen when the download began
+ * against the one offered now, and a validator held only in a local was a
+ * validator that died with the process. The bytes survived a restart and the
+ * comparison could not, so the resume was refused for the one reason that is
+ * not recoverable -- "the server offers no ETag or Last-Modified" -- and hours
+ * of transfer were deleted by the attempt that was meant to continue them.
+ *
+ * Written beside the partial rather than into the catalog because that is where
+ * it is true: the pair is the resumable thing, and neither half means anything
+ * without the other.
+ * @param {string} target - The file being written.
+ * @param {string} url - The source, so a stale sidecar cannot be mistaken for this one's.
+ * @param {Headers} headers - The response the download began with.
+ * @returns {Promise<void>} - Resolves once written.
+ */
+async function rememberValidator(target, url, headers) {
+  const etag = headers?.get('etag') ?? null;
+  const lastModified = headers?.get('last-modified') ?? null;
+  // Nothing worth remembering: with neither validator a resume would be
+  // refused anyway, and an empty sidecar only invites believing in it.
+  if (!etag && !lastModified) return;
+  await fs
+    .writeFile(
+      resumePathFor(target),
+      JSON.stringify({ url, etag, lastModified, at: new Date().toISOString() }),
+    )
+    .catch((error) => {
+      // Not fatal. The download still works; only a restart costs more.
+      console.warn(
+        `[fetch] could not record resume data for ${url}: ${error.message}`,
+      );
+    });
+}
+
+/**
+ * Reads back what a previous attempt saw, if it was for this same URL.
+ * @param {string} target - The file being written.
+ * @param {string} url - The source now being fetched.
+ * @returns {Promise<Headers | null>} - Headers to compare against, or null.
+ */
+async function recallValidator(target, url) {
+  try {
+    const saved = JSON.parse(await fs.readFile(resumePathFor(target), 'utf8'));
+    // A staging directory is named for its URL, so a mismatch here means the
+    // file has been reused for something else. Refusing is the safe read.
+    if (saved.url !== url) return null;
+    const headers = new Headers();
+    if (saved.etag) headers.set('etag', saved.etag);
+    if (saved.lastModified) headers.set('last-modified', saved.lastModified);
+    return headers.has('etag') || headers.has('last-modified') ? headers : null;
+  } catch {
+    // No sidecar, or an unreadable one. Both mean the same thing: there is
+    // nothing to compare against, so this behaves as it did before.
+    return null;
+  }
+}
+
+/**
+ * Drops a partial download and the validator that described it.
+ * @param {string} target - The file being written.
+ * @returns {Promise<void>} - Resolves once both are gone.
+ */
+async function discardPartial(target) {
+  await fs.rm(target, { force: true });
+  await fs.rm(resumePathFor(target), { force: true }).catch(() => {});
+}
+
 /**
  * Streams a URL to a file, resuming where a previous attempt stopped.
  *
@@ -337,6 +416,19 @@ async function downloadTo(url, target, onProgress, signal, options = {}) {
   // is not progress.
   let consumed = 0;
   let best = await bytesOnDisk(target);
+
+  // Picked up from the last process where there was one. This is what makes a
+  // partial survive a restart rather than only a stall: the bytes were always
+  // kept, but without the validator that described them the resume below had
+  // nothing to compare and threw them away.
+  if (best > 0) {
+    firstHeaders = await recallValidator(target, url);
+    if (firstHeaders) {
+      console.log(
+        `[fetch] ${url}: continuing from ${best} bytes left by an earlier run`,
+      );
+    }
+  }
   // Reset does mean an unlucky download can go round more times than the
   // budget names, which is the point, so there is a ceiling as well: a source
   // dribbling a few bytes before dropping every time would otherwise retry
@@ -349,7 +441,10 @@ async function downloadTo(url, target, onProgress, signal, options = {}) {
     const from = await bytesOnDisk(target);
     // A file already at full length is one a previous attempt finished, and
     // that the caller died before renaming. Re-fetching it buys nothing.
-    if (total && from >= total) return from;
+    if (total && from >= total) {
+      await fs.rm(resumePathFor(target), { force: true }).catch(() => {});
+      return from;
+    }
 
     let response;
     try {
@@ -403,14 +498,20 @@ async function downloadTo(url, target, onProgress, signal, options = {}) {
         // because it looks like a complete download. The next attempt sees an
         // empty target, sends no Range, and gets the whole file.
         await response.body.cancel().catch(() => {});
-        await fs.rm(target, { force: true });
+        await discardPartial(target);
         total = 0;
+        best = 0;
         firstHeaders = null;
         continue;
       }
     }
 
-    if (!firstHeaders) firstHeaders = response.headers;
+    if (!firstHeaders) {
+      firstHeaders = response.headers;
+      // Recorded before a byte is written, because the useful moment to have
+      // it is the one where this process is no longer running.
+      await rememberValidator(target, url, firstHeaders);
+    }
     const length = Number(response.headers.get('content-length') ?? 0);
     // On a 206 the length is what remains, not the size of the whole file.
     if (length) total = appending ? from + length : length;
@@ -437,6 +538,9 @@ async function downloadTo(url, target, onProgress, signal, options = {}) {
         { signal },
       );
       onProgress?.({ received, total, done: true });
+      // Nothing left to resume. Left behind it would also stop the staging
+      // directory being removed, since that is only unlinked once empty.
+      await fs.rm(resumePathFor(target), { force: true }).catch(() => {});
       return received;
     } catch (error) {
       if (signal?.aborted) throw error;

package/src/web/index.html

@@ -688,8 +688,10 @@
             <div class="sub">
               For a quick manual check. The torrent already verifies the
               archive, and per piece rather than as a whole — this is the
-              familiar convenience, not the integrity guarantee.
+              familiar convenience, not the integrity guarantee. It costs a
+              second read of the whole file, so the add takes twice as long.
             </div>
+            <div class="sub" id="md5-default"></div>
           </div>
         </fieldset>
 
@@ -902,12 +904,16 @@
       }
 
       /**
-       * Shows downloads that have no archive yet.
+       * Shows adds that have no archive yet.
        *
        * An archive added from a URL is fetched whole before there is anything
        * to hash a torrent out of, so for the length of that download there is
        * no catalog entry and nothing in the table. For a planet build that is
        * hours in which a watched location looks like it silently did nothing.
+       *
+       * A local file has the same gap without the download: the bytes are
+       * already here, but they still all go past the hasher, and until that
+       * finishes there is no infohash to file it under either.
        * @param {object[]} running - From /api/adds.
        * @returns {void}
        */
@@ -917,30 +923,48 @@
           box.innerHTML = '';
           return;
         }
+        const cancellable = running.filter((add) => add.cancellable !== false);
         box.innerHTML = `
-          <h3 style="margin:1.2rem 0 0.5rem">Downloading, before a torrent exists</h3>
+          <h3 style="margin:1.2rem 0 0.5rem">Being added, before a torrent exists</h3>
           <div class="sub" style="margin-bottom:0.6rem">
-            These have no infohash yet — an archive has to be here in full
-            before it can be hashed. They appear in the table above once that
+            These have no infohash yet — an archive has to be read end to end
+            before it can be hashed, whether it is arriving from a URL or
+            already on the disk. They appear in the table above once that
             finishes.
           </div>
           ${running
             .map((add) => {
-              const pct = add.total ? (add.received / add.total) * 100 : null;
+              // Hashing reports nothing until it is done, so there is no
+              // percentage to draw — only how big the file is and how long it
+              // has been going. A bar pinned at zero for twenty minutes says
+              // "stuck" when the honest answer is "no idea, still working".
+              const hashing = add.phase === 'hashing';
+              const pct =
+                add.total && add.received != null
+                  ? (add.received / add.total) * 100
+                  : null;
+              const elapsed = add.startedAt
+                ? duration(Date.now() - new Date(add.startedAt).getTime())
+                : null;
+              const value = hashing
+                ? `hashing${add.total ? ` ${bytes(add.total)}` : ''}${
+                    elapsed ? ` · ${elapsed}` : ''
+                  }`
+                : pct == null
+                  ? bytes(add.received)
+                  : `${pct.toFixed(1)}%`;
               return `
                 <div class="piecerow">
                   <span class="label" title="${escapeHtml(add.url)}">${escapeHtml(
                     add.name ?? add.url.split('/').pop() ?? add.url,
                   )}</span>
                   <span class="track"><i style="width:${pct == null ? 0 : pct.toFixed(1)}%"></i></span>
-                  <span class="value">${
-                    pct == null ? bytes(add.received) : `${pct.toFixed(1)}%`
-                  }</span>
+                  <span class="value">${value}</span>
                 </div>`;
             })
             .join('')}
           <div class="bar" style="margin-top:0.5rem">
-            ${running
+            ${cancellable
               .map(
                 (add) =>
                   `<button data-cancel="${escapeHtml(add.url)}">Cancel ${escapeHtml(
@@ -2032,7 +2056,7 @@ Every piece is hashed against the ` +
        */
       function locationPicker(id) {
         return `
-          <div class="field">
+          <div class="field" id="${id}-field">
             <label for="${id}-select">Save location</label>
             <select id="${id}-select"></select>
             <input id="${id}-path" hidden placeholder="M:\\archives" style="margin-top:0.4rem" />
@@ -2202,6 +2226,11 @@ Every piece is hashed against the ` +
         // Only a torrent created here can carry a digest; joining one takes
         // whatever that torrent already says.
         $('md5-options').hidden = kind !== 'url' && kind !== 'path';
+        // A local file is hashed where it lies and seeded from there:
+        // addLocalArchive registers the file's own directory as the save path,
+        // whatever was picked here. Offering the choice invited the reasonable
+        // reading that the archive was about to be copied or moved somewhere.
+        $('add-loc-field').hidden = kind === 'path';
         // Only joining has a mode to choose. Creating a torrent here always
         // means holding the file, so the question does not arise.
         $('join-options').hidden = kind !== 'magnet' && kind !== 'torrentUrl';
@@ -2238,8 +2267,13 @@ Every piece is hashed against the ` +
         $('extra-seeds').value = '';
         $('add-error').textContent = '';
         $('use-source-seed').checked = true;
-        $('want-md5').checked = false;
         $('extra-trackers').value = '';
+        // Set from the node's own setting below, not assumed here. The box is
+        // what gets sent now, so starting it at false on a node configured for
+        // MD5 would quietly turn the setting off for every add made from the
+        // console — the same disagreement the other way round.
+        $('want-md5').checked = false;
+        $('md5-default').textContent = '';
 
         try {
           const { config } = await api('/api/config');
@@ -2247,6 +2281,10 @@ Every piece is hashed against the ` +
           $('default-trackers').textContent = defaults.length
             ? `Every torrent created here announces to: ${defaults.join(', ')}`
             : 'No default trackers are configured.';
+          $('want-md5').checked = Boolean(config.md5);
+          $('md5-default').textContent = config.md5
+            ? 'This node computes one by default; unticking it applies to this add only.'
+            : '';
         } catch {
           $('default-trackers').textContent =
             'Defaults come from the trackers setting.';
@@ -2299,7 +2337,11 @@ Every piece is hashed against the ` +
           .filter(Boolean);
         if (trackers.length > 0) body.addTrackers = trackers;
 
-        if ($('want-md5').checked) body.md5 = true;
+        // Sent either way, because the server reads a missing md5 as "unspecified"
+        // and falls back to the node's configured default. An unticked box was
+        // therefore not a decision not to, and on a node configured for md5 the
+        // log reported hashing that the dialog appeared to have turned off.
+        body.md5 = $('want-md5').checked;
         if (kind === 'magnet' || kind === 'torrentUrl') {
           body.mode = document.querySelector('input[name="join-mode"]:checked').value;
         }
@@ -2321,7 +2363,16 @@ Every piece is hashed against the ` +
             body: { ...body, ...chosenLocation('add-loc') },
           });
           $('add-dialog').close();
-          toast(kind === 'url' ? 'fetching — watch the log' : 'added');
+          // The two slow kinds close on validation, not on completion, so
+          // saying "added" would be claiming something that has not happened
+          // yet. What has happened is that it was accepted and started.
+          toast(
+            kind === 'url'
+              ? 'fetching — watch the log'
+              : kind === 'path'
+                ? 'hashing — watch the log'
+                : 'added',
+          );
           refresh();
         } catch (error) {
           $('add-error').textContent = error.message;