pmtiles-swarm 0.43.0 → 0.45.0

package/CHANGELOG.md

@@ -7,6 +7,37 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.45.0
+### ✨ Features and improvements
+- **Requires pmtiles-torrent 0.9.1**, which stops the sidecar rewriting resume data that has not
+  changed. A hybrid torrent carries a merkle tree of 32 bytes per 16 KiB block in its resume data —
+  a few hundred megabytes for a 128 GiB archive — and it was being restaged, fsynced and renamed
+  every five minutes for every archive at once, to record that nothing had moved. It also gives each
+  torrent its own share of the save budget instead of five seconds for the whole library, which is
+  why a node with four archives persisted two of them and a different two next time.
+- **Running as a service documents a newer libtorrent than the distribution ships.** `apt` gives you
+  whatever your release froze on, `pip install` into the system Python is refused under PEP 668, and
+  `--break-system-packages` is not the way round it. A virtualenv plus the existing
+  `libtorrent.python` key is, reversibly and without touching apt. The upgrade section now also says
+  that libtorrent is a third thing an update never moves, and how to check which one the service
+  actually imports.
+
+### 🐞 Bug fixes
+
+## 0.44.0
+### ✨ Features and improvements
+- **Web seeds are added where they are listed.** The **HTTP sources** tab now has a field and an Add
+  button under the list, so adding one shows what is already there and what the addition did. It
+  replaces the **Add web seed** button in Actions, which opened a `prompt()` a long way from the list
+  it changed — one way to do this rather than two.
+
+  The tab now also says why this is safe on a published archive: `url-list` sits outside the info
+  dictionary, so the infohash is unchanged and every magnet and peer already relying on it keeps
+  working. The result distinguishes "added" from "the engine has taken it", which are different
+  claims and the second is the one that matters.
+
+### 🐞 Bug fixes
+
 ## 0.43.0
 ### ✨ Features and improvements
 - **Six detail tabs are now five, and the two bars that matter lead the General tab.** How much of an

package/docs/publishing.md

@@ -151,6 +151,22 @@ Adding from a **local path** never publishes a seed unless you ask: a web seed
 appears only when `webSeedBase` is set, and `publishDir` moves the file without
 implying one.
 
+### Adding one later
+
+An archive already in circulation can be given a web seed at any time, from the
+**HTTP sources** tab of its details or with
+`POST /api/torrents/<infohash>/webseeds`. This is safe on a published torrent:
+`url-list` sits outside the info dictionary, so the infohash does not change and
+every magnet, tracker and peer already relying on it keeps working.
+
+The URL has to serve the whole archive, byte for byte, under the name the torrent
+gives it. A seed that answers with anything else is not a faster path to the
+data — peers will fetch from it, fail the piece hash, and fall back to the swarm
+having wasted the transfer.
+
+Worth doing whenever an archive gets a public home after the fact: a swarm with
+one web seed starts cold in seconds rather than waiting for a peer to appear.
+
 ## Building from a feed of source data
 
 Subscribing is not only for finished archives. The OpenStreetMap project

package/docs/running-as-a-service.md

@@ -553,6 +553,36 @@ rather than exiting. Name the interpreter explicitly when the service user's
 { "libtorrent": { "python": "/usr/bin/python3" } }
 ```
 
+### A newer libtorrent than the distribution ships
+
+`apt` gives you whatever your release froze on, which on Ubuntu 24.04 is
+libtorrent 2.0.x. `pip install libtorrent` into the system Python is refused —
+`externally-managed-environment`, because apt owns that Python — and
+`--break-system-packages` is not the way round it.
+
+Give the service account a virtualenv instead and point the sidecar at it. The
+same `python` key does the work, so nothing else changes:
+
+```sh
+sudo -u pmtiles-swarm python3 -m venv /var/lib/pmtiles-swarm/venv
+sudo -u pmtiles-swarm /var/lib/pmtiles-swarm/venv/bin/pip install libtorrent
+sudo -u pmtiles-swarm /var/lib/pmtiles-swarm/venv/bin/python   -c "import libtorrent; print(libtorrent.__version__)"
+```
+
+```json
+{ "libtorrent": { "python": "/var/lib/pmtiles-swarm/venv/bin/python" } }
+```
+
+Reversible in one line: delete the key and the node is back on the distribution
+package, with no uninstall and nothing done to apt. `ReadWritePaths` already
+covers `/var/lib/pmtiles-swarm`, so the venv needs no unit change — put it
+anywhere else and it does.
+
+Worth doing for a node seeding very large hybrid archives, where v2 support has
+had the most work since 2.0. Change one thing at a time, though: take a version
+of the node first, confirm what it did, then move libtorrent, so a difference in
+behaviour has one candidate rather than two.
+
 ## Ports
 
 Five listeners, and only the peer ports want a firewall rule. See
@@ -615,6 +645,22 @@ sudo -u pmtiles-swarm -H npm ls --prefix /var/lib/pmtiles-swarm --depth 1 \
   pmtiles-swarm pmtiles-torrent
 ```
 
+**libtorrent itself is a third thing, and an update never touches it.** The
+sidecar is Python that ships with the package; libtorrent is the binding it
+imports, which came from `apt` or from a virtualenv you made. Neither moves when
+the node does:
+
+```sh
+sudo -u pmtiles-swarm python3 -c "import libtorrent; print(libtorrent.__version__)"
+```
+
+Run that against whatever `libtorrent.python` names, not your login shell's
+Python — they are frequently different interpreters, and the one that matters is
+the one the service uses. See
+[a newer libtorrent than the distribution ships](#a-newer-libtorrent-than-the-distribution-ships)
+for moving it, and move it on its own: a node upgrade and a libtorrent upgrade in
+one restart leaves two candidates for whatever happens next.
+
 An install runs the dependency install scripts again, so check WebRTC survived
 it — see [the allowScripts warning](#the-allowscripts-warning):
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.43.0",
+  "version": "0.45.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -46,7 +46,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.9.0",
+    "pmtiles-torrent": "^0.9.1",
     "webtorrent": "^3.0.21"
   },
   "engines": {

package/src/web/index.html

@@ -1728,12 +1728,63 @@
           if (name === 'sources') {
             const seeds = entry.webSeeds ?? [];
             pane.innerHTML =
-              table(['URL'], seeds.map((url) => [escapeHtml(url)])) +
-              `<div class="sub" style="margin-top:0.5rem">
+              table(
+                ['URL'],
+                seeds.map((url) => [escapeHtml(url)]),
+              ) +
+              `<div style="display:flex; gap:0.4rem; margin-top:0.75rem">
+                 <input
+                   id="seed-url"
+                   class="grow"
+                   placeholder="https://example.org/${escapeHtml(entry.name)}"
+                   spellcheck="false"
+                 />
+                 <button id="seed-add">Add</button>
+               </div>
+               <div class="sub" style="margin-top:0.5rem">
                  A web seed serves the archive over plain HTTP, which is what
                  makes it usable before it has peers — and the difference
-                 between a cold tile taking tens of seconds and under one.
+                 between a cold tile taking tens of seconds and under one. One
+                 can be added at any time: <code>url-list</code> sits outside
+                 the info dictionary, so the infohash is unchanged and every
+                 magnet and peer already relying on it keeps working. The URL
+                 must serve the whole archive, byte for byte, under the name
+                 the torrent gives it.
                </div>`;
+
+            // Attached here rather than in the panel, because this pane is
+            // rebuilt whenever it is shown and the handler has to survive that.
+            // Safe: sources is not one of the panes the refresh redraws.
+            const add = async () => {
+              const url = pane.querySelector('#seed-url').value.trim();
+              if (!url) return;
+              const button = pane.querySelector('#seed-add');
+              button.disabled = true;
+              try {
+                const result = await api(
+                  `/api/torrents/${infoHash}/webseeds`,
+                  { method: 'POST', body: { webSeeds: [url] } },
+                );
+                // "Added" and "the engine is using it" are different claims,
+                // and the second is the one somebody adding a seed wants.
+                toast(
+                  result.live
+                    ? 'web seed added and live'
+                    : 'web seed added; the engine has not taken it yet',
+                );
+                renderDetail(infoHash);
+              } catch (error) {
+                toast(error.message);
+                button.disabled = false;
+              }
+            };
+
+            pane.querySelector('#seed-add').onclick = add;
+            pane.querySelector('#seed-url').onkeydown = (event) => {
+              // Enter, because a single field with a button beside it is a
+              // form in everything but name.
+              if (event.key === 'Enter') add();
+            };
             return;
           }
 
@@ -1976,7 +2027,6 @@
           <div class="actions">
             ${servable ? '<button id="warm">Warm region</button>' : ''}
             <button id="set-location">Set location…</button>
-            <button id="add-seed">Add web seed</button>
             <button id="clear-cache" ${mode === 'cache' ? '' : 'disabled title="only cache-mode archives have a cache to clear"'}>Clear cache</button>
             <button id="recheck" title="Hash the files on disk again and believe the result. For an archive that reads 0% next to a file that is plainly there, or one that claims to be complete and is not — every other figure comes from something written down earlier, and this is the only thing that goes and looks.">Recheck files</button>
             <button id="pause">${entry.paused ? 'Resume' : 'Pause'}</button>
@@ -2248,24 +2298,6 @@
           };
         };
 
-        $('add-seed').onclick = async () => {
-          const url = window.prompt(
-            'Web seed URL (adding one does not change the infohash):',
-            `https://example.org/${entry.name}`,
-          );
-          if (!url) return;
-          try {
-            const result = await api(`/api/torrents/${infoHash}/webseeds`, {
-              method: 'POST',
-              body: { webSeeds: [url] },
-            });
-            toast(result.live ? 'web seed added and live' : 'web seed added');
-            renderDetail(infoHash);
-          } catch (error) {
-            toast(error.message);
-          }
-        };
-
         $('clear-cache').onclick = async () => {
           if (!window.confirm('Discard this archive’s cached pieces?')) return;
           try {