pmtiles-swarm 0.46.0 → 0.48.0

package/CHANGELOG.md

@@ -7,6 +7,50 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.48.0
+### ✨ Features and improvements
+- **An archive can now be read as a file, by byte range.** `GET /archives/<infohash>/archive.pmtiles`
+  answers `Accept-Ranges: bytes`, a `206` with `Content-Range` for a range and `416` for one it cannot
+  satisfy. Every PMTiles consumer there is — pmtiles.js, tileserver-gl, go-pmtiles, QGIS — reads one
+  file over HTTP that way, and until now this node offered tiles, which is a different protocol: using
+  it as an origin meant a copy of the file somewhere else. This is the file, at an address that does
+  not depend on knowing where the node keeps it.
+
+  Complete archives only. A partial file answers a range with whatever is at that offset, which for a
+  torrent's sparse allocation is zeroes — worse than a refusal, because it looks like data. An
+  incomplete archive answers `409` and says so.
+
+  It is also, by construction, a valid BEP 19 web seed: that specification is "an HTTP URL that serves
+  the file and honours Range". Publishing it as one is a separate decision and a later change — a node
+  is not obliged to offer 700 GiB to strangers because it can.
+
+### 🐞 Bug fixes
+
+## 0.47.1
+### ✨ Features and improvements
+
+### 🐞 Bug fixes
+- **The magnet button copies a magnet, not a link to one.** `/latest/<category>/magnet` answers a
+  magnet URI as `text/plain`, and both pages were copying that endpoint's own address — so what
+  landed on the clipboard was an `http://` URL, which a torrent client cannot open. The button now
+  fetches the endpoint and copies what it answers. Every other endpoint is still copied by address,
+  because for those the address is the thing.
+
+## 0.47.0
+### ✨ Features and improvements
+- **The console's Categories tab now matches the public page.** They were two views of one thing that
+  had drifted into two shapes: a label/value table with Copy and Open on every row here, links and a
+  printed style URL there, different labels for the same endpoints, and a preview link on one and not
+  the other.
+
+  Both now offer each endpoint the way it is used — `.torrent` downloads, preview opens, and
+  TileJSON, magnet, RSS and the style URL copy. The console keeps `RSS, newest only`, which is its
+  own, and gains the preview link it was missing. The style URL is no longer printed on either page:
+  here it was truncated to 96 characters, which is long enough to fill the row and too short to be
+  the thing anybody wanted.
+
+### 🐞 Bug fixes
+
 ## 0.46.0
 ### ✨ Features and improvements
 - **Each category endpoint on the public page is now offered the way it is used.** A `.torrent` is a

package/README.md

@@ -783,6 +783,7 @@ which the endpoint answers 501.
 | `GET`                 | `/health`                                                   | 200 when this node can serve, 503 when its engine cannot — **public**, no credential, for a load balancer                                                                                                                                 |
 | `GET`                 | `/archives/:infoHash/ready`                                 | Whether this node can serve _this_ archive yet: 200 ready, 503 not yet, 415 never — **public**                                                                                                                                            |
 | `GET`                 | `/archives/:infoHash/archive.torrent`                       | The `.torrent` a torrent-aware client joins with — **public**                                                                                                                                                                             |
+| `GET`                 | `/archives/:infoHash/archive.pmtiles`                       | The archive itself, by byte range — what any PMTiles reader wants, and a valid web seed. Complete archives only — **public**                                                                                                              |
 | `GET`                 | `/archives/:infoHash/preview`                               | Map preview for one archive — admin, not public                                                                                                                                                                                           |
 | `GET`                 | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is |
 | `GET`                 | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public**                                                                                                                                                                                                                          |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.46.0",
+  "version": "0.48.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -2429,6 +2429,68 @@ export function createApp({
     }),
   );
 
+  /**
+   * The archive itself, by byte range.
+   *
+   * Every PMTiles consumer there is — pmtiles.js, tileserver-gl, go-pmtiles,
+   * QGIS — reads one file over HTTP with Range requests. Until now this node
+   * offered tiles, which is a different protocol, so using it as an origin
+   * meant either its own tile endpoint or a copy of the file somewhere else.
+   * This is the file, at an address that does not depend on knowing where the
+   * node keeps it.
+   *
+   * It is also, by construction, a valid BEP 19 web seed: that specification
+   * is "an HTTP URL that serves the file and honours Range", which is exactly
+   * this. Publishing it is a separate decision and a later one — a node is not
+   * obliged to offer 700 GiB to strangers because it can.
+   *
+   * Complete archives only. A partial file would answer a range with whatever
+   * happened to be at that offset, which for a torrent's sparse allocation is
+   * zeroes: worse than a refusal, because it looks like data.
+   */
+  app.get('/archives/:infoHash/archive.pmtiles', (req, res) => {
+    const entry = catalog.get(req.params.infoHash);
+    if (!entry) return res.status(404).json({ error: 'not found' });
+
+    if (entry.complete === false) {
+      return res.status(409).json({
+        error:
+          'this archive is not complete here, so a byte range would answer ' +
+          'with unwritten space rather than data',
+      });
+    }
+    const file = entry.savePath ? path.join(entry.savePath, entry.name) : null;
+    if (!file) return res.status(404).json({ error: 'no file for it here' });
+
+    // sendFile does the range work: 206 with Content-Range, 416 for one that
+    // cannot be satisfied, HEAD, and the conditional headers a cache needs.
+    // Reimplementing that is how a subtly wrong Content-Range gets shipped.
+    res.sendFile(
+      file,
+      {
+        acceptRanges: true,
+        // A tile archive is immutable — it is addressed by the hash of its
+        // own contents, so a byte at an offset is the same byte for ever.
+        maxAge: '1y',
+        immutable: true,
+        headers: { 'content-type': 'application/octet-stream' },
+      },
+      (error) => {
+        if (!error || res.headersSent) return;
+        const missing = error.code === 'ENOENT';
+        // sendFile reports a range it cannot satisfy as a 416 on the error,
+        // which is an answer rather than a fault — passing it through as 500
+        // would tell a client its request was our problem.
+        const status = missing ? 404 : (error.status ?? 500);
+        res.status(status).json({
+          error: missing
+            ? 'the catalog has this archive but its file is not there'
+            : error.message,
+        });
+      },
+    );
+  });
+
   app.get(
     '/archives/:infoHash/archive.torrent',
     route(async (req, res) => {

package/src/web/index.html

@@ -224,6 +224,23 @@
       }
       .tag.on { background: var(--accent); color: #fff; border-color: var(--accent); }
       #add-error { color: var(--bad); min-height: 1.2rem; }
+      /* One row of endpoints per category, matching the public page. */
+      #category-list .links {
+        display: flex;
+        flex-wrap: wrap;
+        gap: 0.4rem 0.9rem;
+        align-items: center;
+      }
+      /* A copy button reads as a link's sibling, not as a form control. */
+      button.copy {
+        border: 1px solid var(--line);
+        border-radius: 999px;
+        background: none;
+        color: var(--muted);
+        font-size: 0.74rem;
+        padding: 0 0.45rem;
+      }
+      button.copy:hover { color: var(--fg); border-color: var(--fg); }
       /* Category endpoints. `th, td` is nowrap for the archive table, where a
          wrapped number is worse than a wide one — but here the cell holds a
          style URL and a paragraph explaining it, so nowrap pushed the table
@@ -3477,20 +3494,40 @@ Every piece is hashed against the ` +
         list.innerHTML = categories
           .map((entry, index) => {
             const newest = entry.newest;
-            const rows = [
-              ['TileJSON', entry.endpoints.tileJson],
-              ['For a style', entry.endpoints.styleUrl],
-              ['Torrent', entry.endpoints.torrent],
-              ['Magnet', entry.endpoints.magnet],
-              ['Feed', entry.endpoints.feed],
-              ['Latest only', entry.endpoints.latestFeed],
-            ];
+            // The same shape as the public page, and for the same reason: what
+            // an endpoint is for decides how it is offered. A `.torrent` is a
+            // file, so it downloads; a preview is a page, so it opens. The
+            // rest are addresses that belong somewhere else — a style, a
+            // torrent client, a feed reader — and following one from here
+            // achieves nothing, so they copy.
+            //
+            // The style URL is not printed. It is a TileJSON URL carrying a
+            // `.torrent` URL and a percent-encoded magnet in its fragment,
+            // several hundred characters of it, and it was truncated to 96
+            // here — long enough to fill the row and too short to be the
+            // thing anybody wanted.
+            const ends = entry.endpoints;
+            const link = (url, text, download) =>
+              url
+                ? `<a href="${escapeHtml(url)}"${download ? ' download' : ' target="_blank" rel="noreferrer"'}>${text}</a>`
+                : '';
+            // `fetched` for an endpoint whose *answer* is the thing worth
+            // having rather than its address. /magnet serves a magnet URI as
+            // text/plain, and copying the URL of that handed somebody a link
+            // to a magnet instead of a magnet — which a torrent client cannot
+            // open.
+            const copyable = (url, text, fetched) =>
+              url
+                ? `<button class="copy" data-copy-url="${escapeHtml(url)}"${
+                    fetched ? ' data-copy-fetch="1"' : ''
+                  } title="${escapeHtml(url)}">${text}</button>`
+                : '';
 
             return `
               <div class="panel" style="margin-bottom:0.8rem">
                 <h2 style="margin-top:0">
                   ${escapeHtml(entry.category)}
-                  <span class="sub">· ${entry.archives} archive${entry.archives === 1 ? '' : 's'}</span>
+                  <span class="sub">· ${entry.archives} build${entry.archives === 1 ? '' : 's'}</span>
                 </h2>
                 <div class="sub" style="margin-bottom:0.6rem">
                   ${
@@ -3499,60 +3536,44 @@ Every piece is hashed against the ` +
                       : 'nothing published under this tag yet'
                   }
                 </div>
-                <table>
-                  <tbody>
-                    ${rows
-                      .map(
-                        ([label, url], row) => `
-                          <tr>
-                            <td style="width:8rem">${label}</td>
-                            <td>${
-                              url
-                                ? `<code>${escapeHtml(
-                                    url.length > 96 ? `${url.slice(0, 96)}…` : url,
-                                  )}</code>${
-                                    label === 'For a style'
-                                      ? `<div class="sub">the .torrent URL and the magnet ride in the
-                                         fragment, which is never sent to the server — an ordinary
-                                         client fetches the TileJSON and ignores them, a swarm-aware
-                                         one reads them before making any call${
-                                           // Matched unanchored because the magnet is
-                                           // percent-encoded in the fragment now that it
-                                           // shares one with the .torrent URL.
-                                           url.includes('btpk')
-                                             ? ', and both follow the category rather than this build'
-                                             : ''
-                                         }</div>`
-                                      : ''
-                                  }`
-                                : '<span class="sub">not a PMTiles archive, so there is no tile endpoint</span>'
-                            }</td>
-                            <td style="width:11rem">${
-                              url
-                                ? `<button data-copy="${index}-${row}">Copy</button>
-                                   <a href="${escapeHtml(url)}" target="_blank" rel="noreferrer"><button type="button">Open</button></a>`
-                                : ''
-                            }</td>
-                          </tr>`,
-                      )
-                      .join('')}
-                  </tbody>
-                </table>
+                ${
+                  ends.tileJson
+                    ? `<div class="links">
+                         ${copyable(ends.tileJson, 'TileJSON')}
+                         ${link(ends.preview, 'preview')}
+                         ${link(ends.torrent, '.torrent', true)}
+                         ${copyable(ends.magnet, 'magnet', true)}
+                         ${copyable(ends.feed, 'RSS')}
+                         ${copyable(ends.latestFeed, 'RSS, newest only')}
+                         ${copyable(ends.styleUrl, 'style URL')}
+                       </div>
+                       <div class="sub" style="margin-top:0.5rem">
+                         The style URL is the TileJSON with the .torrent URL and
+                         the magnet in its fragment, which is never sent to the
+                         server — an ordinary client fetches the TileJSON and
+                         ignores them, a swarm-aware one reads them first, and
+                         both follow the category rather than this build.
+                       </div>`
+                    : '<div class="sub">not a PMTiles archive, so there is no tile endpoint</div>'
+                }
               </div>`;
           })
           .join('');
 
-        for (const button of list.querySelectorAll('[data-copy]')) {
-          const [index, row] = button.dataset.copy.split('-').map(Number);
-          const url = [
-            categories[index].endpoints.tileJson,
-            categories[index].endpoints.styleUrl,
-            categories[index].endpoints.torrent,
-            categories[index].endpoints.magnet,
-            categories[index].endpoints.feed,
-            categories[index].endpoints.latestFeed,
-          ][row];
-          button.onclick = () => copy(url, 'URL');
+        for (const button of list.querySelectorAll('[data-copy-url]')) {
+          button.onclick = async () => {
+            const url = button.dataset.copyUrl;
+            if (!button.dataset.copyFetch) return copy(url, 'URL');
+            try {
+              // A plain fetch, not `api`: that one parses JSON, and this
+              // endpoint answers text/plain.
+              const response = await fetch(url);
+              if (!response.ok) throw new Error(response.statusText);
+              copy((await response.text()).trim(), 'magnet');
+            } catch (error) {
+              toast(error.message);
+            }
+          };
         }
       }
 

package/src/web/public.html

@@ -408,7 +408,11 @@
           // preview is a page to visit. The rest are addresses that belong
           // somewhere else — a style, a torrent client, a feed reader — and
           // following one here achieves nothing, so those copy.
-          const copy = (href, text) => {
+          // `fetched` for an endpoint whose *answer* is the thing worth
+          // having rather than its address. /magnet serves a magnet URI as
+          // text/plain, and copying the URL of that handed somebody a link to
+          // a magnet instead of a magnet — which a torrent client cannot open.
+          const copy = (href, text, fetched) => {
             const button = el('button', 'copy', text);
             button.type = 'button';
             const absolute = new URL(href, location.href).href;
@@ -418,7 +422,10 @@
             button.title = absolute;
             button.onclick = async () => {
               try {
-                await navigator.clipboard.writeText(absolute);
+                const value = fetched
+                  ? (await (await fetch(absolute)).text()).trim()
+                  : absolute;
+                await navigator.clipboard.writeText(value);
                 button.classList.add('done');
                 button.textContent = 'copied';
                 setTimeout(() => {
@@ -444,7 +451,7 @@
             a.download = '';
             links.append(a);
           }
-          if (ends.magnet) copy(ends.magnet, 'magnet');
+          if (ends.magnet) copy(ends.magnet, 'magnet', true);
           if (ends.feed) copy(ends.feed, 'RSS');
           // The one worth copying and the one nobody wants to read: a TileJSON
           // URL carrying the .torrent URL and a percent-encoded magnet in its