pmtiles-swarm 0.17.0 → 0.18.0

package/CHANGELOG.md

@@ -2,10 +2,28 @@
 
 ## master
 ### ✨ Features and improvements
+- **A watched folder can ask for its stable name to be a hard link.**
+  `latestLinkType: "hard"` reverses the order the two kinds are attempted in,
+  for a folder whose archives are read *through* that name rather than followed
+  to see where it points. A hard link still resolves after the build it names
+  is retired; a symlink is left pointing at nothing, which a tile endpoint
+  reports as a missing archive while the bytes are still on the disk. The other
+  kind remains the fallback in both directions, and the log now always says
+  which was made rather than only mentioning the unexpected one.
+- **The default trackers now include WebSocket ones.** They were two `udp://` entries, so an
+  archive created with stock configuration was undiscoverable from a browser — healthy in a
+  desktop client, invisible from a page, with nothing in either to say why. A browser speaks
+  WebRTC only and has no DHT, PeX or local discovery to fall back on, so `wss://` is not
+  redundancy with the rest of the list, it is the whole of that path. Two are listed because it
+  has no backstop. Every entry was checked for a completed handshake before being added.
 - _...Add new stuff here..._
 
 ### 🐞 Bug fixes
-- _...Add new stuff here..._
+- **Retention no longer reaches across a folder's other entries.** A watched folder's family was
+  built from the directory alone, so several entries sharing one — which `match` exists to make
+  possible — were treated as a single family. With `keep: 1`, importing this week's `monthly`
+  retired `10yrplus` and deleted its data. The family is now scoped by the entry's glob as well
+  as its path. Introduced in 0.17.0; anything affected must be regenerated.
 
 ## 0.17.0
 ### ✨ Features and improvements

package/docs/engines.md

@@ -285,31 +285,31 @@ load becomes swarm capacity instead of passing through the same pipe.
 This is a configuration requirement, not an automatic one, and it is easy to miss because
 everything looks healthy without it.
 
-The default trackers are UDP:
+The default list carries both kinds:
 
 ```json
 "trackers": [
   "udp://tracker.opentrackr.org:1337/announce",
-  "udp://tracker.torrent.eu.org:451/announce"
+  "udp://tracker.torrent.eu.org:451/announce",
+  "udp://tracker-udp.gbitt.info:80/announce",
+  "wss://tracker.openwebtorrent.com",
+  "wss://tracker.webtorrent.dev"
 ]
 ```
 
-**A browser cannot use either of those, and cannot use the DHT.** Both need UDP sockets and raw
+**A browser cannot use the UDP ones, and cannot use the DHT.** Both need UDP sockets and raw
 TCP, which a page does not have. A browser's only route to discovery is a `wss://` tracker — so a
 torrent announcing to UDP trackers alone is one no browser can find a peer for, however many
 WebTorrent nodes are seeding it. WebTorrent's own default WebSocket trackers do not fill this gap
 either: they are compiled into its **browser** bundle, and a Node client adds nothing.
 
-To actually reach browser peers, announce to at least one WebSocket tracker as well:
+Two `wss://` entries are listed rather than one because that path has no backstop: when the UDP
+side loses a tracker there are still the others, the DHT, PeX and local discovery, and when the
+WebSocket side loses one a browser has nothing left.
 
-```json
-"trackers": [
-  "udp://tracker.opentrackr.org:1337/announce",
-  "udp://tracker.torrent.eu.org:451/announce",
-  "wss://tracker.openwebtorrent.com",
-  "wss://tracker.webtorrent.dev"
-]
-```
+If you replace this list, keep at least one `wss://` entry — and check it answers, rather than
+copying one from a list. Public WebSocket trackers come and go, and a dead one fails in exactly
+the way this section describes: silently, and only for browsers.
 
 Trackers live outside the torrent's `info` dictionary, so adding them **does not change the
 infohash** — but it only applies to torrents created after the change. An existing archive keeps

package/docs/publishing.md

@@ -377,6 +377,24 @@ A new `.pmtiles` appearing in a watched folder is imported automatically.
 
 Editable in **Settings → Monitored folders** as a table, rather than by hand.
 
+### What kind of link the stable name is
+
+`latestLink` is made as a symlink by default. Set `latestLinkType: "hard"` on
+the folder when something reads the archive *through* that name — a tile
+server, or anything else that opens it rather than following it to see where it
+points.
+
+The difference only shows when the build the name refers to goes away. A hard
+link is another name for the same bytes, so it still resolves; a symlink is
+left pointing at nothing. Retention repoints the name as it retires a build, so
+neither case arises from ordinary housekeeping — but a rebuild that fails
+between the two steps leaves a reader working in one arrangement and broken in
+the other.
+
+The other kind stays the fallback either way: Windows refuses symlinks without
+elevation, and a hard link cannot cross a filesystem, so the name is still made
+whichever was asked for. The log says which it got.
+
 ### One folder, several kinds of build
 
 Categories and retention are decided per entry, not per directory. A generator

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.17.0",
+  "version": "0.18.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/config.js

@@ -236,10 +236,27 @@ const DEFAULTS = {
    *
    * Only applies to torrents created here. Joining an existing one uses the
    * trackers that torrent already carries.
+   *
+   * The `wss://` entries are not redundancy with the others — they are the
+   * only ones a browser can use. A page speaks WebRTC and nothing else: it
+   * cannot reach a UDP or HTTP tracker, and it has no DHT, PeX or local
+   * discovery to fall back on. An archive announced only to `udp://` trackers
+   * is perfectly healthy in a desktop client and invisible from a browser,
+   * with nothing in either to say why. Two are listed because that single
+   * path has no backstop.
+   *
+   * Keep this list short and answering. A tracker that no longer resolves
+   * costs an announce attempt per interval per torrent and finds nobody —
+   * unlike one that is merely unreachable from here, which can still
+   * introduce peers on networks that can reach it, since the list travels
+   * with the torrent.
    */
   trackers: [
     'udp://tracker.opentrackr.org:1337/announce',
     'udp://tracker.torrent.eu.org:451/announce',
+    'udp://tracker-udp.gbitt.info:80/announce',
+    'wss://tracker.openwebtorrent.com',
+    'wss://tracker.webtorrent.dev',
   ],
   /**
    * Copy every .torrent we create into this directory as well.
@@ -596,6 +613,15 @@ const DEFAULTS = {
    * ever touch archives this same folder imported, and never remove the
    * newest build however old it gets.
    *
+   * `latestLinkType` picks how `latestLink` is made: `symbolic` (the default),
+   * or `hard` for a name something reads the archive through. A hard link
+   * still resolves after the build it names is retired, where a symlink is
+   * left pointing at nothing — so a tile endpoint opening that name keeps
+   * working across a rebuild either way, and only one of them keeps working if
+   * the rebuild fails halfway. The other kind stays the fallback in both
+   * directions, since Windows refuses symlinks without elevation and a hard
+   * link cannot cross a filesystem.
+   *
    * `match` is a glob tested against the filename, for a folder holding more
    * than one kind of build. Categories are decided per entry, so several
    * entries can watch one folder and each take only its own archives:

package/src/latest-link.js

@@ -20,21 +20,35 @@ import path from 'node:path';
  * this names is eventually deleted: a symlink is left dangling, while a hard
  * link keeps the bytes alive until it too is gone. Retention never removes the
  * newest build, so neither case arises from this node's own housekeeping.
+ *
+ * `type` chooses which is preferred, because which one is better depends on
+ * what opens the name. Something reading the archive through this link — a
+ * tile endpoint, say — is better served by a hard link: if the build it names
+ * is removed before the name is repointed, a hard link still resolves to the
+ * bytes while a symlink resolves to nothing. The other order is the default
+ * only because it is what has always happened here.
  * @param {object} options - What to link and how to say so.
  * @param {string} options.target - The build the name should resolve to.
  * @param {string} options.name - The stable name, absolute or beside the target.
  * @param {string} options.label - How to name the caller in the log.
+ * @param {string} [options.type] - 'symbolic' (default), 'hard', or 'auto'.
  * @returns {Promise<string|undefined>} - The link made, or undefined on failure.
  */
-export async function linkLatest({ target, name, label }) {
+export async function linkLatest({ target, name, label, type = 'symbolic' }) {
   const link = linkPathFor(target, name);
 
-  const attempts = [
+  const symbolic = [
     // The type is autodetected from the target on Windows and ignored
     // everywhere else, which is right: the target is always a file here.
-    ['symlink', () => fs.symlink(target, link)],
-    ['hard link', () => fs.link(target, link)],
+    'symlink',
+    () => fs.symlink(target, link),
   ];
+  const hard = ['hard link', () => fs.link(target, link)];
+
+  // Both orders keep the other as a fallback rather than failing outright:
+  // Windows refuses symlinks without elevation, and a hard link cannot cross a
+  // filesystem. Whichever is asked for, the name still gets made.
+  const attempts = type === 'hard' ? [hard, symbolic] : [symbolic, hard];
 
   for (const [kind, make] of attempts) {
     try {
@@ -42,10 +56,10 @@ export async function linkLatest({ target, name, label }) {
       // per build and the name by definition already exists after the first.
       await fs.rm(link, { force: true });
       await make();
-      console.log(
-        `${label} latest -> ${path.basename(target)}` +
-          (kind === 'symlink' ? '' : ` (${kind})`),
-      );
+      // Always says which kind. Reporting only the unexpected one meant the
+      // common case was silent, and the only way to learn what you had was to
+      // list the directory and read the mode bits.
+      console.log(`${label} latest -> ${path.basename(target)} (${kind})`);
       return link;
     } catch (error) {
       // Try the next kind rather than giving up on the first refusal; only the

package/src/watch.js

@@ -134,7 +134,7 @@ export class WatchManager {
           return !this.#isLatestLink(file, entry);
         });
 
-        if (owner) this.#import(file, owner.entry);
+        if (owner) this.#import(file, owner.entry, owner.match);
       });
       watcher.on('error', (error) => {
         console.error(`[watch] ${directory}: ${error.message}`);
@@ -172,9 +172,10 @@ export class WatchManager {
    * Imports one archive, guarding against overlapping imports of the same file.
    * @param {string} file - Path to the archive.
    * @param {object} folder - The watch-folder configuration.
+   * @param {RegExp|null} [match] - The entry's filename filter, if it has one.
    * @returns {Promise<void>} - Resolves once imported or skipped.
    */
-  async #import(file, folder) {
+  async #import(file, folder, match = null) {
     if (this.#importing.has(file)) return;
     this.#importing.add(file);
     try {
@@ -213,6 +214,10 @@ export class WatchManager {
           target: entry.retainedAt ?? path.join(entry.savePath, entry.name),
           name: folder.latestLink,
           label: `[watch] ${folder.path}`,
+          // Per folder: a folder whose archives are opened through this
+          // name wants a hard link, one that is only a convenience for a
+          // person browsing the directory does not care.
+          type: folder.latestLinkType,
         });
       }
 
@@ -223,9 +228,20 @@ export class WatchManager {
       if (!retains(folder)) return;
       await retire({
         library: this.#library,
+        // Scoped by the entry, not just by the directory. Several entries
+        // describing one directory is the whole point of `match`, and a family
+        // built from the path alone puts every bucket in one: importing
+        // this week's `monthly` would retire `10yrplus` under keep:1, deleting
+        // an archive that has nothing to do with it. Re-applying the glob is
+        // what splits one directory back into the families the config
+        // describes.
         family: this.#library.catalog
           .list()
-          .filter((candidate) => candidate.source?.watch === folder.path),
+          .filter(
+            (candidate) =>
+              candidate.source?.watch === folder.path &&
+              (!match || match.test(path.basename(candidate.name ?? ''))),
+          ),
         entry,
         keep: folder.keep,
         keepDays: folder.keepDays,

package/src/web/index.html

@@ -3427,6 +3427,14 @@
               label: 'Stable name',
               placeholder: 'off',
             },
+            {
+              field: 'latestLinkType',
+              label: 'Stable name is a',
+              options: [
+                ['', 'symlink'],
+                ['hard', 'hard link'],
+              ],
+            },
             {
               field: 'keep',
               label: 'Builds to keep',