pmtiles-swarm 0.16.0 → 0.17.0

package/CHANGELOG.md

@@ -7,6 +7,32 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.17.0
+### ✨ Features and improvements
+- **A monitored folder can filter by filename.** `match` takes a glob, so one directory can be
+  described by several entries and each take only its own archives — which is what a generator
+  writing `monthly-20260813.pmtiles` and `10yrplus-20260813.pmtiles` side by side needs, since
+  categories and retention are decided per entry. Without it every entry claimed every archive,
+  and because imports are deduplicated by path the file landed under whichever entry won the
+  race: not a duplicate, which would at least have been visible, but one import under an
+  arbitrary category.
+
+### 🐞 Bug fixes
+- **Entries sharing a directory now share one watcher.** Each previously started its own, and two
+  chokidar instances over one path made ownership a race. They are now grouped, so the first
+  entry whose `match` accepts a name takes it — decided by the order they appear in the config.
+
+## 0.16.1
+### 🐞 Bug fixes
+- **Client addresses are recorded without the IPv4-mapped prefix.** A dual-stack listener reports
+  IPv4 peers as `::ffff:172.16.1.2`, so the Traffic tab showed an address nobody types — and
+  would have counted one client twice had it reached the node over both stacks.
+
+  Only a display and counting matter: `trustProxy` matching is unaffected, since Express compares
+  mapped addresses against plain IPv4 entries correctly. Verified rather than assumed, because
+  `"trustProxy": "172.16.1.2, 172.16.1.3"` looking like it should not match a `::ffff:` socket is
+  exactly the sort of thing that would have been quietly wrong.
+
 ## 0.16.0
 ### ✨ Features and improvements
 - **A Traffic tab in the console**, which is where the statistics added in 0.10.0 should have been

package/README.md

@@ -272,7 +272,9 @@ Three ways in, all editable from the Settings screen:
 - **Monitored folders** (`watch`) — a directory scanned for new archives, the way a torrent client
   watches for `.torrent` files. `publishDir` moves each one into the directory a web server serves
   before the torrent is built, and `webSeedBase` is the URL that directory answers on, so a
-  brand-new archive has a working web seed before any peer holds a copy.
+  brand-new archive has a working web seed before any peer holds a copy. `match` is a filename
+  glob, for a generator that writes several kinds of build into one directory: give the folder an
+  entry per kind, each matching its own names, and each gets its own categories and retention.
 - **A URL template** (`sources[].url`) — for an upstream that publishes at a predictable address,
   like `https://build.protomaps.com/{YYYYMMDD}.pmtiles`. Expanded per candidate date and probed
   with a `HEAD`. The more reliable of the two web options: it asks a direct question, and needs the

package/docs/publishing.md

@@ -377,6 +377,44 @@ A new `.pmtiles` appearing in a watched folder is imported automatically.
 
 Editable in **Settings → Monitored folders** as a table, rather than by hand.
 
+### One folder, several kinds of build
+
+Categories and retention are decided per entry, not per directory. A generator
+that writes more than one kind of archive into one place therefore needs an
+entry per kind, and `match` — a filename glob — is what tells them apart:
+
+```json
+{
+  "watch": [
+    {
+      "path": "/srv/out/pmtiles",
+      "match": "monthly-*.pmtiles",
+      "categories": ["monthly"],
+      "latestLink": "monthly.pmtiles",
+      "keep": 1
+    },
+    {
+      "path": "/srv/out/pmtiles",
+      "match": "10yrplus-*.pmtiles",
+      "categories": ["10yrplus"],
+      "latestLink": "10yrplus.pmtiles",
+      "keep": 1
+    }
+  ]
+}
+```
+
+The glob is matched against the filename, is anchored at both ends — so
+`monthly-*` does not claim `cell_monthly-*` — and treats `.` and other regex
+punctuation as literal text. An entry with no `match` takes anything the
+entries above it have not already claimed, and the first entry whose glob
+accepts a name is the one that imports it, so config order decides.
+
+Leaving `match` off where several entries share a directory does not produce
+duplicates, which would at least be visible. Imports are deduplicated by path,
+so each archive is imported exactly once — under whichever entry got to it
+first, with nothing in the config or the log saying which.
+
 `comment` goes into every torrent this folder produces, and is where the attribution and
 licence belong — it is the one field a torrent carries that says what the thing is, and
 it reaches anyone who opens the file in any client:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.16.0",
+  "version": "0.17.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/config.js

@@ -595,6 +595,12 @@ const DEFAULTS = {
    * that it takes the torrent with the data. Both are off unless set, only
    * ever touch archives this same folder imported, and never remove the
    * newest build however old it gets.
+   *
+   * `match` is a glob tested against the filename, for a folder holding more
+   * than one kind of build. Categories are decided per entry, so several
+   * entries can watch one folder and each take only its own archives:
+   * `{ path: '/out/pmtiles', match: 'monthly-*.pmtiles', categories: ['monthly'] }`.
+   * Without it every entry imports every archive, once under each category.
    */
   watch: [],
   /**

package/src/tile-stats.js

@@ -103,6 +103,12 @@ export class TileStats {
     const { infoHash, z, status } = entry;
     if (!infoHash) return;
 
+    // A dual-stack listener reports IPv4 peers as ::ffff:172.16.1.2, which is
+    // the same client written two ways -- so counting it as written would
+    // split one address across two entries the moment anything reached the
+    // node over plain IPv4 as well.
+    const ip = entry.ip ? entry.ip.replace(/^::ffff:/, '') : entry.ip;
+
     let archive = this.#archives.get(infoHash);
     if (!archive) {
       archive = {
@@ -135,20 +141,20 @@ export class TileStats {
     if (Number.isFinite(status)) {
       archive.byStatus.set(status, (archive.byStatus.get(status) ?? 0) + 1);
     }
-    if (entry.ip) {
+    if (ip) {
       // Counted rather than listed: a busy node sees a handful of distinct
       // sources — the proxy, a few LAN clients — and the count is the answer
       // to "is this arriving directly or through HAProxy".
-      const seen = archive.clients.get(entry.ip) ?? { requests: 0, bytes: 0 };
+      const seen = archive.clients.get(ip) ?? { requests: 0, bytes: 0 };
       seen.requests += 1;
       seen.bytes += bytes;
-      archive.clients.set(entry.ip, seen);
+      archive.clients.set(ip, seen);
     }
 
     if (this.#limit === 0) return;
     const row = {
       at: archive.lastSeen,
-      ip: entry.ip ?? null,
+      ip: ip ?? null,
       infoHash,
       name: archive.name ?? null,
       z: Number.isInteger(z) ? z : null,

package/src/watch.js

@@ -14,6 +14,30 @@ import { retains, retire } from './retention.js';
  * deliberately generous, because a stalled network copy can pause for a long
  * time mid-file.
  */
+/**
+ * Compiles a shell-style glob into an anchored regular expression.
+ *
+ * Deliberately a glob and not a regular expression: a watch folder's filter is
+ * something an operator writes in a JSON config beside a filename, and
+ * `monthly-*.pmtiles` is what they mean. Every other character is escaped, so
+ * a name containing regex punctuation matches itself rather than becoming a
+ * pattern by accident.
+ *
+ * @param {string} pattern - A glob, matched against the basename.
+ * @returns {RegExp} - Anchored, case-insensitive.
+ */
+export function globToRegExp(pattern) {
+  const body = pattern
+    .split('')
+    .map((character) => {
+      if (character === '*') return '.*';
+      if (character === '?') return '.';
+      return character.replace(/[.+^${}()|[\]\\]/g, '\\$&');
+    })
+    .join('');
+  return new RegExp(`^${body}$`, 'i');
+}
+
 export class WatchManager {
   #library;
   #watchers = [];
@@ -33,8 +57,33 @@ export class WatchManager {
    * @returns {void}
    */
   start(folders = []) {
+    // One watcher per directory, not per entry. Several entries describing one
+    // directory is the normal way to give each build its own category, and two
+    // chokidar instances over the same path do not reliably both report a
+    // file — so an archive could go unimported with nothing to say why.
+    // Grouping also makes the choice of entry deterministic: the first whose
+    // `match` accepts a name takes it, decided by config order rather than by
+    // whichever watcher fired first.
+    const groups = new Map();
     for (const folder of folders) {
-      const stability = (folder.stabilitySeconds ?? 30) * 1000;
+      if (!groups.has(folder.path)) groups.set(folder.path, []);
+      groups.get(folder.path).push(folder);
+    }
+
+    for (const [directory, entries] of groups) {
+      // The watcher is shared, so its settings have to suit every entry:
+      // the longest stability threshold any of them asked for, and polling if
+      // any of them needs it, at the shortest interval requested.
+      const folder = {
+        ...entries[0],
+        stabilitySeconds: Math.max(...entries.map((e) => e.stabilitySeconds ?? 30)),
+        pollSeconds: Math.min(
+          ...entries.map((e) => e.pollSeconds ?? 0).filter((s) => s > 0),
+          ...(entries.some((e) => (e.pollSeconds ?? 0) > 0) ? [] : [0]),
+        ),
+        path: directory,
+      };
+      const stability = folder.stabilitySeconds * 1000;
 
       // A local directory needs no interval: the filesystem says when
       // something lands and the archive is picked up as it appears. A network
@@ -63,27 +112,44 @@ export class WatchManager {
         },
       });
 
+      const matchers = entries.map((entry) => ({
+        entry,
+        // A directory holding several kinds of build needs one entry each,
+        // because categories are decided per entry. `match` is what tells them
+        // apart; an entry without one takes anything not already claimed.
+        match: entry.match ? globToRegExp(entry.match) : null,
+      }));
+
       watcher.on('add', (file) => {
         if (!/\.pmtiles$/i.test(file)) return;
-        // The folder's own `latestLink`, which is a .pmtiles in a watched
-        // folder like any other and would otherwise be imported as a second
-        // archive — a whole extra torrent for the same bytes under a name that
-        // changes every build. A hard link is indistinguishable from the file
-        // it names, so the name is the only thing that can tell them apart.
-        if (this.#isLatestLink(file, folder)) return;
-        this.#import(file, folder);
+
+        const name = path.basename(file);
+        const owner = matchers.find(({ entry, match }) => {
+          if (match && !match.test(name)) return false;
+          // The entry's own `latestLink`, a .pmtiles like any other that would
+          // otherwise be imported as a second archive — a whole extra torrent
+          // for the same bytes, under a name that changes every build. A hard
+          // link is indistinguishable from the file it names, so the name is
+          // the only thing that can tell them apart.
+          return !this.#isLatestLink(file, entry);
+        });
+
+        if (owner) this.#import(file, owner.entry);
       });
       watcher.on('error', (error) => {
-        console.error(`[watch] ${folder.path}: ${error.message}`);
+        console.error(`[watch] ${directory}: ${error.message}`);
       });
 
       this.#watchers.push(watcher);
-      const tags = normalizeCategories(folder);
-      console.log(
-        `[watch] watching ${folder.path}` +
-          (tags.length > 0 ? ` as "${tags.join('", "')}"` : '') +
-          (pollSeconds > 0 ? ` (polling every ${pollSeconds}s)` : ''),
-      );
+      for (const entry of entries) {
+        const tags = normalizeCategories(entry);
+        console.log(
+          `[watch] watching ${directory}` +
+            (entry.match ? ` matching ${entry.match}` : '') +
+            (tags.length > 0 ? ` as "${tags.join('", "')}"` : '') +
+            (pollSeconds > 0 ? ` (polling every ${pollSeconds}s)` : ''),
+        );
+      }
     }
   }
 

package/src/web/index.html

@@ -3408,6 +3408,11 @@
                 ...(config.locations ?? []).map((entry) => [entry.path, entry.name]),
               ],
             },
+            {
+              field: 'match',
+              label: 'Only files matching',
+              placeholder: 'everything',
+            },
             { field: 'publishDir', label: 'Publish to', placeholder: '/var/www/pmtiles' },
             { field: 'webSeedBase', label: 'Web seed base', placeholder: 'https://…/files' },
             {