pmtiles-swarm 0.3.1 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,79 @@
7
7
  ### 🐞 Bug fixes
8
8
  - _...Add new stuff here..._
9
9
 
10
+ ## 0.4.0
11
+ ### ✨ Features and improvements
12
+ - **A watched folder can set the torrent comment**, which is where attribution and licence belong —
13
+ it is the one field a torrent carries that says what the thing is, and it reaches anyone who
14
+ opens the file in any client. The setting was passed through from the start and offered nowhere,
15
+ so it could only be reached by editing the config by hand.
16
+ - **Feeds can be followed with the controls a torrent client gives them.** A subscription takes
17
+ one item per check by default, counting from the newest — `newest` raises the cap and `0` lifts
18
+ it — because a feed like the one OpenStreetMap publishes for the planet dumps lists five of them,
19
+ and taking the lot is four hundred gigabytes nobody asked for. `enabled: false` switches one feed
20
+ off without deleting it, `subscriptionsEnabled` switches off all of them at once, and both are in
21
+ the console alongside the check interval.
22
+ - **Watched folders can retire what they have outgrown**, with the `keep` and `keepDays` that
23
+ until now existed only on scheduled sources. A folder receiving a daily 137 GB planet build
24
+ fills any disk within the week, and the alternative was a `find -mtime +35` sweep in the
25
+ generation script — which deletes the file but leaves this node advertising a torrent for it,
26
+ so every peer that asks fails. Retirement takes the two together.
27
+ - **`keepDays` retires by age rather than by count**, on both watched folders and scheduled
28
+ sources, which is what a `find -mtime` sweep actually said. Set alongside `keep` the two are a
29
+ union: whichever rule says a build has to go, it goes. Neither removes the newest build however
30
+ old it is — a source that stops publishing would otherwise erase itself, and a last build going
31
+ stale is a thing to notice rather than a thing to fix by deleting it.
32
+ - **A watched folder can give the newest build a stable name**, with the `latestLink` that
33
+ until now existed only on scheduled sources — `planetiler-openmaptiles-latest.pmtiles`, the
34
+ `ln -sfn latest` a generation script used to run. Off unless set. The dated file stays the
35
+ real one and keeps its own torrent, and the link costs no extra space: a symlink where the
36
+ platform allows one, a hard link where it does not, since Windows refuses symlinks without
37
+ elevation or developer mode. The watcher ignores that one name — a hard link is
38
+ indistinguishable from the file it names, so without that it would be imported as a second
39
+ archive of bytes already being seeded.
40
+ - **The torrent link for a category can be named whatever reads best.**
41
+ `/latest/openmaptiles/planetiler-openmaptiles-latest.torrent` is the same route as
42
+ `/latest/openmaptiles/archive.torrent` — fine in an API, poor in an href on a page. The name
43
+ is cosmetic: the redirect still ends at the immutable URL, which names the download after the
44
+ build it actually is, because a URL that could choose that would be a link on your own domain
45
+ that saves a file called anything at all.
46
+
47
+ ### 🐞 Bug fixes
48
+ - **`/archives/<infohash>/archive.torrent` says it can be cached.** An infohash names those
49
+ bytes and no others, so the URL can never answer differently — the tile routes have always
50
+ said so and this one did not, which meant a cache or reverse proxy in front of a node had to
51
+ re-fetch every download from it. `/latest/<category>/archive.torrent` gets a short one, since
52
+ it moves on every build.
53
+ - **A `subscriptionIntervalSeconds` of zero no longer polls as fast as the event loop allows.**
54
+ Zero reads as off everywhere else in the configuration; here it reached `setInterval` unchanged,
55
+ which is not a stopped timer.
56
+
57
+ ### 📚 Documentation
58
+ - **Following someone else's RSS feed**, using the one OpenStreetMap publishes for the planet
59
+ dumps as the worked example, including handing what lands to a generation script with
60
+ `onComplete`.
61
+ - **Piece size and network equipment.** Larger pieces are widely assumed to be gentler on a
62
+ router and mostly are not: peers request 16 KiB blocks whatever the piece size, so packet
63
+ volume for the same bytes is identical. Simultaneous connections are what exhaust a NAT table,
64
+ and `maxConnections` is the setting for that.
65
+
66
+ ## 0.3.2
67
+ ### ✨ Features and improvements
68
+ - **The console has a footer naming the version it is running**, beside `© <year> TechIdiots LLC`
69
+ and a link to the source. The version comes from `package.json` through `/api/status` rather than
70
+ being written into the page, since the number on screen is the one somebody quotes when reporting
71
+ a problem.
72
+
73
+ ### 📚 Documentation
74
+ - **How to update an installed service**, which was missing: reinstall into the same prefix and
75
+ restart. The restart is not optional — the Python sidecar is started with the process and lives
76
+ as long as it does, so a new one sits on disk doing nothing until then, and most of what changes
77
+ between releases is in there.
78
+ - The WebRTC check given in two places imported a directory path, which ESM refuses whatever the
79
+ state of the install — so it reported a failure that was never about WebRTC. It now imports
80
+ `node-datachannel` by name from the install directory, which is the binary the install script
81
+ fetches.
82
+
10
83
  ## 0.3.1
11
84
 
12
85
  Depends on pmtiles-torrent 0.3.0, which is what carries the resume-data fix below to an installed
package/README.md CHANGED
@@ -614,7 +614,7 @@ matters there is `maxConnections`, since every peer holds a NAT table entry. See
614
614
  | `GET` | `/archives/:infoHash/:z/:x/:y.:ext` | One tile — **public** |
615
615
  | `GET` | `/archives/:infoHash/archive.torrent` | The `.torrent` a torrent-aware client joins with — **public** |
616
616
  | `GET` | `/archives/:infoHash/preview` | Map preview for one archive — admin, not public |
617
- | `GET` | `/latest/:category/tiles.json`, `/archive.torrent`, `/magnet` | The newest build in a category — **public** |
617
+ | `GET` | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is |
618
618
  | `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public** |
619
619
 
620
620
  Everything under `/api/` is guarded once a credential is configured; tiles, TileJSON and the feeds
@@ -377,6 +377,16 @@ A new `.pmtiles` appearing in a watched folder is imported automatically.
377
377
 
378
378
  Editable in **Settings → Monitored folders** as a table, rather than by hand.
379
379
 
380
+ `comment` goes into every torrent this folder produces, and is where the attribution and
381
+ licence belong — it is the one field a torrent carries that says what the thing is, and
382
+ it reaches anyone who opens the file in any client:
383
+
384
+ ```json
385
+ {
386
+ "comment": "Planetiler OpenMapTiles export. OpenStreetMap contributors, under ODbL 1.0. OpenMapTiles under BSD 3-Clause / CC-BY 4.0"
387
+ }
388
+ ```
389
+
380
390
  `stabilitySeconds` is the important one. A map build writes its output over minutes or
381
391
  hours, and hashing a half-written archive produces a torrent for bytes that no longer
382
392
  exist. Nothing is imported until the file has stopped changing for that long. Raise it if
@@ -453,8 +463,23 @@ today's URL and see whether it exists yet:
453
463
  loses that build permanently.
454
464
 
455
465
  `latestLink`
456
- A symlink pointing at the newest build. The dated file stays the real one, so it
457
- remains seedable under its own torrent while consumers can reference a fixed path.
466
+ A stable name pointing at the newest build `planetiler-protomaps-latest.pmtiles`.
467
+ The dated file stays the real one, so it remains seedable under its own torrent while
468
+ a page links to a name that does not change. **Off unless set.**
469
+
470
+ A symlink where the platform allows one, and a hard link where it does not: Windows
471
+ refuses symlinks with `EPERM` unless the process is elevated or the machine is in
472
+ developer mode, which is not a reasonable thing to require of a daemon. Neither costs
473
+ extra space — both are another name for the same bytes rather than a copy, which for a
474
+ 137 GB archive is the whole point — and both need the name and the build to be on one
475
+ filesystem, which a link beside the file it names always is.
476
+
477
+ Available on watched folders too, where a generation script writes the builds. There
478
+ the link lands in the folder being watched, so the watcher is taught to ignore that one
479
+ name — a hard link is indistinguishable from the file it names, and without that it
480
+ would be imported as a second archive of bytes already being seeded. It follows that
481
+ **the name must not collide with a real build's**, and that changing it leaves the old
482
+ one behind as a file the watcher will then import.
458
483
 
459
484
  `at`, `everyHours`, `everyMinutes`
460
485
  When to look. `at` is a time of day in UTC — `"03:30"`, or a list of them — for an
@@ -507,6 +532,23 @@ today's URL and see whether it exists yet:
507
532
  `newest` is above 1, because a poll takes candidates newest first and the *last* build
508
533
  imported is then the oldest of them.
509
534
 
535
+ `keepDays`
536
+ The same thing said as a window rather than a count: how many days old a build may
537
+ get. `keepDays: 35` is the `find -mtime +35` sweep such a script would otherwise run
538
+ itself, except that this takes the torrent with the data instead of leaving the node
539
+ advertising an archive that is gone.
540
+
541
+ Age is read from the date in the build's name where there is one, and from when this
542
+ node took it otherwise. A build that can be dated neither way is never removed — a
543
+ guess is not good enough to delete several hundred gigabytes on.
544
+
545
+ **The newest build is never removed, however old it is.** A source that stops
546
+ publishing would otherwise erase itself, and a last build going stale is a thing to
547
+ notice rather than a thing to fix by deleting it.
548
+
549
+ Set alongside `keep` the two are a union: whichever rule says a build has to go, it
550
+ goes. `keep: 10, keepDays: 35` holds at most ten builds and at most five weeks.
551
+
510
552
  `seeding`
511
553
  A seeding limit for this source's builds, in the same shape as the global one:
512
554
  `{ "ratio": 2, "minutes": 4320, "then": "stop" }`. Useful where one source's archives
@@ -520,6 +562,27 @@ what you want — old builds stay seedable for as long as anyone still wants the
520
562
  Every candidate URL is checked with a HEAD, so a build that has not been published yet
521
563
  costs one request.
522
564
 
565
+ ### Retention on a watched folder
566
+
567
+ `keep` and `keepDays` work the same way on a watched folder, where builds arrive from a
568
+ generation script rather than from a URL:
569
+
570
+ ```json
571
+ {
572
+ "watch": [
573
+ {
574
+ "path": "/mnt/store/generated/openmaptiles/pmtiles",
575
+ "categories": ["openmaptiles", "planet"],
576
+ "keepDays": 35
577
+ }
578
+ ]
579
+ }
580
+ ```
581
+
582
+ The family there is the folder rather than a named source: only archives this same folder
583
+ imported are ever considered, so one dropped into the same directory by hand — or moved
584
+ elsewhere by `publishDir` and belonging to a different folder — is not caught up in it.
585
+
523
586
  ### Watching a directory instead
524
587
 
525
588
  Where the naming is not predictable enough to write as a template, give an `index` — a
@@ -104,8 +104,8 @@ the prebuilt binary WebTorrent needs for WebRTC, and it is not in the published
104
104
  tarball. Check it landed:
105
105
 
106
106
  ```sh
107
- sudo -u pmtiles-swarm -H node -e \
108
- "import('/var/lib/pmtiles-swarm/node_modules/webtorrent').then(() => console.log('ok'))"
107
+ cd /var/lib/pmtiles-swarm && sudo -u pmtiles-swarm -H node -e \
108
+ "import('node-datachannel').then(() => console.log('webrtc ok'))"
109
109
  ```
110
110
 
111
111
  `npm approve-scripts node-datachannel` records the approval in `package.json`
@@ -243,6 +243,43 @@ Four listeners, and only the peer ports want a firewall rule. See
243
243
  | `port` — 8090 | your proxy or CDN |
244
244
  | `adminPort` — 8091, bound to `127.0.0.1` | nothing; that is the point |
245
245
 
246
+ ## Updating
247
+
248
+ ```sh
249
+ sudo -u pmtiles-swarm -H npm install --prefix /var/lib/pmtiles-swarm pmtiles-swarm@latest
250
+ sudo systemctl restart pmtiles-swarm
251
+ ```
252
+
253
+ **The restart is not optional.** The Python sidecar is started with the process
254
+ and lives as long as it does, so a new sidecar sits on disk doing nothing until
255
+ the service is restarted. Most of what changes between releases is in there.
256
+
257
+ Nothing under `/etc/pmtiles-swarm` is touched, and restarting does not re-check
258
+ the archives: a clean stop writes resume data, and `TimeoutStopSec` above
259
+ leaves room for it.
260
+
261
+ Confirm both halves moved, since the sidecar has its own version:
262
+
263
+ ```sh
264
+ sudo -u pmtiles-swarm -H npm ls --prefix /var/lib/pmtiles-swarm --depth 1 \
265
+ pmtiles-swarm pmtiles-torrent
266
+ ```
267
+
268
+ An install runs the dependency install scripts again, so check WebRTC survived
269
+ it — see [the allowScripts warning](#the-allowscripts-warning):
270
+
271
+ ```sh
272
+ cd /var/lib/pmtiles-swarm && sudo -u pmtiles-swarm -H node -e \
273
+ "import('node-datachannel').then(() => console.log('webrtc ok'))"
274
+ ```
275
+
276
+ To pin a version, or to go back to one:
277
+
278
+ ```sh
279
+ sudo -u pmtiles-swarm -H npm install --prefix /var/lib/pmtiles-swarm pmtiles-swarm@0.3.0
280
+ sudo systemctl restart pmtiles-swarm
281
+ ```
282
+
246
283
  ## Checking it
247
284
 
248
285
  ```sh
@@ -41,11 +41,14 @@ Its feed is then at `https://maps.example.org/feed.xml`, with per-category feeds
41
41
  "subscriptions": [
42
42
  { "url": "https://maps.example.org/feed.xml", "mode": "cache" }
43
43
  ],
44
+ "subscriptionsEnabled": true,
44
45
  "subscriptionIntervalSeconds": 900
45
46
  }
46
47
  ```
47
48
 
48
- Poll immediately rather than waiting for the interval:
49
+ `subscriptionsEnabled` is the master switch and `subscriptionIntervalSeconds`
50
+ is how often each feed is checked; zero or less turns polling off. Poll
51
+ immediately rather than waiting for the interval:
49
52
 
50
53
  ```sh
51
54
  curl -X POST localhost:8090/api/subscriptions/refresh
@@ -261,6 +264,61 @@ One feed can serve subscribers with different appetites:
261
264
 
262
265
  `filter` is a case-insensitive regular expression matched against the item title.
263
266
 
267
+ ## Following someone else's RSS feed
268
+
269
+ A subscription does not have to be another swarm node. Any RSS feed with
270
+ `application/x-bittorrent` enclosures works, which includes the one
271
+ OpenStreetMap publishes for the planet dumps:
272
+
273
+ ```json
274
+ {
275
+ "subscriptions": [
276
+ {
277
+ "url": "https://planet.openstreetmap.org/pbf/planet-pbf-rss.xml",
278
+ "protocol": "rss",
279
+ "mode": "mirror",
280
+ "newest": 1,
281
+ "categories": ["osm-planet"],
282
+ "savePath": "/mnt/raid0/work/planet"
283
+ }
284
+ ]
285
+ }
286
+ ```
287
+
288
+ `newest` is how many items one check may take, counting from the newest, and
289
+ it is **1 by default**. That feed lists five planet dumps, so without a cap the
290
+ first poll is roughly four hundred gigabytes. `0` takes everything it lists.
291
+
292
+ The items are `.osm.pbf`, not map archives, and that is fine: joining an
293
+ existing torrent does not require the content to be anything in particular —
294
+ only *creating* one does. The archive simply is not servable as tiles, and
295
+ nothing tries.
296
+
297
+ `enabled: false` switches one feed off without deleting it, and
298
+ `subscriptionsEnabled: false` switches off all of them at once, for when a disk
299
+ is filling or a build has gone wrong.
300
+
301
+ Nothing is downloaded twice: an item already in the catalog is skipped, so a
302
+ feed that keeps listing last week's dump costs one comparison per check.
303
+
304
+ ### Feeding a generation script
305
+
306
+ The point of taking a `.osm.pbf` is usually to build something from it. That is
307
+ what `onComplete` is for — it runs when a download finishes, so a planet dump
308
+ landing can start the render that turns it into archives this node publishes:
309
+
310
+ ```json
311
+ {
312
+ "onComplete": {
313
+ "command": "/usr/local/bin/planetilerdump-swarm.sh",
314
+ "args": ["%N", "%F", "%I"]
315
+ }
316
+ }
317
+ ```
318
+
319
+ Which is the same handover qBittorrent's "run external program on torrent
320
+ finished" provided, with the feed replacing its RSS auto-downloader.
321
+
264
322
  ## Subscribing with plain qBittorrent
265
323
 
266
324
  The feed is ordinary RSS 2.0 with `application/x-bittorrent` enclosures, which is exactly
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pmtiles-swarm",
3
- "version": "0.3.1",
3
+ "version": "0.4.0",
4
4
  "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
5
5
  "type": "module",
6
6
  "main": "src/index.js",
package/src/api.js CHANGED
@@ -1,4 +1,5 @@
1
1
  import crypto from 'node:crypto';
2
+ import fsSync from 'node:fs';
2
3
  import fs from 'node:fs/promises';
3
4
  import { createRequire } from 'node:module';
4
5
  import path from 'node:path';
@@ -24,6 +25,17 @@ import { TileReadError } from './tiles.js';
24
25
 
25
26
  const here = path.dirname(fileURLToPath(import.meta.url));
26
27
 
28
+ /**
29
+ * This package's version, for the console to show.
30
+ *
31
+ * Read from package.json rather than repeated in a constant: a version in two
32
+ * places is a version that disagrees with itself, and the one on screen is the
33
+ * one somebody will quote when reporting a problem.
34
+ */
35
+ const VERSION = JSON.parse(
36
+ fsSync.readFileSync(path.join(here, '..', 'package.json'), 'utf8'),
37
+ ).version;
38
+
27
39
  /**
28
40
  * Wraps an async route so a rejection becomes a 500 rather than an unhandled
29
41
  * rejection that takes the process down.
@@ -395,6 +407,7 @@ export function createApp({
395
407
  engineError = error.message;
396
408
  }
397
409
  res.json({
410
+ version: VERSION,
398
411
  engine: { name: engine.name, ok: engineOk, error: engineError },
399
412
  archives: catalog.list().length,
400
413
  categories: catalog.categories(),
@@ -1584,15 +1597,28 @@ export function createApp({
1584
1597
 
1585
1598
  // Redirects rather than serving, so what arrives is the immutable URL and a
1586
1599
  // client that keeps it keeps a specific build rather than a moving target.
1587
- app.get('/latest/:category/archive.torrent', (req, res) => {
1600
+ // The name in the path is yours to choose: `/latest/openmaptiles/
1601
+ // planetiler-openmaptiles-latest.torrent` is the same route as
1602
+ // `.../archive.torrent`, and reads as what it is where it matters most —
1603
+ // in an href on a page. It does not name the download. That is decided at
1604
+ // the immutable URL this redirects to, which calls the file after the build
1605
+ // it actually is, and a URL that could choose it would be a link on this
1606
+ // domain that saves a file called anything at all.
1607
+ app.get('/latest/:category/:name.torrent', (req, res) => {
1588
1608
  const entry = newestIn(req.params.category, req);
1589
1609
  if (!entry) return res.status(404).json({ error: 'no such category' });
1610
+ // Short, and said explicitly. A 302 is not cacheable unless a response
1611
+ // says so, but "unless it says so" is a thing intermediaries have been
1612
+ // known to disagree about — and this one moves on every build, which is
1613
+ // the whole point of it.
1614
+ res.setHeader('cache-control', 'public, max-age=300');
1590
1615
  res.redirect(302, `${baseUrl(req)}/archives/${entry.infoHash}/archive.torrent`);
1591
1616
  });
1592
1617
 
1593
1618
  app.get('/latest/:category/magnet', (req, res) => {
1594
1619
  const entry = newestIn(req.params.category, req);
1595
1620
  if (!entry) return res.status(404).json({ error: 'no such category' });
1621
+ res.setHeader('cache-control', 'public, max-age=300');
1596
1622
  res.type('text/plain').send(entry.magnet ?? '');
1597
1623
  });
1598
1624
 
@@ -1756,6 +1782,12 @@ export function createApp({
1756
1782
  if (!body) return res.status(404).json({ error: 'torrent file missing' });
1757
1783
  res.setHeader('access-control-allow-origin', '*');
1758
1784
  res.type('application/x-bittorrent');
1785
+ // An infohash names these bytes and no others, so this URL can never
1786
+ // answer differently — the same reason the tile routes say it. Worth
1787
+ // saying out loud where a link to it is published: a cache or a reverse
1788
+ // proxy in front of this then serves the download without touching the
1789
+ // node at all.
1790
+ res.setHeader('cache-control', 'public, max-age=31536000, immutable');
1759
1791
  res.setHeader(
1760
1792
  'content-disposition',
1761
1793
  `attachment; filename="${entry.name}.torrent"`,
package/src/config.js CHANGED
@@ -475,6 +475,18 @@ const DEFAULTS = {
475
475
  *
476
476
  * `webSeedBase` on its own assumes the watched folder is already the web
477
477
  * root, since nothing is moved.
478
+ *
479
+ * `latestLink` gives the newest build a stable second name — a symlink, or
480
+ * a hard link where the platform refuses one. The watcher ignores that name
481
+ * so the link is never imported as an archive of its own.
482
+ *
483
+ * `keep` and `keepDays` retire what the folder has outgrown — the newest N
484
+ * builds, or a window in days. A folder receiving a daily 137 GB planet
485
+ * build fills any disk within the week, and this is the `find -mtime +35`
486
+ * sweep that would otherwise have to sit in the generation script, except
487
+ * that it takes the torrent with the data. Both are off unless set, only
488
+ * ever touch archives this same folder imported, and never remove the
489
+ * newest build however old it gets.
478
490
  */
479
491
  watch: [],
480
492
  /**
@@ -661,7 +673,15 @@ const DEFAULTS = {
661
673
  allowHooksFromApi: false,
662
674
  /** How often to look for finished downloads, in seconds. */
663
675
  onCompleteCheckIntervalSeconds: 60,
664
- /** How often to poll subscribed feeds, in seconds. */
676
+ /**
677
+ * Whether to follow feeds at all.
678
+ *
679
+ * The master switch, separate from the per-feed one: turning it off stops
680
+ * every feed without editing any of them, which is what you want when a
681
+ * disk is filling or a run has gone wrong.
682
+ */
683
+ subscriptionsEnabled: true,
684
+ /** How often to poll subscribed feeds, in seconds. Zero or less is off. */
665
685
  subscriptionIntervalSeconds: 900,
666
686
  /** Republish interval for BEP 46 records, in seconds. DHT items expire. */
667
687
  republishIntervalSeconds: 3600,
@@ -886,6 +906,7 @@ export const RELOADABLE = new Map([
886
906
  ['sourceCheckIntervalHours', 'sources'],
887
907
  ['subscriptions', 'subscriptions'],
888
908
  ['subscriptionIntervalSeconds', 'subscriptions'],
909
+ ['subscriptionsEnabled', 'subscriptions'],
889
910
  ['seeding', 'seeding'],
890
911
  // Applied to a running session, so a schedule can be corrected at the moment
891
912
  // it turns out to be wrong rather than at the next convenient restart.
@@ -0,0 +1,72 @@
1
+ import fs from 'node:fs/promises';
2
+ import path from 'node:path';
3
+
4
+ /**
5
+ * Points a stable name at the newest build.
6
+ *
7
+ * The dated file stays the real one, so it remains seedable under its own
8
+ * torrent while a page links to a name that does not change — which is what an
9
+ * `ln -sfn latest` in a generation script was for.
10
+ *
11
+ * A symlink first, then a hard link. **Windows refuses symlinks with EPERM**
12
+ * unless the process is elevated or the machine is in developer mode, which is
13
+ * not a reasonable thing to require of a daemon — and a hard link needs
14
+ * neither. It costs no extra space either way, since both are another name for
15
+ * the same bytes rather than a copy, and for a 137 GB archive that distinction
16
+ * is the whole point. Both only work within one filesystem, which is where a
17
+ * link beside the file it names always is.
18
+ *
19
+ * The two are not quite equivalent, and the difference shows when the build
20
+ * this names is eventually deleted: a symlink is left dangling, while a hard
21
+ * link keeps the bytes alive until it too is gone. Retention never removes the
22
+ * newest build, so neither case arises from this node's own housekeeping.
23
+ * @param {object} options - What to link and how to say so.
24
+ * @param {string} options.target - The build the name should resolve to.
25
+ * @param {string} options.name - The stable name, absolute or beside the target.
26
+ * @param {string} options.label - How to name the caller in the log.
27
+ * @returns {Promise<string|undefined>} - The link made, or undefined on failure.
28
+ */
29
+ export async function linkLatest({ target, name, label }) {
30
+ const link = linkPathFor(target, name);
31
+
32
+ const attempts = [
33
+ // The type is autodetected from the target on Windows and ignored
34
+ // everywhere else, which is right: the target is always a file here.
35
+ ['symlink', () => fs.symlink(target, link)],
36
+ ['hard link', () => fs.link(target, link)],
37
+ ];
38
+
39
+ for (const [kind, make] of attempts) {
40
+ try {
41
+ // The previous build's link, which is the usual case — this runs once
42
+ // per build and the name by definition already exists after the first.
43
+ await fs.rm(link, { force: true });
44
+ await make();
45
+ console.log(
46
+ `${label} latest -> ${path.basename(target)}` +
47
+ (kind === 'symlink' ? '' : ` (${kind})`),
48
+ );
49
+ return link;
50
+ } catch (error) {
51
+ // Try the next kind rather than giving up on the first refusal; only the
52
+ // last one is worth reporting.
53
+ if (make === attempts.at(-1)[1]) {
54
+ console.warn(
55
+ `${label} could not point ${path.basename(link)} at ` +
56
+ `${path.basename(target)}: ${error.message}`,
57
+ );
58
+ }
59
+ }
60
+ }
61
+ return undefined;
62
+ }
63
+
64
+ /**
65
+ * Where a stable name resolves to, absolute or beside the build it names.
66
+ * @param {string} target - The build being named.
67
+ * @param {string} name - The configured name.
68
+ * @returns {string} - The link's path.
69
+ */
70
+ export function linkPathFor(target, name) {
71
+ return path.isAbsolute(name) ? name : path.join(path.dirname(target), name);
72
+ }
package/src/library.js CHANGED
@@ -197,7 +197,11 @@ export class Library {
197
197
 
198
198
  return this.#register(created, {
199
199
  categories: options.categories ?? options.category,
200
- source: { type: 'file', location: absolute },
200
+ // `watch` names the folder that imported this, where one did. Not the
201
+ // same thing as the directory it sits in: with publishDir it has already
202
+ // moved somewhere else, and an archive dropped into a watched folder by
203
+ // hand is still that folder's to retire.
204
+ source: { type: 'file', location: absolute, watch: options.watch },
201
205
  // The torrent names the file, so the save path is its parent directory.
202
206
  savePath: path.dirname(absolute),
203
207
  pmtiles: summary,
@@ -0,0 +1,144 @@
1
+ const DAY_MS = 24 * 60 * 60 * 1000;
2
+
3
+ /**
4
+ * When an archive was built, as a timestamp.
5
+ *
6
+ * The date in the name where there is one, since that is what a dated build
7
+ * means by "old" — falling back to when this node took it, which for a folder
8
+ * receiving builds as they are made is close enough to the same thing.
9
+ * @param {object} entry - A catalog entry.
10
+ * @returns {number|undefined} - Milliseconds, or undefined if it cannot be told.
11
+ */
12
+ function builtAt(entry) {
13
+ const parsed = Date.parse(entry.buildDate ?? entry.createdAt ?? '');
14
+ return Number.isNaN(parsed) ? undefined : parsed;
15
+ }
16
+
17
+ /**
18
+ * Whether any retention rule is set at all.
19
+ *
20
+ * Worth asking before gathering the family, because gathering it means walking
21
+ * the whole catalog — and retention is off unless somebody turned it on, so
22
+ * that walk is usually for nothing.
23
+ * @param {object} rules - Something carrying keep and keepDays.
24
+ * @returns {boolean} - True if either rule would do something.
25
+ */
26
+ export function retains({ keep, keepDays } = {}) {
27
+ return Number(keep) >= 1 || Number(keepDays) >= 1;
28
+ }
29
+
30
+ /**
31
+ * Which of a family of builds have outlived the rules set for them.
32
+ *
33
+ * Pure, so the decision to delete several hundred gigabytes can be tested
34
+ * without any of it existing.
35
+ * @param {object} options - The family and the rules.
36
+ * @param {object[]} options.family - Builds, newest first.
37
+ * @param {number} [options.keep] - How many of the newest to hold.
38
+ * @param {number} [options.keepDays] - How old a build may get, in days.
39
+ * @param {number} [options.now] - The current time, for testing.
40
+ * @returns {object[]} - The entries to retire, in the family's order.
41
+ */
42
+ export function expired({ family, keep, keepDays, now = Date.now() }) {
43
+ const doomed = new Set();
44
+
45
+ const count = Number(keep);
46
+ if (Number.isFinite(count) && count >= 1) {
47
+ for (const entry of family.slice(count)) doomed.add(entry);
48
+ }
49
+
50
+ const days = Number(keepDays);
51
+ if (Number.isFinite(days) && days >= 1) {
52
+ const cutoff = now - days * DAY_MS;
53
+ // Never the newest, however old it is. A folder that stops receiving
54
+ // builds would otherwise empty itself and leave the node serving nothing —
55
+ // and "the last build is stale" is a thing to notice, not a thing to fix
56
+ // by deleting it. `find -mtime +35` had no such qualm.
57
+ for (const entry of family.slice(1)) {
58
+ const at = builtAt(entry);
59
+ // No date at all means no way to tell how old it is, and a guess is not
60
+ // good enough to delete on.
61
+ if (at !== undefined && at < cutoff) doomed.add(entry);
62
+ }
63
+ }
64
+
65
+ return family.filter((entry) => doomed.has(entry));
66
+ }
67
+
68
+ /**
69
+ * Retires the builds a family has outgrown, and removes their data.
70
+ *
71
+ * Deliberately narrow, because this deletes data:
72
+ *
73
+ * * Off unless `keep` or `keepDays` is set. Silence has to mean "keep
74
+ * everything", since the alternative is deleting archives nobody asked to
75
+ * lose.
76
+ * * Only the family it is given — whatever the caller can prove came from
77
+ * the same source or folder. An archive added by hand, adopted from a
78
+ * client, or taken from a peer is never touched, even in the same
79
+ * directory.
80
+ * * Never the newest build, and never the one just imported.
81
+ *
82
+ * Nothing is deleted until the new build is the one being served. A category's
83
+ * feed and its `/latest/<category>/tiles.json` resolve to the newest archive in
84
+ * it, and that is what consumers point at; once they resolve to this build, the
85
+ * ones it replaced are no longer where anyone is being sent. Before that —
86
+ * while an older build is still the answer — deleting it would break the very
87
+ * URL the feed is advertising. It also covers the case that makes this
88
+ * necessary at all: an import run that takes several builds takes them newest
89
+ * first, so an *older* one can be the most recent import. It has superseded
90
+ * nothing and must retire nothing.
91
+ *
92
+ * The torrent goes with the data. Leaving a catalog entry whose file is gone
93
+ * would leave the node advertising an archive it cannot serve, and every peer
94
+ * that asked would fail.
95
+ * @param {object} options - What to retire and how to say so.
96
+ * @param {object} options.library - The library, for removal.
97
+ * @param {object[]} options.family - Builds from one source, newest first.
98
+ * @param {object} options.entry - The build just imported.
99
+ * @param {number} [options.keep] - How many of the newest to hold.
100
+ * @param {number} [options.keepDays] - How old a build may get, in days.
101
+ * @param {string} options.label - How to name this family in the log.
102
+ * @param {number} [options.now] - The current time, for testing.
103
+ * @returns {Promise<string[]>} - The infohashes removed.
104
+ */
105
+ export async function retire({
106
+ library,
107
+ family,
108
+ entry,
109
+ keep,
110
+ keepDays,
111
+ label,
112
+ now,
113
+ }) {
114
+ const doomed = expired({ family, keep, keepDays, now });
115
+ if (doomed.length === 0) return [];
116
+
117
+ if (family[0]?.infoHash !== entry.infoHash) {
118
+ console.log(
119
+ `${label}: ${entry.name} is not the newest build here, so nothing is retired`,
120
+ );
121
+ return [];
122
+ }
123
+
124
+ const held = [
125
+ Number(keep) >= 1 ? `the newest ${Number(keep)}` : undefined,
126
+ Number(keepDays) >= 1 ? `${Number(keepDays)} days` : undefined,
127
+ ]
128
+ .filter(Boolean)
129
+ .join(' and ');
130
+
131
+ const removed = [];
132
+ for (const old of doomed) {
133
+ try {
134
+ await library.remove(old.infoHash, { deleteData: true });
135
+ removed.push(old.infoHash);
136
+ console.log(`${label}: retired ${old.name} — keeping ${held}`);
137
+ } catch (error) {
138
+ // Worth saying rather than swallowing: the disk this exists to protect
139
+ // is now not being protected.
140
+ console.error(`${label}: could not retire ${old.name}: ${error.message}`);
141
+ }
142
+ }
143
+ return removed;
144
+ }
package/src/sources.js CHANGED
@@ -28,6 +28,8 @@
28
28
 
29
29
  import fs from 'node:fs/promises';
30
30
  import path from 'node:path';
31
+ import { linkLatest } from './latest-link.js';
32
+ import { retains, retire } from './retention.js';
31
33
 
32
34
  /**
33
35
  * Expands date placeholders in a template.
@@ -622,79 +624,37 @@ export class ScheduledSourceManager {
622
624
  }
623
625
 
624
626
  /**
625
- * Removes older builds from the same source, where `keep` says to.
627
+ * Removes older builds from the same source, where retention says to.
626
628
  *
627
629
  * A daily planet build is 137 GB. Kept for ever, a source like that fills
628
630
  * any disk within the week — and the older ones are rarely what anyone
629
631
  * wants, because the whole point of a dated build is that a newer one
630
- * replaces it. `keep: 1` means only the newest is held.
632
+ * replaces it. `keep: 1` holds only the newest; `keepDays: 35` holds five
633
+ * weeks of them.
631
634
  *
632
- * Deliberately narrow, because this deletes data:
633
- *
634
- * * Only archives this same named source imported. An archive added by
635
- * hand, adopted from a client, or taken from a peer is never touched,
636
- * even if it sits in the same directory.
637
- * * Never the one just imported, and never more than the count allows.
638
- * * Off unless `keep` is set. Silence has to mean "keep everything",
639
- * since the alternative is deleting archives nobody asked to lose.
640
- *
641
- * The torrent goes with the data. Leaving a catalog entry whose file is
642
- * gone would leave the node advertising an archive it cannot serve, and
643
- * every peer that asked would fail.
635
+ * The family is the archives this same *named* source imported, and nothing
636
+ * else — one added by hand, adopted from a client, or taken from a peer is
637
+ * never touched, even in the same directory. See `retire` for the rest of
638
+ * what this will not do.
644
639
  * @param {object} source - The source definition.
645
640
  * @param {object} entry - The build just imported.
646
641
  * @returns {Promise<string[]>} - The infohashes removed.
647
642
  */
648
643
  async #retire(source, entry) {
649
- const keep = Number(source.keep);
650
- if (!Number.isFinite(keep) || keep < 1) return [];
651
644
  if (!source.name) return [];
652
-
653
- // The catalog's own order, which is exactly what `/latest` follows.
654
- const family = this.#catalog
655
- .list()
656
- .filter((candidate) => candidate.source?.name === source.name);
657
-
658
- // Nothing is deleted until the new build is the one being served.
659
- //
660
- // A category's feed and its `/latest/<category>/tiles.json` resolve to the
661
- // newest archive in it, and that is what consumers point at. Once they
662
- // resolve to this build, the ones it replaced are no longer where anyone
663
- // is being sent, and retiring them costs nobody a fetch they were about to
664
- // make. Before that — while an older build is still the answer — deleting
665
- // it would break the very URL the feed is advertising.
666
- //
667
- // It also covers the case that makes this necessary at all: a poll taking
668
- // several builds takes them newest first, so an *older* one can be the
669
- // most recent import. It has superseded nothing and must retire nothing.
670
- if (family[0]?.infoHash !== entry.infoHash) {
671
- console.log(
672
- `[source] ${source.name}: ${entry.name} is not the newest build here, ` +
673
- 'so nothing is retired',
674
- );
675
- return [];
676
- }
677
-
678
- const doomed = family.slice(keep);
679
- const removed = [];
680
-
681
- for (const old of doomed) {
682
- try {
683
- await this.#library.remove(old.infoHash, { deleteData: true });
684
- removed.push(old.infoHash);
685
- console.log(
686
- `[source] ${source.name}: retired ${old.name} — keeping the newest ` +
687
- `${keep}`,
688
- );
689
- } catch (error) {
690
- // Worth saying rather than swallowing: the disk this exists to protect
691
- // is now not being protected.
692
- console.error(
693
- `[source] ${source.name}: could not retire ${old.name}: ${error.message}`,
694
- );
695
- }
696
- }
697
- return removed;
645
+ if (!retains(source)) return [];
646
+
647
+ return retire({
648
+ library: this.#library,
649
+ // The catalog's own order, which is exactly what `/latest` follows.
650
+ family: this.#catalog
651
+ .list()
652
+ .filter((candidate) => candidate.source?.name === source.name),
653
+ entry,
654
+ keep: source.keep,
655
+ keepDays: source.keepDays,
656
+ label: `[source] ${source.name}`,
657
+ });
698
658
  }
699
659
 
700
660
  /**
@@ -703,48 +663,16 @@ export class ScheduledSourceManager {
703
663
  * The dated file stays the real one either way, so it remains seedable under
704
664
  * its own torrent while consumers reference a fixed path.
705
665
  *
706
- * A symlink first, then a hard link. Windows refuses symlinks with EPERM
707
- * unless the process is elevated or the machine is in developer mode, which
708
- * is not a reasonable thing to require of a daemon — and a hard link needs
709
- * neither. It costs no extra space, since it is another name for the same
710
- * bytes rather than a copy, and for a 137 GB archive that distinction is the
711
- * whole point. It only works within one filesystem, which is where a link
712
- * beside the file it names always is.
713
666
  * @param {object} source - The source definition.
714
667
  * @param {object} entry - The freshly imported entry.
715
668
  * @returns {Promise<void>} - Resolves once linked, or logs and continues.
716
669
  */
717
670
  async #linkLatest(source, entry) {
718
- const target = entry.retainedAt ?? path.join(entry.savePath, entry.name);
719
- const link = path.isAbsolute(source.latestLink)
720
- ? source.latestLink
721
- : path.join(path.dirname(target), source.latestLink);
722
-
723
- const attempts = [
724
- ['symlink', () => fs.symlink(target, link)],
725
- ['hard link', () => fs.link(target, link)],
726
- ];
727
-
728
- for (const [kind, make] of attempts) {
729
- try {
730
- await fs.rm(link, { force: true });
731
- await make();
732
- console.log(
733
- `[source] latest -> ${path.basename(target)}` +
734
- (kind === 'symlink' ? '' : ` (${kind})`),
735
- );
736
- return;
737
- } catch (error) {
738
- // Try the next kind rather than giving up on the first refusal; only
739
- // the last one is worth reporting.
740
- if (make === attempts.at(-1)[1]) {
741
- console.warn(
742
- `[source] could not point ${path.basename(link)} at ` +
743
- `${path.basename(target)}: ${error.message}`,
744
- );
745
- }
746
- }
747
- }
671
+ await linkLatest({
672
+ target: entry.retainedAt ?? path.join(entry.savePath, entry.name),
673
+ name: source.latestLink,
674
+ label: '[source]',
675
+ });
748
676
  }
749
677
 
750
678
  /**
@@ -39,10 +39,23 @@ export class SubscriptionManager {
39
39
  start() {
40
40
  const feeds = this.#config.subscriptions ?? [];
41
41
 
42
+ if (this.#config.subscriptionsEnabled === false) {
43
+ console.log('[feed] following is switched off');
44
+ return;
45
+ }
46
+
42
47
  // The timer runs even with nothing to follow. Every refresh reads the list
43
48
  // fresh, so this is what lets a peer added through the console start
44
49
  // working without a restart; an empty pass costs nothing.
45
- const intervalMs = (this.#config.subscriptionIntervalSeconds ?? 900) * 1000;
50
+ const seconds = this.#config.subscriptionIntervalSeconds ?? 900;
51
+ // Zero reads as off everywhere else in the configuration, and it has to
52
+ // read as off here too: setInterval(fn, 0) is not a stopped timer, it is
53
+ // one that fires as fast as the loop will let it.
54
+ if (seconds <= 0) {
55
+ console.log('[feed] interval is zero; not polling');
56
+ return;
57
+ }
58
+ const intervalMs = seconds * 1000;
46
59
  // Poll once at startup, then on the interval.
47
60
  this.refresh().catch((error) =>
48
61
  console.error(`[feed] initial refresh failed: ${error.message}`),
@@ -67,7 +80,12 @@ export class SubscriptionManager {
67
80
  */
68
81
  async refresh() {
69
82
  const added = [];
83
+ // Checked here as well as in start(), so the switch means the same thing
84
+ // to a refresh asked for through the API as it does to the timer.
85
+ if (this.#config.subscriptionsEnabled === false) return added;
70
86
  for (const subscription of this.#config.subscriptions ?? []) {
87
+ // Off, but kept — what qBittorrent's per-feed checkbox does.
88
+ if (subscription.enabled === false) continue;
71
89
  try {
72
90
  added.push(...(await this.#poll(subscription)));
73
91
  } catch (error) {
@@ -103,7 +121,20 @@ export class SubscriptionManager {
103
121
  const items = parseFeed(await response.text());
104
122
  const added = [];
105
123
 
124
+ // How many items one poll may take.
125
+ //
126
+ // One by default, because a feed's items are whole files and some of them
127
+ // are enormous: planet.openstreetmap.org lists five planet dumps, and
128
+ // taking the lot on the first poll is four hundred gigabytes nobody asked
129
+ // for. `newest: 0` lifts the cap for feeds where that is what you want.
130
+ //
131
+ // Items arrive newest first, so the cap keeps the newest.
132
+ const limit = subscription.newest === 0
133
+ ? Number.POSITIVE_INFINITY
134
+ : Math.max(1, subscription.newest ?? 1);
135
+
106
136
  for (const item of items) {
137
+ if (added.length >= limit) break;
107
138
  // A regex filter lets one feed serve several consumers with different
108
139
  // appetites, e.g. only taking Europe extracts.
109
140
  if (subscription.filter && !new RegExp(subscription.filter, 'i').test(item.title)) {
package/src/watch.js CHANGED
@@ -1,5 +1,8 @@
1
+ import path from 'node:path';
1
2
  import chokidar from 'chokidar';
2
3
  import { normalizeCategories } from './catalog.js';
4
+ import { linkLatest, linkPathFor } from './latest-link.js';
5
+ import { retains, retire } from './retention.js';
3
6
 
4
7
  /**
5
8
  * Watches folders for new PMTiles archives and imports them automatically.
@@ -26,7 +29,7 @@ export class WatchManager {
26
29
 
27
30
  /**
28
31
  * Starts watching the configured folders.
29
- * @param {object[]} folders - Entries of {path, categories, webSeedBase, publishDir, sparse, trackers, addTrackers, stabilitySeconds, pollSeconds}.
32
+ * @param {object[]} folders - Entries of {path, categories, webSeedBase, publishDir, sparse, trackers, addTrackers, stabilitySeconds, pollSeconds, keep, keepDays, latestLink}.
30
33
  * @returns {void}
31
34
  */
32
35
  start(folders = []) {
@@ -62,6 +65,12 @@ export class WatchManager {
62
65
 
63
66
  watcher.on('add', (file) => {
64
67
  if (!/\.pmtiles$/i.test(file)) return;
68
+ // The folder's own `latestLink`, which is a .pmtiles in a watched
69
+ // folder like any other and would otherwise be imported as a second
70
+ // archive — a whole extra torrent for the same bytes under a name that
71
+ // changes every build. A hard link is indistinguishable from the file
72
+ // it names, so the name is the only thing that can tell them apart.
73
+ if (this.#isLatestLink(file, folder)) return;
65
74
  this.#import(file, folder);
66
75
  });
67
76
  watcher.on('error', (error) => {
@@ -78,6 +87,21 @@ export class WatchManager {
78
87
  }
79
88
  }
80
89
 
90
+ /**
91
+ * Whether a file is this folder's own "latest" name.
92
+ * @param {string} file - The file that appeared.
93
+ * @param {object} folder - The watch-folder configuration.
94
+ * @returns {boolean} - True when it is the link, not a build.
95
+ */
96
+ #isLatestLink(file, folder) {
97
+ if (!folder.latestLink) return false;
98
+ // Compared as a path rather than a basename, so an absolute latestLink
99
+ // pointing somewhere else entirely does not silently exclude a real build
100
+ // that happens to share its name.
101
+ const target = path.join(folder.path, 'any.pmtiles');
102
+ return path.resolve(file) === path.resolve(linkPathFor(target, folder.latestLink));
103
+ }
104
+
81
105
  /**
82
106
  * Imports one archive, guarding against overlapping imports of the same file.
83
107
  * @param {string} file - Path to the archive.
@@ -110,8 +134,37 @@ export class WatchManager {
110
134
  // not on the node.
111
135
  pieceLength: folder.pieceLength,
112
136
  comment: folder.comment,
137
+ // Marks this as the folder's, so retention below has a family to work
138
+ // within and nothing outside it can be caught up in one.
139
+ watch: folder.path,
113
140
  });
114
141
  console.log(`[watch] imported ${entry.name} (${entry.infoHash})`);
142
+
143
+ // Before retirement, so the stable name is already pointing at the new
144
+ // build by the time anything older is considered for removal.
145
+ if (folder.latestLink) {
146
+ await linkLatest({
147
+ target: entry.retainedAt ?? path.join(entry.savePath, entry.name),
148
+ name: folder.latestLink,
149
+ label: `[watch] ${folder.path}`,
150
+ });
151
+ }
152
+
153
+ // A folder receiving a daily planet build fills any disk within the
154
+ // week. This is the `find -mtime +35` sweep that used to sit in the
155
+ // generation script, except that it takes the torrent with the data
156
+ // rather than leaving the node advertising an archive that is gone.
157
+ if (!retains(folder)) return;
158
+ await retire({
159
+ library: this.#library,
160
+ family: this.#library.catalog
161
+ .list()
162
+ .filter((candidate) => candidate.source?.watch === folder.path),
163
+ entry,
164
+ keep: folder.keep,
165
+ keepDays: folder.keepDays,
166
+ label: `[watch] ${folder.path}`,
167
+ });
115
168
  } catch (error) {
116
169
  console.error(`[watch] failed to import ${file}: ${error.message}`);
117
170
  } finally {
@@ -254,6 +254,20 @@
254
254
  }
255
255
  .choice > span { flex: 1; }
256
256
 
257
+ footer {
258
+ margin-top: 2.5rem;
259
+ padding: 1rem 0 0.5rem;
260
+ border-top: 1px solid var(--line);
261
+ color: var(--muted);
262
+ font-size: 0.8rem;
263
+ display: flex;
264
+ gap: 0.6rem;
265
+ flex-wrap: wrap;
266
+ align-items: baseline;
267
+ }
268
+ footer a { color: var(--muted); }
269
+ footer .version { font-variant-numeric: tabular-nums; }
270
+
257
271
  /* Speed limits. Declared after .field on purpose: `.field input` is
258
272
  width:100%, which would push the unit label onto its own line. */
259
273
  .rates { display: flex; gap: 1rem; flex-wrap: wrap; }
@@ -362,6 +376,14 @@
362
376
  </section>
363
377
  </main>
364
378
 
379
+ <footer>
380
+ <span>&copy; <span id="year"></span> TechIdiots LLC</span>
381
+ <span>·</span>
382
+ <span>pmtiles-swarm <span class="version" id="version">…</span></span>
383
+ <span>·</span>
384
+ <a href="https://github.com/TechIdiots-LLC/pmtiles-swarm" rel="noreferrer">source</a>
385
+ </footer>
386
+
365
387
  <dialog id="move-dialog">
366
388
  <form method="dialog" id="move-form">
367
389
  <h2>Set location</h2>
@@ -859,6 +881,7 @@
859
881
  ]);
860
882
  renderSpeed(speed);
861
883
  renderFetching(adds?.running);
884
+ if (status.version) $('version').textContent = `v${status.version}`;
862
885
  const engine = status.engine;
863
886
  $('status').innerHTML =
864
887
  `engine <b>${engine.name}</b> ${engine.ok ? 'ready' : 'unavailable'}` +
@@ -3101,9 +3124,46 @@
3101
3124
  },
3102
3125
  { field: 'publishDir', label: 'Publish to', placeholder: '/var/www/pmtiles' },
3103
3126
  { field: 'webSeedBase', label: 'Web seed base', placeholder: 'https://…/files' },
3127
+ {
3128
+ field: 'comment',
3129
+ label: 'Torrent comment',
3130
+ placeholder: 'attribution and licence',
3131
+ wide: true,
3132
+ },
3104
3133
  { field: 'pollSeconds', label: 'Poll every (s)', placeholder: 'events', number: true },
3134
+ {
3135
+ field: 'latestLink',
3136
+ label: 'Stable name',
3137
+ placeholder: 'off',
3138
+ },
3139
+ {
3140
+ field: 'keep',
3141
+ label: 'Builds to keep',
3142
+ placeholder: 'all',
3143
+ number: true,
3144
+ },
3145
+ {
3146
+ field: 'keepDays',
3147
+ label: 'Keep for (days)',
3148
+ placeholder: 'for ever',
3149
+ number: true,
3150
+ },
3105
3151
  ],
3106
3152
  rows: config.watch ?? [],
3153
+ footnote:
3154
+ 'Stable name gives the newest build a second name that does not ' +
3155
+ 'change — planet-latest.pmtiles — so a page can link to it. The ' +
3156
+ 'dated file stays the real one and keeps its own torrent; the ' +
3157
+ 'link costs no extra space, and is a symlink or a hard link ' +
3158
+ 'depending on what the platform allows. Do not give it a name a ' +
3159
+ 'real build could have. ' +
3160
+ 'Builds to keep and Keep for are both off unless set, and both ' +
3161
+ 'delete the archive as well as forgetting it — a catalog entry ' +
3162
+ 'whose file is gone leaves this node advertising something it ' +
3163
+ 'cannot serve. They only ever touch archives this same folder ' +
3164
+ 'imported, never the newest one however old it is, and nothing at ' +
3165
+ 'all until the new build is the one the feed resolves to. Set ' +
3166
+ 'both and whichever says a build has to go wins.',
3107
3167
  });
3108
3168
 
3109
3169
  renderRowEditor({
@@ -3170,6 +3230,12 @@
3170
3230
  placeholder: 'all',
3171
3231
  number: true,
3172
3232
  },
3233
+ {
3234
+ field: 'keepDays',
3235
+ label: 'Keep for (days)',
3236
+ placeholder: 'for ever',
3237
+ number: true,
3238
+ },
3173
3239
  ],
3174
3240
  rows: config.sources ?? [],
3175
3241
  preview: true,
@@ -3200,17 +3266,50 @@
3200
3266
  'archives this same source imported.',
3201
3267
  });
3202
3268
 
3269
+ const feedGlobals = document.createElement('div');
3270
+ feedGlobals.className = 'panel';
3271
+ feedGlobals.style.marginBottom = '1rem';
3272
+ feedGlobals.innerHTML = `
3273
+ <h2 style="margin-top:0">Feeds</h2>
3274
+ <div class="sub" style="margin-bottom:0.7rem">
3275
+ Applies to every feed below.
3276
+ </div>
3277
+ <div class="field">
3278
+ <label for="cfg-subscriptionsEnabled">Follow feeds</label>
3279
+ <select id="cfg-subscriptionsEnabled" data-key="subscriptionsEnabled" data-scalar="1">
3280
+ <option value="true"${config.subscriptionsEnabled === false ? '' : ' selected'}>on</option>
3281
+ <option value="false"${config.subscriptionsEnabled === false ? ' selected' : ''}>off</option>
3282
+ </select>
3283
+ <div class="sub">
3284
+ Off stops all of them without editing any, for when a disk is
3285
+ filling or a build has gone wrong.
3286
+ </div>
3287
+ </div>
3288
+ <div class="field">
3289
+ <label for="cfg-subscriptionIntervalSeconds">Check every (seconds)</label>
3290
+ <input id="cfg-subscriptionIntervalSeconds" data-key="subscriptionIntervalSeconds"
3291
+ data-scalar="1" style="width:9rem"
3292
+ value="${escapeHtml(config.subscriptionIntervalSeconds ?? 900)}" />
3293
+ <div class="sub">
3294
+ A poll is one request per feed. Publishers post on their own
3295
+ schedule, so checking faster than they publish finds nothing.
3296
+ </div>
3297
+ </div>`;
3298
+ body.append(feedGlobals);
3299
+
3203
3300
  renderRowEditor({
3204
3301
  into: body,
3205
3302
  key: 'subscriptions',
3206
3303
  title: 'Remote nodes',
3207
3304
  blurb:
3208
- 'Peers whose archives this node follows. An RSS feed says "here ' +
3209
- 'is what is new" and is bounded by the publisher, so a node ' +
3210
- 'offline long enough misses things for good; a catalog URL says ' +
3211
- '"here is everything", which is what makes reconciling possible. ' +
3212
- 'A token, where the peer issued one, may get you more than it ' +
3213
- 'publishes to the world.',
3305
+ 'Feeds this node follows: another swarm node, or any RSS feed ' +
3306
+ 'that carries torrents planet.openstreetmap.org publishes one ' +
3307
+ 'for the planet dumps. An RSS feed says "here is what is new" ' +
3308
+ 'and is bounded by the publisher, so a node offline long enough ' +
3309
+ 'misses things for good; a catalog URL says "here is ' +
3310
+ 'everything", which is what makes reconciling possible. A token, ' +
3311
+ 'where the peer issued one, may get you more than it publishes ' +
3312
+ 'to the world.',
3214
3313
  columns: [
3215
3314
  {
3216
3315
  field: 'url',
@@ -3235,6 +3334,12 @@
3235
3334
  ['mirror', 'mirror — whole copy'],
3236
3335
  ],
3237
3336
  },
3337
+ {
3338
+ field: 'newest',
3339
+ label: 'Items per check',
3340
+ placeholder: '1',
3341
+ number: true,
3342
+ },
3238
3343
  { field: 'category', label: 'Tag as', placeholder: 'from-peer' },
3239
3344
  { field: 'filter', label: 'Name filter', placeholder: 'terrain' },
3240
3345
  { field: 'token', label: 'Token', placeholder: 'if issued one', secret: true },
@@ -3248,6 +3353,15 @@
3248
3353
  ['delete', 'forget and delete'],
3249
3354
  ],
3250
3355
  },
3356
+ {
3357
+ field: 'enabled',
3358
+ label: 'On',
3359
+ boolean: true,
3360
+ options: [
3361
+ ['', 'yes'],
3362
+ ['false', 'no'],
3363
+ ],
3364
+ },
3251
3365
  ],
3252
3366
  rows: config.subscriptions ?? [],
3253
3367
  peerPreview: true,
@@ -3257,7 +3371,10 @@
3257
3371
  'every archive the peer lists. Dropping stays off unless chosen, ' +
3258
3372
  'only ever considers archives this peer sent, and never acts on a ' +
3259
3373
  'filtered or partial view — watch a new peer on "report only" ' +
3260
- 'before trusting it with anything more.',
3374
+ 'before trusting it with anything more. Items per check is 1 by ' +
3375
+ 'default and counts from the newest, because a feed listing five ' +
3376
+ 'planet dumps is four hundred gigabytes if you take the lot; 0 ' +
3377
+ 'takes everything it lists.',
3261
3378
  });
3262
3379
 
3263
3380
  for (const [key, value] of Object.entries(config)) {
@@ -3266,6 +3383,8 @@
3266
3383
  // Shown as real fields above rather than as raw JSON.
3267
3384
  if (key === 'seeding' || key === 'speed') continue;
3268
3385
  if (key === 'watch' || key === 'sources' || key === 'subscriptions') continue;
3386
+ if (key === 'subscriptionsEnabled') continue;
3387
+ if (key === 'subscriptionIntervalSeconds') continue;
3269
3388
  if (key === 'onAdded' || key === 'onComplete') continue;
3270
3389
  const needsRestart = restartKeys.has(key);
3271
3390
  const field = document.createElement('div');
@@ -3436,6 +3555,11 @@
3436
3555
  if (name === 'settings') loadSettings().catch((e) => toast(e.message));
3437
3556
  if (name === 'categories') loadCategories().catch((e) => toast(e.message));
3438
3557
  };
3558
+ // The footer's year, set from the clock rather than typed into a file
3559
+ // nobody will remember to edit. The version beside it arrives with the
3560
+ // first status, and reads "…" until then.
3561
+ $('year').textContent = String(new Date().getFullYear());
3562
+
3439
3563
  $('tab-archives').onclick = () => showTab('archives');
3440
3564
  $('tab-categories').onclick = () => showTab('categories');
3441
3565
  $('tab-settings').onclick = () => showTab('settings');
@@ -71,8 +71,18 @@
71
71
  }
72
72
  ],
73
73
 
74
+ "subscriptionsEnabled": true,
75
+ "subscriptionIntervalSeconds": 900,
74
76
  "subscriptions": [
75
- { "url": "https://peer.example.org/api/catalog", "protocol": "api", "mode": "cache" }
77
+ { "url": "https://peer.example.org/api/catalog", "protocol": "api", "mode": "cache" },
78
+ {
79
+ "url": "https://feed.example.org/pbf/planet-pbf-rss.xml",
80
+ "protocol": "rss",
81
+ "mode": "mirror",
82
+ "newest": 1,
83
+ "categories": ["osm-planet"],
84
+ "enabled": false
85
+ }
76
86
  ],
77
87
 
78
88
  "tiles": { "sparse": null },