pmtiles-swarm 0.69.0 → 0.70.0

package/CHANGELOG.md

@@ -7,6 +7,33 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.70.0
+### ✨ Features and improvements
+- **An export works on the disk the archive is going to.** It worked under the data directory and
+  moved the finished file afterwards, which is the wrong disk twice over. The bytes have to go
+  where there is room for them, and a 700 GiB archive is not something a data directory is sized
+  for - while the disk chosen to hold the finished archive is, by definition.
+
+  The second problem was the move. `publish` renames within a filesystem and copies across one, so
+  a bake that worked elsewhere ended with a full copy of the whole archive - and until that
+  finished, the buffered tile data and the finished file both sat on the data directory's disk.
+  Working at the destination makes the last step a rename and removes the second full-size write.
+
+  Nothing appears at the destination until the end either way: the archive is assembled in one
+  pass at finalize, so until then `<destination>/bakes/<stack>/` holds the buffered tiles and the
+  checkpoint and no `.pmtiles` at all.
+
+### 🐞 Bug fixes
+- **A stack said "without sharp installed, its tiles answer 501" whether or not sharp was
+  installed.** It is a conditional the console can resolve and the reader cannot: nothing on the
+  page said which way it fell, so the line told somebody with a working stack that it might not
+  be, and somebody with a broken one that it might be fine.
+
+  `/api/stacks` has always reported the codec. Now the stack row uses it. With one installed there
+  is nothing to say, so nothing is said - and the "needs a codec" badge is gone too, because a
+  warning on something that works teaches people to ignore warnings. Without one it says the tiles
+  answer 501 *here*, names the field responsible, and says what to install.
+
 ## 0.69.0
 ### ✨ Features and improvements
 - **An export merges several tiles at once.** It did one at a time, start to finish, before
@@ -40,9 +67,6 @@
   while it ran was never looked at again, and the export completed as though nobody had asked it
   to stop. Checked around each batch as well now.
 
-### 🐞 Bug fixes
-- _...Add new stuff here..._
-
 ## 0.68.3
 ### 🐞 Bug fixes
 - **Exporting a stack failed on its first merged tile.** "Cannot transfer object of unsupported

package/docs/tile-stacks.md

@@ -789,6 +789,20 @@ One bake per stack at a time. Two runs of one recipe write the same checkpoint
 files over each other, and the second would resume the first's work believing it
 were its own.
 
+The work happens **on the filesystem the archive is going to** — under
+`<destination>/bakes/<stack>/` — not under the data directory. The bytes have
+to go where there is room for them, and a 700 GiB archive is not something a
+data directory is sized for, while the disk chosen to hold the finished archive
+is by definition. It also turns the last step of a long job from a copy of the
+whole archive into a rename, and means the buffered tile data and the finished
+archive never both have to fit on the data directory's disk.
+
+Nothing appears at the destination itself until the end. The archive is
+assembled in one pass at finalize, so until then the directory holds the
+buffered tiles and the checkpoint and no `.pmtiles` at all. Choosing a different
+location for a resumed bake looks for the checkpoint in the new place and finds
+none, so it starts over.
+
 ### What exists now
 
 `src/pmtiles-write.js` writes archives, `src/pmtiles-scan.js` reads back what one

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.69.0",
+  "version": "0.70.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/bake-jobs.js

@@ -24,9 +24,36 @@ import { outputFormat } from './stack-tile.js';
  * first's work as if it were its own.
  */
 
-/** Where unfinished bakes live, under the data directory. */
+/** What the working directory is called, wherever it sits. */
 const WORK_DIR = 'bakes';
 
+/**
+ * Where a bake does its work.
+ *
+ * On the filesystem the archive is going to, not under the data directory.
+ * Two reasons, and the first is the one that decides it: the bytes have to go
+ * somewhere there is room for them, and a 700 GiB archive is not something a
+ * data directory is sized for -- while the disk chosen to hold the finished
+ * archive is, by definition.
+ *
+ * The second falls out of the first. `publish` renames the finished file into
+ * place where it can and copies it where it cannot, so working on the
+ * destination's filesystem turns the last step of a long job from a full copy
+ * of the whole archive into a rename. That also removes the second full-size
+ * write, since the buffered tile data and the finished archive no longer have
+ * to exist on the data directory's disk at the same time.
+ *
+ * The directory is removed when the bake finishes. A cancelled one keeps it,
+ * which is the point of it.
+ * @param {object} job - The job, carrying where it will publish to.
+ * @param {object} config - The node's configuration.
+ * @returns {string} - The working directory.
+ */
+export function workDirFor(job, config = {}) {
+  const root = job.publishDir ?? config.savePath ?? config.dataDir ?? './data';
+  return path.join(root, WORK_DIR, job.stackId);
+}
+
 /** A bake that has finished is kept this long, so the console can report it. */
 const KEEP_FINISHED_MS = 10 * 60 * 1000;
 
@@ -173,11 +200,7 @@ export class BakeManager {
    * @returns {Promise<void>} - Resolves when the archive is in the catalog.
    */
   async #run(job, resolved, codec, options) {
-    const workDir = path.join(
-      this.#config.dataDir ?? './data',
-      WORK_DIR,
-      job.stackId,
-    );
+    const workDir = workDirFor(job, this.#config);
     const destination = path.join(workDir, job.name);
     const format = outputFormat(resolved);
 

package/src/web/index.html

@@ -5225,7 +5225,7 @@ Every piece is hashed against the ` +
             },
             {
               key: 'stacks.bakePauseMs',
-              label: 'Pause between tiles while exporting',
+              label: 'Pause between batches while exporting',
               type: 'number',
               placeholder: '0',
               unit: 'ms',
@@ -7042,11 +7042,15 @@ Every piece is hashed against the ` +
       // The shapes this node has, so a source can be clipped to one by name
       // rather than by remembering a filename.
       let stackCutlines = [];
+      // Which codec is installed, or null. A stack row needs it to say whether
+      // its pixel work is a problem here or merely a fact about the recipe.
+      let stackCodec = null;
 
       const loadStacks = async () => {
         const body = await api('/api/stacks');
         const { stacks, codec } = body;
         stackCutlines = body.cutlines ?? [];
+        stackCodec = codec ?? null;
         // A category resolving to a new build changes this, which is the
         // reason to poll at all -- but most polls change nothing, and
         // repainting a list somebody is about to click is worse than not.
@@ -7108,8 +7112,12 @@ Every piece is hashed against the ` +
         if (unresolved.length) {
           badges.push('<span class="pill warn">sources missing</span>');
         }
-        if (stack.needsCodec) {
-          badges.push('<span class="pill warn">needs a codec</span>');
+        // Only where it is actually a problem. With a codec installed the
+        // recipe asking for pixel work is a fact about the recipe, not
+        // something wrong with it, and a warning badge on a working stack
+        // teaches people to ignore warning badges.
+        if (stack.needsCodec && !stackCodec) {
+          badges.push('<span class="pill warn">no codec for this</span>');
         }
         if (stack.pinned) badges.push('<span class="pill">pinned</span>');
 
@@ -7191,10 +7199,10 @@ Every piece is hashed against the ` +
             }
             ${bakeLine}
             ${
-              stack.needsCodec
-                ? `<p class="muted">Serving this means decoding pixels
-                   (<code>${escapeHtml(stack.needsCodec)}</code>). Without
-                   <code>sharp</code> installed, its tiles answer 501.</p>`
+              stack.needsCodec && !stackCodec
+                ? `<p class="bad">Its tiles answer 501 here: serving this means
+                   decoding pixels (<code>${escapeHtml(stack.needsCodec)}</code>)
+                   and this node has no codec. <code>npm install sharp</code>.</p>`
                 : ''
             }
             <table class="tight">