c3b-assets 0.2.0 → 0.4.0

package/README.md

@@ -12,7 +12,8 @@ Run `c3b-assets u` in a project root whenever you add, change or remove an image
    `colab-asset-manager/hero-banner.3f2a9c1e.webp`. Change the image and it gets a new name automatically, so it can be
    cached forever and a stale copy can never be served.
 2. Uploads the ones the storage zone doesn't have yet (checksum-verified).
-3. Writes `d8e-asset-manifest.json`: each image's path under `src/` and its CDN URL. **Commit this file** — it's a few KB.
+3. Writes `d8e-asset-manifest.json`: each image's path under `src/` and its CDN URL. **Commit this file** — it's a few
+   KB.
 4. Deletes files nobody references any more, **7 days** after they stopped being referenced, so pages in browser
    caches, sent emails and older staging builds keep working in the meantime.
 
@@ -21,7 +22,8 @@ local dev still works by opening `src/`. At deploy, d8e rewrites those reference
 images out of `dist/`.
 
 It only ever lists, uploads to and deletes inside the zone's `colab-asset-manager/` folder (the tool's old name,
-kept so existing zones need no re-upload) — files uploaded by hand anywhere else in the zone are never touched. Supported: `avif`, `gif`, `jpeg`, `jpg`, `png`, `svg`, `webp`.
+kept so existing zones need no re-upload) — files uploaded by hand anywhere else in the zone are never touched.
+Supported: `avif`, `gif`, `jpeg`, `jpg`, `png`, `svg`, `webp`.
 
 It runs on your machine only, never at deploy: the storage key stays off Heroku and Vercel, and staging can't delete
 production's images.
@@ -73,11 +75,18 @@ your shell profile or a password manager — never in the repo.
 
 ```
 c3b-assets             # status: src/asset/ against the manifest — new, changed, missing, retired. Changes nothing, no key needed
-c3b-assets u           # upload: fingerprint, upload new images, write d8e-asset-manifest.json, delete retired files
+c3b-assets u           # upload: fingerprint, upload new images, write d8e-asset-manifest.json (deletes need --allow-removals)
 c3b-assets s           # sync: download the images the manifest lists that are missing locally, then upload as "u"
+c3b-assets a           # adopt: no manifest yet (a repo started over)? rebuild it from what the zone holds, then download
 c3b-assets --help      # the commands and flags
 ```
 
+`a` is for a checkout that has no `d8e-asset-manifest.json` but whose zone already has the images — a repo
+started over, or a project joining an existing zone: it lists the tool's folder in the zone, reads every stored
+image back to its manifest key (`colab-asset-manager/arch/x.2ed3ab61.webp` → `asset/arch/x.webp`), writes the
+manifest and downloads as `s` does. A key stored under several fingerprints keeps the most recently changed copy
+and retires the rest from today. It refuses to run over an existing manifest.
+
 Flags: `--dry-run` (report, change nothing), `--allow-removals` (see Safety checks). The version is printed first, so
 you can tell which install is running.
 
@@ -89,7 +98,8 @@ upload or delete afterwards. So a fresh clone on a machine without the key still
 1. **Fresh clone?** Run `c3b-assets s` first — images aren't in git, so `src/asset/` starts empty.
 2. Replace, add or remove the file in `src/asset/` (kebab-case names). Keeping the same name for a replacement is
    fine: the fingerprint changes anyway. Check it locally by opening `src/`.
-3. `c3b-assets` to see what will happen, `c3b-assets u --dry-run` if you want the exact Bunny paths, then `c3b-assets u`.
+3. `c3b-assets` to see what will happen, `c3b-assets u --dry-run` if you want the exact Bunny paths, then
+   `c3b-assets u`.
 4. Commit `d8e-asset-manifest.json` (plus any code change that references a new image) and deploy as usual.
 
 If two people sync on different branches, `d8e-asset-manifest.json` conflicts on merge: pull both sets of images
@@ -100,10 +110,19 @@ into `src/asset/` (`c3b-assets s` on each) and run `c3b-assets u` again — it r
 - **Nothing is deleted just because it's missing locally.** If images the manifest lists aren't in `src/asset/` (a
   fresh clone, a half-copied folder), `u` refuses and says to run `s`. If you removed them on purpose, add
   `--allow-removals`.
-- The download step of `s` never overwrites a local file that differs from Bunny — that's usually a change that hasn't been uploaded
+- The download step of `s` never overwrites a local file that differs from Bunny — that's usually a change that hasn't
+  been uploaded
   yet. It only fills in missing ones, and checks each download against its fingerprint.
 - Deletions only happen inside the zone's `colab-asset-manager/` folder, 7 days after a file stopped being
-  referenced. A folder left empty by those deletions is deleted with them (Bunny keeps empty folders otherwise).
+  referenced, and only with `--allow-removals`: "unreferenced" is judged from the checkout you run in, which may be
+  a branch that dropped images `main` still serves. A plain `u` reports what is due and stops. A folder left empty by
+  those deletions is deleted with them (Bunny keeps empty folders otherwise).
+- Only files this project's manifest has listed are ever retired or deleted. If two projects share a storage zone,
+  neither touches the other's images.
+- The download step of `s` only fetches image files, and only from this project's `cdnUrl` folder — the manifest is
+  a committed file, so a tampered entry can't plant something else in `src/asset/`.
+- `storageHost` must be a Bunny storage endpoint (`storage.bunnycdn.com`, optionally region-prefixed): the storage
+  key is sent to it, and the config is committed.
 
 ## Limitations
 

package/adopt.js

@@ -0,0 +1,34 @@
+// `c3b-assets a` (adopt): the manifest rebuilt from what the zone holds, for a checkout that has
+// none — a repo started over, or a project joining a zone that already has its images. Every
+// file under the tool's folder is read back to its manifest key: colab-asset-manager/arch/x.2ed3ab61.webp
+// is asset/arch/x.webp at ${ cdnUrl }/colab-asset-manager/arch/x.2ed3ab61.webp. Only images; a
+// key stored under more than one fingerprint keeps the most recently changed copy and retires
+// the rest from today, as an upload would have.
+export function manifestFromZone(entries, { cdnUrl, folder, imageExtensions, today }) {
+    const chosen = new Map();
+    const retired = {};
+    const skipped = [];
+
+    for (const entry of entries) {
+        const match = entry.path.match(/^(.*)\.([0-9a-f]{8})\.([^./]+)$/);
+        if (!match || !imageExtensions.includes(match[3].toLowerCase()) || !entry.path.startsWith(`${ folder }/`)) {
+            skipped.push(entry.path);
+            continue;
+        }
+        const key = `asset/${ match[1].slice(folder.length + 1) }.${ match[3] }`;
+        const current = chosen.get(key);
+        if (!current) {
+            chosen.set(key, entry);
+        } else if ((entry.changed ?? '') > (current.changed ?? '')) {
+            retired[current.path] = today;
+            chosen.set(key, entry);
+        } else {
+            retired[entry.path] = today;
+        }
+    }
+
+    const assets = Object.fromEntries([...chosen].sort(([a], [b]) => a.localeCompare(b))
+            .map(([key, entry]) => [key, `${ cdnUrl }/${ encodeURI(entry.path) }`]));
+
+    return { assets, retired: Object.fromEntries(Object.entries(retired).sort(([a], [b]) => a.localeCompare(b))), skipped };
+}

package/bunny.js

@@ -13,8 +13,9 @@ export function createStorage({ host, zone, accessKey }) {
         });
     }
 
-    // Every file under `folder`, as paths relative to the zone root (e.g. colab-asset-manager/asset/logo.1a2b3c4d.svg).
-    async function listFiles(folder) {
+    // Every file under `folder`, with when it last changed, as paths relative to the zone root
+    // (e.g. colab-asset-manager/asset/logo.1a2b3c4d.svg).
+    async function listEntries(folder) {
         const response = await request('GET', folder + '/');
         // a folder that doesn't exist yet just means nothing has been uploaded
         if (response.status === 404) {
@@ -28,15 +29,19 @@ export function createStorage({ host, zone, accessKey }) {
         for (const entry of await response.json()) {
             const path = folder + '/' + entry.ObjectName;
             if (entry.IsDirectory) {
-                files.push(...await listFiles(path));
+                files.push(...await listEntries(path));
             } else {
-                files.push(path);
+                files.push({ path, changed: entry.LastChanged ?? null });
             }
         }
 
         return files;
     }
 
+    async function listFiles(folder) {
+        return (await listEntries(folder)).map((entry) => entry.path);
+    }
+
     async function upload(path, content) {
         // Bunny rejects the upload if the stored bytes don't match this, so a truncated upload can't
         // end up being served from a URL that's cached forever.
@@ -58,5 +63,5 @@ export function createStorage({ host, zone, accessKey }) {
         }
     }
 
-    return { listFiles, upload, remove };
+    return { listEntries, listFiles, upload, remove };
 }

package/c3b-assets.js

@@ -4,6 +4,7 @@ import { existsSync, globSync, mkdirSync, readFileSync, writeFileSync } from 'fs
 import { basename, dirname, extname, join, relative, resolve, sep } from 'path';
 import { fileURLToPath } from 'url';
 import { createStorage } from './bunny.js';
+import { manifestFromZone } from './adopt.js';
 
 // Printed first so it's obvious which installed version is running (same as d8e does).
 const packageJsonPath = join(dirname(fileURLToPath(import.meta.url)), 'package.json');
@@ -38,16 +39,18 @@ const DAY_MS = 24 * 60 * 60 * 1000;
 const COMMANDS = {
     'u': 'upload: fingerprint src/asset/, upload new images, write d8e-asset-manifest.json, delete retired files',
     's': 'sync: download images the manifest lists that are missing locally, then upload (as "u")',
+    'a': 'adopt: for a checkout with no manifest, rebuild d8e-asset-manifest.json from what the zone holds, then download'
 };
 const FLAGS = {
     '--dry-run': 'report what would be uploaded, downloaded, deleted and written, change nothing',
-    '--allow-removals': 'allow manifest entries with no local file to be dropped',
-    '--help': 'print this help',
+    '--allow-removals': 'allow manifest entries with no local file to be dropped, and retired files past the grace '
+            + 'period to be deleted from Bunny',
+    '--help': 'print this help'
 };
 
 function usage() {
     console.log('Usage: c3b-assets [command] [flags]\n\nWith no command, prints the status of src/asset/ against '
-        + 'd8e-asset-manifest.json and changes nothing.\n\nCommands:');
+            + 'd8e-asset-manifest.json and changes nothing.\n\nCommands:');
     for (const [command, description] of Object.entries(COMMANDS)) {
         console.log(`  ${ command.padEnd(18) }${ description }`);
     }
@@ -99,9 +102,16 @@ function readConfig() {
         fail('c3b-assets.json needs "storageZone" and "cdnUrl"');
     }
 
+    // The storage key goes to this host in every request, and the config is a committed file: only
+    // Bunny's own storage endpoints (optionally region-prefixed) may receive it.
+    const storageHost = config.storageHost ?? 'storage.bunnycdn.com';
+    if (!/^([a-z]{2,3}\.)?storage\.bunnycdn\.com$/.test(storageHost)) {
+        fail(`c3b-assets.json "storageHost" must be a Bunny storage endpoint, not ${ storageHost }`);
+    }
+
     return {
         storageZone: config.storageZone,
-        storageHost: config.storageHost ?? 'storage.bunnycdn.com',
+        storageHost,
         cdnUrl: config.cdnUrl.replace(/\/+$/, '')
     };
 }
@@ -130,13 +140,15 @@ function scanImages() {
 
 // Which unreferenced files to delete now, and the updated record of when each one stopped being
 // referenced. That record is kept in the manifest, so the grace period survives between runs and
-// machines.
-function planDeletions(remoteFiles, referenced, previouslyRetired, now) {
+// machines. Only files this project's manifest has known about (`known`) are candidates: a zone
+// shared with another project holds files this checkout never referenced, and they are not ours
+// to delete.
+function planDeletions(remoteFiles, referenced, known, previouslyRetired, now) {
     const retired = {};
     const toDelete = [];
 
     for (const path of remoteFiles) {
-        if (referenced.has(path)) {
+        if (referenced.has(path) || !known.has(path)) {
             continue;
         }
 
@@ -162,6 +174,10 @@ function sortedObject(entries) {
 
 const config = readConfig();
 
+// Every manifest URL is `${ cdnUrl }/${ encodeURI(remotePath) }`; anything else is not ours.
+const CDN_PREFIX = `${ config.cdnUrl }/${ FOLDER }/`;
+const remotePathOf = (url) => (url.startsWith(CDN_PREFIX) ? decodeURI(url.slice(config.cdnUrl.length + 1)) : null);
+
 // Where src/asset/ stands against the manifest, without touching Bunny: images not in the manifest
 // yet, images whose content no longer matches the URL the manifest holds, and manifest entries with
 // no local file. The manifest is only written after every upload succeeded, so "in the manifest"
@@ -197,6 +213,7 @@ function status() {
     console.log(`zone ${ config.storageZone } -> ${ config.cdnUrl }`);
     if (!existsSync(ASSET_DIR)) {
         console.log(`${ ASSET_DIR } not found — run "c3b-assets s" to download the manifest's images`);
+
         return;
     }
 
@@ -225,10 +242,10 @@ function status() {
         console.log('Everything in src/asset/ is uploaded and in the manifest.');
     } else if (missing.length > 0) {
         console.log(`Run "c3b-assets s" to download ${ missing.length } missing image(s)`
-            + (toUpload > 0 ? ` and upload ${ toUpload }.` : '.'));
+                + (toUpload > 0 ? ` and upload ${ toUpload }.` : '.'));
     } else {
         console.log(`Run "c3b-assets u" to upload ${ toUpload } image(s)`
-            + (due > 0 ? ` and delete ${ due } retired file(s).` : '.'));
+                + (due > 0 ? `, with --allow-removals to delete ${ due } retired file(s).` : '.'));
     }
 }
 
@@ -247,17 +264,28 @@ async function upload(accessKey) {
     const missing = Object.keys(previous.assets ?? {}).filter((key) => !localKeys.has(key));
     if (missing.length > 0 && !allowRemovals) {
         fail(`${ missing.length } image(s) in the manifest aren't in src/asset/ (e.g. ${ missing[0] }). `
-            + 'Run "c3b-assets s" to download them, or add --allow-removals if you removed them on purpose.');
+                + 'Run "c3b-assets s" to download them, or add --allow-removals if you removed them on purpose.');
     }
 
     const remoteFiles = new Set(await storage.listFiles(FOLDER));
 
     const toUpload = images.filter((image) => !remoteFiles.has(image.remotePath));
     const referenced = new Set(images.map((image) => image.remotePath));
-    const { retired, toDelete } = planDeletions(remoteFiles, referenced, previous.retired ?? {}, new Date());
+    const known = new Set([
+        ...Object.values(previous.assets ?? {}).map(remotePathOf),
+        ...Object.keys(previous.retired ?? {})
+    ]);
+    const { retired, toDelete } = planDeletions(remoteFiles, referenced, known, previous.retired ?? {}, new Date());
+
+    // Deleting is driven by this checkout, which may be a branch that dropped images main still
+    // serves — so it is never a side effect of a plain upload.
+    if (toDelete.length > 0 && !allowRemovals) {
+        fail(`${ toDelete.length } retired file(s) are past the ${ GRACE_DAYS }-day grace period. `
+                + 'Add --allow-removals to delete them from Bunny, once this branch is what production serves.');
+    }
 
     console.log(`${ images.length } images, ${ toUpload.length } to upload, ${ toDelete.length } to delete, `
-        + `${ Object.keys(retired).length } waiting out the ${ GRACE_DAYS }-day grace period`);
+            + `${ Object.keys(retired).length } waiting out the ${ GRACE_DAYS }-day grace period`);
 
     for (const image of toUpload) {
         console.log(`  upload  ${ image.remotePath }`);
@@ -285,9 +313,11 @@ async function upload(accessKey) {
     }
 
     // Bunny leaves a folder behind when its last file goes, so the zone slowly fills with empty
-    // ones. Deepest first, so a parent is empty by the time it is checked.
+    // ones. Deepest first, so a parent is empty by the time it is checked. The listing predates
+    // this run's uploads, so they are added back in — a folder that only just received a new
+    // image is not empty.
     const deleted = new Set(toDelete);
-    const remaining = [...remoteFiles].filter((path) => !deleted.has(path));
+    const remaining = [...remoteFiles, ...referenced].filter((path) => !deleted.has(path));
     for (const folder of emptiedFolders(toDelete, remaining)) {
         console.log(`  delete  ${ folder }/`);
         if (!dryRun) {
@@ -311,8 +341,8 @@ function emptiedFolders(deleted, remaining) {
     }
 
     return [...candidates]
-        .sort((a, b) => b.length - a.length)
-        .filter((folder) => !remaining.some((path) => path.startsWith(folder + '/')));
+            .sort((a, b) => b.length - a.length)
+            .filter((folder) => !remaining.some((path) => path.startsWith(folder + '/')));
 }
 
 // Pull, then upload. The pull needs no storage key (the CDN is public), so a fresh clone without the
@@ -323,16 +353,17 @@ async function sync() {
     const accessKey = process.env.BUNNY_STORAGE_KEY;
     if (accessKey) {
         await upload(accessKey);
+
         return;
     }
 
     const previous = existsSync(MANIFEST) ? readJson(MANIFEST) : {};
     const { added, changed } = localChanges(existsSync(ASSET_DIR) ? scanImages() : [], previous);
     const due = Object.values(previous.retired ?? {})
-        .filter((since) => (Date.now() - new Date(since)) / DAY_MS >= GRACE_DAYS).length;
+            .filter((since) => (Date.now() - new Date(since)) / DAY_MS >= GRACE_DAYS).length;
     if (added.length + changed.length + due > 0) {
         fail(`${ added.length + changed.length } image(s) to upload and ${ due } retired file(s) to delete — `
-            + 'set BUNNY_STORAGE_KEY to the storage zone password (Bunny dashboard: Storage > FTP & API Access)');
+                + 'set BUNNY_STORAGE_KEY to the storage zone password (Bunny dashboard: Storage > FTP & API Access)');
     }
     console.log('Nothing to upload.');
 }
@@ -343,6 +374,7 @@ async function sync() {
 async function pullImages() {
     if (!existsSync(MANIFEST)) {
         console.log('No manifest yet — nothing to download.');
+
         return;
     }
 
@@ -356,6 +388,14 @@ async function pullImages() {
         if (!target.startsWith(ASSET_DIR + sep)) {
             fail(`refusing to write outside src/asset/: ${ key }`);
         }
+        // The manifest is a committed file, so what it points at is only trusted as far as this
+        // project's own CDN folder, and only ever written as an image.
+        if (!IMAGE_EXTENSIONS.includes(extname(key).slice(1).toLowerCase())) {
+            fail(`refusing to download ${ key }: not an image`);
+        }
+        if (remotePathOf(url) === null) {
+            fail(`refusing to download ${ key } from outside ${ CDN_PREFIX }: ${ url }`);
+        }
 
         // the fingerprint in the stored name (hero.0100d254.webp) is the content hash we expect
         const expected = decodeURI(url).match(/\.([0-9a-f]{8})\.[^./]+$/)?.[1];
@@ -391,11 +431,38 @@ async function pullImages() {
     }
 
     console.log(`${ downloaded } downloaded, ${ upToDate } already up to date, ${ kept } kept (local changes)`
-        + (dryRun ? ' — dry run, nothing written' : ''));
+            + (dryRun ? ' — dry run, nothing written' : ''));
+}
+
+// A checkout with no manifest (a repo started over) adopts what the zone already holds: the
+// manifest is rebuilt from the zone's listing and the images downloaded as "s" would.
+async function adopt() {
+    if (existsSync(MANIFEST)) {
+        fail('d8e-asset-manifest.json already exists — "a" is for a checkout without one; use "s" to sync');
+    }
+    const storage = createStorage({ host: config.storageHost, zone: config.storageZone, accessKey: requireStorageKey() });
+    const entries = await storage.listEntries(FOLDER);
+    const today = new Date().toISOString().slice(0, 10);
+    const manifest = manifestFromZone(entries, { cdnUrl: config.cdnUrl, folder: FOLDER, imageExtensions: IMAGE_EXTENSIONS, today });
+
+    for (const key of Object.keys(manifest.assets)) {
+        console.log(`  adopt   ${ key }`);
+    }
+    for (const path of manifest.skipped) {
+        console.log(`  skip    ${ path } (not an image c3b-assets stored)`);
+    }
+    console.log(`${ Object.keys(manifest.assets).length } image(s) on Bunny, ${ Object.keys(manifest.retired).length } older copies retired`
+            + (dryRun ? ' — dry run, nothing written' : ''));
+    if (dryRun) {
+        return;
+    }
+    writeFileSync(MANIFEST, JSON.stringify({ assets: manifest.assets, retired: manifest.retired }, null, 2) + '\n');
+    await pullImages();
 }
 
 // Bunny API errors (wrong key, wrong zone or region) print as one line rather than a stack trace
 const run = command === 'u' ? () => upload(requireStorageKey())
-    : command === 's' ? sync
-    : async () => status();
+        : command === 's' ? sync
+                : command === 'a' ? adopt
+                        : async () => status();
 run().catch((error) => fail(error.message));

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "c3b-assets",
-  "version": "0.2.0",
+  "version": "0.4.0",
   "description": "Fingerprints a project's images and syncs them to a Bunny storage zone, writing the d8e-asset-manifest.json d8e reads.",
   "license": "GPL-3.0",
   "type": "module",
@@ -9,7 +9,8 @@
   },
   "files": [
     "c3b-assets.js",
-    "bunny.js"
+    "bunny.js",
+    "adopt.js"
   ],
   "engines": {
     "node": ">=22"
@@ -22,7 +23,10 @@
     "registry": "https://registry.npmjs.org"
   },
   "scripts": {
-    "test": "node --test test/*.test.js",
+    "test": "c3b-lint && node --test test/*.test.js",
     "deploy": "npm run test && npm publish"
+  },
+  "devDependencies": {
+    "c3b-lint": "0.0.8"
   }
 }