c3b-assets 0.0.0-stage → 0.1.0

package/README.md

@@ -1,3 +1,122 @@
-# Temporary Holding Version
+# c3b-assets
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Fingerprints a project's images and syncs them to a [Bunny](https://bunny.net) storage zone, so images never live in
+git and never need uploading by hand. Writes `d8e-asset-manifest.json`, which [d8e](https://codeberg.org/colab47/d8e)
+(1.2.0+) uses to point every reference at the CDN.
+
+## How it works
+
+Run `c3b-assets` in a project root whenever you add, change or remove an image under `src/asset/`. It:
+
+1. Fingerprints each image from its content — `asset/hero-banner.webp` is stored as
+   `colab-asset-manager/hero-banner.3f2a9c1e.webp`. Change the image and it gets a new name automatically, so it can be
+   cached forever and a stale copy can never be served.
+2. Uploads the ones the storage zone doesn't have yet (checksum-verified).
+3. Writes `d8e-asset-manifest.json`: each image's path under `src/` and its CDN URL. **Commit this file** — it's a few KB.
+4. Deletes files nobody references any more, **7 days** after they stopped being referenced, so pages in browser
+   caches, sent emails and older staging builds keep working in the meantime.
+
+Code keeps referencing images normally (`<img src="asset/hero-banner.webp">`, `url(../asset/hero-banner.webp)`), so
+local dev still works by opening `src/`. At deploy, d8e rewrites those references to the CDN URLs and leaves the
+images out of `dist/`.
+
+It only ever lists, uploads to and deletes inside the zone's `colab-asset-manager/` folder (the tool's old name,
+kept so existing zones need no re-upload) — files uploaded by hand anywhere else in the zone are never touched. Supported: `avif`, `gif`, `jpeg`, `jpg`, `png`, `svg`, `webp`.
+
+It runs on your machine only, never at deploy: the storage key stays off Heroku and Vercel, and staging can't delete
+production's images.
+
+## Setup
+
+### 1. Install
+
+```
+npm install -g c3b-assets@latest
+```
+
+### 2. Add `c3b-assets.json` to the project root
+
+```json
+{
+  "storageZone": "viking-gear",
+  "storageHost": "jh.storage.bunnycdn.com",
+  "cdnUrl": "https://viking-gear-pz.b-cdn.net"
+}
+```
+
+- `storageZone`: the storage zone's name.
+- `storageHost`: optional, defaults to `storage.bunnycdn.com` (Frankfurt). Use the zone's primary region:
+  `uk.`, `ny.`, `la.`, `sg.`, `se.`, `br.`, `jh.` (Johannesburg) or `syd.storage.bunnycdn.com`.
+- `cdnUrl`: the pull zone connected to that storage zone.
+
+### 3. Gitignore the images
+
+```
+src/asset/**/*.avif
+src/asset/**/*.gif
+src/asset/**/*.jpeg
+src/asset/**/*.jpg
+src/asset/**/*.png
+src/asset/**/*.svg
+src/asset/**/*.webp
+```
+
+If the images are already committed, also run `git rm -r --cached src/asset` once (it keeps your local copies). Their
+old versions stay in git history.
+
+### 4. Set the storage key
+
+`BUNNY_STORAGE_KEY` is the storage zone password: Bunny dashboard → Storage → your zone → FTP & API Access. Keep it in
+your shell profile or a password manager — never in the repo.
+
+## Usage
+
+```
+c3b-assets --help      # lists every flag
+c3b-assets --pull      # downloads the images the manifest lists into src/asset/ (no key needed)
+c3b-assets --dry-run   # shows what would be uploaded and deleted; changes nothing
+c3b-assets             # syncs and writes d8e-asset-manifest.json
+```
+
+The version is printed first, so you can tell which install is running.
+
+### Changing an image
+
+1. **Fresh clone?** Run `c3b-assets --pull` first — images aren't in git, so `src/asset/` starts empty.
+2. Replace, add or remove the file in `src/asset/` (kebab-case names). Keeping the same name for a replacement is
+   fine: the fingerprint changes anyway. Check it locally by opening `src/`.
+3. `c3b-assets --dry-run`, then `c3b-assets`.
+4. Commit `d8e-asset-manifest.json` (plus any code change that references a new image) and deploy as usual.
+
+If two people sync on different branches, `d8e-asset-manifest.json` conflicts on merge: pull both sets of images
+into `src/asset/` (`--pull`) and run `c3b-assets` again — it rebuilds the manifest from what's there.
+
+### Safety checks
+
+- **Nothing is deleted just because it's missing locally.** If images the manifest lists aren't in `src/asset/` (a
+  fresh clone, a half-copied folder), syncing refuses and says to `--pull`. If you removed them on purpose, add
+  `--allow-removals`.
+- `--pull` never overwrites a local file that differs from Bunny — that's usually a change that hasn't been uploaded
+  yet. It only fills in missing ones, and checks each download against its fingerprint.
+- Deletions only happen inside the zone's `colab-asset-manager/` folder, 7 days after a file stopped being referenced.
+
+## Limitations
+
+- Bunny storage zones have one full-access password (plus a read-only one, which isn't enough to upload). Anyone who
+  uploads needs the full one: share it through a password manager, and regenerate it in Bunny if someone leaves.
+- Videos and other large files stay where they are (Mux / Bunny Stream).
+
+---
+
+## Tests
+
+`npm test` — Node's built-in test runner, no extra dependency. `test/c3b-assets.test.js` runs the real CLI
+against fixture projects and a fake Bunny (`test/fake-bunny.js`, installed as the global `fetch`), so nothing reaches
+the network and no key is needed.
+
+## Deploying to npm
+
+1. **FIRST TIME ONLY**: run `npm login`
+2. Bump the version in `package.json`
+3. `npm test`
+4. Run `npm publish`

package/bunny.js

@@ -0,0 +1,62 @@
+import { createHash } from 'crypto';
+
+// Thin wrapper over Bunny's Edge Storage HTTP API (https://bunny.net/docs/storage/http) — only the
+// three calls c3b-assets needs. Uses the global fetch, so there are no dependencies.
+export function createStorage({ host, zone, accessKey }) {
+    const base = `https://${ host }/${ zone }/`;
+
+    function request(method, path, options = {}) {
+        return fetch(base + encodeURI(path), {
+            ...options,
+            method,
+            headers: { AccessKey: accessKey, ...options.headers }
+        });
+    }
+
+    // Every file under `folder`, as paths relative to the zone root (e.g. colab-asset-manager/asset/logo.1a2b3c4d.svg).
+    async function listFiles(folder) {
+        const response = await request('GET', folder + '/');
+        // a folder that doesn't exist yet just means nothing has been uploaded
+        if (response.status === 404) {
+            return [];
+        }
+        if (!response.ok) {
+            throw new Error(`Listing ${ folder }/ failed: ${ response.status } ${ await response.text() }`);
+        }
+
+        const files = [];
+        for (const entry of await response.json()) {
+            const path = folder + '/' + entry.ObjectName;
+            if (entry.IsDirectory) {
+                files.push(...await listFiles(path));
+            } else {
+                files.push(path);
+            }
+        }
+
+        return files;
+    }
+
+    async function upload(path, content) {
+        // Bunny rejects the upload if the stored bytes don't match this, so a truncated upload can't
+        // end up being served from a URL that's cached forever.
+        const checksum = createHash('sha256').update(content).digest('hex').toUpperCase();
+        const response = await request('PUT', path, {
+            body: content,
+            headers: { 'Content-Type': 'application/octet-stream', Checksum: checksum }
+        });
+        if (response.status !== 201) {
+            throw new Error(`Uploading ${ path } failed: ${ response.status } ${ await response.text() }`);
+        }
+    }
+
+    async function remove(path) {
+        const response = await request('DELETE', path);
+        // already gone is the outcome we wanted anyway
+        if (!response.ok && response.status !== 404) {
+            throw new Error(`Deleting ${ path } failed: ${ response.status } ${ await response.text() }`);
+        }
+    }
+
+    return { listFiles, upload, remove };
+}

package/c3b-assets.js

@@ -0,0 +1,273 @@
+#!/usr/bin/env node
+import { createHash } from 'crypto';
+import { existsSync, globSync, mkdirSync, readFileSync, writeFileSync } from 'fs';
+import { basename, dirname, extname, join, relative, resolve, sep } from 'path';
+import { fileURLToPath } from 'url';
+import { createStorage } from './bunny.js';
+
+// Printed first so it's obvious which installed version is running (same as d8e does).
+const packageJsonPath = join(dirname(fileURLToPath(import.meta.url)), 'package.json');
+const { version } = JSON.parse(readFileSync(packageJsonPath, 'utf8'));
+console.log(`c3b-assets version: ${ version }`);
+
+// Fingerprints every image under src/asset/, uploads the ones Bunny doesn't have yet, and writes
+// d8e-asset-manifest.json (asset path under src/ -> CDN URL) for d8e to rewrite references with.
+// Files in Bunny that nothing references any more are deleted once they've been unreferenced
+// for GRACE_DAYS — pages already in browser caches, sent emails and older staging builds still
+// point at them for a while.
+//
+// Runs on your machine only (not at deploy), so the storage key never has to live on Heroku or
+// Vercel, and staging can't delete production's images.
+
+const SRC = resolve('src');
+const ASSET_DIR = resolve(SRC, 'asset');
+const CONFIG = resolve('c3b-assets.json');
+const MANIFEST = resolve('d8e-asset-manifest.json');
+const IMAGE_EXTENSIONS = ['avif', 'gif', 'jpeg', 'jpg', 'png', 'svg', 'webp'];
+
+// Everything c3b-assets uploads lives under this folder in the storage zone, and it only ever
+// lists or deletes inside it — files uploaded by hand elsewhere in the zone are never touched.
+// The folder keeps the tool's old name: every existing zone has its images there, and renaming it
+// would re-upload every image and orphan the old copies.
+const FOLDER = 'colab-asset-manager';
+const GRACE_DAYS = 7;
+const DAY_MS = 24 * 60 * 60 * 1000;
+
+// Every flag the tool accepts, with its one-line description. `--help` prints them; anything not
+// listed here is rejected rather than silently ignored.
+const FLAGS = {
+    '--dry-run': 'report what would be uploaded, deleted and written, change nothing',
+    '--pull': 'download images listed in the manifest that are missing locally',
+    '--allow-removals': 'allow manifest entries with no local file to be dropped',
+    '--help': 'print this help',
+};
+
+function usage() {
+    console.log(`Usage: c3b-assets [flags]\n\nFingerprints images under src/asset/, syncs them to Bunny, writes d8e-asset-manifest.json.\n\nFlags:`);
+    for (const [flag, description] of Object.entries(FLAGS)) {
+        console.log(`  ${ flag.padEnd(18) }${ description }`);
+    }
+}
+
+const args = process.argv.slice(2);
+if (args.includes('--help') || args.includes('-h')) {
+    usage();
+    process.exit(0);
+}
+for (const arg of args) {
+    if (!(arg in FLAGS)) {
+        console.error(`c3b-assets: unknown option ${ arg }\n`);
+        usage();
+        process.exit(1);
+    }
+}
+
+const dryRun = args.includes('--dry-run');
+const pull = args.includes('--pull');
+const allowRemovals = args.includes('--allow-removals');
+
+function fail(message) {
+    console.error(`c3b-assets: ${ message }`);
+    process.exit(1);
+}
+
+function readJson(path) {
+    try {
+        return JSON.parse(readFileSync(path, 'utf8'));
+    } catch (error) {
+        fail(`could not read ${ path }: ${ error.message }`);
+    }
+}
+
+const toPosix = (p) => p.split(sep).join('/');
+
+function readConfig() {
+    if (!existsSync(CONFIG)) {
+        fail(`${ CONFIG } not found — see the README for its shape`);
+    }
+
+    const config = readJson(CONFIG);
+    if (!config.storageZone || !config.cdnUrl) {
+        fail('c3b-assets.json needs "storageZone" and "cdnUrl"');
+    }
+
+    return {
+        storageZone: config.storageZone,
+        storageHost: config.storageHost ?? 'storage.bunnycdn.com',
+        cdnUrl: config.cdnUrl.replace(/\/+$/, '')
+    };
+}
+
+// Each image's manifest key (its path under src/) and the fingerprinted path it's stored at in
+// Bunny. The fingerprint comes from the content, so changing an image always gives it a new name.
+function scanImages() {
+    const images = [];
+    for (const file of globSync(`${ ASSET_DIR }/**/*.{${ IMAGE_EXTENSIONS.join(',') }}`)) {
+        const content = readFileSync(file);
+        const hash = fingerprint(content);
+        const ext = extname(file);
+        const key = toPosix(relative(SRC, file));
+        const hashedName = basename(file, ext) + '.' + hash + ext;
+        // relative to src/asset/, not src/ — everything is under asset/, so repeating it in Bunny is
+        // just an extra folder level (asset/arch/x.webp is stored as colab-asset-manager/arch/x.<hash>.webp)
+        const remoteDir = toPosix(relative(ASSET_DIR, dirname(file)));
+        const remotePath = remoteDir ? `${ FOLDER }/${ remoteDir }/${ hashedName }` : `${ FOLDER }/${ hashedName }`;
+
+        images.push({ key, content, remotePath });
+    }
+    images.sort((a, b) => a.key.localeCompare(b.key));
+
+    return images;
+}
+
+// Which unreferenced files to delete now, and the updated record of when each one stopped being
+// referenced. That record is kept in the manifest, so the grace period survives between runs and
+// machines.
+function planDeletions(remoteFiles, referenced, previouslyRetired, now) {
+    const retired = {};
+    const toDelete = [];
+
+    for (const path of remoteFiles) {
+        if (referenced.has(path)) {
+            continue;
+        }
+
+        const since = previouslyRetired[path] ?? now.toISOString().slice(0, 10);
+        const ageDays = (now - new Date(since)) / DAY_MS;
+        if (ageDays >= GRACE_DAYS) {
+            toDelete.push(path);
+        } else {
+            retired[path] = since;
+        }
+    }
+
+    return { retired, toDelete };
+}
+
+function fingerprint(content) {
+    return createHash('md5').update(content).digest('hex').slice(0, 8);
+}
+
+function sortedObject(entries) {
+    return Object.fromEntries(entries.sort(([a], [b]) => a.localeCompare(b)));
+}
+
+const config = readConfig();
+
+async function sync() {
+    const accessKey = process.env.BUNNY_STORAGE_KEY;
+    if (!accessKey) {
+        fail('set BUNNY_STORAGE_KEY to the storage zone password (Bunny dashboard: Storage > FTP & API Access)');
+    }
+    if (!existsSync(ASSET_DIR)) {
+        fail(`${ ASSET_DIR } not found — on a fresh clone, run "c3b-assets --pull" first`);
+    }
+
+    const storage = createStorage({ host: config.storageHost, zone: config.storageZone, accessKey });
+    const images = scanImages();
+    const previous = existsSync(MANIFEST) ? readJson(MANIFEST) : {};
+
+    // Images aren't in git, so a fresh clone (or a half-copied folder) looks exactly like someone
+    // deleting them — and syncing would then delete them from Bunny too. Removals have to be asked for.
+    const localKeys = new Set(images.map((image) => image.key));
+    const missing = Object.keys(previous.assets ?? {}).filter((key) => !localKeys.has(key));
+    if (missing.length > 0 && !allowRemovals) {
+        fail(`${ missing.length } image(s) in the manifest aren't in src/asset/ (e.g. ${ missing[0] }). `
+            + 'Run "c3b-assets --pull" to download them, or add --allow-removals if you removed them on purpose.');
+    }
+
+    const remoteFiles = new Set(await storage.listFiles(FOLDER));
+
+    const toUpload = images.filter((image) => !remoteFiles.has(image.remotePath));
+    const referenced = new Set(images.map((image) => image.remotePath));
+    const { retired, toDelete } = planDeletions(remoteFiles, referenced, previous.retired ?? {}, new Date());
+
+    console.log(`${ images.length } images, ${ toUpload.length } to upload, ${ toDelete.length } to delete, `
+        + `${ Object.keys(retired).length } waiting out the ${ GRACE_DAYS }-day grace period`);
+
+    for (const image of toUpload) {
+        console.log(`  upload  ${ image.remotePath }`);
+        if (!dryRun) {
+            await storage.upload(image.remotePath, image.content);
+        }
+    }
+
+    // Uploads go first and the manifest is only written once they've all succeeded, so a failed run
+    // leaves the previous manifest pointing at files that are still there.
+    const manifest = {
+        // encodeURI so a name with a space (or similar) is still a valid URL in srcset and unquoted CSS url()
+        assets: sortedObject(images.map((image) => [image.key, `${ config.cdnUrl }/${ encodeURI(image.remotePath) }`])),
+        retired: sortedObject(Object.entries(retired))
+    };
+    if (!dryRun) {
+        writeFileSync(MANIFEST, JSON.stringify(manifest, null, 2) + '\n');
+    }
+
+    for (const path of toDelete) {
+        console.log(`  delete  ${ path }`);
+        if (!dryRun) {
+            await storage.remove(path);
+        }
+    }
+
+    console.log(dryRun ? 'Dry run — nothing uploaded, deleted or written.' : `Wrote ${ MANIFEST }`);
+}
+
+// Downloads every image in the manifest that src/asset/ is missing, straight from the CDN — so it
+// needs no storage key, just the committed manifest. A local file that differs from the manifest is
+// kept, not overwritten: it's most likely a change that hasn't been uploaded yet.
+async function pullImages() {
+    if (!existsSync(MANIFEST)) {
+        fail(`${ MANIFEST } not found — nothing to pull`);
+    }
+
+    const assets = readJson(MANIFEST).assets ?? {};
+    let downloaded = 0;
+    let upToDate = 0;
+    let kept = 0;
+
+    for (const [key, url] of Object.entries(assets)) {
+        const target = resolve(SRC, key);
+        if (!target.startsWith(ASSET_DIR + sep)) {
+            fail(`refusing to write outside src/asset/: ${ key }`);
+        }
+
+        // the fingerprint in the stored name (hero.0100d254.webp) is the content hash we expect
+        const expected = decodeURI(url).match(/\.([0-9a-f]{8})\.[^./]+$/)?.[1];
+
+        if (existsSync(target)) {
+            if (fingerprint(readFileSync(target)) === expected) {
+                upToDate++;
+            } else {
+                console.log(`  keep    ${ key } (differs from Bunny — not uploaded yet?)`);
+                kept++;
+            }
+            continue;
+        }
+
+        console.log(`  pull    ${ key }`);
+        if (dryRun) {
+            downloaded++;
+            continue;
+        }
+
+        const response = await fetch(url);
+        if (!response.ok) {
+            throw new Error(`Downloading ${ url } failed: ${ response.status }`);
+        }
+        const content = Buffer.from(await response.arrayBuffer());
+        if (fingerprint(content) !== expected) {
+            throw new Error(`Downloaded ${ key } doesn't match its fingerprint — not saved`);
+        }
+
+        mkdirSync(dirname(target), { recursive: true });
+        writeFileSync(target, content);
+        downloaded++;
+    }
+
+    console.log(`${ downloaded } downloaded, ${ upToDate } already up to date, ${ kept } kept (local changes)`
+        + (dryRun ? ' — dry run, nothing written' : ''));
+}
+
+// Bunny API errors (wrong key, wrong zone or region) print as one line rather than a stack trace
+(pull ? pullImages() : sync()).catch((error) => fail(error.message));

package/package.json

@@ -1,6 +1,28 @@
 {
   "name": "c3b-assets",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.0",
+  "description": "Fingerprints a project's images and syncs them to a Bunny storage zone, writing the d8e-asset-manifest.json d8e reads.",
+  "license": "GPL-3.0",
+  "type": "module",
+  "bin": {
+    "c3b-assets": "c3b-assets.js"
+  },
+  "files": [
+    "c3b-assets.js",
+    "bunny.js"
+  ],
+  "engines": {
+    "node": ">=22"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://codeberg.org/colab47/c3b-assets.git"
+  },
+  "publishConfig": {
+    "registry": "https://registry.npmjs.org"
+  },
+  "scripts": {
+    "test": "node --test test/*.test.js",
+    "deploy": "npm run test && npm publish"
+  }
+}