@lad-sapienza/scms-core 0.1.0-alpha.11 → 0.1.0-alpha.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +90 -0
- package/bin/lib/optimize-images.mjs +440 -0
- package/bin/optimize-images.mjs +95 -0
- package/package.json +4 -2
- package/template/README.md.tmpl +13 -0
- package/template/package.json.tmpl +2 -1
package/README.md
CHANGED
|
@@ -89,6 +89,96 @@ A consuming site typically wires these up as `npm run add-collection` / `npm run
|
|
|
89
89
|
|
|
90
90
|
Both commands are multilingual-aware. `scms-add-collection` optionally takes a list of locale codes and scaffolds one sample per `src/content/<name>/<locale>/` folder; `scms-add-content` detects those language folders (every sub-folder is a locale code, nothing loose at the collection root) and asks which language the new file belongs in — a single locale, a comma-separated subset, or `all`.
|
|
91
91
|
|
|
92
|
+
## Image optimization
|
|
93
|
+
|
|
94
|
+
`scms-optimize-images` is a maintenance command (also a `bin`, run from the site's project root) that keeps the images in your content light: it converts every JPG/JPEG/PNG under the content folder to WebP, scales them down to a maximum size, deletes the originals and rewrites every reference to them in your `.md`/`.mdx` files. It uses [sharp](https://sharp.pixelplumbing.com/), so nothing else needs to be installed on the machine.
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
npx scms-optimize-images # convert, delete the originals, rewrite references
|
|
98
|
+
npx scms-optimize-images --dry-run # print the estimated savings, touch nothing
|
|
99
|
+
npx scms-optimize-images --check # touch nothing; exit 1 if something is left to do
|
|
100
|
+
npx scms-optimize-images --only a.jpg b.png # limit the work to these files
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Wire it up in the site's `package.json`:
|
|
104
|
+
|
|
105
|
+
```json
|
|
106
|
+
{
|
|
107
|
+
"scripts": {
|
|
108
|
+
"images": "scms-optimize-images",
|
|
109
|
+
"prebuild": "scms-optimize-images --check"
|
|
110
|
+
}
|
|
111
|
+
}
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
`--check` never changes anything — the build must not modify or delete source files. It exits with a non-zero code (and lists the offenders) if it finds a JPG/PNG that could be converted, a WebP larger than the maximum size, or a stale reference from code (see below), so a forgotten image fails the build or the CI instead of being published at 5 MB. Fix it locally with `npm run images`, review the diff, commit.
|
|
115
|
+
|
|
116
|
+
### What it does
|
|
117
|
+
|
|
118
|
+
- **Resize**: the long side is limited to `maxSize` (2000 px); images are only ever scaled down. EXIF orientation is applied and metadata stripped.
|
|
119
|
+
- **JPG → lossy WebP** at quality 82. **PNG → lossless WebP**; if the result is still larger than 50% of the original *and* the original is over 500 KB, it falls back to lossy WebP (quality 85, alpha quality 100).
|
|
120
|
+
- **Not lighter, not converted**: when the WebP would not be smaller than the original, the original is left as it is. The verdict is stored (file hash + settings) in `.scms-optimize-images.json` at the project root — **commit it** — so such images are neither re-encoded on every run nor reported again by `--check`. Replace the file, or change a setting, and it is evaluated again.
|
|
121
|
+
- **References** in `.md`/`.mdx` files (frontmatter such as `img:`, ``, relative and absolute paths) are rewritten. Absolute paths are rooted at the content folder, as in `contentAssetsIntegration`. `%20` is decoded and names are compared case-insensitively.
|
|
122
|
+
- **Languages**: pages in a non-default language usually have no assets of their own and reuse those of the default-language twin (`blog/en/post/index.md` citing `a.jpg` that lives in `blog/it/post/`). Such references are resolved by swapping the language folder for the default one — using the languages declared in `userConfig.i18n` (below), not fixed codes.
|
|
123
|
+
- **Name conflicts**: `bdus.jpg` + `bdus.png` would both become `bdus.webp`. The one cited by a `.md`/`.mdx` in the same folder (or its twin in another language) is kept, the other is deleted; if that does not single one out, both are skipped and reported.
|
|
124
|
+
- **WebP already larger than `maxSize`** are scaled down in place (lossless stays lossless, lossy is re-encoded at `quality`), and reported by `--check`.
|
|
125
|
+
- **Idempotent**: a second run changes nothing (and takes a fraction of a second).
|
|
126
|
+
- The report lists converted files, skipped ones (with the reason), deleted duplicates and converted WebP that nothing cites (images in `gallery/` folders are not expected to be cited).
|
|
127
|
+
|
|
128
|
+
### References in code
|
|
129
|
+
|
|
130
|
+
Only `.md`/`.mdx` files are rewritten. Images can also be referenced from `.astro`, `.jsx`, `.tsx`, `.ts`, `.js` and `.mjs` files under `src/` (`<img src="/didattica/cover.jpg">`, `import x from '@content/…'`): those are **never rewritten** — every one that points at an image that is, or was, converted is reported with file and line, and the command exits with an error:
|
|
131
|
+
|
|
132
|
+
```
|
|
133
|
+
src/pages/[locale]/didattica/index.astro:22 /didattica/cover.jpg is now broken: the image was converted → use /didattica/cover.webp
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
The report keeps coming back (also from `--check`) until the code is fixed, so a broken link cannot slip through silently. Paths are resolved from the content folder (`/…`), from the file (`./…`, `../…`) and through the `@content/` and `@user/` aliases.
|
|
137
|
+
|
|
138
|
+
### Configuration
|
|
139
|
+
|
|
140
|
+
All optional; defaults shown. In `src/user.config.mjs`:
|
|
141
|
+
|
|
142
|
+
```js
|
|
143
|
+
export const userConfig = {
|
|
144
|
+
// Languages of the site. Pages in a language other than defaultLocale may reuse
|
|
145
|
+
// the images of the defaultLocale twin. Omit for a monolingual site.
|
|
146
|
+
i18n: {
|
|
147
|
+
defaultLocale: 'it',
|
|
148
|
+
locales: ['it', 'en'],
|
|
149
|
+
},
|
|
150
|
+
|
|
151
|
+
images: {
|
|
152
|
+
contentDir: 'src/content', // scanned for images and .md/.mdx files
|
|
153
|
+
srcDir: 'src', // scanned for references from code
|
|
154
|
+
maxSize: 2000, // px, long side
|
|
155
|
+
quality: 82, // WebP quality for JPGs (and for re-encoded lossy WebP)
|
|
156
|
+
pngLossyRatio: 0.5, // PNG: go lossy if lossless WebP > this fraction of the original...
|
|
157
|
+
pngLossyMinSize: 500, // ...and the original is larger than this many KB
|
|
158
|
+
pngLossyQuality: 85, // quality of that lossy fallback
|
|
159
|
+
exclude: [], // folders left alone: paths relative to contentDir, or plain folder names
|
|
160
|
+
},
|
|
161
|
+
};
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
Keep `i18n` in sync with the site's own locale list (e.g. `LOCALES`/`DEFAULT_LOCALE` in `src/utils/i18n.ts`). Images outside the content folder (e.g. `public/`) are never touched.
|
|
165
|
+
|
|
166
|
+
### Optional: pre-commit hook
|
|
167
|
+
|
|
168
|
+
Nothing is installed automatically. If you want images checked at commit time, add `.git/hooks/pre-commit` (or the equivalent in husky/lefthook) and make it executable. It only looks at the images staged in the commit:
|
|
169
|
+
|
|
170
|
+
```sh
|
|
171
|
+
#!/bin/sh
|
|
172
|
+
# Block the commit if a staged image is not optimized
|
|
173
|
+
git diff --cached --name-only --diff-filter=AM -z -- '*.jpg' '*.jpeg' '*.png' '*.JPG' '*.JPEG' '*.PNG' '*.webp' \
|
|
174
|
+
| xargs -0 npx scms-optimize-images --check --only || {
|
|
175
|
+
echo "Run 'npm run images', review the changes and stage them." >&2
|
|
176
|
+
exit 1
|
|
177
|
+
}
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
An empty list is fine: `--only` with no files does nothing and exits 0. To convert instead of just blocking, run `npx scms-optimize-images --only <files>` and then `git add` the converted `.webp` files and the rewritten `.md`/`.mdx` — the command never stages anything itself.
|
|
181
|
+
|
|
92
182
|
## Development
|
|
93
183
|
|
|
94
184
|
```bash
|
|
@@ -0,0 +1,440 @@
|
|
|
1
|
+
// optimize-images engine — shared by bin/optimize-images.mjs and the tests.
|
|
2
|
+
//
|
|
3
|
+
// Converts the JPG/PNG images found under the content directory to WebP (max
|
|
4
|
+
// `maxSize` px on the long side), deletes the originals and rewrites every
|
|
5
|
+
// reference to them in the .md/.mdx files. Pure Node + sharp: no side effects
|
|
6
|
+
// besides the file system, no console output (the CLI prints the report).
|
|
7
|
+
//
|
|
8
|
+
// See the "Image optimization" section of the package README for the rules.
|
|
9
|
+
|
|
10
|
+
import crypto from 'node:crypto';
|
|
11
|
+
import fs from 'node:fs';
|
|
12
|
+
import path from 'node:path';
|
|
13
|
+
import sharp from 'sharp';
|
|
14
|
+
|
|
15
|
+
export const DEFAULTS = {
|
|
16
|
+
/** Folder (relative to the project root) scanned for images and .md/.mdx files. */
|
|
17
|
+
contentDir: 'src/content',
|
|
18
|
+
/** Folder (relative to the project root) scanned for code files referencing images. */
|
|
19
|
+
srcDir: 'src',
|
|
20
|
+
/** Max size, in px, of the long side. Images are only ever scaled down. */
|
|
21
|
+
maxSize: 2000,
|
|
22
|
+
/** WebP quality for JPGs. */
|
|
23
|
+
quality: 82,
|
|
24
|
+
/** PNG: fall back to lossy when the lossless WebP is larger than this fraction of the original... */
|
|
25
|
+
pngLossyRatio: 0.5,
|
|
26
|
+
/** ...and the original is larger than this many KB. */
|
|
27
|
+
pngLossyMinSize: 500,
|
|
28
|
+
/** Quality of the lossy PNG fallback. */
|
|
29
|
+
pngLossyQuality: 85,
|
|
30
|
+
/** Folders (relative to contentDir, or plain folder names) left untouched. */
|
|
31
|
+
exclude: [],
|
|
32
|
+
};
|
|
33
|
+
|
|
34
|
+
export const MANIFEST_NAME = '.scms-optimize-images.json';
|
|
35
|
+
|
|
36
|
+
const IMAGE_RE = /\.(jpe?g|png)$/i;
|
|
37
|
+
const TEXT_RE = /\.mdx?$/i;
|
|
38
|
+
const CODE_RE = /\.(astro|jsx|tsx|mjs|ts|js)$/i;
|
|
39
|
+
// A reference token inside a markdown/frontmatter line (no whitespace, quotes or brackets)
|
|
40
|
+
const MD_REF_RE = /[^\s"'()<>[\]=]+\.(?:jpe?g|png)(?![\w.])/gi;
|
|
41
|
+
// Same for code, where the token can also be delimited by template-literal / JSX / object punctuation
|
|
42
|
+
const CODE_REF_RE = /[^\s"'`()<>[\]={},;]+\.(?:jpe?g|png)(?![\w.])/gi;
|
|
43
|
+
|
|
44
|
+
const posix = (p) => p.split(path.sep).join('/');
|
|
45
|
+
const kb = (bytes) => (bytes / 1024).toFixed(0);
|
|
46
|
+
|
|
47
|
+
/** Merges `userConfig.images` / `userConfig.i18n` (either may be undefined) over the defaults. */
|
|
48
|
+
export function resolveConfig(root, userConfig = {}) {
|
|
49
|
+
const images = { ...DEFAULTS, ...(userConfig.images ?? {}) };
|
|
50
|
+
const i18n = userConfig.i18n ?? {};
|
|
51
|
+
const locales = Array.isArray(i18n.locales) ? i18n.locales : [];
|
|
52
|
+
const defaultLocale = i18n.defaultLocale ?? locales[0];
|
|
53
|
+
return {
|
|
54
|
+
root,
|
|
55
|
+
contentRoot: path.resolve(root, images.contentDir),
|
|
56
|
+
srcRoot: path.resolve(root, images.srcDir),
|
|
57
|
+
maxSize: images.maxSize,
|
|
58
|
+
quality: images.quality,
|
|
59
|
+
pngLossyRatio: images.pngLossyRatio,
|
|
60
|
+
pngLossyMinSize: images.pngLossyMinSize,
|
|
61
|
+
pngLossyQuality: images.pngLossyQuality,
|
|
62
|
+
exclude: (images.exclude ?? []).map((e) => posix(e).replace(/^\.?\/+|\/+$/g, '')),
|
|
63
|
+
locales,
|
|
64
|
+
defaultLocale,
|
|
65
|
+
};
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
function walk(dir) {
|
|
69
|
+
if (!fs.existsSync(dir)) return [];
|
|
70
|
+
return fs
|
|
71
|
+
.readdirSync(dir, { withFileTypes: true })
|
|
72
|
+
.sort((a, b) => a.name.localeCompare(b.name))
|
|
73
|
+
.flatMap((e) => {
|
|
74
|
+
if (e.name === 'node_modules' || e.name.startsWith('.')) return [];
|
|
75
|
+
const p = path.join(dir, e.name);
|
|
76
|
+
return e.isDirectory() ? walk(p) : [p];
|
|
77
|
+
});
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
function isExcluded(c, rel) {
|
|
81
|
+
const segments = rel.split('/');
|
|
82
|
+
return c.exclude.some((e) => rel === e || rel.startsWith(`${e}/`) || (!e.includes('/') && segments.slice(0, -1).includes(e)));
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
// ─── Locale twins ────────────────────────────────────────────────────────────
|
|
86
|
+
|
|
87
|
+
/**
|
|
88
|
+
* Alternative locations of a path: pages in a non-default locale have no assets
|
|
89
|
+
* of their own and reuse the files of their default-locale twin, so the first
|
|
90
|
+
* locale segment of a non-default language is swapped for the default one.
|
|
91
|
+
* `relPath` is relative to the content root, posix-style.
|
|
92
|
+
*/
|
|
93
|
+
function defaultLocaleTwin(c, relPath) {
|
|
94
|
+
if (!c.defaultLocale) return null;
|
|
95
|
+
const segments = relPath.split('/');
|
|
96
|
+
const i = segments.findIndex((s) => s !== c.defaultLocale && c.locales.includes(s));
|
|
97
|
+
if (i === -1) return null;
|
|
98
|
+
segments[i] = c.defaultLocale;
|
|
99
|
+
return segments.join('/');
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
/** The folder itself plus its twins in the other languages (used to find which images a page cites). */
|
|
103
|
+
function twinDirs(c, dir) {
|
|
104
|
+
const rel = posix(path.relative(c.contentRoot, dir));
|
|
105
|
+
const segments = rel.split('/');
|
|
106
|
+
const out = [dir];
|
|
107
|
+
const i = c.defaultLocale ? segments.indexOf(c.defaultLocale) : -1;
|
|
108
|
+
if (i !== -1) {
|
|
109
|
+
for (const l of c.locales.filter((l) => l !== c.defaultLocale)) {
|
|
110
|
+
const s = [...segments];
|
|
111
|
+
s[i] = l;
|
|
112
|
+
out.push(path.join(c.contentRoot, ...s));
|
|
113
|
+
}
|
|
114
|
+
}
|
|
115
|
+
return out;
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
// ─── Manifest (images known not to benefit) ──────────────────────────────────
|
|
119
|
+
|
|
120
|
+
const sha1 = (file) => crypto.createHash('sha1').update(fs.readFileSync(file)).digest('hex');
|
|
121
|
+
|
|
122
|
+
function settingsHash(c) {
|
|
123
|
+
const s = [c.maxSize, c.quality, c.pngLossyRatio, c.pngLossyMinSize, c.pngLossyQuality].join('|');
|
|
124
|
+
return crypto.createHash('sha1').update(s).digest('hex').slice(0, 10);
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
function loadManifest(c) {
|
|
128
|
+
const file = path.join(c.root, MANIFEST_NAME);
|
|
129
|
+
try {
|
|
130
|
+
const m = JSON.parse(fs.readFileSync(file, 'utf8'));
|
|
131
|
+
return m && m.settings === settingsHash(c) && m.kept ? m.kept : {};
|
|
132
|
+
} catch {
|
|
133
|
+
return {};
|
|
134
|
+
}
|
|
135
|
+
}
|
|
136
|
+
|
|
137
|
+
function saveManifest(c, kept) {
|
|
138
|
+
const file = path.join(c.root, MANIFEST_NAME);
|
|
139
|
+
const keys = Object.keys(kept).sort();
|
|
140
|
+
if (!keys.length) {
|
|
141
|
+
fs.rmSync(file, { force: true });
|
|
142
|
+
return;
|
|
143
|
+
}
|
|
144
|
+
const sorted = Object.fromEntries(keys.map((k) => [k, kept[k]]));
|
|
145
|
+
fs.writeFileSync(file, `${JSON.stringify({ settings: settingsHash(c), kept: sorted }, null, 2)}\n`);
|
|
146
|
+
}
|
|
147
|
+
|
|
148
|
+
// ─── sharp ───────────────────────────────────────────────────────────────────
|
|
149
|
+
|
|
150
|
+
// Inputs are always passed to sharp as buffers, never as paths: libvips caches by file path
|
|
151
|
+
// and would serve stale metadata for a file rewritten earlier in the same process.
|
|
152
|
+
function pipeline(input, c) {
|
|
153
|
+
return sharp(Buffer.isBuffer(input) ? input : fs.readFileSync(input), { failOn: 'error' })
|
|
154
|
+
.rotate()
|
|
155
|
+
.resize({ width: c.maxSize, height: c.maxSize, fit: 'inside', withoutEnlargement: true });
|
|
156
|
+
}
|
|
157
|
+
|
|
158
|
+
/** Encodes a JPG/PNG to WebP. Same lossless/lossy logic as the original ImageMagick script. */
|
|
159
|
+
async function encode(src, c) {
|
|
160
|
+
if (!/\.png$/i.test(src)) return pipeline(src, c).webp({ quality: c.quality }).toBuffer();
|
|
161
|
+
// PNGs are mostly screenshots and graphics: lossless first, lossy if that is still heavy
|
|
162
|
+
const original = fs.statSync(src).size;
|
|
163
|
+
let out = await pipeline(src, c).webp({ lossless: true, effort: 6 }).toBuffer();
|
|
164
|
+
if (out.length > original * c.pngLossyRatio && original > c.pngLossyMinSize * 1024) {
|
|
165
|
+
out = await pipeline(src, c).webp({ quality: c.pngLossyQuality, alphaQuality: 100 }).toBuffer();
|
|
166
|
+
}
|
|
167
|
+
return out;
|
|
168
|
+
}
|
|
169
|
+
|
|
170
|
+
/** True for a lossless WebP (VP8L chunk), false for lossy (VP8 chunk). */
|
|
171
|
+
function webpIsLossless(buf) {
|
|
172
|
+
let offset = 12;
|
|
173
|
+
while (offset + 8 <= buf.length) {
|
|
174
|
+
const fourcc = buf.toString('ascii', offset, offset + 4);
|
|
175
|
+
if (fourcc === 'VP8L') return true;
|
|
176
|
+
if (fourcc === 'VP8 ') return false;
|
|
177
|
+
const size = buf.readUInt32LE(offset + 4);
|
|
178
|
+
offset += 8 + size + (size & 1);
|
|
179
|
+
}
|
|
180
|
+
return false;
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
async function resizeWebp(file, c) {
|
|
184
|
+
const buf = fs.readFileSync(file);
|
|
185
|
+
const opts = webpIsLossless(buf) ? { lossless: true, effort: 6 } : { quality: c.quality, alphaQuality: 100 };
|
|
186
|
+
return { before: buf.length, out: await pipeline(buf, c).webp(opts).toBuffer() };
|
|
187
|
+
}
|
|
188
|
+
|
|
189
|
+
async function pixelSize(file) {
|
|
190
|
+
const m = await sharp(fs.readFileSync(file), { failOn: 'error' }).metadata();
|
|
191
|
+
return { max: Math.max(m.width ?? 0, m.height ?? 0), animated: (m.pages ?? 1) > 1 };
|
|
192
|
+
}
|
|
193
|
+
|
|
194
|
+
// ─── Main ────────────────────────────────────────────────────────────────────
|
|
195
|
+
|
|
196
|
+
/**
|
|
197
|
+
* @param {ReturnType<typeof resolveConfig>} c
|
|
198
|
+
* @param {{ dryRun?: boolean, check?: boolean, only?: string[] | null }} [mode]
|
|
199
|
+
* `check`: touch nothing and convert nothing, just list what is left to do.
|
|
200
|
+
* `only`: restrict the images that get converted/checked to these files.
|
|
201
|
+
*/
|
|
202
|
+
export async function optimizeImages(c, { dryRun = false, check = false, only = null } = {}) {
|
|
203
|
+
const readOnly = dryRun || check;
|
|
204
|
+
const rel = (f) => posix(path.relative(c.contentRoot, f));
|
|
205
|
+
const onlySet = only ? new Set(only.map((f) => path.resolve(f).toLowerCase())) : null;
|
|
206
|
+
const inScope = (f) => !onlySet || onlySet.has(f.toLowerCase());
|
|
207
|
+
|
|
208
|
+
const report = {
|
|
209
|
+
converted: [], // { rel, before, after }
|
|
210
|
+
skipped: [], // { rel, reason }
|
|
211
|
+
dropped: [], // unused duplicates (rel)
|
|
212
|
+
resized: [], // oversized WebPs scaled down: { rel, before, after }
|
|
213
|
+
pending: [], // --check: JPG/PNG still to convert (rel)
|
|
214
|
+
oversized: [], // --check: WebP larger than maxSize (rel)
|
|
215
|
+
unreferenced: [],
|
|
216
|
+
codeRefs: [], // { file, line, ref, kind, replacement }
|
|
217
|
+
keptCount: 0, // already known not to benefit, left alone
|
|
218
|
+
rewritten: 0,
|
|
219
|
+
before: 0,
|
|
220
|
+
after: 0,
|
|
221
|
+
problems: 0,
|
|
222
|
+
};
|
|
223
|
+
|
|
224
|
+
const all = walk(c.contentRoot).filter((f) => !isExcluded(c, rel(f)));
|
|
225
|
+
const textFiles = all.filter((f) => TEXT_RE.test(f));
|
|
226
|
+
const allImages = all.filter((f) => IMAGE_RE.test(f));
|
|
227
|
+
const webps = all.filter((f) => /\.webp$/i.test(f));
|
|
228
|
+
|
|
229
|
+
const kept = loadManifest(c);
|
|
230
|
+
const keptKey = (f) => rel(f);
|
|
231
|
+
const isKept = (f) => {
|
|
232
|
+
const e = kept[keptKey(f)];
|
|
233
|
+
return Boolean(e) && e.sha1 === sha1(f);
|
|
234
|
+
};
|
|
235
|
+
const markKept = (f, reason) => {
|
|
236
|
+
kept[keptKey(f)] = { sha1: sha1(f), reason };
|
|
237
|
+
};
|
|
238
|
+
|
|
239
|
+
const converted = new Map(); // old absolute path -> new absolute path
|
|
240
|
+
|
|
241
|
+
if (check) {
|
|
242
|
+
for (const f of allImages) {
|
|
243
|
+
if (!inScope(f)) continue;
|
|
244
|
+
if (isKept(f)) report.keptCount++;
|
|
245
|
+
else report.pending.push(rel(f));
|
|
246
|
+
}
|
|
247
|
+
for (const f of webps) {
|
|
248
|
+
if (!inScope(f) || isKept(f)) continue;
|
|
249
|
+
const { max, animated } = await pixelSize(f).catch(() => ({ max: 0, animated: true }));
|
|
250
|
+
if (!animated && max > c.maxSize) report.oversized.push(rel(f));
|
|
251
|
+
}
|
|
252
|
+
for (const f of allImages.filter((f) => inScope(f) && !isKept(f))) converted.set(f, f.replace(IMAGE_RE, '.webp'));
|
|
253
|
+
} else {
|
|
254
|
+
// Same name, different extension in one folder (e.g. bdus.jpg + bdus.png): they would both
|
|
255
|
+
// become bdus.webp. Keep only the one a sibling .md/.mdx (or its locale twin) points to.
|
|
256
|
+
let images = allImages;
|
|
257
|
+
const byTarget = new Map();
|
|
258
|
+
for (const f of images) {
|
|
259
|
+
const k = f.replace(IMAGE_RE, '');
|
|
260
|
+
byTarget.set(k, [...(byTarget.get(k) ?? []), f]);
|
|
261
|
+
}
|
|
262
|
+
for (const group of [...byTarget.values()].filter((g) => g.length > 1)) {
|
|
263
|
+
const dirs = twinDirs(c, path.dirname(group[0]));
|
|
264
|
+
const text = dirs
|
|
265
|
+
.flatMap((d) => textFiles.filter((t) => path.dirname(t) === d))
|
|
266
|
+
.map((t) => fs.readFileSync(t, 'utf8'))
|
|
267
|
+
.join('\n');
|
|
268
|
+
const used = group.filter((g) => text.includes(path.basename(g)));
|
|
269
|
+
if (used.length !== 1) continue; // ambiguous: reported as skipped below
|
|
270
|
+
for (const unused of group.filter((g) => g !== used[0])) {
|
|
271
|
+
if (!inScope(unused)) continue;
|
|
272
|
+
report.dropped.push(rel(unused));
|
|
273
|
+
if (!readOnly) fs.rmSync(unused);
|
|
274
|
+
}
|
|
275
|
+
images = images.filter((i) => !group.includes(i) || i === used[0]);
|
|
276
|
+
}
|
|
277
|
+
|
|
278
|
+
for (const src of images) {
|
|
279
|
+
if (!inScope(src)) continue;
|
|
280
|
+
const dest = src.replace(IMAGE_RE, '.webp');
|
|
281
|
+
const name = rel(src);
|
|
282
|
+
|
|
283
|
+
if (isKept(src)) {
|
|
284
|
+
report.keptCount++;
|
|
285
|
+
continue;
|
|
286
|
+
}
|
|
287
|
+
if (fs.existsSync(dest) || images.some((o) => o !== src && o.replace(IMAGE_RE, '.webp') === dest)) {
|
|
288
|
+
report.skipped.push({ rel: name, reason: 'target .webp already exists or name clashes with another image' });
|
|
289
|
+
continue;
|
|
290
|
+
}
|
|
291
|
+
|
|
292
|
+
const origSize = fs.statSync(src).size;
|
|
293
|
+
let out;
|
|
294
|
+
try {
|
|
295
|
+
const { animated } = await pixelSize(src);
|
|
296
|
+
if (animated) {
|
|
297
|
+
report.skipped.push({ rel: name, reason: 'animated image, left as is' });
|
|
298
|
+
continue;
|
|
299
|
+
}
|
|
300
|
+
out = await encode(src, c);
|
|
301
|
+
} catch (err) {
|
|
302
|
+
report.skipped.push({ rel: name, reason: `conversion failed (${err.message.split('\n')[0]})` });
|
|
303
|
+
continue;
|
|
304
|
+
}
|
|
305
|
+
if (out.length >= origSize) {
|
|
306
|
+
report.skipped.push({ rel: name, reason: `WebP is not smaller (${kb(origSize)} -> ${kb(out.length)} KB), kept as is` });
|
|
307
|
+
if (!readOnly) markKept(src, 'webp-not-smaller');
|
|
308
|
+
continue;
|
|
309
|
+
}
|
|
310
|
+
|
|
311
|
+
report.before += origSize;
|
|
312
|
+
report.after += out.length;
|
|
313
|
+
converted.set(src, dest);
|
|
314
|
+
report.converted.push({ rel: name, before: origSize, after: out.length });
|
|
315
|
+
if (!readOnly) {
|
|
316
|
+
const tmp = `${dest}.tmp`;
|
|
317
|
+
fs.writeFileSync(tmp, out);
|
|
318
|
+
fs.renameSync(tmp, dest);
|
|
319
|
+
fs.rmSync(src);
|
|
320
|
+
}
|
|
321
|
+
}
|
|
322
|
+
|
|
323
|
+
// WebPs that were already there but exceed the max size are scaled down in place
|
|
324
|
+
// (same mode as the source: lossless stays lossless, lossy is re-encoded at `quality`).
|
|
325
|
+
for (const f of webps) {
|
|
326
|
+
if (!inScope(f)) continue;
|
|
327
|
+
if (isKept(f)) {
|
|
328
|
+
report.keptCount++;
|
|
329
|
+
continue;
|
|
330
|
+
}
|
|
331
|
+
try {
|
|
332
|
+
const { max, animated } = await pixelSize(f);
|
|
333
|
+
if (animated || max <= c.maxSize) continue;
|
|
334
|
+
const { before, out } = await resizeWebp(f, c);
|
|
335
|
+
if (out.length >= before) {
|
|
336
|
+
report.skipped.push({ rel: rel(f), reason: 'oversized WebP, but scaling it down does not make it smaller, kept as is' });
|
|
337
|
+
if (!readOnly) markKept(f, 'resize-not-smaller');
|
|
338
|
+
continue;
|
|
339
|
+
}
|
|
340
|
+
report.resized.push({ rel: rel(f), before, after: out.length });
|
|
341
|
+
if (!readOnly) fs.writeFileSync(f, out);
|
|
342
|
+
} catch (err) {
|
|
343
|
+
report.skipped.push({ rel: rel(f), reason: `could not inspect WebP (${err.message.split('\n')[0]})` });
|
|
344
|
+
}
|
|
345
|
+
}
|
|
346
|
+
|
|
347
|
+
if (!readOnly) {
|
|
348
|
+
// prune entries whose file is gone (or no longer the one that was evaluated)
|
|
349
|
+
for (const k of Object.keys(kept)) {
|
|
350
|
+
const abs = path.join(c.contentRoot, ...k.split('/'));
|
|
351
|
+
if (!fs.existsSync(abs) || sha1(abs) !== kept[k].sha1) delete kept[k];
|
|
352
|
+
}
|
|
353
|
+
saveManifest(c, kept);
|
|
354
|
+
}
|
|
355
|
+
|
|
356
|
+
// Rewrite references in .md/.mdx
|
|
357
|
+
const lookup = new Map([...converted].map(([o, n]) => [o.toLowerCase(), n]));
|
|
358
|
+
for (const file of textFiles) {
|
|
359
|
+
const dir = path.dirname(file);
|
|
360
|
+
const text = fs.readFileSync(file, 'utf8');
|
|
361
|
+
const next = text.replace(MD_REF_RE, (ref) => {
|
|
362
|
+
if (/^(https?:)?\/\//i.test(ref)) return ref;
|
|
363
|
+
let decoded = ref;
|
|
364
|
+
try {
|
|
365
|
+
decoded = decodeURIComponent(ref);
|
|
366
|
+
} catch {
|
|
367
|
+
/* keep raw */
|
|
368
|
+
}
|
|
369
|
+
// Relative to the file, or absolute from the content root (see contentAssetsIntegration)
|
|
370
|
+
const target = decoded.startsWith('/') ? path.join(c.contentRoot, decoded) : path.resolve(dir, decoded);
|
|
371
|
+
// Pages in a non-default language have no assets of their own: they reuse the twin's files
|
|
372
|
+
const twinRel = defaultLocaleTwin(c, rel(target));
|
|
373
|
+
const twin = twinRel ? path.join(c.contentRoot, ...twinRel.split('/')) : null;
|
|
374
|
+
const hit = lookup.get(target.toLowerCase()) ?? (twin ? lookup.get(twin.toLowerCase()) : undefined);
|
|
375
|
+
return hit ? ref.replace(IMAGE_RE, '.webp') : ref;
|
|
376
|
+
});
|
|
377
|
+
if (next !== text) {
|
|
378
|
+
report.rewritten++;
|
|
379
|
+
if (!readOnly) fs.writeFileSync(file, next);
|
|
380
|
+
}
|
|
381
|
+
}
|
|
382
|
+
}
|
|
383
|
+
|
|
384
|
+
// ─── References from code (reported, never rewritten) ──────────────────────
|
|
385
|
+
const codeFiles = walk(c.srcRoot).filter((f) => CODE_RE.test(f) && !f.startsWith(c.contentRoot + path.sep));
|
|
386
|
+
const existing = new Set(all.map((f) => f.toLowerCase()));
|
|
387
|
+
const convertedLower = new Map([...converted].map(([o, n]) => [o.toLowerCase(), n]));
|
|
388
|
+
const kind = check || dryRun ? 'to-convert' : 'converted';
|
|
389
|
+
|
|
390
|
+
for (const file of codeFiles) {
|
|
391
|
+
const dir = path.dirname(file);
|
|
392
|
+
const lines = fs.readFileSync(file, 'utf8').split('\n');
|
|
393
|
+
lines.forEach((line, i) => {
|
|
394
|
+
for (const [ref] of line.matchAll(CODE_REF_RE)) {
|
|
395
|
+
if (/^(https?:)?\/\//i.test(ref)) continue;
|
|
396
|
+
let decoded = ref;
|
|
397
|
+
try {
|
|
398
|
+
decoded = decodeURIComponent(ref);
|
|
399
|
+
} catch {
|
|
400
|
+
/* keep raw */
|
|
401
|
+
}
|
|
402
|
+
const candidates = decoded.startsWith('@content/')
|
|
403
|
+
? [path.join(c.contentRoot, decoded.slice(9))]
|
|
404
|
+
: decoded.startsWith('@user/')
|
|
405
|
+
? [path.join(c.srcRoot, decoded.slice(6))]
|
|
406
|
+
: decoded.startsWith('/')
|
|
407
|
+
? [path.join(c.contentRoot, decoded)]
|
|
408
|
+
: [path.resolve(dir, decoded), path.join(c.contentRoot, decoded)];
|
|
409
|
+
let found = null;
|
|
410
|
+
for (const cand of candidates) {
|
|
411
|
+
const lower = cand.toLowerCase();
|
|
412
|
+
if (convertedLower.has(lower)) {
|
|
413
|
+
found = kind;
|
|
414
|
+
break;
|
|
415
|
+
}
|
|
416
|
+
// a JPG/PNG that no longer exists although its .webp does: left over from an earlier conversion
|
|
417
|
+
if (!existing.has(lower) && existing.has(lower.replace(IMAGE_RE, '.webp'))) {
|
|
418
|
+
found = 'stale';
|
|
419
|
+
break;
|
|
420
|
+
}
|
|
421
|
+
}
|
|
422
|
+
if (found) {
|
|
423
|
+
report.codeRefs.push({ file: posix(path.relative(c.root, file)), line: i + 1, ref, kind: found, replacement: ref.replace(IMAGE_RE, '.webp') });
|
|
424
|
+
}
|
|
425
|
+
}
|
|
426
|
+
});
|
|
427
|
+
}
|
|
428
|
+
|
|
429
|
+
// Sanity check: every converted file must be referenced or live in a gallery/ folder
|
|
430
|
+
if (!readOnly) {
|
|
431
|
+
const corpus = [...textFiles, ...codeFiles].map((f) => fs.readFileSync(f, 'utf8')).join('\n');
|
|
432
|
+
report.unreferenced = [...converted.values()]
|
|
433
|
+
.filter((d) => !rel(d).split('/').includes('gallery'))
|
|
434
|
+
.filter((d) => !corpus.includes(path.basename(d)))
|
|
435
|
+
.map(rel);
|
|
436
|
+
}
|
|
437
|
+
|
|
438
|
+
report.problems = check ? report.pending.length + report.oversized.length + report.codeRefs.length : report.codeRefs.length;
|
|
439
|
+
return report;
|
|
440
|
+
}
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// optimize-images.mjs — Convert content images to WebP and rewrite their references.
|
|
3
|
+
// Usage: npx scms-optimize-images [--dry-run | --check] [--only <file>...]
|
|
4
|
+
// or: npm run images (from a consuming site's package.json)
|
|
5
|
+
|
|
6
|
+
import fs from 'node:fs';
|
|
7
|
+
import path from 'node:path';
|
|
8
|
+
import { pathToFileURL } from 'node:url';
|
|
9
|
+
import { optimizeImages, resolveConfig } from './lib/optimize-images.mjs';
|
|
10
|
+
|
|
11
|
+
const G = '\x1b[32m', Y = '\x1b[33m', R = '\x1b[31m';
|
|
12
|
+
const B = '\x1b[1m', X = '\x1b[0m';
|
|
13
|
+
|
|
14
|
+
const HELP = `scms-optimize-images — convert content JPG/PNG images to WebP
|
|
15
|
+
|
|
16
|
+
scms-optimize-images convert, delete the originals, rewrite references
|
|
17
|
+
scms-optimize-images --dry-run report what would happen, touch nothing
|
|
18
|
+
scms-optimize-images --check touch nothing; exit 1 if anything is left to convert
|
|
19
|
+
(JPG/PNG, WebP larger than the max size, stale code references).
|
|
20
|
+
Meant for "prebuild" scripts and CI.
|
|
21
|
+
scms-optimize-images --only <files> limit conversion/checks to these image files
|
|
22
|
+
(e.g. the ones staged in a pre-commit hook)
|
|
23
|
+
|
|
24
|
+
Settings live in src/user.config.mjs (userConfig.images, userConfig.i18n); see the scms-core README.`;
|
|
25
|
+
|
|
26
|
+
const args = process.argv.slice(2);
|
|
27
|
+
if (args.includes('--help') || args.includes('-h')) {
|
|
28
|
+
console.log(HELP);
|
|
29
|
+
process.exit(0);
|
|
30
|
+
}
|
|
31
|
+
const dryRun = args.includes('--dry-run');
|
|
32
|
+
const check = args.includes('--check');
|
|
33
|
+
const onlyIdx = args.indexOf('--only');
|
|
34
|
+
const only = onlyIdx === -1 ? null : args.slice(onlyIdx + 1).filter((a) => !a.startsWith('--'));
|
|
35
|
+
const unknown = args.filter((a) => a.startsWith('-') && !['--dry-run', '--check', '--only'].includes(a));
|
|
36
|
+
if (unknown.length) {
|
|
37
|
+
console.error(`${R}✖${X} Unknown option: ${unknown.join(' ')}\n\n${HELP}`);
|
|
38
|
+
process.exit(2);
|
|
39
|
+
}
|
|
40
|
+
if (only && only.length === 0) {
|
|
41
|
+
console.log('No files given to --only, nothing to do.');
|
|
42
|
+
process.exit(0);
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
// This file ships inside the @lad-sapienza/scms-core package (bin/): the project is wherever it was run from.
|
|
46
|
+
const ROOT = process.cwd();
|
|
47
|
+
const configFile = path.join(ROOT, 'src', 'user.config.mjs');
|
|
48
|
+
let userConfig = {};
|
|
49
|
+
if (fs.existsSync(configFile)) {
|
|
50
|
+
try {
|
|
51
|
+
userConfig = (await import(pathToFileURL(configFile).href)).userConfig ?? {};
|
|
52
|
+
} catch (err) {
|
|
53
|
+
console.error(`${R}✖${X} Cannot load src/user.config.mjs: ${err.message}`);
|
|
54
|
+
process.exit(2);
|
|
55
|
+
}
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
const config = resolveConfig(ROOT, userConfig);
|
|
59
|
+
if (!fs.existsSync(config.contentRoot)) {
|
|
60
|
+
console.error(`${R}✖${X} Content folder not found: ${config.contentRoot}`);
|
|
61
|
+
process.exit(2);
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
const r = await optimizeImages(config, { dryRun, check, only });
|
|
65
|
+
|
|
66
|
+
const kb = (n) => (n / 1024).toFixed(0);
|
|
67
|
+
const mb = (n) => (n / 1048576).toFixed(1);
|
|
68
|
+
const list = (title, items) => items.length && console.log(`\n${title} (${items.length}):\n ${items.join('\n ')}`);
|
|
69
|
+
const tag = dryRun ? '[dry run] ' : '';
|
|
70
|
+
|
|
71
|
+
if (check) {
|
|
72
|
+
list(`${R}JPG/PNG images to convert${X}`, r.pending);
|
|
73
|
+
list(`${R}WebP images larger than ${config.maxSize}px${X}`, r.oversized);
|
|
74
|
+
} else {
|
|
75
|
+
for (const c of r.converted) console.log(`${c.rel}: ${kb(c.before)} -> ${kb(c.after)} KB`);
|
|
76
|
+
for (const c of r.resized) console.log(`${c.rel}: scaled to max ${config.maxSize}px, ${kb(c.before)} -> ${kb(c.after)} KB`);
|
|
77
|
+
console.log(`\n${tag}${r.converted.length} images converted, ${r.resized.length} resized, ${r.rewritten} text files updated`);
|
|
78
|
+
console.log(`Size: ${mb(r.before)} MB -> ${mb(r.after)} MB${r.before ? ` (-${(100 - (r.after / r.before) * 100).toFixed(0)}%)` : ''}`);
|
|
79
|
+
list(`${B}Unused duplicates ${dryRun ? 'to delete' : 'deleted'}${X}`, r.dropped);
|
|
80
|
+
list(`${Y}Skipped${X}`, r.skipped.map((s) => `${s.rel}: ${s.reason}`));
|
|
81
|
+
list(`${Y}Converted but not referenced anywhere (check)${X}`, r.unreferenced);
|
|
82
|
+
if (r.keptCount) console.log(`\n${r.keptCount} image(s) already evaluated and kept as they are (see .scms-optimize-images.json)`);
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
if (r.codeRefs.length) {
|
|
86
|
+
const verb = { 'to-convert': 'will break once the image is converted', converted: 'is now broken: the image was converted', stale: 'points to an image that was converted earlier' };
|
|
87
|
+
console.error(`\n${R}✖${X} ${r.codeRefs.length} reference(s) in code must be updated by hand (code is never rewritten):`);
|
|
88
|
+
for (const c of r.codeRefs) console.error(` ${c.file}:${c.line} ${c.ref} ${verb[c.kind]} → use ${c.replacement}`);
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
if (r.problems) {
|
|
92
|
+
console.error(`\n${R}✖${X} ${r.problems} problem(s) found.${check ? ' Run `npx scms-optimize-images` to fix the images.' : ''}`);
|
|
93
|
+
process.exit(1);
|
|
94
|
+
}
|
|
95
|
+
if (check) console.log(`${G}✔${X} Images are optimized.`);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@lad-sapienza/scms-core",
|
|
3
|
-
"version": "0.1.0-alpha.
|
|
3
|
+
"version": "0.1.0-alpha.12",
|
|
4
4
|
"description": "Framework components and Astro integrations for s:CMS — DataTb, Gallery, Map, ZoteroGeoViewer, Record.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "BSD-0-Clause",
|
|
@@ -13,7 +13,8 @@
|
|
|
13
13
|
"bin": {
|
|
14
14
|
"scms-add-collection": "./bin/add-collection.mjs",
|
|
15
15
|
"scms-add-content": "./bin/add-content.mjs",
|
|
16
|
-
"scms-create": "./bin/create.mjs"
|
|
16
|
+
"scms-create": "./bin/create.mjs",
|
|
17
|
+
"scms-optimize-images": "./bin/optimize-images.mjs"
|
|
17
18
|
},
|
|
18
19
|
"files": [
|
|
19
20
|
"index.ts",
|
|
@@ -54,6 +55,7 @@
|
|
|
54
55
|
"papaparse": "^5.5.3",
|
|
55
56
|
"photoswipe": "^5.4.4",
|
|
56
57
|
"react-photoswipe-gallery": "^4.0.0",
|
|
58
|
+
"sharp": "^0.35.3",
|
|
57
59
|
"zod": "^4.3.6"
|
|
58
60
|
},
|
|
59
61
|
"devDependencies": {
|
package/template/README.md.tmpl
CHANGED
|
@@ -14,6 +14,19 @@ Built with [s:CMS](https://github.com/lad-sapienza/scms-core), on top of [Astro]
|
|
|
14
14
|
| `npm run preview` | Preview the production build locally |
|
|
15
15
|
| `npm run add-collection` | Scaffold a new content collection |
|
|
16
16
|
| `npm run add-content` | Add a new content file to an existing collection |
|
|
17
|
+
| `npm run images` | Convert content images to WebP (max 2000 px) and rewrite their references |
|
|
18
|
+
|
|
19
|
+
## Optimizing images
|
|
20
|
+
|
|
21
|
+
Drop JPG/PNG images next to your content as usual, then run `npm run images`: they are converted to WebP, scaled down to 2000 px, the originals are deleted and the `.md`/`.mdx` references are rewritten (`--dry-run` shows the savings without touching anything). References from `.astro`/`.tsx` files are only reported — update them by hand.
|
|
22
|
+
|
|
23
|
+
To make the build fail when an unoptimized image slips in, add a check to `package.json`; it never modifies any file:
|
|
24
|
+
|
|
25
|
+
```json
|
|
26
|
+
"prebuild": "scms-optimize-images --check"
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Limits, quality and the site languages are configured in `src/user.config.mjs` (`userConfig.images`, `userConfig.i18n`); see the [scms-core README](https://github.com/lad-sapienza/scms-core#image-optimization) for all the options, including an optional pre-commit hook. Commit the `.scms-optimize-images.json` file it may create.
|
|
17
30
|
|
|
18
31
|
## Updating the framework
|
|
19
32
|
|
|
@@ -10,7 +10,8 @@
|
|
|
10
10
|
"preview": "astro preview",
|
|
11
11
|
"astro": "astro",
|
|
12
12
|
"add-collection": "scms-add-collection",
|
|
13
|
-
"add-content": "scms-add-content"
|
|
13
|
+
"add-content": "scms-add-content",
|
|
14
|
+
"images": "scms-optimize-images"
|
|
14
15
|
},
|
|
15
16
|
"dependencies": {
|
|
16
17
|
"@astrojs/check": "^0.9.10",
|