@tideworthy/tidysites 0.0.0-stage → 0.1.1

package/README.md

@@ -1,3 +1,81 @@
-# Temporary Holding Version
+# tidysites
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+A Tidysites website as files. Pull every record and page to a folder, edit
+them (by hand, with an editor, or with an assistant), check them against the
+website's content model, push the changes back in one write, and look at
+the result. The website is the remote: pull, edit, push.
+
+```
+npx -y @tideworthy/tidysites@latest init <site_id> --host https://app.tideworthy.com --key <key>
+```
+
+The connector prints that command, with the key filled in, from its
+`connect_tidysites_cli` tool. The key reaches the workspace's websites and
+nothing else; it lives in `.tidysites/key`, which is kept out of git.
+
+## Commands
+
+| | |
+|---|---|
+| `tidysites init <site_id> --host <url> --key <key>` | connect this folder to a website and pull it |
+| `tidysites pull [--collections pages,services]` | the website's content to files; files for entries the website no longer has are removed |
+| `tidysites status` | what changed here since the pull |
+| `tidysites validate` | every file against the content model, with the file and the field in each message |
+| `tidysites push [--prune] [--force] [--dry] [--only pages]` | validate, then write the created and updated entries back (with `--prune`, delete the entries whose files are gone), then pull |
+| `tidysites shot <path or url> [--mobile] [--desktop] [--first-screen] [--live]` | a screenshot of a page of the preview site, the live site, or any address, saved under `.tidysites/shots/` |
+| `tidysites pages` | every address the website renders |
+| `tidysites page <path> [--html]` | the rendered page, section by section, with the key each section carries |
+| `tidysites schema [type or collection]` | the section types and layouts, or one type's or collection's fields |
+| `tidysites gallery [type]` | a screenshot of every section type and layout in this website's theme, or the first screen of one type |
+| `tidysites fetch <url> [--out dir]` | a page of the site being ported: cleaned HTML, its outline (headings, navigation, images, phones, fonts, colours) and its stylesheets |
+| `tidysites publish --yes` | the live site becomes a copy of the preview site |
+
+A push that would overwrite an entry changed on the website since the pull
+is refused with the list; pull again (your files are overwritten, so copy
+what you need first) or push `--force`.
+
+## The folder
+
+```
+<collection>/<slug>.json      { "id", "slug", "status", "updated_at", "data": { ... } }
+<collection>/<slug>.css       data.custom_css as a file (for rows: data.css)
+rows/<slug>.html              data.html as a file
+menus/<name>.json             a menu as a tree: { "label", "items": [{ "label", "url", "children": [...] }] }
+.tidysites/                   config.json, key, snapshot.json (the last pull), schema.json (the content model), shots/, source/
+```
+
+A new entry is a new `<collection>/<slug>.json` with at least `"data"`; the
+slug is the file name, the status is `published` unless the file says
+`draft`. Delete a file and `push --prune` removes the entry. A field taken
+out of a file is cleared on the website at the next push. `id` and
+`updated_at` are the website's; leave them as pulled. A pull never overwrites
+a file you changed and have not pushed; it keeps the file and the difference
+shows in `status`.
+
+Images: an image field, a hero's image or a Columns image slot takes
+`{ "$media": { "url": "https://...", "alt": "what it shows" } }` and the
+push fetches the file into the website's library. An image already in the
+library keeps the value the pull gave it.
+
+References: a record names another by its id (what the pull shows), or by
+`"$ref:<slug>"` for one created in the same push.
+
+Rich text (a section's `copy`, a post's `content`) is markdown.
+
+Custom HTML (a row's `.html` file, a Columns HTML slot) reads the records:
+`{{ business.phone }}`, `{{ page.title }}`, `{{#each services as s}}…{{ s.url }}…{{/each}}`,
+`{{#if page.collection == "services"}}…{{/if}}`, and the drop-ins `{{> menu "primary"}}`,
+`{{> logo}}`, `{{> phone}}`, `{{> address}}`, `{{> hours}}`, `{{> social}}`, `{{> year}}`,
+`{{> button "Label" "/url" style=outline}}`, `{{> form "Name"}}`. A link to `#form:<Name>`
+opens that form in a dialog. So a custom header is written once against `page.*` and
+`business.*` and fits every page; `frame/frame.json` names it as `header_row`, or a page
+names its own in `header_row`.
+
+## Building a site from these files
+
+1. `tidysites schema` for the section types, `tidysites gallery` to see them in this theme.
+2. Records first: `business/business.json` (the company's name, tagline, phone, hours, brand), `locations/`, `services/`, `people/`, `reviews/`, `faqs/`.
+3. `frame/frame.json` for the top bar, the call bar, the footer headings, the wording; `menus/primary.json` and `menus/footer.json` for the menus (an item with `children` is a dropdown).
+4. Pages as sections: `pages/<slug>.json`, each a list in `data.sections`. Columns is the section for a band with no record behind it.
+5. The look: brand colours and fonts on the Business record, `business/business.css` for what the tokens cannot say. The theme's own rules are written as `.ts.ts-accent` and `.ts.ts-ink .ts-prose a`, so a rule of yours needs the same weight (`.ts.ts-accent …`) to win.
+6. `tidysites validate`, `tidysites push`, `tidysites shot /` and `tidysites shot / --mobile`, and look.

package/bin/tidysites.mjs

@@ -0,0 +1,506 @@
+#!/usr/bin/env node
+/**
+ * tidysites: a Tidysites website as files.
+ *
+ *   init <site_id> --host <url> --key <key>   connect this folder to a website, then pull
+ *   pull [--collections a,b]                  the website's content to files
+ *   status                                    what changed here since the pull
+ *   validate                                  check the files against the content model
+ *   push [--prune] [--force] [--dry]          write the changes back, then pull
+ *   shot <path|url> [--mobile|--desktop] [--first-screen]   screenshot to .tidysites/shots
+ *   pages                                     every address the website renders
+ *   page <path> [--html]                      the rendered page, section by section
+ *   schema [name]                             section types and layouts, or one type's or collection's fields
+ *   gallery [type]                            screenshot of every section type in this theme, or the first screen of one type
+ *   fetch <url> [--out dir]                   a page of the site being ported: cleaned HTML and its outline
+ *   publish --yes                             the live site becomes a copy of the preview site
+ *
+ * The site is the remote: pull, edit, push. A push that would overwrite an
+ * entry changed on the website since the pull is refused until you pull
+ * again (or say --force).
+ */
+import { existsSync, mkdirSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs';
+import { dirname, join, relative, resolve } from 'node:path';
+import { callTool, ApiError } from '../lib/api.mjs';
+import { diffEntries, entryToFiles, filesToEntry, FILE_NAME, menuItems, menuTree, pushDocument, sameEntry, snapshotOf, SIDECARS } from '../lib/folder.mjs';
+import { validateAll, indexSchema } from '../lib/validate.mjs';
+import { cleanHtml, cssSummary, fetchPage, outline, outlineText } from '../lib/fetch.mjs';
+
+const META = '.tidysites';
+const argv = process.argv.slice(2);
+const command = argv[0];
+const flags = {};
+const positional = [];
+for (let i = 1; i < argv.length; i++) {
+	const a = argv[i];
+	if (a.startsWith('--')) {
+		const [k, inline] = a.slice(2).split('=');
+		if (inline !== undefined) flags[k] = inline;
+		else if (argv[i + 1] !== undefined && !argv[i + 1].startsWith('--')) flags[k] = argv[++i];
+		else flags[k] = true;
+	} else positional.push(a);
+}
+
+const say = (...a) => console.log(...a);
+const fail = (msg, code = 1) => {
+	console.error(msg);
+	process.exit(code);
+};
+
+function root() {
+	let dir = resolve(flags.dir ?? process.cwd());
+	for (;;) {
+		if (existsSync(join(dir, META, 'config.json'))) return dir;
+		const up = dirname(dir);
+		if (up === dir) return resolve(flags.dir ?? process.cwd());
+		dir = up;
+	}
+}
+
+function config(required = true) {
+	const dir = root();
+	const path = join(dir, META, 'config.json');
+	if (!existsSync(path)) {
+		if (required) fail(`Not a Tidysites folder (no ${META}/config.json). Run \`tidysites init <site_id> --host <url> --key <key>\` first; the connector's connect_tidysites_cli tool prints that command.`);
+		return { dir, cfg: null };
+	}
+	const cfg = JSON.parse(readFileSync(path, 'utf8'));
+	const keyPath = join(dir, META, 'key');
+	cfg.key = process.env.TIDYSITES_KEY || (existsSync(keyPath) ? readFileSync(keyPath, 'utf8').trim() : '');
+	if (!cfg.key) fail(`No credential: put it in ${META}/key or TIDYSITES_KEY.`);
+	return { dir, cfg };
+}
+
+function writeFiles(dir, files) {
+	for (const [rel, content] of Object.entries(files)) {
+		const p = join(dir, rel);
+		mkdirSync(dirname(p), { recursive: true });
+		writeFileSync(p, content);
+	}
+}
+
+/** Every entry in the folder, by "collection/slug", reading sidecars beside each JSON. */
+function readLocal(dir, only = null) {
+	const local = {};
+	const problems = [];
+	for (const name of readdirSync(dir)) {
+		if (name.startsWith('.') || name === 'node_modules') continue;
+		const colDir = join(dir, name);
+		if (!statSync(colDir).isDirectory()) continue;
+		if (!/^[a-z][a-z0-9_]*$/.test(name) || name === 'menus') continue;
+		if (only && !only.includes(name)) continue;
+		const sidecarExts = Object.keys(SIDECARS[name] ?? SIDECARS.default);
+		for (const file of readdirSync(colDir)) {
+			if (!file.endsWith('.json')) continue;
+			const slug = file.slice(0, -5);
+			if (!FILE_NAME.test(slug)) {
+				problems.push(`${name}/${file}: the file name is the slug; use letters, digits, - or _`);
+				continue;
+			}
+			const sidecars = {};
+			for (const ext of sidecarExts) {
+				const p = join(colDir, `${slug}.${ext}`);
+				if (existsSync(p)) sidecars[ext] = readFileSync(p, 'utf8');
+			}
+			try {
+				local[`${name}/${slug}`] = filesToEntry(name, slug, readFileSync(join(colDir, file), 'utf8'), sidecars);
+			} catch (e) {
+				problems.push(e.message);
+			}
+		}
+	}
+	return { local, problems };
+}
+
+function readJson(path, fallback = null) {
+	return existsSync(path) ? JSON.parse(readFileSync(path, 'utf8')) : fallback;
+}
+
+function describeError(e) {
+	let msg = e.message;
+	if (e.details?.conflicts) {
+		msg += '\n' + e.details.conflicts.map((c) => `  ${c.collection}/${c.slug}: ${c.kind === 'changed' ? `changed on the website (${c.site_updated_at})` : c.kind === 'gone' ? 'removed on the website' : 'already exists on the website'}`).join('\n');
+		msg += '\n\nRun `tidysites pull` to take the website\'s version (a file you changed here is kept and shows as changed against the new snapshot), or push --force to overwrite the website.';
+	} else if (e.details) msg += `\n${JSON.stringify(e.details, null, 2).slice(0, 4000)}`;
+	return msg;
+}
+
+async function tool(cfg, name, args) {
+	try {
+		return await callTool(cfg, name, args);
+	} catch (e) {
+		if (e instanceof ApiError) fail(describeError(e));
+		throw e;
+	}
+}
+
+/** The site's menus as trees, by name. */
+async function readMenus(cfg) {
+	const { json: list } = await tool(cfg, 'call_website_site_tool', { site_id: cfg.site, tool: 'menu_list', arguments: {} });
+	const rows = list?.result ?? list?.menus ?? [];
+	const menus = {};
+	for (const m of Array.isArray(rows) ? rows : []) {
+		if (!m?.name) continue;
+		const { json: one } = await tool(cfg, 'call_website_site_tool', { site_id: cfg.site, tool: 'menu_get', arguments: { name: m.name } });
+		const menu = one?.result ?? one;
+		menus[m.name] = { name: m.name, label: menu?.label ?? m.label ?? m.name, items: menuTree(Array.isArray(menu?.items) ? menu.items : []) };
+	}
+	return menus;
+}
+
+function readLocalMenus(dir) {
+	const out = {};
+	const d = join(dir, 'menus');
+	if (!existsSync(d)) return out;
+	for (const f of readdirSync(d)) {
+		if (!f.endsWith('.json')) continue;
+		const name = f.slice(0, -5);
+		try {
+			const j = JSON.parse(readFileSync(join(d, f), 'utf8'));
+			out[name] = { name, label: j.label ?? name, items: Array.isArray(j.items) ? j.items : [] };
+		} catch (e) {
+			out[name] = { name, error: `menus/${f}: ${e.message}` };
+		}
+	}
+	return out;
+}
+
+async function pull({ dir, cfg }, collections = null) {
+	const want = collections ?? (flags.collections ? String(flags.collections).split(',').map((s) => s.trim()).filter(Boolean) : null);
+	const [{ json: doc }, { json: schema }] = await Promise.all([
+		tool(cfg, 'pull_website', { site_id: cfg.site, ...(want ? { collections: want.join(',') } : {}) }),
+		tool(cfg, 'get_website_schema', { site_id: cfg.site })
+	]);
+	if (!doc?.content) fail('The pull came back without content.');
+	const pulled = new Set(doc.collections.map((c) => c.slug));
+	const { local } = readLocal(dir, [...pulled]);
+	const prev = readJson(join(dir, META, 'snapshot.json'));
+	// A file changed here since the last pull and not pushed yet is kept; the snapshot moves to the
+	// website's version underneath it, so status shows the difference and a push carries the new stamp.
+	const files = {};
+	const kept = [];
+	for (const [collection, entries] of Object.entries(doc.content)) {
+		for (const e of entries) {
+			const key = `${collection}/${e.slug ?? e.id}`;
+			const mine = local[key];
+			const was = prev?.entries?.[key];
+			if (mine && was && !sameEntry(mine, was) && !sameEntry(mine, { status: e.status, data: e.data })) {
+				kept.push(key);
+				continue;
+			}
+			Object.assign(files, entryToFiles(collection, e));
+		}
+	}
+	let removed = 0;
+	for (const key of Object.keys(local)) {
+		const [collection, slug] = key.split('/');
+		// Only a file the last pull wrote goes; a file you added here and never pushed stays.
+		if (prev?.entries?.[key] && !doc.content[collection]?.some((e) => (e.slug ?? e.id) === slug)) {
+			for (const ext of ['json', ...Object.keys(SIDECARS[collection] ?? SIDECARS.default)]) {
+				const p = join(dir, collection, `${slug}.${ext}`);
+				if (existsSync(p)) rmSync(p);
+			}
+			removed++;
+		}
+	}
+	writeFiles(dir, files);
+	const snap = snapshotOf(doc);
+	if (!want) {
+		const menus = await readMenus(cfg);
+		snap.menus = menus;
+		const localMenus = readLocalMenus(dir);
+		for (const [name, menu] of Object.entries(menus)) {
+			const mine = localMenus[name];
+			const wasMenu = prev?.menus?.[name];
+			if (mine && !mine.error && wasMenu && JSON.stringify(mine.items) !== JSON.stringify(wasMenu.items) && JSON.stringify(mine.items) !== JSON.stringify(menu.items)) {
+				kept.push(`menus/${name}`);
+				continue;
+			}
+			writeFiles(dir, { [`menus/${name}.json`]: `${JSON.stringify({ label: menu.label, items: menu.items }, null, '\t')}\n` });
+		}
+	} else if (prev?.menus) snap.menus = prev.menus;
+	if (want && prev) {
+		// A partial pull refreshes only its collections in the snapshot.
+		for (const [k, v] of Object.entries(prev.entries)) if (!pulled.has(k.split('/')[0])) snap.entries[k] = v;
+		snap.settings = prev.settings;
+		snap.collections = [...new Set([...(prev.collections ?? []), ...snap.collections])];
+	}
+	writeFileSync(join(dir, META, 'snapshot.json'), JSON.stringify(snap, null, '\t'));
+	writeFileSync(join(dir, META, 'schema.json'), JSON.stringify(schema, null, '\t'));
+	const n = doc.collections.reduce((a, c) => a + c.count, 0);
+	say(`Pulled ${n} entr${n === 1 ? 'y' : 'ies'} in ${doc.collections.length} collections from ${doc.site.preview_url ?? cfg.site}${removed ? `; ${removed} file${removed === 1 ? '' : 's'} removed (gone on the website)` : ''}.`);
+	for (const c of doc.collections.filter((c) => c.count)) say(`  ${c.slug.padEnd(20)} ${c.count}`);
+	if (snap.menus) say(`  ${'menus'.padEnd(20)} ${Object.keys(snap.menus).length}`);
+	for (const k of kept) say(`  kept your unpushed changes in ${k}`);
+}
+
+function status({ dir }) {
+	const snap = readJson(join(dir, META, 'snapshot.json'));
+	if (!snap) fail('Nothing pulled yet. Run `tidysites pull`.');
+	const { local, problems } = readLocal(dir);
+	const d = diffEntries(local, snap);
+	const menus = readLocalMenus(dir);
+	const menusChanged = [];
+	for (const [name, m] of Object.entries(menus)) {
+		if (m.error) problems.push(m.error);
+		else if (JSON.stringify(m.items) !== JSON.stringify(snap.menus?.[name]?.items ?? null) || m.label !== (snap.menus?.[name]?.label ?? m.label)) menusChanged.push(name);
+	}
+	return { snap, local, problems, diff: d, menus, menusChanged };
+}
+
+function printStatus(s) {
+	for (const p of s.problems) say(`! ${p}`);
+	for (const k of s.diff.created) say(`+ ${k}`);
+	for (const k of s.diff.updated) say(`~ ${k}`);
+	for (const k of s.diff.removed) say(`- ${k} (file deleted; push --prune removes it from the website)`);
+	for (const m of s.menusChanged) say(`~ menus/${m}`);
+	const n = s.diff.created.length + s.diff.updated.length + s.menusChanged.length;
+	if (!n && !s.diff.removed.length && !s.problems.length) say(`Nothing changed since the pull (${s.diff.unchanged.length} entries).`);
+	return n;
+}
+
+function validate({ dir }, local = null) {
+	const schema = readJson(join(dir, META, 'schema.json'));
+	if (!schema) fail('No content model here yet. Run `tidysites pull`.');
+	const read = local ? { local, problems: [] } : readLocal(dir);
+	const errors = [...read.problems, ...validateAll(read.local, schema)];
+	return errors;
+}
+
+async function push(ctx) {
+	const s = status(ctx);
+	const errors = validate(ctx, s.local);
+	if (errors.length) {
+		for (const e of errors) say(`✗ ${e}`);
+		fail(`\n${errors.length} problem${errors.length === 1 ? '' : 's'}; nothing pushed.`);
+	}
+	const only = flags.only ? String(flags.only).split(',') : null;
+	const keys = [...s.diff.created, ...s.diff.updated].filter((k) => !only || only.includes(k.split('/')[0]));
+	const removeKeys = flags.prune ? s.diff.removed.filter((k) => !only || only.includes(k.split('/')[0])) : [];
+	const menus = s.menusChanged.filter((m) => !only || only.includes('menus'));
+	if (!keys.length && !removeKeys.length && !menus.length) {
+		say('Nothing to push.');
+		return;
+	}
+	const doc = pushDocument(s.local, s.snap, { keys, removeKeys });
+	if (flags.force) doc.force = true;
+	if (flags.dry) {
+		say(`Would push ${keys.length} entr${keys.length === 1 ? 'y' : 'ies'}${removeKeys.length ? `, remove ${removeKeys.length}` : ''}${menus.length ? `, rewrite menu${menus.length === 1 ? '' : 's'} ${menus.join(', ')}` : ''}:`);
+		printStatus(s);
+		if (flags.json) say(JSON.stringify(doc, null, 2));
+		return;
+	}
+	printStatus(s);
+	const touched = [...new Set([...keys, ...removeKeys].map((k) => k.split('/')[0]))];
+	if (keys.length || removeKeys.length) {
+		let r;
+		try {
+			({ json: r } = await callTool(ctx.cfg, 'push_website', { site_id: ctx.cfg.site, ...doc }));
+		} catch (e) {
+			if (!(e instanceof ApiError)) throw e;
+			say(describeError(e));
+			if (e.status !== 409 && e.status !== 422) {
+				// The website writes entry by entry, so a failure part way leaves some of them written.
+				say('\nThe website may have taken part of the push. Pulling what it has now; a file of yours that still differs is kept and stays listed in status.');
+				await pull(ctx, touched);
+			}
+			process.exit(1);
+		}
+		say(`Pushed: ${r.content.created} created, ${r.content.updated} updated${r.images ? `, ${r.images} image${r.images === 1 ? '' : 's'} added` : ''}${r.removed ? `, ${r.removed} removed` : ''}.`);
+		if (r.missing?.length) say(`Not found to remove: ${r.missing.map((m) => `${m.collection}/${m.slug}`).join(', ')}`);
+	}
+	for (const name of menus) {
+		const m = s.menus[name];
+		let items;
+		try {
+			items = menuItems(m.items);
+		} catch (e) {
+			fail(`menus/${name}.json: ${e.message}`);
+		}
+		await tool(ctx.cfg, 'call_website_site_tool', { site_id: ctx.cfg.site, tool: 'menu_set_items', arguments: { name, items } });
+		say(`Menu ${name}: ${items.length} item${items.length === 1 ? '' : 's'} written.`);
+	}
+	// Refresh only what went: a partial push leaves the other collections' files exactly as they are.
+	await pull(ctx, menus.length && !touched.length ? null : touched.length ? touched : null);
+}
+
+async function shot(ctx, target = positional[0] ?? '/', opts = {}) {
+	const isUrl = /^https?:\/\//.test(target);
+	const view = flags.mobile ? 'mobile' : flags.desktop ? 'desktop' : (opts.view ?? 'both');
+	const args = { site_id: ctx.cfg.site, view, full_page: !(flags['first-screen'] || opts.firstScreen), ...(isUrl ? { url: target } : { path: target, site: flags.live ? 'live' : 'preview' }) };
+	const { json, images } = await tool(ctx.cfg, 'screenshot_website', args);
+	const outDir = join(ctx.dir, META, 'shots');
+	mkdirSync(outDir, { recursive: true });
+	const stamp = new Date().toISOString().replace(/[-:.]/g, '').slice(4, 18);
+	const base = (opts.name ?? (isUrl ? new URL(target).hostname + new URL(target).pathname : target)).replace(/[^a-z0-9]+/gi, '-').replace(/^-|-$/g, '') || 'home';
+	const named = Array.isArray(json?.captures) ? json.captures : [];
+	images.forEach((img, i) => {
+		const v = named[i]?.view ?? (view === 'both' ? ['desktop', 'mobile'][i] : view);
+		const p = join(outDir, `${base}-${v}-${stamp}.${img.mimeType.includes('png') ? 'png' : 'jpg'}`);
+		writeFileSync(p, Buffer.from(img.base64, 'base64'));
+		say(relative(process.cwd(), p));
+	});
+	if (!images.length) say(json ? JSON.stringify(json).slice(0, 500) : 'No image came back.');
+}
+
+async function pages(ctx) {
+	const { json } = await tool(ctx.cfg, 'list_website_pages', { site_id: ctx.cfg.site });
+	const list = json?.pages ?? json?.items ?? [];
+	for (const p of list) say(`${(p.path ?? '').padEnd(40)} ${p.kind ?? ''}${p.collection ? ` ${p.collection}/${p.slug ?? ''}` : ''}  ${p.title ?? ''}`);
+	if (!list.length) say(JSON.stringify(json).slice(0, 800));
+}
+
+async function page(ctx) {
+	const path = positional[0] ?? '/';
+	const { json } = await tool(ctx.cfg, 'get_website_page', { site_id: ctx.cfg.site, path });
+	if (!json) fail('No page came back.');
+	say(`${json.title ?? ''}${json.h1 ? `  (H1: ${json.h1})` : ''}`);
+	const sections = json.sections ?? json.blocks ?? [];
+	for (const s of sections) say(`  ${String(s.key ?? s._key ?? '').padEnd(14)} ${String(s.type ?? s._type ?? '').padEnd(12)} ${(s.html ?? '').length} chars${s.heading ? `  ${s.heading}` : ''}`);
+	if (flags.html) {
+		const outDir = join(ctx.dir, META, 'pages');
+		mkdirSync(outDir, { recursive: true });
+		const p = join(outDir, `${path.replace(/[^a-z0-9]+/gi, '-').replace(/^-|-$/g, '') || 'home'}.html`);
+		writeFileSync(p, [json.header ?? '', ...sections.map((s) => `<!-- ${s.key ?? ''} ${s.type ?? ''} -->\n${s.html ?? ''}`), json.footer ?? ''].join('\n\n'));
+		say(relative(process.cwd(), p));
+	}
+}
+
+function schema(ctx) {
+	const s = readJson(join(ctx.dir, META, 'schema.json'));
+	if (!s) fail('No content model here yet. Run `tidysites pull`.');
+	const model = indexSchema(s);
+	const name = positional[0];
+	const fieldLine = (f) => {
+		const opts = f.validation?.options ?? f.options;
+		const sub = f.validation?.subFields ?? f.subFields;
+		let line = `  ${f.slug.padEnd(22)} ${f.type}${f.required ? ' (required)' : ''}${f.default !== undefined ? `  default: ${JSON.stringify(f.default)}` : ''}`;
+		if (f.label && f.label !== f.slug) line += `  — ${f.label}`;
+		if (opts) line += `\n${' '.repeat(25)}one of: ${opts.join(' | ')}`;
+		if (sub) line += `\n${' '.repeat(25)}items: ${sub.map((x) => `${x.slug}:${x.type}${x.options ? `(${x.options.join('|')})` : ''}`).join(', ')}`;
+		if (f.validation?.allowedTypes) line += `\n${' '.repeat(25)}section types: ${f.validation.allowedTypes.join(', ')}`;
+		return line;
+	};
+	if (!name) {
+		say('Section types (a page\'s sections list holds these; `tidysites schema <type>` for the fields):');
+		for (const t of model.sectionTypes.values()) say(`  ${t.slug.padEnd(16)} ${t.description ?? t.label ?? ''}`);
+		say('\nCollections (`tidysites schema <collection>` for the fields):');
+		for (const c of model.collections.values()) say(`  ${c.slug.padEnd(20)} ${c.label ?? ''}${c.description ? ` — ${c.description}` : ''}`);
+		return;
+	}
+	const t = model.sectionTypes.get(name);
+	const c = model.collections.get(name);
+	if (!t && !c) fail(`No section type or collection named "${name}".`);
+	if (t) {
+		say(`Section type ${t.slug}: ${t.description ?? ''}`);
+		for (const f of t.fields ?? []) say(fieldLine(f));
+	}
+	if (c) {
+		say(`Collection ${c.slug}: ${c.description ?? c.label ?? ''}`);
+		for (const f of c.fields ?? []) say(fieldLine(f));
+	}
+}
+
+async function fetchCmd(ctx) {
+	const url = positional[0];
+	if (!url) fail('Usage: tidysites fetch <url> [--out dir]');
+	const outDir = resolve(flags.out ?? join(ctx?.dir ?? process.cwd(), META, 'source'));
+	mkdirSync(outDir, { recursive: true });
+	const { url: final, html } = await fetchPage(url);
+	const o = outline(html, final);
+	const base = (new URL(final).pathname.replace(/\/$/, '').split('/').pop() || 'home').replace(/[^a-z0-9]+/gi, '-');
+	const cssTexts = [];
+	for (const href of o.stylesheets.slice(0, 8)) {
+		try {
+			const r = await fetch(href, { headers: { 'user-agent': 'Mozilla/5.0' } });
+			if (r.ok) {
+				const css = await r.text();
+				cssTexts.push(css);
+				writeFileSync(join(outDir, `${base}-${cssTexts.length}.css`), css);
+			}
+		} catch {
+			/* a stylesheet that will not download is not the port's problem */
+		}
+	}
+	if (cssTexts.length) o.css = cssSummary(cssTexts.join('\n'));
+	writeFileSync(join(outDir, `${base}.raw.html`), html);
+	writeFileSync(join(outDir, `${base}.html`), cleanHtml(html));
+	writeFileSync(join(outDir, `${base}.outline.json`), JSON.stringify(o, null, '\t'));
+	const text = outlineText(o);
+	writeFileSync(join(outDir, `${base}.outline.txt`), text);
+	say(text);
+	say(`Saved to ${relative(process.cwd(), outDir)}/: ${base}.html (cleaned), ${base}.raw.html, ${base}.outline.json${cssTexts.length ? `, ${cssTexts.length} stylesheet${cssTexts.length === 1 ? '' : 's'}` : ''}`);
+}
+
+async function publish(ctx) {
+	if (!flags.yes) fail('Publishing makes the live site a copy of the preview site. Look at it first (`tidysites shot /`), then run `tidysites publish --yes`.');
+	const { json, text } = await tool(ctx.cfg, 'publish_website', { site_id: ctx.cfg.site });
+	say(json ? JSON.stringify(json, null, 2) : text);
+}
+
+function help() {
+	say(readFileSync(new URL(import.meta.url), 'utf8').split('\n').slice(2, 20).map((l) => l.replace(/^ \* ?/, '')).join('\n'));
+	say(`\nFiles:\n  <collection>/<slug>.json      { "id", "slug", "status", "updated_at", "data": {...} }; a new entry is a new file with "data"\n  <collection>/<slug>.css       data.custom_css as a file (rows: data.css); rows/<slug>.html is data.html\n  menus/<name>.json             a menu as a tree: { label, items: [{ label, url, children: [...] }] }\n  ${META}/                   config, key, the last pull, the content model, screenshots\n\nA field taken out of a file is cleared on the website at the next push; a file you changed here is never overwritten by a pull.\nImages: an image field or a Columns image slot takes { "$media": { "url": "https://...", "alt": "..." } }; the push fetches it into the library.\nReferences: a record names another by its id, or "$ref:<slug-or-id>" for one in the same push.`);
+}
+
+async function main() {
+	switch (command) {
+		case 'init': {
+			const site = positional[0];
+			const host = flags.host;
+			const key = flags.key || process.env.TIDYSITES_KEY;
+			if (!site || !host || !key) fail('Usage: tidysites init <site_id> --host <url> --key <key> [--dir folder]');
+			const dir = resolve(flags.dir ?? process.cwd());
+			mkdirSync(join(dir, META), { recursive: true });
+			writeFileSync(join(dir, META, 'config.json'), `${JSON.stringify({ host: String(host).replace(/\/+$/, ''), site }, null, '\t')}\n`);
+			writeFileSync(join(dir, META, 'key'), `${key}\n`, { mode: 0o600 });
+			writeFileSync(join(dir, META, '.gitignore'), 'key\nshots/\nsource/\npages/\n');
+			say(`Connected ${relative(process.cwd(), dir) || '.'} to ${site} on ${host}.`);
+			await pull(config());
+			return;
+		}
+		case 'pull':
+			return pull(config());
+		case 'status': {
+			printStatus(status(config()));
+			return;
+		}
+		case 'validate': {
+			const errors = validate(config());
+			if (errors.length) {
+				for (const e of errors) say(`✗ ${e}`);
+				fail(`\n${errors.length} problem${errors.length === 1 ? '' : 's'}.`);
+			}
+			say('Every file fits the content model.');
+			return;
+		}
+		case 'push':
+			return push(config());
+		case 'shot':
+			return shot(config());
+		case 'gallery': {
+			// One type at a time is readable; the whole gallery is a very tall picture.
+			const type = positional[0];
+			return shot(config(), `/_tidy/gallery${type ? `#${type}` : ''}`, { view: 'desktop', name: type ? `gallery-${type}` : 'gallery', firstScreen: Boolean(type) });
+		}
+		case 'pages':
+			return pages(config());
+		case 'page':
+			return page(config());
+		case 'schema':
+			return schema(config());
+		case 'fetch':
+			return fetchCmd(config(false));
+		case 'publish':
+			return publish(config());
+		case 'help':
+		case '--help':
+		case '-h':
+		case undefined:
+			return help();
+		default:
+			fail(`Unknown command "${command}". \`tidysites help\` lists them.`);
+	}
+}
+
+main().catch((e) => fail(e?.stack ?? String(e)));

package/lib/api.mjs

@@ -0,0 +1,68 @@
+/**
+ * The website connector over HTTP. The command line is one more client of
+ * the Tidysites connector (`<host>/mcp/tidysites`): every call is a tool
+ * call with the credential as a bearer, so whatever the connector can do
+ * for an assistant, the command line can do from a shell, and nothing is
+ * opened for it that was not open already.
+ */
+
+export class ApiError extends Error {
+	constructor(message, { status = null, details = null } = {}) {
+		super(message);
+		this.status = status;
+		this.details = details;
+	}
+}
+
+/**
+ * Call one connector tool. Resolves to { text, json, images }: the text
+ * content, the same parsed as JSON when it is JSON, and any images
+ * (base64 with their media type).
+ */
+export async function callTool(cfg, name, args = {}, { fetchImpl = fetch } = {}) {
+	const url = `${cfg.host.replace(/\/+$/, '')}/mcp/tidysites`;
+	const res = await fetchImpl(url, {
+		method: 'POST',
+		headers: { authorization: `Bearer ${cfg.key}`, 'content-type': 'application/json', accept: 'application/json', 'user-agent': 'tidysites-cli' },
+		body: JSON.stringify({ jsonrpc: '2.0', id: 1, method: 'tools/call', params: { name, arguments: args } })
+	});
+	const bodyText = await res.text();
+	let rpc;
+	try {
+		rpc = JSON.parse(bodyText);
+	} catch {
+		throw new ApiError(`${url} answered ${res.status} with something that is not JSON: ${bodyText.slice(0, 200)}`, { status: res.status });
+	}
+	if (res.status === 401) throw new ApiError('The credential was refused. Run `tidysites init` again with a fresh command from the connector (connect_tidysites_cli).', { status: 401 });
+	if (rpc.error) throw new ApiError(`${name}: ${rpc.error.message ?? JSON.stringify(rpc.error)}`, { status: res.status, details: rpc.error });
+	const content = Array.isArray(rpc.result?.content) ? rpc.result.content : [];
+	const text = content.filter((c) => c.type === 'text').map((c) => c.text).join('\n');
+	const images = content.filter((c) => c.type === 'image').map((c) => ({ base64: c.data, mimeType: c.mimeType || 'image/png' }));
+	if (rpc.result?.isError) {
+		// "HTTP 422 — invalid_content: ... {details}" is how the connector words a refusal; keep the details readable.
+		const m = text.match(/^HTTP (\d+) — ([a-z_]+): ([\s\S]*)$/);
+		if (m) {
+			const [, status, code, rest] = m;
+			const brace = rest.indexOf(' {');
+			let details = null;
+			let message = rest;
+			if (brace > 0) {
+				try {
+					details = JSON.parse(rest.slice(brace + 1));
+					message = rest.slice(0, brace);
+				} catch {
+					/* leave the text whole */
+				}
+			}
+			throw new ApiError(`${code}: ${message}`, { status: Number(status), details, code });
+		}
+		throw new ApiError(text || `${name} failed`, { status: res.status });
+	}
+	let json = null;
+	try {
+		json = JSON.parse(text);
+	} catch {
+		/* not JSON: fine, text stands */
+	}
+	return { text, json, images };
+}

package/lib/fetch.mjs

@@ -0,0 +1,144 @@
+/**
+ * Read a page of the site being ported: the HTML with scripts, styles and
+ * SVG noise taken out, and an outline (title, description, headings in
+ * order, navigation, images with their alt, phone numbers and emails, the
+ * stylesheets) so a builder can see the page's bones before deciding which
+ * section each band becomes.
+ */
+
+const UA = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36';
+
+export async function fetchPage(url, { fetchImpl = fetch } = {}) {
+	const res = await fetchImpl(url, { headers: { 'user-agent': UA, accept: 'text/html,*/*' }, redirect: 'follow' });
+	const html = await res.text();
+	if (!res.ok) throw new Error(`${url} answered ${res.status}`);
+	return { url: res.url || url, html };
+}
+
+const decode = (s) =>
+	s
+		.replace(/ /g, ' ')
+		.replace(/&/g, '&')
+		.replace(/&lt;/g, '<')
+		.replace(/&gt;/g, '>')
+		.replace(/&quot;/g, '"')
+		.replace(/&#39;|&apos;/g, "'")
+		.replace(/&#(\d+);/g, (_, n) => String.fromCodePoint(Number(n)))
+		.replace(/&#x([0-9a-f]+);/gi, (_, n) => String.fromCodePoint(parseInt(n, 16)));
+
+const text = (s) => decode(s.replace(/<[^>]+>/g, ' ')).replace(/\s+/g, ' ').trim();
+
+/** The HTML without what a builder never needs to read. */
+export function cleanHtml(html) {
+	return html
+		.replace(/<!--[\s\S]*?-->/g, '')
+		.replace(/<(script|style|noscript|template|iframe)\b[\s\S]*?<\/\1>/gi, '')
+		.replace(/<svg\b[\s\S]*?<\/svg>/gi, '<svg></svg>')
+		.replace(/\son[a-z]+="[^"]*"/gi, '')
+		.replace(/\sstyle="[^"]*"/gi, '')
+		.replace(/\s(data-[a-z0-9-]+|class|id)="[^"]*"/gi, (m, name) => (name === 'class' || name === 'id' ? m : ''))
+		.replace(/[ \t]+\n/g, '\n')
+		.replace(/\n{3,}/g, '\n\n');
+}
+
+function abs(base, href) {
+	try {
+		return new URL(href, base).toString();
+	} catch {
+		return href;
+	}
+}
+
+/** What the page is made of, in order. */
+export function outline(html, base) {
+	const out = { url: base, title: null, description: null, headings: [], nav: [], images: [], phones: [], emails: [], stylesheets: [], fonts: [], colors: [] };
+	const t = html.match(/<title[^>]*>([\s\S]*?)<\/title>/i);
+	if (t) out.title = text(t[1]);
+	const d = html.match(/<meta[^>]+name=["']description["'][^>]*content=["']([^"']*)["']/i) || html.match(/<meta[^>]+content=["']([^"']*)["'][^>]*name=["']description["']/i);
+	if (d) out.description = decode(d[1]).trim();
+	for (const m of html.matchAll(/<(h[1-4])\b[^>]*>([\s\S]*?)<\/\1>/gi)) {
+		const s = text(m[2]);
+		if (s) out.headings.push({ level: Number(m[1][1]), text: s });
+	}
+	const navBlocks = [...html.matchAll(/<nav\b[\s\S]*?<\/nav>/gi)].map((m) => m[0]);
+	const navSource = navBlocks.length ? navBlocks.join('\n') : (html.match(/<header\b[\s\S]*?<\/header>/i)?.[0] ?? '');
+	const seen = new Set();
+	for (const m of navSource.matchAll(/<a\b[^>]*href=["']([^"'#][^"']*)["'][^>]*>([\s\S]*?)<\/a>/gi)) {
+		const label = text(m[2]);
+		const href = abs(base, m[1]);
+		if (label && !seen.has(href)) {
+			seen.add(href);
+			out.nav.push({ label, href });
+		}
+	}
+	for (const m of html.matchAll(/<img\b[^>]*>/gi)) {
+		const tag = m[0];
+		const src = tag.match(/\s(?:data-src|data-lazy-src|src)=["']([^"']+)["']/i)?.[1];
+		if (!src || /^data:/.test(src)) continue;
+		const alt = decode(tag.match(/\salt=["']([^"']*)["']/i)?.[1] ?? '');
+		const width = tag.match(/\swidth=["']?(\d+)/i)?.[1];
+		const height = tag.match(/\sheight=["']?(\d+)/i)?.[1];
+		out.images.push({ src: abs(base, src), alt, ...(width ? { width: Number(width) } : {}), ...(height ? { height: Number(height) } : {}) });
+	}
+	const body = text(html.replace(/<(script|style)\b[\s\S]*?<\/\1>/gi, ''));
+	out.phones = [...new Set([...body.matchAll(/(?:\+?1[\s.-]?)?\(?\d{3}\)?[\s.-]\d{3}[\s.-]\d{4}/g)].map((m) => m[0].trim()))].slice(0, 10);
+	out.emails = [...new Set([...body.matchAll(/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}/gi)].map((m) => m[0]))].slice(0, 10);
+	for (const m of html.matchAll(/<link\b[^>]*rel=["']stylesheet["'][^>]*>/gi)) {
+		const href = m[0].match(/\shref=["']([^"']+)["']/i)?.[1];
+		if (href) out.stylesheets.push(abs(base, href));
+	}
+	for (const m of html.matchAll(/fonts\.googleapis\.com\/css2?\?[^"'<>\s]+/gi)) {
+		const fams = [...m[0].matchAll(/family=([^&:]+)/g)].map((f) => decodeURIComponent(f[1]).replace(/\+/g, ' '));
+		out.fonts.push(...fams);
+	}
+	out.fonts = [...new Set(out.fonts)];
+	return out;
+}
+
+/** The colours and font families a stylesheet leans on, most used first. */
+export function cssSummary(css) {
+	const colors = new Map();
+	for (const m of css.matchAll(/#(?:[0-9a-f]{3}){1,2}\b|rgba?\([^)]*\)/gi)) {
+		const c = m[0].toLowerCase();
+		if (/^#(?:fff|ffffff|000|000000)$/.test(c)) continue;
+		colors.set(c, (colors.get(c) ?? 0) + 1);
+	}
+	const fonts = new Map();
+	for (const m of css.matchAll(/font-family\s*:\s*([^;}]+)/gi)) {
+		const f = m[1].split(',')[0].replace(/["']/g, '').trim();
+		if (f && !/^(inherit|initial|unset|var\()/.test(f)) fonts.set(f, (fonts.get(f) ?? 0) + 1);
+	}
+	const top = (map, n) => [...map.entries()].sort((a, b) => b[1] - a[1]).slice(0, n).map(([value, count]) => ({ value, count }));
+	return { colors: top(colors, 12), fonts: top(fonts, 6) };
+}
+
+export function outlineText(o) {
+	const lines = [];
+	lines.push(`URL: ${o.url}`);
+	if (o.title) lines.push(`Title: ${o.title}`);
+	if (o.description) lines.push(`Description: ${o.description}`);
+	if (o.nav.length) {
+		lines.push('', 'Navigation:');
+		for (const n of o.nav) lines.push(`  ${n.label}  →  ${n.href}`);
+	}
+	if (o.headings.length) {
+		lines.push('', 'Headings, in order:');
+		for (const h of o.headings) lines.push(`  ${'#'.repeat(h.level)} ${h.text}`);
+	}
+	if (o.images.length) {
+		lines.push('', `Images (${o.images.length}):`);
+		for (const i of o.images.slice(0, 60)) lines.push(`  ${i.src}${i.width ? ` (${i.width}x${i.height ?? '?'})` : ''}${i.alt ? `  alt: ${i.alt}` : ''}`);
+	}
+	if (o.phones.length) lines.push('', `Phones: ${o.phones.join(', ')}`);
+	if (o.emails.length) lines.push(`Emails: ${o.emails.join(', ')}`);
+	if (o.fonts.length) lines.push(`Google fonts: ${o.fonts.join(', ')}`);
+	if (o.css) {
+		lines.push('', 'Stylesheet colours (most used first):', `  ${o.css.colors.map((c) => `${c.value} ×${c.count}`).join('  ')}`);
+		lines.push('Stylesheet fonts:', `  ${o.css.fonts.map((c) => `${c.value} ×${c.count}`).join('  ')}`);
+	}
+	if (o.stylesheets.length) {
+		lines.push('', 'Stylesheets:');
+		for (const s of o.stylesheets) lines.push(`  ${s}`);
+	}
+	return `${lines.join('\n')}\n`;
+}

package/lib/folder.mjs

@@ -0,0 +1,220 @@
+/**
+ * The folder: a Tidysite's content as files.
+ *
+ *   <collection>/<slug>.json    { "id", "slug", "status", "updated_at", "data": { ... } }
+ *   <collection>/<slug>.css     data.custom_css, as a file (rows: data.css)
+ *   rows/<slug>.html            data.html, as a file
+ *   .tidysites/                 config, key, the last pull (snapshot), the content model, screenshots
+ *
+ * A new entry is a new <collection>/<slug>.json with at least { "data": { ... } };
+ * the slug is the file name. Everything here is pure: reading and writing the
+ * disk is the command's job, so the rules are testable on their own.
+ */
+
+export const SIDECARS = {
+	default: { css: 'custom_css' },
+	rows: { css: 'css', html: 'html' }
+};
+
+export const FILE_NAME = /^[a-z0-9][a-z0-9_-]*$/i;
+
+function sidecarsFor(collection) {
+	return SIDECARS[collection] ?? SIDECARS.default;
+}
+
+/** A pulled entry as the files it becomes: the JSON and its sidecars. */
+export function entryToFiles(collection, entry) {
+	const name = entry.slug ?? entry.id;
+	const data = { ...entry.data };
+	const files = {};
+	for (const [ext, field] of Object.entries(sidecarsFor(collection))) {
+		const v = data[field];
+		if (typeof v === 'string' && v.trim()) {
+			// Always one newline added on write and one taken on read, so the text round-trips exactly.
+			files[`${collection}/${name}.${ext}`] = `${v}\n`;
+			delete data[field];
+		} else if (v === '' || v === null) {
+			delete data[field];
+		}
+	}
+	const json = { id: entry.id, slug: entry.slug ?? null, status: entry.status, updated_at: entry.updated_at ?? null, ...(entry.has_draft ? { has_draft: true } : {}), data };
+	files[`${collection}/${name}.json`] = `${JSON.stringify(json, null, '\t')}\n`;
+	return files;
+}
+
+/** The whole document as files: every entry. The site title and tagline a theme prints are the Business record's, so there is no settings file. */
+export function docToFiles(doc) {
+	const files = {};
+	for (const [collection, entries] of Object.entries(doc.content ?? {})) {
+		for (const e of entries) Object.assign(files, entryToFiles(collection, e));
+	}
+	return files;
+}
+
+/**
+ * One entry read back from its JSON text and the sidecar texts beside it
+ * ({ css?, html? }). Throws with a message that names the file.
+ */
+export function filesToEntry(collection, name, jsonText, sidecars = {}) {
+	const file = `${collection}/${name}.json`;
+	let parsed;
+	try {
+		parsed = JSON.parse(jsonText);
+	} catch (e) {
+		throw new Error(`${file}: not valid JSON (${e.message})`);
+	}
+	if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) throw new Error(`${file}: must be an object with a data field`);
+	if (!parsed.data || typeof parsed.data !== 'object' || Array.isArray(parsed.data)) throw new Error(`${file}: needs "data": { ... }`);
+	if (parsed.slug != null && parsed.slug !== name) throw new Error(`${file}: slug "${parsed.slug}" does not match the file name; rename the file or the slug`);
+	if (!FILE_NAME.test(name)) throw new Error(`${file}: the file name is the slug; use letters, digits, - or _`);
+	const data = { ...parsed.data };
+	for (const [ext, field] of Object.entries(sidecarsFor(collection))) {
+		const text = sidecars[ext];
+		if (typeof text === 'string') {
+			if (typeof data[field] === 'string' && data[field] !== '') throw new Error(`${file}: "${field}" is set both in the JSON and in ${collection}/${name}.${ext}; keep one`);
+			data[field] = text.endsWith('\n') ? text.slice(0, -1) : text;
+		}
+	}
+	const status = parsed.status === 'draft' ? 'draft' : parsed.status === 'published' || parsed.status === undefined ? 'published' : null;
+	if (status === null) throw new Error(`${file}: status is published or draft`);
+	return {
+		id: typeof parsed.id === 'string' && parsed.id ? parsed.id : name,
+		slug: name,
+		status,
+		updated_at: typeof parsed.updated_at === 'string' ? parsed.updated_at : null,
+		data
+	};
+}
+
+/** The snapshot kept from a pull: what the site had, by "collection/slug". */
+export function snapshotOf(doc) {
+	const entries = {};
+	for (const [collection, list] of Object.entries(doc.content ?? {})) {
+		for (const e of list) entries[`${collection}/${e.slug ?? e.id}`] = { id: e.id, status: e.status, updated_at: e.updated_at ?? null, data: e.data };
+	}
+	return { pulled_at: doc.site?.exported_at ?? new Date().toISOString(), site: doc.site ?? null, settings: doc.settings ?? {}, collections: (doc.collections ?? []).map((c) => c.slug), entries };
+}
+
+function stable(value) {
+	if (Array.isArray(value)) return `[${value.map(stable).join(',')}]`;
+	if (value && typeof value === 'object') return `{${Object.keys(value).sort().map((k) => `${JSON.stringify(k)}:${stable(value[k])}`).join(',')}}`;
+	return JSON.stringify(value);
+}
+
+/** An empty field and a missing one say the same thing to a push (the site keeps what it has). */
+function bare(data) {
+	const out = {};
+	for (const [k, v] of Object.entries(data ?? {})) if (v !== null && v !== '' && v !== undefined) out[k] = v;
+	return out;
+}
+
+/** Whether two entries would write the same thing. */
+export function sameEntry(a, b) {
+	return a.status === b.status && stable(bare(a.data)) === stable(bare(b.data));
+}
+
+/**
+ * Local entries (by "collection/slug") against the snapshot: what a push
+ * would create, update, and (with prune) remove. Only collections the
+ * snapshot covered can report removals; a collection never pulled is new.
+ */
+export function diffEntries(local, snapshot) {
+	const out = { created: [], updated: [], removed: [], unchanged: [] };
+	for (const [key, entry] of Object.entries(local)) {
+		const was = snapshot.entries[key];
+		if (!was) out.created.push(key);
+		else if (!sameEntry(entry, was)) out.updated.push(key);
+		else out.unchanged.push(key);
+	}
+	for (const key of Object.keys(snapshot.entries)) if (!local[key]) out.removed.push(key);
+	out.created.sort();
+	out.updated.sort();
+	out.removed.sort();
+	return out;
+}
+
+/** The document a push sends for the given keys, with expect from the snapshot. */
+export function pushDocument(local, snapshot, { keys, removeKeys = [], settings = null }) {
+	const content = {};
+	const expect = {};
+	for (const key of keys) {
+		const [collection, slug] = splitKey(key);
+		const e = local[key];
+		const was = snapshot.entries[key];
+		// A field taken out of the file is cleared on the site: the site keeps what a push does not name,
+		// so the names the snapshot had and the file no longer has go as null.
+		const data = { ...e.data };
+		if (was?.data) for (const k of Object.keys(was.data)) if (!(k in data) && was.data[k] !== null && was.data[k] !== '') data[k] = null;
+		(content[collection] ??= []).push({ id: e.id, slug, status: e.status, data });
+		if (was?.updated_at) expect[key] = was.updated_at;
+	}
+	const remove = {};
+	for (const key of removeKeys) {
+		const [collection, slug] = splitKey(key);
+		(remove[collection] ??= []).push(slug);
+		const was = snapshot.entries[key];
+		if (was?.updated_at) expect[key] = was.updated_at;
+	}
+	const doc = {};
+	if (settings && Object.keys(settings).length) doc.settings = settings;
+	if (Object.keys(content).length) doc.content = content;
+	if (Object.keys(remove).length) doc.remove = remove;
+	if (Object.keys(expect).length) doc.expect = expect;
+	return doc;
+}
+
+export function splitKey(key) {
+	const i = key.indexOf('/');
+	return [key.slice(0, i), key.slice(i + 1)];
+}
+
+// ---------------------------------------------------------------------------
+// Menus: menus/<name>.json, a tree of items; the site keeps a flat list with parent indexes.
+// ---------------------------------------------------------------------------
+
+/** The site's flat item list (parentId, sortOrder) as the tree the file holds. */
+export function menuTree(items) {
+	const byParent = new Map();
+	for (const it of [...items].sort((a, b) => (a.sortOrder ?? 0) - (b.sortOrder ?? 0))) {
+		const k = it.parentId ?? null;
+		(byParent.get(k) ?? byParent.set(k, []).get(k)).push(it);
+	}
+	const node = (it) => {
+		const out = { label: it.label };
+		if (it.type === 'custom' || !it.type) out.url = it.customUrl ?? '';
+		else {
+			out.type = it.type;
+			if (it.referenceCollection) out.collection = it.referenceCollection;
+			if (it.referenceId) out.id = it.referenceId;
+		}
+		if (it.titleAttr) out.title = it.titleAttr;
+		if (it.target) out.target = it.target;
+		if (it.cssClasses) out.classes = it.cssClasses;
+		const kids = byParent.get(it.id) ?? [];
+		if (kids.length) out.children = kids.map(node);
+		return out;
+	};
+	return (byParent.get(null) ?? []).map(node);
+}
+
+/** The tree flattened to what menu_set_items takes: each child names its parent's index. */
+export function menuItems(tree) {
+	const out = [];
+	const walk = (nodes, parentIndex) => {
+		for (const n of nodes) {
+			if (!n || typeof n.label !== 'string' || !n.label) throw new Error('every menu item needs a label');
+			const item = { label: n.label, type: n.url !== undefined || !n.type ? 'custom' : n.type };
+			if (item.type === 'custom') item.customUrl = n.url ?? '#';
+			if (n.collection) item.referenceCollection = n.collection;
+			if (n.id) item.referenceId = n.id;
+			if (n.title) item.titleAttr = n.title;
+			if (n.target) item.target = n.target;
+			if (n.classes) item.cssClasses = n.classes;
+			if (parentIndex !== null) item.parentIndex = parentIndex;
+			out.push(item);
+			if (Array.isArray(n.children) && n.children.length) walk(n.children, out.length - 1);
+		}
+	};
+	walk(tree, null);
+	return out;
+}

package/lib/validate.mjs

@@ -0,0 +1,195 @@
+/**
+ * Check entries against the website's content model before anything is
+ * sent: unknown fields, unknown section types and section fields, select
+ * values outside their options, repeater items with fields the row does not
+ * have, values of the wrong shape. The website runs its own validator too;
+ * this one says it first, with the file path, so a build is fixed locally.
+ *
+ * The model is what get_website_schema returns:
+ *   collections.items[]   { slug, label, fields: [{ slug, type, required, validation }] }
+ *   section_types.items[] { slug, label, description, fields: [{ slug, type, validation, default }] }
+ * with validation.options on a select, validation.allowedTypes on a blocks
+ * field, validation.subFields on a repeater.
+ */
+
+const BLOCK_META = new Set(['_type', '_key', '_version']);
+
+export function indexSchema(schema) {
+	const collections = new Map();
+	for (const c of schema?.collections?.items ?? []) if (c?.slug) collections.set(c.slug, c);
+	const sectionTypes = new Map();
+	for (const t of schema?.section_types?.items ?? []) if (t?.slug) sectionTypes.set(t.slug, t);
+	return { collections, sectionTypes };
+}
+
+const isMedia = (v) => v && typeof v === 'object' && !Array.isArray(v) && (typeof v.id === 'string' || (v.$media && typeof v.$media === 'object'));
+const empty = (v) => v === null || v === undefined || v === '';
+
+function checkScalar(type, value, at, errors) {
+	if (empty(value)) return;
+	switch (type) {
+		case 'string':
+		case 'text':
+		case 'url':
+		case 'email':
+		case 'portableText':
+		case 'richText':
+		case 'markdown':
+			if (typeof value !== 'string') {
+				// A rich-text field may travel as blocks already; a plain field may not.
+				if (type === 'portableText' && Array.isArray(value)) return;
+				errors.push(`${at}: expected text, got ${describe(value)}`);
+			}
+			return;
+		case 'boolean':
+			// The site stores a switch as 1 or 0, so a pull shows numbers; both spellings are fine.
+			if (typeof value !== 'boolean' && value !== 0 && value !== 1) errors.push(`${at}: expected true or false, got ${describe(value)}`);
+			return;
+		case 'integer':
+		case 'number':
+			if (typeof value !== 'number' && !(typeof value === 'string' && /^-?\d+(\.\d+)?$/.test(value))) errors.push(`${at}: expected a number, got ${describe(value)}`);
+			return;
+		case 'image':
+		case 'media':
+		case 'file':
+			if (!isMedia(value)) errors.push(`${at}: an image is the stored value (from the pull or add_website_image) or { "$media": { "url": "https://...", "alt": "..." } }`);
+			else if (value.$media && (typeof value.$media.url !== 'string' || !/^https?:\/\//.test(value.$media.url))) errors.push(`${at}: $media.url must be an http(s) address`);
+			else if (value.$media && typeof value.$media.alt !== 'string') errors.push(`${at}: $media.alt is required (what the image shows; "" for a decorative one)`);
+			return;
+		case 'json':
+			return;
+		default:
+			return;
+	}
+}
+
+function checkSelect(field, value, at, errors) {
+	if (empty(value)) return;
+	const options = field.validation?.options ?? field.options;
+	if (!Array.isArray(options) || options.length === 0) return;
+	if (typeof value !== 'string' || !options.includes(value)) errors.push(`${at}: "${String(value)}" is not one of ${options.map((o) => `"${o}"`).join(', ')}`);
+}
+
+function checkRepeater(field, value, at, errors) {
+	if (empty(value)) return;
+	if (!Array.isArray(value)) {
+		errors.push(`${at}: expected a list of items, got ${describe(value)}`);
+		return;
+	}
+	const sub = field.validation?.subFields ?? field.subFields ?? [];
+	const known = new Map(sub.map((f) => [f.slug, f]));
+	value.forEach((item, i) => {
+		if (!item || typeof item !== 'object' || Array.isArray(item)) {
+			errors.push(`${at}[${i}]: expected an object`);
+			return;
+		}
+		for (const [k, v] of Object.entries(item)) {
+			if (k === '_key') continue;
+			const f = known.get(k);
+			if (!f) {
+				errors.push(`${at}[${i}].${k}: not a field of this row (fields: ${[...known.keys()].join(', ')})`);
+				continue;
+			}
+			checkField(f, v, `${at}[${i}].${k}`, errors, null);
+		}
+	});
+}
+
+function checkField(field, value, at, errors, ctx) {
+	switch (field.type) {
+		case 'select':
+			checkSelect(field, value, at, errors);
+			return;
+		case 'repeater':
+			checkRepeater(field, value, at, errors);
+			return;
+		case 'blocks':
+			checkBlocks(field, value, at, errors, ctx);
+			return;
+		case 'reference':
+		case 'relation':
+			if (!empty(value) && typeof value !== 'string' && !Array.isArray(value)) errors.push(`${at}: a reference is the entry id (or $ref:<id>) as a string`);
+			return;
+		default:
+			checkScalar(field.type, value, at, errors);
+	}
+}
+
+function checkBlocks(field, value, at, errors, ctx) {
+	if (empty(value)) return;
+	if (!Array.isArray(value)) {
+		errors.push(`${at}: sections are a list`);
+		return;
+	}
+	const allowed = field.validation?.allowedTypes;
+	value.forEach((block, i) => {
+		const where = `${at}[${i}]`;
+		if (!block || typeof block !== 'object' || Array.isArray(block)) {
+			errors.push(`${where}: a section is an object with _type`);
+			return;
+		}
+		const type = block._type;
+		if (typeof type !== 'string' || !type) {
+			errors.push(`${where}: missing _type`);
+			return;
+		}
+		const def = ctx?.sectionTypes.get(type);
+		if (!def || (Array.isArray(allowed) && !allowed.includes(type))) {
+			errors.push(`${where}: "${type}" is not a section type this field takes (run \`tidysites schema\` for the list)`);
+			return;
+		}
+		const fields = new Map((def.fields ?? []).map((f) => [f.slug, f]));
+		for (const [k, v] of Object.entries(block)) {
+			if (BLOCK_META.has(k)) continue;
+			const f = fields.get(k);
+			if (!f) {
+				errors.push(`${where} (${type}).${k}: not a field of ${type} (fields: ${[...fields.keys()].join(', ')})`);
+				continue;
+			}
+			checkField(f, v, `${where} (${type}).${k}`, errors, ctx);
+		}
+	});
+}
+
+function describe(v) {
+	if (v === null) return 'null';
+	if (Array.isArray(v)) return 'a list';
+	return typeof v === 'object' ? 'an object' : typeof v;
+}
+
+/**
+ * Errors for one entry, each prefixed with the file. An empty list means the
+ * website's own validator should accept it too (its rules are the same model).
+ */
+export function validateEntry(collection, entry, model, file = `${collection}/${entry.slug}.json`) {
+	const errors = [];
+	const col = model.collections.get(collection);
+	if (!col) {
+		errors.push(`${file}: no collection named "${collection}" on this website (collections: ${[...model.collections.keys()].join(', ')})`);
+		return errors;
+	}
+	const fields = new Map((col.fields ?? []).map((f) => [f.slug, f]));
+	for (const f of col.fields ?? []) {
+		if (f.required && empty(entry.data[f.slug]) && !(f.slug in entry.data && f.type === 'blocks')) errors.push(`${file}: data.${f.slug} is required`);
+	}
+	for (const [k, v] of Object.entries(entry.data)) {
+		const f = fields.get(k);
+		if (!f) {
+			errors.push(`${file}: data.${k} is not a field of ${collection} (fields: ${[...fields.keys()].join(', ')})`);
+			continue;
+		}
+		checkField(f, v, `${file}: data.${k}`, errors, model);
+	}
+	return errors;
+}
+
+/** Every entry (by "collection/slug") checked; the errors in file order. */
+export function validateAll(local, schema) {
+	const model = indexSchema(schema);
+	const errors = [];
+	for (const key of Object.keys(local).sort()) {
+		const i = key.indexOf('/');
+		errors.push(...validateEntry(key.slice(0, i), local[key], model));
+	}
+	return errors;
+}

package/package.json

@@ -1,6 +1,27 @@
 {
   "name": "@tideworthy/tidysites",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.1",
+  "description": "Pull a Tidysites website to files, edit, validate against its content model, push it back, screenshot the result.",
+  "type": "module",
+  "bin": {
+    "tidysites": "./bin/tidysites.mjs"
+  },
+  "files": [
+    "bin",
+    "lib",
+    "README.md"
+  ],
+  "engines": {
+    "node": ">=20"
+  },
+  "scripts": {
+    "test": "node --test test/*.test.mjs",
+    "check": "node --check bin/tidysites.mjs && npm test"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/4sons/tidysites-platform.git",
+    "directory": "cli"
+  },
+  "license": "UNLICENSED"
+}