cornhsu-polymigrate 1.1.0

package/README.md

@@ -0,0 +1,138 @@
+# PolyMigrate
+
+[![NuGet](https://img.shields.io/nuget/v/Cornhsu.PolyMigrate.svg?label=Cornhsu.PolyMigrate)](https://www.nuget.org/packages/Cornhsu.PolyMigrate)
+[![Downloads](https://img.shields.io/nuget/dt/Cornhsu.PolyMigrate.svg)](https://www.nuget.org/packages/Cornhsu.PolyMigrate)
+[![CI](https://github.com/HSU-YU-MING/cornhsu-polymigrate/actions/workflows/ci.yml/badge.svg)](https://github.com/HSU-YU-MING/cornhsu-polymigrate/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+> **The i18n-first static-site migrator** — the only site migrator that pairs your multilingual pages automatically.
+
+**[Project write-up](https://cornhsu.com/polymigrate) · [NuGet](https://www.nuget.org/packages/Cornhsu.PolyMigrate) · [繁體中文說明](README.zh-Hant.md) · MIT**
+
+PolyMigrate turns legacy dynamic sites (old PHP sites and the like) into clean, static-site-ready
+Markdown — and it treats multilingual content as a first-class concern, not an afterthought.
+Config-driven, fully offline-rerunnable, built on .NET.
+
+**Status: 1.0.** The extraction pipeline, pairing, verification, thumbnails and orphan-page
+recovery are complete and validated against a real full-site migration (see below). The CLI
+surface and the Phase output contracts are now stable: new features bump the minor version,
+fixes bump the patch version.
+
+## Why
+
+Every multilingual institution site (governments, universities, NGOs, religious organizations)
+faces the same painful migration step: matching up the language versions of every page by hand.
+No existing tool solves this — general-purpose scrapers extract pages one at a time and leave
+the pairing to you. PolyMigrate:
+
+- **pairs automatically** where filenames are symmetric (`/ch/news/x` ↔ `/en/news/x` share a
+  `translation_key`),
+- **suggests pairs heuristically** where they are not — shared photo albums, normalized dates
+  hidden in slugs (`20240121` vs `01212024`), title similarity,
+- **honestly reports what it cannot pair**, producing a review-ready gap inventory instead of
+  guessing wrong.
+
+Languages are not limited to two: declare any number in `lang_map` and every output
+(frontmatter, inventories, pairing) expands accordingly. All locale output is standard BCP-47.
+
+## Battle-tested defaults
+
+The extraction pipeline bakes in fixes for real problems found during a real migration —
+things generic tools and LLM extractors silently get wrong:
+
+| Real-world pit | Built-in handling |
+|---|---|
+| Phone photos sideways/upside-down in thumbnails | EXIF auto-orientation before resizing |
+| Titles with colons / numeric slugs with leading zeros | YAML-library escaping, forced quoting |
+| `%20` double-encoding in image paths | decoded on disk, single-encoded in URLs |
+| Markdown converters dropping videos / iframes / PDFs | placeholder round-trip keeps embeds in place |
+| `<title>` polluted with dates and site name | body-first title extraction + configurable cleanup |
+| Mixed date formats (`YYYYMMDD` / `MMDDYYYY` / `DDMMYYYY`) | all recognized and normalized |
+| Old articles removed from indexes but still served | orphan probing (per-day URL candidates + suffix variants) |
+| Broken images on the source site | detected, recorded, never blocks the run |
+| Bot protection (JS cookie challenge → 409) | declarative cookie workaround in config |
+| Legacy encodings (Big5, GB2312, …) | declared or defaulted per site |
+
+## Case study: full temple-site migration
+
+PolyMigrate's pipeline is the productized version of a completed real migration
+(a bilingual Buddhist temple site, Chinese/English):
+
+- **516 pages** mirrored and extracted, **4.6 GB** of media
+- **281 translation keys**; **231 bilingual articles paired automatically** by symmetric paths
+- built-in verifier: **1,269 internal links + 4,116 media references checked — 0 errors**
+- 13 orphaned articles recovered via date probing; 141 EXIF-rotated photos fixed in thumbnails
+
+The original Python prototype's output was used as the golden baseline while porting: 466/516
+extracted bodies are byte-identical after whitespace normalization, and the remainder are
+render-equivalent or strictly more faithful.
+
+Two more numbers from that run:
+
+- **Re-runs are ~7× faster**: media hashes are cached by `(size, mtime)`, taking a full re-run
+  of the 4.6 GB site from **30.1 s down to 4.6 s**.
+- **The output is deployable as-is**: `redirect_map` is auto-filled with the new paths, and
+  PolyMigrate emits both an **nginx conf** and a **Netlify `_redirects`** file — turning a
+  half-day of hand-written 301s into copying one file.
+
+## Install & use
+
+Two channels — pick whichever runtime you already have. **Same tool, same behaviour.**
+
+```
+npx cornhsu-polymigrate extract site.yaml   # Node — no .NET install needed
+dotnet tool install -g Cornhsu.PolyMigrate  # .NET
+```
+
+> The npm build ships a self-contained native binary and only downloads the one
+> matching your platform (win32-x64 / linux-x64 / darwin-x64 / darwin-arm64).
+> Migrating a site is usually a one-off job — installing a whole SDK for a single
+> run is friction most people won't accept, so the tool meets you where you are.
+
+```
+dotnet tool install -g Cornhsu.PolyMigrate
+polymigrate extract site.yaml               # mirror HTML -> frontmatter Markdown + inventories
+polymigrate verify out/                     # link/media/frontmatter audit, CI-friendly exit codes
+polymigrate thumbs site.yaml                # EXIF-corrected, width-capped thumbnails
+polymigrate probe-orphans site.yaml --section news --years 2021-2023
+polymigrate fetch-orphans site.yaml --section news
+```
+
+One YAML config per site describes everything site-specific — see
+[examples/ibps-austin.yaml](examples/ibps-austin.yaml) for a fully-annotated real example.
+
+```yaml
+config_version: 1
+site:
+  base_url: https://legacy.example.org
+url_pattern:
+  lang_map: { ch: zh-Hant, en: en }     # any number of languages
+  default_lang: zh-Hant
+  strip_extensions: [.php]
+extract:
+  content: "section[id]:not(#header):not(#footer)"
+pairing:
+  strategy: symmetric_path
+  fallback: [shared_media, date, title_similarity]
+```
+
+## Layout
+
+| Path | Contents |
+|---|---|
+| `src/PolyMigrate.Core` | extraction / pairing / verification library (NuGet: `Cornhsu.PolyMigrate.Core`) |
+| `src/PolyMigrate.Cli` | the `polymigrate` CLI (NuGet tool package: `Cornhsu.PolyMigrate`) |
+| `tests/` | 119 unit/integration tests + an offline fixture site with golden-file baselines |
+| `docs/contracts.md` | file-format contracts between pipeline phases |
+
+## Development
+
+```
+dotnet build
+dotnet test
+dotnet run --project src/PolyMigrate.Cli -- --help
+```
+
+License: [MIT](LICENSE). All dependencies are MIT/BSD/Apache-2.0
+(imaging via **Magick.NET**; ImageSharp was dropped when its 4.x line began requiring a
+license key at build time).

package/bin/polymigrate.js

@@ -0,0 +1,68 @@
+#!/usr/bin/env node
+// npm 通路的啟動腳本。
+//
+// 為什麼要有這一層(而不是直接把執行檔當 bin):
+// npm 靠 optionalDependencies 的 os/cpu 只下載使用者當前平台的那一包,
+// 但執行時仍需要解析出它實際被裝到哪裡。
+//
+// 為什麼要有 npm 通路:PolyMigrate 的使用者是「要把舊網站搬成靜態站的人」,
+// 那些人多半在 Hugo / Eleventy / Astro / Next.js 的生態裡,手邊有 Node、
+// 不一定有 .NET SDK。而且搬站是一次性任務 —— 為了用一次而裝一整套 SDK,
+// 摩擦成本高到多數人會直接放棄。
+//
+// 刻意不做 postinstall 下載:那會被 `npm ci --ignore-scripts` 擋掉,
+// 而不少公司的 CI 預設就是關腳本的。
+
+"use strict";
+
+const { spawnSync } = require("node:child_process");
+const path = require("node:path");
+
+// 平台 → 子套件名 / 執行檔名
+const TARGETS = {
+  "win32 x64": { pkg: "@cornhsu/polymigrate-win32-x64", bin: "polymigrate.exe" },
+  "linux x64": { pkg: "@cornhsu/polymigrate-linux-x64", bin: "polymigrate" },
+  "darwin x64": { pkg: "@cornhsu/polymigrate-darwin-x64", bin: "polymigrate" },
+  "darwin arm64": { pkg: "@cornhsu/polymigrate-darwin-arm64", bin: "polymigrate" },
+};
+
+const key = `${process.platform} ${process.arch}`;
+const target = TARGETS[key];
+
+if (!target) {
+  console.error(
+    `Cornhsu.PolyMigrate 沒有 ${key} 的預先建置版本。\n` +
+      `目前支援:${Object.keys(TARGETS).join("、")}\n` +
+      `其他平台請改用 .NET 版本:dotnet tool install -g Cornhsu.PolyMigrate`
+  );
+  process.exit(1);
+}
+
+let binary;
+try {
+  // 從子套件的 package.json 反推安裝位置,不用猜 node_modules 的層級
+  // (pnpm / yarn 的實體結構與 npm 不同,require.resolve 才可靠)
+  const manifest = require.resolve(`${target.pkg}/package.json`);
+  binary = path.join(path.dirname(manifest), "bin", target.bin);
+} catch {
+  console.error(
+    `找不到平台套件 ${target.pkg}。\n` +
+      `可能是安裝時加了 --no-optional,或安裝過程中斷。\n` +
+      `請重新安裝:npm install cornhsu-polymigrate`
+  );
+  process.exit(1);
+}
+
+const result = spawnSync(binary, process.argv.slice(2), { stdio: "inherit" });
+
+if (result.error) {
+  console.error(`無法執行 ${binary}:${result.error.message}`);
+  process.exit(1);
+}
+
+// 被訊號中斷時 status 會是 null;轉成慣例的 128+signal,別讓 CI 誤判為成功
+if (result.status === null && result.signal) {
+  process.exit(128 + (require("node:os").constants.signals[result.signal] ?? 0));
+}
+
+process.exit(result.status ?? 0);

package/package.json

@@ -0,0 +1,43 @@
+{
+  "name": "cornhsu-polymigrate",
+  "version": "1.1.0",
+  "description": "The i18n-first static-site migrator — the only one that pairs your multilingual pages automatically. Mirrors a legacy site into frontmatter Markdown, auto-matches language versions, and honestly lists what it could not pair.",
+  "keywords": [
+    "i18n",
+    "internationalization",
+    "multilingual",
+    "site-migration",
+    "static-site",
+    "markdown",
+    "web-scraping",
+    "content-migration",
+    "hugo",
+    "eleventy",
+    "astro"
+  ],
+  "homepage": "https://cornhsu.com/polymigrate",
+  "bugs": "https://github.com/HSU-YU-MING/cornhsu-polymigrate/issues",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/HSU-YU-MING/cornhsu-polymigrate.git"
+  },
+  "license": "MIT",
+  "author": "許彧銘 Hsu Yu-Ming (https://cornhsu.com/)",
+  "type": "commonjs",
+  "bin": {
+    "polymigrate": "bin/polymigrate.js"
+  },
+  "files": [
+    "bin/polymigrate.js",
+    "README.md"
+  ],
+  "engines": {
+    "node": ">=18"
+  },
+  "optionalDependencies": {
+    "@cornhsu/polymigrate-win32-x64": "1.1.0",
+    "@cornhsu/polymigrate-linux-x64": "1.1.0",
+    "@cornhsu/polymigrate-darwin-x64": "1.1.0",
+    "@cornhsu/polymigrate-darwin-arm64": "1.1.0"
+  }
+}