@signalk/charts-plugin 3.7.0 → 3.8.1

package/.prettierignore

@@ -5,3 +5,4 @@ coverage
 plugin
 reports
 .stryker-tmp
+public/leaflet

package/CHANGELOG.md

@@ -0,0 +1,109 @@
+# Changelog
+
+This file tracks user-visible changes between released versions. For the
+full commit history, see the [git log](https://github.com/SignalK/charts-plugin/commits/master)
+or the [GitHub releases](https://github.com/SignalK/charts-plugin/releases).
+
+## v3.7.0 (unreleased)
+
+The big change in 3.7 is the `ChartDownloader` rewrite (PR #84) plus the
+follow-on architectural cleanup. Highlights:
+
+### Added
+
+- **Tile cache moved to mbtiles SQLite.** Per-provider working files at
+  `<cachePath>/.working/<id>/cache.mbtiles` replace the per-tile PNG layout
+  used in 3.6 and earlier. Faster on slow storage (Pi SD cards), survives
+  filesystem hiccups better, no inode pressure.
+- **Region snapshot exports** with `options.mbtiles: true` on a seeding
+  job. Output lands in `<cachePath>/exports/<region>_<provider-id>.mbtiles`
+  and is excluded from the chart scanner so it can be safely copied to
+  another Signal K server (e.g. via USB).
+- **Leaflet-based webapp** with live region drawing, multi-region seeding,
+  refresh-all-tiles and vacuum options, persistent regions storage, and
+  mobile-friendly layout. Replaces the earlier form-based UI.
+- **Declarative token providers** (`tokenProviders` config) for chart
+  sources that need a rotating bearer token fetched from a separate URL
+  (Navionics-via-Garmin, ArcGIS key rotation, OAuth client_credentials).
+  Token fetched on demand, cached for `ttlSeconds`, templated into the
+  tile URL and headers via `{token.<field>}` placeholders. Replaces the
+  early-iteration `.js` chart-provider auto-loader (dropped for security).
+  See README for the schema.
+- **Legacy PNG → mbtiles migration tool** at
+  `POST /signalk/chart-tiles/cache/<id>/migrate`. Walks an existing
+  pre-3.7 per-file cache and inserts each tile into the new mbtiles.
+  Idempotent, fire-and-forget, with status polling at the same URL via
+  GET. Solves the upgrade-and-lose-cache regression.
+- **New endpoints**: `GET /cache/stats` (per-provider hit/miss/failure
+  counters), `GET/POST /cache/regions` (persistent region storage for
+  the webapp), `GET /cache/<id>/migrate` (migration status).
+- **CHANGELOG.md** (this file).
+
+### Changed
+
+- Chart-folder scanner excludes `.working/`, `exports/`, and any
+  dot-prefixed entry at depth 0. User charts at any other path are
+  scanned as before.
+- Scanned mbtiles files are opened read-only (already in 3.6 via
+  `e7eb431`); 3.7 pairs that with the working/exports directory split
+  so the proxy and the scanner never compete for write access on the
+  same file.
+- Tile lists during seeding are generators rather than
+  pre-materialised arrays. Bounds memory on large bboxes; fixes the
+  OOM seen at high zoom levels (#48).
+- Disk-space safety check moved into the worker hot path (every 1000
+  tiles). When space frees up, caching resumes — the previous
+  one-way `CachingDisabled` latch is gone.
+- `POST /signalk/chart-tiles/cache/jobs/<id>` with `action='start'`
+  rejects non-Idle jobs with `409 Conflict`. The webapp already hid
+  the button; the endpoint is now consistent with that. Remove and
+  recreate the job to seed again.
+- Token fetches are coalesced: concurrent tile requests share one
+  in-flight token fetch instead of N parallel ones. An upstream
+  401/403 invalidates the cached token so the next request refetches.
+
+### Fixed / Security
+
+- The legacy-cache migration endpoint now refuses a `sourceName` that
+  resolves onto an installed directory chart (one carrying
+  `tilemapresource.xml` / `metadata.json`). Because `cachePath` defaults
+  to the chart-scan path and directory charts share the
+  `<z>/<x>/<y>` tile layout the migrator walks, `deleteSource: true`
+  could otherwise have deleted a user's installed chart.
+- Seed and delete jobs are fire-and-forget; a failure inside one
+  (locked DB, disk error) is now caught and recorded on the job instead
+  of surfacing as an unhandled promise rejection, which on Node 22+
+  would terminate the Signal K server.
+- `remoteUrl` and request `headers` are no longer included in the
+  `/resources/charts` view. For token providers these resolve a live
+  bearer token; clients fetch tiles through the proxy path and never
+  need the upstream target, so exposing them leaked the credential.
+- `@signalk/mbtiles` (and its `node:sqlite` dependency) is loaded
+  lazily, so a server on Node < 22 shows the plugin's "requires Node
+  22" message instead of crashing on `ERR_UNKNOWN_BUILTIN_MODULE`
+  before the guard can run. `engines.node` is set accordingly.
+- The writable mbtiles handle attaches an `error` listener so a runtime
+  SQLite error (e.g. disk full on an SD card) is logged rather than
+  crashing the process.
+
+### Migration / upgrade notes
+
+- On first start after upgrade, the plugin moves any pre-3.7
+  `<cachePath>/<id>.mbtiles_` and `<cachePath>/mbtiles/` files into
+  the new `.working/` and `exports/` layout. The migration is
+  idempotent, no admin action required.
+- Existing per-file PNG caches are NOT automatically imported. To
+  preserve them, run `POST /signalk/chart-tiles/cache/<id>/migrate`
+  per provider after upgrading. See README.
+- The `.js` chart-provider auto-loader from early PR #84 iterations
+  was removed. Replace any such modules with a `tokenProviders`
+  config entry.
+
+### Internal
+
+- `ChartDownloader` split across `src/chartDownloader.ts`,
+  `src/chartDownloaderTileHelpers.ts`, and
+  `src/chartDownloaderMBTilesHelpers.ts` for testability.
+- ~200 unit + HTTP integration tests added covering tile-coordinate
+  math, the SQLite helpers, token-provider TTL/template/401 paths,
+  cache-layout migration, and the new HTTP endpoints.

package/README.md

@@ -99,6 +99,78 @@ Example: NOAA Chart Display Service (S-57 ENC) as a cacheable tile source — se
 https://gis.charttools.noaa.gov/arcgis/rest/services/MCS/NOAAChartDisplay/MapServer/exts/MaritimeChartService/WMSServer?service=WMS&version=1.3.0&request=GetMap&layers=0,1,2,3,4,5,6,7,8,9,10,11,12&styles=&crs=EPSG:3857&bbox={bbox_3857}&width=256&height=256&format=image/png&transparent=true
 ```
 
+### Token providers (declarative dynamic providers)
+
+Some commercial chart sources require a short-lived bearer token fetched from a separate endpoint and rotated periodically (Navionics-via-Garmin, ArcGIS-style key rotation, OAuth client_credentials, etc). Configure these via `tokenProviders` in the plugin config. The plugin fetches the token, caches it for `ttlSeconds`, and templates it into the tile URL and headers per request. No code execution — declarative JSON only.
+
+```json
+{
+  "tokenProviders": [
+    {
+      "identifier": "navionics",
+      "name": "Navionics",
+      "type": "tilelayer",
+      "format": "png",
+      "scale": 50000,
+      "minzoom": 2,
+      "maxzoom": 19,
+      "bounds": [-180, -90, 180, 90],
+      "tokenEndpoint": {
+        "url": "https://example.com/api/token",
+        "method": "GET",
+        "headers": { "User-Agent": "..." },
+        "ttlSeconds": 600
+      },
+      "tile": {
+        "url": "https://tile{1-5}.example.com/v1/tile/{z}/{x}/{y}?config={token.configuration_token}",
+        "headers": { "Authorization": "Bearer {token.access_token}" }
+      }
+    }
+  ]
+}
+```
+
+Template placeholders inside `tile.url` and `tile.headers`:
+
+| Placeholder       | Replaced with                                                      |
+| ----------------- | ------------------------------------------------------------------ |
+| `{token.<field>}` | A string-valued field from the latest token-endpoint JSON response |
+| `{a-b}`           | A random integer in `[a, b]` inclusive (sharded tile hostnames)    |
+| `{z}` `{x}` `{y}` | Tile coords (substituted by the standard tile fetch path)          |
+
+Token fetches are coalesced — concurrent tile requests share one in-flight token fetch — and an upstream 401/403 invalidates the cached token so the next request refetches it. A token-endpoint failure is logged but doesn't take down tile requests; subsequent fetches retry.
+
+> [!WARNING]
+> The plugin executes only the configured HTTP requests. The configs themselves describe how to talk to a third-party service: confirm the provider's terms of use allow it before deploying. SignalK itself ships no provider configs that may break a third party's TOS.
+
+### Cache directory layout
+
+When `cachePath` is set (or defaulted to `~/.signalk/charts/`) the plugin organises its files into three subdirectories that the chart scanner treats differently:
+
+| Subdir      | Scanned? | Purpose                                                                                                                                           |
+| ----------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `.working/` | No       | Per-provider working mbtiles caches (`<id>/cache.mbtiles`); written to by the proxy.                                                              |
+| `exports/`  | No       | Region snapshot mbtiles produced by seeding jobs with the "separate file" option. Intended for offline transfer (USB stick to another SK server). |
+| (any other) | Yes      | Charts the user installed; opened read-only by the scanner.                                                                                       |
+
+The scanner also skips any top-level entry whose name starts with `.`. On startup the plugin migrates pre-3.7 layouts (`<id>.mbtiles_` and `mbtiles/` subdir) into this shape automatically; the migration is idempotent so a repeat startup is a no-op.
+
+### Migrating a pre-3.7 PNG cache
+
+Versions before 3.7 stored proxy-cached tiles as one PNG per tile under `<cachePath>/<provider.name>/<z>/<x>/<y>.<format>`. The 3.7 release moves all caching into mbtiles. To reuse an existing PNG cache instead of re-downloading:
+
+```bash
+# Kick off the migration (returns 202 immediately)
+curl -X POST http://localhost:3000/signalk/chart-tiles/cache/<provider-id>/migrate \
+  -H 'Content-Type: application/json' \
+  -d '{"sourceName": "<legacy-dir-name>", "deleteSource": false}'
+
+# Poll progress / final counts
+curl http://localhost:3000/signalk/chart-tiles/cache/<provider-id>/migrate
+```
+
+`sourceName` defaults to `provider.name` (the human-readable name the old cache was keyed under). `deleteSource: true` removes each PNG after a successful insert; the default keeps the source files in place so a botched migration is recoverable. The migration is idempotent — re-running after a failure picks up where the last attempt left off.
+
 ### Supported chart formats
 
 - [MBTiles](https://github.com/mapbox/mbtiles-spec) files

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@signalk/charts-plugin",
-  "version": "3.7.0",
+  "version": "3.8.1",
   "description": "Signal K plugin to provide chart support for Signal K server",
   "main": "plugin/index.js",
   "scripts": {
@@ -19,6 +19,9 @@
     "prepublishOnly": "npm run build"
   },
   "license": "Apache-2.0",
+  "engines": {
+    "node": ">=24"
+  },
   "keywords": [
     "signalk",
     "signalk-node-server-plugin",
@@ -36,7 +39,9 @@
     "displayName": "SK Charts",
     "appIcon": "./logo.svg",
     "screenshots": [
-      "./docs/screenshots/1_placeholder.png"
+      "./docs/screenshots/1_overview.png",
+      "./docs/screenshots/2_region.png",
+      "./docs/screenshots/3_jobs.png"
     ]
   },
   "dependencies": {
@@ -45,7 +50,7 @@
     "@turf/bbox": "^7.2.0",
     "@turf/boolean-intersects": "^7.2.0",
     "@turf/helpers": "^7.2.0",
-    "check-disk-space": "^3.4.0",
+    "check-disk-space": "^4.0.0",
     "fast-xml-parser": "^5.7.1",
     "geojson-antimeridian-cut": "^0.1.0",
     "lodash": "^4.17.11",
@@ -53,24 +58,24 @@
   },
   "repository": {
     "type": "git",
-    "url": "https://github.com/SignalK/charts-plugin"
+    "url": "git+https://github.com/SignalK/charts-plugin.git"
   },
   "devDependencies": {
-    "@stryker-mutator/core": "^9.6.1",
-    "@stryker-mutator/mocha-runner": "^9.6.1",
+    "@stryker-mutator/core": "^10.0.0",
+    "@stryker-mutator/mocha-runner": "^10.0.0",
     "@types/chai": "^5.2.3",
     "@types/express": "^5.0.6",
     "@types/lodash": "^4.14.191",
     "@types/mocha": "^10.0.10",
-    "@types/node": "^25.6.0",
-    "c8": "^11.0.0",
-    "chai": "6.2.2",
+    "@types/node": "^26.1.0",
+    "c8": "^12.0.0",
+    "chai": "6.3.0",
     "chai-http": "^5.1.2",
     "express": "^5.2.1",
-    "mocha": "^11.7.1",
+    "mocha": "^12.0.0",
     "prettier": "^3.8.2",
     "rimraf": "^6.1.3",
     "tsx": "^4.20.3",
-    "typescript": "^6.0.3"
+    "typescript": "^7.0.2"
   }
 }

package/plugin/cacheLayout.d.ts

@@ -0,0 +1,72 @@
+/**
+ * Cache directory layout.
+ *
+ * Three roles inside `cachePath`, kept as separate subdirectories so the
+ * scanner can tell them apart from user-installed charts:
+ *
+ *   ${cachePath}/
+ *     .working/<provider-id>/cache.mbtiles   - proxy working cache (RW), not scanned
+ *     exports/<name>_<provider-id>.mbtiles   - region snapshots for offline transfer, not scanned
+ *
+ * The scanner (src/charts.ts) excludes `.working` and `exports` by name at
+ * depth 0, plus any dot-prefixed entry. Anything else under `cachePath` is
+ * treated as a user chart and read-opened.
+ *
+ * `migrateLegacyCacheLayout()` is run once per startup to move any
+ * flat-layout proxy files (`<id>.mbtiles_`) and the older `mbtiles/`
+ * subdirectory of region snapshots into the layout above.
+ */
+import type { MBTilesHandle } from './types';
+export declare const WORKING_DIR = ".working";
+export declare const EXPORTS_DIR = "exports";
+/** Absolute path to a proxy provider's working mbtiles cache. */
+export declare function workingMbtilesPath(cachePath: string, providerIdentifier: string): string;
+/** Absolute path to a region-snapshot export file. */
+export declare function exportMbtilesPath(cachePath: string, regionName: string, providerIdentifier: string): string;
+/**
+ * Move any flat-layout proxy / export files into the .working/ and
+ * exports/ subdirs the scanner expects. Idempotent: re-running after
+ * success is a no-op. Failures on individual files are logged and
+ * skipped so a partial migration doesn't take the plugin out — the next
+ * startup retries.
+ *
+ * Migrates:
+ *   ${cachePath}/${id}.mbtiles_                  ->  .working/${id}/cache.mbtiles
+ *   ${cachePath}/mbtiles/${name}_${id}.mbtiles   ->  exports/${name}_${id}.mbtiles
+ */
+export declare function migrateLegacyCacheLayout(cachePath: string, app: {
+    debug: (msg: string) => void;
+}): Promise<void>;
+/**
+ * Counts returned by migrateLegacyTileCache. `migrated` is tiles inserted
+ * into mbtiles; `skipped` is non-matching files (subdirs / extensions we
+ * don't recognise / a stray README); `failed` is files we tried to migrate
+ * and couldn't (read or putTile errored). Distinguishing skip from fail
+ * matters when the admin reads the response: a high `failed` count is a
+ * real problem, a high `skipped` count usually isn't.
+ */
+export interface LegacyMigrationCounts {
+    migrated: number;
+    skipped: number;
+    failed: number;
+}
+/**
+ * Walk a flat-file tile cache rooted at `legacyDir`,
+ * `<root>/<z>/<x>/<y>.<ext>`, and insert each tile into the provider's
+ * working mbtiles via the standard putTile callback API.
+ *
+ * Idempotent: re-running after partial failure picks up where the last
+ * run left off (mbtiles' INSERT OR IGNORE-style write makes duplicate
+ * inserts cheap). `deleteSource: true` removes successfully migrated
+ * files so the source directory drains over time; the directory itself
+ * is left in place even when empty (admin tooling can rmdir later).
+ *
+ * Failures on individual tiles are counted and the walk continues:
+ * one corrupt PNG shouldn't block 49,999 healthy ones.
+ */
+export declare function migrateLegacyTileCache(legacyDir: string, mbtilesHandle: MBTilesHandle, opts?: {
+    deleteSource?: boolean;
+    onProgress?: (counts: LegacyMigrationCounts) => void;
+    signal?: AbortSignal;
+}): Promise<LegacyMigrationCounts>;
+//# sourceMappingURL=cacheLayout.d.ts.map

package/plugin/cacheLayout.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cacheLayout.d.ts","sourceRoot":"","sources":["../src/cacheLayout.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;GAiBG;AAIH,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,SAAS,CAAA;AAE5C,eAAO,MAAM,WAAW,aAAa,CAAA;AACrC,eAAO,MAAM,WAAW,YAAY,CAAA;AAEpC,iEAAiE;AACjE,wBAAgB,kBAAkB,CAChC,SAAS,EAAE,MAAM,EACjB,kBAAkB,EAAE,MAAM,GACzB,MAAM,CAER;AAED,sDAAsD;AACtD,wBAAgB,iBAAiB,CAC/B,SAAS,EAAE,MAAM,EACjB,UAAU,EAAE,MAAM,EAClB,kBAAkB,EAAE,MAAM,GACzB,MAAM,CAMR;AAED;;;;;;;;;;GAUG;AACH,wBAAsB,wBAAwB,CAC5C,SAAS,EAAE,MAAM,EACjB,GAAG,EAAE;IAAE,KAAK,EAAE,CAAC,GAAG,EAAE,MAAM,KAAK,IAAI,CAAA;CAAE,GACpC,OAAO,CAAC,IAAI,CAAC,CAgEf;AAED;;;;;;;GAOG;AACH,MAAM,WAAW,qBAAqB;IACpC,QAAQ,EAAE,MAAM,CAAA;IAChB,OAAO,EAAE,MAAM,CAAA;IACf,MAAM,EAAE,MAAM,CAAA;CACf;AAID;;;;;;;;;;;;;GAaG;AACH,wBAAsB,sBAAsB,CAC1C,SAAS,EAAE,MAAM,EACjB,aAAa,EAAE,aAAa,EAC5B,IAAI,GAAE;IACJ,YAAY,CAAC,EAAE,OAAO,CAAA;IACtB,UAAU,CAAC,EAAE,CAAC,MAAM,EAAE,qBAAqB,KAAK,IAAI,CAAA;IAKpD,MAAM,CAAC,EAAE,WAAW,CAAA;CAChB,GACL,OAAO,CAAC,qBAAqB,CAAC,CA+FhC"}

package/plugin/cacheLayout.js

@@ -0,0 +1,243 @@
+"use strict";
+/**
+ * Cache directory layout.
+ *
+ * Three roles inside `cachePath`, kept as separate subdirectories so the
+ * scanner can tell them apart from user-installed charts:
+ *
+ *   ${cachePath}/
+ *     .working/<provider-id>/cache.mbtiles   - proxy working cache (RW), not scanned
+ *     exports/<name>_<provider-id>.mbtiles   - region snapshots for offline transfer, not scanned
+ *
+ * The scanner (src/charts.ts) excludes `.working` and `exports` by name at
+ * depth 0, plus any dot-prefixed entry. Anything else under `cachePath` is
+ * treated as a user chart and read-opened.
+ *
+ * `migrateLegacyCacheLayout()` is run once per startup to move any
+ * flat-layout proxy files (`<id>.mbtiles_`) and the older `mbtiles/`
+ * subdirectory of region snapshots into the layout above.
+ */
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.EXPORTS_DIR = exports.WORKING_DIR = void 0;
+exports.workingMbtilesPath = workingMbtilesPath;
+exports.exportMbtilesPath = exportMbtilesPath;
+exports.migrateLegacyCacheLayout = migrateLegacyCacheLayout;
+exports.migrateLegacyTileCache = migrateLegacyTileCache;
+const path_1 = __importDefault(require("path"));
+const fs_1 = require("fs");
+exports.WORKING_DIR = '.working';
+exports.EXPORTS_DIR = 'exports';
+/** Absolute path to a proxy provider's working mbtiles cache. */
+function workingMbtilesPath(cachePath, providerIdentifier) {
+    return path_1.default.join(cachePath, exports.WORKING_DIR, providerIdentifier, 'cache.mbtiles');
+}
+/** Absolute path to a region-snapshot export file. */
+function exportMbtilesPath(cachePath, regionName, providerIdentifier) {
+    return path_1.default.join(cachePath, exports.EXPORTS_DIR, `${regionName}_${providerIdentifier}.mbtiles`);
+}
+/**
+ * Move any flat-layout proxy / export files into the .working/ and
+ * exports/ subdirs the scanner expects. Idempotent: re-running after
+ * success is a no-op. Failures on individual files are logged and
+ * skipped so a partial migration doesn't take the plugin out — the next
+ * startup retries.
+ *
+ * Migrates:
+ *   ${cachePath}/${id}.mbtiles_                  ->  .working/${id}/cache.mbtiles
+ *   ${cachePath}/mbtiles/${name}_${id}.mbtiles   ->  exports/${name}_${id}.mbtiles
+ */
+async function migrateLegacyCacheLayout(cachePath, app) {
+    // Flat-layout proxy files (.mbtiles_ trailing underscore).
+    let entries;
+    try {
+        entries = await fs_1.promises.readdir(cachePath, { withFileTypes: true });
+    }
+    catch {
+        // Cache path doesn't exist yet — nothing to migrate.
+        return;
+    }
+    for (const entry of entries) {
+        if (!entry.isFile())
+            continue;
+        const m = entry.name.match(/^(.+)\.mbtiles_$/);
+        if (!m)
+            continue;
+        const id = m[1];
+        const src = path_1.default.join(cachePath, entry.name);
+        const dst = workingMbtilesPath(cachePath, id);
+        try {
+            await fs_1.promises.mkdir(path_1.default.dirname(dst), { recursive: true });
+            await fs_1.promises.rename(src, dst);
+            app.debug(`Migrated proxy cache ${src} -> ${dst}`);
+        }
+        catch (err) {
+            app.debug(`Could not migrate ${src} -> ${dst}: ${err.message}`);
+        }
+    }
+    // Flat-layout region-snapshot directory (cachePath/mbtiles/).
+    const oldExportDir = path_1.default.join(cachePath, 'mbtiles');
+    let oldExports = [];
+    try {
+        oldExports = await fs_1.promises.readdir(oldExportDir, { withFileTypes: true });
+    }
+    catch (err) {
+        // ENOENT: legacy dir doesn't exist, nothing to migrate. Any other
+        // error (EACCES, EBUSY, EIO) means the dir might exist but we
+        // couldn't read it — surface that so the admin sees half-migrated
+        // state instead of a silent skip.
+        const code = err.code;
+        if (code !== 'ENOENT') {
+            console.warn(`[charts-plugin] could not read legacy export dir ${oldExportDir}: ` +
+                `${err.message} ` +
+                `(proxy-cache migration above may have completed; manual cleanup needed)`);
+        }
+        return;
+    }
+    for (const entry of oldExports) {
+        if (!entry.isFile() || !entry.name.match(/\.mbtiles$/i))
+            continue;
+        const src = path_1.default.join(oldExportDir, entry.name);
+        const dst = path_1.default.join(cachePath, exports.EXPORTS_DIR, entry.name);
+        try {
+            await fs_1.promises.mkdir(path_1.default.dirname(dst), { recursive: true });
+            await fs_1.promises.rename(src, dst);
+            app.debug(`Migrated export ${src} -> ${dst}`);
+        }
+        catch (err) {
+            app.debug(`Could not migrate ${src} -> ${dst}: ${err.message}`);
+        }
+    }
+    // Best-effort cleanup of the now-empty legacy export dir; ignore failure
+    // (e.g. user dropped a non-mbtiles file in there manually).
+    try {
+        await fs_1.promises.rmdir(oldExportDir);
+    }
+    catch {
+        // leave it; harmless
+    }
+}
+const TILE_EXTENSION = /\.(png|jpe?g|webp|pbf|mvt)$/i;
+/**
+ * Walk a flat-file tile cache rooted at `legacyDir`,
+ * `<root>/<z>/<x>/<y>.<ext>`, and insert each tile into the provider's
+ * working mbtiles via the standard putTile callback API.
+ *
+ * Idempotent: re-running after partial failure picks up where the last
+ * run left off (mbtiles' INSERT OR IGNORE-style write makes duplicate
+ * inserts cheap). `deleteSource: true` removes successfully migrated
+ * files so the source directory drains over time; the directory itself
+ * is left in place even when empty (admin tooling can rmdir later).
+ *
+ * Failures on individual tiles are counted and the walk continues:
+ * one corrupt PNG shouldn't block 49,999 healthy ones.
+ */
+async function migrateLegacyTileCache(legacyDir, mbtilesHandle, opts = {}) {
+    const counts = {
+        migrated: 0,
+        skipped: 0,
+        failed: 0
+    };
+    let zEntries;
+    try {
+        zEntries = await fs_1.promises.readdir(legacyDir, { withFileTypes: true });
+    }
+    catch (err) {
+        // Missing legacyDir is the same as "nothing to migrate" — return zeros.
+        if (err.code === 'ENOENT')
+            return counts;
+        throw err;
+    }
+    for (const zEntry of zEntries) {
+        if (opts.signal?.aborted)
+            return counts;
+        if (!zEntry.isDirectory()) {
+            counts.skipped++;
+            continue;
+        }
+        const z = parseInt(zEntry.name, 10);
+        if (!Number.isFinite(z)) {
+            counts.skipped++;
+            continue;
+        }
+        const zDir = path_1.default.join(legacyDir, zEntry.name);
+        let xEntries;
+        try {
+            xEntries = await fs_1.promises.readdir(zDir, { withFileTypes: true });
+        }
+        catch {
+            counts.failed++;
+            continue;
+        }
+        for (const xEntry of xEntries) {
+            if (opts.signal?.aborted)
+                return counts;
+            if (!xEntry.isDirectory()) {
+                counts.skipped++;
+                continue;
+            }
+            const x = parseInt(xEntry.name, 10);
+            if (!Number.isFinite(x)) {
+                counts.skipped++;
+                continue;
+            }
+            const xDir = path_1.default.join(zDir, xEntry.name);
+            let yEntries;
+            try {
+                yEntries = await fs_1.promises.readdir(xDir, { withFileTypes: true });
+            }
+            catch {
+                counts.failed++;
+                continue;
+            }
+            for (const yEntry of yEntries) {
+                if (opts.signal?.aborted)
+                    return counts;
+                if (!yEntry.isFile() || !TILE_EXTENSION.test(yEntry.name)) {
+                    counts.skipped++;
+                    continue;
+                }
+                const yBase = yEntry.name.replace(TILE_EXTENSION, '');
+                const y = parseInt(yBase, 10);
+                if (!Number.isFinite(y)) {
+                    counts.skipped++;
+                    continue;
+                }
+                const tilePath = path_1.default.join(xDir, yEntry.name);
+                try {
+                    const buffer = await fs_1.promises.readFile(tilePath);
+                    await new Promise((resolve, reject) => {
+                        mbtilesHandle.putTile(z, x, y, buffer, (err) => {
+                            if (err)
+                                reject(err);
+                            else
+                                resolve();
+                        });
+                    });
+                    counts.migrated++;
+                    if (opts.deleteSource) {
+                        try {
+                            await fs_1.promises.unlink(tilePath);
+                        }
+                        catch {
+                            // Source delete is best-effort; mbtiles already has the tile.
+                        }
+                    }
+                }
+                catch {
+                    counts.failed++;
+                }
+                // Periodic progress signal (every 500 processed) so a long-running
+                // migration can surface progress to the caller without flooding.
+                const total = counts.migrated + counts.skipped + counts.failed;
+                if (opts.onProgress && total % 500 === 0)
+                    opts.onProgress(counts);
+            }
+        }
+    }
+    opts.onProgress?.(counts);
+    return counts;
+}
+//# sourceMappingURL=cacheLayout.js.map

package/plugin/cacheLayout.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cacheLayout.js","sourceRoot":"","sources":["../src/cacheLayout.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;;;;GAiBG;;;;;;;;;;AAEH,gDAAuB;AACvB,2BAAmC;AAGtB,QAAA,WAAW,GAAG,UAAU,CAAA;AACxB,QAAA,WAAW,GAAG,SAAS,CAAA;AAEpC,iEAAiE;AACjE,4BACE,SAAiB,EACjB,kBAA0B;IAE1B,OAAO,cAAI,CAAC,IAAI,CAAC,SAAS,EAAE,QAAA,WAAW,EAAE,kBAAkB,EAAE,eAAe,CAAC,CAAA;AAC/E,CAAC;AAED,sDAAsD;AACtD,2BACE,SAAiB,EACjB,UAAkB,EAClB,kBAA0B;IAE1B,OAAO,cAAI,CAAC,IAAI,CACd,SAAS,EACT,QAAA,WAAW,EACX,GAAG,UAAU,IAAI,kBAAkB,UAAU,CAC9C,CAAA;AACH,CAAC;AAED;;;;;;;;;;GAUG;AACI,KAAK,mCACV,SAAiB,EACjB,GAAqC;IAErC,2DAA2D;IAC3D,IAAI,OAA8B,CAAA;IAClC,IAAI,CAAC;QACH,OAAO,GAAG,MAAM,aAAE,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,CAAA;IAChE,CAAC;IAAC,MAAM,CAAC;QACP,qDAAqD;QACrD,OAAM;IACR,CAAC;IACD,KAAK,MAAM,KAAK,IAAI,OAAO,EAAE,CAAC;QAC5B,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE;YAAE,SAAQ;QAC7B,MAAM,CAAC,GAAG,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,kBAAkB,CAAC,CAAA;QAC9C,IAAI,CAAC,CAAC;YAAE,SAAQ;QAChB,MAAM,EAAE,GAAG,CAAC,CAAC,CAAC,CAAE,CAAA;QAChB,MAAM,GAAG,GAAG,cAAI,CAAC,IAAI,CAAC,SAAS,EAAE,KAAK,CAAC,IAAI,CAAC,CAAA;QAC5C,MAAM,GAAG,GAAG,kBAAkB,CAAC,SAAS,EAAE,EAAE,CAAC,CAAA;QAC7C,IAAI,CAAC;YACH,MAAM,aAAE,CAAC,KAAK,CAAC,cAAI,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAA;YACtD,MAAM,aAAE,CAAC,MAAM,CAAC,GAAG,EAAE,GAAG,CAAC,CAAA;YACzB,GAAG,CAAC,KAAK,CAAC,wBAAwB,GAAG,OAAO,GAAG,EAAE,CAAC,CAAA;QACpD,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,GAAG,CAAC,KAAK,CAAC,qBAAqB,GAAG,OAAO,GAAG,KAAM,GAAa,CAAC,OAAO,EAAE,CAAC,CAAA;QAC5E,CAAC;IACH,CAAC;IAED,8DAA8D;IAC9D,MAAM,YAAY,GAAG,cAAI,CAAC,IAAI,CAAC,SAAS,EAAE,SAAS,CAAC,CAAA;IACpD,IAAI,UAAU,GAA0B,EAAE,CAAA;IAC1C,IAAI,CAAC;QACH,UAAU,GAAG,MAAM,aAAE,CAAC,OAAO,CAAC,YAAY,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,CAAA;IACtE,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,kEAAkE;QAClE,8DAA8D;QAC9D,kEAAkE;QAClE,kCAAkC;QAClC,MAAM,IAAI,GAAI,GAA6B,CAAC,IAAI,CAAA;QAChD,IAAI,IAAI,KAAK,QAAQ,EAAE,CAAC;YACtB,OAAO,CAAC,IAAI,CACV,oDAAoD,YAAY,IAAI;gBAClE,GAAI,GAAa,CAAC,OAAO,GAAG;gBAC5B,yEAAyE,CAC5E,CAAA;QACH,CAAC;QACD,OAAM;IACR,CAAC;IACD,KAAK,MAAM,KAAK,IAAI,UAAU,EAAE,CAAC;QAC/B,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,aAAa,CAAC;YAAE,SAAQ;QACjE,MAAM,GAAG,GAAG,cAAI,CAAC,IAAI,CAAC,YAAY,EAAE,KAAK,CAAC,IAAI,CAAC,CAAA;QAC/C,MAAM,GAAG,GAAG,cAAI,CAAC,IAAI,CAAC,SAAS,EAAE,QAAA,WAAW,EAAE,KAAK,CAAC,IAAI,CAAC,CAAA;QACzD,IAAI,CAAC;YACH,MAAM,aAAE,CAAC,KAAK,CAAC,cAAI,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAA;YACtD,MAAM,aAAE,CAAC,MAAM,CAAC,GAAG,EAAE,GAAG,CAAC,CAAA;YACzB,GAAG,CAAC,KAAK,CAAC,mBAAmB,GAAG,OAAO,GAAG,EAAE,CAAC,CAAA;QAC/C,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,GAAG,CAAC,KAAK,CAAC,qBAAqB,GAAG,OAAO,GAAG,KAAM,GAAa,CAAC,OAAO,EAAE,CAAC,CAAA;QAC5E,CAAC;IACH,CAAC;IACD,yEAAyE;IACzE,4DAA4D;IAC5D,IAAI,CAAC;QACH,MAAM,aAAE,CAAC,KAAK,CAAC,YAAY,CAAC,CAAA;IAC9B,CAAC;IAAC,MAAM,CAAC;QACP,qBAAqB;IACvB,CAAC;AACH,CAAC;AAgBD,MAAM,cAAc,GAAG,8BAA8B,CAAA;AAErD;;;;;;;;;;;;;GAaG;AACI,KAAK,iCACV,SAAiB,EACjB,aAA4B,EAC5B,IAAI,GAQA,EAAE;IAEN,MAAM,MAAM,GAA0B;QACpC,QAAQ,EAAE,CAAC;QACX,OAAO,EAAE,CAAC;QACV,MAAM,EAAE,CAAC;KACV,CAAA;IAED,IAAI,QAA+B,CAAA;IACnC,IAAI,CAAC;QACH,QAAQ,GAAG,MAAM,aAAE,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,CAAA;IACjE,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,wEAAwE;QACxE,IAAK,GAA6B,CAAC,IAAI,KAAK,QAAQ;YAAE,OAAO,MAAM,CAAA;QACnE,MAAM,GAAG,CAAA;IACX,CAAC;IAED,KAAK,MAAM,MAAM,IAAI,QAAQ,EAAE,CAAC;QAC9B,IAAI,IAAI,CAAC,MAAM,EAAE,OAAO;YAAE,OAAO,MAAM,CAAA;QACvC,IAAI,CAAC,MAAM,CAAC,WAAW,EAAE,EAAE,CAAC;YAC1B,MAAM,CAAC,OAAO,EAAE,CAAA;YAChB,SAAQ;QACV,CAAC;QACD,MAAM,CAAC,GAAG,QAAQ,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,CAAC,CAAA;QACnC,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,CAAC;YACxB,MAAM,CAAC,OAAO,EAAE,CAAA;YAChB,SAAQ;QACV,CAAC;QACD,MAAM,IAAI,GAAG,cAAI,CAAC,IAAI,CAAC,SAAS,EAAE,MAAM,CAAC,IAAI,CAAC,CAAA;QAC9C,IAAI,QAA+B,CAAA;QACnC,IAAI,CAAC;YACH,QAAQ,GAAG,MAAM,aAAE,CAAC,OAAO,CAAC,IAAI,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,CAAA;QAC5D,CAAC;QAAC,MAAM,CAAC;YACP,MAAM,CAAC,MAAM,EAAE,CAAA;YACf,SAAQ;QACV,CAAC;QACD,KAAK,MAAM,MAAM,IAAI,QAAQ,EAAE,CAAC;YAC9B,IAAI,IAAI,CAAC,MAAM,EAAE,OAAO;gBAAE,OAAO,MAAM,CAAA;YACvC,IAAI,CAAC,MAAM,CAAC,WAAW,EAAE,EAAE,CAAC;gBAC1B,MAAM,CAAC,OAAO,EAAE,CAAA;gBAChB,SAAQ;YACV,CAAC;YACD,MAAM,CAAC,GAAG,QAAQ,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,CAAC,CAAA;YACnC,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,CAAC;gBACxB,MAAM,CAAC,OAAO,EAAE,CAAA;gBAChB,SAAQ;YACV,CAAC;YACD,MAAM,IAAI,GAAG,cAAI,CAAC,IAAI,CAAC,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,CAAA;YACzC,IAAI,QAA+B,CAAA;YACnC,IAAI,CAAC;gBACH,QAAQ,GAAG,MAAM,aAAE,CAAC,OAAO,CAAC,IAAI,EAAE,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,CAAA;YAC5D,CAAC;YAAC,MAAM,CAAC;gBACP,MAAM,CAAC,MAAM,EAAE,CAAA;gBACf,SAAQ;YACV,CAAC;YACD,KAAK,MAAM,MAAM,IAAI,QAAQ,EAAE,CAAC;gBAC9B,IAAI,IAAI,CAAC,MAAM,EAAE,OAAO;oBAAE,OAAO,MAAM,CAAA;gBACvC,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE,IAAI,CAAC,cAAc,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC;oBAC1D,MAAM,CAAC,OAAO,EAAE,CAAA;oBAChB,SAAQ;gBACV,CAAC;gBACD,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,cAAc,EAAE,EAAE,CAAC,CAAA;gBACrD,MAAM,CAAC,GAAG,QAAQ,CAAC,KAAK,EAAE,EAAE,CAAC,CAAA;gBAC7B,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,CAAC;oBACxB,MAAM,CAAC,OAAO,EAAE,CAAA;oBAChB,SAAQ;gBACV,CAAC;gBACD,MAAM,QAAQ,GAAG,cAAI,CAAC,IAAI,CAAC,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,CAAA;gBAC7C,IAAI,CAAC;oBACH,MAAM,MAAM,GAAG,MAAM,aAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAA;oBAC1C,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;wBAC1C,aAAa,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,GAAG,EAAE,EAAE;4BAC7C,IAAI,GAAG;gCAAE,MAAM,CAAC,GAAG,CAAC,CAAA;;gCACf,OAAO,EAAE,CAAA;wBAChB,CAAC,CAAC,CAAA;oBACJ,CAAC,CAAC,CAAA;oBACF,MAAM,CAAC,QAAQ,EAAE,CAAA;oBACjB,IAAI,IAAI,CAAC,YAAY,EAAE,CAAC;wBACtB,IAAI,CAAC;4BACH,MAAM,aAAE,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAA;wBAC3B,CAAC;wBAAC,MAAM,CAAC;4BACP,8DAA8D;wBAChE,CAAC;oBACH,CAAC;gBACH,CAAC;gBAAC,MAAM,CAAC;oBACP,MAAM,CAAC,MAAM,EAAE,CAAA;gBACjB,CAAC;gBACD,mEAAmE;gBACnE,iEAAiE;gBACjE,MAAM,KAAK,GAAG,MAAM,CAAC,QAAQ,GAAG,MAAM,CAAC,OAAO,GAAG,MAAM,CAAC,MAAM,CAAA;gBAC9D,IAAI,IAAI,CAAC,UAAU,IAAI,KAAK,GAAG,GAAG,KAAK,CAAC;oBAAE,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,CAAA;YACnE,CAAC;QACH,CAAC;IACH,CAAC;IACD,IAAI,CAAC,UAAU,EAAE,CAAC,MAAM,CAAC,CAAA;IACzB,OAAO,MAAM,CAAA;AACf,CAAC"}