@walkthru-earth/objex-utils 1.0.0 → 1.2.0

package/docs/README.md

@@ -0,0 +1,102 @@
+# `@walkthru-earth/objex-utils` Developer Docs
+
+Professional reference documentation for every public export of the package.
+
+Each page lists the exact TypeScript signature, parameter semantics, return shape, peer-dependency requirements, and non-obvious behavior. Intended as the single source of truth for integrators.
+
+## Install
+
+```bash
+pnpm add @walkthru-earth/objex-utils
+# or
+npm install @walkthru-earth/objex-utils
+```
+
+Minimum: ES2022, modern browser or Node 18+. Ships both ESM (`dist/index.js`) and CJS (`dist/index.cjs`) with full `.d.ts`.
+
+## Optional peer dependencies
+
+All heavy dependencies are **optional** peer dependencies. Install only the ones you touch.
+
+| Peer | Version | Required by |
+|------|---------|-------------|
+| `apache-arrow` | `>=14` | `buildGeoArrowTables`, `normalizeGeomType` (indirectly) |
+| `hyparquet` | `>=1.25` | `readParquetMetadata` and its extractors |
+| `hyparquet-compressors` | `>=1.1` | `readParquetMetadata` (for SNAPPY/ZSTD/GZIP/etc.) |
+| `yaml` | `>=2` | `parseMarkdownDocument` (lazy-loaded, only when YAML frontmatter is present) |
+
+As of v1.2, `yaml` is loaded via dynamic `import()` inside `parseMarkdownDocument`. Consumers who do not call that function never touch `yaml` at load time.
+
+## Reference pages
+
+| Page | Covers |
+|------|--------|
+| [`geometry.md`](./geometry.md) | WKB parser, GeoArrow table builder, geometry-column detection |
+| [`cog.md`](./cog.md) | Cloud-Optimized GeoTIFF helpers (pipeline selection, band configs, color ramps, bounds clamping) |
+| [`parquet-metadata.md`](./parquet-metadata.md) | `readParquetMetadata` + CRS / bounds / geometry-types extractors |
+| [`storage.md`](./storage.md) | URL parsing (`parseStorageUrl`, `resolveCloudUrl`), provider registry, `StorageAdapter` interface, `UrlAdapter` |
+| [`query-engine.md`](./query-engine.md) | `QueryEngine` interface and associated result/handle types |
+| [`file-types.md`](./file-types.md) | File-type registry (`getFileTypeInfo`, `getViewerKind`, `getDuckDbReadFn`, ...) |
+| [`formatting.md`](./formatting.md) | Display formatters, column-type classification, hex dump, CSV/JSON export |
+| [`file-sort.md`](./file-sort.md) | `sortFileEntries`, `toggleSortField` |
+| [`markdown-sql.md`](./markdown-sql.md) | Markdown + SQL block parsing (Evidence-compatible syntax) |
+| [`local-storage.md`](./local-storage.md) | SSR-safe `loadFromStorage` / `persistToStorage` |
+| [`errors.md`](./errors.md) | `handleLoadError` |
+| [`types-constants.md`](./types-constants.md) | `Connection`, `Tab`, `FileEntry`, `WriteResult`, `Theme`, shared constants |
+
+## Quick recipes
+
+### Render a GeoParquet file on deck.gl
+
+```ts
+import {
+  readParquetMetadata,
+  extractEpsgFromGeoMeta,
+  extractGeometryTypes,
+  buildGeoArrowTables
+} from '@walkthru-earth/objex-utils';
+
+// 1. Read footer metadata (~512KB range requests, no DuckDB)
+const meta = await readParquetMetadata(url);
+const sourceCrs = extractEpsgFromGeoMeta(meta.geo!);   // e.g. 'EPSG:27700' or null
+
+// 2. Get WKB buffers + attributes from your SQL engine (DuckDB-WASM, etc.)
+const { wkbArrays, attributes } = await myEngine.queryForMap(...);
+
+// 3. Build GeoArrow tables (one per geometry type)
+const tables = buildGeoArrowTables(wkbArrays, attributes);
+
+// 4. Feed into deck.gl GeoArrowLayer(s)
+```
+
+### Parse a cloud storage URL a user pasted
+
+```ts
+import { parseStorageUrl, looksLikeUrl, describeParseResult } from '@walkthru-earth/objex-utils';
+
+const parsed = parseStorageUrl(input);
+// { provider: 's3', bucket: 'my-bucket', region: 'us-east-1', endpoint: '...', prefix: '...' }
+
+console.log(describeParseResult(parsed));
+```
+
+### Read a binary blob as a hex dump
+
+```ts
+import { generateHexDump } from '@walkthru-earth/objex-utils';
+
+const rows = generateHexDump(bytes);
+// rows: [{ offset: '00000000', hex: ['48','65','6c',...], ascii: 'Hello...' }, ...]
+```
+
+## Upstream source
+
+All modules re-export from `src/lib/` in the [objex](https://github.com/walkthru-earth/objex) monorepo. The re-export list lives at [`packages/objex-utils/src/index.ts`](../src/index.ts).
+
+## Versioning & releases
+
+Both `@walkthru-earth/objex` and `@walkthru-earth/objex-utils` ship together via Changesets with a `fixed` config, so their versions stay in lockstep. See [`RELEASE.md`](../../../RELEASE.md) in the monorepo root.
+
+## License
+
+[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

package/docs/cog.md

@@ -0,0 +1,303 @@
+# Cloud-Optimized GeoTIFF (COG) helpers
+
+Pure helpers plus lazy wrappers for rendering Cloud-Optimized GeoTIFFs with [`@developmentseed/deck.gl-geotiff`](https://github.com/developmentseed/deck.gl-geotiff) v0.5 and MapLibre GL.
+
+The "pure" subset (types, safe-math, data-type labels, color ramps) has no native dependencies. The rendering-oriented functions pull in `@developmentseed/geotiff`, `@developmentseed/deck.gl-geotiff`, `maplibre-gl`, `proj4`, and `@developmentseed/proj` — treat all of those as **optional** peers.
+
+Source: `src/lib/utils/cog.ts`.
+
+## Types
+
+### `GeoBounds`
+
+```ts
+interface GeoBounds {
+  west: number;
+  south: number;
+  east: number;
+  north: number;
+}
+```
+
+### `CogInfo`
+
+```ts
+interface CogInfo {
+  width: number;
+  height: number;
+  bandCount: number;
+  dataType: string;      // e.g. 'uint8', 'float32', 'int16'
+  bounds: GeoBounds;
+  downsampled?: boolean; // true when renderNonTiledBitmap decimated the array
+}
+```
+
+### `BandConfig`
+
+```ts
+interface BandConfig {
+  mode: 'rgb' | 'single';
+  rBand: number;           // 0-indexed
+  gBand: number;
+  bBand: number;
+  band: number;            // single-band mode selection
+  colorRamp: ColorRampId;
+}
+```
+
+### `RescaleConfig`
+
+```ts
+interface RescaleConfig {
+  min: number;
+  max: number;
+}
+```
+
+### `CogTagInfo` / `ResolvedCogPipeline`
+
+Internal-but-exported descriptors returned by `inspectCogTags` / `selectCogPipeline`. See source for the full field list; the common shape is "metadata about which rendering pipeline should be used and why."
+
+## Pure helpers (no peer deps)
+
+### `SF_LABELS`
+
+```ts
+const SF_LABELS: Record<number, string> = {
+  1: 'uint', 2: 'int', 3: 'float', 4: 'void',
+  5: 'complex int', 6: 'complex float',
+};
+```
+
+Human labels for the GeoTIFF SampleFormat tag.
+
+### `safeClamp(v, lo, hi, fallback)`
+
+```ts
+function safeClamp(v: number, lo: number, hi: number, fallback: number): number
+```
+
+Clamp `v` into `[lo, hi]`. If `v` is `NaN` or `±Infinity`, return `fallback`. Use instead of `Math.min(hi, Math.max(lo, v))` — NaN would otherwise propagate silently.
+
+### `clampBounds(b)`
+
+```ts
+function clampBounds(b: GeoBounds): GeoBounds
+```
+
+Clamp geographic bounds to valid MapLibre Web Mercator range: ±180° longitude, ±85.051129° latitude.
+
+### `buildDataTypeLabel(sampleFormat, bitsPerSample)`
+
+```ts
+function buildDataTypeLabel(
+  sampleFormat: number,
+  bitsPerSample: number
+): string
+```
+
+Combine the GeoTIFF `SampleFormat` tag with `BitsPerSample` into `'uint8'`, `'int16'`, `'float32'`, etc.
+
+### `defaultBandConfig(bandCount, sampleFormat)`
+
+```ts
+function defaultBandConfig(bandCount: number, sampleFormat: number): BandConfig
+```
+
+Sensible initial band config: RGB mode for ≥3 bands, single-band with a terrain ramp for int/float single-band imagery.
+
+### `isDefaultBandConfig(config, bandCount, sampleFormat)`
+
+```ts
+function isDefaultBandConfig(
+  config: BandConfig,
+  bandCount: number,
+  sampleFormat: number
+): boolean
+```
+
+`true` when `config` equals the default for this COG — used to short-circuit custom-pipeline selection.
+
+### Color ramps
+
+```ts
+type ColorRampId = 'grayscale' | 'terrain' | 'viridis' | 'magma' | 'turbo' | 'spectral';
+
+const COLOR_RAMP_STOPS: Record<ColorRampId, [number, number, number][]>;
+
+function interpolateRamp(
+  stops: [number, number, number][],
+  t: number
+): [number, number, number];
+
+function rampToGradientCss(id: ColorRampId): string;
+```
+
+`interpolateRamp(stops, t)` returns an RGB triple for normalized `t ∈ [0, 1]` (clamped). `rampToGradientCss` produces a CSS `linear-gradient(...)` string for UI previews.
+
+## Pipeline helpers (require peer deps)
+
+### `inspectCogTags(geotiff)`
+
+```ts
+function inspectCogTags(geotiff: GeoTIFF): CogTagInfo
+```
+
+Centralized reader for the TIFF tags that determine which render pipeline to use (Photometric.Palette, SampleFormat, ColorMap, etc.).
+
+### `needsCustomPipeline(geotiff)`
+
+```ts
+function needsCustomPipeline(geotiff: GeoTIFF): boolean
+```
+
+`true` when the library's `inferRenderPipeline` can't handle this COG (non-uint sample formats). Triggers the custom JS pipeline.
+
+### `needsCustomPipelineForConfig(geotiff, config)`
+
+```ts
+function needsCustomPipelineForConfig(
+  geotiff: GeoTIFF,
+  config: BandConfig
+): boolean
+```
+
+`true` when `config` deviates from what the library default pipeline supports (non-standard RGB band order, single-band mode, non-uint, non-palette).
+
+### `createCustomGetTileData(geotiff)` / `customRenderTile(data)`
+
+```ts
+function createCustomGetTileData(
+  geotiff: GeoTIFF
+): (image, options) => Promise<CustomTileData>;
+
+function customRenderTile(data: CustomTileData): { image: ImageData };
+```
+
+Band-0-only fallback path for non-uint COGs. Uses GDAL statistics when available, otherwise per-tile adaptive stretch, and applies the `terrain` color ramp.
+
+### `createConfigurableGetTileData(geotiff, config)`
+
+```ts
+function createConfigurableGetTileData(
+  geotiff: GeoTIFF,
+  config: BandConfig
+): (image, options) => Promise<CustomTileData>
+```
+
+Band-selectable variant. Honors `BandConfig.mode` (`'rgb'` vs `'single'`) and `colorRamp`.
+
+### `isRescaleActive(cfg)` / `createRescaledPipeline(geotiff, rescale)`
+
+```ts
+function isRescaleActive(cfg: RescaleConfig): boolean;
+
+function createRescaledPipeline(
+  geotiff: GeoTIFF,
+  rescale: RescaleConfig
+): { getTileData: Function; renderTile: Function };
+```
+
+GPU-side LinearRescale module appended to the library default uint pipeline. Only meaningful for uint COGs whose shape / colormap falls inside the library default — see `selectCogPipeline` for the dispatch rules.
+
+### `selectCogPipeline(geotiff, opts?)`
+
+```ts
+function selectCogPipeline(
+  geotiff: GeoTIFF,
+  opts?: SelectCogPipelineOptions
+): ResolvedCogPipeline
+```
+
+Single entry point that returns the correct `{ getTileData, renderTile }` pair for a given COG + config. Four outcomes in priority order:
+
+1. Custom configurable — band swap or color ramp change is active and the config is non-default.
+2. Custom non-uint — Int/Float COG without an explicit config yet.
+3. Library default + LinearRescale — uint COG with `rescale` active.
+4. Library default — everything else (empty object is returned; `COGLayer` uses its built-ins).
+
+### `normalizeCogGeotiff(geotiff)`
+
+```ts
+function normalizeCogGeotiff(geotiff: GeoTIFF): void
+```
+
+Apply in-place upstream workarounds:
+
+- Drop overview IFDs smaller than one tile (cause proj4 NaN on polar projections).
+- Clamp EPSG:4326 bbox to ±85.051129° latitude (Web Mercator safe range).
+
+Idempotent — safe to call repeatedly.
+
+### `createEpsgResolver()`
+
+```ts
+function createEpsgResolver(): (code: number) => Promise<ProjectionDefinition>
+```
+
+Async factory returning an `epsgResolver` compatible with `@developmentseed/deck.gl-geotiff`'s `COGLayer` prop of the same name. Looks up numeric EPSG codes in the bundled `@developmentseed/epsg` gzipped CSV and parses each WKT with `parseWkt()` from `@developmentseed/proj`. Throws a clear `Error` if the code is not present.
+
+Replaces runtime `epsg.io` fetches — first COG per session pulls the CSV (~200 KB gzipped) once.
+
+### Pixel / CRS helpers
+
+```ts
+function readPixelAtLngLat(
+  geotiff: GeoTIFF,
+  lng: number,
+  lat: number,
+  proj4Def: string | null,
+  pool: any,
+  signal?: AbortSignal
+): Promise<PixelValue | null>;
+
+function resolveProj4Def(
+  crs: number | unknown,
+  signal: AbortSignal
+): Promise<string | null>;
+```
+
+`readPixelAtLngLat` converts `(lng, lat)` into source CRS, then pixel coords, and reads every band. Returns `null` when outside the raster.
+
+`resolveProj4Def` returns a proj4-compatible WKT/ProjJSON string for a CRS identifier, or `null` for WGS84 / unknown.
+
+### MapLibre helpers
+
+```ts
+function getMaxTextureSize(map: maplibregl.Map): number;     // fallback 4096
+
+function fitCogBounds(map: maplibregl.Map, b: GeoBounds): void;
+
+function cleanupNativeBitmap(map: maplibregl.Map): void;    // idempotent
+
+function renderNonTiledBitmap(options: {
+  url: string;
+  map: maplibregl.Map;
+  signal?: AbortSignal;
+  geotiff?: GeoTIFF;
+}): Promise<CogInfo>;
+```
+
+`renderNonTiledBitmap` is the fallback path for non-tiled GeoTIFFs: opens the file, reads band 0, normalizes to grayscale RGBA (terrain ramp for single-band int/float), adds a MapLibre native `image` source. Throws when the raster exceeds 100 M pixels or projection bounds cannot be computed.
+
+## Usage outline
+
+```ts
+import {
+  defaultBandConfig,
+  selectCogPipeline,
+  normalizeCogGeotiff,
+  createEpsgResolver,
+  fitCogBounds,
+  clampBounds,
+} from '@walkthru-earth/objex-utils';
+
+normalizeCogGeotiff(geotiff);
+const config = defaultBandConfig(geotiff.bandCount, sampleFormat);
+const pipeline = selectCogPipeline(geotiff, { bandConfig: config });
+const resolver = createEpsgResolver();
+
+// feed into COGLayer({ ...pipeline, epsgResolver: resolver })
+
+fitCogBounds(map, clampBounds(bounds));
+```

package/docs/errors.md

@@ -0,0 +1,34 @@
+# Error handling
+
+One function. Normalizes thrown values into a displayable string, with special-casing for `AbortError`.
+
+Source: `src/lib/utils/error.ts`.
+
+## `handleLoadError(err)`
+
+```ts
+function handleLoadError(err: unknown): string | null
+```
+
+Extract a display message from an unknown caught value.
+
+| Input | Output | Meaning |
+|-------|--------|---------|
+| `DOMException` with `name === 'AbortError'` | `null` | Caller should silently return — the user cancelled. |
+| `Error` with `name === 'AbortError'` (fetch cancel) | `null` | Same as above. |
+| Other `Error` | `err.message` | Standard error. |
+| Everything else | `String(err)` | Best-effort coercion. |
+
+## Usage pattern
+
+```ts
+import { handleLoadError } from '@walkthru-earth/objex-utils';
+
+try {
+  data = await adapter.read(path, 0, undefined, signal);
+} catch (err) {
+  const msg = handleLoadError(err);
+  if (msg === null) return;    // aborted — do nothing
+  errorState = msg;
+}
+```

package/docs/file-sort.md

@@ -0,0 +1,67 @@
+# File sorting
+
+Sort a `FileEntry[]` with directories pinned on top. Pure — no mutation of the input.
+
+Source: `src/lib/utils/file-sort.ts`.
+
+## Types
+
+### `SortField`
+
+```ts
+type SortField = 'name' | 'size' | 'modified' | 'extension';
+```
+
+### `SortDirection`
+
+```ts
+type SortDirection = 'asc' | 'desc';
+```
+
+### `SortConfig`
+
+```ts
+interface SortConfig {
+  field: SortField;
+  direction: SortDirection;
+}
+```
+
+## Functions
+
+### `sortFileEntries(entries, config)`
+
+```ts
+function sortFileEntries(entries: FileEntry[], config: SortConfig): FileEntry[]
+```
+
+Return a **new** array sorted according to `config`.
+
+- Directories always sort before files regardless of direction.
+- `name` uses locale-aware comparison.
+- `size` and `modified` use numeric comparison.
+- `extension` falls back to `name` when equal.
+- Stable for equal keys.
+
+### `toggleSortField(current, field)`
+
+```ts
+function toggleSortField(current: SortConfig, field: SortField): SortConfig
+```
+
+UI helper for clickable column headers:
+
+- If `field === current.field`: flip direction.
+- Otherwise: switch to `{ field, direction: 'asc' }`.
+
+## Example
+
+```ts
+import { sortFileEntries, toggleSortField } from '@walkthru-earth/objex-utils';
+
+let config = { field: 'modified', direction: 'desc' };
+const sorted = sortFileEntries(entries, config);
+
+// onClick a column header:
+config = toggleSortField(config, 'size');
+```

package/docs/file-types.md

@@ -0,0 +1,141 @@
+# File-type registry
+
+Central mapping between file extensions and everything objex needs to render or query them — icon, color, viewer, MIME type, DuckDB read function.
+
+Source: `src/lib/file-icons/index.ts`.
+
+## Types
+
+### `FileCategory`
+
+```ts
+type FileCategory =
+  | 'data' | 'geo' | 'code' | 'document' | 'config'
+  | 'image' | 'video' | 'audio' | 'archive'
+  | 'database' | '3d' | 'other';
+```
+
+### `ViewerKind`
+
+```ts
+type ViewerKind =
+  | 'table' | 'image' | 'video' | 'audio' | 'markdown' | 'code'
+  | 'cog' | 'pmtiles' | 'flatgeobuf' | 'pdf' | '3d' | 'archive'
+  | 'database' | 'zarr' | 'copc' | 'notebook' | 'raw';
+```
+
+Use this to drive viewer routing in your own UI.
+
+### `DuckDbReadFn`
+
+```ts
+type DuckDbReadFn = 'read_parquet' | 'read_csv' | 'read_json' | 'ST_Read';
+```
+
+Which DuckDB table function should be used to ingest the file. `ST_Read` covers GDAL-backed vector formats (Shapefile, GeoPackage, FlatGeobuf, KML, GML).
+
+### `FileTypeInfo`
+
+```ts
+interface FileTypeInfo {
+  icon: string;            // Lucide icon name
+  color: string;           // Tailwind color classes (light + dark)
+  label: string;           // human-readable type label
+  category: FileCategory;
+  viewer: ViewerKind;
+  queryable: boolean;
+  duckdbReadFn: DuckDbReadFn | null;
+  mimeType: string;
+}
+```
+
+## Functions
+
+### `getFileTypeInfo(extension, isDir?)`
+
+```ts
+function getFileTypeInfo(extension: string, isDir?: boolean): FileTypeInfo
+```
+
+Return a fully populated `FileTypeInfo` for an extension.
+
+- `extension` is matched with or without a leading dot, case-insensitive.
+- `isDir === true` short-circuits to a directory entry (folder icon, `category: 'other'`, `viewer: 'raw'`).
+- Unknown extensions return the `raw` entry.
+
+### `getViewerKind(extension)`
+
+```ts
+function getViewerKind(extension: string): ViewerKind
+```
+
+Shorthand that returns `info.viewer` only. `extension` may be a full filename or an extension.
+
+### `getMimeType(extension)`
+
+```ts
+function getMimeType(extension: string): string
+```
+
+Return the MIME type (`application/octet-stream` fallback). Useful for `Content-Type` on uploads and `<a download>` links.
+
+### `isQueryable(extension)`
+
+```ts
+function isQueryable(extension: string): boolean
+```
+
+`true` when the format can be queried with DuckDB (Parquet, CSV, TSV, JSONL, NDJSON, Shapefile, GeoPackage, FlatGeobuf, KML, GML, etc.).
+
+### `getDuckDbReadFn(pathOrExt)`
+
+```ts
+function getDuckDbReadFn(pathOrExt: string): string
+```
+
+Return the DuckDB table-function name (`'read_parquet'`, `'read_csv'`, `'read_json'`, `'ST_Read'`). Falls back to `'read_parquet'` when unknown.
+
+Accepts either a file path or a bare extension.
+
+### `isCloudNativeFormat(pathOrExt)`
+
+```ts
+function isCloudNativeFormat(pathOrExt: string): boolean
+```
+
+`true` for formats DuckDB can query directly over HTTP range requests without buffering the whole file (`.parquet`, `.geoparquet`, `.gpq`, `.gparquet`, `.ducklake`).
+
+### `buildDuckDbSource(pathOrExt, url)`
+
+```ts
+function buildDuckDbSource(pathOrExt: string, url: string): string
+```
+
+Return a ready-to-paste FROM-clause expression. Key behavior:
+
+| Ext | Output |
+|-----|--------|
+| `.parquet` / `.geoparquet` | `read_parquet('<url>')` |
+| `.csv` / `.tsv` | `read_csv('<url>')` |
+| `.jsonl` / `.ndjson` | `read_json('<url>')` |
+| `.json` | Unnested expression that flattens GeoJSON `features[*]` into rows with property columns + a `geometry` column |
+| `.shp`, `.gpkg`, `.fgb`, `.kml`, `.gml` | `ST_Read('<url>')` |
+| Unknown | `read_parquet('<url>')` fallback |
+
+## Example
+
+```ts
+import {
+  getFileTypeInfo,
+  isQueryable,
+  buildDuckDbSource,
+} from '@walkthru-earth/objex-utils';
+
+const info = getFileTypeInfo('streets.fgb');
+// { viewer: 'flatgeobuf', category: 'geo', queryable: true, duckdbReadFn: 'ST_Read', ... }
+
+if (isQueryable('streets.fgb')) {
+  const sql = `SELECT COUNT(*) FROM ${buildDuckDbSource('streets.fgb', url)}`;
+  // SELECT COUNT(*) FROM ST_Read('https://.../streets.fgb')
+}
+```