@walkthru-earth/objex-utils 1.1.0 → 1.2.1

package/docs/README.md

@@ -0,0 +1,102 @@
+# `@walkthru-earth/objex-utils` Developer Docs
+
+Professional reference documentation for every public export of the package.
+
+Each page lists the exact TypeScript signature, parameter semantics, return shape, peer-dependency requirements, and non-obvious behavior. Intended as the single source of truth for integrators.
+
+## Install
+
+```bash
+pnpm add @walkthru-earth/objex-utils
+# or
+npm install @walkthru-earth/objex-utils
+```
+
+Minimum: ES2022, modern browser or Node 18+. Ships both ESM (`dist/index.js`) and CJS (`dist/index.cjs`) with full `.d.ts`.
+
+## Optional peer dependencies
+
+All heavy dependencies are **optional** peer dependencies. Install only the ones you touch.
+
+| Peer | Version | Required by |
+|------|---------|-------------|
+| `apache-arrow` | `>=14` | `buildGeoArrowTables`, `normalizeGeomType` (indirectly) |
+| `hyparquet` | `>=1.25` | `readParquetMetadata` and its extractors |
+| `hyparquet-compressors` | `>=1.1` | `readParquetMetadata` (for SNAPPY/ZSTD/GZIP/etc.) |
+| `yaml` | `>=2` | `parseMarkdownDocument` (lazy-loaded, only when YAML frontmatter is present) |
+
+As of v1.2, `yaml` is loaded via dynamic `import()` inside `parseMarkdownDocument`. Consumers who do not call that function never touch `yaml` at load time.
+
+## Reference pages
+
+| Page | Covers |
+|------|--------|
+| [`geometry.md`](./geometry.md) | WKB parser, GeoArrow table builder, geometry-column detection |
+| [`cog.md`](./cog.md) | Cloud-Optimized GeoTIFF helpers (pipeline selection, band configs, color ramps, bounds clamping) |
+| [`parquet-metadata.md`](./parquet-metadata.md) | `readParquetMetadata` + CRS / bounds / geometry-types extractors |
+| [`storage.md`](./storage.md) | URL parsing (`parseStorageUrl`, `resolveCloudUrl`), provider registry, `StorageAdapter` interface, `UrlAdapter` |
+| [`query-engine.md`](./query-engine.md) | `QueryEngine` interface and associated result/handle types |
+| [`file-types.md`](./file-types.md) | File-type registry (`getFileTypeInfo`, `getViewerKind`, `getDuckDbReadFn`, ...) |
+| [`formatting.md`](./formatting.md) | Display formatters, column-type classification, hex dump, CSV/JSON export |
+| [`file-sort.md`](./file-sort.md) | `sortFileEntries`, `toggleSortField` |
+| [`markdown-sql.md`](./markdown-sql.md) | Markdown + SQL block parsing (Evidence-compatible syntax) |
+| [`local-storage.md`](./local-storage.md) | SSR-safe `loadFromStorage` / `persistToStorage` |
+| [`errors.md`](./errors.md) | `handleLoadError` |
+| [`types-constants.md`](./types-constants.md) | `Connection`, `Tab`, `FileEntry`, `WriteResult`, `Theme`, shared constants |
+
+## Quick recipes
+
+### Render a GeoParquet file on deck.gl
+
+```ts
+import {
+  readParquetMetadata,
+  extractEpsgFromGeoMeta,
+  extractGeometryTypes,
+  buildGeoArrowTables
+} from '@walkthru-earth/objex-utils';
+
+// 1. Read footer metadata (~512KB range requests, no DuckDB)
+const meta = await readParquetMetadata(url);
+const sourceCrs = extractEpsgFromGeoMeta(meta.geo!);   // e.g. 'EPSG:27700' or null
+
+// 2. Get WKB buffers + attributes from your SQL engine (DuckDB-WASM, etc.)
+const { wkbArrays, attributes } = await myEngine.queryForMap(...);
+
+// 3. Build GeoArrow tables (one per geometry type)
+const tables = buildGeoArrowTables(wkbArrays, attributes);
+
+// 4. Feed into deck.gl GeoArrowLayer(s)
+```
+
+### Parse a cloud storage URL a user pasted
+
+```ts
+import { parseStorageUrl, looksLikeUrl, describeParseResult } from '@walkthru-earth/objex-utils';
+
+const parsed = parseStorageUrl(input);
+// { provider: 's3', bucket: 'my-bucket', region: 'us-east-1', endpoint: '...', prefix: '...' }
+
+console.log(describeParseResult(parsed));
+```
+
+### Read a binary blob as a hex dump
+
+```ts
+import { generateHexDump } from '@walkthru-earth/objex-utils';
+
+const rows = generateHexDump(bytes);
+// rows: [{ offset: '00000000', hex: ['48','65','6c',...], ascii: 'Hello...' }, ...]
+```
+
+## Upstream source
+
+All modules re-export from `src/lib/` in the [objex](https://github.com/walkthru-earth/objex) monorepo. The re-export list lives at [`packages/objex-utils/src/index.ts`](../src/index.ts).
+
+## Versioning & releases
+
+Both `@walkthru-earth/objex` and `@walkthru-earth/objex-utils` ship together via Changesets with a `fixed` config, so their versions stay in lockstep. See [`RELEASE.md`](../../../RELEASE.md) in the monorepo root.
+
+## License
+
+[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

package/docs/cog.md

@@ -0,0 +1,92 @@
+# Cloud-Optimized GeoTIFF (COG) helpers
+
+Pure, framework-agnostic helpers for working with Cloud-Optimized GeoTIFF metadata and bounds. No Svelte, MapLibre, deck.gl, or GeoTIFF library dependency.
+
+Source: `src/lib/utils/cog.ts`.
+
+> The render-pipeline helpers (`selectCogPipeline`, `createConfigurableGetTileData`, `normalizeCogGeotiff`, `createEpsgResolver`, `fitCogBounds`, `renderNonTiledBitmap`, etc.) live in the same source file but are **not** re-exported from `@walkthru-earth/objex-utils` because they pull in `@developmentseed/deck.gl-geotiff`, `@developmentseed/geotiff`, `@developmentseed/proj`, `proj4`, and `maplibre-gl`. If you need them, depend on the full Svelte package [`@walkthru-earth/objex`](https://www.npmjs.com/package/@walkthru-earth/objex) (they are re-exported from `src/lib/index.ts`) and install those optional peers yourself.
+
+## Types
+
+### `GeoBounds`
+
+```ts
+interface GeoBounds {
+  west: number;
+  south: number;
+  east: number;
+  north: number;
+}
+```
+
+### `CogInfo`
+
+```ts
+interface CogInfo {
+  width: number;
+  height: number;
+  bandCount: number;
+  dataType: string;      // e.g. 'uint8', 'float32', 'int16'
+  bounds: GeoBounds;
+  downsampled?: boolean; // true when renderNonTiledBitmap decimated the array
+}
+```
+
+## Constants
+
+### `SF_LABELS`
+
+```ts
+const SF_LABELS: Record<number, string> = {
+  1: 'uint', 2: 'int', 3: 'float', 4: 'void',
+  5: 'complex int', 6: 'complex float',
+};
+```
+
+Human labels for the GeoTIFF SampleFormat tag.
+
+## Functions
+
+### `safeClamp(v, lo, hi, fallback)`
+
+```ts
+function safeClamp(v: number, lo: number, hi: number, fallback: number): number
+```
+
+Clamp `v` into `[lo, hi]`. If `v` is `NaN` or `±Infinity`, return `fallback`. Use instead of `Math.min(hi, Math.max(lo, v))` — NaN would otherwise propagate silently.
+
+### `clampBounds(b)`
+
+```ts
+function clampBounds(b: GeoBounds): GeoBounds
+```
+
+Clamp geographic bounds to valid MapLibre Web Mercator range: ±180° longitude, ±85.051129° latitude.
+
+### `buildDataTypeLabel(sampleFormat, bitsPerSample)`
+
+```ts
+function buildDataTypeLabel(
+  sampleFormat: number,
+  bitsPerSample: number
+): string
+```
+
+Combine the GeoTIFF `SampleFormat` tag with `BitsPerSample` into `'uint8'`, `'int16'`, `'float32'`, etc.
+
+## Usage outline
+
+```ts
+import {
+  buildDataTypeLabel,
+  clampBounds,
+  safeClamp,
+  SF_LABELS,
+  type CogInfo,
+  type GeoBounds,
+} from '@walkthru-earth/objex-utils';
+
+const label = buildDataTypeLabel(sampleFormat, bitsPerSample); // 'float32'
+const safeBounds = clampBounds(bounds);
+const nice = safeClamp(value, 0, 1, 0);
+```

package/docs/errors.md

@@ -0,0 +1,34 @@
+# Error handling
+
+One function. Normalizes thrown values into a displayable string, with special-casing for `AbortError`.
+
+Source: `src/lib/utils/error.ts`.
+
+## `handleLoadError(err)`
+
+```ts
+function handleLoadError(err: unknown): string | null
+```
+
+Extract a display message from an unknown caught value.
+
+| Input | Output | Meaning |
+|-------|--------|---------|
+| `DOMException` with `name === 'AbortError'` | `null` | Caller should silently return — the user cancelled. |
+| `Error` with `name === 'AbortError'` (fetch cancel) | `null` | Same as above. |
+| Other `Error` | `err.message` | Standard error. |
+| Everything else | `String(err)` | Best-effort coercion. |
+
+## Usage pattern
+
+```ts
+import { handleLoadError } from '@walkthru-earth/objex-utils';
+
+try {
+  data = await adapter.read(path, 0, undefined, signal);
+} catch (err) {
+  const msg = handleLoadError(err);
+  if (msg === null) return;    // aborted — do nothing
+  errorState = msg;
+}
+```

package/docs/file-sort.md

@@ -0,0 +1,67 @@
+# File sorting
+
+Sort a `FileEntry[]` with directories pinned on top. Pure — no mutation of the input.
+
+Source: `src/lib/utils/file-sort.ts`.
+
+## Types
+
+### `SortField`
+
+```ts
+type SortField = 'name' | 'size' | 'modified' | 'extension';
+```
+
+### `SortDirection`
+
+```ts
+type SortDirection = 'asc' | 'desc';
+```
+
+### `SortConfig`
+
+```ts
+interface SortConfig {
+  field: SortField;
+  direction: SortDirection;
+}
+```
+
+## Functions
+
+### `sortFileEntries(entries, config)`
+
+```ts
+function sortFileEntries(entries: FileEntry[], config: SortConfig): FileEntry[]
+```
+
+Return a **new** array sorted according to `config`.
+
+- Directories always sort before files regardless of direction.
+- `name` uses locale-aware comparison.
+- `size` and `modified` use numeric comparison.
+- `extension` falls back to `name` when equal.
+- Stable for equal keys.
+
+### `toggleSortField(current, field)`
+
+```ts
+function toggleSortField(current: SortConfig, field: SortField): SortConfig
+```
+
+UI helper for clickable column headers:
+
+- If `field === current.field`: flip direction.
+- Otherwise: switch to `{ field, direction: 'asc' }`.
+
+## Example
+
+```ts
+import { sortFileEntries, toggleSortField } from '@walkthru-earth/objex-utils';
+
+let config = { field: 'modified', direction: 'desc' };
+const sorted = sortFileEntries(entries, config);
+
+// onClick a column header:
+config = toggleSortField(config, 'size');
+```

package/docs/file-types.md

@@ -0,0 +1,141 @@
+# File-type registry
+
+Central mapping between file extensions and everything objex needs to render or query them — icon, color, viewer, MIME type, DuckDB read function.
+
+Source: `src/lib/file-icons/index.ts`.
+
+## Types
+
+### `FileCategory`
+
+```ts
+type FileCategory =
+  | 'data' | 'geo' | 'code' | 'document' | 'config'
+  | 'image' | 'video' | 'audio' | 'archive'
+  | 'database' | '3d' | 'other';
+```
+
+### `ViewerKind`
+
+```ts
+type ViewerKind =
+  | 'table' | 'image' | 'video' | 'audio' | 'markdown' | 'code'
+  | 'cog' | 'pmtiles' | 'flatgeobuf' | 'pdf' | '3d' | 'archive'
+  | 'database' | 'zarr' | 'copc' | 'notebook' | 'raw';
+```
+
+Use this to drive viewer routing in your own UI.
+
+### `DuckDbReadFn`
+
+```ts
+type DuckDbReadFn = 'read_parquet' | 'read_csv' | 'read_json' | 'ST_Read';
+```
+
+Which DuckDB table function should be used to ingest the file. `ST_Read` covers GDAL-backed vector formats (Shapefile, GeoPackage, FlatGeobuf, KML, GML).
+
+### `FileTypeInfo`
+
+```ts
+interface FileTypeInfo {
+  icon: string;            // Lucide icon name
+  color: string;           // Tailwind color classes (light + dark)
+  label: string;           // human-readable type label
+  category: FileCategory;
+  viewer: ViewerKind;
+  queryable: boolean;
+  duckdbReadFn: DuckDbReadFn | null;
+  mimeType: string;
+}
+```
+
+## Functions
+
+### `getFileTypeInfo(extension, isDir?)`
+
+```ts
+function getFileTypeInfo(extension: string, isDir?: boolean): FileTypeInfo
+```
+
+Return a fully populated `FileTypeInfo` for an extension.
+
+- `extension` is matched with or without a leading dot, case-insensitive.
+- `isDir === true` short-circuits to a directory entry (folder icon, `category: 'other'`, `viewer: 'raw'`).
+- Unknown extensions return the `raw` entry.
+
+### `getViewerKind(extension)`
+
+```ts
+function getViewerKind(extension: string): ViewerKind
+```
+
+Shorthand that returns `info.viewer` only. `extension` may be a full filename or an extension.
+
+### `getMimeType(extension)`
+
+```ts
+function getMimeType(extension: string): string
+```
+
+Return the MIME type (`application/octet-stream` fallback). Useful for `Content-Type` on uploads and `<a download>` links.
+
+### `isQueryable(extension)`
+
+```ts
+function isQueryable(extension: string): boolean
+```
+
+`true` when the format can be queried with DuckDB (Parquet, CSV, TSV, JSONL, NDJSON, Shapefile, GeoPackage, FlatGeobuf, KML, GML, etc.).
+
+### `getDuckDbReadFn(pathOrExt)`
+
+```ts
+function getDuckDbReadFn(pathOrExt: string): string
+```
+
+Return the DuckDB table-function name (`'read_parquet'`, `'read_csv'`, `'read_json'`, `'ST_Read'`). Falls back to `'read_parquet'` when unknown.
+
+Accepts either a file path or a bare extension.
+
+### `isCloudNativeFormat(pathOrExt)`
+
+```ts
+function isCloudNativeFormat(pathOrExt: string): boolean
+```
+
+`true` for formats DuckDB can query directly over HTTP range requests without buffering the whole file (`.parquet`, `.geoparquet`, `.gpq`, `.gparquet`, `.ducklake`).
+
+### `buildDuckDbSource(pathOrExt, url)`
+
+```ts
+function buildDuckDbSource(pathOrExt: string, url: string): string
+```
+
+Return a ready-to-paste FROM-clause expression. Key behavior:
+
+| Ext | Output |
+|-----|--------|
+| `.parquet` / `.geoparquet` | `read_parquet('<url>')` |
+| `.csv` / `.tsv` | `read_csv('<url>')` |
+| `.jsonl` / `.ndjson` | `read_json('<url>')` |
+| `.json` | Unnested expression that flattens GeoJSON `features[*]` into rows with property columns + a `geometry` column |
+| `.shp`, `.gpkg`, `.fgb`, `.kml`, `.gml` | `ST_Read('<url>')` |
+| Unknown | `read_parquet('<url>')` fallback |
+
+## Example
+
+```ts
+import {
+  getFileTypeInfo,
+  isQueryable,
+  buildDuckDbSource,
+} from '@walkthru-earth/objex-utils';
+
+const info = getFileTypeInfo('streets.fgb');
+// { viewer: 'flatgeobuf', category: 'geo', queryable: true, duckdbReadFn: 'ST_Read', ... }
+
+if (isQueryable('streets.fgb')) {
+  const sql = `SELECT COUNT(*) FROM ${buildDuckDbSource('streets.fgb', url)}`;
+  // SELECT COUNT(*) FROM ST_Read('https://.../streets.fgb')
+}
+```

package/docs/formatting.md

@@ -0,0 +1,192 @@
+# Formatting, column types, hex, export
+
+Display formatters, column-type classification, hex dump, and data serialization. All sync, pure, no browser APIs required unless noted.
+
+Sources:
+
+- `src/lib/utils/format.ts`
+- `src/lib/utils/column-types.ts`
+- `src/lib/utils/hex.ts`
+- `src/lib/utils/export.ts`
+
+## Formatters
+
+### `formatFileSize(bytes)`
+
+```ts
+function formatFileSize(bytes: number): string
+```
+
+1024-based human-readable byte count. Integer for raw bytes (`'512 B'`), one decimal for everything else (`'1.5 MB'`). Returns `'0 B'` for `0`, `'-'` for negative.
+
+### `formatDate(timestamp)`
+
+```ts
+function formatDate(timestamp: number): string
+```
+
+Format a unix timestamp **in milliseconds** as a human-readable date string.
+
+- Recent timestamps render relative: `'just now'`, `'5m ago'`, `'2h ago'`, `'3d ago'`.
+- Older render as `'YYYY-MM-DD'`.
+- Missing / invalid → `'--'`.
+
+### `formatValue(value)`
+
+```ts
+function formatValue(value: unknown): string
+```
+
+Format any value for display in tables, panels, or exports.
+
+| Input | Output |
+|-------|--------|
+| `null` / `undefined` | `''` |
+| `bigint` | `value.toString()` |
+| `Date` | `value.toISOString()` |
+| Object (incl. arrays) | `JSON.stringify(value, jsonReplacerBigInt)` |
+| Everything else | `String(value)` |
+
+### `getFileExtension(filename)`
+
+```ts
+function getFileExtension(filename: string): string
+```
+
+Return the extension **including** the leading dot (`'data.parquet' → '.parquet'`). Empty string if no extension. Case-preserving.
+
+### `jsonReplacerBigInt(_key, value)`
+
+```ts
+function jsonReplacerBigInt(_key: string, value: unknown): unknown
+```
+
+`JSON.stringify` replacer that coerces `BigInt` to decimal strings so DuckDB `BIGINT`s don't explode serialization.
+
+```ts
+JSON.stringify(row, jsonReplacerBigInt);
+```
+
+## Column-type classification
+
+### `TypeCategory`
+
+```ts
+type TypeCategory =
+  | 'number' | 'string' | 'date' | 'boolean'
+  | 'geo' | 'binary' | 'json' | 'other';
+```
+
+### `classifyType(duckdbType)`
+
+```ts
+function classifyType(duckdbType: string): TypeCategory
+```
+
+Classify a DuckDB/Arrow type string. Handles parameterized types (`DECIMAL(18,3)`, `VARCHAR(100)`), compound types (`STRUCT`, `MAP`, `LIST`), and fuzzy keyword matching as a fallback.
+
+| Example inputs | Result |
+|----------------|--------|
+| `BIGINT`, `DOUBLE`, `DECIMAL(18,3)` | `'number'` |
+| `VARCHAR`, `STRING`, `UTF8` | `'string'` |
+| `TIMESTAMP`, `DATE`, `TIME` | `'date'` |
+| `BOOLEAN` | `'boolean'` |
+| `GEOMETRY`, `GEOMETRY('EPSG:27700')`, `POINT`, `WKB_BLOB` | `'geo'` |
+| `BLOB`, `BINARY`, `VARBINARY` | `'binary'` |
+| `STRUCT<...>`, `MAP<...>`, `LIST<...>`, `JSON` | `'json'` |
+| Otherwise | `'other'` |
+
+### `typeColor(category)`
+
+```ts
+function typeColor(category: TypeCategory): string
+```
+
+Tailwind text-color class (e.g. `'text-blue-500'`) for dots / badges.
+
+### `typeBadgeClass(category)`
+
+```ts
+function typeBadgeClass(category: TypeCategory): string
+```
+
+Tailwind classes for a pill-style badge (background + text + border).
+
+### `typeLabel(category)`
+
+```ts
+function typeLabel(category: TypeCategory): string
+```
+
+Short symbol (`'#'` for number, `'{}'` for json, `'geo'` for geo, etc.) for compact badges.
+
+## Hex dump
+
+### `HexRow`
+
+```ts
+interface HexRow {
+  offset: string;     // hex offset, zero-padded (e.g. '00000000')
+  hex: string[];      // per-byte hex (e.g. '48', '65')
+  ascii: string;      // printable ASCII, '.' for non-printable
+}
+```
+
+### `generateHexDump(data, bytesPerRow?)`
+
+```ts
+function generateHexDump(data: Uint8Array, bytesPerRow?: number): HexRow[]
+```
+
+Build a structured hex dump. `bytesPerRow` defaults to `16`. Pure — no DOM.
+
+Rendering is left to the caller; `HexRow` maps directly to `<tr>` / CSV / JSON.
+
+## Data export
+
+### `escapeCsvField(value)`
+
+```ts
+function escapeCsvField(value: string): string
+```
+
+Escape a single CSV field per RFC 4180: wraps in double quotes and doubles any embedded quote **only when** the value contains `,`, `"`, `\n`, or `\r`. Otherwise returns the value unchanged.
+
+### `serializeToCsv(columns, rows)`
+
+```ts
+function serializeToCsv(
+  columns: string[],
+  rows: Record<string, unknown>[]
+): string
+```
+
+Pure serializer. Uses `formatValue` internally, so `bigint` / `Date` / objects round-trip correctly. Row terminator is `'\n'`.
+
+### `serializeToJson(columns, rows)`
+
+```ts
+function serializeToJson(
+  columns: string[],
+  rows: Record<string, unknown>[]
+): string
+```
+
+Pure. Returns pretty-printed JSON (`2`-space indent). `Date` → ISO, `BigInt` → decimal string, `null` preserved.
+
+### `exportToCsv(columns, rows, filename)` / `exportToJson(columns, rows, filename)`
+
+```ts
+function exportToCsv(
+  columns: string[],
+  rows: Record<string, unknown>[],
+  filename: string
+): void;
+function exportToJson(
+  columns: string[],
+  rows: Record<string, unknown>[],
+  filename: string
+): void;
+```
+
+Browser-only wrappers that build the blob and trigger a download via a hidden `<a>`. Throw `ReferenceError` in non-DOM environments — use the `serializeToCsv` / `serializeToJson` variants server-side.