@walkthru-earth/objex-utils 1.1.0 → 1.2.0

package/docs/formatting.md

@@ -0,0 +1,192 @@
+# Formatting, column types, hex, export
+
+Display formatters, column-type classification, hex dump, and data serialization. All sync, pure, no browser APIs required unless noted.
+
+Sources:
+
+- `src/lib/utils/format.ts`
+- `src/lib/utils/column-types.ts`
+- `src/lib/utils/hex.ts`
+- `src/lib/utils/export.ts`
+
+## Formatters
+
+### `formatFileSize(bytes)`
+
+```ts
+function formatFileSize(bytes: number): string
+```
+
+1024-based human-readable byte count. Integer for raw bytes (`'512 B'`), one decimal for everything else (`'1.5 MB'`). Returns `'0 B'` for `0`, `'-'` for negative.
+
+### `formatDate(timestamp)`
+
+```ts
+function formatDate(timestamp: number): string
+```
+
+Format a unix timestamp **in milliseconds** as a human-readable date string.
+
+- Recent timestamps render relative: `'just now'`, `'5m ago'`, `'2h ago'`, `'3d ago'`.
+- Older render as `'YYYY-MM-DD'`.
+- Missing / invalid → `'--'`.
+
+### `formatValue(value)`
+
+```ts
+function formatValue(value: unknown): string
+```
+
+Format any value for display in tables, panels, or exports.
+
+| Input | Output |
+|-------|--------|
+| `null` / `undefined` | `''` |
+| `bigint` | `value.toString()` |
+| `Date` | `value.toISOString()` |
+| Object (incl. arrays) | `JSON.stringify(value, jsonReplacerBigInt)` |
+| Everything else | `String(value)` |
+
+### `getFileExtension(filename)`
+
+```ts
+function getFileExtension(filename: string): string
+```
+
+Return the extension **including** the leading dot (`'data.parquet' → '.parquet'`). Empty string if no extension. Case-preserving.
+
+### `jsonReplacerBigInt(_key, value)`
+
+```ts
+function jsonReplacerBigInt(_key: string, value: unknown): unknown
+```
+
+`JSON.stringify` replacer that coerces `BigInt` to decimal strings so DuckDB `BIGINT`s don't explode serialization.
+
+```ts
+JSON.stringify(row, jsonReplacerBigInt);
+```
+
+## Column-type classification
+
+### `TypeCategory`
+
+```ts
+type TypeCategory =
+  | 'number' | 'string' | 'date' | 'boolean'
+  | 'geo' | 'binary' | 'json' | 'other';
+```
+
+### `classifyType(duckdbType)`
+
+```ts
+function classifyType(duckdbType: string): TypeCategory
+```
+
+Classify a DuckDB/Arrow type string. Handles parameterized types (`DECIMAL(18,3)`, `VARCHAR(100)`), compound types (`STRUCT`, `MAP`, `LIST`), and fuzzy keyword matching as a fallback.
+
+| Example inputs | Result |
+|----------------|--------|
+| `BIGINT`, `DOUBLE`, `DECIMAL(18,3)` | `'number'` |
+| `VARCHAR`, `STRING`, `UTF8` | `'string'` |
+| `TIMESTAMP`, `DATE`, `TIME` | `'date'` |
+| `BOOLEAN` | `'boolean'` |
+| `GEOMETRY`, `GEOMETRY('EPSG:27700')`, `POINT`, `WKB_BLOB` | `'geo'` |
+| `BLOB`, `BINARY`, `VARBINARY` | `'binary'` |
+| `STRUCT<...>`, `MAP<...>`, `LIST<...>`, `JSON` | `'json'` |
+| Otherwise | `'other'` |
+
+### `typeColor(category)`
+
+```ts
+function typeColor(category: TypeCategory): string
+```
+
+Tailwind text-color class (e.g. `'text-blue-500'`) for dots / badges.
+
+### `typeBadgeClass(category)`
+
+```ts
+function typeBadgeClass(category: TypeCategory): string
+```
+
+Tailwind classes for a pill-style badge (background + text + border).
+
+### `typeLabel(category)`
+
+```ts
+function typeLabel(category: TypeCategory): string
+```
+
+Short symbol (`'#'` for number, `'{}'` for json, `'geo'` for geo, etc.) for compact badges.
+
+## Hex dump
+
+### `HexRow`
+
+```ts
+interface HexRow {
+  offset: string;     // hex offset, zero-padded (e.g. '00000000')
+  hex: string[];      // per-byte hex (e.g. '48', '65')
+  ascii: string;      // printable ASCII, '.' for non-printable
+}
+```
+
+### `generateHexDump(data, bytesPerRow?)`
+
+```ts
+function generateHexDump(data: Uint8Array, bytesPerRow?: number): HexRow[]
+```
+
+Build a structured hex dump. `bytesPerRow` defaults to `16`. Pure — no DOM.
+
+Rendering is left to the caller; `HexRow` maps directly to `<tr>` / CSV / JSON.
+
+## Data export
+
+### `escapeCsvField(value)`
+
+```ts
+function escapeCsvField(value: string): string
+```
+
+Escape a single CSV field per RFC 4180: wraps in double quotes and doubles any embedded quote **only when** the value contains `,`, `"`, `\n`, or `\r`. Otherwise returns the value unchanged.
+
+### `serializeToCsv(columns, rows)`
+
+```ts
+function serializeToCsv(
+  columns: string[],
+  rows: Record<string, unknown>[]
+): string
+```
+
+Pure serializer. Uses `formatValue` internally, so `bigint` / `Date` / objects round-trip correctly. Row terminator is `'\n'`.
+
+### `serializeToJson(columns, rows)`
+
+```ts
+function serializeToJson(
+  columns: string[],
+  rows: Record<string, unknown>[]
+): string
+```
+
+Pure. Returns pretty-printed JSON (`2`-space indent). `Date` → ISO, `BigInt` → decimal string, `null` preserved.
+
+### `exportToCsv(columns, rows, filename)` / `exportToJson(columns, rows, filename)`
+
+```ts
+function exportToCsv(
+  columns: string[],
+  rows: Record<string, unknown>[],
+  filename: string
+): void;
+function exportToJson(
+  columns: string[],
+  rows: Record<string, unknown>[],
+  filename: string
+): void;
+```
+
+Browser-only wrappers that build the blob and trigger a download via a hidden `<a>`. Throw `ReferenceError` in non-DOM environments — use the `serializeToCsv` / `serializeToJson` variants server-side.

package/docs/geometry.md

@@ -0,0 +1,198 @@
+# Geometry
+
+WKB parsing, geometry-column detection, and GeoArrow table construction. Zero-copy where possible.
+
+Source: `src/lib/utils/wkb.ts`, `src/lib/utils/geoarrow.ts`.
+
+## Types
+
+### `GeoType`
+
+```ts
+type GeoType =
+  | 'Point'
+  | 'LineString'
+  | 'Polygon'
+  | 'MultiPoint'
+  | 'MultiLineString'
+  | 'MultiPolygon'
+  | 'Unknown';
+```
+
+`'Unknown'` is returned for unsupported WKB types (GeometryCollections, TINs, triangles, etc.).
+
+### `ParsedGeometry`
+
+```ts
+interface ParsedGeometry {
+  type: GeoType;
+  coordinates: number[] | number[][] | number[][][] | number[][][][];
+}
+```
+
+Coordinate nesting follows GeoJSON conventions.
+
+| Type | Shape |
+|------|-------|
+| `Point` | `[x, y]` |
+| `MultiPoint` / `LineString` | `[[x, y], ...]` |
+| `MultiLineString` / `Polygon` | `[[[x, y], ...], ...]` |
+| `MultiPolygon` | `[[[[x, y], ...], ...], ...]` |
+
+### `GeoArrowGeomType`
+
+```ts
+type GeoArrowGeomType =
+  | 'point'
+  | 'linestring'
+  | 'polygon'
+  | 'multipoint'
+  | 'multilinestring'
+  | 'multipolygon';
+```
+
+Lowercase normalized form used by the GeoArrow builder and `@geoarrow/deck.gl-layers`.
+
+### `GeoArrowResult`
+
+```ts
+interface GeoArrowResult {
+  table: Table;                                      // apache-arrow Table
+  geometryType: GeoArrowGeomType;
+  bounds: [number, number, number, number];          // [minX, minY, maxX, maxY]
+  sourceIndices: number[];                           // table row i → original wkbArrays[sourceIndices[i]]
+}
+```
+
+`sourceIndices` lets callers map picked rows back to the original row order when mixed geometry types force a split.
+
+## Functions
+
+### `toBinary(value)`
+
+```ts
+function toBinary(value: unknown): Uint8Array | null
+```
+
+Normalize an arbitrary "possibly-binary" value to a `Uint8Array`.
+
+| Input | Handling |
+|-------|----------|
+| `Uint8Array` | Returned as-is |
+| `ArrayBuffer` | Wrapped in `new Uint8Array(buf)` |
+| `number[]` | `new Uint8Array(arr)` |
+| Hex string (even length, `[0-9a-fA-F]`) | Decoded to bytes |
+| DuckDB `toJSON()` blob object `{0: b0, 1: b1, ...}` | Reassembled into bytes |
+| Anything else | `null` |
+
+Returns **`null`** rather than throwing on unrecognized input, so callers can fall through.
+
+### `parseWKB(data)`
+
+```ts
+function parseWKB(data: Uint8Array): ParsedGeometry | null
+```
+
+Parse a WKB byte blob.
+
+- Supports standard WKB, ISO WKB with Z/M flags, and EWKB with SRID prefix (PostGIS). Z/M ordinates are **dropped** — only X/Y is returned.
+- Supports Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon.
+- Returns **`null`** for truncated buffers, invalid byte-order flags, or malformed geometry.
+- GeometryCollections (WKB type 7) return `{ type: 'Unknown', coordinates: [] }`.
+
+### `findGeoColumn(schema)`
+
+```ts
+function findGeoColumn(
+  schema: { name: string; type: string }[]
+): string | null
+```
+
+Schema-only heuristic for locating the geometry column. Checks in priority order:
+
+1. **Type** contains a geometry keyword (`GEOMETRY`, `POINT`, `WKB`, ...) — match wins immediately.
+2. Exact well-known **name** (`geometry`, `geom`, `the_geom`, ...) **with** a binary-ish type (`BLOB`, `BINARY`, `VARBINARY`, `BYTES`).
+3. Exact well-known **name**, any type.
+4. Name contains a geo hint (`geom`, `geo_`, `wkb`, `wkt`, `shape`, `spatial`) with binary-ish type.
+5. Name contains geo hint, any type.
+
+Returns the first matching `name`, or `null` if no heuristic hits. Use `findGeoColumnFromRows` as a fallback when the schema is not informative.
+
+### `findGeoColumnFromRows(rows, schema)`
+
+```ts
+function findGeoColumnFromRows(
+  rows: Record<string, unknown>[],
+  schema: { name: string; type: string }[]
+): string | null
+```
+
+Row-based probe. Samples the first row and classifies values:
+
+- Binary-typed columns: tests for WKB magic bytes (endian flag 0x00/0x01 + valid geometry type).
+- Other columns: probes hex-encoded WKB, WKT strings starting with `POINT(`, `LINESTRING(`, etc., and GeoJSON geometry objects (`{type: 'Point', coordinates: [...]}`).
+
+Returns the first column whose value looks geometry-shaped, or `null`.
+
+### `normalizeGeomType(raw)`
+
+```ts
+function normalizeGeomType(raw: string): GeoArrowGeomType
+```
+
+Map a DuckDB `ST_GeometryType()` result (`'POINT'`, `'ST_Polygon'`, etc., case-insensitive, optional `ST_` prefix) to a `GeoArrowGeomType`. Unknown input falls back to `'polygon'`.
+
+### `buildGeoArrowTables(wkbArrays, attributes, knownGeomType?)`
+
+```ts
+function buildGeoArrowTables(
+  wkbArrays: Uint8Array[],
+  attributes: Map<string, { values: any[]; type: string }>,
+  knownGeomType?: GeoArrowGeomType
+): GeoArrowResult[]
+```
+
+Build one or more Arrow `Table` objects keyed by geometry type, ready for `@geoarrow/deck.gl-layers`.
+
+**Parameters**
+
+| Name | Type | Meaning |
+|------|------|---------|
+| `wkbArrays` | `Uint8Array[]` | Per-row WKB binary. Entries may be empty / invalid — they are skipped. |
+| `attributes` | `Map<name, { values, type }>` | Non-geometry columns. `values.length` must equal `wkbArrays.length`. `type` is a DuckDB/Arrow type string used for Arrow dtype inference (numeric → Float64, bool → Bool, everything else → Utf8). |
+| `knownGeomType` | optional | If provided (e.g. from GeoParquet metadata), classification is skipped — all WKBs are assumed to share this type, and one `GeoArrowResult` is returned. |
+
+**Behavior**
+
+- **Zero-copy geometry ingest.** A 5-byte peek classifies each WKB; coordinates are read directly into pre-allocated `Float64Array` backings with no intermediate JS objects.
+- **Mixed-type splitting.** If `knownGeomType` is not provided and the rows span multiple types (e.g. `Polygon` and `MultiPolygon`), rows are partitioned and one `GeoArrowResult` is emitted per non-empty group.
+- **Bounds.** Each result carries a computed `[minX, minY, maxX, maxY]`. When splitting, each group sees its own tight bounds.
+- **Attribute propagation.** Attribute values follow the split via `sourceIndices`.
+- **Empty / unknown WKBs** are silently dropped.
+
+**Returns** `GeoArrowResult[]` — empty array if no rows have parseable geometries.
+
+**Peer dependencies**
+
+- `apache-arrow` (used to construct the returned `Table`).
+
+**Example**
+
+```ts
+import { buildGeoArrowTables, normalizeGeomType } from '@walkthru-earth/objex-utils';
+
+const wkbArrays: Uint8Array[] = rows.map(r => r.geometry);
+const attributes = new Map([
+  ['id', { values: rows.map(r => r.id), type: 'BIGINT' }],
+  ['name', { values: rows.map(r => r.name), type: 'VARCHAR' }]
+]);
+
+// When you know the type from GeoParquet metadata:
+const [result] = buildGeoArrowTables(wkbArrays, attributes, normalizeGeomType('POLYGON'));
+
+// When you do not:
+const results = buildGeoArrowTables(wkbArrays, attributes);
+for (const r of results) {
+  console.log(r.geometryType, r.table.numRows, r.bounds);
+}
+```

package/docs/local-storage.md

@@ -0,0 +1,51 @@
+# localStorage helpers
+
+SSR-safe JSON persistence on top of `window.localStorage`.
+
+Source: `src/lib/utils/local-storage.ts`.
+
+## Functions
+
+### `loadFromStorage<T>(key, defaultValue)`
+
+```ts
+function loadFromStorage<T>(key: string, defaultValue: T): T
+```
+
+Load a JSON value from localStorage. Returns `defaultValue` when:
+
+- Running in SSR (`typeof window === 'undefined'`).
+- The key is not present.
+- `JSON.parse` throws (stored value corrupted).
+
+Never throws.
+
+### `persistToStorage(key, value)`
+
+```ts
+function persistToStorage(key: string, value: unknown): void
+```
+
+Write a JSON-serializable value to localStorage. No-ops silently when:
+
+- Running in SSR.
+- `localStorage.setItem` throws (quota exceeded, private-browsing restrictions, Safari storage partitioning).
+
+Pair with the [`STORAGE_KEYS`](./types-constants.md#storage_keys) constants to keep key strings consistent across the app.
+
+## Example
+
+```ts
+import { loadFromStorage, persistToStorage, STORAGE_KEYS } from '@walkthru-earth/objex-utils';
+
+interface Settings {
+  theme: 'light' | 'dark' | 'system';
+  locale: 'en' | 'ar';
+}
+
+const defaults: Settings = { theme: 'system', locale: 'en' };
+let settings = loadFromStorage<Settings>(STORAGE_KEYS.SETTINGS, defaults);
+
+// later…
+persistToStorage(STORAGE_KEYS.SETTINGS, settings);
+```

package/docs/markdown-sql.md

@@ -0,0 +1,109 @@
+# Markdown + SQL parsing
+
+Parse markdown documents with YAML frontmatter and Evidence-style SQL code blocks, run interpolation, and stub blocks for server-side rendering.
+
+Source: `src/lib/utils/markdown-sql.ts`.
+
+## Peer dependency
+
+- `yaml >= 2` — **dynamically imported** only when frontmatter is detected. If `yaml` is not installed and the document has frontmatter, parsing silently returns `frontmatter: {}` and continues. Consumers who never call `parseMarkdownDocument` do not need `yaml` at all.
+
+## Types
+
+### `SqlBlock`
+
+```ts
+interface SqlBlock {
+  name: string;       // block identifier from ```sql <name>
+  sql: string;        // raw SQL body
+  startLine: number;  // 0-indexed line index of the opening ``` fence
+  endLine: number;    // 0-indexed line index of the closing ``` fence
+}
+```
+
+### `ParsedMarkdownDocument`
+
+```ts
+interface ParsedMarkdownDocument {
+  frontmatter: Record<string, any>;
+  content: string;     // markdown with frontmatter stripped
+  sqlBlocks: SqlBlock[];
+}
+```
+
+## Functions
+
+### `parseMarkdownDocument(markdown)`
+
+```ts
+async function parseMarkdownDocument(
+  markdown: string
+): Promise<ParsedMarkdownDocument>
+```
+
+**Async** (since v1.2). Parses:
+
+- YAML frontmatter delimited by `---\n ... \n---\n` at the top of the file.
+- SQL blocks of the form:
+
+  ```markdown
+  ```sql some_query_name
+  SELECT * FROM table
+  ```
+  ```
+
+  The identifier must match `/^\w[\w-]*$/`. Content is captured verbatim until the matching closing fence.
+
+**Behavior**
+
+- Missing or malformed frontmatter → `frontmatter = {}`.
+- Missing `yaml` peer dep → `frontmatter = {}` (silent).
+- No SQL blocks → `sqlBlocks = []`.
+
+### `interpolateTemplates(text, queryResults)`
+
+```ts
+function interpolateTemplates(
+  text: string,
+  queryResults: Map<string, Record<string, any>[]>
+): string
+```
+
+Replace references of the form `{queryName.rows[N].columnName}` in `text` with the corresponding value from `queryResults`.
+
+- Unknown query name → leave untouched.
+- Missing row or column → leave untouched.
+- Non-string values are coerced via `String(value)`.
+
+### `markSqlBlocks(content)`
+
+```ts
+function markSqlBlocks(content: string): string
+```
+
+Replace every `` ```sql <name> ... ``` `` block in `content` with `<div data-sql-block="<name>"></div>`. Useful when streaming `content` through a markdown renderer — the caller can later hydrate each placeholder with the actual query result.
+
+## Pattern
+
+```ts
+import {
+  parseMarkdownDocument,
+  interpolateTemplates,
+  markSqlBlocks,
+} from '@walkthru-earth/objex-utils';
+
+const parsed = await parseMarkdownDocument(rawMarkdown);
+
+if (parsed.sqlBlocks.length) {
+  const results = new Map<string, Record<string, any>[]>();
+  await Promise.all(
+    parsed.sqlBlocks.map(async (b) => {
+      results.set(b.name, await myEngine.query(b.sql));
+    })
+  );
+
+  const interpolated = interpolateTemplates(parsed.content, results);
+  const withPlaceholders = markSqlBlocks(interpolated);
+  // hand withPlaceholders to your markdown renderer
+}
+```

package/docs/parquet-metadata.md

@@ -0,0 +1,133 @@
+# Parquet metadata
+
+Lightweight GeoParquet-aware metadata reader for remote Parquet files. Uses [`hyparquet`](https://github.com/hyparam/hyparquet) via HTTP range requests (~512 KB) so you can inspect schemas and geo metadata **before** DuckDB-WASM finishes booting.
+
+Source: `src/lib/utils/parquet-metadata.ts`.
+
+## Peer dependencies
+
+- `hyparquet >= 1.25`
+- `hyparquet-compressors >= 1.1` (SNAPPY, ZSTD, GZIP, LZ4, BROTLI)
+
+## Types
+
+### `GeoColumnMeta`
+
+```ts
+interface GeoColumnMeta {
+  encoding: string;              // e.g. 'WKB'
+  geometryTypes: string[];       // e.g. ['Polygon', 'MultiPolygon']
+  crs: any | null;               // raw ProjJSON or EPSG identifier
+  bbox?: number[];               // [minX, minY, maxX, maxY]  (or with Z: 6 values)
+}
+```
+
+### `GeoParquetMeta`
+
+```ts
+interface GeoParquetMeta {
+  primaryColumn: string;
+  columns: Record<string, GeoColumnMeta>;
+}
+```
+
+### `ParquetFileMetadata`
+
+```ts
+interface ParquetFileMetadata {
+  rowCount: number;
+  schema: { name: string; type: string }[];
+  geo: GeoParquetMeta | null;      // null for non-geo Parquet
+  legacyGeoParquet: boolean;       // true for pre-1.0 (schema_version without "version" field)
+  createdBy: string | null;
+  numRowGroups: number;
+  compression: string | null;      // e.g. 'SNAPPY', 'ZSTD'
+}
+```
+
+## Functions
+
+### `readParquetMetadata(url)`
+
+```ts
+async function readParquetMetadata(url: string): Promise<ParquetFileMetadata>
+```
+
+Read the Parquet footer from a remote URL via range requests.
+
+**Parameters**
+
+| Name | Type | Meaning |
+|------|------|---------|
+| `url` | `string` | Full HTTPS URL. Must be CORS-accessible. `s3://` / `gs://` URIs must be resolved first (see [`resolveCloudUrl`](./storage.md#resolvecloudurl)). |
+
+**Returns** `Promise<ParquetFileMetadata>`.
+
+**Throws** a native error if the URL is not reachable, CORS is blocked, or the footer is malformed.
+
+**Notes**
+
+- The `geo` field contains the parsed `"geo"` key from Parquet file-level KV metadata. For legacy files (`schema_version` but no `version`), it is still parsed; `legacyGeoParquet` is set so callers can apply fallbacks.
+- `compression` comes from the first row group's first column and is reported capitalized.
+
+### `extractEpsgFromGeoMeta(geo)`
+
+```ts
+function extractEpsgFromGeoMeta(geo: GeoParquetMeta): string | null
+```
+
+Extract an EPSG authority code from a GeoParquet CRS block. Returns `null` for WGS84/CRS84 (no reprojection needed) or when no EPSG identifier is embedded.
+
+**Return examples**
+
+- `'EPSG:27700'` (British National Grid)
+- `'EPSG:3857'` (Web Mercator)
+- `null` (WGS84 or CRS absent)
+
+### `extractGeometryTypes(geo)`
+
+```ts
+function extractGeometryTypes(
+  geo: GeoParquetMeta
+): GeoArrowGeomType[]
+```
+
+Pull the `geometry_types` array from the primary column's metadata and normalize it into [`GeoArrowGeomType`](./geometry.md#geoarrowgeomtype). Useful to skip per-row `ST_GeometryType()` calls.
+
+### `extractBounds(geo)`
+
+```ts
+function extractBounds(
+  geo: GeoParquetMeta
+): [number, number, number, number] | null
+```
+
+Extract the `bbox` from the primary column. Returns `null` when absent. If the bbox has Z (`[minX, minY, minZ, maxX, maxY, maxZ]`), only the XY extent is returned.
+
+## End-to-end example
+
+```ts
+import {
+  readParquetMetadata,
+  extractEpsgFromGeoMeta,
+  extractGeometryTypes,
+  extractBounds,
+} from '@walkthru-earth/objex-utils';
+
+const meta = await readParquetMetadata(
+  'https://example.com/data.parquet'
+);
+
+console.log({
+  rows: meta.rowCount,
+  compression: meta.compression,
+  schema: meta.schema,
+});
+
+if (meta.geo) {
+  const crs = extractEpsgFromGeoMeta(meta.geo);          // null means WGS84
+  const types = extractGeometryTypes(meta.geo);          // ['polygon']
+  const bbox = extractBounds(meta.geo);                  // [minX, minY, maxX, maxY] | null
+  console.log({ crs, types, bbox, legacy: meta.legacyGeoParquet });
+}
+```