@walkthru-earth/objex-utils 1.1.0 → 1.2.1

package/docs/geometry.md

@@ -0,0 +1,198 @@
+# Geometry
+
+WKB parsing, geometry-column detection, and GeoArrow table construction. Zero-copy where possible.
+
+Source: `src/lib/utils/wkb.ts`, `src/lib/utils/geoarrow.ts`.
+
+## Types
+
+### `GeoType`
+
+```ts
+type GeoType =
+  | 'Point'
+  | 'LineString'
+  | 'Polygon'
+  | 'MultiPoint'
+  | 'MultiLineString'
+  | 'MultiPolygon'
+  | 'Unknown';
+```
+
+`'Unknown'` is returned for unsupported WKB types (GeometryCollections, TINs, triangles, etc.).
+
+### `ParsedGeometry`
+
+```ts
+interface ParsedGeometry {
+  type: GeoType;
+  coordinates: number[] | number[][] | number[][][] | number[][][][];
+}
+```
+
+Coordinate nesting follows GeoJSON conventions.
+
+| Type | Shape |
+|------|-------|
+| `Point` | `[x, y]` |
+| `MultiPoint` / `LineString` | `[[x, y], ...]` |
+| `MultiLineString` / `Polygon` | `[[[x, y], ...], ...]` |
+| `MultiPolygon` | `[[[[x, y], ...], ...], ...]` |
+
+### `GeoArrowGeomType`
+
+```ts
+type GeoArrowGeomType =
+  | 'point'
+  | 'linestring'
+  | 'polygon'
+  | 'multipoint'
+  | 'multilinestring'
+  | 'multipolygon';
+```
+
+Lowercase normalized form used by the GeoArrow builder and `@geoarrow/deck.gl-layers`.
+
+### `GeoArrowResult`
+
+```ts
+interface GeoArrowResult {
+  table: Table;                                      // apache-arrow Table
+  geometryType: GeoArrowGeomType;
+  bounds: [number, number, number, number];          // [minX, minY, maxX, maxY]
+  sourceIndices: number[];                           // table row i → original wkbArrays[sourceIndices[i]]
+}
+```
+
+`sourceIndices` lets callers map picked rows back to the original row order when mixed geometry types force a split.
+
+## Functions
+
+### `toBinary(value)`
+
+```ts
+function toBinary(value: unknown): Uint8Array | null
+```
+
+Normalize an arbitrary "possibly-binary" value to a `Uint8Array`.
+
+| Input | Handling |
+|-------|----------|
+| `Uint8Array` | Returned as-is |
+| `ArrayBuffer` | Wrapped in `new Uint8Array(buf)` |
+| `number[]` | `new Uint8Array(arr)` |
+| Hex string (even length, `[0-9a-fA-F]`) | Decoded to bytes |
+| DuckDB `toJSON()` blob object `{0: b0, 1: b1, ...}` | Reassembled into bytes |
+| Anything else | `null` |
+
+Returns **`null`** rather than throwing on unrecognized input, so callers can fall through.
+
+### `parseWKB(data)`
+
+```ts
+function parseWKB(data: Uint8Array): ParsedGeometry | null
+```
+
+Parse a WKB byte blob.
+
+- Supports standard WKB, ISO WKB with Z/M flags, and EWKB with SRID prefix (PostGIS). Z/M ordinates are **dropped** — only X/Y is returned.
+- Supports Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon.
+- Returns **`null`** for truncated buffers, invalid byte-order flags, or malformed geometry.
+- GeometryCollections (WKB type 7) return `{ type: 'Unknown', coordinates: [] }`.
+
+### `findGeoColumn(schema)`
+
+```ts
+function findGeoColumn(
+  schema: { name: string; type: string }[]
+): string | null
+```
+
+Schema-only heuristic for locating the geometry column. Checks in priority order:
+
+1. **Type** contains a geometry keyword (`GEOMETRY`, `POINT`, `WKB`, ...) — match wins immediately.
+2. Exact well-known **name** (`geometry`, `geom`, `the_geom`, ...) **with** a binary-ish type (`BLOB`, `BINARY`, `VARBINARY`, `BYTES`).
+3. Exact well-known **name**, any type.
+4. Name contains a geo hint (`geom`, `geo_`, `wkb`, `wkt`, `shape`, `spatial`) with binary-ish type.
+5. Name contains geo hint, any type.
+
+Returns the first matching `name`, or `null` if no heuristic hits. Use `findGeoColumnFromRows` as a fallback when the schema is not informative.
+
+### `findGeoColumnFromRows(rows, schema)`
+
+```ts
+function findGeoColumnFromRows(
+  rows: Record<string, unknown>[],
+  schema: { name: string; type: string }[]
+): string | null
+```
+
+Row-based probe. Samples the first row and classifies values:
+
+- Binary-typed columns: tests for WKB magic bytes (endian flag 0x00/0x01 + valid geometry type).
+- Other columns: probes hex-encoded WKB, WKT strings starting with `POINT(`, `LINESTRING(`, etc., and GeoJSON geometry objects (`{type: 'Point', coordinates: [...]}`).
+
+Returns the first column whose value looks geometry-shaped, or `null`.
+
+### `normalizeGeomType(raw)`
+
+```ts
+function normalizeGeomType(raw: string): GeoArrowGeomType
+```
+
+Map a DuckDB `ST_GeometryType()` result (`'POINT'`, `'ST_Polygon'`, etc., case-insensitive, optional `ST_` prefix) to a `GeoArrowGeomType`. Unknown input falls back to `'polygon'`.
+
+### `buildGeoArrowTables(wkbArrays, attributes, knownGeomType?)`
+
+```ts
+function buildGeoArrowTables(
+  wkbArrays: Uint8Array[],
+  attributes: Map<string, { values: any[]; type: string }>,
+  knownGeomType?: GeoArrowGeomType
+): GeoArrowResult[]
+```
+
+Build one or more Arrow `Table` objects keyed by geometry type, ready for `@geoarrow/deck.gl-layers`.
+
+**Parameters**
+
+| Name | Type | Meaning |
+|------|------|---------|
+| `wkbArrays` | `Uint8Array[]` | Per-row WKB binary. Entries may be empty / invalid — they are skipped. |
+| `attributes` | `Map<name, { values, type }>` | Non-geometry columns. `values.length` must equal `wkbArrays.length`. `type` is a DuckDB/Arrow type string used for Arrow dtype inference (numeric → Float64, bool → Bool, everything else → Utf8). |
+| `knownGeomType` | optional | If provided (e.g. from GeoParquet metadata), classification is skipped — all WKBs are assumed to share this type, and one `GeoArrowResult` is returned. |
+
+**Behavior**
+
+- **Zero-copy geometry ingest.** A 5-byte peek classifies each WKB; coordinates are read directly into pre-allocated `Float64Array` backings with no intermediate JS objects.
+- **Mixed-type splitting.** If `knownGeomType` is not provided and the rows span multiple types (e.g. `Polygon` and `MultiPolygon`), rows are partitioned and one `GeoArrowResult` is emitted per non-empty group.
+- **Bounds.** Each result carries a computed `[minX, minY, maxX, maxY]`. When splitting, each group sees its own tight bounds.
+- **Attribute propagation.** Attribute values follow the split via `sourceIndices`.
+- **Empty / unknown WKBs** are silently dropped.
+
+**Returns** `GeoArrowResult[]` — empty array if no rows have parseable geometries.
+
+**Peer dependencies**
+
+- `apache-arrow` (used to construct the returned `Table`).
+
+**Example**
+
+```ts
+import { buildGeoArrowTables, normalizeGeomType } from '@walkthru-earth/objex-utils';
+
+const wkbArrays: Uint8Array[] = rows.map(r => r.geometry);
+const attributes = new Map([
+  ['id', { values: rows.map(r => r.id), type: 'BIGINT' }],
+  ['name', { values: rows.map(r => r.name), type: 'VARCHAR' }]
+]);
+
+// When you know the type from GeoParquet metadata:
+const [result] = buildGeoArrowTables(wkbArrays, attributes, normalizeGeomType('POLYGON'));
+
+// When you do not:
+const results = buildGeoArrowTables(wkbArrays, attributes);
+for (const r of results) {
+  console.log(r.geometryType, r.table.numRows, r.bounds);
+}
+```

package/docs/local-storage.md

@@ -0,0 +1,51 @@
+# localStorage helpers
+
+SSR-safe JSON persistence on top of `window.localStorage`.
+
+Source: `src/lib/utils/local-storage.ts`.
+
+## Functions
+
+### `loadFromStorage<T>(key, defaultValue)`
+
+```ts
+function loadFromStorage<T>(key: string, defaultValue: T): T
+```
+
+Load a JSON value from localStorage. Returns `defaultValue` when:
+
+- Running in SSR (`typeof window === 'undefined'`).
+- The key is not present.
+- `JSON.parse` throws (stored value corrupted).
+
+Never throws.
+
+### `persistToStorage(key, value)`
+
+```ts
+function persistToStorage(key: string, value: unknown): void
+```
+
+Write a JSON-serializable value to localStorage. No-ops silently when:
+
+- Running in SSR.
+- `localStorage.setItem` throws (quota exceeded, private-browsing restrictions, Safari storage partitioning).
+
+Pair with the [`STORAGE_KEYS`](./types-constants.md#storage_keys) constants to keep key strings consistent across the app.
+
+## Example
+
+```ts
+import { loadFromStorage, persistToStorage, STORAGE_KEYS } from '@walkthru-earth/objex-utils';
+
+interface Settings {
+  theme: 'light' | 'dark' | 'system';
+  locale: 'en' | 'ar';
+}
+
+const defaults: Settings = { theme: 'system', locale: 'en' };
+let settings = loadFromStorage<Settings>(STORAGE_KEYS.SETTINGS, defaults);
+
+// later…
+persistToStorage(STORAGE_KEYS.SETTINGS, settings);
+```

package/docs/markdown-sql.md

@@ -0,0 +1,109 @@
+# Markdown + SQL parsing
+
+Parse markdown documents with YAML frontmatter and Evidence-style SQL code blocks, run interpolation, and stub blocks for server-side rendering.
+
+Source: `src/lib/utils/markdown-sql.ts`.
+
+## Peer dependency
+
+- `yaml >= 2` — **dynamically imported** only when frontmatter is detected. If `yaml` is not installed and the document has frontmatter, parsing silently returns `frontmatter: {}` and continues. Consumers who never call `parseMarkdownDocument` do not need `yaml` at all.
+
+## Types
+
+### `SqlBlock`
+
+```ts
+interface SqlBlock {
+  name: string;       // block identifier from ```sql <name>
+  sql: string;        // raw SQL body
+  startLine: number;  // 0-indexed line index of the opening ``` fence
+  endLine: number;    // 0-indexed line index of the closing ``` fence
+}
+```
+
+### `ParsedMarkdownDocument`
+
+```ts
+interface ParsedMarkdownDocument {
+  frontmatter: Record<string, any>;
+  content: string;     // markdown with frontmatter stripped
+  sqlBlocks: SqlBlock[];
+}
+```
+
+## Functions
+
+### `parseMarkdownDocument(markdown)`
+
+```ts
+async function parseMarkdownDocument(
+  markdown: string
+): Promise<ParsedMarkdownDocument>
+```
+
+**Async** (since v1.2). Parses:
+
+- YAML frontmatter delimited by `---\n ... \n---\n` at the top of the file.
+- SQL blocks of the form:
+
+  ```markdown
+  ```sql some_query_name
+  SELECT * FROM table
+  ```
+  ```
+
+  The identifier must match `/^\w[\w-]*$/`. Content is captured verbatim until the matching closing fence.
+
+**Behavior**
+
+- Missing or malformed frontmatter → `frontmatter = {}`.
+- Missing `yaml` peer dep → `frontmatter = {}` (silent).
+- No SQL blocks → `sqlBlocks = []`.
+
+### `interpolateTemplates(text, queryResults)`
+
+```ts
+function interpolateTemplates(
+  text: string,
+  queryResults: Map<string, Record<string, any>[]>
+): string
+```
+
+Replace references of the form `{queryName.rows[N].columnName}` in `text` with the corresponding value from `queryResults`.
+
+- Unknown query name → leave untouched.
+- Missing row or column → leave untouched.
+- Non-string values are coerced via `String(value)`.
+
+### `markSqlBlocks(content)`
+
+```ts
+function markSqlBlocks(content: string): string
+```
+
+Replace every `` ```sql <name> ... ``` `` block in `content` with `<div data-sql-block="<name>"></div>`. Useful when streaming `content` through a markdown renderer — the caller can later hydrate each placeholder with the actual query result.
+
+## Pattern
+
+```ts
+import {
+  parseMarkdownDocument,
+  interpolateTemplates,
+  markSqlBlocks,
+} from '@walkthru-earth/objex-utils';
+
+const parsed = await parseMarkdownDocument(rawMarkdown);
+
+if (parsed.sqlBlocks.length) {
+  const results = new Map<string, Record<string, any>[]>();
+  await Promise.all(
+    parsed.sqlBlocks.map(async (b) => {
+      results.set(b.name, await myEngine.query(b.sql));
+    })
+  );
+
+  const interpolated = interpolateTemplates(parsed.content, results);
+  const withPlaceholders = markSqlBlocks(interpolated);
+  // hand withPlaceholders to your markdown renderer
+}
+```

package/docs/parquet-metadata.md

@@ -0,0 +1,133 @@
+# Parquet metadata
+
+Lightweight GeoParquet-aware metadata reader for remote Parquet files. Uses [`hyparquet`](https://github.com/hyparam/hyparquet) via HTTP range requests (~512 KB) so you can inspect schemas and geo metadata **before** DuckDB-WASM finishes booting.
+
+Source: `src/lib/utils/parquet-metadata.ts`.
+
+## Peer dependencies
+
+- `hyparquet >= 1.25`
+- `hyparquet-compressors >= 1.1` (SNAPPY, ZSTD, GZIP, LZ4, BROTLI)
+
+## Types
+
+### `GeoColumnMeta`
+
+```ts
+interface GeoColumnMeta {
+  encoding: string;              // e.g. 'WKB'
+  geometryTypes: string[];       // e.g. ['Polygon', 'MultiPolygon']
+  crs: any | null;               // raw ProjJSON or EPSG identifier
+  bbox?: number[];               // [minX, minY, maxX, maxY]  (or with Z: 6 values)
+}
+```
+
+### `GeoParquetMeta`
+
+```ts
+interface GeoParquetMeta {
+  primaryColumn: string;
+  columns: Record<string, GeoColumnMeta>;
+}
+```
+
+### `ParquetFileMetadata`
+
+```ts
+interface ParquetFileMetadata {
+  rowCount: number;
+  schema: { name: string; type: string }[];
+  geo: GeoParquetMeta | null;      // null for non-geo Parquet
+  legacyGeoParquet: boolean;       // true for pre-1.0 (schema_version without "version" field)
+  createdBy: string | null;
+  numRowGroups: number;
+  compression: string | null;      // e.g. 'SNAPPY', 'ZSTD'
+}
+```
+
+## Functions
+
+### `readParquetMetadata(url)`
+
+```ts
+async function readParquetMetadata(url: string): Promise<ParquetFileMetadata>
+```
+
+Read the Parquet footer from a remote URL via range requests.
+
+**Parameters**
+
+| Name | Type | Meaning |
+|------|------|---------|
+| `url` | `string` | Full HTTPS URL. Must be CORS-accessible. `s3://` / `gs://` URIs must be resolved first (see [`resolveCloudUrl`](./storage.md#resolvecloudurl)). |
+
+**Returns** `Promise<ParquetFileMetadata>`.
+
+**Throws** a native error if the URL is not reachable, CORS is blocked, or the footer is malformed.
+
+**Notes**
+
+- The `geo` field contains the parsed `"geo"` key from Parquet file-level KV metadata. For legacy files (`schema_version` but no `version`), it is still parsed; `legacyGeoParquet` is set so callers can apply fallbacks.
+- `compression` comes from the first row group's first column and is reported capitalized.
+
+### `extractEpsgFromGeoMeta(geo)`
+
+```ts
+function extractEpsgFromGeoMeta(geo: GeoParquetMeta): string | null
+```
+
+Extract an EPSG authority code from a GeoParquet CRS block. Returns `null` for WGS84/CRS84 (no reprojection needed) or when no EPSG identifier is embedded.
+
+**Return examples**
+
+- `'EPSG:27700'` (British National Grid)
+- `'EPSG:3857'` (Web Mercator)
+- `null` (WGS84 or CRS absent)
+
+### `extractGeometryTypes(geo)`
+
+```ts
+function extractGeometryTypes(
+  geo: GeoParquetMeta
+): GeoArrowGeomType[]
+```
+
+Pull the `geometry_types` array from the primary column's metadata and normalize it into [`GeoArrowGeomType`](./geometry.md#geoarrowgeomtype). Useful to skip per-row `ST_GeometryType()` calls.
+
+### `extractBounds(geo)`
+
+```ts
+function extractBounds(
+  geo: GeoParquetMeta
+): [number, number, number, number] | null
+```
+
+Extract the `bbox` from the primary column. Returns `null` when absent. If the bbox has Z (`[minX, minY, minZ, maxX, maxY, maxZ]`), only the XY extent is returned.
+
+## End-to-end example
+
+```ts
+import {
+  readParquetMetadata,
+  extractEpsgFromGeoMeta,
+  extractGeometryTypes,
+  extractBounds,
+} from '@walkthru-earth/objex-utils';
+
+const meta = await readParquetMetadata(
+  'https://example.com/data.parquet'
+);
+
+console.log({
+  rows: meta.rowCount,
+  compression: meta.compression,
+  schema: meta.schema,
+});
+
+if (meta.geo) {
+  const crs = extractEpsgFromGeoMeta(meta.geo);          // null means WGS84
+  const types = extractGeometryTypes(meta.geo);          // ['polygon']
+  const bbox = extractBounds(meta.geo);                  // [minX, minY, maxX, maxY] | null
+  console.log({ crs, types, bbox, legacy: meta.legacyGeoParquet });
+}
+```

package/docs/query-engine.md

@@ -0,0 +1,140 @@
+# Query engine
+
+Interfaces only. `@walkthru-earth/objex-utils` does not ship a DuckDB-WASM implementation; the objex app provides one (`src/lib/query/wasm.ts`). Use `QueryEngine` as the shape your own engine must implement, or that downstream consumers of your library can depend on.
+
+Source: `src/lib/query/engine.ts`.
+
+## Types
+
+### `QueryResult`
+
+```ts
+interface QueryResult {
+  columns: string[];
+  types: string[];
+  rowCount: number;
+  rows: Record<string, any>[];   // already materialized
+}
+```
+
+Pre-parsed rows avoid the Arrow version-mismatch surface in DuckDB-WASM (ships Arrow v17 internally while the project may use Arrow v21).
+
+### `MapQueryResult`
+
+```ts
+interface MapQueryResult {
+  wkbArrays: Uint8Array[];                                     // geometry column as raw WKB
+  geometryType: string;                                        // e.g. 'POLYGON'
+  attributes: Map<string, { values: any[]; type: string }>;    // non-geometry columns, columnar
+  rowCount: number;
+}
+```
+
+Raw columnar shape for map rendering — feed straight into [`buildGeoArrowTables`](./geometry.md#buildgeoarrowtables).
+
+### `SchemaField`
+
+```ts
+interface SchemaField {
+  name: string;
+  type: string;       // DuckDB type string, e.g. 'VARCHAR', 'GEOMETRY(EPSG:27700)'
+  nullable: boolean;
+}
+```
+
+### `QuerySource`
+
+```ts
+interface QuerySource {
+  ref: string;          // FROM-clause expression, e.g. read_parquet('...url...') or "db"."schema"."table"
+  filePath?: string;    // optional shortcut for Parquet file metadata queries
+}
+```
+
+### Cancellation
+
+```ts
+interface QueryHandle {
+  result: Promise<QueryResult>;
+  cancel(): Promise<boolean>;    // true if cancelled, false if already completed
+}
+
+interface MapQueryHandle {
+  result: Promise<MapQueryResult>;
+  cancel(): Promise<boolean>;
+}
+
+class QueryCancelledError extends Error {
+  name: 'QueryCancelledError';
+}
+```
+
+The cancellable variants are how `objex` drives long-running queries — `cancel()` invokes DuckDB's `conn.send()` cancel path so the single worker isn't held hostage by an abandoned tab.
+
+## `QueryEngine` interface
+
+```ts
+interface QueryEngine {
+  query(connId: string, sql: string): Promise<QueryResult>;
+
+  queryForMap(
+    connId: string,
+    sql: string,
+    geomCol: string,
+    geomColType: string,
+    sourceCrs?: string | null
+  ): Promise<MapQueryResult>;
+
+  getSchema(connId: string, source: QuerySource): Promise<SchemaField[]>;
+  getRowCount(connId: string, source: QuerySource): Promise<number>;
+  detectCrs(
+    connId: string,
+    source: QuerySource,
+    geomCol: string
+  ): Promise<string | null>;
+
+  getSchemaAndCrs?(
+    connId: string,
+    source: QuerySource,
+    findGeoCol: (schema: SchemaField[]) => string | null
+  ): Promise<{ schema: SchemaField[]; geomCol: string | null; crs: string | null }>;
+
+  queryCancellable?(connId: string, sql: string): QueryHandle;
+  queryForMapCancellable?(
+    connId: string,
+    sql: string,
+    geomCol: string,
+    geomColType: string,
+    sourceCrs?: string | null
+  ): MapQueryHandle;
+
+  forceCancel?(): Promise<void>;
+  registerFileBuffer?(name: string, buffer: Uint8Array): Promise<void>;
+  dropFile?(name: string): Promise<void>;
+
+  releaseMemory(): Promise<void>;
+  dispose(): Promise<void>;
+}
+```
+
+### Methods at a glance
+
+| Method | Required? | Notes |
+|--------|-----------|-------|
+| `query` | yes | Fire-and-forget query returning pre-parsed rows. |
+| `queryForMap` | yes | Geometry-aware query. `geomColType` lets the engine decide whether to wrap in `ST_AsWKB` / `ST_Transform`. `sourceCrs` (e.g. `'EPSG:27700'`) is only used when the geometry column does not have a parameterized GEOMETRY type; pass `null` for WGS84. |
+| `getSchema` / `getRowCount` | yes | Metadata helpers. `QuerySource.ref` is the FROM expression — works for both files and catalog tables. |
+| `detectCrs` | yes | Returns e.g. `'EPSG:27700'` or `null` (WGS84 / unknown). |
+| `getSchemaAndCrs` | optional | Single round-trip combining the three above. `findGeoCol` is injected so the engine can inspect the schema without owning the heuristic. |
+| `queryCancellable` / `queryForMapCancellable` | optional | Prefer these in UIs. Falls back to non-cancellable when absent. |
+| `forceCancel` | optional | Kill any in-flight query across all handles. |
+| `registerFileBuffer` / `dropFile` | optional | Register an in-memory file in DuckDB's VFS (used for ATTACH'd catalogs, drag-and-drop). |
+| `releaseMemory` | yes | Trim DuckDB buffer pools (e.g. after closing large tabs). |
+| `dispose` | yes | Tear down the connection fully. |
+
+### Implementation checklist
+
+- Serialize rows with BigInt-safe JSON (see [`jsonReplacerBigInt`](./formatting.md#jsonreplacerbigint)).
+- Wrap geometry selects with `ST_AsWKB(...)` before Arrow export — DuckDB-WASM cannot Arrow-export `GEOMETRY` yet ([duckdb/duckdb-wasm#2187](https://github.com/duckdb/duckdb-wasm/issues/2187)).
+- Return `rowCount` even when the result is streamed; consumers display progress.
+- Surface cancellation via `QueryCancelledError` so UIs can distinguish "user aborted" from real failures.