@walkthru-earth/objex-utils 1.1.0 → 1.2.0

package/docs/query-engine.md

@@ -0,0 +1,140 @@
+# Query engine
+
+Interfaces only. `@walkthru-earth/objex-utils` does not ship a DuckDB-WASM implementation; the objex app provides one (`src/lib/query/wasm.ts`). Use `QueryEngine` as the shape your own engine must implement, or that downstream consumers of your library can depend on.
+
+Source: `src/lib/query/engine.ts`.
+
+## Types
+
+### `QueryResult`
+
+```ts
+interface QueryResult {
+  columns: string[];
+  types: string[];
+  rowCount: number;
+  rows: Record<string, any>[];   // already materialized
+}
+```
+
+Pre-parsed rows avoid the Arrow version-mismatch surface in DuckDB-WASM (ships Arrow v17 internally while the project may use Arrow v21).
+
+### `MapQueryResult`
+
+```ts
+interface MapQueryResult {
+  wkbArrays: Uint8Array[];                                     // geometry column as raw WKB
+  geometryType: string;                                        // e.g. 'POLYGON'
+  attributes: Map<string, { values: any[]; type: string }>;    // non-geometry columns, columnar
+  rowCount: number;
+}
+```
+
+Raw columnar shape for map rendering — feed straight into [`buildGeoArrowTables`](./geometry.md#buildgeoarrowtables).
+
+### `SchemaField`
+
+```ts
+interface SchemaField {
+  name: string;
+  type: string;       // DuckDB type string, e.g. 'VARCHAR', 'GEOMETRY(EPSG:27700)'
+  nullable: boolean;
+}
+```
+
+### `QuerySource`
+
+```ts
+interface QuerySource {
+  ref: string;          // FROM-clause expression, e.g. read_parquet('...url...') or "db"."schema"."table"
+  filePath?: string;    // optional shortcut for Parquet file metadata queries
+}
+```
+
+### Cancellation
+
+```ts
+interface QueryHandle {
+  result: Promise<QueryResult>;
+  cancel(): Promise<boolean>;    // true if cancelled, false if already completed
+}
+
+interface MapQueryHandle {
+  result: Promise<MapQueryResult>;
+  cancel(): Promise<boolean>;
+}
+
+class QueryCancelledError extends Error {
+  name: 'QueryCancelledError';
+}
+```
+
+The cancellable variants are how `objex` drives long-running queries — `cancel()` invokes DuckDB's `conn.send()` cancel path so the single worker isn't held hostage by an abandoned tab.
+
+## `QueryEngine` interface
+
+```ts
+interface QueryEngine {
+  query(connId: string, sql: string): Promise<QueryResult>;
+
+  queryForMap(
+    connId: string,
+    sql: string,
+    geomCol: string,
+    geomColType: string,
+    sourceCrs?: string | null
+  ): Promise<MapQueryResult>;
+
+  getSchema(connId: string, source: QuerySource): Promise<SchemaField[]>;
+  getRowCount(connId: string, source: QuerySource): Promise<number>;
+  detectCrs(
+    connId: string,
+    source: QuerySource,
+    geomCol: string
+  ): Promise<string | null>;
+
+  getSchemaAndCrs?(
+    connId: string,
+    source: QuerySource,
+    findGeoCol: (schema: SchemaField[]) => string | null
+  ): Promise<{ schema: SchemaField[]; geomCol: string | null; crs: string | null }>;
+
+  queryCancellable?(connId: string, sql: string): QueryHandle;
+  queryForMapCancellable?(
+    connId: string,
+    sql: string,
+    geomCol: string,
+    geomColType: string,
+    sourceCrs?: string | null
+  ): MapQueryHandle;
+
+  forceCancel?(): Promise<void>;
+  registerFileBuffer?(name: string, buffer: Uint8Array): Promise<void>;
+  dropFile?(name: string): Promise<void>;
+
+  releaseMemory(): Promise<void>;
+  dispose(): Promise<void>;
+}
+```
+
+### Methods at a glance
+
+| Method | Required? | Notes |
+|--------|-----------|-------|
+| `query` | yes | Fire-and-forget query returning pre-parsed rows. |
+| `queryForMap` | yes | Geometry-aware query. `geomColType` lets the engine decide whether to wrap in `ST_AsWKB` / `ST_Transform`. `sourceCrs` (e.g. `'EPSG:27700'`) is only used when the geometry column does not have a parameterized GEOMETRY type; pass `null` for WGS84. |
+| `getSchema` / `getRowCount` | yes | Metadata helpers. `QuerySource.ref` is the FROM expression — works for both files and catalog tables. |
+| `detectCrs` | yes | Returns e.g. `'EPSG:27700'` or `null` (WGS84 / unknown). |
+| `getSchemaAndCrs` | optional | Single round-trip combining the three above. `findGeoCol` is injected so the engine can inspect the schema without owning the heuristic. |
+| `queryCancellable` / `queryForMapCancellable` | optional | Prefer these in UIs. Falls back to non-cancellable when absent. |
+| `forceCancel` | optional | Kill any in-flight query across all handles. |
+| `registerFileBuffer` / `dropFile` | optional | Register an in-memory file in DuckDB's VFS (used for ATTACH'd catalogs, drag-and-drop). |
+| `releaseMemory` | yes | Trim DuckDB buffer pools (e.g. after closing large tabs). |
+| `dispose` | yes | Tear down the connection fully. |
+
+### Implementation checklist
+
+- Serialize rows with BigInt-safe JSON (see [`jsonReplacerBigInt`](./formatting.md#jsonreplacerbigint)).
+- Wrap geometry selects with `ST_AsWKB(...)` before Arrow export — DuckDB-WASM cannot Arrow-export `GEOMETRY` yet ([duckdb/duckdb-wasm#2187](https://github.com/duckdb/duckdb-wasm/issues/2187)).
+- Return `rowCount` even when the result is streamed; consumers display progress.
+- Surface cancellation via `QueryCancelledError` so UIs can distinguish "user aborted" from real failures.

package/docs/storage.md

@@ -0,0 +1,251 @@
+# Storage
+
+URL parsing, provider registry, and the `StorageAdapter` contract.
+
+Sources:
+
+- `src/lib/utils/storage-url.ts` — generic URL / bucket parser
+- `src/lib/utils/cloud-url.ts` — cloud-scheme → HTTPS resolver
+- `src/lib/storage/providers.ts` — provider registry + access-mode logic
+- `src/lib/storage/adapter.ts` — the `StorageAdapter` interface
+- `src/lib/storage/url-adapter.ts` — adapter for arbitrary HTTPS URLs
+
+## URL parsing
+
+### `parseStorageUrl(input, defaults?)`
+
+```ts
+function parseStorageUrl(
+  input: string,
+  defaults?: Defaults
+): ParsedStorageUrl
+```
+
+Universal parser for any bucket / cloud URL a user might paste. Accepted formats:
+
+- URI schemes: `s3://`, `s3a://`, `s3n://`, `r2://`, `gs://`, `gcs://`, `azure://`, `az://`, `abfs://`, `abfss://`, `wasbs://`, `swift://`, `file://`
+- AWS S3 virtual-hosted, path-style, global URLs (region auto-detected from host)
+- Cloudflare R2, Google Cloud Storage, Azure, DigitalOcean Spaces, Wasabi, Backblaze B2, Alibaba OSS, Tencent COS, Yandex, Storj, Contabo, Hetzner, Linode, OVHcloud, MinIO
+- Generic custom endpoints
+- Plain bucket names (defaults to `s3`)
+
+**Types**
+
+```ts
+interface ParsedStorageUrl {
+  bucket: string;
+  region: string;
+  endpoint: string;
+  provider: StorageProvider;   // same ID space as ProviderId (see below)
+  prefix: string;              // path after bucket (may be '')
+}
+
+interface Defaults {
+  region?: string;
+  endpoint?: string;
+  provider?: StorageProvider;
+}
+```
+
+**Example**
+
+```ts
+parseStorageUrl('s3://us-west-2.my-bucket/data/*.parquet');
+// {
+//   provider: 's3',
+//   bucket: 'my-bucket',
+//   region: 'us-west-2',
+//   endpoint: 'https://s3.us-west-2.amazonaws.com',
+//   prefix: 'data/*.parquet'
+// }
+```
+
+### `looksLikeUrl(input)`
+
+```ts
+function looksLikeUrl(input: string): boolean
+```
+
+Return `true` if `input` resembles a URL/URI rather than a plain bucket name. Useful to decide whether to pass it to `parseStorageUrl`.
+
+### `describeParseResult(parsed)`
+
+```ts
+function describeParseResult(parsed: ParsedStorageUrl): string
+```
+
+Build a human-readable summary of a parse result (bucket, endpoint, region, provider, prefix). Used in the objex UI but safe to render anywhere.
+
+### `resolveCloudUrl(url)`
+
+```ts
+function resolveCloudUrl(url: string): string
+```
+
+Convert a cloud-protocol URL to an HTTPS URL that any fetch client can handle.
+
+| Input | Output |
+|-------|--------|
+| `s3://bucket/key` | `https://bucket.s3.<region>.amazonaws.com/key` — region auto-detected from host prefix (e.g. `us-west-2.opendata.source.coop`), or `us-east-1` as fallback |
+| `gs://bucket/key` | `https://storage.googleapis.com/bucket/key` |
+| `http(s)://...` | returned unchanged |
+| Anything else | returned unchanged |
+
+### `getNativeScheme(provider)`
+
+```ts
+function getNativeScheme(provider: string): string
+```
+
+Map a provider ID to its canonical URI scheme prefix (first entry in the provider registry). Falls back to `'s3'` for unknown S3-compatible providers.
+
+### `safeDecodeURIComponent(s)`
+
+```ts
+function safeDecodeURIComponent(s: string): string
+```
+
+Percent-decode a URL component without throwing on malformed input. Returns the original string if decoding fails.
+
+## Provider registry
+
+### `ProviderId`
+
+```ts
+type ProviderId =
+  | 's3' | 'gcs' | 'r2' | 'minio' | 'azure' | 'storj' | 'b2'
+  | 'digitalocean' | 'wasabi' | 'contabo' | 'hetzner' | 'linode' | 'ovhcloud';
+```
+
+### `ProviderDef`
+
+```ts
+interface ProviderDef {
+  label: string;                          // "Amazon S3"
+  description: string;                    // short helper text for UI
+  authMethod: 'sigv4' | 'sas-token';
+  needsRegion: boolean;
+  needsEndpoint: boolean;
+  defaultRegion: string;
+  endpointTemplate: string | null;        // may contain {region}
+  regions: ProviderRegion[];
+  bucketLabel?: string;                   // e.g. Azure uses "Container"
+  endpointPlaceholder: string;
+  schemes: string[];                      // e.g. ['s3', 's3a', 's3n']
+}
+
+interface ProviderRegion {
+  code: string;
+  label: string;
+}
+```
+
+### Registry exports
+
+```ts
+const PROVIDERS: Record<ProviderId, ProviderDef>;
+const PROVIDER_IDS: ProviderId[];
+```
+
+`PROVIDER_IDS` is the display order used in the objex connection dialog.
+
+### Helpers
+
+```ts
+function getProvider(id: string): ProviderDef;
+function buildEndpointFromTemplate(id: ProviderId, region: string): string;
+function buildProviderBaseUrl(
+  provider: ProviderId,
+  endpoint: string,
+  bucket: string,
+  region: string
+): string;
+function isGcsProvider(provider: string, endpoint: string): boolean;
+```
+
+| Function | Semantics |
+|----------|-----------|
+| `getProvider` | Unknown IDs fall back to the S3 entry (never throws). |
+| `buildEndpointFromTemplate` | Substitute `{region}` in the provider's template. |
+| `buildProviderBaseUrl` | Produce the HTTPS base URL for API requests (endpoint + bucket, correctly interleaved for virtual-host vs path-style). |
+| `isGcsProvider` | `true` when the connection uses the GCS JSON API rather than S3 XML — used to pick adapter implementation. |
+
+### Access mode
+
+```ts
+type AccessMode = 'public-https' | 'sas-https' | 'signed-s3';
+
+function getAccessMode(conn: AccessModeInput): AccessMode;
+function isPubliclyStreamable(conn: AccessModeInput): boolean;
+```
+
+`AccessMode` is the single source of truth for how an HTTP client (DuckDB httpfs, MapLibre, fetch) should read the connection's files:
+
+| Mode | Meaning |
+|------|---------|
+| `'public-https'` | Plain HTTPS, no signing — anonymous S3, GCS, R2, public MinIO, etc. |
+| `'sas-https'` | HTTPS with SAS token appended to the URL — Azure. |
+| `'signed-s3'` | Requires SigV4 signing — authenticated S3-compatible connections. |
+
+`isPubliclyStreamable` is `true` for `'public-https'` and `'sas-https'` (anything a plain `fetch()` / `<img>` / `<video>` can reach directly).
+
+## StorageAdapter
+
+### Interface
+
+```ts
+interface StorageAdapter {
+  list(path: string, signal?: AbortSignal): Promise<FileEntry[]>;
+  read(
+    path: string,
+    offset?: number,
+    length?: number,
+    signal?: AbortSignal
+  ): Promise<Uint8Array>;
+  head(path: string, signal?: AbortSignal): Promise<FileEntry>;
+  listPage?(
+    path: string,
+    continuationToken?: string,
+    pageSize?: number,
+    signal?: AbortSignal
+  ): Promise<ListPage>;
+  put(key: string, data: Uint8Array, contentType?: string): Promise<WriteResult>;
+  delete(key: string): Promise<void>;
+  deletePrefix(prefix: string): Promise<{ deleted: number }>;
+  copy(srcKey: string, destKey: string): Promise<WriteResult>;
+  readonly supportsWrite: boolean;
+}
+
+interface ListPage {
+  entries: FileEntry[];
+  continuationToken?: string;
+  hasMore: boolean;
+}
+```
+
+See [`types-constants.md`](./types-constants.md#fileentry) for `FileEntry` and `WriteResult`.
+
+**Conventions**
+
+- `signal` is propagated all the way to the underlying `fetch()`. Callers should always pass an `AbortController.signal` so tab switches / cleanups don't leak requests.
+- `read(path, offset, length)` uses HTTP Range when supported; omitting offset/length reads the whole object.
+- `listPage` is optional — read-heavy viewers (e.g. paginated browser) should feature-detect it.
+- Read-only adapters should throw a native `Error` from write methods and set `supportsWrite = false`.
+
+### `UrlAdapter`
+
+```ts
+class UrlAdapter implements StorageAdapter {
+  readonly supportsWrite = false;
+
+  read(url: string, offset?: number, length?: number, signal?: AbortSignal): Promise<Uint8Array>;
+  head(url: string, signal?: AbortSignal): Promise<FileEntry>;
+  list(): Promise<FileEntry[]>;    // always []
+  put(): Promise<WriteResult>;      // throws
+  delete(): Promise<void>;           // throws
+  deletePrefix(): Promise<{ deleted: number }>;  // throws
+  copy(): Promise<WriteResult>;     // throws
+}
+```
+
+Minimal adapter for arbitrary HTTPS URLs (`tab.source === 'url'`). `path` is the full URL. Supports `read()` (Range requests) and `head()` only. Listing returns an empty array, writes throw. Use when you have a raw HTTPS link and do not need connection credentials.

package/docs/types-constants.md

@@ -0,0 +1,173 @@
+# Core types & constants
+
+Shared data shapes (connections, tabs, files) and package-wide constants.
+
+Sources: `src/lib/types.ts`, `src/lib/constants.ts`.
+
+## Types
+
+### `FileEntry`
+
+```ts
+interface FileEntry {
+  name: string;
+  path: string;
+  is_dir: boolean;
+  size: number;
+  modified: number;   // unix timestamp in milliseconds
+  extension: string;
+}
+```
+
+A single file or directory entry returned by any `StorageAdapter.list` / `list_page` / `head`. `extension` is lowercase without the leading dot (empty string for directories and extensionless files).
+
+### `Connection`
+
+```ts
+interface Connection {
+  id: string;
+  name: string;
+  provider: string;       // same ID space as ProviderId
+  endpoint: string;
+  bucket: string;
+  region: string;
+  anonymous: boolean;
+  authMethod?: 'sigv4' | 'sas-token';
+  rootPrefix?: string;
+}
+```
+
+Persisted connection record. **No credentials** — secrets live only in the session `ConnectionConfig`.
+
+### `ConnectionConfig`
+
+```ts
+interface ConnectionConfig {
+  name: string;
+  provider: string;
+  endpoint: string;
+  bucket: string;
+  region: string;
+  access_key?: string;
+  secret_key?: string;
+  sas_token?: string;
+  anonymous: boolean;
+  authMethod?: 'sigv4' | 'sas-token';
+  rootPrefix?: string;
+}
+```
+
+Transient form with credentials. Never persist this directly.
+
+### `Tab`
+
+```ts
+interface Tab {
+  id: string;
+  name: string;
+  path: string;
+  source: 'remote' | 'url';
+  connectionId?: string;
+  extension: string;       // lowercase, no leading dot
+  size?: number;
+  sourceRef?: string;      // FROM-clause ref when reading from a catalog table
+}
+```
+
+Represents one open viewer tab. When `sourceRef` is set the tab reads from a DuckDB/DuckLake FROM expression rather than a file URL — file-only metadata paths (hyparquet prefetch, `parquet_kv_metadata`) are skipped.
+
+### `WriteResult`
+
+```ts
+interface WriteResult {
+  key: string;
+  size: number;
+  e_tag?: string;
+}
+```
+
+Returned from every `StorageAdapter` write method (`put`, `copy`).
+
+### `Theme`
+
+```ts
+type Theme = 'light' | 'dark' | 'system';
+```
+
+## Constants
+
+### `STORAGE_KEYS`
+
+```ts
+const STORAGE_KEYS = {
+  SETTINGS: 'obstore-explore-settings',
+  CONNECTIONS: 'obstore-explore-connections',
+  QUERY_HISTORY: 'obstore-explore-query-history',
+} as const;
+```
+
+Namespace for localStorage keys. Always use these when persisting app state with [`persistToStorage`](./local-storage.md#persisttostorage-key-value) / [`loadFromStorage`](./local-storage.md#loadfromstoragetkey-defaultvalue).
+
+### `WGS84_CODES`
+
+```ts
+const WGS84_CODES: Set<number>;   // { 4326, 4979 }
+```
+
+EPSG codes considered equivalent to WGS84. Use to short-circuit reprojection.
+
+### `DEFAULT_TARGET_CRS`
+
+```ts
+const DEFAULT_TARGET_CRS = 'OGC:CRS84';
+```
+
+Canonical target for DuckDB `ST_Transform`. OGC:CRS84 (longitude, latitude) matches GeoParquet 1.1+ and is functionally equivalent to EPSG:4326 under `geometry_always_xy = true`.
+
+### `DUCKDB_INIT_TIMEOUT_MS`
+
+```ts
+const DUCKDB_INIT_TIMEOUT_MS = 30_000;
+```
+
+Max milliseconds the DuckDB-WASM worker has to boot before the UI surfaces an error.
+
+### `MAX_QUERY_HISTORY_ENTRIES`
+
+```ts
+const MAX_QUERY_HISTORY_ENTRIES = 200;
+```
+
+LRU cap for persisted query history.
+
+### `SQL_PREVIEW_LENGTH`
+
+```ts
+const SQL_PREVIEW_LENGTH = 120;
+```
+
+Characters used for SQL previews in the query history list.
+
+### `VIEWER_DIR_EXTENSIONS`
+
+```ts
+const VIEWER_DIR_EXTENSIONS: Set<string>;   // { 'zarr', 'zr3' }
+```
+
+Extensions that open as a viewer even though the path is a directory (Zarr stores).
+
+### `LAYER_HUE_MULTIPLIER`
+
+```ts
+const LAYER_HUE_MULTIPLIER = 137;
+```
+
+Golden-angle-based hue step (137° ≈ 360° × (1 − 1/φ)). Multiply by layer index to get visually distinct hues.
+
+### `COPY_FEEDBACK_MS`
+
+```ts
+const COPY_FEEDBACK_MS = 2000;
+```
+
+Duration of the "Copied!" confirmation state on copy-to-clipboard buttons.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@walkthru-earth/objex-utils",
-  "version": "1.1.0",
+  "version": "1.2.0",
   "description": "Pure TypeScript utilities from objex — WKB parser, GeoArrow builder, storage URL parser, file type registry",
   "author": "Youssef Harby <yharby@walkthru.earth>",
   "license": "CC-BY-4.0",
@@ -22,6 +22,7 @@
   },
   "files": [
     "dist",
+    "docs",
     "README.md"
   ],
   "peerDependencies": {