@walkthru-earth/objex-utils 1.3.0 → 1.4.0

package/docs/stac-pushdown.md

@@ -0,0 +1,145 @@
+# stac-pushdown
+
+Translate a `FacetState` into a native STAC API query plus CQL2-JSON, gated by what the endpoint advertises in `conformsTo`. Pair with [`stac-facets`](./stac-facets.md): push as much as the server supports, then apply the residual client-side.
+
+```ts
+import {
+  type StacApiCapabilities,
+  type StacNativeQuery,
+  type ToNativeQueryOptions,
+  sniffApiCapabilities,
+  toNativeQuery,
+  toCql2Filter,
+  residualState,
+} from '@walkthru-earth/objex-utils';
+```
+
+Pure TypeScript, zero runtime deps.
+
+## Concept
+
+A STAC API `/conformance` (or the `conformsTo` array on the landing page) advertises which extensions the endpoint implements. Different endpoints support different subsets:
+
+- Earth Search v1: OGC API Features + STAC API Item Search + Filter (CQL2).
+- Microsoft Planetary Computer: same as Earth Search plus Sortables / Queryables.
+- A vanilla `pystac-server` with no Filter ext: bbox + datetime only, the rest must be done client-side.
+
+`sniffApiCapabilities(conformsTo)` returns a flag set the rest of the module branches on. `toNativeQuery` emits whatever the endpoint can honor as native query params (`bbox`, `datetime`, `collections`); `toCql2Filter` emits a CQL2-JSON expression for the cloud-cover / GSD / platform / constellation / instruments / collection-when-no-native-cap dimensions; `residualState` returns the subset of `FacetState` neither was able to push, so the caller can apply it client-side via `applyFacets` (from `stac-facets`).
+
+## Types
+
+### `StacApiCapabilities`
+
+```ts
+interface StacApiCapabilities {
+  bbox: boolean;
+  datetime: boolean;
+  collections: boolean;
+  cql2: boolean;
+  queryables: boolean;
+}
+```
+
+Every flag matches a regex against entries in the `conformsTo` array. False when the URI is absent, true when present.
+
+### `StacNativeQuery`
+
+```ts
+interface StacNativeQuery {
+  bbox?: [number, number, number, number];
+  datetime?: string;
+  collections?: string[];
+  limit?: number;
+  filter?: unknown;
+  'filter-lang'?: 'cql2-json';
+}
+```
+
+Superset of the standard `StacItemsQuery`. `filter` carries the CQL2-JSON object when the endpoint supports the Filter ext; absent otherwise.
+
+### `ToNativeQueryOptions`
+
+```ts
+interface ToNativeQueryOptions {
+  limit?: number;
+  collections?: string[];
+}
+```
+
+`collections` lets the caller force a single-collection scope when classifying a Collection or Catalog (the Filter ext is not the right tool to constrain collection at the protocol level).
+
+## Functions
+
+### `sniffApiCapabilities(conformsTo)`
+
+```ts
+function sniffApiCapabilities(conformsTo: readonly string[] | undefined): StacApiCapabilities;
+```
+
+Regex-matches OGC API Features + STAC API Item Search + Filter extension URIs. Returns all-false when `conformsTo` is missing or empty.
+
+### `toNativeQuery(state, caps, opts?)`
+
+```ts
+function toNativeQuery(
+  state: FacetState,
+  caps: StacApiCapabilities,
+  opts?: ToNativeQueryOptions
+): StacNativeQuery;
+```
+
+Translates a `FacetState` into the native query shape. Drops anything the endpoint cannot honor. The caller still applies the residual client-side via `applyFacets(views, residualState(state, caps))`.
+
+When `caps.cql2 === true` the cloud-cover / GSD / platform / constellation / instruments dimensions are baked into a CQL2-JSON `filter` plus `'filter-lang': 'cql2-json'`. When `caps.cql2 === false` they stay in the residual.
+
+### `toCql2Filter(state, caps)`
+
+```ts
+function toCql2Filter(state: FacetState, caps: StacApiCapabilities): unknown | null;
+```
+
+Emits the CQL2-JSON expression for the dimensions `caps.cql2` covers (cloud cover, GSD, platform, constellation, instruments, and collection when there is no native `collections=` cap). Returns `null` when nothing is pushable. Useful when callers want to compose the filter into a different request shape (POST `/search` with extra fields, etc.).
+
+### `residualState(state, caps)`
+
+```ts
+function residualState(state: FacetState, caps: StacApiCapabilities): FacetState;
+```
+
+Subtract everything `toNativeQuery` (or a hypothetical `toCql2Filter`) would push, return the rest. Pass to `applyFacets(views, residual)` to handle the client-side leftovers.
+
+## Recipe: STAC API source with push-down
+
+```ts
+import {
+  sniffApiCapabilities,
+  toNativeQuery,
+  residualState,
+  applyFacets,
+  extractItemView,
+  type FacetState,
+} from '@walkthru-earth/objex-utils';
+
+async function loadFiltered(endpoint: string, state: FacetState) {
+  const landing = await fetch(endpoint).then((r) => r.json());
+  const caps = sniffApiCapabilities(landing.conformsTo);
+
+  const query = toNativeQuery(state, caps, { limit: 250 });
+  const url = new URL(`${endpoint}/search`);
+  for (const [k, v] of Object.entries(query)) {
+    url.searchParams.set(k, Array.isArray(v) ? v.join(',') : String(v));
+  }
+
+  const fc = await fetch(url).then((r) => r.json());
+  const views = fc.features.map(extractItemView);
+  const residual = residualState(state, caps);
+  return applyFacets(views, residual);
+}
+```
+
+## Caveats
+
+- The CQL2-JSON shape is the **JSON encoding** (RFC 8259 / OGC 21-065). Some servers only accept the **text encoding** at `/search` query strings, in which case POST to `/search` with `Content-Type: application/json` and a body of `{ "filter": <cql2-json>, "filter-lang": "cql2-json", ... }`.
+- `sniffApiCapabilities` is regex-based; it does not fetch `/queryables` to verify which property keys the server actually exposes. If your caller enables push-down for a property the server does not index, the server may reject the request or silently return zero results. Test against your target catalogs.
+- This module does **not** retry, paginate, or AbortSignal-handle. Wrap the request layer yourself.
+- For a fully-orchestrated viewer (link-walking, abort, atomic-swap, item caps, footprint hover), see `StacMosaicViewer` in the main `@walkthru-earth/objex` package.

package/docs/stac-source.md

@@ -0,0 +1,224 @@
+# stac-source
+
+The `StacSource` contract, a unified async-iterable interface over the three STAC ingestion paths (STAC API, stac-geoparquet, self-contained static catalog) so a viewer has one orchestration loop and the UI can branch on capability flags instead of hard-coded discovery modes. Pure TypeScript, zero Svelte / maplibre / deck.gl / DuckDB on the import graph.
+
+Source: `packages/objex-utils/src/stac-source.ts`.
+
+```ts
+import {
+  type StacSource,
+  type StacSourceKind,
+  type StacSourceCapabilities,
+  type StacSourceRequest,
+  type StacSourceBatch,
+  emptyPushdown,
+  createApiSource,
+  createStaticSource,
+} from '@walkthru-earth/objex-utils';
+```
+
+## Concept
+
+A `StacSource` is constructed synchronously (no await, so the orchestrator can read `capabilities.kind` before any I/O) and exposes a single `query(req)` returning an `AsyncIterable<StacSourceBatch>`. Each batch reports the subset of the filter the engine pushed down (`pushedDown`) and the subset the caller must still apply client-side (`residual`, via `applyFacets` from [`stac-facets`](./stac-facets.md)). The split is per batch, not per source: a parquet file with a STRUCT `properties` column can push `eo:cloud_cover` while a sibling with an opaque `properties` cannot.
+
+Two implementations ship here, `createApiSource` and `createStaticSource`. The third path, the stac-geoparquet `StacSource`, lives in the Svelte components package (`@walkthru-earth/objex`) under `query/`, because it needs DuckDB-WASM. It is deliberately NOT exported from `objex-utils` so the `utils/` side stays free of heavy deps.
+
+## Types
+
+### `StacSourceKind`
+
+```ts
+type StacSourceKind = 'api' | 'parquet' | 'static';
+```
+
+Which underlying engine drives the source. Used by the viewer to pick atomic-swap-vs-append, by the UI to choose copy and badges, and by tests.
+
+### `StacSourceCapabilities`
+
+```ts
+interface StacSourceCapabilities {
+  kind: StacSourceKind;
+  label: string;
+  countAvailable: boolean;
+  streaming: boolean;
+  hivePartitioned?: boolean;
+  pushdown: {
+    bbox: boolean;
+    datetime: boolean;
+    collection: boolean;
+    cloudCover: boolean;
+    gsd: boolean;
+    epsg: boolean;
+    platform: boolean;
+    constellation: boolean;
+    instruments: boolean;
+    assetRoles: boolean;
+  };
+}
+```
+
+Read at construction (synchronous) and by the filter UI to decide which controls to disable or badge as "client-side only".
+
+- `label` -- human-readable HUD label, e.g. `"STAC API"`, `"stac-geoparquet"`, `"Static catalog"`.
+- `countAvailable` -- true when a cheap `count(filter, bbox)` exists, so the UI can surface "Y of X".
+- `streaming` -- true when `query()` yields multiple batches before completing.
+- `hivePartitioned` -- true when the source is a hive-partitioned parquet directory (set by the parquet source). Lets the viewer hint at the discovery model without inspecting `kind === 'parquet'` alone, since the same `kind` also covers single-file stac-geoparquet.
+- `pushdown` -- the *ceiling* of what this source kind can push. Exhaustive: every `FacetState` facet has a flag, so adding a new facet is a compile-time error in every consumer until handled. The actual per-request push-down is reported in each batch's `pushedDown`.
+
+### `StacSourceRequest`
+
+```ts
+interface StacSourceRequest {
+  bbox: [number, number, number, number];
+  filter: FacetState;
+  limit: number;
+  pageSize?: number;
+  signal: AbortSignal;
+}
+```
+
+Per-query inputs.
+
+- `bbox` -- WGS84 viewport `[west, south, east, north]`, required. Sources that cannot push bbox still receive it, they stream the whole set and rely on the caller's residual filter.
+- `filter` -- the active `FacetState` (from `stac-facets`).
+- `limit` -- hard item cap for the request.
+- `pageSize` -- optional per-page hint for paginating sources, the server may ignore it.
+- `signal` -- required. Sources MUST throw `DOMException("Aborted", "AbortError")` on abort, never silently complete.
+
+### `StacSourceBatch`
+
+```ts
+interface StacSourceBatch {
+  items: StacItem[];
+  pushedDown: FacetState;
+  residual: FacetState;
+  done: boolean;
+  totalHinted?: number;
+}
+```
+
+One yielded batch.
+
+- `items` -- the items in this batch (empty on the terminal `done` batch).
+- `pushedDown` -- the subset of `filter` the source / engine already applied, reported so the UI can show "pushed".
+- `residual` -- the subset the caller must still apply via `applyFacets(views, residual)`.
+- `done` -- true on the final batch. The iterator's own end-of-iteration also signals completion, this flag lets a caller break the loop the moment a single-yield source finishes.
+- `totalHinted` -- best-effort total matching count, when the source knows it.
+
+### `StacSource`
+
+```ts
+interface StacSource {
+  capabilities: StacSourceCapabilities;
+  query(req: StacSourceRequest): AsyncIterable<StacSourceBatch>;
+  count?(filter: FacetState, bbox: StacSourceRequest['bbox'], signal: AbortSignal): Promise<number>;
+}
+```
+
+The contract every implementation satisfies. `count` is optional, surfaced as "Y of X" only when present (and `capabilities.countAvailable` is true). Neither shipped implementation here defines `count`.
+
+## Functions
+
+### `emptyPushdown()`
+
+```ts
+function emptyPushdown(): StacSourceCapabilities['pushdown'];
+```
+
+Return an all-false push-down flag set, a terse base for capability declarations. Spread it and flip on only the flags a source supports, e.g. `{ ...emptyPushdown(), bbox: true, datetime: true }`.
+
+## Implementations
+
+### `createApiSource(kind, deps)` (`stac-source-api.ts`)
+
+```ts
+function createApiSource(kind: StacRoutableKind, deps: StacApiSourceDeps): StacSource;
+
+interface StacApiSourceDeps {
+  adapter: StorageAdapter;
+  baseHref: string;
+  urlToKey?: (absoluteUrl: string) => string | null;
+  concurrency?: number;
+}
+```
+
+Wraps `hydrateStacItems` (see [`stac-hydrate`](./stac-hydrate.md)) with `itemsQuery` push-down. `kind` is the classified payload (a Collection / Catalog with a `rel="items"` endpoint, or a STAC API `item-collection` page), the factory validates before dispatching, this function does not re-validate.
+
+Advertised `capabilities.pushdown` is the CQL2 ceiling: `bbox`, `datetime`, `collection`, `cloudCover`, `gsd`, `platform`, `constellation`, `instruments`. `epsg` and `assetRoles` stay false. `streaming` is true, `countAvailable` is false.
+
+Per-request behavior:
+
+- On the first `query()`, lazily sniffs the source's `conformsTo` (via [`sniffApiCapabilities`](./stac-pushdown.md)) once and caches the resolved promise, later queries never re-fetch the root. The sniff reads `conformsTo` inline from the Collection / Catalog / item-collection payload when present, else fetches `baseHref` (routing through the adapter when `urlToKey` resolves it). Any failure degrades to a slice-1 fallback of `bbox` + `datetime` only, so the source never throws on construction or first query.
+- Translates `req.filter` into a native query via `toNativeQuery` (bbox + datetime + CQL2 filter), maps it onto a `StacItemsQuery`, and threads it into `hydrateStacItems` so it rides every `rel="next"` page.
+- Reports the actually-pushed `FacetState` by inverting `residualState`: `residual = residualState(req.filter, caps)` and `pushedDown = filter - residual`. Both ride every batch unchanged. `collections` cannot be stamped on `/items` (hydrate walks per-collection), so a collection constraint stays in the residual for the caller to apply client-side.
+- Bridges hydrate's callback-based `onBatch` into the async iterable via an internal promise queue. Abort pushes a terminal `DOMException("Aborted", "AbortError")` that the iterator throws.
+
+### `createStaticSource(kind, deps)` (`stac-source-static.ts`)
+
+```ts
+function createStaticSource(kind: StacRoutableKind, deps: StacStaticSourceDeps): StacSource;
+
+interface StacStaticSourceDeps {
+  adapter: StorageAdapter;
+  baseHref: string;
+  urlToKey?: (absoluteUrl: string) => string | null;
+  concurrency?: number;
+}
+```
+
+Wraps `hydrateStacItems` with NO `itemsQuery`. The entire advertised tree is fetched and the caller filters everything client-side. Capabilities advertise an all-false `pushdown` (via `emptyPushdown()`), `label: 'Static catalog'`, `streaming: true`, `countAvailable: false`. Every batch reports `pushedDown: {}` and `residual: req.filter`. Same internal promise-queue bridge and the same abort contract as the API source.
+
+### The parquet source (not exported here)
+
+The stac-geoparquet `StacSource` (`kind: 'parquet'`) lives in the Svelte components package `@walkthru-earth/objex` under `query/`, because it needs DuckDB-WASM to read parquet and push down predicates. It is intentionally absent from `objex-utils` so this package's import graph stays free of heavy deps. To use the row-level transforms it relies on, see [`stac-geoparquet`](./stac-geoparquet.md), which is pure and does ship here.
+
+## Example
+
+```ts
+import {
+  classifyStac,
+  createApiSource,
+  createStaticSource,
+  hasStacItemsEndpoint,
+  applyFacets,
+  extractItemView,
+  emptyFacetState,
+  type StacSource,
+  type StacSourceBatch,
+} from '@walkthru-earth/objex-utils';
+import type { StorageAdapter } from '@walkthru-earth/objex-utils';
+
+async function build(url: string, adapter: StorageAdapter): Promise<StacSource> {
+  const json = await fetch(url).then((r) => r.json());
+  const kind = classifyStac(json);
+  const serverBacked =
+    (kind.kind === 'collection' || kind.kind === 'catalog') &&
+    hasStacItemsEndpoint(kind.payload);
+  const deps = { adapter, baseHref: url };
+  return serverBacked ? createApiSource(kind, deps) : createStaticSource(kind, deps);
+}
+
+async function run(source: StacSource, bbox: [number, number, number, number]) {
+  const filter = emptyFacetState();
+  const controller = new AbortController();
+  const collected: ReturnType<typeof extractItemView>[] = [];
+
+  for await (const batch of source.query({
+    bbox,
+    filter,
+    limit: 2000,
+    signal: controller.signal,
+  })) {
+    // Only re-filter what the engine could NOT push down.
+    const views = batch.items.map(extractItemView);
+    collected.push(...applyFacets(views, batch.residual));
+    if (batch.done) break;
+  }
+
+  return collected;
+}
+```
+
+## Peer dependencies
+
+None bundled. The implementations depend only on the caller-supplied `StorageAdapter` interface (re-exported from this package's host-side surface) and the pure sibling modules [`stac-hydrate`](./stac-hydrate.md), [`stac-pushdown`](./stac-pushdown.md), and [`stac-facets`](./stac-facets.md). No DuckDB, Svelte, maplibre, or deck.gl.

package/docs/stac-storage-extension.md

@@ -0,0 +1,146 @@
+# STAC Storage Extension
+
+Parser for the STAC Storage Extension (v1.0.0 and v2.0.0) that extracts connection-relevant hints (region, requester-pays, custom-S3 endpoint) from a STAC Item. Pure TypeScript, no fetch, no Svelte dependency.
+
+Source: `packages/objex-utils/src/stac-storage-extension.ts`.
+
+Inspired by lazycogs's `_storage_ext.py`. Recognized schema URLs:
+
+- `https://stac-extensions.github.io/storage/v1.0.0/schema.json`
+- `https://stac-extensions.github.io/storage/v2.0.0/schema.json`
+
+The two versions carry storage metadata in different places:
+
+- **v1** stores fields directly on item `properties` and/or per-asset (`storage:platform`, `storage:region`, `storage:requester_pays`, `storage:tier`). Asset-level fields take precedence over item-level fields. `storage:tier` is ignored (no obstore equivalent).
+- **v2** stores a scheme map at `properties.storage:schemes` keyed by ref name, and each asset references one or more schemes via `storage:refs`. The first matching ref wins.
+
+## Types
+
+### `StorageExtensionVersion`
+
+```ts
+type StorageExtensionVersion = '1.0.0' | '2.0.0';
+```
+
+The two Storage Extension schema versions this parser recognizes.
+
+### `StorageHints`
+
+```ts
+interface StorageHints {
+  platform: string | null;
+  region: string | null;
+  requesterPays: boolean;
+  endpoint: string | null;
+}
+```
+
+Connection-relevant hints extracted from the Storage Extension. All fields are nullable (except `requesterPays`, which is a plain boolean) so callers can merge selectively into existing config without clobbering user-set values.
+
+| Field | Meaning |
+|-------|---------|
+| `platform` | Cloud platform, e.g. `'AWS'`, `'GCP'`, `'AZURE'`, `'MINIO'`. Uppercased. Null when absent. |
+| `region` | Region code, e.g. `'us-west-2'`. Null when absent. |
+| `requesterPays` | `true` when requester-pays must be set. `false` when absent or false. |
+| `endpoint` | Concrete S3-compatible endpoint URL. Null unless a v2 `custom-s3` scheme carries a non-templated value. |
+
+## Functions
+
+### `emptyStorageHints()`
+
+```ts
+function emptyStorageHints(): StorageHints
+```
+
+Return an empty hints record (`platform: null`, `region: null`, `requesterPays: false`, `endpoint: null`). This is what `extractStorageHints` returns when the extension is absent or unparseable, so callers can treat the absent case and the present-but-empty case identically.
+
+### `detectStorageExtensionVersion(item)`
+
+```ts
+function detectStorageExtensionVersion(item: StacItem): StorageExtensionVersion | null
+```
+
+Scan `item.stac_extensions[]` for the Storage Extension schema URL and return its parsed version.
+
+**Parameters**
+
+| Name | Type | Meaning |
+|------|------|---------|
+| `item` | `StacItem` | The STAC Item to inspect. `stac_extensions` is read defensively, so a non-array or missing value returns `null`. |
+
+**Returns** `StorageExtensionVersion | null`. Null when the extension is absent or the version is not one we recognize.
+
+**Notes**
+
+- Both the trailing `/schema.json` suffix and a leading `v` on the version segment are stripped before matching.
+- An exact `'1.0.0'` or `'2.0.0'` is returned directly. For an unknown patch or minor (a hypothetical `v1.0.1`), the parser falls back to major-version detection: major `1` maps to `'1.0.0'` and major `2` maps to `'2.0.0'`, so the appropriate parse path still runs.
+
+### `extractStorageHints(item, assetKey?)`
+
+```ts
+function extractStorageHints(item: StacItem, assetKey?: string): StorageHints
+```
+
+Extract connection hints from a STAC Item. Dispatches on the detected Storage Extension version and returns `emptyStorageHints()` when the extension is absent or fails to parse.
+
+**Parameters**
+
+| Name | Type | Meaning |
+|------|------|---------|
+| `item` | `StacItem` | The STAC Item to parse. |
+| `assetKey` | `string` (optional) | Scope the lookup to a specific asset. |
+
+**`assetKey` behavior**
+
+- **v1**: when given, that asset's `storage:*` overrides take precedence over item-level fields. When omitted, only item-level fields are read.
+- **v2**: when given, that asset's `storage:refs[0]` (first ref present on the asset) resolves the item scheme. When omitted, the first scheme reachable from any asset's refs wins.
+
+**Returns** `StorageHints`.
+
+**v2 endpoint resolution**
+
+The `endpoint` field is only populated for a scheme whose `type` is `'custom-s3'`. The parser prefers the scheme's explicit `endpoint` string, then falls back to its `platform` string, but only treats either as a concrete endpoint when it contains no URI-template variable (no `{`, e.g. `{region}`). Templated values are left as `null` so the caller does not wire a non-resolvable URL.
+
+### `applyStorageHintsToConnection(conn, hints)`
+
+```ts
+function applyStorageHintsToConnection<T extends { region?: string; endpoint?: string }>(
+  conn: T,
+  hints: StorageHints
+): T
+```
+
+Merge `region` and `endpoint` hints into a connection-shaped object, filling each field only when the existing value is empty. Returns a shallow copy. The generic accepts any object with optional `region` / `endpoint` keys, so it stays decoupled from the concrete `Connection` type.
+
+**Notes**
+
+- Only `region` and `endpoint` are merged. `platform` and `requesterPays` are not written by this helper.
+- An existing non-empty `region` or `endpoint` on `conn` is never overwritten.
+- The input object is not mutated; the returned value is a new object.
+
+## Example
+
+```ts
+import {
+  detectStorageExtensionVersion,
+  extractStorageHints,
+  applyStorageHintsToConnection,
+} from '@walkthru-earth/objex-utils';
+
+const version = detectStorageExtensionVersion(item); // '1.0.0' | '2.0.0' | null
+
+const hints = extractStorageHints(item, 'visual');
+// {
+//   platform: 'AWS',
+//   region: 'us-west-2',
+//   requesterPays: false,
+//   endpoint: null
+// }
+
+// Pre-fill a new connection without clobbering anything the user already set.
+const conn = applyStorageHintsToConnection(
+  { region: '', endpoint: '' },
+  hints
+);
+// { region: 'us-west-2', endpoint: '' }
+```