flatsql 3.11.1 → 3.12.0

package/docs/agents/response-artifacts.md

@@ -1,65 +0,0 @@
-# Agent Guide: Response Bandwidth
-
-Use this guide when modifying FlatSQL response caching, SDS stress tests, or Space Data Network FILE_ID behavior.
-
-## Rules
-
-- Do not add FlatSQL-specific fields or annotations to canonical SDS schemas.
-- Treat `FILE_ID` as a Space Data Network publish-event policy key.
-- Use runtime indexes or external SDN index profiles for lookup speed.
-- Measure response bytes separately from query latency and cache hits.
-- Prefer raw FlatBuffer artifacts for repeated high-volume responses.
-- Tune native query-template cache limits with `configureQueryCache({ maxEntries, maxRows })` when hot FILE_ID traffic needs more than the defaults.
-
-## Library Responsibilities
-
-FlatSQL response bandwidth work belongs in these primitives:
-
-- native `buildResponseArtifactCacheKey(...)` bindings for deterministic query/result cache keys
-- `createQueryResponseArtifact(...)` for immutable byte artifacts
-- `getResponseArtifactChunk(...)` for range-addressable chunk reads
-- `MemoryResponseArtifactCache.getOrCreateByKey(...)` for bounded process-local artifact reuse with native keys
-
-Deployment code is responsible for CDN, browser Cache API, WasmEdge host filesystem cache, auth, quotas, routing, and regional replication.
-
-## Required Tests
-
-When changing response artifacts, run:
-
-```bash
-npm test -- --runInBand --runTestsByPath test/response-artifact.test.ts
-npm test -- --runInBand --runTestsByPath test/wasm-query-params.test.ts test/standalone-shim.test.ts test/standalone-artifact-builder.test.ts test/wasmedge-runner.test.ts
-npm test -- --runInBand --runTestsByPath test/stress-harness.test.ts
-npm run build
-```
-
-Before claiming production readiness, also run:
-
-```bash
-npm test -- --runInBand
-npm run stress:sds:full-config -- --run-id sds-full-config-check
-docker compose -f stress/results/sds-full-config-check/docker-compose.full.yml config
-git diff --check
-```
-
-## SDS Stress Interpretation
-
-The SDS harness reports measured events. A cache hit is not enough. If `responseBytes` remains high, repeated clients are still consuming bandwidth unless the response artifact is served from edge/browser/WasmEdge host cache.
-
-For hot publish events, look at:
-
-- `hot-file-id-cache.responseBytes`
-- `raw-flatbuffer-retrieval.responseBytes`
-- `sql-projection-query.responseBytes`
-- `bulk-streaming-ingest.requestBytes`
-- aggregate `ingestRecordsPerSecond` and `minNodeIngestRecordsPerSecond`; every node must stay at or above 75 fps for production readiness
-- `bandwidth-constrained-streaming.requestBytes`
-- per-node `storageBytes` and storage fill percentage; every node must reach at least 95% of its configured storage target
-
-## Common Mistakes
-
-- Calling a run successful because query latency improved while response bytes remain unchanged.
-- Requiring fields like `file_identifier`, `indexed`, `key`, or `id` in SDS schemas.
-- Returning JSON for large repeated query results when raw FlatBuffer bytes are acceptable.
-- Building a process-local cache and assuming it solves 2M-user egress.
-- Omitting schema version, projection, format, or publish-event key from a cache key.

package/docs/response-artifacts.md

@@ -1,86 +0,0 @@
-# Response Artifacts
-
-FlatSQL reduces response bandwidth by producing cacheable response artifacts. The library owns deterministic byte output, cache keys, hashes, ETags, and chunk metadata. Deployment infrastructure owns CDN placement, global invalidation, routing, authorization, and quotas.
-
-## Boundary
-
-FlatSQL should do:
-
-- build deterministic cache keys in the C++/WASM core from query shape, schema version, format, projection, params, and publish-event key
-- encode raw FlatBuffer query results as size-prefixed byte streams
-- encode JSON results only when callers explicitly request JSON
-- produce content hashes and ETags for immutable artifacts
-- split artifacts into range-addressable chunks
-- expose small process-local caches for repeated query artifacts
-
-FlatSQL should not do:
-
-- add database-specific annotations to canonical SDS schemas
-- implement a CDN or global cache-invalidation service
-- decide authorization or tenant-specific cache policy
-- route users to regions or nodes
-- hide response byte cost behind a query-cache hit counter
-
-## Recommended FILE_ID Flow
-
-For Space Data Network publish events, `FILE_ID` is a network policy key, not a schema requirement.
-
-1. Ingest records for a publish event.
-2. Apply `FILE_ID` indexes through runtime policy or SDN index profiles.
-3. Query the publish event with a stable projection and format.
-4. Create a response artifact.
-5. Serve the artifact by `ETag` or content hash from edge, browser, WasmEdge host storage, or a CDN.
-6. Recompute only when the publish event, schema version, projection, or format changes.
-
-```ts
-import {
-  createQueryResponseArtifact,
-  MemoryResponseArtifactCache,
-} from 'flatsql/response';
-import initFlatSQL from 'flatsql/wasm';
-
-const flatsql = await initFlatSQL();
-const sql = "SELECT _data FROM PublishEventRecord WHERE FILE_ID = ?";
-const cacheKey = flatsql.buildResponseArtifactCacheKey('PublishEventRecord', '1', sql, {
-  format: 'raw-flatbuffer-stream',
-  publishEventKey: 'publish-1',
-  projection: ['_data'],
-  params: ['publish-1'],
-});
-const result = db.query(sql);
-const cache = new MemoryResponseArtifactCache();
-
-const artifact = cache.getOrCreateByKey(cacheKey, () => createQueryResponseArtifact(result, {
-  schemaName: 'PublishEventRecord',
-  schemaVersion: 1,
-  sql,
-  params: ['publish-1'],
-  format: 'raw-flatbuffer-stream',
-  publishEventKey: 'publish-1',
-  projection: ['_data'],
-  cacheKey,
-  chunkBytes: 1024 * 1024,
-}));
-
-console.log(cacheKey);
-console.log(artifact.metadata.etag);
-console.log(artifact.metadata.chunks);
-```
-
-## Production Checks
-
-Before relying on response artifacts in production:
-
-- verify `cacheKey` includes every dimension that can change bytes
-- verify browser and WasmEdge runtimes produce the same native response cache key
-- prefer `raw-flatbuffer-stream` for high-volume SDS responses
-- verify `responseBytes` and `flatbufferBytes`, not only query latency
-- tune native `configureQueryCache({ maxEntries, maxRows })` limits for hot FILE_ID query templates
-- set chunk sizes based on edge/cache/range-request behavior
-- cap in-process caches with `maxEntries` and `maxBytes`
-- keep authorization outside shared immutable artifacts unless all callers are allowed to read identical bytes
-- test hot `FILE_ID` traffic separately from cold fanout
-
-## Runtime Notes
-
-Browser deployments should call the native `buildResponseArtifactCacheKey(...)` binding through `flatsql/wasm` or `flatsql/standalone`, then map artifact metadata to HTTP cache headers and the browser Cache API where appropriate. WasmEdge deployments should call the same binding through `flatsql/artifacts/standalone` and map artifacts to host filesystem or edge-local object cache. FlatSQL response artifacts are intentionally plain bytes plus metadata so either runtime can use the same cache key and ETag semantics.

package/scripts/stress-docker.sh

@@ -1,279 +0,0 @@
-#!/usr/bin/env bash
-
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
-
-COMMAND="smoke"
-if [[ $# -gt 0 ]]; then
-  COMMAND="$1"
-  shift
-fi
-
-RUN_ID="${RUN_ID:-sds-$(date -u +%Y%m%dT%H%M%SZ)}"
-SDS_SCHEMA_ROOT="${SDS_SCHEMA_ROOT:-$PROJECT_ROOT/../spacedatastandards.org/schema}"
-if [[ -n "${OUTPUT_DIR:-}" ]]; then
-  OUTPUT_DIR_EXPLICIT=1
-else
-  OUTPUT_DIR_EXPLICIT=0
-  OUTPUT_DIR="$PROJECT_ROOT/stress/results/$RUN_ID"
-fi
-
-EXTRA_RUNNER_ARGS=()
-while [[ $# -gt 0 ]]; do
-  case "$1" in
-    --schema-root)
-      SDS_SCHEMA_ROOT="$2"
-      shift 2
-      ;;
-    --schema-root=*)
-      SDS_SCHEMA_ROOT="${1#*=}"
-      shift
-      ;;
-    --output-dir)
-      OUTPUT_DIR="$2"
-      OUTPUT_DIR_EXPLICIT=1
-      shift 2
-      ;;
-    --output-dir=*)
-      OUTPUT_DIR="${1#*=}"
-      OUTPUT_DIR_EXPLICIT=1
-      shift
-      ;;
-    --nodes)
-      NODE_COUNT="$2"
-      shift 2
-      ;;
-    --nodes=*)
-      NODE_COUNT="${1#*=}"
-      shift
-      ;;
-    --node-id-offset)
-      NODE_ID_OFFSET="$2"
-      shift 2
-      ;;
-    --node-id-offset=*)
-      NODE_ID_OFFSET="${1#*=}"
-      shift
-      ;;
-    --storage-gb)
-      NODE_STORAGE_GB="$2"
-      shift 2
-      ;;
-    --storage-gb=*)
-      NODE_STORAGE_GB="${1#*=}"
-      shift
-      ;;
-    --records-per-node)
-      RECORDS_PER_NODE="$2"
-      shift 2
-      ;;
-    --records-per-node=*)
-      RECORDS_PER_NODE="${1#*=}"
-      shift
-      ;;
-    --batch-bytes)
-      BATCH_BYTES="$2"
-      shift 2
-      ;;
-    --batch-bytes=*)
-      BATCH_BYTES="${1#*=}"
-      shift
-      ;;
-    --query-concurrency)
-      QUERY_CONCURRENCY="$2"
-      shift 2
-      ;;
-    --query-concurrency=*)
-      QUERY_CONCURRENCY="${1#*=}"
-      shift
-      ;;
-    --hot-query-ratio)
-      HOT_QUERY_RATIO="$2"
-      shift 2
-      ;;
-    --hot-query-ratio=*)
-      HOT_QUERY_RATIO="${1#*=}"
-      shift
-      ;;
-    --runtime)
-      RUNTIME="$2"
-      shift 2
-      ;;
-    --runtime=*)
-      RUNTIME="${1#*=}"
-      shift
-      ;;
-    --node-isolation)
-      NODE_ISOLATION="$2"
-      shift 2
-      ;;
-    --node-isolation=*)
-      NODE_ISOLATION="${1#*=}"
-      shift
-      ;;
-    --run-id)
-      RUN_ID="$2"
-      shift 2
-      ;;
-    --run-id=*)
-      RUN_ID="${1#*=}"
-      shift
-      ;;
-    *)
-      EXTRA_RUNNER_ARGS+=("$1")
-      shift
-      ;;
-  esac
-done
-if [[ "$OUTPUT_DIR_EXPLICIT" -eq 0 ]]; then
-  OUTPUT_DIR="$PROJECT_ROOT/stress/results/$RUN_ID"
-fi
-if [[ "${#EXTRA_RUNNER_ARGS[@]}" -gt 0 ]]; then
-  set -- "${EXTRA_RUNNER_ARGS[@]}"
-else
-  set --
-fi
-
-run_local_smoke() {
-  local runner_args=(
-    --mode "${STRESS_MODE:-smoke}"
-    --schema-root "$SDS_SCHEMA_ROOT"
-    --output-dir "$OUTPUT_DIR"
-    --nodes "${NODE_COUNT:-2}"
-    --storage-gb "${NODE_STORAGE_GB:-0.01}"
-    --records-per-node "${RECORDS_PER_NODE:-25}"
-    --run-id "$RUN_ID"
-  )
-  if [[ -n "${NODE_ID_OFFSET:-}" ]]; then runner_args+=(--node-id-offset "$NODE_ID_OFFSET"); fi
-  if [[ -n "${BATCH_BYTES:-}" ]]; then runner_args+=(--batch-bytes "$BATCH_BYTES"); fi
-  if [[ -n "${QUERY_CONCURRENCY:-}" ]]; then runner_args+=(--query-concurrency "$QUERY_CONCURRENCY"); fi
-  if [[ -n "${HOT_QUERY_RATIO:-}" ]]; then runner_args+=(--hot-query-ratio "$HOT_QUERY_RATIO"); fi
-  if [[ -n "${RUNTIME:-}" ]]; then runner_args+=(--runtime "$RUNTIME"); fi
-  if [[ -n "${NODE_ISOLATION:-}" ]]; then runner_args+=(--node-isolation "$NODE_ISOLATION"); fi
-  if [[ "$#" -gt 0 ]]; then runner_args+=("$@"); fi
-
-  npm run build
-  node "$PROJECT_ROOT/stress/sds/run.mjs" "${runner_args[@]}"
-}
-
-run_docker() {
-  docker info >/dev/null
-  local schema_root_abs
-  schema_root_abs="$(cd "$SDS_SCHEMA_ROOT" && pwd)"
-  mkdir -p "$OUTPUT_DIR"
-  local output_dir_abs
-  output_dir_abs="$(cd "$OUTPUT_DIR" && pwd)"
-  SDS_SCHEMA_ROOT_HOST="$schema_root_abs" \
-  OUTPUT_DIR_HOST="$output_dir_abs" \
-  RUN_ID="$RUN_ID" \
-  STRESS_MODE="${STRESS_MODE:-smoke}" \
-  NODE_COUNT="${NODE_COUNT:-2}" \
-  NODE_STORAGE_GB="${NODE_STORAGE_GB:-0.01}" \
-  RECORDS_PER_NODE="${RECORDS_PER_NODE:-25}" \
-  BATCH_BYTES="${BATCH_BYTES:-1048576}" \
-  QUERY_CONCURRENCY="${QUERY_CONCURRENCY:-4}" \
-  HOT_QUERY_RATIO="${HOT_QUERY_RATIO:-0.9}" \
-  RUNTIME="${RUNTIME:-standalone}" \
-  NODE_ISOLATION="${NODE_ISOLATION:-logical}" \
-  docker compose -f "$PROJECT_ROOT/stress/docker/docker-compose.yml" up \
-    --build \
-    --abort-on-container-exit \
-    --exit-code-from controller \
-    controller
-}
-
-prepare_container_fleet_compose() {
-  npm run build
-
-  local schema_root_abs
-  schema_root_abs="$(cd "$SDS_SCHEMA_ROOT" && pwd)"
-  mkdir -p "$OUTPUT_DIR"
-  local output_dir_abs
-  output_dir_abs="$(cd "$OUTPUT_DIR" && pwd)"
-  FLEET_COMPOSE_FILE="$output_dir_abs/docker-compose.full.yml"
-
-  node "$PROJECT_ROOT/stress/sds/write-compose.mjs" \
-    --output-file "$FLEET_COMPOSE_FILE" \
-    --project-root "$PROJECT_ROOT" \
-    --schema-root-host "$schema_root_abs" \
-    --output-dir-host "$output_dir_abs" \
-    --run-id "$RUN_ID" \
-    --mode "${STRESS_MODE:-smoke}" \
-    --nodes "${NODE_COUNT:-2}" \
-    --storage-gb "${NODE_STORAGE_GB:-0.01}" \
-    --records-per-node "${RECORDS_PER_NODE:-25}" \
-    --batch-bytes "${BATCH_BYTES:-1048576}" \
-    --query-concurrency "${QUERY_CONCURRENCY:-4}" \
-    --hot-query-ratio "${HOT_QUERY_RATIO:-0.9}" \
-    --runtime "${RUNTIME:-standalone}"
-}
-
-run_container_fleet() {
-  prepare_container_fleet_compose
-  docker info >/dev/null
-
-  local node_count="${NODE_COUNT:-2}"
-  local worker_services=()
-  local node_id
-  for ((node_id = 0; node_id < node_count; node_id++)); do
-    worker_services+=("worker-$node_id")
-  done
-
-  docker compose -f "$FLEET_COMPOSE_FILE" up --build "${worker_services[@]}"
-  docker compose -f "$FLEET_COMPOSE_FILE" build aggregator
-  docker compose -f "$FLEET_COMPOSE_FILE" run --no-deps --rm aggregator
-}
-
-run_full() {
-  if [[ "${CONFIRM_FULL_STRESS:-}" != "1" ]]; then
-    echo "Full stress requires CONFIRM_FULL_STRESS=1 because it targets 100 nodes and 1 GB/node." >&2
-    exit 2
-  fi
-
-  local parent
-  parent="$(dirname "$OUTPUT_DIR")"
-  mkdir -p "$parent"
-  local available_kb
-  available_kb="$(df -Pk "$parent" | awk 'NR == 2 { print $4 }')"
-  local required_kb=$((110 * 1024 * 1024))
-  if [[ "$available_kb" -lt "$required_kb" ]]; then
-    echo "Full stress requires at least 110 GB free near $parent; available KB: $available_kb." >&2
-    exit 3
-  fi
-
-  RECORDS_PER_NODE="${RECORDS_PER_NODE:-20000}"
-  QUERY_CONCURRENCY="${QUERY_CONCURRENCY:-64}"
-  HOT_QUERY_RATIO="${HOT_QUERY_RATIO:-0.01}"
-  STRESS_MODE=full NODE_COUNT=100 NODE_STORAGE_GB=1 NODE_ISOLATION=container run_container_fleet
-}
-
-case "$COMMAND" in
-  smoke)
-    run_local_smoke "$@"
-    ;;
-  docker-smoke)
-    run_docker "$@"
-    ;;
-  docker-fleet)
-    STRESS_MODE="${STRESS_MODE:-smoke}" NODE_ISOLATION=container run_container_fleet "$@"
-    ;;
-  full-config)
-    RECORDS_PER_NODE="${RECORDS_PER_NODE:-20000}" \
-      QUERY_CONCURRENCY="${QUERY_CONCURRENCY:-64}" \
-      HOT_QUERY_RATIO="${HOT_QUERY_RATIO:-0.01}" \
-      STRESS_MODE=full \
-      NODE_COUNT="${NODE_COUNT:-100}" \
-      NODE_STORAGE_GB="${NODE_STORAGE_GB:-1}" \
-      NODE_ISOLATION=container \
-      prepare_container_fleet_compose
-    ;;
-  full)
-    run_full "$@"
-    ;;
-  *)
-    echo "Usage: scripts/stress-docker.sh [smoke|docker-smoke|docker-fleet|full-config|full] [runner args]" >&2
-    exit 2
-    ;;
-esac

package/sdm/index.d.ts

@@ -1,88 +0,0 @@
-/**
- * FlatSQL Spatial Engine — Space Data Module (SDM)
- * TypeScript type definitions
- *
- * @see https://github.com/DigitalArsenal/space-data-module-sdk
- */
-
-export interface ECEFPoint {
-  x: number;
-  y: number;
-  z: number;
-}
-
-export interface GeodeticPoint {
-  lat: number;
-  lon: number;
-  alt: number;
-}
-
-export interface MethodResult {
-  status: number;
-  output: string;
-}
-
-export interface PluginManifest {
-  pluginId: string;
-  name: string;
-  version: string;
-  pluginFamily: string;
-  description: string;
-  capabilities: string[];
-  externalInterfaces: string[];
-  methods: Array<{
-    methodId: string;
-    description: string;
-    inputPorts: Array<{
-      portId: string;
-      schemaNames: string[];
-      maxItems: number;
-    }>;
-    outputPorts: Array<{
-      portId: string;
-      schemaNames: string[];
-      maxItems: number;
-    }>;
-    maxBatch: number;
-    drainPolicy: string;
-  }>;
-  runtimeTargets: string[];
-}
-
-export interface FlatSQLSpatialSDM {
-  /** Get the embedded plugin manifest */
-  getManifest(): PluginManifest;
-
-  /** Compute haversine distance between two WGS84 points (km) */
-  computeDistance(lat1: number, lon1: number, lat2: number, lon2: number): number;
-
-  /** Compute forward azimuth between two WGS84 points (degrees 0-360) */
-  computeBearing(lat1: number, lon1: number, lat2: number, lon2: number): number;
-
-  /** Test if a WGS84 point is inside a WKT polygon */
-  pointInPolygon(wkt: string, lat: number, lon: number): boolean;
-
-  /** Compute intersection of two WKT polygons */
-  polygonIntersection(wkt1: string, wkt2: string): string;
-
-  /** Compute union of two WKT polygons */
-  polygonUnion(wkt1: string, wkt2: string): string;
-
-  /** Compute Voronoi diagram from multipoint WKT within bounds */
-  computeVoronoi(multipointWkt: string, boundsWkt: string): string;
-
-  /** Compute Delaunay triangulation from multipoint WKT */
-  computeDelaunay(multipointWkt: string): string;
-
-  /** Transform WGS84 geodetic to ECEF */
-  toECEF(lat: number, lon: number, alt: number): ECEFPoint;
-
-  /** Transform ECEF to WGS84 geodetic */
-  fromECEF(x: number, y: number, z: number): GeodeticPoint;
-
-  /** Low-level method call */
-  callMethod(methodName: string, input: string): MethodResult;
-}
-
-/** Initialize the FlatSQL Spatial SDM module */
-export function initSpatialSDM(): Promise<FlatSQLSpatialSDM>;