flatsql 0.4.2 → 1.0.0

package/README.md

@@ -75,11 +75,118 @@ FlatBuffer → Query (via virtual table) → FlatBuffer
 ## Runtime Choices & Performance Gates
 
 - Production should prefer the SQLite-backed native/WASM path (`initFlatSQL` → `FlatSQL` → `createDatabase`). The pure TypeScript `FlatSQLDatabase` is preserved only as an explicit fallback/reference implementation for environments that cannot load the WASM module.
+- Standalone deployments use the C++ WASI reactor at `flatsql/wasi.wasm`. Browser hosts load it through `flatsql/standalone`, while Node/WasmEdge hosts use a resident runner from `flatsql/standalone/wasmedge`.
+- Artifact-builder cache behavior belongs to C++: query templates, result-cache keys, invalidation generation, hit/miss counters, SQL execution, and raw FlatBuffer lookup all live in the WASM database instance. JavaScript only marshals bytes and fills host capability gaps.
 - Use `db.ingestBuffers([...])` on the WASM path when your input is already a set of raw FlatBuffers. It feeds the native bulk-stream ingest path instead of looping through `ingestOne(...)`.
 - Run `npm run bench` (alias `npm run bench:perf`) as the single-command benchmark matrix for FlatSQL JS vs FlatSQL WASM. Use `npm run bench:perf:profile` to print the WASM ingest phase breakdown (`pack`, `decode`, `append`, `index`, `verify`) alongside the gate table.
   Profiling mode is diagnostic and includes instrumentation overhead; use the non-profile benchmark for merge-gate decisions.
 - Run `npm run test:cluster` for the native WAL-backed cluster validation workload, or `npm run test:cluster:smoke` for the short preflight run.
 
+## SDS Stress Harness
+
+The Space Data Standards stress harness exercises every discoverable `main.fbs` under `../spacedatastandards.org/schema`, records bandwidth and storage-fill metrics, and keeps canonical schemas database-neutral. Results are written under `stress/results/<run-id>/`.
+
+```bash
+# Local smoke path, no Docker required.
+npm run stress:sds:smoke -- --run-id sds-smoke
+
+# Existing single-controller Docker smoke path.
+npm run stress:sds:docker-smoke -- --run-id sds-docker-smoke
+
+# Container-per-node fleet path for small probes, e.g. 10 isolated nodes.
+npm run stress:sds:docker-fleet -- --run-id sds-10-node --nodes 10 --storage-gb 1 --records-per-node 100000
+
+# Generate and validate the 100-node / 1 GB-per-node production compose file without running it.
+npm run stress:sds:full-config -- --run-id sds-full
+docker compose -f stress/results/sds-full/docker-compose.full.yml config
+
+# Run the full production gate. Requires Docker and enough local storage.
+CONFIRM_FULL_STRESS=1 npm run stress:sds:full -- --run-id sds-full
+```
+
+The production gate only passes when the aggregate report shows container isolation, Docker transport, at least 100 nodes, no metric errors, measured native/standalone/WasmEdge events, complete SDS schema-sweep coverage, at least 95% storage fill against the 1 GB/node target on every node, and bulk-ingest throughput at or above 75 fps on every node.
+
+Full mode defaults to 20,000 records per node, or 2 million records across the 100-node fleet. In full mode the generated PublishEventRecord payload size scales from the configured storage budget so the run targets the requested per-node storage fill without requiring hundreds of millions of tiny records.
+
+## Agent Runtime Notes
+
+Use these entrypoints when wiring agents or deployments:
+
+| Import | Runtime | Use for |
+|--------|---------|---------|
+| `flatsql/wasm` | Emscripten JS wrapper + C++ WASM | Existing browser/Node API |
+| `flatsql/standalone` | Raw `flatsql-wasi.wasm` + small JS WASI shim | Browser or Node direct WebAssembly API |
+| `flatsql/artifacts/standalone` | C++ artifact builder facade | Cache-centric artifact workflows |
+| `flatsql/response` | Response artifact bytes | ETags, chunks, and raw byte artifacts keyed by native runtime cache keys |
+| `flatsql/standalone/wasmedge` | Native WasmEdge runner builder/client | Persistent standalone runtime in Node |
+| `flatsql` `FlatSQLDatabase` | Pure TypeScript fallback | Reference or no-WASM environments only |
+
+For repeated FILE_ID-style traffic, register a cacheable query template once and call `queryTemplate(...)` with positional params. Identical query IDs and params hit the C++ result cache; ingest, load, template changes, and explicit `clearQueryCache()` invalidate stale entries by generation.
+
+```javascript
+import { createStandaloneArtifactBuilder } from 'flatsql/artifacts/standalone';
+
+const builder = await createStandaloneArtifactBuilder(schema, { runtime: 'standalone' });
+await builder.registerFileId('USER', 'User');
+await builder.enableDemoExtractors();
+await builder.ingestBuffers(buffers);
+await builder.configureQueryCache({ maxEntries: 4096, maxRows: 5000 });
+await builder.registerQueryTemplate('userByEmail', 'SELECT id FROM User WHERE email = ?', true);
+
+await builder.queryTemplate('userByEmail', ['alice@example.com']); // miss
+await builder.queryTemplate('userByEmail', ['alice@example.com']); // hit
+console.log(await builder.getQueryCacheStats());
+```
+
+The native cache defaults to 1024 entries and 1000 rows per cached result. Tune `maxEntries` and `maxRows` for production FILE_ID traffic instead of recompiling the C++ core.
+
+For WasmEdge, build the small host runner and pass it to the same builder. The runner keeps one WASI module instance resident so cache state survives across requests.
+
+```javascript
+import { createStandaloneArtifactBuilder } from 'flatsql/artifacts/standalone';
+import { buildFlatSQLWasmEdgeRunner } from 'flatsql/standalone/wasmedge';
+
+const runner = await buildFlatSQLWasmEdgeRunner({
+  outputPath: './build/flatsql-wasmedge-runner',
+});
+
+const builder = await createStandaloneArtifactBuilder(schema, {
+  runtime: 'wasmedge',
+  wasmEdgeRunnerBinary: runner.outputPath,
+});
+```
+
+WasmEdge must support WebAssembly exception handling for this artifact. The test runner uses `wasmedge --enable-exception-handling --reactor wasm/flatsql-wasi.wasm _initialize` for the smoke check and the native C API runner enables the same proposal programmatically. Some Node 24 builds require `node --experimental-wasm-exnref` when directly instantiating the standalone WASI artifact.
+
+For high-volume repeated FILE_ID responses, build the cache key through the C++ runtime, wrap query results as response artifacts, and let edge/browser/WasmEdge host caches serve the immutable bytes by ETag or content hash. FlatSQL creates deterministic native cache keys and chunk metadata; deployment code owns CDN placement, routing, authorization, and global invalidation.
+
+```javascript
+import { createQueryResponseArtifact } from 'flatsql/response';
+
+const sql = 'SELECT _data FROM PublishEventRecord WHERE FILE_ID = ?';
+const cacheKey = await builder.buildResponseArtifactCacheKey('PublishEventRecord', '1', sql, {
+  format: 'raw-flatbuffer-stream',
+  publishEventKey: 'publish-1',
+  projection: ['_data'],
+  params: ['publish-1'],
+});
+const result = await builder.query(sql, ['publish-1']);
+const artifact = createQueryResponseArtifact(result, {
+  schemaName: 'PublishEventRecord',
+  schemaVersion: 1,
+  sql,
+  params: ['publish-1'],
+  format: 'raw-flatbuffer-stream',
+  publishEventKey: 'publish-1',
+  projection: ['_data'],
+  cacheKey,
+});
+
+console.log(artifact.metadata.cacheKey, artifact.metadata.etag);
+```
+
+See `docs/response-artifacts.md` and `docs/agents/response-artifacts.md` for production and agent guidance.
+
 
 ## Source Code
 
@@ -457,7 +564,7 @@ The index stores `(key, sequence) → (offset, length)` mappings, allowing O(log
 
 ### Prerequisites
 
-- Node.js 18+
+- Node.js 24+
 - CMake 3.20+ (for WASM builds)
 - Emscripten (for WASM builds)
 
@@ -475,13 +582,16 @@ npm test
 # Clone DA-FlatBuffers (required dependency)
 git clone https://github.com/DigitalArsenal/flatbuffers.git ../flatbuffers
 
-# Build WASM
-cd cpp
-emcmake cmake -B build-wasm -DCMAKE_BUILD_TYPE=Release
-cmake --build build-wasm
+# Build browser, standalone WASI, and SDM artifacts
+npm run build:wasm
+
+# Verify standalone and package surfaces
+npm run test:wasi
+npm run test:wasmedge
+npm test -- --runInBand --runTestsByPath test/package-exports.test.ts
 ```
 
-Output: `wasm/flatsql.js` and `wasm/flatsql.wasm`
+Output: `wasm/flatsql.js`, `wasm/flatsql.wasm`, `wasm/flatsql-wasi.wasm`, `sdm/flatsql-spatial.wasm`, and docs copies for GitHub Pages.
 
 ### Run Demo Locally
 

package/dist/artifacts/builder.d.ts

@@ -0,0 +1,31 @@
+import { type DatabaseSchema } from '../schema/index.js';
+import { type ArtifactFieldExtractor } from './demo-extractors.js';
+import type { ArtifactBuilderOptions, ArtifactIngestOptions, ArtifactQueryParams, ArtifactQuerySpec, ArtifactIngestResult, ArtifactQueryResult } from './types.js';
+export declare class FlatSQLArtifactBuilder {
+    private readonly schema;
+    private readonly db;
+    private readonly beginTransactionSql;
+    private readonly tableByName;
+    private readonly fileIdToTable;
+    private readonly extractors;
+    private readonly queryCache;
+    private readonly queryResultCache;
+    private sequence;
+    static fromSchema(source: string, options: ArtifactBuilderOptions): FlatSQLArtifactBuilder;
+    constructor(schema: DatabaseSchema, options: ArtifactBuilderOptions);
+    registerFileId(fileId: string, tableName: string): void;
+    setFieldExtractor(tableName: string, extractor: ArtifactFieldExtractor): void;
+    enableDemoExtractors(): void;
+    ingestBuffers(buffers: Uint8Array[], options?: ArtifactIngestOptions): ArtifactIngestResult;
+    query(sql: string, params?: ArtifactQueryParams): ArtifactQueryResult;
+    queryMany(queries: readonly ArtifactQuerySpec[]): ArtifactQueryResult[];
+    close(): void;
+    private createIndexTables;
+    private indexTableName;
+    private recordTableName;
+    private coveringIndexName;
+    private createCoveringIndexes;
+    private dropCoveringIndexes;
+    private createIngestPlan;
+}
+//# sourceMappingURL=builder.d.ts.map

package/dist/artifacts/builder.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"builder.d.ts","sourceRoot":"","sources":["../../src/artifacts/builder.ts"],"names":[],"mappings":"AAGA,OAAO,EAIL,KAAK,cAAc,EAEpB,MAAM,oBAAoB,CAAC;AAC5B,OAAO,EAIL,KAAK,sBAAsB,EAC5B,MAAM,sBAAsB,CAAC;AAC9B,OAAO,KAAK,EACV,sBAAsB,EACtB,qBAAqB,EACrB,mBAAmB,EACnB,iBAAiB,EACjB,oBAAoB,EAEpB,mBAAmB,EACpB,MAAM,YAAY,CAAC;AAyYpB,qBAAa,sBAAsB;IACjC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAiB;IACxC,OAAO,CAAC,QAAQ,CAAC,EAAE,CAAe;IAClC,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAS;IAC7C,OAAO,CAAC,QAAQ,CAAC,WAAW,CAA+B;IAC3D,OAAO,CAAC,QAAQ,CAAC,aAAa,CAA6B;IAC3D,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA6C;IACxE,OAAO,CAAC,QAAQ,CAAC,UAAU,CAItB;IACL,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAA0C;IAC3E,OAAO,CAAC,QAAQ,CAAK;IAErB,MAAM,CAAC,UAAU,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,sBAAsB,GAAG,sBAAsB;gBAW9E,MAAM,EAAE,cAAc,EAAE,OAAO,EAAE,sBAAsB;IAcnE,cAAc,CAAC,MAAM,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,GAAG,IAAI;IAIvD,iBAAiB,CAAC,SAAS,EAAE,MAAM,EAAE,SAAS,EAAE,sBAAsB,GAAG,IAAI;IAI7E,oBAAoB,IAAI,IAAI;IAM5B,aAAa,CAAC,OAAO,EAAE,UAAU,EAAE,EAAE,OAAO,GAAE,qBAA0B,GAAG,oBAAoB;IA6C/F,KAAK,CAAC,GAAG,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,mBAAmB,GAAG,mBAAmB;IAyCrE,SAAS,CAAC,OAAO,EAAE,SAAS,iBAAiB,EAAE,GAAG,mBAAmB,EAAE;IAIvE,KAAK,IAAI,IAAI;IAMb,OAAO,CAAC,iBAAiB;IA6BzB,OAAO,CAAC,cAAc;IAItB,OAAO,CAAC,eAAe;IAIvB,OAAO,CAAC,iBAAiB;IAIzB,OAAO,CAAC,qBAAqB;IAa7B,OAAO,CAAC,mBAAmB;IAU3B,OAAO,CAAC,gBAAgB;CAyCzB"}