flatsql 3.11.0 → 3.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +61 -795
- package/bin/flatsql.mjs +76 -0
- package/cpp/README.md +18 -0
- package/cpp/include/flatsql/flatsql.h +162 -0
- package/cpp/include/flatsql/flatsql_io.h +159 -0
- package/docs/API-3.12.md +114 -0
- package/package.json +28 -123
- package/wasm/flatsql.wasm +0 -0
- package/wasm/index.d.ts +61 -548
- package/wasm/index.js +280 -1281
- package/wasm/integrity.json +6 -26
- package/wasm/runtime/LICENSE.sdk.txt +190 -0
- package/wasm/runtime/NOTICE.txt +2 -0
- package/wasm/runtime/browser-guest.mjs +1 -0
- package/wasm/runtime/browser-host.mjs +1 -0
- package/wasm/runtime/node-guest.mjs +2 -0
- package/wasm/runtime/node-host.mjs +5 -0
- package/wasm/runtime/opfsIoWorker.mjs +1 -0
- package/wasm/runtime/wasiThreadBrowserWorker.mjs +1 -0
- package/dist/artifacts/index.d.ts +0 -3
- package/dist/artifacts/index.d.ts.map +0 -1
- package/dist/artifacts/index.js +0 -3
- package/dist/artifacts/index.js.map +0 -1
- package/dist/artifacts/transport.d.ts +0 -5
- package/dist/artifacts/transport.d.ts.map +0 -1
- package/dist/artifacts/transport.js +0 -45
- package/dist/artifacts/transport.js.map +0 -1
- package/dist/artifacts/types.d.ts +0 -34
- package/dist/artifacts/types.d.ts.map +0 -1
- package/dist/artifacts/types.js +0 -2
- package/dist/artifacts/types.js.map +0 -1
- package/dist/btree/btree.d.ts +0 -34
- package/dist/btree/btree.d.ts.map +0 -1
- package/dist/btree/btree.js +0 -234
- package/dist/btree/btree.js.map +0 -1
- package/dist/btree/index.d.ts +0 -3
- package/dist/btree/index.d.ts.map +0 -1
- package/dist/btree/index.js +0 -3
- package/dist/btree/index.js.map +0 -1
- package/dist/btree/types.d.ts +0 -36
- package/dist/btree/types.d.ts.map +0 -1
- package/dist/btree/types.js +0 -55
- package/dist/btree/types.js.map +0 -1
- package/dist/cluster/index.d.ts +0 -11
- package/dist/cluster/index.d.ts.map +0 -1
- package/dist/cluster/index.js +0 -59
- package/dist/cluster/index.js.map +0 -1
- package/dist/response/artifact.d.ts +0 -4
- package/dist/response/artifact.d.ts.map +0 -1
- package/dist/response/artifact.js +0 -102
- package/dist/response/artifact.js.map +0 -1
- package/dist/response/cache-key.d.ts +0 -6
- package/dist/response/cache-key.d.ts.map +0 -1
- package/dist/response/cache-key.js +0 -114
- package/dist/response/cache-key.js.map +0 -1
- package/dist/response/index.d.ts +0 -5
- package/dist/response/index.d.ts.map +0 -1
- package/dist/response/index.js +0 -5
- package/dist/response/index.js.map +0 -1
- package/dist/response/memory-cache.d.ts +0 -17
- package/dist/response/memory-cache.d.ts.map +0 -1
- package/dist/response/memory-cache.js +0 -76
- package/dist/response/memory-cache.js.map +0 -1
- package/dist/response/types.d.ts +0 -57
- package/dist/response/types.d.ts.map +0 -1
- package/dist/response/types.js +0 -2
- package/dist/response/types.js.map +0 -1
- package/dist/schema/index.d.ts +0 -3
- package/dist/schema/index.d.ts.map +0 -1
- package/dist/schema/index.js +0 -3
- package/dist/schema/index.js.map +0 -1
- package/dist/schema/parser.d.ts +0 -25
- package/dist/schema/parser.d.ts.map +0 -1
- package/dist/schema/parser.js +0 -474
- package/dist/schema/parser.js.map +0 -1
- package/dist/schema/types.d.ts +0 -33
- package/dist/schema/types.d.ts.map +0 -1
- package/dist/schema/types.js +0 -29
- package/dist/schema/types.js.map +0 -1
- package/dist/standalone/index.d.ts +0 -3
- package/dist/standalone/index.d.ts.map +0 -1
- package/dist/standalone/index.js +0 -3
- package/dist/standalone/index.js.map +0 -1
- package/dist/standalone/process-client.d.ts +0 -85
- package/dist/standalone/process-client.d.ts.map +0 -1
- package/dist/standalone/process-client.js +0 -537
- package/dist/standalone/process-client.js.map +0 -1
- package/dist/standalone/wasmedge-runner.d.ts +0 -19
- package/dist/standalone/wasmedge-runner.d.ts.map +0 -1
- package/dist/standalone/wasmedge-runner.js +0 -108
- package/dist/standalone/wasmedge-runner.js.map +0 -1
- package/dist/storage/index.d.ts +0 -3
- package/dist/storage/index.d.ts.map +0 -1
- package/dist/storage/index.js +0 -3
- package/dist/storage/index.js.map +0 -1
- package/dist/storage/stacked-flatbuffers.d.ts +0 -38
- package/dist/storage/stacked-flatbuffers.d.ts.map +0 -1
- package/dist/storage/stacked-flatbuffers.js +0 -253
- package/dist/storage/stacked-flatbuffers.js.map +0 -1
- package/dist/storage/types.d.ts +0 -23
- package/dist/storage/types.d.ts.map +0 -1
- package/dist/storage/types.js +0 -27
- package/dist/storage/types.js.map +0 -1
- package/dist/stress/aggregate.d.ts +0 -3
- package/dist/stress/aggregate.d.ts.map +0 -1
- package/dist/stress/aggregate.js +0 -61
- package/dist/stress/aggregate.js.map +0 -1
- package/dist/stress/docker-compose.d.ts +0 -17
- package/dist/stress/docker-compose.d.ts.map +0 -1
- package/dist/stress/docker-compose.js +0 -156
- package/dist/stress/docker-compose.js.map +0 -1
- package/dist/stress/index.d.ts +0 -10
- package/dist/stress/index.d.ts.map +0 -1
- package/dist/stress/index.js +0 -10
- package/dist/stress/index.js.map +0 -1
- package/dist/stress/metrics.d.ts +0 -3
- package/dist/stress/metrics.d.ts.map +0 -1
- package/dist/stress/metrics.js +0 -83
- package/dist/stress/metrics.js.map +0 -1
- package/dist/stress/native-publish-event-workload.d.ts +0 -3
- package/dist/stress/native-publish-event-workload.d.ts.map +0 -1
- package/dist/stress/native-publish-event-workload.js +0 -465
- package/dist/stress/native-publish-event-workload.js.map +0 -1
- package/dist/stress/production-gate.d.ts +0 -4
- package/dist/stress/production-gate.d.ts.map +0 -1
- package/dist/stress/production-gate.js +0 -12
- package/dist/stress/production-gate.js.map +0 -1
- package/dist/stress/report.d.ts +0 -3
- package/dist/stress/report.d.ts.map +0 -1
- package/dist/stress/report.js +0 -174
- package/dist/stress/report.js.map +0 -1
- package/dist/stress/sds-discovery.d.ts +0 -4
- package/dist/stress/sds-discovery.d.ts.map +0 -1
- package/dist/stress/sds-discovery.js +0 -40
- package/dist/stress/sds-discovery.js.map +0 -1
- package/dist/stress/smoke-runner.d.ts +0 -3
- package/dist/stress/smoke-runner.d.ts.map +0 -1
- package/dist/stress/smoke-runner.js +0 -64
- package/dist/stress/smoke-runner.js.map +0 -1
- package/dist/stress/types.d.ts +0 -219
- package/dist/stress/types.d.ts.map +0 -1
- package/dist/stress/types.js +0 -2
- package/dist/stress/types.js.map +0 -1
- package/dist/stress/workload.d.ts +0 -6
- package/dist/stress/workload.d.ts.map +0 -1
- package/dist/stress/workload.js +0 -144
- package/dist/stress/workload.js.map +0 -1
- package/docs/PARTITION-STORE-WASM.md +0 -338
- package/docs/PARTITION-STORE.md +0 -2380
- package/docs/agents/response-artifacts.md +0 -65
- package/docs/response-artifacts.md +0 -86
- package/scripts/stress-docker.sh +0 -279
- package/sdm/flatsql-spatial.wasm +0 -0
- package/sdm/index.d.ts +0 -88
- package/sdm/index.js +0 -253
- package/sdm/manifest.json +0 -402
- package/stress/docker/Dockerfile +0 -11
- package/stress/docker/docker-compose.yml +0 -51
- package/stress/sds/aggregate.mjs +0 -97
- package/stress/sds/run.mjs +0 -105
- package/stress/sds/write-compose.mjs +0 -96
- package/wasm/flatsql-client.js +0 -280
- package/wasm/flatsql-io.d.ts +0 -97
- package/wasm/flatsql-io.js +0 -621
- package/wasm/flatsql-p4-threads.wasm +0 -0
- package/wasm/flatsql-ps-threads.wasm +0 -0
- package/wasm/flatsql-wasi-noeh.wasm +0 -0
- package/wasm/flatsql-wasi.wasm +0 -0
- package/wasm/flatsql.js +0 -2
- package/wasm/flatsql.worker.js +0 -209
- package/wasm/native/flatsql_wasmedge_runner.cpp +0 -858
- package/wasm/ps-node-fault.d.mts +0 -23
- package/wasm/ps-node-fault.mjs +0 -381
- package/wasm/ps-node-guest.mjs +0 -173
- package/wasm/ps-node-host.d.mts +0 -95
- package/wasm/ps-node-host.mjs +0 -476
- package/wasm/ps-node-imports.mjs +0 -847
- package/wasm/ps.d.ts +0 -21
- package/wasm/ps.js +0 -75
- package/wasm/wasi.d.ts +0 -11
- package/wasm/wasi.js +0 -51
package/dist/stress/workload.js
DELETED
|
@@ -1,144 +0,0 @@
|
|
|
1
|
-
export const DEFAULT_USE_CASES = [
|
|
2
|
-
{
|
|
3
|
-
id: 'schema-sweep',
|
|
4
|
-
label: 'SDS schema sweep',
|
|
5
|
-
operation: 'schema-parse',
|
|
6
|
-
description: 'Discover every SDS FlatBuffer schema and record compatibility metadata.',
|
|
7
|
-
},
|
|
8
|
-
{
|
|
9
|
-
id: 'bulk-streaming-ingest',
|
|
10
|
-
label: 'Bulk streaming ingest',
|
|
11
|
-
operation: 'stream-ingest',
|
|
12
|
-
description: 'Stream generated FlatBuffers in size-prefixed batches.',
|
|
13
|
-
},
|
|
14
|
-
{
|
|
15
|
-
id: 'hot-file-id-cache',
|
|
16
|
-
label: 'Hot FILE_ID query cache',
|
|
17
|
-
operation: 'query-template',
|
|
18
|
-
description: 'Repeat the same PNM/DPM-style FILE_ID lookup to measure cache behavior.',
|
|
19
|
-
},
|
|
20
|
-
{
|
|
21
|
-
id: 'cold-file-id-fanout',
|
|
22
|
-
label: 'Cold FILE_ID fanout',
|
|
23
|
-
operation: 'query-template',
|
|
24
|
-
description: 'Fan out mostly unique FILE_ID lookups and measure cache churn.',
|
|
25
|
-
},
|
|
26
|
-
{
|
|
27
|
-
id: 'mixed-hot-cold-query',
|
|
28
|
-
label: 'Mixed hot/cold query distribution',
|
|
29
|
-
operation: 'query-template',
|
|
30
|
-
description: 'Run a skewed query mix with hot and long-tail FILE_ID values.',
|
|
31
|
-
},
|
|
32
|
-
{
|
|
33
|
-
id: 'raw-flatbuffer-retrieval',
|
|
34
|
-
label: 'Raw FlatBuffer retrieval',
|
|
35
|
-
operation: 'raw-flatbuffer',
|
|
36
|
-
description: 'Return original FlatBuffer bytes through indexed lookup paths.',
|
|
37
|
-
},
|
|
38
|
-
{
|
|
39
|
-
id: 'sql-projection-query',
|
|
40
|
-
label: 'SQL projection query',
|
|
41
|
-
operation: 'parameterized-sql',
|
|
42
|
-
description: 'Return a narrow SQL projection to compare against binary payload transfer.',
|
|
43
|
-
},
|
|
44
|
-
{
|
|
45
|
-
id: 'raw-sql-query',
|
|
46
|
-
label: 'Raw SQL query',
|
|
47
|
-
operation: 'raw-sql',
|
|
48
|
-
description: 'Execute untemplated SQL and compare request overhead with query templates.',
|
|
49
|
-
},
|
|
50
|
-
{
|
|
51
|
-
id: 'large-result-query',
|
|
52
|
-
label: 'Large result query',
|
|
53
|
-
operation: 'large-result',
|
|
54
|
-
description: 'Return more rows than the result-cache row cap and measure transport cost.',
|
|
55
|
-
},
|
|
56
|
-
{
|
|
57
|
-
id: 'node-fanout-query',
|
|
58
|
-
label: 'Node fanout query',
|
|
59
|
-
operation: 'fanout-query',
|
|
60
|
-
description: 'Send one logical query across many nodes and measure tail latency.',
|
|
61
|
-
},
|
|
62
|
-
{
|
|
63
|
-
id: 'backfill-sync',
|
|
64
|
-
label: 'Backfill sync',
|
|
65
|
-
operation: 'backfill-stream',
|
|
66
|
-
description: 'Bootstrap a node by streaming records from another source.',
|
|
67
|
-
},
|
|
68
|
-
{
|
|
69
|
-
id: 'restart-rebuild',
|
|
70
|
-
label: 'Restart and rebuild',
|
|
71
|
-
operation: 'restart-rebuild',
|
|
72
|
-
description: 'Restart a node and measure rebuild or reload behavior.',
|
|
73
|
-
},
|
|
74
|
-
{
|
|
75
|
-
id: 'cache-invalidation-churn',
|
|
76
|
-
label: 'Cache invalidation churn',
|
|
77
|
-
operation: 'ingest-query-churn',
|
|
78
|
-
description: 'Alternate ingest and hot queries to expose broad invalidation cost.',
|
|
79
|
-
},
|
|
80
|
-
{
|
|
81
|
-
id: 'bandwidth-constrained-streaming',
|
|
82
|
-
label: 'Bandwidth-constrained streaming',
|
|
83
|
-
operation: 'throttled-stream',
|
|
84
|
-
description: 'Measure chunked streaming behavior under an artificial transport limit.',
|
|
85
|
-
},
|
|
86
|
-
{
|
|
87
|
-
id: 'sds-domain-query-pack',
|
|
88
|
-
label: 'SDS domain query pack',
|
|
89
|
-
operation: 'domain-query',
|
|
90
|
-
description: 'Exercise named space-domain lookups such as catalog, ephemeris, sensor, and weather queries.',
|
|
91
|
-
},
|
|
92
|
-
{
|
|
93
|
-
id: 'udl-style-api-shape',
|
|
94
|
-
label: 'UDL-style API comparison shape',
|
|
95
|
-
operation: 'api-shape',
|
|
96
|
-
description: 'Run centralized API-shaped requests without calling an external UDL service.',
|
|
97
|
-
},
|
|
98
|
-
];
|
|
99
|
-
export function normalizeStressRunConfig(config = {}) {
|
|
100
|
-
const mode = config.mode ?? 'smoke';
|
|
101
|
-
return {
|
|
102
|
-
mode,
|
|
103
|
-
runId: config.runId ?? `sds-${mode}-${Date.now()}`,
|
|
104
|
-
nodeCount: config.nodeCount ?? (mode === 'full' ? 100 : 2),
|
|
105
|
-
nodeIdOffset: config.nodeIdOffset ?? 0,
|
|
106
|
-
nodeStorageGb: config.nodeStorageGb ?? (mode === 'full' ? 1 : 0.01),
|
|
107
|
-
recordsPerNode: config.recordsPerNode ?? (mode === 'full' ? 20_000 : 100),
|
|
108
|
-
batchBytes: config.batchBytes ?? 1024 * 1024,
|
|
109
|
-
queryConcurrency: config.queryConcurrency ?? (mode === 'full' ? 64 : 4),
|
|
110
|
-
hotQueryRatio: config.hotQueryRatio ?? 0.9,
|
|
111
|
-
runtime: config.runtime ?? 'standalone',
|
|
112
|
-
transport: config.transport ?? 'in-process',
|
|
113
|
-
nodeIsolation: config.nodeIsolation ?? 'logical',
|
|
114
|
-
seed: config.seed ?? 0x5d5,
|
|
115
|
-
};
|
|
116
|
-
}
|
|
117
|
-
export function buildNodeAssignments(schemas, config) {
|
|
118
|
-
const schemaNames = schemas.map((schema) => schema.name);
|
|
119
|
-
return Array.from({ length: config.nodeCount }, (_, localNodeIndex) => {
|
|
120
|
-
const nodeId = config.nodeIdOffset + localNodeIndex;
|
|
121
|
-
const assigned = schemaNames.filter((_, index) => index % config.nodeCount === localNodeIndex);
|
|
122
|
-
if (assigned.length === 0 && schemaNames.length > 0) {
|
|
123
|
-
assigned.push(schemaNames[nodeId % schemaNames.length]);
|
|
124
|
-
}
|
|
125
|
-
return {
|
|
126
|
-
nodeId,
|
|
127
|
-
storageGb: config.nodeStorageGb,
|
|
128
|
-
schemas: assigned,
|
|
129
|
-
};
|
|
130
|
-
});
|
|
131
|
-
}
|
|
132
|
-
export function buildWorkloadManifest(config, schemas) {
|
|
133
|
-
const normalized = normalizeStressRunConfig(config);
|
|
134
|
-
const sortedSchemas = [...schemas].sort((left, right) => left.name.localeCompare(right.name));
|
|
135
|
-
return {
|
|
136
|
-
...normalized,
|
|
137
|
-
createdAt: new Date().toISOString(),
|
|
138
|
-
schemaCount: sortedSchemas.length,
|
|
139
|
-
schemas: sortedSchemas,
|
|
140
|
-
useCases: DEFAULT_USE_CASES,
|
|
141
|
-
assignments: buildNodeAssignments(sortedSchemas, normalized),
|
|
142
|
-
};
|
|
143
|
-
}
|
|
144
|
-
//# sourceMappingURL=workload.js.map
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
{"version":3,"file":"workload.js","sourceRoot":"","sources":["../../src/stress/workload.ts"],"names":[],"mappings":"AASA,MAAM,CAAC,MAAM,iBAAiB,GAAoB;IAChD;QACE,EAAE,EAAE,cAAc;QAClB,KAAK,EAAE,kBAAkB;QACzB,SAAS,EAAE,cAAc;QACzB,WAAW,EAAE,yEAAyE;KACvF;IACD;QACE,EAAE,EAAE,uBAAuB;QAC3B,KAAK,EAAE,uBAAuB;QAC9B,SAAS,EAAE,eAAe;QAC1B,WAAW,EAAE,wDAAwD;KACtE;IACD;QACE,EAAE,EAAE,mBAAmB;QACvB,KAAK,EAAE,yBAAyB;QAChC,SAAS,EAAE,gBAAgB;QAC3B,WAAW,EAAE,yEAAyE;KACvF;IACD;QACE,EAAE,EAAE,qBAAqB;QACzB,KAAK,EAAE,qBAAqB;QAC5B,SAAS,EAAE,gBAAgB;QAC3B,WAAW,EAAE,gEAAgE;KAC9E;IACD;QACE,EAAE,EAAE,sBAAsB;QAC1B,KAAK,EAAE,mCAAmC;QAC1C,SAAS,EAAE,gBAAgB;QAC3B,WAAW,EAAE,+DAA+D;KAC7E;IACD;QACE,EAAE,EAAE,0BAA0B;QAC9B,KAAK,EAAE,0BAA0B;QACjC,SAAS,EAAE,gBAAgB;QAC3B,WAAW,EAAE,gEAAgE;KAC9E;IACD;QACE,EAAE,EAAE,sBAAsB;QAC1B,KAAK,EAAE,sBAAsB;QAC7B,SAAS,EAAE,mBAAmB;QAC9B,WAAW,EAAE,4EAA4E;KAC1F;IACD;QACE,EAAE,EAAE,eAAe;QACnB,KAAK,EAAE,eAAe;QACtB,SAAS,EAAE,SAAS;QACpB,WAAW,EAAE,4EAA4E;KAC1F;IACD;QACE,EAAE,EAAE,oBAAoB;QACxB,KAAK,EAAE,oBAAoB;QAC3B,SAAS,EAAE,cAAc;QACzB,WAAW,EAAE,4EAA4E;KAC1F;IACD;QACE,EAAE,EAAE,mBAAmB;QACvB,KAAK,EAAE,mBAAmB;QAC1B,SAAS,EAAE,cAAc;QACzB,WAAW,EAAE,oEAAoE;KAClF;IACD;QACE,EAAE,EAAE,eAAe;QACnB,KAAK,EAAE,eAAe;QACtB,SAAS,EAAE,iBAAiB;QAC5B,WAAW,EAAE,4DAA4D;KAC1E;IACD;QACE,EAAE,EAAE,iBAAiB;QACrB,KAAK,EAAE,qBAAqB;QAC5B,SAAS,EAAE,iBAAiB;QAC5B,WAAW,EAAE,wDAAwD;KACtE;IACD;QACE,EAAE,EAAE,0BAA0B;QAC9B,KAAK,EAAE,0BAA0B;QACjC,SAAS,EAAE,oBAAoB;QAC/B,WAAW,EAAE,qEAAqE;KACnF;IACD;QACE,EAAE,EAAE,iCAAiC;QACrC,KAAK,EAAE,iCAAiC;QACxC,SAAS,EAAE,kBAAkB;QAC7B,WAAW,EAAE,yEAAyE;KACvF;IACD;QACE,EAAE,EAAE,uBAAuB;QAC3B,KAAK,EAAE,uBAAuB;QAC9B,SAAS,EAAE,cAAc;QACzB,WAAW,EAAE,8FAA8F;KAC5G;IACD;QACE,EAAE,EAAE,qBAAqB;QACzB,KAAK,EAAE,gCAAgC;QACvC,SAAS,EAAE,WAAW;QACtB,WAAW,EAAE,8EAA8E;KAC5F;CACF,CAAC;AAEF,MAAM,UAAU,wBAAwB,CAAC,SAA0B,EAAE;IACnE,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,IAAI,OAAO,CAAC;IACpC,OAAO;QACL,IAAI;QACJ,KAAK,EAAE,MAAM,CAAC,KAAK,IAAI,OAAO,IAAI,IAAI,IAAI,CAAC,GAAG,EAAE,EAAE;QAClD,SAAS,EAAE,MAAM,CAAC,SAAS,IAAI,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;QAC1D,YAAY,EAAE,MAAM,CAAC,YAAY,IAAI,CAAC;QACtC,aAAa,EAAE,MAAM,CAAC,aAAa,IAAI,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;QACnE,cAAc,EAAE,MAAM,CAAC,cAAc,IAAI,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC;QACzE,UAAU,EAAE,MAAM,CAAC,UAAU,IAAI,IAAI,GAAG,IAAI;QAC5C,gBAAgB,EAAE,MAAM,CAAC,gBAAgB,IAAI,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;QACvE,aAAa,EAAE,MAAM,CAAC,aAAa,IAAI,GAAG;QAC1C,OAAO,EAAE,MAAM,CAAC,OAAO,IAAI,YAAY;QACvC,SAAS,EAAE,MAAM,CAAC,SAAS,IAAI,YAAY;QAC3C,aAAa,EAAE,MAAM,CAAC,aAAa,IAAI,SAAS;QAChD,IAAI,EAAE,MAAM,CAAC,IAAI,IAAI,KAAK;KAC3B,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,oBAAoB,CAClC,OAAiC,EACjC,MAAuF;IAEvF,MAAM,WAAW,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;IACzD,OAAO,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC,EAAE,cAAc,EAAE,EAAE;QACpE,MAAM,MAAM,GAAG,MAAM,CAAC,YAAY,GAAG,cAAc,CAAC;QACpD,MAAM,QAAQ,GAAG,WAAW,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,EAAE,CAAC,KAAK,GAAG,MAAM,CAAC,SAAS,KAAK,cAAc,CAAC,CAAC;QAC/F,IAAI,QAAQ,CAAC,MAAM,KAAK,CAAC,IAAI,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YACpD,QAAQ,CAAC,IAAI,CAAC,WAAW,CAAC,MAAM,GAAG,WAAW,CAAC,MAAM,CAAC,CAAC,CAAC;QAC1D,CAAC;QACD,OAAO;YACL,MAAM;YACN,SAAS,EAAE,MAAM,CAAC,aAAa;YAC/B,OAAO,EAAE,QAAQ;SAClB,CAAC;IACJ,CAAC,CAAC,CAAC;AACL,CAAC;AAED,MAAM,UAAU,qBAAqB,CACnC,MAAuB,EACvB,OAAiC;IAEjC,MAAM,UAAU,GAAG,wBAAwB,CAAC,MAAM,CAAC,CAAC;IACpD,MAAM,aAAa,GAAG,CAAC,GAAG,OAAO,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC;IAC9F,OAAO;QACL,GAAG,UAAU;QACb,SAAS,EAAE,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE;QACnC,WAAW,EAAE,aAAa,CAAC,MAAM;QACjC,OAAO,EAAE,aAAa;QACtB,QAAQ,EAAE,iBAAiB;QAC3B,WAAW,EAAE,oBAAoB,CAAC,aAAa,EAAE,UAAU,CAAC;KAC7D,CAAC;AACJ,CAAC"}
|
|
@@ -1,338 +0,0 @@
|
|
|
1
|
-
# FlatSQL partition store: the wasm artifact and the Node host (T4)
|
|
2
|
-
|
|
3
|
-
The partition store engine (docs/PARTITION-STORE.md, T1 and T2) built as one
|
|
4
|
-
`wasm32-wasip1-threads` artifact, `wasm/flatsql-ps-threads.wasm`, and the
|
|
5
|
-
Node host that runs it and the engine's test suites with real threads. The
|
|
6
|
-
design is the stack's `docs/architecture/flatsql-partition-store.md`: §5.5,
|
|
7
|
-
§5.6, §18 T4 and the §22 amendments naming T4 (A1, A34, 22.3a-6/7). This file
|
|
8
|
-
records what was built, how to run it, the measured acceptance, and every
|
|
9
|
-
departure from the design.
|
|
10
|
-
|
|
11
|
-
## 1. The artifact
|
|
12
|
-
|
|
13
|
-
| | |
|
|
14
|
-
|---|---|
|
|
15
|
-
| Build | `bash scripts/build-wasm.sh --ps` (artifact) or `--ps-tests` (plus the test commands). CMake: `cpp/cmake/wasi-threads-toolchain.cmake` and `cpp/cmake/flatsql_ps_wasm.cmake`, included from `cpp/CMakeLists.txt` only under the WASI toolchain. Engine and test sources are globbed (`src/ps/*.cpp`, `test/ps/*_test.cpp`), so files other tasks add build here unchanged. |
|
|
16
|
-
| Toolchain | wasi-sdk 30.0 (clang 21.1.4), target `wasm32-wasip1-threads`, `-pthread -matomics -mbulk-memory -fno-exceptions`, Release (`-O3 -DNDEBUG`), `-ffile-prefix-map` for the source, flatbuffers and SDK paths, `--strip-debug`. The build script refuses any other wasi-sdk version. |
|
|
17
|
-
| Bytes | The released artifact is the Linux wasi-sdk 30 build: Linux arm64 (Docker) and the x86_64 CI runner produce the same sha256 from different checkout paths. The macOS wasi-sdk 30 tarball ships a different sysroot (libc, libc++), compiler-rt and clang build and links different (equivalent) bytes; on macOS `bash scripts/build-wasm.sh --ps --linux` builds the released bytes in Docker (the Linux tarball, sha256-pinned). `npm-publish.yml` rebuilds and refuses to publish unless the build equals the committed file; CI warns when it differs. `wasm/integrity.json` holds the sha256, sha384 SRI and size of every shipped wasm (`scripts/write-integrity.mjs`, `npm run check:integrity`). |
|
|
18
|
-
| Shape | A reactor (`-mexec-model=reactor`). A host instantiates one instance per writer or reader instance; every guest thread is a new instance of the same module over the same shared memory. |
|
|
19
|
-
| Imports | `wasi_snapshot_preview1`: `clock_time_get fd_close fd_prestat_dir_name fd_prestat_get fd_seek fd_write proc_exit random_get sched_yield` (all provided by the SDK's browser and Node pool hosts); `wasi.thread-spawn`; `env.memory` (shared, 512 initial, 32768 maximum pages); the seven `env.flatsql_io_*`. Nothing else: `scripts/check-wasm-imports.mjs` and `test/standalone-wasm-imports.test.ts` fail on any change. |
|
|
20
|
-
| Exports | `flatsql_ps_init start stop pump wake layout reader_layout stats register_type register_partition ring` (docs/PARTITION-STORE.md §6), `flatsql_ps_alloc`/`flatsql_ps_free` (host-owned buffers for config TLVs and paths), `wasi_thread_start`, `_initialize`, `memory`. |
|
|
21
|
-
| SQLite | The lanes' SQL layer (ruling 1): `SQLITE_THREADSAFE=2`, `DEFAULT_MEMSTATUS=0`, `OMIT_WAL`, `TEMP_STORE=3`, `SQLITE_OS_OTHER`. No unix VFS (it would import WASI file calls): `src/ps/capi_wasm.cpp` registers a default VFS that opens no file, and installs pthread mutexes, because `SQLITE_OS_OTHER` compiles only no-op ones. The lanes open `:memory:` on `flatsql_ps_null` as natively. |
|
|
22
|
-
| Threads | `std::thread` defaults to 1 MiB stacks (wasi-libc's default is 128 KiB and a shadow stack has no guard page); lanes set their own (`ReaderConfig::stackBytes`). `sleepNs` is a timed `memory.atomic.wait32`, so the artifact never imports `poll_oneoff`. |
|
|
23
|
-
| Guard | `assertPthreadArtifact` (space-data-module-sdk 0.8.24) passes: shared memory, 1,636 atomic instructions, the wasi-threads contract, no emscripten thread hooks. |
|
|
24
|
-
|
|
25
|
-
### Host obligations
|
|
26
|
-
|
|
27
|
-
- Create `env.memory` shared, with a maximum no larger than the import's
|
|
28
|
-
(browser writer 256 MiB, reader 128 MiB, design §5.5).
|
|
29
|
-
- **V8 hosts (Node, browsers): grow the heap to the memory's maximum before
|
|
30
|
-
`flatsql_ps_start`, so it never grows while guest threads run.** Allocate
|
|
31
|
-
through `flatsql_ps_alloc` in steps from 1 GiB down to 1 MiB until each
|
|
32
|
-
fails, then free them all (the allocator keeps the blocks; wasm memory never
|
|
33
|
-
shrinks; the guest allocator grows at most 2 GiB per call). V8 bounds-checks
|
|
34
|
-
`memory.fill`/`memory.copy` and atomics against each thread's cached view of
|
|
35
|
-
a shared memory's size, and another thread's `memory.grow` updates that view
|
|
36
|
-
only at this thread's next stack check. A thread that fills a block carved
|
|
37
|
-
from memory another thread has just grown then traps "memory access out of
|
|
38
|
-
bounds" (§4.3). A partial pre-grow only moves the failure: the 1,536 MiB the
|
|
39
|
-
test main used to grow was passed at 1,651 MiB in CI. Stop 16 MiB short of
|
|
40
|
-
4 GiB with a 4 GiB maximum: a heap segment ending at 2^32 wraps the
|
|
41
|
-
allocator's pointer arithmetic. `createPsNodeInstance` does this
|
|
42
|
-
(`growHeap: false` opts out, `heapBytes` caps it), and the Node host reports
|
|
43
|
-
`memoryGrownWhileThreadsRan` for every guest and warns when it is not 0.
|
|
44
|
-
- **WasmEdge hosts do not pre-grow** (and SDN recycles reader instances on
|
|
45
|
-
their page count). WasmEdge keeps one page count shared by every thread's
|
|
46
|
-
instance and grows in place in a fixed reservation, so a grow is visible to
|
|
47
|
-
every thread at once. WasmEdge 0.16.4's AOT compiler must carry SDN's
|
|
48
|
-
`04-atomic-memarg-offset` patch (§4.3).
|
|
49
|
-
- Give every guest thread a distinct `tid` below 2^29 (wasi-libc keeps it in
|
|
50
|
-
mutex words).
|
|
51
|
-
|
|
52
|
-
## 2. The Node host (design §5.6)
|
|
53
|
-
|
|
54
|
-
`wasm/ps-node-host.mjs` (types `ps-node-host.d.mts`), with
|
|
55
|
-
`ps-node-guest.mjs`, `ps-node-imports.mjs` and `ps-node-fault.mjs`. Node 24+,
|
|
56
|
-
optional peer dependency `space-data-module-sdk` ≥ 0.8.24.
|
|
57
|
-
|
|
58
|
-
- **Supervisor.** The main thread never runs guest code. The guest's own
|
|
59
|
-
thread (`_start` for a command, the exec thread for a reactor instance) is a
|
|
60
|
-
worker; so is every guest thread.
|
|
61
|
-
- **wasi.thread-spawn** claims a pre-started worker from a pool the supervisor
|
|
62
|
-
keeps (16 warm, grown on demand to a cap of 512). The claim is a CAS on the
|
|
63
|
-
pool's SharedArrayBuffer and needs no event loop, so any guest thread can
|
|
64
|
-
spawn, and workers are reused: a crash harness that starts and stops engines
|
|
65
|
-
12,000 times runs on 17 workers. Tids come from one shared counter.
|
|
66
|
-
- **flatsql_io** is the SDK's Node sync-fs provider (`createNodeSyncFsIo`: a
|
|
67
|
-
virtual-handle table in a SharedArrayBuffer, lazily opened fds per worker,
|
|
68
|
-
`fdatasync`/`F_FULLFSYNC`, `CREATE_PARENTS`, `UNLINK_IF_UNUSED`, A23
|
|
69
|
-
revocation) over the host root.
|
|
70
|
-
- **WASI preview1** is the host's own: a descriptor table in a
|
|
71
|
-
SharedArrayBuffer (a descriptor opened on one thread works on all, as in a
|
|
72
|
-
process), preopens, process-wide clocks (`process.hrtime`; a worker's
|
|
73
|
-
`performance.now()` has its own origin, which would break the engine's
|
|
74
|
-
cross-thread time comparisons), seeded `random_get` in deterministic mode,
|
|
75
|
-
`poll_oneoff` clock waits, `proc_exit` from any thread ending the guest.
|
|
76
|
-
- **File system.** `root` is the guest's "/" for WASI and for flatsql_io;
|
|
77
|
-
`mounts` preopen more host directories (the CLI mounts the checkout at its
|
|
78
|
-
own path so the test commands find their vectors). Default: a private
|
|
79
|
-
temporary root, removed afterwards.
|
|
80
|
-
- **Faults.** A guest thread that traps writes the trap to stderr from its own
|
|
81
|
-
worker and sets a shared fault word; the supervisor ends the run (the thread
|
|
82
|
-
that would join the dead one may be blocked forever).
|
|
83
|
-
- **API.** `runPsCommand(wasm, options)` (commands: exit code, kill, timeout,
|
|
84
|
-
thread report); `createPsNodeInstance(wasm, options)` (one artifact instance:
|
|
85
|
-
`call`, `write`, `read`, `threads`, `close`); CLI `node wasm/ps-node-host.mjs
|
|
86
|
-
<command.wasm> [args]`.
|
|
87
|
-
|
|
88
|
-
### Fault overlay (§19, 22.3a-7)
|
|
89
|
-
|
|
90
|
-
`wasm/ps-node-fault.mjs`: a SharedArrayBuffer page table every worker
|
|
91
|
-
consults, interposed under the SDK provider's syscalls. It records the
|
|
92
|
-
pre-image of every 4 KiB page written since the file's last sync (pages wholly
|
|
93
|
-
past the synced size need none), the synced size, and files created since
|
|
94
|
-
their parent directory's last sync. The supervisor freezes it (later mutating
|
|
95
|
-
calls fail with EIO), stops the guest, and applies a crash: `dropAll` (every
|
|
96
|
-
unsynced page and size change undone, unsynced directory entries gone),
|
|
97
|
-
`pageSubset` (each unsynced page survives or not, in any order; the size is
|
|
98
|
-
the synced or the current one), `keepAll` (kill -9). Overflowing its page
|
|
99
|
-
budget fails the crash instead of making it partial.
|
|
100
|
-
|
|
101
|
-
## 3. Test commands
|
|
102
|
-
|
|
103
|
-
| Command | Contents |
|
|
104
|
-
|---|---|
|
|
105
|
-
| `cpp/build-ps-wasm/flatsql-ps-test.wasm` | `flatsql_ps_test` (every test in `cpp/test/ps`) as a wasi-threads command; host I/O through the seven imports. `--list` prints the tests. |
|
|
106
|
-
| `cpp/build-ps-wasm/flatsql-ps-test-memio.wasm` | The same command with the seven functions defined in the guest (`test/ps/wasm_io_stub.cpp`, all ACCESS): imports only WASI, thread-spawn and memory, so a WASI-only host (the SDK WasmEdge C runner) runs every in-memory test. |
|
|
107
|
-
|
|
108
|
-
Added tests (`PS_SLOW_TEST`, run only when named): `parity_canonical_dump`,
|
|
109
|
-
`parity_deterministic_bytes` (`cpp/test/ps/parity_test.cpp`) and
|
|
110
|
-
`host_crash_ingest` / `host_crash_verify` (`host_crash_test.cpp`). The wasm
|
|
111
|
-
test main grows the heap to the memory's maximum before any thread starts
|
|
112
|
-
(§1; `PS_WASM_HEAP_MB` caps it).
|
|
113
|
-
|
|
114
|
-
```
|
|
115
|
-
npm run build:wasm:ps-tests # artifact + test commands
|
|
116
|
-
node wasm/ps-node-host.mjs cpp/build-ps-wasm/flatsql-ps-test.wasm --test=lane_
|
|
117
|
-
npm run test:ps-wasm # scripts/ps-wasm-suite.mjs: every default test + 1,000 crashes
|
|
118
|
-
FLATSQL_PS_WASM_SUITE=1 npm test -- test/ps-node-threads.test.ts
|
|
119
|
-
npm test -- test/ps-host-fault.test.ts test/ps-parity-vectors.test.ts
|
|
120
|
-
FLATSQL_PS_WASMEDGE=1 npm test -- test/ps-parity-vectors.test.ts # + the C runner lane
|
|
121
|
-
```
|
|
122
|
-
|
|
123
|
-
## 4. Acceptance (§18 T4 as amended)
|
|
124
|
-
|
|
125
|
-
Machine: the owner's Mac Studio (Darwin 25.3.0, arm64, 28 hardware threads,
|
|
126
|
-
APFS), Node 25.4.0, shared with other lanes: 1-minute load average 18–25
|
|
127
|
-
during these runs (37–41 for the release rerun, §4.2). Native builds
|
|
128
|
-
RelWithDebInfo; wasm builds as §1.
|
|
129
|
-
|
|
130
|
-
| # | Acceptance | Result |
|
|
131
|
-
|---|---|---|
|
|
132
|
-
| 1 | Imports exactly WASI p1 + `wasi.thread-spawn` + shared `env.memory` (≤ 32768 pages) + the 7 `env.flatsql_io_*`; exports `wasi_thread_start`; 0 emscripten imports; passes `assertPthreadArtifact` | **Pass.** §1 table; `scripts/check-wasm-imports.mjs`, `test/standalone-wasm-imports.test.ts` (the SDK guard runs in a child process). |
|
|
133
|
-
| 2 | The T1 and T2 suites, including 1,000 fault crashes, pass under the Node host with distinct OS thread ids ≥ writers + lanes | **Pass.** `scripts/ps-wasm-suite.mjs`: all 53 default-suite tests, each in its own guest process, plus `crash_faults_T1_1 --crash-trials=1000` (1,000 trials over the five modes, 298,923 acked records verified, 41.8 s). `readers_under_saturating_writers_T2_1` (8 writers, 16 lanes) ran 54 guest threads at once on 54 distinct worker (OS) threads. `test/ps-node-threads.test.ts`: the shipped artifact as a writer instance (4 writers) and a reader instance (4 lanes), each running thread on its own worker. `writer_backpressure_flood_T1_5` runs with `--bp-samples=40` (deviation 3). `crash_faults_T1_1_full` (10,000 trials, 64 types, 256 partitions): §4.1. |
|
|
134
|
-
| 3 | Parity: canonical dumps (by (pid, pseq)) and query results identical native vs Node; deterministic mode byte-identical across native, Node and the SDK WasmEdge C runner (patched 0.16.4) | **Pass.** Canonical dump, 4 writers, 11,313 lines (rows, counters, lane counters and six queries, partition and type level): sha256 `f3b419a2…` natively and under Node, three runs each. Deterministic mode, 122 files: in memory `e7ab8592…` natively, under Node and under the C runner (18.7 s, interpreted); on the host's files `c7e34c55…` natively and under Node. `test/ps-parity-vectors.test.ts`. |
|
|
135
|
-
| 4 | A published npm version with provenance; the wasm sha256 in `integrity.json`; `npm ls sql.js` empty; the package root exports only the wasm bindings | **Pass.** `flatsql@3.0.0`, §4.2 and §6. |
|
|
136
|
-
|
|
137
|
-
Host-level crashes (the overlay, §2): `test/ps-host-fault.test.ts` kills the
|
|
138
|
-
guest mid-ingest at a random 0.4–2.0 s, crashes the store in the three modes
|
|
139
|
-
in turn and verifies every acked record (bytes by CID), gap-free pseqs,
|
|
140
|
-
counters and 0 `d-*` bytes read at open; 9 cycles, half of them with the A8
|
|
141
|
-
journal. A negative control (half a data segment cut off) fails the verifier.
|
|
142
|
-
|
|
143
|
-
### 4.1 Slow runs
|
|
144
|
-
|
|
145
|
-
- `crash_faults_T1_1_full` under the Node host (10,000 trials, 64 types, 256
|
|
146
|
-
partitions, the five crash modes, an A4 second crash every other trial, half
|
|
147
|
-
the stores journaled): pass in 377 s; 2,122,844 acked records verified,
|
|
148
|
-
410,159 merges, 1,167,880 journal records replayed, 125,016 guest threads
|
|
149
|
-
spawned on 10 pooled workers. Mac, load 24 rising to 35 during the run
|
|
150
|
-
(natively 243 s).
|
|
151
|
-
- The whole default suite under the host, one guest process per test:
|
|
152
|
-
155 s of test time; the most concurrent guest threads was 54
|
|
153
|
-
(`readers_under_saturating_writers_T2_1`), each on its own worker thread;
|
|
154
|
-
the largest guest memory 1,902 MiB (the 1,536 MiB pre-grown heap included).
|
|
155
|
-
- Deterministic vector under the C runner: 18.7 s interpreted (0.12 s natively).
|
|
156
|
-
|
|
157
|
-
### 4.2 Release
|
|
158
|
-
|
|
159
|
-
- `flatsql@3.0.0`, published by `npm-publish.yml` from tag `v3.0.0` =
|
|
160
|
-
`79fd694` (run 36486543061) with a signed provenance statement
|
|
161
|
-
(`npm audit signatures`: verified attestation); `gitHead` 79fd694, dist-tag
|
|
162
|
-
`latest`, `dependencies` `{}`.
|
|
163
|
-
- `wasm/flatsql-ps-threads.wasm`: 2,074,707 bytes, sha256
|
|
164
|
-
`572a4bc38345d07ff91c345fc8d81e28873d72efa0fb8864a2edc5242145f9f5`, the same
|
|
165
|
-
in `wasm/integrity.json`, from the Linux arm64 Docker build, the x86_64
|
|
166
|
-
publish runner and the installed tarball.
|
|
167
|
-
- A fresh `npm install flatsql@3.0.0`: `npm ls sql.js` is empty; the root's
|
|
168
|
-
exports equal `flatsql/wasm`'s; `loadFlatSQLPsThreads()` checks the bytes
|
|
169
|
-
against `integrity.json`.
|
|
170
|
-
- At the release sources (`dac8d89` engine), under the Node host: the 55
|
|
171
|
-
default-suite tests and `crash_faults_T1_1 --crash-trials=1000` pass (1,000
|
|
172
|
-
trials, 403,798 acked records verified, 47.3 s; load 37–41). Main CI green
|
|
173
|
-
on `79fd694` (run 36482613908).
|
|
174
|
-
- Two engine defects the Node host found before the release, fixed in the
|
|
175
|
-
sources the artifact builds: the open path dropped the lane checkpoint
|
|
176
|
-
pointer of a head with more than 32 live lanes (`5c7e769`; the 1,000-trial
|
|
177
|
-
crash run, trial 317), and the type owner never merged its L0 directory
|
|
178
|
-
when `mergeL0Blocks` exceeded the directory cap, so labeling, partition
|
|
179
|
-
merges and acks stopped (`dac8d89`).
|
|
180
|
-
|
|
181
|
-
### 4.3 The shared-memory grow race (V8) and AOT wait/notify (WasmEdge)
|
|
182
|
-
|
|
183
|
-
**V8.** CI ps-wasm run 36603876540 (attempt 1) trapped "memory access out of
|
|
184
|
-
bounds" in `writeTypeMergeOutputs` on the type-merge helper thread during
|
|
185
|
-
`readers_under_saturating_writers_T2_1`, with memory at 1,651 MiB, past the
|
|
186
|
-
test main's 1,536 MiB pre-grow. A Docker build with the Linux wasi-sdk 30 at
|
|
187
|
-
the CI checkout path reproduces the CI test command exactly (its function
|
|
188
|
-
indices match the trace, and its `flatsql-ps-threads.wasm` is the committed
|
|
189
|
-
artifact's sha256). The trap offset `0x18d31a` is a `memory.fill`: the
|
|
190
|
-
zero-fill of `blocks[i].resize(de.l0Len)` right after its `operator new`.
|
|
191
|
-
Each thread's instance caches the memory size, and V8 bounds-checks
|
|
192
|
-
`memory.fill`/`memory.copy` and atomics against that cache. Another thread's
|
|
193
|
-
`memory.grow` updates the cache only at this thread's next stack check, so a
|
|
194
|
-
block the allocator carves from memory another thread has just grown fails
|
|
195
|
-
that check. Plain loads and stores are checked by guard pages and never trap.
|
|
196
|
-
Measured on the Mac, Node 25.4.0, load 35–65:
|
|
197
|
-
|
|
198
|
-
| Heap before threads start | T2_1 runs | Traps |
|
|
199
|
-
|---|---|---|
|
|
200
|
-
| none (`PS_WASM_HEAP_MB=0`) | 31 | 8, every one a `memory.fill`/`memory.copy` right after an allocation (vector push_back, string copy, the resize above) |
|
|
201
|
-
| 2,000 MiB (past the test's peak) | 30 | 0 |
|
|
202
|
-
| the memory's maximum (§1, now the default) | 500 | 0, and 0 runs grew memory after a thread started (1 run ended in a Node 25.4.0 SIGILL inside V8's JIT code, a host crash) |
|
|
203
|
-
|
|
204
|
-
Natively the same test under ASan+UBSan (Docker linux/arm64, 4 CPUs, a CPU
|
|
205
|
-
hog) found no memory error in 488 runs: the engine never overran.
|
|
206
|
-
|
|
207
|
-
**WasmEdge, the SDN host.** Every thread's instance imports one shared
|
|
208
|
-
`MemoryInstance`. On 64-bit hosts it is reserved in full up front (12 GiB,
|
|
209
|
-
`lib/system/allocator.cpp`) and `memory.grow` maps pages in place, so the
|
|
210
|
-
base never moves. The page count is one field that `memory.fill`/`copy`/
|
|
211
|
-
`init` and atomic wait/notify check live, and plain loads and stores are
|
|
212
|
-
checked by guard pages. The SDN C host I/O resolves every guest pointer per
|
|
213
|
-
call (`WasmEdge_MemoryInstanceGetPointer`), and the Go accessor re-reads the
|
|
214
|
-
page count on a miss. A grow is visible to every thread when the growing
|
|
215
|
-
thread's `memory.grow` returns, and the guest allocator's lock orders it
|
|
216
|
-
before any other thread can allocate from it. Measured: the same T2_1 with
|
|
217
|
-
no pre-grow on the patched WasmEdge 0.16.4, AOT, had 0 traps in 26 completed
|
|
218
|
-
runs.
|
|
219
|
-
|
|
220
|
-
**WasmEdge AOT wait/notify.** Those WasmEdge runs also hung at teardown: 4 of
|
|
221
|
-
30 with no pre-grow, 5 of 30 with a 1,536 MiB pre-grow. WasmEdge 0.16.4's AOT
|
|
222
|
-
compiler passed `memory.atomic.notify` and `memory.atomic.wait32/64` the bare
|
|
223
|
-
address operand and dropped the instruction's memarg offset; the interpreter
|
|
224
|
-
adds it. wasi-libc addresses its thread-list lock as `i32.const 0;
|
|
225
|
-
memory.atomic.notify offset=<lock>`, so those wakeups went to address 0 and
|
|
226
|
-
thread exit and join hung. A plain spawn/join guest with no flatsql code, 50
|
|
227
|
-
rounds of 16 threads, hung 11 of 20 runs AOT, 0 of 30 interpreted and 0 of 20
|
|
228
|
-
under V8. `sleepNs` (a timed wait on a stack local whose frame offset the
|
|
229
|
-
compiler folds into the memarg) compared another stack word and returned at
|
|
230
|
-
once: 100 × 10 ms took 0.000 s AOT and 1.159 s interpreted. SDN's static
|
|
231
|
-
WasmEdge carries patch `04-atomic-memarg-offset` from SDN `74db37e14`, and
|
|
232
|
-
its substrate self-test checks it (tag `sdn3`, so threaded AOT artifacts
|
|
233
|
-
recompile). With the patch, the plain guest ran 40 of 40, and T2_1 AOT ran
|
|
234
|
-
140 times with no hang. Two of those runs failed. One ran out of the 4 GiB
|
|
235
|
-
address space: the memio command keeps every file in guest memory, and
|
|
236
|
-
passing runs end near 2.7 GiB. The other run's output was not kept.
|
|
237
|
-
`memory_safety_sleep_and_notify_at_folded_offsets` passes. Without it, that test fails (20 of 20 folded-offset wakeups lost).
|
|
238
|
-
`sleepNs` now passes its address through a volatile, so its wait has a zero
|
|
239
|
-
memarg offset and sleeps even on an unpatched runtime. The artifact still has
|
|
240
|
-
4 folded wait/notify sites, all in wasi-libc's thread code.
|
|
241
|
-
|
|
242
|
-
## 5. Design deviations
|
|
243
|
-
|
|
244
|
-
1. **The Node host's thread spawn is its own pool, not the SDK's Node
|
|
245
|
-
branch** (§5.6: "The SDK's Node branch (lazy worker_threads) runs the guest
|
|
246
|
-
threads"). `createWasiThreadSpawn` (0.8.24) removes a finished worker only on
|
|
247
|
-
its `exit` event, which runs on the spawning thread's event loop; that
|
|
248
|
-
thread is inside the guest, blocked, for the whole command. After `poolSize`
|
|
249
|
-
spawns every later spawn is declined, and without a pool the finished
|
|
250
|
-
threads are never joined. The crash harness spawns more than 12,000 guest
|
|
251
|
-
threads per run. The host keeps a pool whose idle/assign handshake is a
|
|
252
|
-
SharedArrayBuffer CAS, so no event loop is involved and any thread can
|
|
253
|
-
spawn. Reported to the module-sdk lane (SDK 0.8.25); the SDK's Node sync-fs
|
|
254
|
-
flatsql_io provider is used as designed.
|
|
255
|
-
2. **WASI is the host's own**, not the SDK shim: the shim keeps descriptors
|
|
256
|
-
and clocks per worker (a worker's `performance.now()` has its own origin),
|
|
257
|
-
has no preopens and no `poll_oneoff`. The artifact itself imports only names
|
|
258
|
-
the SDK shim provides (§1), so browser hosts are unaffected.
|
|
259
|
-
3. **The flood test runs 40 samples under wasm32** (`--bp-samples=40`, default
|
|
260
|
-
150): its in-memory host keeps every flooded byte twice for as long as the
|
|
261
|
-
timed measurement lasts (12 GB resident natively with 150 samples), past
|
|
262
|
-
wasm32's 4 GiB. Its ring, credit, pool and zero-loss checks are unchanged;
|
|
263
|
-
the ack-p99 ratio is reported (1.066) and, as on any box with fewer than 8
|
|
264
|
-
hardware threads, not enforced: WASI reports one CPU. The same holds for the
|
|
265
|
-
T2 latency bounds (`latencyBoxQuiet` is false under wasm): they are native
|
|
266
|
-
Linux-8 acceptance numbers. `lane_needs_bulk_under_1ms_zero_rows_T2_4`
|
|
267
|
-
enforced its 1 ms p99 on every machine; it now follows the same rule
|
|
268
|
-
(enforced on a quiet box, reported elsewhere) after a Debug build on the
|
|
269
|
-
loaded 3-vCPU macOS CI runner and the Node host on the Linux runner
|
|
270
|
-
measured past it. Its status, rows and examined counts are enforced
|
|
271
|
-
everywhere. `type_arrivals_segments_and_fence_A15` caps type commits at 40
|
|
272
|
-
rows: a slow type owner labeled more than 40 rows in one commit and filled
|
|
273
|
-
an empty arrivals segment past its 40-entry seal (a batch never splits
|
|
274
|
-
across segments), which failed on the macOS runner (main 8aea463) and
|
|
275
|
-
under the Node host.
|
|
276
|
-
4. **The C runner lane runs the memio command.** The SDK's WasmEdge
|
|
277
|
-
wasi-threads C runner provides WASI and thread-spawn only, so the byte
|
|
278
|
-
vectors there use the in-memory host, which is also compared natively and
|
|
279
|
-
under Node; the host-file vector is compared natively and under Node.
|
|
280
|
-
Adding flatsql_io to the runner (as the SDN C host module does) is an SDK
|
|
281
|
-
change.
|
|
282
|
-
5. **Parity inputs are built deterministically.** A fixture expression such
|
|
283
|
-
as `a + b * c` is contracted to a fused multiply-add on arm64 and not in
|
|
284
|
-
wasm, which changes the input bytes (1 ULP), not the engine's output; the
|
|
285
|
-
parity workload builds its frames from exact doubles. `buildRecordAttr`
|
|
286
|
-
(`src/ps/writer_pool.cpp`) creates its strings inside one call's argument
|
|
287
|
-
list, whose evaluation order C++ leaves unspecified: GCC on x86_64 builds
|
|
288
|
-
them in the opposite order from clang and from GCC on arm64, so the same
|
|
289
|
-
attributes serialize to different bytes (measured on the CI runner and under
|
|
290
|
-
x86_64 emulation). The parity workload builds its attributes in a fixed
|
|
291
|
-
order; the helper itself belongs to the writer files T3 holds (reported). The canonical dump masks
|
|
292
|
-
`kRowAttrInM` (where attributes are stored until a merge moves them, like
|
|
293
|
-
offsets) and injects the engine's wall clock (TOMB arrival times).
|
|
294
|
-
6. **Deterministic mode injects both clocks in the guest** (`setTestClock`,
|
|
295
|
-
`src/ps/platform.cpp`): the store UUID hashes the monotonic clock, and a
|
|
296
|
-
frozen clock keeps every time-based trigger out of the byte stream. A host
|
|
297
|
-
running the artifact deterministically injects `clock_time_get` and
|
|
298
|
-
`random_get` instead (the Node host's `deterministic` option seeds
|
|
299
|
-
`random_get`).
|
|
300
|
-
7. **A wasm32 defect fixed in the lane arena**: the 256 GiB request bound was
|
|
301
|
-
`size_t(1) << 38`, undefined where `size_t` is 32 bits; every lane
|
|
302
|
-
allocation failed and every lane died at `sqlite3_open`.
|
|
303
|
-
|
|
304
|
-
## 6. Package 3.0.0 (A1, A34)
|
|
305
|
-
|
|
306
|
-
Semver-major (A34). The package root is the wasm bindings: `wasm/index.js`
|
|
307
|
-
(the SQLite engine, as `flatsql/wasm`) plus the partition store locator
|
|
308
|
-
(`getFlatSQLPsThreadsURL`, `loadFlatSQLPsThreads`,
|
|
309
|
-
`FLATSQL_PS_THREADS_SHA256`). New subpaths: `flatsql/ps-threads.wasm`,
|
|
310
|
-
`flatsql/ps/node-host`, `flatsql/ps/fault`, `flatsql/integrity.json`.
|
|
311
|
-
|
|
312
|
-
Removed (A1 ledger row T4):
|
|
313
|
-
|
|
314
|
-
| Removed | Replaced by |
|
|
315
|
-
|---|---|
|
|
316
|
-
| `src/core/` (the TS engine) and the root exports `FlatSQLDatabase`, `DirectAccessor`, `FlatcAccessor`, `TableStore` | the wasm engines |
|
|
317
|
-
| the `node:sqlite` artifact builder (`FlatSQLArtifactBuilder`, its worker) and the `flatsql/artifacts/standalone` builder | — |
|
|
318
|
-
| `wasm/standalone.js` and `flatsql/standalone` | `flatsql/ps/node-host` for the partition store. The legacy WASI engine's own tests keep a test-only copy (`test/support/legacy-standalone.js`, not shipped) until T8 retires that engine. |
|
|
319
|
-
| `sql.js`, and the npm `flatbuffers` package (its only user was the deleted TS workload) | — |
|
|
320
|
-
|
|
321
|
-
The BTree, stacked store, schema parser and cluster detection stay, as
|
|
322
|
-
`flatsql/btree`, `flatsql/storage`, `flatsql/schema` and `flatsql/cluster`.
|
|
323
|
-
|
|
324
|
-
**A1 parity check (the receipt).** A1 allows the deletion when each consumer's
|
|
325
|
-
suite passes on the new bindings or the consumer pins the last pre-T4 major.
|
|
326
|
-
Every consumer pins an earlier major, so none resolves 3.0.0:
|
|
327
|
-
|
|
328
|
-
| Consumer (stack pin) | flatsql |
|
|
329
|
-
|---|---|
|
|
330
|
-
| space-data-module-sdk 0.8.24 (450aa78; `src/runtime-host/flatsqlRuntimeStore.js`) | `^2.0.0` |
|
|
331
|
-
| OrbPro (216083d024; `packages/wasm-engine`, root) | `^1.4.4` |
|
|
332
|
-
| OrbPro `packages/space-data-module-sdk` 0.8.23 | `^2.0.0` |
|
|
333
|
-
| spacedatastandards.org 1.228.0 (cd86467de5; `test-flatsql*.js`) | `^1.3.0` |
|
|
334
|
-
| spacedatastandards `packages/standards-explorer` | `^0.3.9` |
|
|
335
|
-
| sdn-js 3.1.0 (space-data-network f20dbc35d) | `2.0.3` |
|
|
336
|
-
|
|
337
|
-
Moving each consumer to 3.x (or recording that it stays on 2.x) is a ripple
|
|
338
|
-
task per repository (A1, A34).
|