@evolu/sqlite-wasm 2.2.4 → 3.53.4-build1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +55 -218
- package/dist/src/CApi.d.ts +2108 -0
- package/dist/src/CApi.d.ts.map +1 -0
- package/dist/src/CApi.js +1919 -0
- package/dist/src/Constants.d.ts +868 -0
- package/dist/src/Constants.d.ts.map +1 -0
- package/dist/src/Constants.js +602 -0
- package/dist/src/Database.d.ts +641 -0
- package/dist/src/Database.d.ts.map +1 -0
- package/dist/src/Database.js +1177 -0
- package/dist/src/Memory.d.ts +119 -0
- package/dist/src/Memory.d.ts.map +1 -0
- package/dist/src/Memory.js +207 -0
- package/dist/src/Pointer.d.ts +100 -0
- package/dist/src/Pointer.d.ts.map +1 -0
- package/dist/src/Pointer.js +15 -0
- package/dist/src/SahPool.d.ts +744 -0
- package/dist/src/SahPool.d.ts.map +1 -0
- package/dist/src/SahPool.js +1985 -0
- package/dist/src/Wasm.d.ts +315 -0
- package/dist/src/Wasm.d.ts.map +1 -0
- package/dist/src/Wasm.js +756 -0
- package/dist/src/WasmUrl.d.ts +21 -0
- package/dist/src/WasmUrl.d.ts.map +1 -0
- package/dist/src/WasmUrl.js +20 -0
- package/dist/src/c-api/index.d.ts +9 -0
- package/dist/src/c-api/index.d.ts.map +1 -0
- package/dist/src/c-api/index.js +8 -0
- package/dist/src/index.d.ts +15 -0
- package/dist/src/index.d.ts.map +1 -0
- package/dist/src/index.js +13 -0
- package/dist/wasm/sqlite3.wasm +0 -0
- package/package.json +60 -60
- package/src/CApi.test.ts +217 -0
- package/src/CApi.ts +3294 -0
- package/src/Constants.ts +803 -0
- package/src/Database.ts +1714 -0
- package/src/Memory.test.ts +319 -0
- package/src/Memory.ts +237 -0
- package/src/Pointer.ts +121 -0
- package/src/SahPool.test.ts +180 -0
- package/src/SahPool.ts +2627 -0
- package/src/Wasm.test.ts +444 -0
- package/src/Wasm.ts +1026 -0
- package/src/WasmUrl.ts +26 -0
- package/src/c-api/index.ts +9 -0
- package/src/index.ts +15 -0
- package/bin/index.js +0 -110
- package/index.d.ts +0 -8118
- package/index.mjs +0 -7
- package/node.mjs +0 -3
- package/sqlite-wasm/jswasm/sqlite3-bundler-friendly.mjs +0 -13659
- package/sqlite-wasm/jswasm/sqlite3-node.mjs +0 -11671
- package/sqlite-wasm/jswasm/sqlite3-opfs-async-proxy.js +0 -691
- package/sqlite-wasm/jswasm/sqlite3-worker1-bundler-friendly.mjs +0 -35
- package/sqlite-wasm/jswasm/sqlite3-worker1-promiser.js +0 -193
- package/sqlite-wasm/jswasm/sqlite3-worker1-promiser.mjs +0 -187
- package/sqlite-wasm/jswasm/sqlite3-worker1.js +0 -46
- package/sqlite-wasm/jswasm/sqlite3.js +0 -13697
- package/sqlite-wasm/jswasm/sqlite3.mjs +0 -13661
- package/sqlite-wasm/jswasm/sqlite3.wasm +0 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2023 Evolu
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,253 +1,90 @@
|
|
|
1
|
-
# SQLite Wasm
|
|
1
|
+
# Evolu SQLite Wasm
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
This package is SQLite compiled to WebAssembly with Evolu's own minimal TypeScript layer for loading and using it. The layer also encrypts databases.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
> [SQLite Wasm](https://github.com/sqlite/sqlite-wasm) that uses
|
|
7
|
-
> [SQLite3MultipleCiphers](https://github.com/utelle/SQLite3MultipleCiphers).
|
|
5
|
+
## Why our own TypeScript
|
|
8
6
|
|
|
9
|
-
SQLite
|
|
7
|
+
SQLite's WebAssembly build ships a general-purpose JavaScript layer. It's about 15,000 lines of source covering the whole C API, several storage VFSes, and a worker API, plus Emscripten's generated runtime.
|
|
10
8
|
|
|
11
|
-
|
|
9
|
+
We replace that JavaScript with our own, written to be as correct as possible. It's TypeScript throughout, with branded pointer types and typed errors, instead of loosely typed JavaScript with hand-written declarations. We ported the tests of the SQLite team and of [wa-sqlite](https://github.com/rhashimoto/wa-sqlite) for what this package ships. We also wrote our own tests for the failure paths that matter for local-first data, such as recovery after an interrupted transaction, storage running out, and lock conflicts between connections. They run in CI in Chromium, Firefox, and WebKit.
|
|
12
10
|
|
|
13
|
-
|
|
14
|
-
>
|
|
15
|
-
> This project wraps the code of
|
|
16
|
-
> [SQLite Wasm](https://sqlite.org/wasm/doc/trunk/index.md) with _no_ changes,
|
|
17
|
-
> apart from added TypeScript types. Please do _not_ file issues or feature
|
|
18
|
-
> requests regarding the underlying SQLite Wasm code here. Instead, please
|
|
19
|
-
> follow the
|
|
20
|
-
> [SQLite bug filing instructions](https://www.sqlite.org/src/wiki?name=Bug+Reports).
|
|
21
|
-
> Filing TypeScript type related issues and feature requests is fine.
|
|
11
|
+
We reimplement SQLite's JavaScript strictly test-first: red, green, refactor. Every behavior starts as a failing test, ported from the SQLite team's or wa-sqlite's tests or written for our failure paths. Only then do we add code, the least that makes the test pass, and we clean it up while the tests stay green.
|
|
22
12
|
|
|
23
|
-
|
|
13
|
+
These failure paths are real. While preparing this package in September 2026, we found that opfs-sahpool, the VFS Evolu relies on, never rolled back a transaction interrupted by closing a tab. The next open saw it half-applied. Fixing that uncovered a second problem: overlapping connections in one worker could silently lose committed data. Evolu opens only one connection, but apps and libraries that open several were exposed. Now they get `SQLITE_BUSY` instead. We reported both, and the SQLite team fixed them within a day. The [forum thread](https://sqlite.org/forum/forumpost/b2fbb61642) has the details and reproductions.
|
|
24
14
|
|
|
25
|
-
|
|
26
|
-
>
|
|
27
|
-
> Node.js is currently only supported for in-memory databases without
|
|
28
|
-
> persistence.
|
|
15
|
+
We also asked for `SQLITE_FULL` when the browser says storage is full, so an app can tell its users instead of showing a generic disk I/O error. The SQLite team [declined](https://sqlite.org/forum/forumpost/e8997c0ed9). Their JavaScript avoids browser-specific behavior, so every failed write stays `SQLITE_IOERR`. But the main signal isn't browser-specific. The [File System Standard](https://fs.spec.whatwg.org/#api-filesystemsyncaccesshandle-write) requires a sync access handle's `write()` and `truncate()` to throw `QuotaExceededError` when the quota would be exceeded. Our layer reports `SQLITE_FULL` for it and for Firefox's short writes. We test that with a real quota in Chromium and Firefox.
|
|
29
16
|
|
|
30
|
-
|
|
17
|
+
Every public C function the WebAssembly exports can be imported as a standalone function from `@evolu/sqlite-wasm/c-api`, so an app pays only for the functions it uses. We generate them from SQLite's own signature table and check them against the binary. The variadic `sqlite3_config`, `sqlite3_db_config` and `sqlite3_vtab_config` go through the build's fixed-argument shims, as in SQLite's JavaScript. A small loader replaces Emscripten's generated runtime. Together, that makes the JavaScript much smaller. The layer is synchronous, and the high-level API returns typed results.
|
|
31
18
|
|
|
32
|
-
|
|
33
|
-
npm install @evolu/sqlite-wasm
|
|
34
|
-
```
|
|
19
|
+
Of SQLite's storage VFSes, we keep only opfs-sahpool. Evolu already uses it, and it fits Evolu best. It's synchronous, reading and writing through OPFS sync access handles directly in the worker, without a helper worker. It doesn't need cross-origin isolation. SQLite's "opfs" VFS needs `SharedArrayBuffer`, which requires the `Cross-Origin-Opener-Policy` and `Cross-Origin-Embedder-Policy` headers and the restrictions they bring. opfs-sahpool needs neither. And it's the fastest. SQLite's documentation calls it the highest-performing of its OPFS VFSes, clearly so for batch operations.
|
|
35
20
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
There are three ways to use SQLite Wasm:
|
|
39
|
-
|
|
40
|
-
- [in the main thread with a wrapped worker](#in-a-wrapped-worker-with-opfs-if-available)
|
|
41
|
-
(🏆 preferred option)
|
|
42
|
-
- [in a worker](#in-a-worker-with-opfs-if-available)
|
|
43
|
-
- [in the main thread](#in-the-main-thread-without-opfs)
|
|
44
|
-
|
|
45
|
-
Only the worker versions allow you to use the origin private file system (OPFS)
|
|
46
|
-
storage back-end.
|
|
47
|
-
|
|
48
|
-
### In a wrapped worker (with OPFS if available):
|
|
49
|
-
|
|
50
|
-
> [!Warning]
|
|
51
|
-
>
|
|
52
|
-
> For this to work, you need to set the following headers on your server:
|
|
53
|
-
>
|
|
54
|
-
> `Cross-Origin-Opener-Policy: same-origin`
|
|
55
|
-
>
|
|
56
|
-
> `Cross-Origin-Embedder-Policy: require-corp`
|
|
57
|
-
|
|
58
|
-
```js
|
|
59
|
-
import { sqlite3Worker1Promiser } from '@evolu/sqlite-wasm';
|
|
60
|
-
|
|
61
|
-
const log = console.log;
|
|
62
|
-
const error = console.error;
|
|
63
|
-
|
|
64
|
-
const initializeSQLite = async () => {
|
|
65
|
-
try {
|
|
66
|
-
log('Loading and initializing SQLite3 module...');
|
|
67
|
-
|
|
68
|
-
const promiser = await new Promise((resolve) => {
|
|
69
|
-
const _promiser = sqlite3Worker1Promiser({
|
|
70
|
-
onready: () => resolve(_promiser),
|
|
71
|
-
});
|
|
72
|
-
});
|
|
73
|
-
|
|
74
|
-
log('Done initializing. Running demo...');
|
|
75
|
-
|
|
76
|
-
const configResponse = await promiser('config-get', {});
|
|
77
|
-
log('Running SQLite3 version', configResponse.result.version.libVersion);
|
|
78
|
-
|
|
79
|
-
const openResponse = await promiser('open', {
|
|
80
|
-
filename: 'file:mydb.sqlite3?vfs=opfs',
|
|
81
|
-
});
|
|
82
|
-
const { dbId } = openResponse;
|
|
83
|
-
log(
|
|
84
|
-
'OPFS is available, created persisted database at',
|
|
85
|
-
openResponse.result.filename.replace(/^file:(.*?)\?vfs=opfs$/, '$1'),
|
|
86
|
-
);
|
|
87
|
-
// Your SQLite code here.
|
|
88
|
-
} catch (err) {
|
|
89
|
-
if (!(err instanceof Error)) {
|
|
90
|
-
err = new Error(err.result.message);
|
|
91
|
-
}
|
|
92
|
-
error(err.name, err.message);
|
|
93
|
-
}
|
|
94
|
-
};
|
|
95
|
-
|
|
96
|
-
initializeSQLite();
|
|
97
|
-
```
|
|
21
|
+
The catch is that only one tab or worker can have a database open at a time. As SQLite documents for opfs-sahpool, one pool may use a directory at a time, across workers and wasm instances. So opens of a directory must be serialized, and Evolu does that with its leader Web Lock. Otherwise, two openers of a new or empty directory can each create their own slots. On the next open, only one version of a file shows, and the other is kept hidden and never reused. We keep opfs-sahpool's slots byte for byte, so existing databases keep working. A database uses the pool only through the `SqliteVfs` and `SqliteEncryptingVfs` interfaces. So if Evolu ever needs another VFS, such as one with an OPFS file per database, new databases can be created on it while existing ones keep working in the pool, without a migration.
|
|
98
22
|
|
|
99
|
-
|
|
100
|
-
[Worker1 API](https://sqlite.org/wasm/doc/trunk/api-worker1.md#worker1-methods).
|
|
23
|
+
Our pool VFS encrypts databases itself, in TypeScript, with Paul Miller's [`@awasm/noble`](https://github.com/paulmillr/awasm-noble). It provides synchronous WebAssembly ciphers and hashes without dependencies. Its noble backend, which wraps the audited `@noble/ciphers` and `@noble/hashes`, can replace the default one. Each worker that encrypts gets the two fixed-size WebAssembly memories of the default backend: 10 MiB for AES-CBC and 20 MiB for SHA-512.
|
|
101
24
|
|
|
102
|
-
|
|
25
|
+
The format is SQLCipher 4's as the `sqlcipher` scheme of SQLite3 Multiple Ciphers writes it. So the databases `@evolu/web` 3 encrypted with `@evolu/sqlite-wasm` 2.2.4 open unchanged, and 2.2.4 opens what we write. Every page, also in a rollback journal, is encrypted with AES-256-CBC and a random IV and authenticated with HMAC-SHA512. The key never reaches SQLite. `src/SahPool.ts` documents the format. `@evolu/web` 1.0.1-preview.6 to 2.4.0 encrypted databases with `PRAGMA legacy = 4`, SQLCipher 4's own format with page 1 encrypted from byte 16. Those databases have not opened since `@evolu/web` 3.0.0, and they don't open here either.
|
|
103
26
|
|
|
104
|
-
> [
|
|
105
|
-
>
|
|
106
|
-
> For this to work, you need to set the following headers on your server:
|
|
107
|
-
>
|
|
108
|
-
> `Cross-Origin-Opener-Policy: same-origin`
|
|
109
|
-
>
|
|
110
|
-
> `Cross-Origin-Embedder-Policy: require-corp`
|
|
27
|
+
`@evolu/web` passed 2.2.4 the key as the SQL text `x'<hex>'`, SQLCipher's notation for a raw key. SQLite3 Multiple Ciphers 2.2.4 treated it as a passphrase because of a [bug](https://github.com/utelle/SQLite3MultipleCiphers/issues/218), fixed in its 2.2.5. So the databases `@evolu/web` 3 encrypted with 2.2.4 open with the key derived from that text, as before. They are never rekeyed automatically. A database we create is encrypted with the key itself, which 2.2.4 opens only when given the key in its raw-key notation, `raw:<hex>`.
|
|
111
28
|
|
|
112
|
-
|
|
113
|
-
// In `main.js`.
|
|
114
|
-
const worker = new Worker('worker.js', { type: 'module' });
|
|
115
|
-
```
|
|
29
|
+
Only a database file and its journal are encrypted. 2.2.4 encrypted the files `VACUUM INTO` and `ATTACH` write with the database's key, but SQLite ignores the `KEY` clause of `ATTACH` and `PRAGMA key`, `rekey` and `cipher`. So here, `VACUUM INTO` and `ATTACH` write an unencrypted file unless a key is registered for its path with the pool's `registerKey` while they open it. A file that `ATTACH` creates can't be encrypted, because SQLite reserves no bytes in it. But `VACUUM INTO` can create an encrypted copy, which `ATTACH` then opens with its key registered. An encrypted database keeps `secure_delete` on, as 2.2.4 turned it on. Unlike 2.2.4, it refuses to turn it off, because a freed page SQLite doesn't write would fail to authenticate. Our tests compare the pool with 2.2.4 itself, in Node.js and in Chromium, Firefox and WebKit, and with an implementation of the format on `node:crypto`.
|
|
116
30
|
|
|
117
|
-
|
|
118
|
-
// In `worker.js`.
|
|
119
|
-
import sqlite3InitModule from '@evolu/sqlite-wasm';
|
|
120
|
-
|
|
121
|
-
const log = console.log;
|
|
122
|
-
const error = console.error;
|
|
123
|
-
|
|
124
|
-
const start = (sqlite3) => {
|
|
125
|
-
log('Running SQLite3 version', sqlite3.version.libVersion);
|
|
126
|
-
const db =
|
|
127
|
-
'opfs' in sqlite3
|
|
128
|
-
? new sqlite3.oo1.OpfsDb('/mydb.sqlite3')
|
|
129
|
-
: new sqlite3.oo1.DB('/mydb.sqlite3', 'ct');
|
|
130
|
-
log(
|
|
131
|
-
'opfs' in sqlite3
|
|
132
|
-
? `OPFS is available, created persisted database at ${db.filename}`
|
|
133
|
-
: `OPFS is not available, created transient database ${db.filename}`,
|
|
134
|
-
);
|
|
135
|
-
// Your SQLite code here.
|
|
136
|
-
};
|
|
137
|
-
|
|
138
|
-
const initializeSQLite = async () => {
|
|
139
|
-
try {
|
|
140
|
-
log('Loading and initializing SQLite3 module...');
|
|
141
|
-
const sqlite3 = await sqlite3InitModule({ print: log, printErr: error });
|
|
142
|
-
log('Done initializing. Running demo...');
|
|
143
|
-
start(sqlite3);
|
|
144
|
-
} catch (err) {
|
|
145
|
-
error('Initialization error:', err.name, err.message);
|
|
146
|
-
}
|
|
147
|
-
};
|
|
148
|
-
|
|
149
|
-
initializeSQLite();
|
|
150
|
-
```
|
|
31
|
+
With encryption in the pool, the binary no longer contains [SQLite3 Multiple Ciphers](https://github.com/utelle/SQLite3MultipleCiphers), which 2.2.4 encrypts with. It's plain SQLite: SQLite's own C from sqlite.org, compiled with SQLite's own makefile, plus Emscripten's output. There's no other C. In SQLite's full-featured build of 3.53.4, here is what leaving out SQLite3 Multiple Ciphers 2.5.1 gives:
|
|
151
32
|
|
|
152
|
-
The `
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
import sqlite3InitModule from '@evolu/sqlite-wasm';
|
|
159
|
-
|
|
160
|
-
const log = console.log;
|
|
161
|
-
const error = console.error;
|
|
162
|
-
|
|
163
|
-
const start = (sqlite3) => {
|
|
164
|
-
log('Running SQLite3 version', sqlite3.version.libVersion);
|
|
165
|
-
const db = new sqlite3.oo1.DB('/mydb.sqlite3', 'ct');
|
|
166
|
-
// Your SQLite code here.
|
|
167
|
-
};
|
|
168
|
-
|
|
169
|
-
const initializeSQLite = async () => {
|
|
170
|
-
try {
|
|
171
|
-
log('Loading and initializing SQLite3 module...');
|
|
172
|
-
const sqlite3 = await sqlite3InitModule({
|
|
173
|
-
print: log,
|
|
174
|
-
printErr: error,
|
|
175
|
-
});
|
|
176
|
-
log('Done initializing. Running demo...');
|
|
177
|
-
start(sqlite3);
|
|
178
|
-
} catch (err) {
|
|
179
|
-
error('Initialization error:', err.name, err.message);
|
|
180
|
-
}
|
|
181
|
-
};
|
|
182
|
-
|
|
183
|
-
initializeSQLite();
|
|
184
|
-
```
|
|
33
|
+
- The C source the build compiles is 100,241 lines shorter. It's SQLite's 269,649-line `sqlite3.c` instead of the 369,890-line SQLite3 Multiple Ciphers amalgamation. About 72,500 of the extra lines were third-party code: its cipher schemes, their cryptography, and bundled libraries such as AEGIS, miniz and Argon2. The rest were SQLite's own code: a second copy of `sqlite3.h`, `sqlite3ext.h`, `tclsqlite.c` and extensions from `ext/misc`.
|
|
34
|
+
- Our encryption instead is 975 lines of TypeScript, 641 without comments and blank lines. In `src/SahPool.ts`, it's 686 lines: the codec at the end of the file, 381 lines, and the parts of the VFS that call it. In `src/Database.ts`, it's 289 lines that declare what an encrypting VFS does, open encrypted connections and fall back to the key 2.2.4 derived. The ciphers and hashes come from `@awasm/noble`.
|
|
35
|
+
- The wasm went from 1,068,015 to 868,886 bytes, and from 473,354 to 401,985 bytes with `gzip -9`. The `@awasm/noble` modules the pool encrypts with add 150,697 bytes of JavaScript minified by esbuild, and 51,843 with `gzip -9`.
|
|
36
|
+
- The wasm exports 216 public C functions instead of 232. The 11 `sqlite3mc_*` functions are gone. So is the API of SQLite's encryption extension that SQLite3 Multiple Ciphers implemented: `sqlite3_key`, `sqlite3_key_v2`, `sqlite3_rekey`, `sqlite3_rekey_v2` and `sqlite3_activate_see`.
|
|
37
|
+
|
|
38
|
+
There's also less to trust. 2.2.4 repackaged a WebAssembly binary another project built, with that project's cipher code and bundled libraries inside. We build ours in CI from SQLite's source, pinned by the hash sqlite.org publishes, and the build reproduces the same bytes. The package's only runtime dependency, `@awasm/noble`, has no dependencies of its own. It comes from Paul Miller, whose `@noble/ciphers`, `@noble/hashes` and `@scure/bip39` Evolu's other cryptography already uses, so it adds no new party to trust. We pin it to an exact version, because any release can regenerate its WebAssembly with a new version of its compiler. We upgrade it only on purpose, after our tests pass with the new binaries. The package also has a peer dependency, `@evolu/common`, Evolu's own, whose `Result` and `Task` its API returns. 2.2.4 had none. That leaves less code to audit and fewer places for a supply-chain attack to hide.
|
|
185
39
|
|
|
186
|
-
The `
|
|
187
|
-
[Object Oriented API #1](https://sqlite.org/wasm/doc/trunk/api-oo1.md).
|
|
40
|
+
We don't ship that full-featured build, though. We ship SQLite's bare-bones configuration, chosen with the makefile's own variables. It comes with JSON, which Evolu queries, and the math functions, which apps can call through Evolu's query builder. It also keeps temporary files, such as sorts, temporary tables and statement journals, in memory. It omits WAL, the authorizer, the progress callback, incremental blob I/O and the introspection pragmas. Evolu's own SQL tests pass on it as they do on better-sqlite3. The C API covers only what we build. FTS5, R\*Tree, the session extension, the preupdate hook, the column metadata functions `sqlite3_column_database_name`, `sqlite3_column_table_name` and `sqlite3_column_origin_name`, and the registration of virtual tables and window functions aren't there. If you need one of them, start a [discussion](https://github.com/evoluhq/evolu/discussions). Compared with the full-featured build without SQLite3 Multiple Ciphers:
|
|
188
41
|
|
|
189
|
-
|
|
42
|
+
- Less C is compiled. Thirteen files of `sqlite3.c`, 49,818 of its 269,649 lines, contribute nothing, among them `fts5.c`, `sqlite3session.c`, `wal.c` and `rtree.c`. After the preprocessor, the C the build compiles has 87,381 non-blank lines instead of 119,158.
|
|
43
|
+
- The wasm went from 868,886 to 649,755 bytes, from 401,985 to 298,537 with `gzip -9`, and from 348,808 to 259,521 with Brotli at quality 11. With SQLite3 Multiple Ciphers, the full-featured build was 1,068,015 bytes, 473,354 with `gzip -9`. The stack overflow check we add, described below, makes the binary we ship 669,403 bytes, 302,491 with `gzip -9` and 262,026 with Brotli.
|
|
44
|
+
- The wasm exports 156 public C functions instead of 216, and imports 32 functions instead of 35, besides its memory and the stack overflow check's handler.
|
|
190
45
|
|
|
191
|
-
|
|
192
|
-
config option in `vite.config.js`:
|
|
46
|
+
Evolu opens one connection per database and runs one operation at a time. There's no `SQLITE_BUSY` to handle and no interleaving to reason about. More connections in one worker wouldn't add parallelism anyway. They all run on the same JavaScript thread, and with a rollback journal, a reader still blocks another connection's commit. And because every call is synchronous, a connection that gets `SQLITE_BUSY` can't even wait for the other one to finish. It can only yield and retry, which is serializing with extra steps. We prefer this simple mental model, but we're open to revisiting it if a real use case appears. Feel free to start a [discussion](https://github.com/evoluhq/evolu/discussions).
|
|
193
47
|
|
|
194
|
-
|
|
195
|
-
import { defineConfig } from 'vite';
|
|
48
|
+
We watch upstream changes to SQLite's JavaScript and incorporate them after our own audit. Our loader also depends on the functions the WebAssembly imports and exports, and those come from the Emscripten version and SQLite's build settings. The generator pins the imports and exports with their wasm types, together with SQLite's constants and struct layouts. A check fails whenever a new build differs in any of them, so every toolchain upgrade gets the same review. The loader refuses a binary whose constants or export names differ from the pinned ones, or that needs an import it doesn't implement. Neither the check nor the loader compares code or build settings that change none of these, such as `SQLITE_TEMP_STORE`. A test pins those with `PRAGMA compile_options`.
|
|
196
49
|
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
});
|
|
50
|
+
## The WebAssembly
|
|
51
|
+
|
|
52
|
+
We build the `.wasm` ourselves with `scripts/build-wasm.mts`. It's SQLite 3.53.4 from sqlite.org's source zip, unpatched, compiled by Emscripten 6.0.3 with SQLite's own makefile, in its bare-bones configuration with the options above, plus Emscripten's stack overflow check. The source is pinned by the SHA3-256 sqlite.org publishes. Emscripten is pinned by its version, and in CI by the image digest too. The output's sha256 covers the rest: the build fails unless it equals the pinned one and the wasm matches the generator's pins. Two builds in fresh work directories gave the same bytes. The `SQLite Wasm` job of the Checks workflow builds it in CI, in the official Emscripten image pinned by digest. When no input changed, the job restores the wasm from a cache instead. Either way, it hands the wasm to the jobs that test it. Built files are never committed.
|
|
53
|
+
|
|
54
|
+
The tests and playgrounds of the Evolu repository load `wasm/sqlite3.wasm` and fail when it's missing or isn't the pinned wasm. To get it without building it, run `pnpm sqlite-wasm:download` in the repository root. It downloads the published package that `scripts/download-wasm.mts` names and writes the package's wasm only if its sha256 is the pinned one. When the pin changes, no published package holds the new wasm until the next release, so the download says to build it. We set that release's version and sha256 in `scripts/download-wasm.mts`, sometimes before it's published, and the download then fails with HTTP 404 until it is. The examples load the wasm too, but they don't check it, because they're templates to copy.
|
|
55
|
+
|
|
56
|
+
To build or verify it, use an [emsdk](https://emscripten.org/docs/getting_started/downloads.html) that has Emscripten 6.0.3 installed and activated. If the build succeeds, it has reproduced the pinned bytes:
|
|
57
|
+
|
|
58
|
+
```sh
|
|
59
|
+
EMSDK=/path/to/emsdk node scripts/build-wasm.mts
|
|
208
60
|
```
|
|
209
61
|
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
62
|
+
SQLite's makefile puts the 512 KiB C stack directly above SQLite's static data. In SQLite's own build, SQL that is legal but nests deeply, such as a chain of 600 triggers where SQLite allows 1,000, runs the stack into that data and overwrites it without an error. With 2.2.4, such an insert never returned. Built without the check, our binary was left failing with `SQLITE_NOMEM`, even to open another database. So we build with `-sSTACK_OVERFLOW_CHECK=2`. With it, every function compares the stack pointer it sets with the stack's limits, and our loader turns an overflow into a `WebAssembly.RuntimeError`, which breaks the instance as any trap does. The check adds 19,648 bytes, 3,954 with `gzip -9`. In Node.js 24 on an Apple M5, our micro benchmarks of `run` and `exec` on an in-memory database take 1% to 5% longer with it.
|
|
63
|
+
|
|
64
|
+
## Bundlers
|
|
213
65
|
|
|
214
|
-
|
|
66
|
+
`sqliteWasmUrl` is `new URL("../wasm/sqlite3.wasm", import.meta.url)`. Vite, webpack, Next.js and other bundlers recognize it. They emit the binary with the app and give the URL of their copy. Pass `fetch(sqliteWasmUrl)` to `createSqliteWasm`. It compiles the binary while it downloads if the server sends it as exactly `application/wasm`, without parameters, which is the type `WebAssembly.compileStreaming` requires. Otherwise, it compiles the binary from its bytes once it has downloaded.
|
|
215
67
|
|
|
216
|
-
|
|
217
|
-
examples of how to use this in the main thread and in a worker. (Note that the
|
|
218
|
-
worker variant requires special HTTP headers, so it can't be hosted on GitHub
|
|
219
|
-
Pages.) An example that shows how to use this with vite is available on
|
|
220
|
-
[StackBlitz](https://stackblitz.com/edit/vitejs-vite-ttrbwh?file=main.js).
|
|
68
|
+
Vite doesn't process the URL in a dependency it prebundles, so add `@evolu/sqlite-wasm` to `optimizeDeps.exclude`. Vite also emits the binary whenever the app imports `@evolu/sqlite-wasm`, even when the app loads its own binary instead, so a PWA that precaches `*.wasm` downloads it too. webpack emits it only when the app uses `sqliteWasmUrl`.
|
|
221
69
|
|
|
222
|
-
##
|
|
70
|
+
## Versions
|
|
223
71
|
|
|
224
|
-
|
|
225
|
-
[npm dependents](https://www.npmjs.com/browse/depended/@evolu/sqlite-wasm) for
|
|
226
|
-
this package.
|
|
72
|
+
As in SQLite's own npm package, the version is the version of SQLite the package builds, followed by `-build<n>`. `3.53.4-build1` is our first build of SQLite 3.53.4, `3.53.4-build2` the next one, and a new SQLite release starts again at `-build1`. The build number counts our releases, and a release may also change the TypeScript API. Pin the exact version and read the release notes before upgrading. npm treats `-build<n>` as a prerelease, so a range such as `^3.53.4-build1` never reaches a build of another SQLite release. Semver compares `build10` with `build9` as text, so a tenth build would sort below the ninth. The repository's `scripts/version-sqlite-wasm.mts` refuses it, so a SQLite release gets at most nine builds, and a tenth needs a new SQLite pin.
|
|
227
73
|
|
|
228
|
-
##
|
|
74
|
+
## Credits
|
|
229
75
|
|
|
230
|
-
(
|
|
76
|
+
- [SQLite](https://sqlite.org), public domain.
|
|
77
|
+
- [`@awasm/noble`](https://github.com/paulmillr/awasm-noble) by Paul Miller, MIT License, which encrypts databases.
|
|
78
|
+
- [Emscripten](https://emscripten.org), MIT License, which compiles the WebAssembly.
|
|
231
79
|
|
|
232
|
-
|
|
233
|
-
[SQLite version number](https://sqlite.org/download.html) and add a build
|
|
234
|
-
identifier suffix like `-build1`. The complete version number should read
|
|
235
|
-
something like `3.41.2-build1`.
|
|
236
|
-
1. Run `npm run build` to build the ES Module. This downloads the latest SQLite
|
|
237
|
-
Wasm binary and builds the ES Module.
|
|
238
|
-
1. Run `npm run deploy` to commit the changes, push to GitHub, and publish the
|
|
239
|
-
new version to npm.
|
|
80
|
+
## Documentation
|
|
240
81
|
|
|
241
|
-
|
|
82
|
+
For detailed information and usage examples, please visit [evolu.dev](https://www.evolu.dev).
|
|
242
83
|
|
|
243
|
-
|
|
84
|
+
## Community
|
|
244
85
|
|
|
245
|
-
|
|
86
|
+
The Evolu community is on [GitHub Discussions](https://github.com/evoluhq/evolu/discussions), where you can ask questions and voice ideas.
|
|
246
87
|
|
|
247
|
-
|
|
248
|
-
conveniently wraps as an ES Module and publishes to npm as
|
|
249
|
-
[`@sqlite.org/sqlite-wasm`](https://www.npmjs.com/package/@sqlite.org/sqlite-wasm).
|
|
88
|
+
To chat with other community members, you can join the [Evolu Discord](https://discord.gg/2J8yyyyxtZ).
|
|
250
89
|
|
|
251
|
-
|
|
252
|
-
[SQLite3MultipleCiphers](https://github.com/utelle/SQLite3MultipleCiphers),
|
|
253
|
-
which is used to support multiple ciphers.
|
|
90
|
+
[](https://x.com/evoluhq)
|