spd-lib 1.3.1 → 1.3.3

package/README.md

@@ -0,0 +1,396 @@
+# spd-lib-ts
+
+**SPD (Secure Packaged Data)** — a post-quantum-hardened encrypted data format for Node.js.
+
+Encrypt any JavaScript value (strings, numbers, objects, typed arrays, Maps, Sets, Dates, etc.) into a compressed, authenticated, tamper-proof container. Supports file storage, base64 string transport, chunked internet transfer, and streaming I/O for files larger than 2 GB.
+
+---
+
+## Security model
+
+| Layer | Algorithm | Post-quantum strength |
+|---|---|---|
+| Key derivation | Argon2id (128 MiB, 6 iterations) | Raises brute-force cost against quantum search |
+| Encryption | XChaCha20-Poly1305 (256-bit key) | 128-bit PQ security via Grover's algorithm |
+| Authentication | HMAC-SHA3-512 (256-bit key, domain-separated) | 128-bit PQ security |
+| Salt wrapping | Argon2id-derived key + XChaCha20-Poly1305 | Same as above |
+
+Keys are domain-separated: the 512-bit Argon2id master secret is split — the first 32 bytes go to encryption, the last 32 bytes go to authentication. They are never used interchangeably.
+
+---
+
+## Install
+
+```sh
+npm install spd-lib-ts
+```
+
+> **Requires Node.js ≥ 18** and a C++ build toolchain for the `argon2` native module.
+
+---
+
+## Quick start
+
+```ts
+import { SPD } from 'spd-lib-ts';
+
+const spd = new SPD();
+await spd.setPassKey('MyStr0ng!Passphrase#2024');
+
+await spd.addData('username', 'alice');
+await spd.addData('apiKey', 'sk-abc123');
+await spd.addData('config', { theme: 'dark', retries: 3 });
+
+// Save to file
+await spd.saveToFile('./vault.spd', 'MyStr0ng!Passphrase#2024');
+
+// Load from file
+const loaded = await SPD.loadFromFile('./vault.spd', 'MyStr0ng!Passphrase#2024');
+const data = await loaded.extractData();
+console.log(data.username); // 'alice'
+```
+
+---
+
+## API reference
+
+### `new SPD()`
+
+Creates a new empty SPD instance. You must call `setPassKey()` before adding or saving data.
+
+---
+
+### `spd.setPassKey(passcode: string): Promise<void>`
+
+Derives encryption and authentication keys from the passcode using Argon2id. Must be called before any data operation.
+
+**Passcode requirements:** minimum 12 characters, must contain at least 3 of: lowercase, uppercase, digits, special characters.
+
+```ts
+await spd.setPassKey('MyStr0ng!Passphrase#2024');
+```
+
+---
+
+### `spd.addData(name: string, value: unknown): Promise<void>`
+
+Encrypts and stores a single value. The name is sanitized (lowercased, non-alphanumeric chars replaced with `_`).
+
+**Supported types:** `string`, `number`, `boolean`, `null`, `object`, `Array`, `Uint8Array`, `Uint16Array`, `Uint32Array`, `BigInt64Array`, `BigUint64Array`, `Float32Array`, `Float64Array`, `Map`, `Set`, `Date`, `RegExp`, `Error`
+
+```ts
+await spd.addData('score', 42);
+await spd.addData('tags', ['a', 'b', 'c']);
+await spd.addData('created', new Date());
+await spd.addData('raw', new Uint8Array([1, 2, 3]));
+await spd.addData('lookup', new Map([['key', 'val']]));
+```
+
+---
+
+### `spd.addMany(items: { name: string; data: unknown }[]): Promise<void>`
+
+Batch version of `addData`. All items are encrypted in parallel.
+
+```ts
+await spd.addMany([
+  { name: 'firstName', data: 'Alice' },
+  { name: 'age',       data: 30 },
+  { name: 'prefs',     data: { darkMode: true } },
+]);
+```
+
+---
+
+### `spd.extractData(): Promise<Record<string, unknown>>`
+
+Decrypts and returns all stored entries as a plain object.
+
+```ts
+const data = await spd.extractData();
+console.log(data.firstName); // 'Alice'
+```
+
+---
+
+### `spd.saveToFile(path: string, passcode: string): Promise<void>`
+
+Saves the encrypted payload to a file (mode `0600`). Suitable for files up to ~1–2 GB.
+
+```ts
+await spd.saveToFile('./secrets.spd', 'MyStr0ng!Passphrase#2024');
+```
+
+---
+
+### `SPD.loadFromFile(path: string, passcode: string): Promise<SPD>`
+
+Loads and verifies an SPD file. The hash algorithm and Argon2 parameters are read from the file itself — no need to pass them.
+
+```ts
+const spd = await SPD.loadFromFile('./secrets.spd', 'MyStr0ng!Passphrase#2024');
+```
+
+---
+
+### `spd.saveToFileStreaming(path: string, passcode: string): Promise<void>`
+### `SPD.loadFromFileStreaming(path: string, passcode: string): Promise<SPD>`
+
+Streaming variants for files larger than 2 GB. Uses a binary wire format with 64 MB write chunks — heap usage stays bounded regardless of file size.
+
+```ts
+await spd.saveToFileStreaming('./large.spd', 'MyStr0ng!Passphrase#2024');
+const spd = await SPD.loadFromFileStreaming('./large.spd', 'MyStr0ng!Passphrase#2024');
+```
+
+---
+
+### `spd.extractDataStreaming(tmpDir: string, callback): Promise<void>`
+
+Memory-efficient extraction using `.ssf` skeleton files. For each encrypted entry:
+
+1. Decrypts + decompresses the entry into `<tmpDir>/<name>.ssf` (mode `0600`)
+2. Reads the `.ssf` file and calls your callback with the name and value
+3. Deletes the `.ssf` file before moving to the next entry
+
+At most one entry's bytes exist on disk or in RAM at a time.
+
+```ts
+import * as fs from 'fs';
+
+fs.mkdirSync('/tmp/spd_scratch', { recursive: true });
+
+await spd.extractDataStreaming('/tmp/spd_scratch', async (name, value) => {
+  console.log(name, value);
+});
+```
+
+> `tmpDir` must exist before calling. Mode `0700` is recommended.
+
+---
+
+### `SPD.extractFromFileStreaming(filePath, passcode, tmpDir, callback): Promise<void>`
+
+True constant-memory extraction from a binary `.spd` file (written by `saveToFileStreaming`). Processes one entry at a time using `.ssf` skeleton files — the full plaintext never lives in RAM.
+
+Steps per entry:
+1. Writes encrypted blob to `<name>_enc.ssf`
+2. Decrypts into `<name>_dec.ssf`
+3. Reads the `.ssf` file and calls your callback
+4. Deletes both `.ssf` files before the next entry
+
+Peak RAM usage is proportional to the largest single entry, not the file size.
+
+```ts
+await SPD.extractFromFileStreaming(
+  './huge.spd',
+  'MyStr0ng!Passphrase#2024',
+  '/tmp/spd_scratch',
+  async (name, value) => {
+    console.log(name, typeof value);
+  }
+);
+```
+
+---
+
+### `spd.saveData(passcode: string): Promise<string>`
+
+Serializes the payload to a base64 string for in-memory or network transport.
+
+```ts
+const b64 = await spd.saveData('MyStr0ng!Passphrase#2024');
+```
+
+---
+
+### `SPD.loadFromString(data: string, passcode: string): Promise<SPD>`
+
+Loads from a base64 string produced by `saveData`.
+
+```ts
+const spd = await SPD.loadFromString(b64, 'MyStr0ng!Passphrase#2024');
+```
+
+---
+
+### Chunked internet transfer
+
+Split a payload into chunks for HTTP uploads or any transport with a body-size limit.
+
+```ts
+// Sender
+const chunks = await spd.saveDataChunked('MyStr0ng!Passphrase#2024', 512 * 1024); // 512 KB chunks
+// chunks is string[] — send each element, then the manifest (last element) last
+
+// Receiver — pass all chunks in order including the manifest
+const spd = await SPD.loadFromChunks(chunks, 'MyStr0ng!Passphrase#2024');
+```
+
+The last element of the array is always a JSON manifest (`{ totalChunks, chunkSize, totalBytes, version }`). The receiver validates chunk count and byte count before decrypting.
+
+Default chunk size is **1 MB**. Tune down for strict API body limits.
+
+---
+
+### `spd.changePasscode(oldPasscode: string, newPasscode: string): Promise<void>`
+
+Re-derives keys under the new passcode and re-encrypts all data. Verifies the old passcode via MAC before doing anything.
+
+```ts
+await spd.changePasscode('MyStr0ng!Passphrase#2024', 'EvenStr0nger!Pass#9999');
+```
+
+---
+
+### `spd.setHash(hash: 'sha3-512' | 'sha256' | 'sha512'): void`
+
+Sets the hash algorithm used for per-entry integrity checks. Default is `'sha3-512'`. Must be set before adding data. The chosen algorithm is embedded in the payload and restored automatically on load.
+
+```ts
+spd.setHash('sha3-512'); // default, recommended
+```
+
+---
+
+### `spd.setCompressionLevel(level: number): void`
+
+Sets the zlib deflate compression level (1–9). Default is `9` (maximum compression).
+
+```ts
+spd.setCompressionLevel(6); // faster, slightly larger
+```
+
+---
+
+### `spd.destroy()` / `spd.clearCache()`
+
+Securely zeros all key material in place using `sodium.memzero()`, then clears all stored data. Call this when you are done with a session.
+
+```ts
+spd.destroy();
+```
+
+---
+
+### `SPD.deriveKeys(passcode, salt)` / `SPD.derivePBK(passcode, salt)`
+
+Low-level static helpers exposed for advanced use cases (e.g. deriving keys outside of an SPD container).
+
+---
+
+## SPDVault — in-memory key vault
+
+`SPDVault` is a time-limited in-memory store for passcodes or keys. Keys expire automatically after a configurable timeout and are cleared on access renewal.
+
+```ts
+import { SPDVault } from 'spd-lib-ts';
+
+const vault = new SPDVault(300_000); // 5-minute TTL
+
+// Generate a random high-entropy key and store it
+vault.genKey('session');
+
+// Store your own key
+vault.pushKey('myKey', 'MyStr0ng!Passphrase#2024');
+
+// Retrieve (resets TTL)
+const key = vault.pullKey('myKey'); // 'MyStr0ng!Passphrase#2024'
+
+// Update (requires old value to match)
+vault.updateKey('myKey', 'MyStr0ng!Passphrase#2024', 'NewPass!99');
+
+// Delete one key
+vault.destroyKey('myKey');
+
+// Stop all timers (keys stay in memory)
+vault.stop();
+
+// Wipe everything
+vault.destroy();
+```
+
+Generated keys are 500–699 characters of cryptographically random characters drawn from a 91-character charset using rejection sampling (no modulo bias).
+
+---
+
+## SPDLegacy
+
+`SPDLegacy` is a backwards-compatible class for reading files produced by SPD v1.x. It uses `crypto_secretbox_easy` (XSalsa20-Poly1305) and PBKDF2 key derivation. **Do not use for new data** — migrate to `SPD` instead.
+
+```ts
+import { SPDLegacy } from 'spd-lib-ts';
+
+const spd = await SPDLegacy.loadFromFile('./old.spd', 'passcode');
+const data = await spd.extractData();
+```
+
+---
+
+## Full example — save and load a config
+
+```ts
+import { SPD } from 'spd-lib-ts';
+import * as path from 'path';
+
+const FILE = path.join(process.env.HOME!, '.myapp', 'config.spd');
+const PASS = process.env.APP_PASS!;
+
+async function saveConfig(config: Record<string, unknown>) {
+  const spd = new SPD();
+  await spd.setPassKey(PASS);
+  await spd.addMany(Object.entries(config).map(([name, data]) => ({ name, data })));
+  await spd.saveToFile(FILE, PASS);
+  spd.destroy();
+}
+
+async function loadConfig(): Promise<Record<string, unknown>> {
+  const spd = await SPD.loadFromFile(FILE, PASS);
+  const data = await spd.extractData();
+  spd.destroy();
+  return data;
+}
+```
+
+---
+
+## Full example — chunked HTTP upload
+
+```ts
+// server sends:
+const spd = new SPD();
+await spd.setPassKey(PASS);
+await spd.addData('payload', largeObject);
+const chunks = await spd.saveDataChunked(PASS, 256 * 1024); // 256 KB per chunk
+spd.destroy();
+
+for (let i = 0; i < chunks.length; i++) {
+  await fetch('https://example.com/upload', {
+    method: 'POST',
+    body: JSON.stringify({ index: i, chunk: chunks[i] }),
+  });
+}
+
+// receiver collects all chunks in order and loads:
+const spd = await SPD.loadFromChunks(receivedChunks, PASS);
+const data = await spd.extractData();
+spd.destroy();
+```
+
+---
+
+## Versioning
+
+| npm version | SPD format version | Notes |
+|---|---|---|
+| 1.3.1 | v26 | Hash algo + Argon2 params embedded in payload, secure key zeroing, `null` type support |
+| 1.3.0 | v25 | 512-bit Argon2id master secret, domain-separated AEAD + HMAC keys, HMAC-SHA3-512 |
+| < 1.3.0 | v24 | 256-bit Argon2id, single key for AEAD + MAC |
+
+SPD format versions are not cross-compatible. Use `SPDLegacy` to read v24 and earlier files.
+
+---
+
+## License
+
+ISC

package/index.d.mts

@@ -98,6 +98,29 @@ declare class SPD {
     addData(dataName: string, value: unknown): Promise<void>;
     addMany(items: DataInput[]): Promise<void>;
     extractData(): Promise<Record<string, unknown>>;
+    /**
+     * Memory-efficient streaming extraction.
+     *
+     * For each encrypted entry, this method:
+     *   1. Decrypts and decompresses the entry into a temp file in `tmpDir`
+     *      (named `<entryName>.ssf`, mode 0600)
+     *   2. Reads the .ssf file and calls your `callback` with the name + value
+     *   3. Immediately deletes the .ssf file before moving to the next entry
+     *
+     * At most one entry's decrypted bytes exist on disk or in RAM at a time,
+     * so this works for arbitrarily large SPD files as long as a single entry
+     * fits in memory (which is always the case — entries are discrete values).
+     *
+     * @param tmpDir   Directory to write temp files into (must exist, mode 0700 recommended)
+     * @param callback Called once per entry in order. Await is respected — the next
+     *                 entry won't be processed until your callback resolves.
+     *
+     * @example
+     * await spd.extractDataStreaming('/tmp/spd_scratch', async (name, value) => {
+     *   console.log(name, value);
+     * });
+     */
+    extractDataStreaming(tmpDir: string, callback: (name: string, value: unknown) => Promise<void> | void): Promise<void>;
     destroy(): void;
     clearCache(): void;
     saveToFile(outputPath: string, passcode: string): Promise<void>;
@@ -115,6 +138,33 @@ declare class SPD {
     saveDataChunked(passcode: string, chunkSize?: number): Promise<string[]>;
     static loadFromFile(filePath: string, passcode: string): Promise<SPD>;
     static loadFromFileStreaming(filePath: string, passcode: string): Promise<SPD>;
+    /**
+     * True constant-memory streaming extract from a binary SPD file.
+     *
+     * Unlike `loadFromFileStreaming` (which buffers the entire plaintext into RAM),
+     * this method pipes the inflate stream through a byte-by-byte parser that:
+     *   1. Reads and verifies the file header + MAC
+     *   2. Parses the metadata block to derive keys
+     *   3. Processes one entry at a time:
+     *      a. Writes the encrypted entry to `<name>_enc.ssf` in `tmpDir` (mode 0600)
+     *      b. Decrypts it, writes the plaintext to `<name>_dec.ssf`
+     *      c. Reads the `.ssf` file and calls your `callback`
+     *      d. Deletes both `.ssf` files before touching the next entry
+     *
+     * Peak RAM usage is proportional to the largest single entry, not the file size.
+     * Files of any size are supported as long as individual entries fit in memory.
+     *
+     * @param filePath Path to the `.spd` file written by `saveToFileStreaming`
+     * @param passcode Passcode used when the file was saved
+     * @param tmpDir   Scratch directory for temp files (must exist, mode 0700 recommended)
+     * @param callback Called once per entry in order. Next entry waits for this to resolve.
+     *
+     * @example
+     * await SPD.extractFromFileStreaming('./huge.spd', 'MyPass!', '/tmp/spd', async (name, value) => {
+     *   console.log(name, typeof value);
+     * });
+     */
+    static extractFromFileStreaming(filePath: string, passcode: string, tmpDir: string, callback: (name: string, value: unknown) => Promise<void> | void): Promise<void>;
     static loadFromString(data: string, passcode: string): Promise<SPD>;
     static loadFromChunks(chunks: string[], passcode: string): Promise<SPD>;
     static derivePBK(passcode: string, salt: Uint8Array, memory?: number, time?: number): Promise<PBKResult>;

package/index.d.ts

@@ -98,6 +98,29 @@ declare class SPD {
     addData(dataName: string, value: unknown): Promise<void>;
     addMany(items: DataInput[]): Promise<void>;
     extractData(): Promise<Record<string, unknown>>;
+    /**
+     * Memory-efficient streaming extraction.
+     *
+     * For each encrypted entry, this method:
+     *   1. Decrypts and decompresses the entry into a temp file in `tmpDir`
+     *      (named `<entryName>.ssf`, mode 0600)
+     *   2. Reads the .ssf file and calls your `callback` with the name + value
+     *   3. Immediately deletes the .ssf file before moving to the next entry
+     *
+     * At most one entry's decrypted bytes exist on disk or in RAM at a time,
+     * so this works for arbitrarily large SPD files as long as a single entry
+     * fits in memory (which is always the case — entries are discrete values).
+     *
+     * @param tmpDir   Directory to write temp files into (must exist, mode 0700 recommended)
+     * @param callback Called once per entry in order. Await is respected — the next
+     *                 entry won't be processed until your callback resolves.
+     *
+     * @example
+     * await spd.extractDataStreaming('/tmp/spd_scratch', async (name, value) => {
+     *   console.log(name, value);
+     * });
+     */
+    extractDataStreaming(tmpDir: string, callback: (name: string, value: unknown) => Promise<void> | void): Promise<void>;
     destroy(): void;
     clearCache(): void;
     saveToFile(outputPath: string, passcode: string): Promise<void>;
@@ -115,6 +138,33 @@ declare class SPD {
     saveDataChunked(passcode: string, chunkSize?: number): Promise<string[]>;
     static loadFromFile(filePath: string, passcode: string): Promise<SPD>;
     static loadFromFileStreaming(filePath: string, passcode: string): Promise<SPD>;
+    /**
+     * True constant-memory streaming extract from a binary SPD file.
+     *
+     * Unlike `loadFromFileStreaming` (which buffers the entire plaintext into RAM),
+     * this method pipes the inflate stream through a byte-by-byte parser that:
+     *   1. Reads and verifies the file header + MAC
+     *   2. Parses the metadata block to derive keys
+     *   3. Processes one entry at a time:
+     *      a. Writes the encrypted entry to `<name>_enc.ssf` in `tmpDir` (mode 0600)
+     *      b. Decrypts it, writes the plaintext to `<name>_dec.ssf`
+     *      c. Reads the `.ssf` file and calls your `callback`
+     *      d. Deletes both `.ssf` files before touching the next entry
+     *
+     * Peak RAM usage is proportional to the largest single entry, not the file size.
+     * Files of any size are supported as long as individual entries fit in memory.
+     *
+     * @param filePath Path to the `.spd` file written by `saveToFileStreaming`
+     * @param passcode Passcode used when the file was saved
+     * @param tmpDir   Scratch directory for temp files (must exist, mode 0700 recommended)
+     * @param callback Called once per entry in order. Next entry waits for this to resolve.
+     *
+     * @example
+     * await SPD.extractFromFileStreaming('./huge.spd', 'MyPass!', '/tmp/spd', async (name, value) => {
+     *   console.log(name, typeof value);
+     * });
+     */
+    static extractFromFileStreaming(filePath: string, passcode: string, tmpDir: string, callback: (name: string, value: unknown) => Promise<void> | void): Promise<void>;
     static loadFromString(data: string, passcode: string): Promise<SPD>;
     static loadFromChunks(chunks: string[], passcode: string): Promise<SPD>;
     static derivePBK(passcode: string, salt: Uint8Array, memory?: number, time?: number): Promise<PBKResult>;