keyframe-store 4.3.1 → 4.4.0

package/README.md

@@ -5,12 +5,14 @@
 [![NPM downloads][badge-npm-downloads]][url-npm]
 [![NPM Unpacked Size (with version)][badge-npm-unpacked-size]][url-npm]
 
-Content-addressed version storage for documents handed over whole. Every version is stored as a zstd **keyframe** — compressed on its own — or as a **delta** compressed against exactly one keyframe, using that keyframe's plaintext as the dictionary. Reconstructing any version is at most two reads, a version whose content is already held costs nothing, and the backend is anything that can store bytes by key.
+Content-addressed version storage for documents handed over whole, where every version is a zstd **keyframe**, compressed on its own, or a **delta** against a keyframe.
+
+Design notes live in [the specs](https://github.com/Esposter/Esposter/tree/main/openspec/specs/keyframe-store).
 
 ## Table of Contents
 
 - 🚀 [Getting Started](#getting-started)
-- 📖 [Documentation](#documentation)
+- 🧩 [Key Exports](#key-exports)
 - ⚖️ [License](#license)
 
 ---
@@ -64,44 +66,20 @@ const plaintext = await keyframeStore.read(second.hash).match(
 );
 ```
 
-## <a name="documentation">📖 Documentation</a>
-
-We highly recommend you take a look at the [documentation](https://github.com/Esposter/Esposter/tree/main/openspec/specs) to level up.
-
-### How it works
-
-- **Addressing.** An object's key is the hex SHA-256 of its plaintext, so two versions with identical content resolve to one object whichever keyframe either would have been encoded against. The store is write-once: a write of content it already holds records nothing and reports itself deduplicated.
-- **Encoding.** Each write compresses the plaintext standalone and, when the lineage has a keyframe, again with that keyframe's plaintext as the dictionary. The delta is kept only if it is at most the **promotion ratio** of the standalone size _and_ the bytes already anchored to the keyframe plus this delta fit the **segment budget**; otherwise the standalone form is written and this version becomes the lineage's next keyframe.
-- **Reading.** A keyframe is one read and one decompression; a delta fetches the keyframe its own header names, decompresses it, then decompresses the delta against it. Every result is hashed and compared with the key, so a truncated, altered or misfiled object is a refused read rather than a plausible document.
-- **Collection.** An object survives while any record names it, as its own hash or as its base. The caller answers that from its own records — `collect` takes the hashes an eviction released and the hashes still named, deletes the difference, and returns what it deleted.
-- **The anchor is the caller's.** The store holds no notion of a lineage: a caller derives one anchor per history — the newest keyframe's hash, and the bytes anchored to it — and resets both whenever a write reports an empty base.
-
-### Key exports
+## <a name="key-exports">🧩 Key Exports</a>
 
 | Export                 | Role                                                                                                                                                                      |
 | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `createKeyframeStore`  | The factory, taking an `ObjectStore` and optional `KeyframeStoreOptions`                                                                                                  |
-| `ObjectStore`          | The backend interface — `read`, `write`, `delete` by key                                                                                                                  |
 | `KeyframeStore`        | `write(plaintext, anchor)`, `read(hash)`, `collect(released, retained)` — every one a `ResultAsync`                                                                       |
+| `KeyframeStoreOptions` | `compressionLevel`, `promotionRatio`, `segmentBudgetRatio`, each defaulted from `constants` — the level from `@esposter/shared`'s, which the app's browser encoder shares |
+| `ObjectStore`          | The backend interface — `read`, `write`, `delete` by key                                                                                                                  |
 | `VersionAnchor`        | What a lineage passes to a write: its keyframe's hash and the bytes anchored to it                                                                                        |
 | `WrittenVersion`       | What a write reports: the hash, its base, both sizes and whether it was deduplicated                                                                                      |
-| `KeyframeStoreOptions` | `compressionLevel`, `promotionRatio`, `segmentBudgetRatio`, each defaulted from `constants` — the level from `@esposter/shared`'s, which the app's browser encoder shares |
-
-### Commands
-
-Run from `packages/keyframe-store/`:
-
-```bash
-pnpm build        # compile to dist/
-pnpm test         # vitest watch mode (coverage is run from the repo root)
-pnpm bench        # the codec benchmark, writing the colocated *.bench.md
-pnpm lint:fix     # auto-fix lint
-pnpm typecheck    # type check
-```
 
 ## <a name="license">⚖️ License</a>
 
-This project is licensed under the [Apache-2.0 license](https://github.com/Esposter/Esposter/blob/main/LICENSE).
+Licensed under [Apache-2.0][url-license].
 
 [badge-license]: https://img.shields.io/github/license/Esposter/Esposter.svg?color=blue
 [url-license]: https://github.com/Esposter/Esposter/blob/main/LICENSE

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "keyframe-store",
-  "version": "4.3.1",
+  "version": "4.4.0",
   "description": "Content-addressed version storage — every version is a zstd keyframe or a delta against one, over any backend that can store bytes by key.",
   "keywords": [
     "compression",
@@ -47,17 +47,17 @@
     "lint": "eslint .",
     "lint:fix": "eslint --fix .",
     "test": "vitest",
-    "typecheck": "tsc"
+    "typecheck": "tsrs --fast"
   },
   "dependencies": {
-    "@esposter/shared": "4.3.1",
+    "@esposter/shared": "4.4.0",
     "neverthrow": "^8.2.0"
   },
   "devDependencies": {
-    "@esposter/configuration": "4.3.1",
-    "@esposter/shared-node": "4.3.1",
-    "@types/node": "^26.6.5",
-    "tsdown": "^0.23.1"
+    "@esposter/configuration": "4.4.0",
+    "@esposter/shared-node": "4.4.0",
+    "@types/node": "^26.6.6",
+    "tsdown": "0.23.2"
   },
-  "gitHead": "777a39636b789c13af33ed13370246ccf6dbaf31"
+  "gitHead": "e1a416c4b91da69cc2aa259b50a4d656f204d272"
 }

package/dist/index.d.ts

@@ -1,82 +0,0 @@
-import { InvalidOperationError } from "@esposter/shared";
-import { ResultAsync } from "neverthrow";
-//#region src/constants.d.ts
-export declare const OBJECT_MAGIC: Uint8Array;
-export declare const OBJECT_FORMAT_VERSION = 1;
-export declare const OBJECT_VERSION_OFFSET = 4;
-export declare const OBJECT_FLAGS_OFFSET = 5;
-export declare const OBJECT_WINDOW_LOG_OFFSET = 6;
-export declare const OBJECT_HEADER_BYTE_COUNT = 8;
-export declare const KEYFRAME_FLAG = 0;
-export declare const DELTA_FLAG = 1;
-export declare const HASH_BYTE_COUNT = 32;
-export declare const DELTA_HEADER_BYTE_COUNT: number;
-export declare const DEFAULT_PROMOTION_RATIO: number;
-export declare const DEFAULT_SEGMENT_BUDGET_RATIO = 1;
-//#endregion
-//#region src/models/VersionAnchor.d.ts
-export interface VersionAnchor {
-  anchoredBytes: number;
-  hash: string;
-}
-//#endregion
-//#region src/models/WrittenVersion.d.ts
-export interface WrittenVersion {
-  baseHash: string;
-  hash: string;
-  isDeduplicated: boolean;
-  plaintextBytes: number;
-  storedBytes: number;
-}
-//#endregion
-//#region src/models/KeyframeStore.d.ts
-export interface KeyframeStore {
-  collect: (releasedHashes: string[], retainedHashes: string[]) => ResultAsync<string[], Error>;
-  read: (hash: string) => ResultAsync<Uint8Array, Error>;
-  write: (plaintext: Uint8Array, anchor: VersionAnchor) => ResultAsync<WrittenVersion, Error>;
-}
-//#endregion
-//#region src/models/KeyframeStoreOptions.d.ts
-export interface KeyframeStoreOptions {
-  compressionLevel?: number;
-  promotionRatio?: number;
-  segmentBudgetRatio?: number;
-}
-//#endregion
-//#region src/models/ObjectStore.d.ts
-export interface ObjectStore {
-  delete: (keys: string[]) => Promise<void>;
-  read: (key: string) => Promise<Uint8Array | undefined>;
-  write: (key: string, bytes: Uint8Array) => Promise<boolean>;
-}
-//#endregion
-//#region src/createKeyframeStore.d.ts
-export declare const createKeyframeStore: (objectStore: ObjectStore, { compressionLevel, promotionRatio, segmentBudgetRatio }?: KeyframeStoreOptions) => KeyframeStore;
-//#endregion
-//#region src/models/ObjectNotStoredError.d.ts
-export declare class ObjectNotStoredError extends InvalidOperationError {
-  constructor(hash: string, message: string);
-}
-//#endregion
-//#region src/models/ParsedObject.d.ts
-export interface ParsedObject {
-  baseHash: string;
-  payload: Uint8Array;
-  windowLog: number;
-}
-//#endregion
-//#region src/services/decodeObject.d.ts
-export declare const decodeObject: (parsedObject: ParsedObject, basePlaintext?: Uint8Array) => Promise<Buffer>;
-//#endregion
-//#region src/services/encodeObject.d.ts
-export declare const encodeObject: (plaintext: Uint8Array, compressionLevel: number, base?: {
-  hash: string;
-  plaintext: Uint8Array;
-}) => Promise<Uint8Array>;
-//#endregion
-//#region src/services/getContentAddress.d.ts
-export declare const getContentAddress: (plaintext: Uint8Array) => string;
-//#endregion
-//#region src/services/parseObject.d.ts
-export declare const parseObject: (hash: string, bytes: Uint8Array) => ParsedObject;
-//#endregion

package/dist/index.js

@@ -1,175 +0,0 @@
-import { DEFAULT_COMPRESSION_LEVEL, InvalidOperationError, Operation, getResultAsync, getWindowLog } from "@esposter/shared";
-import { promisify } from "node:util";
-import { constants, zstdCompress, zstdDecompress } from "node:zlib";
-import { createHash } from "node:crypto";
-//#region src/constants.ts
-const OBJECT_MAGIC = new Uint8Array([
-	75,
-	70,
-	83,
-	84
-]);
-const OBJECT_FORMAT_VERSION = 1;
-const OBJECT_VERSION_OFFSET = 4;
-const OBJECT_FLAGS_OFFSET = 5;
-const OBJECT_WINDOW_LOG_OFFSET = 6;
-const OBJECT_HEADER_BYTE_COUNT = 8;
-const KEYFRAME_FLAG = 0;
-const DELTA_FLAG = 1;
-const HASH_BYTE_COUNT = 32;
-const DELTA_HEADER_BYTE_COUNT = 40;
-const DEFAULT_PROMOTION_RATIO = 1 / 3;
-const DEFAULT_SEGMENT_BUDGET_RATIO = 1;
-//#endregion
-//#region src/models/ObjectNotStoredError.ts
-var ObjectNotStoredError = class extends InvalidOperationError {
-	constructor(hash, message) {
-		super(Operation.Read, hash, message);
-		this.name = "ObjectNotStoredError";
-	}
-};
-//#endregion
-//#region src/services/decodeObject.ts
-const decompress = promisify(zstdDecompress);
-const decodeObject = (parsedObject, basePlaintext) => decompress(parsedObject.payload, {
-	...basePlaintext ? { dictionary: basePlaintext } : {},
-	params: { [constants.ZSTD_d_windowLogMax]: parsedObject.windowLog }
-});
-//#endregion
-//#region src/services/encodeObject.ts
-const compress = promisify(zstdCompress);
-const encodeObject = async (plaintext, compressionLevel, base) => {
-	const windowLog = getWindowLog(base?.plaintext.byteLength ?? 0, plaintext.byteLength);
-	const payload = await compress(plaintext, {
-		...base ? { dictionary: base.plaintext } : {},
-		params: {
-			[constants.ZSTD_c_compressionLevel]: compressionLevel,
-			[constants.ZSTD_c_windowLog]: windowLog
-		}
-	});
-	const headerByteCount = base ? 40 : 8;
-	const bytes = new Uint8Array(headerByteCount + payload.byteLength);
-	bytes.set(OBJECT_MAGIC);
-	bytes[4] = 1;
-	bytes[5] = base ? 1 : 0;
-	bytes[6] = windowLog;
-	if (base) bytes.set(Buffer.from(base.hash, "hex").subarray(0, 32), 8);
-	bytes.set(payload, headerByteCount);
-	return bytes;
-};
-//#endregion
-//#region src/services/getContentAddress.ts
-const getContentAddress = (plaintext) => createHash("sha256").update(plaintext).digest("hex");
-//#endregion
-//#region src/services/parseObject.ts
-const parseObject = (hash, bytes) => {
-	if (!(bytes.byteLength >= 8 && OBJECT_MAGIC.every((byte, index) => bytes[index] === byte))) throw new InvalidOperationError(Operation.Read, hash, "not a keyframe store object");
-	else if (bytes[4] !== 1) throw new InvalidOperationError(Operation.Read, hash, `unknown object format version ${bytes[4]}`);
-	const windowLog = bytes[6] ?? 0;
-	const flags = bytes[5];
-	if (flags === 0) return {
-		baseHash: "",
-		payload: bytes.subarray(8),
-		windowLog
-	};
-	else if (flags !== 1) throw new InvalidOperationError(Operation.Read, hash, `unknown object flags ${flags}`);
-	else if (bytes.byteLength < 40) throw new InvalidOperationError(Operation.Read, hash, "delta object is truncated before its base hash");
-	return {
-		baseHash: Buffer.from(bytes.subarray(8, 40)).toString("hex"),
-		payload: bytes.subarray(40),
-		windowLog
-	};
-};
-//#endregion
-//#region src/createKeyframeStore.ts
-const decodeVerifiedObject = async (hash, parsedObject, basePlaintext) => {
-	const plaintext = await decodeObject(parsedObject, basePlaintext);
-	if (getContentAddress(plaintext) !== hash) throw new InvalidOperationError(Operation.Read, hash, "object does not hash to its key");
-	return plaintext;
-};
-const createKeyframeStore = (objectStore, { compressionLevel = DEFAULT_COMPRESSION_LEVEL, promotionRatio = DEFAULT_PROMOTION_RATIO, segmentBudgetRatio = 1 } = {}) => {
-	const readKeyframe = async (hash) => {
-		const bytes = await objectStore.read(hash);
-		if (!bytes) return void 0;
-		const parsedObject = parseObject(hash, bytes);
-		if (parsedObject.baseHash) throw new InvalidOperationError(Operation.Read, hash, `is a delta against ${parsedObject.baseHash}, not a keyframe`);
-		return {
-			plaintext: await decodeVerifiedObject(hash, parsedObject),
-			storedBytes: bytes.byteLength
-		};
-	};
-	const readStoredObject = async (hash) => {
-		const bytes = await objectStore.read(hash);
-		if (!bytes) return void 0;
-		const parsedObject = parseObject(hash, bytes);
-		if (!parsedObject.baseHash) return {
-			baseHash: "",
-			plaintext: await decodeVerifiedObject(hash, parsedObject)
-		};
-		const keyframe = await readKeyframe(parsedObject.baseHash);
-		if (!keyframe) throw new ObjectNotStoredError(hash, `keyframe ${parsedObject.baseHash} is not stored`);
-		return {
-			baseHash: parsedObject.baseHash,
-			plaintext: await decodeVerifiedObject(hash, parsedObject, keyframe.plaintext)
-		};
-	};
-	return {
-		collect: (releasedHashes, retainedHashes) => getResultAsync(async () => {
-			const retainedHashSet = new Set(retainedHashes);
-			const collectableHashes = [...new Set(releasedHashes)].filter((hash) => !retainedHashSet.has(hash));
-			if (collectableHashes.length > 0) await objectStore.delete(collectableHashes);
-			return collectableHashes;
-		}),
-		read: (hash) => getResultAsync(async () => {
-			const storedObject = await readStoredObject(hash);
-			if (!storedObject) throw new ObjectNotStoredError(hash, "object is not stored");
-			return storedObject.plaintext;
-		}),
-		write: (plaintext, anchor) => getResultAsync(async () => {
-			const hash = getContentAddress(plaintext);
-			const plaintextBytes = plaintext.byteLength;
-			const adoptStored = (baseHash) => ({
-				baseHash,
-				hash,
-				isDeduplicated: true,
-				plaintextBytes,
-				storedBytes: 0
-			});
-			const storedObject = await readStoredObject(hash);
-			if (storedObject) return adoptStored(storedObject.baseHash);
-			const writeObject = async (bytes, baseHash) => {
-				if (await objectStore.write(hash, bytes)) return {
-					baseHash,
-					hash,
-					isDeduplicated: false,
-					plaintextBytes,
-					storedBytes: bytes.byteLength
-				};
-				const twinObject = await readStoredObject(hash);
-				if (!twinObject) throw new InvalidOperationError(Operation.Create, hash, "refused as already stored, but nothing is stored");
-				return adoptStored(twinObject.baseHash);
-			};
-			const encodeDelta = async () => {
-				const keyframe = await readKeyframe(anchor.hash);
-				if (!keyframe) return void 0;
-				return {
-					bytes: await encodeObject(plaintext, compressionLevel, {
-						hash: anchor.hash,
-						plaintext: keyframe.plaintext
-					}),
-					keyframeStoredBytes: keyframe.storedBytes
-				};
-			};
-			const keyframeEncoding = encodeObject(plaintext, compressionLevel);
-			const deltaEncoding = anchor.hash ? encodeDelta() : void 0;
-			const [keyframeBytes, delta] = await Promise.all([keyframeEncoding, deltaEncoding]);
-			if (!delta) return writeObject(keyframeBytes, "");
-			const isWithinPromotionRatio = delta.bytes.byteLength <= keyframeBytes.byteLength * promotionRatio;
-			const isWithinSegmentBudget = anchor.anchoredBytes + delta.bytes.byteLength <= delta.keyframeStoredBytes * segmentBudgetRatio;
-			if (!isWithinPromotionRatio || !isWithinSegmentBudget) return writeObject(keyframeBytes, "");
-			else return writeObject(delta.bytes, anchor.hash);
-		})
-	};
-};
-//#endregion
-export { DEFAULT_PROMOTION_RATIO, DEFAULT_SEGMENT_BUDGET_RATIO, DELTA_FLAG, DELTA_HEADER_BYTE_COUNT, HASH_BYTE_COUNT, KEYFRAME_FLAG, OBJECT_FLAGS_OFFSET, OBJECT_FORMAT_VERSION, OBJECT_HEADER_BYTE_COUNT, OBJECT_MAGIC, OBJECT_VERSION_OFFSET, OBJECT_WINDOW_LOG_OFFSET, ObjectNotStoredError, createKeyframeStore, decodeObject, encodeObject, getContentAddress, parseObject };