keyframe-store 4.3.3 → 4.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. package/README.md +8 -30
  2. package/dist/index.d.ts +13 -12
  3. package/dist/index.js +17 -16
  4. package/package.json +8 -8
package/README.md CHANGED
@@ -5,12 +5,14 @@
5
5
  [![NPM downloads][badge-npm-downloads]][url-npm]
6
6
  [![NPM Unpacked Size (with version)][badge-npm-unpacked-size]][url-npm]
7
7
 
8
- Content-addressed version storage for documents handed over whole. Every version is stored as a zstd **keyframe** — compressed on its own — or as a **delta** compressed against exactly one keyframe, using that keyframe's plaintext as the dictionary. Reconstructing any version is at most two reads, a version whose content is already held costs nothing, and the backend is anything that can store bytes by key.
8
+ This package stores content-addressed versions of whole documents. Each version is either a zstd **keyframe**, compressed on its own, or a **delta** against a keyframe.
9
+
10
+ Design notes live in [the specs](https://github.com/Esposter/Esposter/tree/main/openspec/specs/keyframe-store).
9
11
 
10
12
  ## Table of Contents
11
13
 
12
14
  - 🚀 [Getting Started](#getting-started)
13
- - 📖 [Documentation](#documentation)
15
+ - 🧩 [Key Exports](#key-exports)
14
16
  - ⚖️ [License](#license)
15
17
 
16
18
  ---
@@ -64,44 +66,20 @@ const plaintext = await keyframeStore.read(second.hash).match(
64
66
  );
65
67
  ```
66
68
 
67
- ## <a name="documentation">📖 Documentation</a>
68
-
69
- We highly recommend you take a look at the [documentation](https://github.com/Esposter/Esposter/tree/main/openspec/specs) to level up.
70
-
71
- ### How it works
72
-
73
- - **Addressing.** An object's key is the hex SHA-256 of its plaintext, so two versions with identical content resolve to one object whichever keyframe either would have been encoded against. The store is write-once: a write of content it already holds records nothing and reports itself deduplicated.
74
- - **Encoding.** Each write compresses the plaintext standalone and, when the lineage has a keyframe, again with that keyframe's plaintext as the dictionary. The delta is kept only if it is at most the **promotion ratio** of the standalone size _and_ the bytes already anchored to the keyframe plus this delta fit the **segment budget**; otherwise the standalone form is written and this version becomes the lineage's next keyframe.
75
- - **Reading.** A keyframe is one read and one decompression; a delta fetches the keyframe its own header names, decompresses it, then decompresses the delta against it. Every result is hashed and compared with the key, so a truncated, altered or misfiled object is a refused read rather than a plausible document.
76
- - **Collection.** An object survives while any record names it, as its own hash or as its base. The caller answers that from its own records — `collect` takes the hashes an eviction released and the hashes still named, deletes the difference, and returns what it deleted.
77
- - **The anchor is the caller's.** The store holds no notion of a lineage: a caller derives one anchor per history — the newest keyframe's hash, and the bytes anchored to it — and resets both whenever a write reports an empty base.
78
-
79
- ### Key exports
69
+ ## <a name="key-exports">🧩 Key Exports</a>
80
70
 
81
71
  | Export | Role |
82
72
  | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
83
73
  | `createKeyframeStore` | The factory, taking an `ObjectStore` and optional `KeyframeStoreOptions` |
84
- | `ObjectStore` | The backend interface — `read`, `write`, `delete` by key |
85
74
  | `KeyframeStore` | `write(plaintext, anchor)`, `read(hash)`, `collect(released, retained)` — every one a `ResultAsync` |
75
+ | `KeyframeStoreOptions` | `compressionLevel`, `promotionRatio`, `segmentBudgetRatio`, each defaulted from `constants` — the level from `@esposter/shared`'s, which the app's browser encoder shares |
76
+ | `ObjectStore` | The backend interface — `read`, `write`, `delete` by key |
86
77
  | `VersionAnchor` | What a lineage passes to a write: its keyframe's hash and the bytes anchored to it |
87
78
  | `WrittenVersion` | What a write reports: the hash, its base, both sizes and whether it was deduplicated |
88
- | `KeyframeStoreOptions` | `compressionLevel`, `promotionRatio`, `segmentBudgetRatio`, each defaulted from `constants` — the level from `@esposter/shared`'s, which the app's browser encoder shares |
89
-
90
- ### Commands
91
-
92
- Run from `packages/keyframe-store/`:
93
-
94
- ```bash
95
- pnpm build # compile to dist/
96
- pnpm test # vitest watch mode (coverage is run from the repo root)
97
- pnpm bench # the codec benchmark, writing the colocated *.bench.md
98
- pnpm lint:fix # auto-fix lint
99
- pnpm typecheck # type check
100
- ```
101
79
 
102
80
  ## <a name="license">⚖️ License</a>
103
81
 
104
- This project is licensed under the [Apache-2.0 license](https://github.com/Esposter/Esposter/blob/main/LICENSE).
82
+ Licensed under [Apache-2.0][url-license].
105
83
 
106
84
  [badge-license]: https://img.shields.io/github/license/Esposter/Esposter.svg?color=blue
107
85
  [url-license]: https://github.com/Esposter/Esposter/blob/main/LICENSE
package/dist/index.d.ts CHANGED
@@ -1,17 +1,5 @@
1
1
  import { InvalidOperationError } from "@esposter/shared";
2
2
  import { ResultAsync } from "neverthrow";
3
- export declare const OBJECT_MAGIC: Uint8Array;
4
- export declare const OBJECT_FORMAT_VERSION = 1;
5
- export declare const OBJECT_VERSION_OFFSET = 4;
6
- export declare const OBJECT_FLAGS_OFFSET = 5;
7
- export declare const OBJECT_WINDOW_LOG_OFFSET = 6;
8
- export declare const OBJECT_HEADER_BYTE_COUNT = 8;
9
- export declare const KEYFRAME_FLAG = 0;
10
- export declare const DELTA_FLAG = 1;
11
- export declare const HASH_BYTE_COUNT = 32;
12
- export declare const DELTA_HEADER_BYTE_COUNT: number;
13
- export declare const DEFAULT_PROMOTION_RATIO: number;
14
- export declare const DEFAULT_SEGMENT_BUDGET_RATIO = 1;
15
3
  export interface VersionAnchor {
16
4
  anchoredBytes: number;
17
5
  hash: string;
@@ -47,7 +35,20 @@ export interface ParsedObject {
47
35
  payload: Uint8Array;
48
36
  windowLog: number;
49
37
  }
38
+ export declare const OBJECT_MAGIC: Uint8Array;
39
+ export declare const OBJECT_FORMAT_VERSION = 1;
40
+ export declare const OBJECT_VERSION_OFFSET = 4;
41
+ export declare const OBJECT_FLAGS_OFFSET = 5;
42
+ export declare const OBJECT_WINDOW_LOG_OFFSET = 6;
43
+ export declare const OBJECT_HEADER_BYTE_COUNT = 8;
44
+ export declare const KEYFRAME_FLAG = 0;
45
+ export declare const DELTA_FLAG = 1;
46
+ export declare const HASH_BYTE_COUNT = 32;
47
+ export declare const DELTA_HEADER_BYTE_COUNT: number;
48
+ export declare const DEFAULT_PROMOTION_RATIO: number;
49
+ export declare const DEFAULT_SEGMENT_BUDGET_RATIO = 1;
50
50
  export declare const decodeObject: (parsedObject: ParsedObject, basePlaintext?: Uint8Array) => Promise<Buffer>;
51
+ export declare const decodeVerifiedObject: (hash: string, parsedObject: ParsedObject, basePlaintext?: Uint8Array) => Promise<Uint8Array>;
51
52
  export declare const encodeObject: (plaintext: Uint8Array, compressionLevel: number, base?: {
52
53
  hash: string;
53
54
  plaintext: Uint8Array;
package/dist/index.js CHANGED
@@ -2,7 +2,14 @@ import { DEFAULT_COMPRESSION_LEVEL, InvalidOperationError, Operation, getResultA
2
2
  import { promisify } from "node:util";
3
3
  import { constants, zstdCompress, zstdDecompress } from "node:zlib";
4
4
  import { createHash } from "node:crypto";
5
- // src/constants.ts
5
+ // src/models/ObjectNotStoredError.ts
6
+ var ObjectNotStoredError = class extends InvalidOperationError {
7
+ constructor(hash, message) {
8
+ super(Operation.Read, hash, message);
9
+ this.name = "ObjectNotStoredError";
10
+ }
11
+ };
12
+ // src/services/constants.ts
6
13
  const OBJECT_MAGIC = new Uint8Array([
7
14
  75,
8
15
  70,
@@ -20,19 +27,20 @@ const HASH_BYTE_COUNT = 32;
20
27
  const DELTA_HEADER_BYTE_COUNT = 40;
21
28
  const DEFAULT_PROMOTION_RATIO = 1 / 3;
22
29
  const DEFAULT_SEGMENT_BUDGET_RATIO = 1;
23
- // src/models/ObjectNotStoredError.ts
24
- var ObjectNotStoredError = class extends InvalidOperationError {
25
- constructor(hash, message) {
26
- super(Operation.Read, hash, message);
27
- this.name = "ObjectNotStoredError";
28
- }
29
- };
30
30
  // src/services/decodeObject.ts
31
31
  const decompress = promisify(zstdDecompress);
32
32
  const decodeObject = (parsedObject, basePlaintext) => decompress(parsedObject.payload, {
33
33
  ...basePlaintext ? { dictionary: basePlaintext } : {},
34
34
  params: { [constants.ZSTD_d_windowLogMax]: parsedObject.windowLog }
35
35
  });
36
+ // src/services/getContentAddress.ts
37
+ const getContentAddress = (plaintext) => createHash("sha256").update(plaintext).digest("hex");
38
+ // src/services/decodeVerifiedObject.ts
39
+ const decodeVerifiedObject = async (hash, parsedObject, basePlaintext) => {
40
+ const plaintext = await decodeObject(parsedObject, basePlaintext);
41
+ if (getContentAddress(plaintext) !== hash) throw new InvalidOperationError(Operation.Read, hash, "object does not hash to its key");
42
+ return plaintext;
43
+ };
36
44
  // src/services/encodeObject.ts
37
45
  const compress = promisify(zstdCompress);
38
46
  const encodeObject = async (plaintext, compressionLevel, base) => {
@@ -54,8 +62,6 @@ const encodeObject = async (plaintext, compressionLevel, base) => {
54
62
  bytes.set(payload, headerByteCount);
55
63
  return bytes;
56
64
  };
57
- // src/services/getContentAddress.ts
58
- const getContentAddress = (plaintext) => createHash("sha256").update(plaintext).digest("hex");
59
65
  // src/services/parseObject.ts
60
66
  const parseObject = (hash, bytes) => {
61
67
  if (!(bytes.byteLength >= 8 && OBJECT_MAGIC.every((byte, index) => bytes[index] === byte))) throw new InvalidOperationError(Operation.Read, hash, "not a keyframe store object");
@@ -76,11 +82,6 @@ const parseObject = (hash, bytes) => {
76
82
  };
77
83
  };
78
84
  // src/createKeyframeStore.ts
79
- const decodeVerifiedObject = async (hash, parsedObject, basePlaintext) => {
80
- const plaintext = await decodeObject(parsedObject, basePlaintext);
81
- if (getContentAddress(plaintext) !== hash) throw new InvalidOperationError(Operation.Read, hash, "object does not hash to its key");
82
- return plaintext;
83
- };
84
85
  const createKeyframeStore = (objectStore, { compressionLevel = DEFAULT_COMPRESSION_LEVEL, promotionRatio = DEFAULT_PROMOTION_RATIO, segmentBudgetRatio = 1 } = {}) => {
85
86
  const readKeyframe = async (hash) => {
86
87
  const bytes = await objectStore.read(hash);
@@ -165,4 +166,4 @@ const createKeyframeStore = (objectStore, { compressionLevel = DEFAULT_COMPRESSI
165
166
  })
166
167
  };
167
168
  };
168
- export { DEFAULT_PROMOTION_RATIO, DEFAULT_SEGMENT_BUDGET_RATIO, DELTA_FLAG, DELTA_HEADER_BYTE_COUNT, HASH_BYTE_COUNT, KEYFRAME_FLAG, OBJECT_FLAGS_OFFSET, OBJECT_FORMAT_VERSION, OBJECT_HEADER_BYTE_COUNT, OBJECT_MAGIC, OBJECT_VERSION_OFFSET, OBJECT_WINDOW_LOG_OFFSET, ObjectNotStoredError, createKeyframeStore, decodeObject, encodeObject, getContentAddress, parseObject };
169
+ export { DEFAULT_PROMOTION_RATIO, DEFAULT_SEGMENT_BUDGET_RATIO, DELTA_FLAG, DELTA_HEADER_BYTE_COUNT, HASH_BYTE_COUNT, KEYFRAME_FLAG, OBJECT_FLAGS_OFFSET, OBJECT_FORMAT_VERSION, OBJECT_HEADER_BYTE_COUNT, OBJECT_MAGIC, OBJECT_VERSION_OFFSET, OBJECT_WINDOW_LOG_OFFSET, ObjectNotStoredError, createKeyframeStore, decodeObject, decodeVerifiedObject, encodeObject, getContentAddress, parseObject };
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "keyframe-store",
3
- "version": "4.3.3",
3
+ "version": "4.5.0",
4
4
  "description": "Content-addressed version storage — every version is a zstd keyframe or a delta against one, over any backend that can store bytes by key.",
5
5
  "keywords": [
6
6
  "compression",
@@ -47,17 +47,17 @@
47
47
  "lint": "eslint .",
48
48
  "lint:fix": "eslint --fix .",
49
49
  "test": "vitest",
50
- "typecheck": "tsc"
50
+ "typecheck": "tsrs --fast"
51
51
  },
52
52
  "dependencies": {
53
- "@esposter/shared": "4.3.3",
53
+ "@esposter/shared": "4.5.0",
54
54
  "neverthrow": "^8.2.0"
55
55
  },
56
56
  "devDependencies": {
57
- "@esposter/configuration": "4.3.3",
58
- "@esposter/shared-node": "4.3.3",
59
- "@types/node": "^26.6.5",
60
- "tsdown": "^0.23.1"
57
+ "@esposter/configuration": "4.5.0",
58
+ "@esposter/shared-node": "4.5.0",
59
+ "@types/node": "^26.6.6",
60
+ "tsdown": "^0.23.5"
61
61
  },
62
- "gitHead": "955bbc130ad9b28877fd4ae8efc73a045141ce01"
62
+ "gitHead": "e30c001f3b4b15693e75fb53092d62a97ade4d2b"
63
63
  }