@bendyline/gezel-knowledge 1.0.7 → 1.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. package/README.md +28 -17
  2. package/dist/index.d.ts +606 -161
  3. package/dist/index.js +2389 -508
  4. package/package.json +6 -4
package/README.md CHANGED
@@ -3,7 +3,7 @@
3
3
  The `.gezk` knowledge-catalog toolchain for
4
4
  [gezel](https://github.com/bendyline/gezel): a deterministic compiler, a
5
5
  verified archive reader, and a read-only catalog handle with two-stage
6
- `bit384+int8` retrieval. One format and one runtime serve a 200-document
6
+ `bit+int8` retrieval. One format and one runtime serve a 200-document
7
7
  personal reference and a 2,000,000-document Wikipedia-class corpus.
8
8
 
9
9
  ```bash
@@ -12,24 +12,29 @@ npm install @bendyline/gezel-knowledge
12
12
 
13
13
  ## What a `.gezk` is
14
14
 
15
- A standard ZIP holding `manifest.json`, a `router.db` (topics, document
16
- directory, brotli bodies, routing centroids), and shard SQLite databases with
17
- FTS and quantized vectors. Every catalog ships a browsable table of contents.
18
- The format is frozen — see `docs/gezk-format-v1.md` in the gezel repo.
15
+ An open format (gezk 0.6, preliminary until 1.0; 0.5 archives still open): a ZIP whose first entry is
16
+ the stored `mimetype` magic (`application/vnd.gezk+zip`), then
17
+ `manifest.json`, `README.md`, `LICENSES/`, a `router.db` (topics, document
18
+ directory, brotli bodies, routing centroids) and shard SQLite databases with
19
+ FTS5 and quantized vectors in plain BLOB tables. Nothing beyond stock SQLite
20
+ is needed to read one. The format definitions are the separate
21
+ [`@bendyline/gezk`](https://www.npmjs.com/package/@bendyline/gezk) package;
22
+ the specification, JSON Schemas, conformance fixtures and a Python reference
23
+ reader live in [bendyline/gezk](https://github.com/bendyline/gezk).
19
24
 
20
25
  ## Building a catalog
21
26
 
22
27
  ```ts
23
28
  import {
24
- GEZEL_BGE_SMALL_EN_V15_1,
25
- GEZEL_MARKDOWN_CHUNKS_2,
29
+ BGE_SMALL_EN_V15_1,
30
+ MARKDOWN_CHUNKS_2,
26
31
  compileKnowledgeCatalog,
27
32
  createProfileEmbedder,
28
33
  loadMarkdownCatalog,
29
34
  } from '@bendyline/gezel-knowledge';
30
35
 
31
36
  const source = await loadMarkdownCatalog('./my-notes', { language: 'en' });
32
- const embedder = await createProfileEmbedder(GEZEL_BGE_SMALL_EN_V15_1);
37
+ const embedder = await createProfileEmbedder(BGE_SMALL_EN_V15_1);
33
38
  await compileKnowledgeCatalog({
34
39
  catalog: {
35
40
  id: 'my-notes',
@@ -38,25 +43,27 @@ await compileKnowledgeCatalog({
38
43
  language: 'en',
39
44
  publisher: { id: 'me', name: 'Me' },
40
45
  createdAt: new Date().toISOString(),
41
- license: { name: 'MIT', attributionRequired: false },
46
+ license: { name: 'MIT', spdx: 'MIT', attributionRequired: false },
42
47
  },
43
48
  topics: source.topics,
44
49
  documents: (async function* () {
45
50
  for (const doc of source.documents) yield doc;
46
51
  })(),
47
52
  outputPath: './my-notes-1.0.0.gezk',
48
- embeddingProfile: GEZEL_BGE_SMALL_EN_V15_1,
49
- chunkingProfile: GEZEL_MARKDOWN_CHUNKS_2,
53
+ embeddingProfile: BGE_SMALL_EN_V15_1,
54
+ chunkingProfile: MARKDOWN_CHUNKS_2,
50
55
  embed: (texts) => embedder.embed(texts),
51
56
  countTokens: (text) => embedder.countTokens(text),
52
57
  workDir: './.gezk-work',
53
58
  });
54
59
  ```
55
60
 
56
- Embedding requires `@huggingface/transformers` at runtime; the module is
57
- imported dynamically so everything else works without it. The `gezel
58
- knowledge` CLI wraps this same API (`init`, `build`, `validate`, `inspect`,
59
- `search`), entirely offline — no daemon.
61
+ Every archive ships a `README.md` and `LICENSES/catalog.txt`; the compiler
62
+ generates minimal ones from the catalog block unless you pass richer texts in
63
+ `extraFiles`. Embedding requires `@huggingface/transformers` at runtime; the
64
+ module is imported dynamically so everything else works without it. The
65
+ `gezel knowledge` CLI wraps this same API (`init`, `build`, `validate`,
66
+ `inspect`, `search`), entirely offline — no daemon.
60
67
 
61
68
  ## Reading a catalog
62
69
 
@@ -73,10 +80,14 @@ handle.close();
73
80
  Extraction verifies every file against the manifest (both directions —
74
81
  undeclared and missing files are equally fatal) and rejects unsafe archive
75
82
  entries. Catalog databases are opened strictly read-only and immutable.
83
+ Semantic search loads a shard's sign-bit rows into memory once (9.6 MB for a
84
+ full 200,000-chunk shard) and scans them with a popcount; the int8 rerank
85
+ reads the candidates by rowid.
76
86
 
77
87
  ## Stability
78
88
 
79
- Published for the gezel toolchain and the Qualla content pipeline. The
89
+ Published for the gezel toolchain and the Bendyline content pipeline. The
80
90
  archive format is versioned (`formatVersion` / `indexSchemaVersion`) and
81
91
  readers never migrate a catalog — incompatibility is a typed, reported
82
- state.
92
+ state. While the format is `0.x`, a minor release may break compatibility;
93
+ each reader names the exact versions it supports.