@bendyline/gezel-knowledge 1.0.7 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +28 -17
- package/dist/index.d.ts +376 -157
- package/dist/index.js +1061 -421
- package/package.json +5 -4
package/README.md
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
The `.gezk` knowledge-catalog toolchain for
|
|
4
4
|
[gezel](https://github.com/bendyline/gezel): a deterministic compiler, a
|
|
5
5
|
verified archive reader, and a read-only catalog handle with two-stage
|
|
6
|
-
`
|
|
6
|
+
`bit+int8` retrieval. One format and one runtime serve a 200-document
|
|
7
7
|
personal reference and a 2,000,000-document Wikipedia-class corpus.
|
|
8
8
|
|
|
9
9
|
```bash
|
|
@@ -12,24 +12,29 @@ npm install @bendyline/gezel-knowledge
|
|
|
12
12
|
|
|
13
13
|
## What a `.gezk` is
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
15
|
+
An open format (gezk 0.5, preliminary until 1.0): a ZIP whose first entry is
|
|
16
|
+
the stored `mimetype` magic (`application/vnd.gezk+zip`), then
|
|
17
|
+
`manifest.json`, `README.md`, `LICENSES/`, a `router.db` (topics, document
|
|
18
|
+
directory, brotli bodies, routing centroids) and shard SQLite databases with
|
|
19
|
+
FTS5 and quantized vectors in plain BLOB tables. Nothing beyond stock SQLite
|
|
20
|
+
is needed to read one. The format definitions are the separate
|
|
21
|
+
[`@bendyline/gezk`](https://www.npmjs.com/package/@bendyline/gezk) package;
|
|
22
|
+
the specification, JSON Schemas, conformance fixtures and a Python reference
|
|
23
|
+
reader live in [bendyline/gezk](https://github.com/bendyline/gezk).
|
|
19
24
|
|
|
20
25
|
## Building a catalog
|
|
21
26
|
|
|
22
27
|
```ts
|
|
23
28
|
import {
|
|
24
|
-
|
|
25
|
-
|
|
29
|
+
BGE_SMALL_EN_V15_1,
|
|
30
|
+
MARKDOWN_CHUNKS_2,
|
|
26
31
|
compileKnowledgeCatalog,
|
|
27
32
|
createProfileEmbedder,
|
|
28
33
|
loadMarkdownCatalog,
|
|
29
34
|
} from '@bendyline/gezel-knowledge';
|
|
30
35
|
|
|
31
36
|
const source = await loadMarkdownCatalog('./my-notes', { language: 'en' });
|
|
32
|
-
const embedder = await createProfileEmbedder(
|
|
37
|
+
const embedder = await createProfileEmbedder(BGE_SMALL_EN_V15_1);
|
|
33
38
|
await compileKnowledgeCatalog({
|
|
34
39
|
catalog: {
|
|
35
40
|
id: 'my-notes',
|
|
@@ -38,25 +43,27 @@ await compileKnowledgeCatalog({
|
|
|
38
43
|
language: 'en',
|
|
39
44
|
publisher: { id: 'me', name: 'Me' },
|
|
40
45
|
createdAt: new Date().toISOString(),
|
|
41
|
-
license: { name: 'MIT', attributionRequired: false },
|
|
46
|
+
license: { name: 'MIT', spdx: 'MIT', attributionRequired: false },
|
|
42
47
|
},
|
|
43
48
|
topics: source.topics,
|
|
44
49
|
documents: (async function* () {
|
|
45
50
|
for (const doc of source.documents) yield doc;
|
|
46
51
|
})(),
|
|
47
52
|
outputPath: './my-notes-1.0.0.gezk',
|
|
48
|
-
embeddingProfile:
|
|
49
|
-
chunkingProfile:
|
|
53
|
+
embeddingProfile: BGE_SMALL_EN_V15_1,
|
|
54
|
+
chunkingProfile: MARKDOWN_CHUNKS_2,
|
|
50
55
|
embed: (texts) => embedder.embed(texts),
|
|
51
56
|
countTokens: (text) => embedder.countTokens(text),
|
|
52
57
|
workDir: './.gezk-work',
|
|
53
58
|
});
|
|
54
59
|
```
|
|
55
60
|
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
61
|
+
Every archive ships a `README.md` and `LICENSES/catalog.txt`; the compiler
|
|
62
|
+
generates minimal ones from the catalog block unless you pass richer texts in
|
|
63
|
+
`extraFiles`. Embedding requires `@huggingface/transformers` at runtime; the
|
|
64
|
+
module is imported dynamically so everything else works without it. The
|
|
65
|
+
`gezel knowledge` CLI wraps this same API (`init`, `build`, `validate`,
|
|
66
|
+
`inspect`, `search`), entirely offline — no daemon.
|
|
60
67
|
|
|
61
68
|
## Reading a catalog
|
|
62
69
|
|
|
@@ -73,10 +80,14 @@ handle.close();
|
|
|
73
80
|
Extraction verifies every file against the manifest (both directions —
|
|
74
81
|
undeclared and missing files are equally fatal) and rejects unsafe archive
|
|
75
82
|
entries. Catalog databases are opened strictly read-only and immutable.
|
|
83
|
+
Semantic search loads a shard's sign-bit rows into memory once (9.6 MB for a
|
|
84
|
+
full 200,000-chunk shard) and scans them with a popcount; the int8 rerank
|
|
85
|
+
reads the candidates by rowid.
|
|
76
86
|
|
|
77
87
|
## Stability
|
|
78
88
|
|
|
79
|
-
Published for the gezel toolchain and the
|
|
89
|
+
Published for the gezel toolchain and the Bendyline content pipeline. The
|
|
80
90
|
archive format is versioned (`formatVersion` / `indexSchemaVersion`) and
|
|
81
91
|
readers never migrate a catalog — incompatibility is a typed, reported
|
|
82
|
-
state.
|
|
92
|
+
state. While the format is `0.x`, a minor release may break compatibility;
|
|
93
|
+
each reader names the exact versions it supports.
|