@urna/cli 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +281 -0
- package/binary-install.js +348 -0
- package/binary.js +124 -0
- package/install.js +4 -0
- package/npm-shrinkwrap.json +52 -0
- package/package.json +126 -0
- package/run-urna.js +4 -0
package/README.md
ADDED
|
@@ -0,0 +1,281 @@
|
|
|
1
|
+

|
|
2
|
+
|
|
3
|
+
# Urna
|
|
4
|
+
|
|
5
|
+
A vector database in one file, with citations that stay valid.
|
|
6
|
+
|
|
7
|
+
A `.urna` file holds the chunks, the embeddings, the source spans, the indices and the search contract. The rust runtime maps it into memory, checks its hashes, and answers with exact cosine scores and a `urna://content_hash/chunk_id` citation for every hit. It works offline and rebuilds byte for byte. Python builds the file, rust serves it.
|
|
8
|
+
|
|
9
|
+
## Install
|
|
10
|
+
|
|
11
|
+
```sh
|
|
12
|
+
brew install hoffresearch/urna/urna
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
```sh
|
|
16
|
+
npm install -g urna
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
```sh
|
|
20
|
+
cargo install urna
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
```sh
|
|
24
|
+
curl -sSf https://raw.githubusercontent.com/hoffresearch/urna/main/scripts/install.sh | sh
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Then run setup once. It downloads the offline embedder, prepares a python env and checks the install:
|
|
28
|
+
|
|
29
|
+
```sh
|
|
30
|
+
urna setup
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Python only, no setup step needed:
|
|
34
|
+
|
|
35
|
+
```sh
|
|
36
|
+
pip install "urna[embed]"
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Windows, docker, `cargo binstall` and how to verify a download are in the [install reference](https://github.com/hoffresearch/urna/blob/main/docs/usage.md#reference).
|
|
40
|
+
|
|
41
|
+
## In the terminal
|
|
42
|
+
|
|
43
|
+
`urna setup` shows the plan before it writes anything and ends on the doctor checks.
|
|
44
|
+
|
|
45
|
+
<img src="https://raw.githubusercontent.com/hoffresearch/urna/main/assets/images/urna-setup.png" alt="urna setup: the verify step with every doctor check passing" width="100%">
|
|
46
|
+
|
|
47
|
+
`urna tui` opens a corpus, validates it, and lets you ask it questions. Each hit shows its score, the stored text and its citation.
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
urna tui my_corpus.urna
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
<img src="https://raw.githubusercontent.com/hoffresearch/urna/main/assets/images/urna-tui.png" alt="urna tui: the ask tab with scored hits and the cited text of the selected one" width="100%">
|
|
54
|
+
|
|
55
|
+
## Quickstart
|
|
56
|
+
|
|
57
|
+
`examples/quickstart/` has twelve short paragraphs and the spec that builds them. From a checkout:
|
|
58
|
+
|
|
59
|
+
```sh
|
|
60
|
+
urna build --spec examples/quickstart/corpus.toml
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
```sh
|
|
64
|
+
urna ask examples/quickstart/out/quickstart.urna "can I use this offline" -k 1
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
```sh
|
|
68
|
+
urna retrieve examples/quickstart/out/quickstart.urna "how do citations work" -k 2 --format jsonl
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
```sh
|
|
72
|
+
urna cite examples/quickstart/out/quickstart.urna 'urna://sha256:1147b256.../sha256:b5dfeb09...'
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
```sh
|
|
76
|
+
urna validate examples/quickstart/out/quickstart.urna
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
`ask` prints the answer with its citation, `retrieve` prints json for another program, `cite` turns a citation back into the stored text, and `validate` checks every hash. To build from your own rows, see [usage section 13](https://github.com/hoffresearch/urna/blob/main/docs/usage.md).
|
|
80
|
+
|
|
81
|
+
## What the file guarantees
|
|
82
|
+
|
|
83
|
+
| Property | How |
|
|
84
|
+
|----------|-----|
|
|
85
|
+
| Self-contained | The file is the whole database. Copy it like a sqlite file. |
|
|
86
|
+
| Verifiable | Sha-256 per section, per file and over the decoded content. `urna cite` resolves any citation. |
|
|
87
|
+
| Reproducible | Same chunks and same model give a byte-identical file on any machine. |
|
|
88
|
+
| Offline | The runtime never opens a socket. A query from the wrong model fails at the `model_hash` check. |
|
|
89
|
+
|
|
90
|
+
## Python
|
|
91
|
+
|
|
92
|
+
```python
|
|
93
|
+
import urna
|
|
94
|
+
from urna.embed_potion import potion_embedder
|
|
95
|
+
|
|
96
|
+
emb = potion_embedder()
|
|
97
|
+
db = urna.open("my_corpus.urna")
|
|
98
|
+
qvec = emb.embed_texts(["can I use this offline"])[0]
|
|
99
|
+
|
|
100
|
+
hits = db.retrieve(qvec, 5, expected_model_hash=emb.model_hash())
|
|
101
|
+
print(hits[0].citation_id, hits[0].score, hits[0].text)
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
<details>
|
|
105
|
+
<summary>Search variants, validate, build</summary>
|
|
106
|
+
|
|
107
|
+
```python
|
|
108
|
+
db.search(qvec, 5) # exact
|
|
109
|
+
db.search_ann(qvec, 5, 100) # hnsw, then exact rerank
|
|
110
|
+
db.search_hybrid(qvec, "vacina contra covid", 5, 100) # bm25 + vectors, exact rerank
|
|
111
|
+
db.search_graph(qvec, 5, hops=2, ef=100) # chunk graph from the seeds
|
|
112
|
+
db.search_space("clip-vit-b32", ivec, 5) # one named multimodal space
|
|
113
|
+
|
|
114
|
+
assert db.validate() is True
|
|
115
|
+
info = db.inspect()
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Each chunk is a dict with `canonical_text`, `source_uri`, `byte_start`, `byte_end` and `embedding`:
|
|
119
|
+
|
|
120
|
+
```python
|
|
121
|
+
urna.build(
|
|
122
|
+
output_path="my_corpus.urna",
|
|
123
|
+
embedding_model="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
|
|
124
|
+
embedding_dim=384,
|
|
125
|
+
chunker_version="fixed-512/1",
|
|
126
|
+
model_hash=model_hash,
|
|
127
|
+
chunks=chunks,
|
|
128
|
+
reproducible=True,
|
|
129
|
+
preset="hybrid",
|
|
130
|
+
)
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
`python examples/quickstart/quickstart.py` runs the whole loop, build to cited hits, with no network.
|
|
134
|
+
|
|
135
|
+
</details>
|
|
136
|
+
|
|
137
|
+
## CLI
|
|
138
|
+
|
|
139
|
+
The engine verbs take a file and a vector and never run python. The agent verbs (`ask`, `retrieve`, `build`) take text and use the offline embedder. `setup` and `tui` are the terminal ui. `urna --help` lists all three groups.
|
|
140
|
+
|
|
141
|
+
<details>
|
|
142
|
+
<summary>Agent verbs</summary>
|
|
143
|
+
|
|
144
|
+
```sh
|
|
145
|
+
urna ask my_corpus.urna "can I use this offline" -k 3
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
```sh
|
|
149
|
+
urna retrieve my_corpus.urna "can I use this offline" -k 5 --format jsonl
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
```sh
|
|
153
|
+
urna build --spec corpus.toml --dry-run
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
`build` reads one toml: the source (sqlite, csv, jsonl, an image dir), the media settings, and one or more embedding models from the registry (`potion`, `clip-vit-b32`, `siglip2`, `wemm-2b`, ...). Each model becomes a named vector space in the same file. The full spec is in [usage section 13](https://github.com/hoffresearch/urna/blob/main/docs/usage.md).
|
|
157
|
+
|
|
158
|
+
</details>
|
|
159
|
+
|
|
160
|
+
<details>
|
|
161
|
+
<summary>Search</summary>
|
|
162
|
+
|
|
163
|
+
```sh
|
|
164
|
+
urna search my_corpus.urna "[0.1, 0.2, ...]" -k 10
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
```sh
|
|
168
|
+
urna search-ann my_corpus.urna "[0.1, 0.2, ...]" -k 10 --ef 200
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
```sh
|
|
172
|
+
urna search-graph my_corpus.urna "[0.1, 0.2, ...]" -k 10 --hops 2 --ef 100
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
```sh
|
|
176
|
+
urna search-space my_corpus.urna "[0.1, ...]" --space "wemm-2b@256" -k 5
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
```sh
|
|
180
|
+
urna search-text my_corpus.urna "vacina contra covid funciona" -k 5
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
</details>
|
|
184
|
+
|
|
185
|
+
<details>
|
|
186
|
+
<summary>Inspect, validate, stats, cite, media, benchmark, doctor</summary>
|
|
187
|
+
|
|
188
|
+
```sh
|
|
189
|
+
urna inspect my_corpus.urna --json
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
```sh
|
|
193
|
+
urna validate my_corpus.urna
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
```sh
|
|
197
|
+
urna stats my_corpus.urna
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
```sh
|
|
201
|
+
urna cite my_corpus.urna 'urna://sha256:1aa9.../sha256:8f314...'
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
```sh
|
|
205
|
+
urna media my_corpus.urna --export DIR
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
```sh
|
|
209
|
+
urna benchmark my_corpus.urna -q 100 -k 10 --ann 100 --madvise-cold
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
```sh
|
|
213
|
+
urna doctor
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
</details>
|
|
217
|
+
|
|
218
|
+
## Benchmarks
|
|
219
|
+
|
|
220
|
+
100,000 x 384 rows, k=10, one thread, same machine for every store.
|
|
221
|
+
|
|
222
|
+
| Store | p50 (ms) | p99 (ms) | Cold open (ms) |
|
|
223
|
+
|-------|---------:|---------:|---------------:|
|
|
224
|
+
| hnswlib | 0.32 | 0.53 | 182 |
|
|
225
|
+
| usearch | 0.67 | 61.4 | 58 |
|
|
226
|
+
| urna hybrid | 0.72 | 1.02 | 356 |
|
|
227
|
+
| urna exact | 7.80 | 8.32 | 292 |
|
|
228
|
+
| lancedb | 16.7 | 19.4 | 612 |
|
|
229
|
+
| sqlite-vec | 19.8 | 24.8 | 50 |
|
|
230
|
+
|
|
231
|
+
Both urna rows return recall@10 = 1.000. Urna's cold open includes checking every section hash before the first answer. Urna does not do updates, filters or concurrent writers. Method and the full table: [docs/benchmarks.md](https://github.com/hoffresearch/urna/blob/main/docs/benchmarks.md).
|
|
232
|
+
|
|
233
|
+
<details>
|
|
234
|
+
<summary>Presets: size vs recall</summary>
|
|
235
|
+
|
|
236
|
+
| Preset | Embeddings | Index | Size | Recall@10 |
|
|
237
|
+
|--------|------------|-------|-----:|----------:|
|
|
238
|
+
| `exact` | float32 | | 1.000 | 1.000 |
|
|
239
|
+
| `compressed` | float16 | | 0.339 | 1.000 |
|
|
240
|
+
| `tiny` | int8 | hnsw | 0.256 | 0.992 |
|
|
241
|
+
| `micro` | mrl256-int8 | hnsw | 0.223 | 0.810 |
|
|
242
|
+
| `nano` | int4 | hnsw | 0.209 | 0.913 |
|
|
243
|
+
| `hybrid` | float32 | hnsw + bm25 | 0.609 | 1.000 |
|
|
244
|
+
|
|
245
|
+
Measured on a 30,725-chunk pt-br corpus. Recall here is rank stability under quantization, not real-query quality. Details in [usage section 6](https://github.com/hoffresearch/urna/blob/main/docs/usage.md).
|
|
246
|
+
|
|
247
|
+
</details>
|
|
248
|
+
|
|
249
|
+
<details>
|
|
250
|
+
<summary>Images: 38,627 magic cards in one file</summary>
|
|
251
|
+
|
|
252
|
+
| Profile | Media | File | Vs the jpeg source |
|
|
253
|
+
|---------|-------|-----:|-------------------:|
|
|
254
|
+
| `archive` | JPEG XL, byte-reversible | 3.61 GB | 1.10x |
|
|
255
|
+
| `stills` | AV1 all-intra crf35 | 1.37 GB | 2.89x |
|
|
256
|
+
| `retrieval` | AV1 all-intra crf50 | 533 MB | 7.46x |
|
|
257
|
+
|
|
258
|
+
Text-to-image hit@1 over every card: siglip2 0.750, wemm-2b 0.744, jina 0.336, clip 0.098. Code and data: [mtg-urna-benchmark](https://github.com/brennercruvinel/mtg-urna-benchmark).
|
|
259
|
+
|
|
260
|
+
</details>
|
|
261
|
+
|
|
262
|
+
## Reference
|
|
263
|
+
|
|
264
|
+
- [docs/usage.md](https://github.com/hoffresearch/urna/blob/main/docs/usage.md): every verb, presets, models, builds, install channels
|
|
265
|
+
- [docs/benchmarks.md](https://github.com/hoffresearch/urna/blob/main/docs/benchmarks.md): how the numbers were measured
|
|
266
|
+
- [docs/SECURITY.md](https://github.com/hoffresearch/urna/blob/main/docs/SECURITY.md): reporting, hardening, data governance
|
|
267
|
+
- [docs/CHANGELOG](https://github.com/hoffresearch/urna/blob/main/docs/CHANGELOG): releases with measured numbers
|
|
268
|
+
- [docs/arc/arc.toml](https://github.com/hoffresearch/urna/blob/main/docs/arc/arc.toml): the architecture map
|
|
269
|
+
- [AGENTS.md](https://github.com/hoffresearch/urna/blob/main/.contracts/.agents/AGENTS.md): notes for contributors and agents
|
|
270
|
+
|
|
271
|
+
The crates are `urna-format` (the container), `urna-runtime` (search), `urna` (the binary) and `urna-python` (the bridge).
|
|
272
|
+
|
|
273
|
+
> Renamed from `nest` after 0.4.0. A `.nest` file written by 0.4.0 still opens.
|
|
274
|
+
|
|
275
|
+
## License
|
|
276
|
+
|
|
277
|
+
MIT, see [docs/LICENSE](https://github.com/hoffresearch/urna/blob/main/docs/LICENSE). [Hoff Research](https://hoffresearch.com)
|
|
278
|
+
|
|
279
|
+
Made it simple, but significant (∂μfμν = jν)
|
|
280
|
+
|
|
281
|
+
Author: Brenner Cruvinel
|
|
@@ -0,0 +1,348 @@
|
|
|
1
|
+
const {
|
|
2
|
+
createWriteStream,
|
|
3
|
+
existsSync,
|
|
4
|
+
mkdirSync,
|
|
5
|
+
mkdtemp,
|
|
6
|
+
rmSync,
|
|
7
|
+
} = require("fs");
|
|
8
|
+
const { join, sep } = require("path");
|
|
9
|
+
const { spawnSync } = require("child_process");
|
|
10
|
+
const { tmpdir } = require("os");
|
|
11
|
+
|
|
12
|
+
const https = require("node:https");
|
|
13
|
+
const http = require("node:http");
|
|
14
|
+
|
|
15
|
+
const tmpDir = tmpdir();
|
|
16
|
+
|
|
17
|
+
const error = (msg) => {
|
|
18
|
+
console.error(msg);
|
|
19
|
+
process.exit(1);
|
|
20
|
+
};
|
|
21
|
+
|
|
22
|
+
function getProxyForUrl(urlString) {
|
|
23
|
+
const url = new URL(urlString);
|
|
24
|
+
const isHttps = url.protocol === "https:";
|
|
25
|
+
|
|
26
|
+
const noProxy = process.env.NO_PROXY || process.env.no_proxy || "";
|
|
27
|
+
if (noProxy === "*") return null;
|
|
28
|
+
if (noProxy) {
|
|
29
|
+
const hostname = url.hostname.toLowerCase();
|
|
30
|
+
const noProxyList = noProxy.split(",").map((s) => s.trim().toLowerCase());
|
|
31
|
+
for (const entry of noProxyList) {
|
|
32
|
+
if (hostname === entry || hostname.endsWith("." + entry)) {
|
|
33
|
+
return null;
|
|
34
|
+
}
|
|
35
|
+
}
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
const proxyEnv = isHttps
|
|
39
|
+
? process.env.HTTPS_PROXY || process.env.https_proxy
|
|
40
|
+
: process.env.HTTP_PROXY || process.env.http_proxy;
|
|
41
|
+
|
|
42
|
+
if (!proxyEnv) return null;
|
|
43
|
+
|
|
44
|
+
const proxyUrl = new URL(proxyEnv);
|
|
45
|
+
|
|
46
|
+
let auth = null;
|
|
47
|
+
if (proxyUrl.username || proxyUrl.password) {
|
|
48
|
+
auth = `${proxyUrl.username}:${proxyUrl.password}`;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
return {
|
|
52
|
+
hostname: proxyUrl.hostname,
|
|
53
|
+
port: proxyUrl.port || (proxyUrl.protocol === "https:" ? 443 : 80),
|
|
54
|
+
auth: auth,
|
|
55
|
+
};
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
function connectThroughProxy(proxy, target) {
|
|
59
|
+
return new Promise((resolve, reject) => {
|
|
60
|
+
const headers = {};
|
|
61
|
+
if (proxy.auth) {
|
|
62
|
+
headers["Proxy-Authorization"] =
|
|
63
|
+
"Basic " + Buffer.from(proxy.auth).toString("base64");
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
const connectReq = http.request({
|
|
67
|
+
hostname: proxy.hostname,
|
|
68
|
+
port: proxy.port,
|
|
69
|
+
method: "CONNECT",
|
|
70
|
+
path: `${target.hostname}:${target.port || 443}`,
|
|
71
|
+
headers,
|
|
72
|
+
});
|
|
73
|
+
connectReq.on("connect", (res, socket) => {
|
|
74
|
+
if (res.statusCode === 200) {
|
|
75
|
+
resolve(socket);
|
|
76
|
+
} else {
|
|
77
|
+
reject(new Error(`Proxy CONNECT failed with status ${res.statusCode}`));
|
|
78
|
+
}
|
|
79
|
+
});
|
|
80
|
+
connectReq.on("error", reject);
|
|
81
|
+
connectReq.end();
|
|
82
|
+
});
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
function download(urlString, maxRedirects) {
|
|
86
|
+
if (maxRedirects === undefined) maxRedirects = 5;
|
|
87
|
+
return new Promise((resolve, reject) => {
|
|
88
|
+
if (maxRedirects < 0) {
|
|
89
|
+
return reject(new Error("Too many redirects"));
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
const parsed = new URL(urlString);
|
|
93
|
+
const isHttps = parsed.protocol === "https:";
|
|
94
|
+
const mod = isHttps ? https : http;
|
|
95
|
+
const proxy = getProxyForUrl(urlString);
|
|
96
|
+
|
|
97
|
+
const doRequest = (extraOptions) => {
|
|
98
|
+
const options = Object.assign(
|
|
99
|
+
{
|
|
100
|
+
hostname: parsed.hostname,
|
|
101
|
+
port: parsed.port || (isHttps ? 443 : 80),
|
|
102
|
+
path: parsed.pathname + parsed.search,
|
|
103
|
+
method: "GET",
|
|
104
|
+
headers: { "User-Agent": "cargo-dist-npm-installer" },
|
|
105
|
+
},
|
|
106
|
+
extraOptions || {},
|
|
107
|
+
);
|
|
108
|
+
|
|
109
|
+
if (proxy && !isHttps) {
|
|
110
|
+
// HTTP through HTTP proxy: request the full URL via the proxy
|
|
111
|
+
options.hostname = proxy.hostname;
|
|
112
|
+
options.port = proxy.port;
|
|
113
|
+
options.path = urlString;
|
|
114
|
+
if (proxy.auth) {
|
|
115
|
+
options.headers["Proxy-Authorization"] =
|
|
116
|
+
"Basic " + Buffer.from(proxy.auth).toString("base64");
|
|
117
|
+
}
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
const req = mod.request(options, (res) => {
|
|
121
|
+
if (
|
|
122
|
+
res.statusCode >= 300 &&
|
|
123
|
+
res.statusCode < 400 &&
|
|
124
|
+
res.headers.location
|
|
125
|
+
) {
|
|
126
|
+
res.resume();
|
|
127
|
+
const nextUrl = new URL(res.headers.location, urlString).toString();
|
|
128
|
+
return download(nextUrl, maxRedirects - 1).then(resolve, reject);
|
|
129
|
+
}
|
|
130
|
+
if (res.statusCode < 200 || res.statusCode >= 300) {
|
|
131
|
+
res.resume();
|
|
132
|
+
return reject(new Error(`HTTP ${res.statusCode} from ${urlString}`));
|
|
133
|
+
}
|
|
134
|
+
resolve(res);
|
|
135
|
+
});
|
|
136
|
+
req.on("error", reject);
|
|
137
|
+
req.end();
|
|
138
|
+
};
|
|
139
|
+
|
|
140
|
+
if (proxy && isHttps) {
|
|
141
|
+
connectThroughProxy(proxy, parsed).then(
|
|
142
|
+
(socket) => doRequest({ socket, agent: false }),
|
|
143
|
+
reject,
|
|
144
|
+
);
|
|
145
|
+
} else {
|
|
146
|
+
doRequest();
|
|
147
|
+
}
|
|
148
|
+
});
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
class Package {
|
|
152
|
+
constructor(platform, name, url, filename, zipExt, binaries) {
|
|
153
|
+
let errors = [];
|
|
154
|
+
if (typeof url !== "string") {
|
|
155
|
+
errors.push("url must be a string");
|
|
156
|
+
} else {
|
|
157
|
+
try {
|
|
158
|
+
new URL(url);
|
|
159
|
+
} catch (e) {
|
|
160
|
+
errors.push(e);
|
|
161
|
+
}
|
|
162
|
+
}
|
|
163
|
+
if (name && typeof name !== "string") {
|
|
164
|
+
errors.push("package name must be a string");
|
|
165
|
+
}
|
|
166
|
+
if (!name) {
|
|
167
|
+
errors.push("You must specify the name of your package");
|
|
168
|
+
}
|
|
169
|
+
if (binaries && typeof binaries !== "object") {
|
|
170
|
+
errors.push("binaries must be a string => string map");
|
|
171
|
+
}
|
|
172
|
+
if (!binaries) {
|
|
173
|
+
errors.push("You must specify the binaries in the package");
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
if (errors.length > 0) {
|
|
177
|
+
let errorMsg =
|
|
178
|
+
"One or more of the parameters you passed to the Binary constructor are invalid:\n";
|
|
179
|
+
errors.forEach((error) => {
|
|
180
|
+
errorMsg += error;
|
|
181
|
+
});
|
|
182
|
+
errorMsg +=
|
|
183
|
+
'\n\nCorrect usage: new Package("my-binary", "https://example.com/binary/download.tar.gz", {"my-binary": "my-binary"})';
|
|
184
|
+
error(errorMsg);
|
|
185
|
+
}
|
|
186
|
+
|
|
187
|
+
this.platform = platform;
|
|
188
|
+
this.url = url;
|
|
189
|
+
this.name = name;
|
|
190
|
+
this.filename = filename;
|
|
191
|
+
this.zipExt = zipExt;
|
|
192
|
+
this.installDirectory = join(__dirname, "node_modules", ".bin_real");
|
|
193
|
+
this.binaries = binaries;
|
|
194
|
+
|
|
195
|
+
if (!existsSync(this.installDirectory)) {
|
|
196
|
+
mkdirSync(this.installDirectory, { recursive: true });
|
|
197
|
+
}
|
|
198
|
+
}
|
|
199
|
+
|
|
200
|
+
exists() {
|
|
201
|
+
for (const binaryName in this.binaries) {
|
|
202
|
+
const binRelPath = this.binaries[binaryName];
|
|
203
|
+
const binPath = join(this.installDirectory, binRelPath);
|
|
204
|
+
if (!existsSync(binPath)) {
|
|
205
|
+
return false;
|
|
206
|
+
}
|
|
207
|
+
}
|
|
208
|
+
return true;
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
install(suppressLogs = false) {
|
|
212
|
+
if (this.exists()) {
|
|
213
|
+
if (!suppressLogs) {
|
|
214
|
+
console.error(
|
|
215
|
+
`${this.name} is already installed, skipping installation.`,
|
|
216
|
+
);
|
|
217
|
+
}
|
|
218
|
+
return Promise.resolve();
|
|
219
|
+
}
|
|
220
|
+
|
|
221
|
+
try {
|
|
222
|
+
rmSync(this.installDirectory, { recursive: true, force: true });
|
|
223
|
+
} catch {
|
|
224
|
+
// ignore - directory may not exist
|
|
225
|
+
}
|
|
226
|
+
|
|
227
|
+
mkdirSync(this.installDirectory, { recursive: true });
|
|
228
|
+
|
|
229
|
+
if (!suppressLogs) {
|
|
230
|
+
console.error(`Downloading release from ${this.url}`);
|
|
231
|
+
}
|
|
232
|
+
|
|
233
|
+
return download(this.url)
|
|
234
|
+
.then((res) => {
|
|
235
|
+
return new Promise((resolve, reject) => {
|
|
236
|
+
mkdtemp(`${tmpDir}${sep}`, (err, directory) => {
|
|
237
|
+
if (err) return reject(err);
|
|
238
|
+
let tempFile = join(directory, this.filename);
|
|
239
|
+
const sink = res.pipe(createWriteStream(tempFile));
|
|
240
|
+
sink.on("error", (err) => reject(err));
|
|
241
|
+
sink.on("close", () => {
|
|
242
|
+
if (/\.tar\.*/.test(this.zipExt)) {
|
|
243
|
+
const result = spawnSync("tar", [
|
|
244
|
+
"xf",
|
|
245
|
+
tempFile,
|
|
246
|
+
// The tarballs are stored with a leading directory
|
|
247
|
+
// component; we strip one component in the
|
|
248
|
+
// shell installers too.
|
|
249
|
+
"--strip-components",
|
|
250
|
+
"1",
|
|
251
|
+
"-C",
|
|
252
|
+
this.installDirectory,
|
|
253
|
+
]);
|
|
254
|
+
if (result.status == 0) {
|
|
255
|
+
resolve();
|
|
256
|
+
} else if (result.error) {
|
|
257
|
+
reject(result.error);
|
|
258
|
+
} else {
|
|
259
|
+
reject(
|
|
260
|
+
new Error(
|
|
261
|
+
`An error occurred untarring the artifact: stdout: ${result.stdout}; stderr: ${result.stderr}`,
|
|
262
|
+
),
|
|
263
|
+
);
|
|
264
|
+
}
|
|
265
|
+
} else if (this.zipExt == ".zip") {
|
|
266
|
+
let result;
|
|
267
|
+
if (this.platform.artifactName.includes("windows")) {
|
|
268
|
+
// Windows does not have "unzip" by default on many installations, instead
|
|
269
|
+
// we use Expand-Archive from powershell
|
|
270
|
+
result = spawnSync("powershell.exe", [
|
|
271
|
+
"-NoProfile",
|
|
272
|
+
"-NonInteractive",
|
|
273
|
+
"-Command",
|
|
274
|
+
`& {
|
|
275
|
+
param([string]$LiteralPath, [string]$DestinationPath)
|
|
276
|
+
Expand-Archive -LiteralPath $LiteralPath -DestinationPath $DestinationPath -Force
|
|
277
|
+
}`,
|
|
278
|
+
tempFile,
|
|
279
|
+
this.installDirectory,
|
|
280
|
+
]);
|
|
281
|
+
} else {
|
|
282
|
+
result = spawnSync("unzip", [
|
|
283
|
+
"-q",
|
|
284
|
+
tempFile,
|
|
285
|
+
"-d",
|
|
286
|
+
this.installDirectory,
|
|
287
|
+
]);
|
|
288
|
+
}
|
|
289
|
+
|
|
290
|
+
if (result.status == 0) {
|
|
291
|
+
resolve();
|
|
292
|
+
} else if (result.error) {
|
|
293
|
+
reject(result.error);
|
|
294
|
+
} else {
|
|
295
|
+
reject(
|
|
296
|
+
new Error(
|
|
297
|
+
`An error occurred unzipping the artifact: stdout: ${result.stdout}; stderr: ${result.stderr}`,
|
|
298
|
+
),
|
|
299
|
+
);
|
|
300
|
+
}
|
|
301
|
+
} else {
|
|
302
|
+
reject(
|
|
303
|
+
new Error(`Unrecognized file extension: ${this.zipExt}`),
|
|
304
|
+
);
|
|
305
|
+
}
|
|
306
|
+
});
|
|
307
|
+
});
|
|
308
|
+
});
|
|
309
|
+
})
|
|
310
|
+
.then(() => {
|
|
311
|
+
if (!suppressLogs) {
|
|
312
|
+
console.error(`${this.name} has been installed!`);
|
|
313
|
+
}
|
|
314
|
+
})
|
|
315
|
+
.catch((e) => {
|
|
316
|
+
error(`Error fetching release: ${e.message}`);
|
|
317
|
+
});
|
|
318
|
+
}
|
|
319
|
+
|
|
320
|
+
run(binaryName) {
|
|
321
|
+
const promise = !this.exists() ? this.install(true) : Promise.resolve();
|
|
322
|
+
|
|
323
|
+
promise
|
|
324
|
+
.then(() => {
|
|
325
|
+
const [, , ...args] = process.argv;
|
|
326
|
+
|
|
327
|
+
const options = { cwd: process.cwd(), stdio: "inherit" };
|
|
328
|
+
|
|
329
|
+
const binRelPath = this.binaries[binaryName];
|
|
330
|
+
if (!binRelPath) {
|
|
331
|
+
error(`${binaryName} is not a known binary in ${this.name}`);
|
|
332
|
+
}
|
|
333
|
+
const binPath = join(this.installDirectory, binRelPath);
|
|
334
|
+
const result = spawnSync(binPath, args, options);
|
|
335
|
+
|
|
336
|
+
if (result.error) {
|
|
337
|
+
error(result.error);
|
|
338
|
+
}
|
|
339
|
+
|
|
340
|
+
process.exit(result.status);
|
|
341
|
+
})
|
|
342
|
+
.catch((e) => {
|
|
343
|
+
error(e.message);
|
|
344
|
+
});
|
|
345
|
+
}
|
|
346
|
+
}
|
|
347
|
+
|
|
348
|
+
module.exports.Package = Package;
|
package/binary.js
ADDED
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
const { Package } = require("./binary-install");
|
|
2
|
+
const os = require("os");
|
|
3
|
+
const libc = require("detect-libc");
|
|
4
|
+
|
|
5
|
+
const error = (msg) => {
|
|
6
|
+
console.error(msg);
|
|
7
|
+
process.exit(1);
|
|
8
|
+
};
|
|
9
|
+
|
|
10
|
+
const {
|
|
11
|
+
name,
|
|
12
|
+
artifactDownloadUrls,
|
|
13
|
+
supportedPlatforms,
|
|
14
|
+
glibcMinimum,
|
|
15
|
+
} = require("./package.json");
|
|
16
|
+
|
|
17
|
+
// FIXME: implement NPM installer handling of fallback download URLs
|
|
18
|
+
const artifactDownloadUrl = artifactDownloadUrls[0];
|
|
19
|
+
const builderGlibcMajorVersion = glibcMinimum.major;
|
|
20
|
+
const builderGlibcMinorVersion = glibcMinimum.series;
|
|
21
|
+
|
|
22
|
+
const getPlatform = () => {
|
|
23
|
+
const rawOsType = os.type();
|
|
24
|
+
const rawArchitecture = os.arch();
|
|
25
|
+
|
|
26
|
+
// We want to use rust-style target triples as the canonical key
|
|
27
|
+
// for a platform, so translate the "os" library's concepts into rust ones
|
|
28
|
+
let osType = "";
|
|
29
|
+
switch (rawOsType) {
|
|
30
|
+
case "Windows_NT":
|
|
31
|
+
osType = "pc-windows-msvc";
|
|
32
|
+
break;
|
|
33
|
+
case "Darwin":
|
|
34
|
+
osType = "apple-darwin";
|
|
35
|
+
break;
|
|
36
|
+
case "Linux":
|
|
37
|
+
osType = "unknown-linux-gnu";
|
|
38
|
+
break;
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
let arch = "";
|
|
42
|
+
switch (rawArchitecture) {
|
|
43
|
+
case "x64":
|
|
44
|
+
arch = "x86_64";
|
|
45
|
+
break;
|
|
46
|
+
case "arm64":
|
|
47
|
+
arch = "aarch64";
|
|
48
|
+
break;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
if (rawOsType === "Linux") {
|
|
52
|
+
if (libc.familySync() == "musl") {
|
|
53
|
+
osType = "unknown-linux-musl-dynamic";
|
|
54
|
+
} else if (libc.isNonGlibcLinuxSync()) {
|
|
55
|
+
console.warn(
|
|
56
|
+
"Your libc is neither glibc nor musl; trying static musl binary instead",
|
|
57
|
+
);
|
|
58
|
+
osType = "unknown-linux-musl-static";
|
|
59
|
+
} else {
|
|
60
|
+
let libcVersion = libc.versionSync();
|
|
61
|
+
let splitLibcVersion = libcVersion.split(".");
|
|
62
|
+
let libcMajorVersion = splitLibcVersion[0];
|
|
63
|
+
let libcMinorVersion = splitLibcVersion[1];
|
|
64
|
+
if (
|
|
65
|
+
libcMajorVersion != builderGlibcMajorVersion ||
|
|
66
|
+
libcMinorVersion < builderGlibcMinorVersion
|
|
67
|
+
) {
|
|
68
|
+
// We can't run the glibc binaries, but we can run the static musl ones
|
|
69
|
+
// if they exist
|
|
70
|
+
console.warn(
|
|
71
|
+
"Your glibc isn't compatible; trying static musl binary instead",
|
|
72
|
+
);
|
|
73
|
+
osType = "unknown-linux-musl-static";
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
// Assume the above succeeded and build a target triple to look things up with.
|
|
79
|
+
// If any of it failed, this lookup will fail and we'll handle it like normal.
|
|
80
|
+
let targetTriple = `${arch}-${osType}`;
|
|
81
|
+
let platform = supportedPlatforms[targetTriple];
|
|
82
|
+
|
|
83
|
+
if (!platform) {
|
|
84
|
+
error(
|
|
85
|
+
`Platform with type "${rawOsType}" and architecture "${rawArchitecture}" is not supported by ${name}.\nYour system must be one of the following:\n\n${Object.keys(
|
|
86
|
+
supportedPlatforms,
|
|
87
|
+
).join(",")}`,
|
|
88
|
+
);
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
return platform;
|
|
92
|
+
};
|
|
93
|
+
|
|
94
|
+
const getPackage = () => {
|
|
95
|
+
const platform = getPlatform();
|
|
96
|
+
const url = `${artifactDownloadUrl}/${platform.artifactName}`;
|
|
97
|
+
let filename = platform.artifactName;
|
|
98
|
+
let ext = platform.zipExt;
|
|
99
|
+
let binary = new Package(platform, name, url, filename, ext, platform.bins);
|
|
100
|
+
|
|
101
|
+
return binary;
|
|
102
|
+
};
|
|
103
|
+
|
|
104
|
+
const install = (suppressLogs) => {
|
|
105
|
+
if (!artifactDownloadUrl || artifactDownloadUrl.length === 0) {
|
|
106
|
+
console.warn("in demo mode, not installing binaries");
|
|
107
|
+
return;
|
|
108
|
+
}
|
|
109
|
+
const pkg = getPackage();
|
|
110
|
+
|
|
111
|
+
return pkg.install(suppressLogs);
|
|
112
|
+
};
|
|
113
|
+
|
|
114
|
+
const run = (binaryName) => {
|
|
115
|
+
const pkg = getPackage();
|
|
116
|
+
|
|
117
|
+
pkg.run(binaryName);
|
|
118
|
+
};
|
|
119
|
+
|
|
120
|
+
module.exports = {
|
|
121
|
+
install,
|
|
122
|
+
run,
|
|
123
|
+
getPackage,
|
|
124
|
+
};
|
package/install.js
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
{
|
|
2
|
+
"lockfileVersion": 3,
|
|
3
|
+
"name": "@urna/cli",
|
|
4
|
+
"packages": {
|
|
5
|
+
"": {
|
|
6
|
+
"bin": {
|
|
7
|
+
"urna": "run-urna.js"
|
|
8
|
+
},
|
|
9
|
+
"dependencies": {
|
|
10
|
+
"detect-libc": "^2.1.2"
|
|
11
|
+
},
|
|
12
|
+
"devDependencies": {
|
|
13
|
+
"prettier": "^3.8.3"
|
|
14
|
+
},
|
|
15
|
+
"engines": {
|
|
16
|
+
"node": ">=14.14",
|
|
17
|
+
"npm": ">=6"
|
|
18
|
+
},
|
|
19
|
+
"hasInstallScript": true,
|
|
20
|
+
"license": "MIT",
|
|
21
|
+
"name": "@urna/cli",
|
|
22
|
+
"version": "0.5.0"
|
|
23
|
+
},
|
|
24
|
+
"node_modules/detect-libc": {
|
|
25
|
+
"engines": {
|
|
26
|
+
"node": ">=8"
|
|
27
|
+
},
|
|
28
|
+
"integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
|
|
29
|
+
"license": "Apache-2.0",
|
|
30
|
+
"resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
|
|
31
|
+
"version": "2.1.2"
|
|
32
|
+
},
|
|
33
|
+
"node_modules/prettier": {
|
|
34
|
+
"bin": {
|
|
35
|
+
"prettier": "bin/prettier.cjs"
|
|
36
|
+
},
|
|
37
|
+
"dev": true,
|
|
38
|
+
"engines": {
|
|
39
|
+
"node": ">=14"
|
|
40
|
+
},
|
|
41
|
+
"funding": {
|
|
42
|
+
"url": "https://github.com/prettier/prettier?sponsor=1"
|
|
43
|
+
},
|
|
44
|
+
"integrity": "sha512-7igPTM53cGHMW8xWuVTydi2KO233VFiTNyF5hLJqpilHfmn8C8gPf+PS7dUT64YcXFbiMGZxS9pCSxL/Dxm/Jw==",
|
|
45
|
+
"license": "MIT",
|
|
46
|
+
"resolved": "https://registry.npmjs.org/prettier/-/prettier-3.8.3.tgz",
|
|
47
|
+
"version": "3.8.3"
|
|
48
|
+
}
|
|
49
|
+
},
|
|
50
|
+
"requires": true,
|
|
51
|
+
"version": "0.5.0"
|
|
52
|
+
}
|
package/package.json
ADDED
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
{
|
|
2
|
+
"artifactDownloadUrls": [
|
|
3
|
+
"https://github.com/hoffresearch/urna/releases/download/v0.5.0"
|
|
4
|
+
],
|
|
5
|
+
"bin": {
|
|
6
|
+
"urna": "run-urna.js"
|
|
7
|
+
},
|
|
8
|
+
"dependencies": {
|
|
9
|
+
"detect-libc": "^2.1.2"
|
|
10
|
+
},
|
|
11
|
+
"description": "sovereign embedded vector database: single-file .urna container with content-addressable citations, offline-first",
|
|
12
|
+
"devDependencies": {
|
|
13
|
+
"prettier": "^3.8.3"
|
|
14
|
+
},
|
|
15
|
+
"engines": {
|
|
16
|
+
"node": ">=14.14",
|
|
17
|
+
"npm": ">=6"
|
|
18
|
+
},
|
|
19
|
+
"glibcMinimum": {
|
|
20
|
+
"major": 2,
|
|
21
|
+
"series": 31
|
|
22
|
+
},
|
|
23
|
+
"homepage": "https://urna.dev",
|
|
24
|
+
"keywords": [
|
|
25
|
+
"command-line-utilities",
|
|
26
|
+
"database-implementations",
|
|
27
|
+
"vector-database",
|
|
28
|
+
"embeddings",
|
|
29
|
+
"search",
|
|
30
|
+
"offline",
|
|
31
|
+
"citations"
|
|
32
|
+
],
|
|
33
|
+
"license": "MIT",
|
|
34
|
+
"name": "@urna/cli",
|
|
35
|
+
"preferUnplugged": true,
|
|
36
|
+
"repository": "https://github.com/hoffresearch/urna",
|
|
37
|
+
"scripts": {
|
|
38
|
+
"fmt": "prettier --write **/*.js",
|
|
39
|
+
"fmt:check": "prettier --check **/*.js",
|
|
40
|
+
"postinstall": "node ./install.js"
|
|
41
|
+
},
|
|
42
|
+
"supportedPlatforms": {
|
|
43
|
+
"aarch64-apple-darwin": {
|
|
44
|
+
"artifactName": "urna-aarch64-apple-darwin.tar.xz",
|
|
45
|
+
"bins": {
|
|
46
|
+
"urna": "urna"
|
|
47
|
+
},
|
|
48
|
+
"zipExt": ".tar.xz"
|
|
49
|
+
},
|
|
50
|
+
"aarch64-pc-windows-msvc": {
|
|
51
|
+
"artifactName": "urna-x86_64-pc-windows-msvc.zip",
|
|
52
|
+
"bins": {
|
|
53
|
+
"urna": "urna.exe"
|
|
54
|
+
},
|
|
55
|
+
"zipExt": ".zip"
|
|
56
|
+
},
|
|
57
|
+
"aarch64-unknown-linux-gnu": {
|
|
58
|
+
"artifactName": "urna-aarch64-unknown-linux-musl.tar.xz",
|
|
59
|
+
"bins": {
|
|
60
|
+
"urna": "urna"
|
|
61
|
+
},
|
|
62
|
+
"zipExt": ".tar.xz"
|
|
63
|
+
},
|
|
64
|
+
"aarch64-unknown-linux-musl-dynamic": {
|
|
65
|
+
"artifactName": "urna-aarch64-unknown-linux-musl.tar.xz",
|
|
66
|
+
"bins": {
|
|
67
|
+
"urna": "urna"
|
|
68
|
+
},
|
|
69
|
+
"zipExt": ".tar.xz"
|
|
70
|
+
},
|
|
71
|
+
"aarch64-unknown-linux-musl-static": {
|
|
72
|
+
"artifactName": "urna-aarch64-unknown-linux-musl.tar.xz",
|
|
73
|
+
"bins": {
|
|
74
|
+
"urna": "urna"
|
|
75
|
+
},
|
|
76
|
+
"zipExt": ".tar.xz"
|
|
77
|
+
},
|
|
78
|
+
"x86_64-apple-darwin": {
|
|
79
|
+
"artifactName": "urna-x86_64-apple-darwin.tar.xz",
|
|
80
|
+
"bins": {
|
|
81
|
+
"urna": "urna"
|
|
82
|
+
},
|
|
83
|
+
"zipExt": ".tar.xz"
|
|
84
|
+
},
|
|
85
|
+
"x86_64-pc-windows-gnu": {
|
|
86
|
+
"artifactName": "urna-x86_64-pc-windows-msvc.zip",
|
|
87
|
+
"bins": {
|
|
88
|
+
"urna": "urna.exe"
|
|
89
|
+
},
|
|
90
|
+
"zipExt": ".zip"
|
|
91
|
+
},
|
|
92
|
+
"x86_64-pc-windows-msvc": {
|
|
93
|
+
"artifactName": "urna-x86_64-pc-windows-msvc.zip",
|
|
94
|
+
"bins": {
|
|
95
|
+
"urna": "urna.exe"
|
|
96
|
+
},
|
|
97
|
+
"zipExt": ".zip"
|
|
98
|
+
},
|
|
99
|
+
"x86_64-unknown-linux-gnu": {
|
|
100
|
+
"artifactName": "urna-x86_64-unknown-linux-musl.tar.xz",
|
|
101
|
+
"bins": {
|
|
102
|
+
"urna": "urna"
|
|
103
|
+
},
|
|
104
|
+
"zipExt": ".tar.xz"
|
|
105
|
+
},
|
|
106
|
+
"x86_64-unknown-linux-musl-dynamic": {
|
|
107
|
+
"artifactName": "urna-x86_64-unknown-linux-musl.tar.xz",
|
|
108
|
+
"bins": {
|
|
109
|
+
"urna": "urna"
|
|
110
|
+
},
|
|
111
|
+
"zipExt": ".tar.xz"
|
|
112
|
+
},
|
|
113
|
+
"x86_64-unknown-linux-musl-static": {
|
|
114
|
+
"artifactName": "urna-x86_64-unknown-linux-musl.tar.xz",
|
|
115
|
+
"bins": {
|
|
116
|
+
"urna": "urna"
|
|
117
|
+
},
|
|
118
|
+
"zipExt": ".tar.xz"
|
|
119
|
+
}
|
|
120
|
+
},
|
|
121
|
+
"version": "0.5.0",
|
|
122
|
+
"volta": {
|
|
123
|
+
"node": "18.14.1",
|
|
124
|
+
"npm": "9.5.0"
|
|
125
|
+
}
|
|
126
|
+
}
|
package/run-urna.js
ADDED