@qvac/harness 0.3.3 → 0.5.0

package/README.md

@@ -48,6 +48,162 @@ dir, ...) are the ones `worker/harness-sidecar.mjs` documents in this repo. The
 in-house hosts don't call this directly — they use the repo's library surface
 (`harnessChild` under bare-supervisor), which owns that argv contract.
 
+## Custom models
+
+A host can let its users add a GGUF the catalog does not carry: a Hugging Face
+link, a direct link to a `.gguf` file, or a local path. The harness checks the
+source, runs it by the name the host registers it under, lists it next to the
+catalog and deletes what it downloaded. Which custom models a device has is the
+host's state; the harness keeps only the set it was last given.
+
+The calls below are on the client `spawnHarness` returns (a `RemoteHarness`) and
+on an in-process `Harness` alike.
+
+### Preview a source before adding it
+
+`previewModelSource` reads the source's Hugging Face metadata and the first
+megabyte of the file. It never downloads the model.
+
+```js
+const preview = await harness.previewModelSource({
+  src: 'https://huggingface.co/bartowski/Nanbeige_Nanbeige4.2-3B-GGUF/resolve/main/Nanbeige_Nanbeige4.2-3B-IQ2_M.gguf',
+  mmprojSrc: undefined // a vision model's projection, checked with it
+})
+
+// {
+//   ok: true,
+//   src: 'https://huggingface.co/.../Nanbeige_Nanbeige4.2-3B-IQ2_M.gguf',
+//   sizeBytes: 1868506272,
+//   contextLength: 262144,
+//   architecture: 'nanbeige',
+//   sizeLabel: '3B',
+//   taskType: 'llm',
+//   license: 'apache-2.0',
+//   fit: 'unknown',
+//   fitReasons: ['...why the fit is unknown']
+// }
+```
+
+A preview is one of three answers:
+
+- **A file**: `ok: true` with `src`, the download link to register. A Hugging
+  Face page link is resolved to its `resolve/` url, and a direct link to one
+  shard of a split model to its first part, sized as the whole set.
+- **A choice**: `ok: true` with `files: [{ src, label, sizeBytes }]` and no
+  `src`, for a repo that holds several weights. Split shards are one entry and
+  projections are left out. Preview the chosen `files[i].src` again.
+- **A refusal**: `ok: false` with `refusal` and a `reason` ready to show. The
+  refusals are `invalid-source`, `not-found`, `private`, `gated`, `web-page`,
+  `unreachable`, `not-gguf`, `not-a-model` (a projection or LoRA adapter),
+  `no-model-files` and `invalid-projection`.
+
+```js
+if (!preview.ok) return showError(preview.reason)
+if (!preview.src) return pickOneOf(preview.files)
+```
+
+`taskType` says what the model does: `llm`, `embeddings`, `speech-to-text`,
+`text-to-speech` or `image-generation`. It comes from the Hugging Face pipeline
+tag, then the card's and the file's tags, then the architecture. The preview
+reports it for any model and refuses none for its task, so the host decides
+what may be added. Only an `llm` runs as a custom model today:
+
+```js
+if (preview.taskType !== 'llm') {
+  return showError(`This is a ${preview.taskType} model. Custom models run as the text model only.`)
+}
+```
+
+`fit` is `likely-fits`, `likely-too-large` or `unknown`, and `fitReasons` says
+why. It comes only from the SDK, for a file whose sha256 its catalog knows, so
+most custom files are `unknown`. It is never a reason to refuse.
+
+`sizeLabel` is the file's own `general.size_label` as written (`0.6B`, `3B`,
+`30B-A3B`), and is absent when the file records none.
+
+### Register the set
+
+`setCustomModels` replaces the whole set. Send it on start and whenever the
+host's store changes; the client sends the last set again after a reconnect.
+
+```js
+await harness.setCustomModels([
+  {
+    name: 'nanbeige-3b',
+    src: preview.src,
+    projectionSrc: preview.mmprojSrc,
+    taskType: preview.taskType, // 'llm'; an entry for another task is not registered
+    expectedSize: preview.sizeBytes,
+    label: 'Nanbeige 4.2 3B', // what the user calls it
+    contextLength: preview.contextLength // the window it was trained on
+  }
+])
+```
+
+A registered name loads like a catalog model wherever an `llm` name is taken,
+and never resolves for another task. A missing `taskType` means `llm`.
+`expectedSize` is what `isCached` compares the downloaded files against.
+`label` is the name to show. The harness also reads the model family from it
+(Qwen, Gemma, LFM and so on) to pick the tool-call format, so a model whose id
+and file name don't carry the family still gets the right one.
+`contextLength` caps the context a custom model loads with by default, so a
+model trained on 40,960 tokens loads with 40,960 rather than the 51,200
+default. A context size the caller asks for still wins.
+
+### List them
+
+`list` returns custom models after the catalog rows, under the same `type` and
+`cachedOnly` filters:
+
+```js
+const models = await harness.list({ type: 'llm' })
+const custom = models.filter((model) => model.registrySource === 'custom')
+
+// {
+//   name: 'nanbeige-3b',
+//   type: 'llm',
+//   registryPath: 'https://huggingface.co/.../Nanbeige_Nanbeige4.2-3B-IQ2_M.gguf',
+//   registrySource: 'custom',
+//   isCached: false,
+//   isLoaded: false,
+//   supportsImages: false,
+//   expectedSize: 1868506272,
+//   label: 'Nanbeige 4.2 3B'
+// }
+```
+
+`registryPath` is the model's `src`, and `supportsImages` is true when it has a
+projection. `isCached` means every file is on disk and, when `expectedSize` is
+known, their sizes add up to it, so a partial download is not cached. A local
+path is checked where it is. A downloaded file is looked for at the path the SDK
+gives a single-file http download under the models cache dir, which the spawned
+sidecar knows from `sdkStorage.models`; an in-process `Harness` needs
+`new Harness({ modelsDir })` for it. A sharded download is looked for in the
+folder the SDK keeps its shards in, and is cached once every shard is there.
+
+### Delete a download
+
+```js
+const { removed, freedBytes } = await harness.deleteModel('nanbeige-3b')
+```
+
+For a custom model this removes the weight and projection it downloaded,
+including every shard of a sharded one, and never a local path: those are the
+user's files. The model stays registered and
+lists as not cached, ready to download again. Like a catalog delete, it does not
+unload a model that is running.
+
+## Freeing memory when idle
+
+```js
+await harness.setModelIdleTimeout(30 * 60_000)
+```
+
+Once no call has been in flight for that long, every idle resident model is
+unloaded, and the next call loads what it needs again. The clock covers the
+whole device: any call resets it, whichever model it uses. `0`, the default, is
+Never.
+
 ## Office extraction
 
 Hosts can supply the bundled Python extractor to their document pipeline: