@qvac/harness 0.4.0 → 0.6.0

package/README.md

@@ -87,7 +87,8 @@ const preview = await harness.previewModelSource({
 A preview is one of three answers:
 
 - **A file**: `ok: true` with `src`, the download link to register. A Hugging
-  Face page link is resolved to its `resolve/` url.
+  Face page link is resolved to its `resolve/` url, and a direct link to one
+  shard of a split model to its first part, sized as the whole set.
 - **A choice**: `ok: true` with `files: [{ src, label, sizeBytes }]` and no
   `src`, for a repo that holds several weights. Split shards are one entry and
   projections are left out. Preview the chosen `files[i].src` again.
@@ -132,7 +133,9 @@ await harness.setCustomModels([
     src: preview.src,
     projectionSrc: preview.mmprojSrc,
     taskType: preview.taskType, // 'llm'; an entry for another task is not registered
-    expectedSize: preview.sizeBytes
+    expectedSize: preview.sizeBytes,
+    label: 'Nanbeige 4.2 3B', // what the user calls it
+    contextLength: preview.contextLength // the window it was trained on
   }
 ])
 ```
@@ -140,6 +143,12 @@ await harness.setCustomModels([
 A registered name loads like a catalog model wherever an `llm` name is taken,
 and never resolves for another task. A missing `taskType` means `llm`.
 `expectedSize` is what `isCached` compares the downloaded files against.
+`label` is the name to show. The harness also reads the model family from it
+(Qwen, Gemma, LFM and so on) to pick the tool-call format, so a model whose id
+and file name don't carry the family still gets the right one.
+`contextLength` caps the context a custom model loads with by default, so a
+model trained on 40,960 tokens loads with 40,960 rather than the 51,200
+default. A context size the caller asks for still wins.
 
 ### List them
 
@@ -158,7 +167,8 @@ const custom = models.filter((model) => model.registrySource === 'custom')
 //   isCached: false,
 //   isLoaded: false,
 //   supportsImages: false,
-//   expectedSize: 1868506272
+//   expectedSize: 1868506272,
+//   label: 'Nanbeige 4.2 3B'
 // }
 ```
 
@@ -168,8 +178,8 @@ known, their sizes add up to it, so a partial download is not cached. A local
 path is checked where it is. A downloaded file is looked for at the path the SDK
 gives a single-file http download under the models cache dir, which the spawned
 sidecar knows from `sdkStorage.models`; an in-process `Harness` needs
-`new Harness({ modelsDir })` for it. A sharded or archive url always reads as
-not cached.
+`new Harness({ modelsDir })` for it. A sharded download is looked for in the
+folder the SDK keeps its shards in, and is cached once every shard is there.
 
 ### Delete a download
 
@@ -177,11 +187,69 @@ not cached.
 const { removed, freedBytes } = await harness.deleteModel('nanbeige-3b')
 ```
 
-For a custom model this removes the weight and projection it downloaded, and
-never a local path: those are the user's files. The model stays registered and
+For a custom model this removes the weight and projection it downloaded,
+including every shard of a sharded one, and never a local path: those are the
+user's files. The model stays registered and
 lists as not cached, ready to download again. Like a catalog delete, it does not
 unload a model that is running.
 
+### Load and unload a model
+
+```js
+import { QWEN_35_4B } from '@qvac/harness'
+
+const { loaded } = await harness.loadModel({ model: QWEN_35_4B, agent: { ctxSize: 8192 } })
+const row = (await harness.list({ cachedOnly: true })).find((model) => model.name === QWEN_35_4B)
+// row.isLoaded === true, and the next completion with the same agent reuses it
+
+const { unloaded } = await harness.unloadModel({ model: QWEN_35_4B })
+```
+
+`loadModel` loads a downloaded model ahead of use, under the key a request for it
+would load under, so the first request does not load it again. Pass the agent a
+chat model will run for, since its context size and KV cache types are part of
+that key. Chat, embedding, speech and transcription models stay loaded; image and
+music models load for each generation and are refused.
+
+A chat model loaded this way is pinned. It does not count against
+`completionModelCap`, so several can be loaded at once, and a request loading
+another model leaves it in place. A request or a manual load for the same model
+under other settings replaces its idle copy rather than loading a second one. A
+manual load never unloads another model to fit: `{ loaded: false, reason: 'memory' }` means the
+SDK's `assessModelFit` estimate says it would not fit beside what is loaded, so
+unload one first. A request always gets its model, though: when that would not
+fit, pinned models no reply is using give way, oldest first. Image or music generation and an idle
+unload still clear every idle chat model, pinned or not.
+
+```js
+const big = await harness.loadModel({ model: QWEN_36_27B, agent: { ctxSize: 262144 } })
+if (big.reason === 'memory') showNotice('Unload a model first')
+```
+
+`unloadModel` waits for any request holding the model to finish, then unloads
+every copy of it. `unloaded` is false when a request still held it after the
+wait, so a caller that wants the model gone at once stops those requests first.
+Both change `isLoaded` in the next `list()`, and emit a `model` event with
+`catalogChanged` set. A load or an unload counts as a call in flight for
+[`setModelIdleTimeout`](#freeing-memory-when-idle), and one that lands during an
+idle unload waits for it, so a load never loads a second copy and an unload
+answers once the memory is free.
+
+`bun run model-residency` (`bare examples/model-residency/run.ts`, with
+`QVAC_MODEL_CACHE_PATH` pointing at the models) runs these calls against the
+models already on disk and prints what each step changed. It never downloads.
+
+## Freeing memory when idle
+
+```js
+await harness.setModelIdleTimeout(30 * 60_000)
+```
+
+Once no call has been in flight for that long, every idle resident model is
+unloaded, and the next call loads what it needs again. The clock covers the
+whole device: any call resets it, whichever model it uses. `0`, the default, is
+Never.
+
 ## Office extraction
 
 Hosts can supply the bundled Python extractor to their document pipeline: