@qvac/harness 0.4.0 → 0.5.0

package/README.md

@@ -87,7 +87,8 @@ const preview = await harness.previewModelSource({
 A preview is one of three answers:
 
 - **A file**: `ok: true` with `src`, the download link to register. A Hugging
-  Face page link is resolved to its `resolve/` url.
+  Face page link is resolved to its `resolve/` url, and a direct link to one
+  shard of a split model to its first part, sized as the whole set.
 - **A choice**: `ok: true` with `files: [{ src, label, sizeBytes }]` and no
   `src`, for a repo that holds several weights. Split shards are one entry and
   projections are left out. Preview the chosen `files[i].src` again.
@@ -132,7 +133,9 @@ await harness.setCustomModels([
     src: preview.src,
     projectionSrc: preview.mmprojSrc,
     taskType: preview.taskType, // 'llm'; an entry for another task is not registered
-    expectedSize: preview.sizeBytes
+    expectedSize: preview.sizeBytes,
+    label: 'Nanbeige 4.2 3B', // what the user calls it
+    contextLength: preview.contextLength // the window it was trained on
   }
 ])
 ```
@@ -140,6 +143,12 @@ await harness.setCustomModels([
 A registered name loads like a catalog model wherever an `llm` name is taken,
 and never resolves for another task. A missing `taskType` means `llm`.
 `expectedSize` is what `isCached` compares the downloaded files against.
+`label` is the name to show. The harness also reads the model family from it
+(Qwen, Gemma, LFM and so on) to pick the tool-call format, so a model whose id
+and file name don't carry the family still gets the right one.
+`contextLength` caps the context a custom model loads with by default, so a
+model trained on 40,960 tokens loads with 40,960 rather than the 51,200
+default. A context size the caller asks for still wins.
 
 ### List them
 
@@ -158,7 +167,8 @@ const custom = models.filter((model) => model.registrySource === 'custom')
 //   isCached: false,
 //   isLoaded: false,
 //   supportsImages: false,
-//   expectedSize: 1868506272
+//   expectedSize: 1868506272,
+//   label: 'Nanbeige 4.2 3B'
 // }
 ```
 
@@ -168,8 +178,8 @@ known, their sizes add up to it, so a partial download is not cached. A local
 path is checked where it is. A downloaded file is looked for at the path the SDK
 gives a single-file http download under the models cache dir, which the spawned
 sidecar knows from `sdkStorage.models`; an in-process `Harness` needs
-`new Harness({ modelsDir })` for it. A sharded or archive url always reads as
-not cached.
+`new Harness({ modelsDir })` for it. A sharded download is looked for in the
+folder the SDK keeps its shards in, and is cached once every shard is there.
 
 ### Delete a download
 
@@ -177,11 +187,23 @@ not cached.
 const { removed, freedBytes } = await harness.deleteModel('nanbeige-3b')
 ```
 
-For a custom model this removes the weight and projection it downloaded, and
-never a local path: those are the user's files. The model stays registered and
+For a custom model this removes the weight and projection it downloaded,
+including every shard of a sharded one, and never a local path: those are the
+user's files. The model stays registered and
 lists as not cached, ready to download again. Like a catalog delete, it does not
 unload a model that is running.
 
+## Freeing memory when idle
+
+```js
+await harness.setModelIdleTimeout(30 * 60_000)
+```
+
+Once no call has been in flight for that long, every idle resident model is
+unloaded, and the next call loads what it needs again. The clock covers the
+whole device: any call resets it, whichever model it uses. `0`, the default, is
+Never.
+
 ## Office extraction
 
 Hosts can supply the bundled Python extractor to their document pipeline: