dsh-plugin-local-infer 0.1.0 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +7 -0
- package/README.md +8 -1
- package/cordis.patch.yml +6 -6
- package/docs/screenshot-20261014.png +0 -0
- package/docs/zh/README.md +100 -0
- package/dsh/client.js +644 -143
- package/dsh/host.js +1 -1
- package/package.json +14 -11
- package/packages/core/lib/adapter.d.ts +75 -4
- package/packages/core/lib/debug.d.ts +1 -1
- package/packages/core/lib/debug.js +1 -1
- package/packages/core/lib/index.d.ts +3 -2
- package/packages/core/lib/index.js +2 -1
- package/packages/core/lib/launch-command.d.ts +13 -0
- package/packages/core/lib/launch-command.js +64 -0
- package/packages/core/lib/models.d.ts +2 -1
- package/packages/core/lib/models.js +10 -1
- package/packages/core/lib/status.d.ts +9 -1
- package/packages/core/lib/status.js +6 -1
- package/packages/core/package.json +1 -1
- package/packages/providers/lib/acquire.d.ts +125 -0
- package/packages/providers/lib/acquire.js +325 -0
- package/packages/providers/lib/adapters.d.ts +43 -6
- package/packages/providers/lib/adapters.js +386 -58
- package/packages/providers/lib/debug.d.ts +1 -1
- package/packages/providers/lib/debug.js +1 -1
- package/packages/providers/lib/deps.d.ts +6 -4
- package/packages/providers/lib/deps.js +17 -3
- package/packages/providers/lib/engines.d.ts +6 -5
- package/packages/providers/lib/engines.js +12 -4
- package/packages/providers/lib/guidelines.d.ts +2 -2
- package/packages/providers/lib/guidelines.js +17 -1
- package/packages/providers/lib/hyperqwen.d.ts +2 -2
- package/packages/providers/lib/hyperqwen.js +1 -1
- package/packages/providers/lib/index.d.ts +10 -4
- package/packages/providers/lib/index.js +9 -5
- package/packages/providers/lib/infernix.d.ts +170 -0
- package/packages/providers/lib/infernix.js +671 -0
- package/packages/providers/lib/llamacpp.d.ts +2 -2
- package/packages/providers/lib/llamacpp.js +1 -1
- package/packages/providers/lib/model-defaults.d.ts +7 -5
- package/packages/providers/lib/model-defaults.js +28 -4
- package/packages/providers/lib/ninfer.d.ts +9 -2
- package/packages/providers/lib/ninfer.js +10 -3
- package/packages/providers/lib/platform.d.ts +31 -0
- package/packages/providers/lib/platform.js +58 -0
- package/packages/providers/lib/presets.d.ts +2 -2
- package/packages/providers/lib/presets.js +1 -1
- package/packages/providers/lib/strata.d.ts +2 -2
- package/packages/providers/lib/strata.js +1 -1
- package/packages/providers/lib/vram.d.ts +2 -2
- package/packages/providers/lib/vram.js +1 -1
- package/packages/providers/package.json +3 -3
- package/packages/service/lib/index.d.ts +63 -20
- package/packages/service/lib/index.js +279 -43
- package/packages/service/lib/types.d.ts +44 -2
- package/packages/service/package.json +2 -2
- package/packages/tool/lib/index.d.ts +1 -1
- package/packages/tool/lib/index.js +1 -1
- package/packages/tool/package.json +3 -3
- package/packages/ui/lib/deps.d.ts +2 -2
- package/packages/ui/lib/deps.js +1 -1
- package/packages/ui/lib/form.d.ts +2 -2
- package/packages/ui/lib/form.js +1 -1
- package/packages/ui/lib/index.d.ts +1 -1
- package/packages/ui/lib/index.js +1 -1
- package/packages/ui/lib/meter.d.ts +2 -2
- package/packages/ui/lib/meter.js +1 -1
- package/packages/ui/lib/route.d.ts +84 -13
- package/packages/ui/lib/route.js +496 -56
- package/packages/ui/lib/warnings.d.ts +2 -2
- package/packages/ui/lib/warnings.js +1 -1
- package/packages/ui/package.json +5 -5
|
@@ -9,9 +9,9 @@
|
|
|
9
9
|
* `LaunchPlan`. Adding a flag here is a data edit, never a core rewire. The
|
|
10
10
|
* engine is not present on the reference host, so the surface is the
|
|
11
11
|
* specified set the plan names, verified when the binary lands.
|
|
12
|
-
* @module
|
|
12
|
+
* @module dsh-plugin-local-infer/providers/llamacpp
|
|
13
13
|
*/
|
|
14
|
-
import type { Config, EnumOption, ParameterSpec } from '
|
|
14
|
+
import type { Config, EnumOption, ParameterSpec } from 'dsh-plugin-local-infer/core';
|
|
15
15
|
/**
|
|
16
16
|
* The KV cache quantization formats llama.cpp serves. `f16` is the full-
|
|
17
17
|
* precision default; the smaller tiers trade a little quality for a longer
|
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
* `LaunchPlan`. Adding a flag here is a data edit, never a core rewire. The
|
|
10
10
|
* engine is not present on the reference host, so the surface is the
|
|
11
11
|
* specified set the plan names, verified when the binary lands.
|
|
12
|
-
* @module
|
|
12
|
+
* @module dsh-plugin-local-infer/providers/llamacpp
|
|
13
13
|
*/
|
|
14
14
|
/**
|
|
15
15
|
* The KV cache quantization formats llama.cpp serves. `f16` is the full-
|
|
@@ -1,16 +1,18 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* The plugin's default model list. One entry per (engine, model) the card's
|
|
3
3
|
* Model dropdown offers: the NInfer nvfp4 27B and 35B-A3B artifacts, the
|
|
4
|
-
* Hyper-Qwen W4A16 27B and AWQ 35B-A3B artifacts,
|
|
5
|
-
*
|
|
6
|
-
*
|
|
4
|
+
* Hyper-Qwen W4A16 27B and AWQ 35B-A3B artifacts, the llama.cpp GGUF 27B and
|
|
5
|
+
* 35B-A3B artifacts, and the Infernix 27B NVIDIA NVFP4 and the Qwen3.8-
|
|
6
|
+
* Flash-Next Dense8 conversion (whose whole-repo download carries the FP8
|
|
7
|
+
* n-gram volume the `--ngram-volume` flag reads). Strata does not focus on
|
|
8
|
+
* the dense Qwen models, so it carries no entry.
|
|
7
9
|
*
|
|
8
10
|
* The list is the single source the card's Model dropdown, the model
|
|
9
11
|
* presence check, and the hf CLI download route read. Adding a model is a
|
|
10
12
|
* data edit here, never a route or card rewire.
|
|
11
|
-
* @module
|
|
13
|
+
* @module dsh-plugin-local-infer/providers/model-defaults
|
|
12
14
|
*/
|
|
13
|
-
import type { DefaultModel } from '
|
|
15
|
+
import type { DefaultModel } from 'dsh-plugin-local-infer/core';
|
|
14
16
|
/**
|
|
15
17
|
* DEFAULT_MODELS is the default model list, in the order the card lists them.
|
|
16
18
|
* The 27B keys reuse the VramModel registry ids; the 35B-A3B entries use the
|
|
@@ -1,14 +1,16 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* The plugin's default model list. One entry per (engine, model) the card's
|
|
3
3
|
* Model dropdown offers: the NInfer nvfp4 27B and 35B-A3B artifacts, the
|
|
4
|
-
* Hyper-Qwen W4A16 27B and AWQ 35B-A3B artifacts,
|
|
5
|
-
*
|
|
6
|
-
*
|
|
4
|
+
* Hyper-Qwen W4A16 27B and AWQ 35B-A3B artifacts, the llama.cpp GGUF 27B and
|
|
5
|
+
* 35B-A3B artifacts, and the Infernix 27B NVIDIA NVFP4 and the Qwen3.8-
|
|
6
|
+
* Flash-Next Dense8 conversion (whose whole-repo download carries the FP8
|
|
7
|
+
* n-gram volume the `--ngram-volume` flag reads). Strata does not focus on
|
|
8
|
+
* the dense Qwen models, so it carries no entry.
|
|
7
9
|
*
|
|
8
10
|
* The list is the single source the card's Model dropdown, the model
|
|
9
11
|
* presence check, and the hf CLI download route read. Adding a model is a
|
|
10
12
|
* data edit here, never a route or card rewire.
|
|
11
|
-
* @module
|
|
13
|
+
* @module dsh-plugin-local-infer/providers/model-defaults
|
|
12
14
|
*/
|
|
13
15
|
/**
|
|
14
16
|
* DEFAULT_MODELS is the default model list, in the order the card lists them.
|
|
@@ -64,6 +66,28 @@ export const DEFAULT_MODELS = [
|
|
|
64
66
|
url: 'https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF',
|
|
65
67
|
file: 'Qwen3.6-35B-A3B-Q4_K_M.gguf',
|
|
66
68
|
},
|
|
69
|
+
{
|
|
70
|
+
engineId: 'infernix',
|
|
71
|
+
modelId: 'qwen3-27b',
|
|
72
|
+
label: 'Infernix Qwen3.8-27B',
|
|
73
|
+
url: 'https://huggingface.co/Wallawalla47/Qwen3.8-27B-NVIDIA-NVFP4-NInferV3',
|
|
74
|
+
file: 'qwen3_8_27b_nvfp4-nvidia.ninfer',
|
|
75
|
+
},
|
|
76
|
+
{
|
|
77
|
+
engineId: 'infernix',
|
|
78
|
+
modelId: 'qwen38-flash-next',
|
|
79
|
+
label: 'Infernix Qwen3.8-Flash-Next (Dense8)',
|
|
80
|
+
// No single file: the download pulls the whole repo, which carries the
|
|
81
|
+
// three model shards, the FP8 n-gram volume, and the conversion metadata.
|
|
82
|
+
url: 'https://huggingface.co/Wallawalla47/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Dense8-Infernix',
|
|
83
|
+
},
|
|
84
|
+
{
|
|
85
|
+
engineId: 'infernix',
|
|
86
|
+
modelId: 'qwen36-35b',
|
|
87
|
+
label: 'Infernix Qwen3.6-35B',
|
|
88
|
+
url: 'https://huggingface.co/neroued/Qwen3.6-35B-A3B-NInfer',
|
|
89
|
+
file: 'qwen3_6_35b_a3b.ninfer',
|
|
90
|
+
},
|
|
67
91
|
];
|
|
68
92
|
/**
|
|
69
93
|
* defaultModelsForEngine reads the default list for one engine id, in list
|
|
@@ -7,9 +7,9 @@
|
|
|
7
7
|
*
|
|
8
8
|
* The data is the surface; the `NInferAdapter` (Phase 5) composes it into a
|
|
9
9
|
* `LaunchPlan`. Adding a flag here is a data edit, never a core rewire.
|
|
10
|
-
* @module
|
|
10
|
+
* @module dsh-plugin-local-infer/providers/ninfer
|
|
11
11
|
*/
|
|
12
|
-
import type { ComboOption, EnumOption, ParameterSpec } from '
|
|
12
|
+
import type { ComboOption, EnumOption, ParameterSpec } from 'dsh-plugin-local-infer/core';
|
|
13
13
|
/**
|
|
14
14
|
* The KV cache quantization formats NInfer serves, in the order the form
|
|
15
15
|
* dropdown lists them. `note` carries the tradeoff text the dropdown shows
|
|
@@ -39,6 +39,13 @@ export declare const CHAT_TEMPLATE_OPTIONS: readonly ComboOption[];
|
|
|
39
39
|
* click, not just the constraint.
|
|
40
40
|
*/
|
|
41
41
|
export declare const SPEC_GATE_MESSAGE = "Pick a speculation backend first (mtp, dflash, or dflash2)";
|
|
42
|
+
/**
|
|
43
|
+
* The gate message the form shows while the LM-head draft field is greyed out.
|
|
44
|
+
* The LM-head drafter is an MTP-backend feature, so the field unlocks only
|
|
45
|
+
* while the MTP speculation backend is selected, for every provider that
|
|
46
|
+
* spells the flag.
|
|
47
|
+
*/
|
|
48
|
+
export declare const LM_HEAD_DRAFT_GATE_MESSAGE = "LM-head draft is only available with the MTP speculation backend";
|
|
42
49
|
/**
|
|
43
50
|
* The NInfer `ParameterSpec` surface, in form order. The `group` field is the
|
|
44
51
|
* form section; specs without a group render under `General`. `requires`
|
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
*
|
|
8
8
|
* The data is the surface; the `NInferAdapter` (Phase 5) composes it into a
|
|
9
9
|
* `LaunchPlan`. Adding a flag here is a data edit, never a core rewire.
|
|
10
|
-
* @module
|
|
10
|
+
* @module dsh-plugin-local-infer/providers/ninfer
|
|
11
11
|
*/
|
|
12
12
|
/**
|
|
13
13
|
* The KV cache quantization formats NInfer serves, in the order the form
|
|
@@ -66,6 +66,13 @@ export const CHAT_TEMPLATE_OPTIONS = [
|
|
|
66
66
|
* click, not just the constraint.
|
|
67
67
|
*/
|
|
68
68
|
export const SPEC_GATE_MESSAGE = 'Pick a speculation backend first (mtp, dflash, or dflash2)';
|
|
69
|
+
/**
|
|
70
|
+
* The gate message the form shows while the LM-head draft field is greyed out.
|
|
71
|
+
* The LM-head drafter is an MTP-backend feature, so the field unlocks only
|
|
72
|
+
* while the MTP speculation backend is selected, for every provider that
|
|
73
|
+
* spells the flag.
|
|
74
|
+
*/
|
|
75
|
+
export const LM_HEAD_DRAFT_GATE_MESSAGE = 'LM-head draft is only available with the MTP speculation backend';
|
|
69
76
|
/**
|
|
70
77
|
* The NInfer `ParameterSpec` surface, in form order. The `group` field is the
|
|
71
78
|
* form section; specs without a group render under `General`. `requires`
|
|
@@ -209,11 +216,11 @@ export const NINFER_PARAMETERS = [
|
|
|
209
216
|
{
|
|
210
217
|
key: 'lmHeadDraft',
|
|
211
218
|
label: 'LM-head draft',
|
|
212
|
-
description: 'Drafts from the small proposal head instead of the full output head, cutting the per-draft projection cost.',
|
|
219
|
+
description: 'Drafts from the small proposal head instead of the full output head, cutting the per-draft projection cost. An MTP-backend feature, so the field unlocks only while MTP is selected.',
|
|
213
220
|
form: { kind: 'boolean' },
|
|
214
221
|
defaultValue: false,
|
|
215
222
|
group: 'Speculation',
|
|
216
|
-
requires: (config) => config.spec
|
|
223
|
+
requires: (config) => config.spec === 'mtp' ? null : LM_HEAD_DRAFT_GATE_MESSAGE,
|
|
217
224
|
toArgv: (value) => (value === true ? ['--lm-head-draft'] : []),
|
|
218
225
|
},
|
|
219
226
|
{
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The per-engine platform defaults the card's Engine platform section seeds:
|
|
3
|
+
* the local build command the deploy step runs in the checkout and the docker
|
|
4
|
+
* image tag the docker deploy pulls. Both are data, so a new engine's
|
|
5
|
+
* defaults are a data edit here, never a card rewire.
|
|
6
|
+
* @module dsh-plugin-local-infer/providers/platform
|
|
7
|
+
*/
|
|
8
|
+
/**
|
|
9
|
+
* EnginePlatformDefaults is the per-engine platform defaults: the local build
|
|
10
|
+
* command (the engine's first acquisition build step) and the docker image tag
|
|
11
|
+
* the docker deploy pulls. An engine that supports neither a mode carries no
|
|
12
|
+
* entry for it.
|
|
13
|
+
*/
|
|
14
|
+
export interface EnginePlatformDefaults {
|
|
15
|
+
defaultBuildCommand?: string;
|
|
16
|
+
defaultDockerTag?: string;
|
|
17
|
+
}
|
|
18
|
+
/**
|
|
19
|
+
* The per-engine platform defaults, keyed by engine id. The build command is
|
|
20
|
+
* the engine's first acquisition build step; the docker tag is the upstream
|
|
21
|
+
* image the adapter pulls. Hyper-Qwen is compose-driven, so its docker tag is
|
|
22
|
+
* the vLLM OpenAI server image; Infernix ships no docker image, so it carries
|
|
23
|
+
* a build command only.
|
|
24
|
+
*/
|
|
25
|
+
export declare const ENGINE_PLATFORM_DEFAULTS: Record<string, EnginePlatformDefaults>;
|
|
26
|
+
/**
|
|
27
|
+
* enginePlatformDefaults reads the registered platform defaults for an engine
|
|
28
|
+
* id, or an empty object when none is registered.
|
|
29
|
+
*/
|
|
30
|
+
export declare function enginePlatformDefaults(engineId: string): EnginePlatformDefaults;
|
|
31
|
+
//# sourceMappingURL=platform.d.ts.map
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The per-engine platform defaults the card's Engine platform section seeds:
|
|
3
|
+
* the local build command the deploy step runs in the checkout and the docker
|
|
4
|
+
* image tag the docker deploy pulls. Both are data, so a new engine's
|
|
5
|
+
* defaults are a data edit here, never a card rewire.
|
|
6
|
+
* @module dsh-plugin-local-infer/providers/platform
|
|
7
|
+
*/
|
|
8
|
+
import { LLAMACPP_DOCKER_IMAGE, NINFER_DOCKER_IMAGE, STRATA_DOCKER_IMAGE, } from "./adapters.js";
|
|
9
|
+
import { engineBuildSteps } from "./acquire.js";
|
|
10
|
+
// The local build command is the engine's first acquisition build step (the
|
|
11
|
+
// cmake build the deploy step runs in the checkout), read through
|
|
12
|
+
// engineBuildSteps so the card's placeholder and the deploy step share one
|
|
13
|
+
// list. A step list that is empty (the compose engines) contributes no
|
|
14
|
+
// build command.
|
|
15
|
+
const NINFER_BUILD_COMMAND = engineBuildSteps('ninfer')[0];
|
|
16
|
+
const LLAMACPP_BUILD_COMMAND = engineBuildSteps('llamacpp')[0];
|
|
17
|
+
const INFERNIX_BUILD_COMMAND = engineBuildSteps('infernix')[0];
|
|
18
|
+
/**
|
|
19
|
+
* The per-engine platform defaults, keyed by engine id. The build command is
|
|
20
|
+
* the engine's first acquisition build step; the docker tag is the upstream
|
|
21
|
+
* image the adapter pulls. Hyper-Qwen is compose-driven, so its docker tag is
|
|
22
|
+
* the vLLM OpenAI server image; Infernix ships no docker image, so it carries
|
|
23
|
+
* a build command only.
|
|
24
|
+
*/
|
|
25
|
+
export const ENGINE_PLATFORM_DEFAULTS = {
|
|
26
|
+
ninfer: {
|
|
27
|
+
...(NINFER_BUILD_COMMAND !== undefined ? { defaultBuildCommand: NINFER_BUILD_COMMAND } : {}),
|
|
28
|
+
defaultDockerTag: NINFER_DOCKER_IMAGE,
|
|
29
|
+
},
|
|
30
|
+
hyperqwen: {
|
|
31
|
+
// The compose setup selects a profile, not an image tag, so the card's
|
|
32
|
+
// docker-tag field seeds the compose arguments the up step runs: the
|
|
33
|
+
// launcher's default `single` profile.
|
|
34
|
+
defaultDockerTag: '--profile single',
|
|
35
|
+
},
|
|
36
|
+
llamacpp: {
|
|
37
|
+
...(LLAMACPP_BUILD_COMMAND !== undefined
|
|
38
|
+
? { defaultBuildCommand: LLAMACPP_BUILD_COMMAND }
|
|
39
|
+
: {}),
|
|
40
|
+
defaultDockerTag: LLAMACPP_DOCKER_IMAGE,
|
|
41
|
+
},
|
|
42
|
+
infernix: {
|
|
43
|
+
...(INFERNIX_BUILD_COMMAND !== undefined
|
|
44
|
+
? { defaultBuildCommand: INFERNIX_BUILD_COMMAND }
|
|
45
|
+
: {}),
|
|
46
|
+
},
|
|
47
|
+
strata: {
|
|
48
|
+
defaultDockerTag: STRATA_DOCKER_IMAGE,
|
|
49
|
+
},
|
|
50
|
+
};
|
|
51
|
+
/**
|
|
52
|
+
* enginePlatformDefaults reads the registered platform defaults for an engine
|
|
53
|
+
* id, or an empty object when none is registered.
|
|
54
|
+
*/
|
|
55
|
+
export function enginePlatformDefaults(engineId) {
|
|
56
|
+
return ENGINE_PLATFORM_DEFAULTS[engineId] ?? {};
|
|
57
|
+
}
|
|
58
|
+
//# sourceMappingURL=platform.js.map
|
|
@@ -11,9 +11,9 @@
|
|
|
11
11
|
* bridge pulls and the pure extractor that turns the fetched page into a
|
|
12
12
|
* preset. The service runs the fetch and tags the result with the URL and the
|
|
13
13
|
* fetch date.
|
|
14
|
-
* @module
|
|
14
|
+
* @module dsh-plugin-local-infer/providers/presets
|
|
15
15
|
*/
|
|
16
|
-
import type { Config, Preset, PresetSource } from '
|
|
16
|
+
import type { Config, Preset, PresetSource } from 'dsh-plugin-local-infer/core';
|
|
17
17
|
/**
|
|
18
18
|
* BUILTIN_PRESETS is the combined preset list the providers register with the
|
|
19
19
|
* seam. Adding a preset is a data edit here, never a resolver change.
|
|
@@ -11,7 +11,7 @@
|
|
|
11
11
|
* bridge pulls and the pure extractor that turns the fetched page into a
|
|
12
12
|
* preset. The service runs the fetch and tags the result with the URL and the
|
|
13
13
|
* fetch date.
|
|
14
|
-
* @module
|
|
14
|
+
* @module dsh-plugin-local-infer/providers/presets
|
|
15
15
|
*/
|
|
16
16
|
/**
|
|
17
17
|
* The hardware class every built-in preset is sourced for: a 24 GiB Ampere
|
|
@@ -4,9 +4,9 @@
|
|
|
4
4
|
* available. The surface is empty until then, so the form renders no fields
|
|
5
5
|
* and the adapter renders a bare docker plan; adding a spec here is a data
|
|
6
6
|
* edit, never an adapter rewire.
|
|
7
|
-
* @module
|
|
7
|
+
* @module dsh-plugin-local-infer/providers/strata
|
|
8
8
|
*/
|
|
9
|
-
import type { ParameterSpec } from '
|
|
9
|
+
import type { ParameterSpec } from 'dsh-plugin-local-infer/core';
|
|
10
10
|
/**
|
|
11
11
|
* The Strata `ParameterSpec` surface, empty until the engine's flag set lands.
|
|
12
12
|
* The `StrataAdapter` composes this (currently nothing) into a docker plan.
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
* available. The surface is empty until then, so the form renders no fields
|
|
5
5
|
* and the adapter renders a bare docker plan; adding a spec here is a data
|
|
6
6
|
* edit, never an adapter rewire.
|
|
7
|
-
* @module
|
|
7
|
+
* @module dsh-plugin-local-infer/providers/strata
|
|
8
8
|
*/
|
|
9
9
|
/**
|
|
10
10
|
* The Strata `ParameterSpec` surface, empty until the engine's flag set lands.
|
|
@@ -6,9 +6,9 @@
|
|
|
6
6
|
* parameter counts. The provenance note on each model names the exact weight
|
|
7
7
|
* artifact the figures apply to, because the runtime also recognizes other
|
|
8
8
|
* weight profiles with different resident sizes.
|
|
9
|
-
* @module
|
|
9
|
+
* @module dsh-plugin-local-infer/providers/vram
|
|
10
10
|
*/
|
|
11
|
-
import type { VramModel } from '
|
|
11
|
+
import type { VramModel } from 'dsh-plugin-local-infer/core';
|
|
12
12
|
/**
|
|
13
13
|
* The NInfer measured VramModels, one per model artifact. The top-level
|
|
14
14
|
* `seqFixedBytes`/`seqPerToken` are the no-speculation baseline; the
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
* parameter counts. The provenance note on each model names the exact weight
|
|
7
7
|
* artifact the figures apply to, because the runtime also recognizes other
|
|
8
8
|
* weight profiles with different resident sizes.
|
|
9
|
-
* @module
|
|
9
|
+
* @module dsh-plugin-local-infer/providers/vram
|
|
10
10
|
*/
|
|
11
11
|
/**
|
|
12
12
|
* DFlash2's CUDA-graph allowance is a step in context, measured at 8,192 /
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
|
-
"name": "
|
|
2
|
+
"name": "dsh-plugin-local-infer-providers",
|
|
3
3
|
"version": "0.1.0",
|
|
4
4
|
"description": "Concrete providers for the dsh local-inference plugin: the ninfer, hyperqwen, llamacpp, and strata engine adapters, the NInfer VramModel, the built-in presets, the built-in guidelines, and the per-OS SystemDep install command data. Each registers itself with ctx.localInfer.",
|
|
5
5
|
"keywords": [
|
|
@@ -31,8 +31,8 @@
|
|
|
31
31
|
"access": "public"
|
|
32
32
|
},
|
|
33
33
|
"dependencies": {
|
|
34
|
-
"
|
|
35
|
-
"
|
|
34
|
+
"dsh-plugin-local-infer-core": "workspace:^",
|
|
35
|
+
"dsh-plugin-local-infer-service": "workspace:^"
|
|
36
36
|
},
|
|
37
37
|
"peerDependencies": {
|
|
38
38
|
"@deepseek-ai/cordis": ">=4.0.1"
|
|
@@ -15,12 +15,12 @@
|
|
|
15
15
|
* Phase 6 over the dsh settings seam, and `resolveDefaults` and
|
|
16
16
|
* `fetchPreset` by Phase 7 over the core defaults resolver and the
|
|
17
17
|
* mod-search bridge.
|
|
18
|
-
* @module
|
|
18
|
+
* @module dsh-plugin-local-infer/service
|
|
19
19
|
*/
|
|
20
20
|
import { Context, Service } from '@deepseek-ai/cordis';
|
|
21
|
-
import type { Config, DefaultModel, ModelDirectoryEntry, EngineAdapter, EngineDepPlan, EnginePlatformConfig, GpuUsageProbe, Guideline, GuidelineResult, HardwareProfile, HardwareProbes, InstallResult, LaunchPlan, ModelRow, Preset, PresetSource, ProcessLine, ProviderLaunchRecord, ProviderRef, ProviderStatus, ReLinkOutcome, RegisterResult, ReserveResolution, ResolvedDefaults, SystemDep, VramModel } from '
|
|
22
|
-
import type { DetectOsReserveOptions, DownloadModelOptions, DownloadResult, EstimateResult, FetchPresetOptions, FetchPresetResult, HfCliOptions, LaunchOptions, LaunchResult, LocalInferService, RegisterModelOptions } from './types.ts';
|
|
23
|
-
export type { DetectOsReserveOptions, DownloadModelOptions, DownloadResult, EstimateResult, FetchPresetOptions, FetchPresetResult, HfCliOptions, LaunchOptions, LaunchResult, LocalInferService, ProviderStatusOptions, RegisterModelOptions, SettingsDescribeResult, SettingsForms, SettingsNamespaceDescriptor, } from './types.ts';
|
|
21
|
+
import type { Config, DefaultModel, ModelDirectoryEntry, EngineAdapter, EngineDepPlan, EnginePlatformConfig, GpuUsageProbe, Guideline, GuidelineResult, HardwareProfile, HardwareProbes, InstallResult, LaunchPlan, ModelRow, Preset, PresetSource, ProcessLine, ProviderLaunchRecord, ProviderRef, ProviderStatus, ReLinkOutcome, RegisterResult, ReserveResolution, ResolvedDefaults, SystemDep, VramModel } from 'dsh-plugin-local-infer/core';
|
|
22
|
+
import type { DetectOsReserveOptions, DownloadModelOptions, DownloadResult, EngineDeathEvent, EstimateResult, FetchPresetOptions, FetchPresetResult, HfCliOptions, LaunchOptions, LaunchResult, LocalInferService, ProviderStatusOptions, RegisterModelOptions } from './types.ts';
|
|
23
|
+
export type { DetectOsReserveOptions, DownloadModelOptions, DownloadResult, EngineDeathEvent, EstimateResult, FetchPresetOptions, FetchPresetResult, HfCliOptions, LaunchOptions, LaunchResult, LocalInferService, ProviderStatusOptions, RegisterModelOptions, SettingsDescribeResult, SettingsForms, SettingsNamespaceDescriptor, } from './types.ts';
|
|
24
24
|
declare module '@deepseek-ai/cordis' {
|
|
25
25
|
interface Context {
|
|
26
26
|
localInfer: LocalInferService;
|
|
@@ -50,6 +50,10 @@ export declare class LocalInfer extends Service implements LocalInferService {
|
|
|
50
50
|
private readonly platformConfigRegistry;
|
|
51
51
|
private readonly launchedEngines;
|
|
52
52
|
private readonly launchedModels;
|
|
53
|
+
private readonly launchedPorts;
|
|
54
|
+
private readonly engineWatchers;
|
|
55
|
+
private readonly handledPids;
|
|
56
|
+
private deathObserver;
|
|
53
57
|
constructor(ctx: Context);
|
|
54
58
|
registerEngine(adapter: EngineAdapter): () => void;
|
|
55
59
|
registerPreset(preset: Preset): () => void;
|
|
@@ -120,22 +124,20 @@ export declare class LocalInfer extends Service implements LocalInferService {
|
|
|
120
124
|
detectOsReserve(config?: Config, options?: DetectOsReserveOptions): Promise<ReserveResolution>;
|
|
121
125
|
/**
|
|
122
126
|
* providerStatuses resolves the running-state the card paints each saved
|
|
123
|
-
* provider's status dot from. It
|
|
124
|
-
*
|
|
125
|
-
*
|
|
126
|
-
*
|
|
127
|
-
*
|
|
128
|
-
*
|
|
129
|
-
* busy
|
|
130
|
-
*
|
|
131
|
-
*
|
|
132
|
-
*
|
|
133
|
-
*
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
gpuUsage?: GpuUsageProbe | null;
|
|
138
|
-
}): Promise<Record<string, ProviderStatus>>;
|
|
127
|
+
* provider's status dot from. It prices each provider's own launch signature
|
|
128
|
+
* through the registered adapter (the build mode's binary, the port the
|
|
129
|
+
* provider's config binds, the env keys), so an engine the plugin launched
|
|
130
|
+
* on a non-default port matches its config instead of reading forever as a
|
|
131
|
+
* process no signature covers, and ties the matched port to the model the
|
|
132
|
+
* plugin launched it with (the launched-models map). The GPU-usage reading
|
|
133
|
+
* feeds the busy-card rule. The answer is the core resolver's: a busy card
|
|
134
|
+
* or no engine running reads as unsure for every provider, an engine the
|
|
135
|
+
* detector saw but could not tie to a model reads as unsure for its configs
|
|
136
|
+
* and stopped for the rest, and a confirmed engine-plus-model reads as
|
|
137
|
+
* running. The options inject the probe results, so a test prices a busy
|
|
138
|
+
* card or a foreign engine without a live nvidia-smi or ps.
|
|
139
|
+
*/
|
|
140
|
+
providerStatuses(providers: readonly ProviderRef[], options?: ProviderStatusOptions): Promise<Record<string, ProviderStatus>>;
|
|
139
141
|
/**
|
|
140
142
|
* unsureBannerKillCommands returns the bash kill commands for the engine
|
|
141
143
|
* processes the detector saw holding VRAM, for the "unsure" banner the card
|
|
@@ -210,6 +212,47 @@ export declare class LocalInfer extends Service implements LocalInferService {
|
|
|
210
212
|
processAlive?: (pid: number) => boolean;
|
|
211
213
|
commandLine?: (pid: number) => string | undefined;
|
|
212
214
|
}): Promise<Record<string, ReLinkOutcome>>;
|
|
215
|
+
/**
|
|
216
|
+
* setEngineDeathObserver registers the route's hook for engine process
|
|
217
|
+
* deaths: a crash or pkill seen through a launched handle's exit event, or a
|
|
218
|
+
* poll-detected death of a process adopted from a stored launch record. The
|
|
219
|
+
* observer persists the provider entry's cleared state so the card reads the
|
|
220
|
+
* provider stopped instead of carrying a stale record. Passing undefined
|
|
221
|
+
* clears it, which the route's effect does on dispose.
|
|
222
|
+
*/
|
|
223
|
+
setEngineDeathObserver(observer: ((event: EngineDeathEvent) => void) | undefined): void;
|
|
224
|
+
/**
|
|
225
|
+
* watchLaunchedHandle ties a launched child's one-shot exit to the live
|
|
226
|
+
* state: the host's close/error event (a crash, a pkill, or the stop kill)
|
|
227
|
+
* reports through observeEngineDeath. A handle whose host cannot watch (no
|
|
228
|
+
* onExit, a stub) is left to the re-link rule to price after a restart.
|
|
229
|
+
*/
|
|
230
|
+
private watchLaunchedHandle;
|
|
231
|
+
/**
|
|
232
|
+
* watchAdoptedEngine re-attaches the exit watch for a process this DSH did
|
|
233
|
+
* not spawn: a surviving engine adopted from a stored launch record has no
|
|
234
|
+
* ChildProcess to listen on after a restart, so an unref'd poll on the
|
|
235
|
+
* liveness probe stands in for the exit event. When the pid dies the watcher
|
|
236
|
+
* stops itself and reports through observeEngineDeath, so the provider's
|
|
237
|
+
* state settles without waiting for the next summary re-link to price the
|
|
238
|
+
* record dead.
|
|
239
|
+
*/
|
|
240
|
+
private watchAdoptedEngine;
|
|
241
|
+
/**
|
|
242
|
+
* stopEngineWatcher cancels one port's adopted-process poll, if it has one.
|
|
243
|
+
*/
|
|
244
|
+
private stopEngineWatcher;
|
|
245
|
+
/**
|
|
246
|
+
* observeEngineDeath settles one engine process's death against the live
|
|
247
|
+
* state: the port-to-model tie drops when the dead engine still owns the
|
|
248
|
+
* port, the port leaves the engine's live ledger, the plugin-launched flag
|
|
249
|
+
* clears with the engine's last live port, the adopted-process watcher for
|
|
250
|
+
* the port stops, and the registered death observer (the route's persistence
|
|
251
|
+
* hook) receives the event. The launch record itself is the route's: the
|
|
252
|
+
* observer clears the provider entry's record so the card reads the provider
|
|
253
|
+
* stopped, not stale.
|
|
254
|
+
*/
|
|
255
|
+
private observeEngineDeath;
|
|
213
256
|
/**
|
|
214
257
|
* stop signals the engine a launch started: it resolves the adapter by the
|
|
215
258
|
* plan's engineId and runs the adapter's stop step (kill the tracked process,
|