@hy-sde-org/dsh-llm-slots 0.1.2-rc.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +82 -0
- package/dist/index.d.ts +168 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +233 -0
- package/dist/index.js.map +1 -0
- package/dist/invariant.d.ts +13 -0
- package/dist/invariant.d.ts.map +1 -0
- package/dist/invariant.js +65 -0
- package/dist/invariant.js.map +1 -0
- package/package.json +69 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 hy-sde
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# @hy-sde-org/dsh-llm-slots
|
|
2
|
+
|
|
3
|
+
Host-wide model-slot admission control (`ctx.modelSlots`) for the DeepSeek
|
|
4
|
+
Harness. A deployment that runs a handful of concurrent model providers behind
|
|
5
|
+
one local inference endpoint (typically 2–3 slots, ~1M context each) needs an
|
|
6
|
+
explicit budget over every model call — otherwise the main agent's turns,
|
|
7
|
+
running subagents, and workflow fan-out stack dozens of simultaneous bursts
|
|
8
|
+
against the endpoint and every call slows to queue latency.
|
|
9
|
+
|
|
10
|
+
Published standalone as `@hy-sde-org/dsh-llm-slots`; it mirrors
|
|
11
|
+
`@deepseek-ai/dsh-llm-slots` in the [hy-sde deepseek-harness fork](https://github.com/hy-sde/deepseek-harness)
|
|
12
|
+
(the fork-only package is not on npm and not in upstream stock).
|
|
13
|
+
|
|
14
|
+
## How it works
|
|
15
|
+
|
|
16
|
+
- **One chokepoint.** Admission decides FIFO at the `llm/stream` waterfall —
|
|
17
|
+
the single boundary every model-backed call crosses (main agent loops,
|
|
18
|
+
in-process subagents, worker-thread children, workflows, title/compaction
|
|
19
|
+
side-requests), regardless of which session or context initiated it.
|
|
20
|
+
- **Host-global budget.** The gate lives in module scope, so every derived
|
|
21
|
+
context and plugin instance shares one pool. Capacity is configured on the
|
|
22
|
+
`llm-slots` row (default 3) and adjustable at runtime:
|
|
23
|
+
`ctx.modelSlots.setCapacity(n)`.
|
|
24
|
+
- **Cancellable waits.** A call waiting for a slot observes its AbortSignal;
|
|
25
|
+
cancellation while queued surfaces as an AbortError upstream and never
|
|
26
|
+
receives a freed slot.
|
|
27
|
+
- **One slot per logical call.** A call holds its slot for its full lifetime,
|
|
28
|
+
including adapter retries, which also prevents a failing endpoint from
|
|
29
|
+
fanning out an unbounded retry storm.
|
|
30
|
+
|
|
31
|
+
## Composition
|
|
32
|
+
|
|
33
|
+
Mount once per host in a shared-row composition:
|
|
34
|
+
|
|
35
|
+
```yaml
|
|
36
|
+
- id: llm-slots
|
|
37
|
+
name: '@hy-sde-org/dsh-llm-slots'
|
|
38
|
+
config:
|
|
39
|
+
enabled: true
|
|
40
|
+
capacity: 3
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Additional mounts share the same global budget and are safe but redundant.
|
|
44
|
+
|
|
45
|
+
## Service surface
|
|
46
|
+
|
|
47
|
+
`ctx.modelSlots` exposes:
|
|
48
|
+
|
|
49
|
+
- `stats()` — `{ enabled, capacity, running, waiting, acquiredTotal }`.
|
|
50
|
+
- `setEnabled(boolean)` — toggle admission without touching capacity.
|
|
51
|
+
- `setCapacity(number)` — change the budget; a shrink applies as calls drain.
|
|
52
|
+
|
|
53
|
+
The optional `./invariant` entry registers package-owned admission-accounting
|
|
54
|
+
checks with a Cordis host's `ctx.invariants` service and needs
|
|
55
|
+
`@deepseek-ai/cordis` + `@deepseek-ai/dsh-invariants` peers only when you use
|
|
56
|
+
that entry.
|
|
57
|
+
|
|
58
|
+
## Development
|
|
59
|
+
|
|
60
|
+
```sh
|
|
61
|
+
pnpm install
|
|
62
|
+
pnpm -r check # strict typecheck (src + tests)
|
|
63
|
+
pnpm -r test # gate + admission tests
|
|
64
|
+
pnpm -r build # tsc -> dist
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
No model-facing tools ship here; the package deliberately reads no LLM service
|
|
68
|
+
state (it only listens on the shared event bus), which keeps it trivially
|
|
69
|
+
testable and safe to mount in any context.
|
|
70
|
+
|
|
71
|
+
## Known Limitations and Deferred Work
|
|
72
|
+
|
|
73
|
+
- The gate is process-global: capacity is not partitioned per provider or per
|
|
74
|
+
session, and the budget never drains below what in-flight calls hold (a
|
|
75
|
+
shrink takes effect as calls finish).
|
|
76
|
+
- `acquire()` grants one slot per logical call; there is no priority or
|
|
77
|
+
fair-share scheduling beyond arrival order (FIFO).
|
|
78
|
+
|
|
79
|
+
## License
|
|
80
|
+
|
|
81
|
+
MIT. Derived from the DeepSeek Harness codebase; see
|
|
82
|
+
`THIRD-PARTY-NOTICES.md` for provenance.
|
package/dist/index.d.ts
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Host-wide model-slot admission control (`ctx.modelSlots`).
|
|
3
|
+
*
|
|
4
|
+
* The deployment runs a handful of concurrent model providers behind one local
|
|
5
|
+
* inference endpoint (typically 2–3 slots, ~1M context each). Without admission
|
|
6
|
+
* control the main agent's own turns, running subagents, and workflow fan-out
|
|
7
|
+
* can stack dozens of simultaneous LLM bursts against that endpoint, so every
|
|
8
|
+
* call slows to endpoint-queue latency and the "N concurrent" budget becomes
|
|
9
|
+
* meaningless. This plugin makes the model budget explicit and host-global:
|
|
10
|
+
*
|
|
11
|
+
* - It decides FIFO at the `llm/stream` waterfall — the single chokepoint
|
|
12
|
+
* every model-backed call crosses (main agent loops, in-process subagents,
|
|
13
|
+
* worker-thread children, workflows, title/compaction side-requests),
|
|
14
|
+
* regardless of which session or context initiated it.
|
|
15
|
+
* - The budget itself lives in a module-global gate, so every derived context
|
|
16
|
+
* and plugin instance shares one host-wide pool. Capacity is configured at
|
|
17
|
+
* mount (`llm-slots` row, default 3) and can be raised/lowered at runtime
|
|
18
|
+
* through `ctx.modelSlots.setCapacity()`.
|
|
19
|
+
* - A call waiting for a slot holds no LLM work; its AbortSignal applies
|
|
20
|
+
* through the wait, so cancellation during admission surfaces as an
|
|
21
|
+
* AbortError upstream instead of leaking a straggler slot.
|
|
22
|
+
* - One logical call holds one slot for its full lifetime (including adapter
|
|
23
|
+
* retries), which also caps retry-storm concurrency: a failing endpoint
|
|
24
|
+
* cannot fan out unbounded retries against the same pool.
|
|
25
|
+
*
|
|
26
|
+
* The plugin deliberately reads no LLM service state: it only listens on the
|
|
27
|
+
* shared event bus, so it mounts in any context (root or child) and is trivially
|
|
28
|
+
* testable without a model adapter.
|
|
29
|
+
*
|
|
30
|
+
* Conceptually inspired by firstmate (https://github.com/kunchenguid/firstmate),
|
|
31
|
+
* MIT, © 2026 Kun Chen — its per-agent harness/model allocation at intake.
|
|
32
|
+
* No firstmate code is included.
|
|
33
|
+
*
|
|
34
|
+
* @module @hy-sde-org/dsh-llm-slots
|
|
35
|
+
*/
|
|
36
|
+
import { Context, Service } from '@deepseek-ai/cordis';
|
|
37
|
+
import z from '@deepseek-ai/schemastery';
|
|
38
|
+
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm';
|
|
39
|
+
/** Cordis plugin name used by Loader diagnostics. */
|
|
40
|
+
export declare const name = "llm-slots";
|
|
41
|
+
/** Plugin configuration on the `llm-slots` row. */
|
|
42
|
+
export interface Config {
|
|
43
|
+
/** Master switch; `false` bypasses admission entirely (default true). */
|
|
44
|
+
enabled?: boolean;
|
|
45
|
+
/** Host-wide concurrent model-slot budget (default 3). */
|
|
46
|
+
capacity?: number;
|
|
47
|
+
}
|
|
48
|
+
export declare const Config: z<Config>;
|
|
49
|
+
/** Schemastery configuration for the slots service (runtime capacity controls). */
|
|
50
|
+
export interface ModelSlotsConfig {
|
|
51
|
+
/** Whether admission is currently enforced. */
|
|
52
|
+
enabled: boolean;
|
|
53
|
+
/** Current host-wide concurrent model-slot budget. */
|
|
54
|
+
capacity: number;
|
|
55
|
+
}
|
|
56
|
+
/** Live admission snapshot exposed for operators and the GUI. */
|
|
57
|
+
export interface ModelSlotsStats {
|
|
58
|
+
/** Whether admission is currently enforced. */
|
|
59
|
+
enabled: boolean;
|
|
60
|
+
/** Current host-wide concurrent model-slot budget. */
|
|
61
|
+
capacity: number;
|
|
62
|
+
/** Slots currently held by in-flight model calls. */
|
|
63
|
+
running: number;
|
|
64
|
+
/** Calls currently queued, waiting FIFO for a slot. */
|
|
65
|
+
waiting: number;
|
|
66
|
+
/** Lifetime acquisitions served by this gate (pool-granted and queue-granted). */
|
|
67
|
+
acquiredTotal: number;
|
|
68
|
+
}
|
|
69
|
+
/**
|
|
70
|
+
* Host-global FIFO model-slot gate. One instance per process (regardless of
|
|
71
|
+
* how many contexts mount {@link apply}), so the budget is genuinely global.
|
|
72
|
+
*/
|
|
73
|
+
export declare class ModelSlotGate {
|
|
74
|
+
/** Whether admission is currently enforced. */
|
|
75
|
+
enabled: boolean;
|
|
76
|
+
/** Current concurrent-slot budget. */
|
|
77
|
+
capacity: number;
|
|
78
|
+
/** Slots currently held by in-flight model calls. */
|
|
79
|
+
running: number;
|
|
80
|
+
/** Calls currently queued, waiting FIFO for a slot. */
|
|
81
|
+
waiting: number;
|
|
82
|
+
/** Lifetime acquisitions served by this gate. */
|
|
83
|
+
acquiredTotal: number;
|
|
84
|
+
private readonly queue;
|
|
85
|
+
/**
|
|
86
|
+
* Acquire one model slot, resolving immediately when one is free or the gate
|
|
87
|
+
* is disabled, and buffering FIFO otherwise. The caller's signal cancels the
|
|
88
|
+
* wait: an aborted waiter is dropped from the queue and its promise rejects
|
|
89
|
+
* with an AbortError — it never receives a freed slot.
|
|
90
|
+
* @param signal - optional caller cancellation observed during the wait only.
|
|
91
|
+
* @returns `true` when the caller holds a slot, or `false` when admission is
|
|
92
|
+
* disabled and the call bypasses the budget.
|
|
93
|
+
*/
|
|
94
|
+
acquire(signal?: AbortSignal): Promise<boolean>;
|
|
95
|
+
/**
|
|
96
|
+
* Release one held slot, granting a waiting acquirer FIFO when one is queued.
|
|
97
|
+
* @throws on a release with nothing held (programmer error — each acquire
|
|
98
|
+
* path must pair with exactly one release).
|
|
99
|
+
*/
|
|
100
|
+
release(): void;
|
|
101
|
+
/**
|
|
102
|
+
* Apply a module-config change (enabled switch / capacity) at runtime.
|
|
103
|
+
* @param config - the new module configuration.
|
|
104
|
+
*/
|
|
105
|
+
reconfigure(config: ModelSlotsConfig): void;
|
|
106
|
+
/**
|
|
107
|
+
* Transient stats snapshot.
|
|
108
|
+
* @returns the current gate statistics.
|
|
109
|
+
*/
|
|
110
|
+
snapshot(): ModelSlotsStats;
|
|
111
|
+
}
|
|
112
|
+
/**
|
|
113
|
+
* Resolve the process-global gate, creating it on first use.
|
|
114
|
+
* @returns the shared process-global gate.
|
|
115
|
+
*/
|
|
116
|
+
export declare function globalModelSlotGate(): ModelSlotGate;
|
|
117
|
+
/** Reset the process-global gate identity. Test-only. */
|
|
118
|
+
export declare function resetGlobalModelSlotGate(): void;
|
|
119
|
+
declare module '@deepseek-ai/cordis' {
|
|
120
|
+
interface Context {
|
|
121
|
+
modelSlots: ModelSlotsService;
|
|
122
|
+
}
|
|
123
|
+
}
|
|
124
|
+
/**
|
|
125
|
+
* Host service exposing the global gate: live stats and runtime capacity
|
|
126
|
+
* controls. Constructed inside {@link apply}; registered with the mounting
|
|
127
|
+
* fiber, so stop/update removes it.
|
|
128
|
+
*/
|
|
129
|
+
export declare class ModelSlotsService extends Service {
|
|
130
|
+
private readonly track;
|
|
131
|
+
constructor(ctx: Context, gate: ModelSlotGate);
|
|
132
|
+
/**
|
|
133
|
+
* Transient admission snapshot for operators and the GUI.
|
|
134
|
+
* @returns the current slot-gate stats.
|
|
135
|
+
*/
|
|
136
|
+
stats(): ModelSlotsStats;
|
|
137
|
+
/**
|
|
138
|
+
* Toggle admission enforcement (independent of capacity).
|
|
139
|
+
* @param enabled - whether admission is enforced.
|
|
140
|
+
*/
|
|
141
|
+
setEnabled(enabled: boolean): void;
|
|
142
|
+
/**
|
|
143
|
+
* Change the host-wide slot budget; a shrink takes effect as calls drain.
|
|
144
|
+
* @param capacity - the new slot budget (clamped to >= 1).
|
|
145
|
+
*/
|
|
146
|
+
setCapacity(capacity: number): void;
|
|
147
|
+
/**
|
|
148
|
+
* Wrap one `llm/stream` waterfall tail in the admission gate: acquire a slot
|
|
149
|
+
* before the first chunk is requested and hold it until the stream settles
|
|
150
|
+
* (finish, error, or consumer abort). Aborting while queued rejects the
|
|
151
|
+
* acquire, which surfaces as a thrown AbortError to the stream consumer.
|
|
152
|
+
* @param options - the full model request; `signal` cancels the wait.
|
|
153
|
+
* @param next - the downstream waterfall tail.
|
|
154
|
+
* @returns a stream that begins only after a slot is held.
|
|
155
|
+
*/
|
|
156
|
+
wrap(options: GenerateOptions, next: () => AsyncIterable<StreamChunk>): AsyncIterable<StreamChunk>;
|
|
157
|
+
}
|
|
158
|
+
/**
|
|
159
|
+
* Install host-wide model-slot admission: configure the global gate, expose
|
|
160
|
+
* `ctx.modelSlots`, and listen at `llm/stream`. Mount exactly once per host in
|
|
161
|
+
* a shared-row composition (`dsh-base` ships the row); additional mounts share
|
|
162
|
+
* the same global budget and are safe but redundant.
|
|
163
|
+
* @param ctx - the mounting context (assumed to be the host root).
|
|
164
|
+
* @param config - admission switch and slot budget.
|
|
165
|
+
*/
|
|
166
|
+
export declare function apply(ctx: Context, config?: Config): void;
|
|
167
|
+
export default apply;
|
|
168
|
+
//# sourceMappingURL=index.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkCG;AAEH,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AACtD,OAAO,CAAC,MAAM,0BAA0B,CAAA;AACxC,OAAO,KAAK,EAAE,eAAe,EAAE,WAAW,EAAE,MAAM,sBAAsB,CAAA;AAExE,qDAAqD;AACrD,eAAO,MAAM,IAAI,cAAc,CAAA;AAE/B,mDAAmD;AACnD,MAAM,WAAW,MAAM;IACrB,yEAAyE;IACzE,OAAO,CAAC,EAAE,OAAO,CAAA;IACjB,0DAA0D;IAC1D,QAAQ,CAAC,EAAE,MAAM,CAAA;CAClB;AAED,eAAO,MAAM,MAAM,EAAE,CAAC,CAAC,MAAM,CAG3B,CAAA;AAEF,mFAAmF;AACnF,MAAM,WAAW,gBAAgB;IAC/B,+CAA+C;IAC/C,OAAO,EAAE,OAAO,CAAA;IAChB,sDAAsD;IACtD,QAAQ,EAAE,MAAM,CAAA;CACjB;AAED,iEAAiE;AACjE,MAAM,WAAW,eAAe;IAC9B,+CAA+C;IAC/C,OAAO,EAAE,OAAO,CAAA;IAChB,sDAAsD;IACtD,QAAQ,EAAE,MAAM,CAAA;IAChB,qDAAqD;IACrD,OAAO,EAAE,MAAM,CAAA;IACf,uDAAuD;IACvD,OAAO,EAAE,MAAM,CAAA;IACf,kFAAkF;IAClF,aAAa,EAAE,MAAM,CAAA;CACtB;AAiBD;;;GAGG;AACH,qBAAa,aAAa;IACxB,+CAA+C;IAC/C,OAAO,UAAO;IACd,sCAAsC;IACtC,QAAQ,SAAI;IACZ,qDAAqD;IACrD,OAAO,SAAI;IACX,uDAAuD;IACvD,OAAO,SAAI;IACX,iDAAiD;IACjD,aAAa,SAAI;IACjB,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAe;IAErC;;;;;;;;OAQG;IACH,OAAO,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,OAAO,CAAC;IAiC/C;;;;OAIG;IACH,OAAO,IAAI,IAAI;IAcf;;;OAGG;IACH,WAAW,CAAC,MAAM,EAAE,gBAAgB,GAAG,IAAI;IAK3C;;;OAGG;IACH,QAAQ,IAAI,eAAe;CAS5B;AAKD;;;GAGG;AACH,wBAAgB,mBAAmB,IAAI,aAAa,CAGnD;AAED,yDAAyD;AACzD,wBAAgB,wBAAwB,IAAI,IAAI,CAE/C;AAED,OAAO,QAAQ,qBAAqB,CAAC;IACnC,UAAU,OAAO;QACf,UAAU,EAAE,iBAAiB,CAAA;KAC9B;CACF;AAED;;;;GAIG;AACH,qBAAa,iBAAkB,SAAQ,OAAO;IAC5C,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAe;gBAEzB,GAAG,EAAE,OAAO,EAAE,IAAI,EAAE,aAAa;IAK7C;;;OAGG;IACH,KAAK,IAAI,eAAe;IAIxB;;;OAGG;IACH,UAAU,CAAC,OAAO,EAAE,OAAO,GAAG,IAAI;IAIlC;;;OAGG;IACH,WAAW,CAAC,QAAQ,EAAE,MAAM,GAAG,IAAI;IAKnC;;;;;;;;OAQG;IACI,IAAI,CAAC,OAAO,EAAE,eAAe,EAAE,IAAI,EAAE,MAAM,aAAa,CAAC,WAAW,CAAC,GAAG,aAAa,CAAC,WAAW,CAAC;CAa1G;AAED;;;;;;;GAOG;AACH,wBAAgB,KAAK,CAAC,GAAG,EAAE,OAAO,EAAE,MAAM,GAAE,MAAW,GAAG,IAAI,CAM7D;AAED,eAAe,KAAK,CAAA"}
|
package/dist/index.js
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Host-wide model-slot admission control (`ctx.modelSlots`).
|
|
3
|
+
*
|
|
4
|
+
* The deployment runs a handful of concurrent model providers behind one local
|
|
5
|
+
* inference endpoint (typically 2–3 slots, ~1M context each). Without admission
|
|
6
|
+
* control the main agent's own turns, running subagents, and workflow fan-out
|
|
7
|
+
* can stack dozens of simultaneous LLM bursts against that endpoint, so every
|
|
8
|
+
* call slows to endpoint-queue latency and the "N concurrent" budget becomes
|
|
9
|
+
* meaningless. This plugin makes the model budget explicit and host-global:
|
|
10
|
+
*
|
|
11
|
+
* - It decides FIFO at the `llm/stream` waterfall — the single chokepoint
|
|
12
|
+
* every model-backed call crosses (main agent loops, in-process subagents,
|
|
13
|
+
* worker-thread children, workflows, title/compaction side-requests),
|
|
14
|
+
* regardless of which session or context initiated it.
|
|
15
|
+
* - The budget itself lives in a module-global gate, so every derived context
|
|
16
|
+
* and plugin instance shares one host-wide pool. Capacity is configured at
|
|
17
|
+
* mount (`llm-slots` row, default 3) and can be raised/lowered at runtime
|
|
18
|
+
* through `ctx.modelSlots.setCapacity()`.
|
|
19
|
+
* - A call waiting for a slot holds no LLM work; its AbortSignal applies
|
|
20
|
+
* through the wait, so cancellation during admission surfaces as an
|
|
21
|
+
* AbortError upstream instead of leaking a straggler slot.
|
|
22
|
+
* - One logical call holds one slot for its full lifetime (including adapter
|
|
23
|
+
* retries), which also caps retry-storm concurrency: a failing endpoint
|
|
24
|
+
* cannot fan out unbounded retries against the same pool.
|
|
25
|
+
*
|
|
26
|
+
* The plugin deliberately reads no LLM service state: it only listens on the
|
|
27
|
+
* shared event bus, so it mounts in any context (root or child) and is trivially
|
|
28
|
+
* testable without a model adapter.
|
|
29
|
+
*
|
|
30
|
+
* Conceptually inspired by firstmate (https://github.com/kunchenguid/firstmate),
|
|
31
|
+
* MIT, © 2026 Kun Chen — its per-agent harness/model allocation at intake.
|
|
32
|
+
* No firstmate code is included.
|
|
33
|
+
*
|
|
34
|
+
* @module @hy-sde-org/dsh-llm-slots
|
|
35
|
+
*/
|
|
36
|
+
import { Service } from '@deepseek-ai/cordis';
|
|
37
|
+
import z from '@deepseek-ai/schemastery';
|
|
38
|
+
/** Cordis plugin name used by Loader diagnostics. */
|
|
39
|
+
export const name = 'llm-slots';
|
|
40
|
+
export const Config = z.object({
|
|
41
|
+
enabled: z.boolean().default(true),
|
|
42
|
+
capacity: z.natural().min(1).max(64).default(3),
|
|
43
|
+
});
|
|
44
|
+
/** Error thrown when a queued admission is cancelled while waiting for a slot. */
|
|
45
|
+
function admissionAborted() {
|
|
46
|
+
return Object.assign(new Error('model-slot wait cancelled'), { name: 'AbortError' });
|
|
47
|
+
}
|
|
48
|
+
/**
|
|
49
|
+
* Host-global FIFO model-slot gate. One instance per process (regardless of
|
|
50
|
+
* how many contexts mount {@link apply}), so the budget is genuinely global.
|
|
51
|
+
*/
|
|
52
|
+
export class ModelSlotGate {
|
|
53
|
+
/** Whether admission is currently enforced. */
|
|
54
|
+
enabled = true;
|
|
55
|
+
/** Current concurrent-slot budget. */
|
|
56
|
+
capacity = 3;
|
|
57
|
+
/** Slots currently held by in-flight model calls. */
|
|
58
|
+
running = 0;
|
|
59
|
+
/** Calls currently queued, waiting FIFO for a slot. */
|
|
60
|
+
waiting = 0;
|
|
61
|
+
/** Lifetime acquisitions served by this gate. */
|
|
62
|
+
acquiredTotal = 0;
|
|
63
|
+
queue = [];
|
|
64
|
+
/**
|
|
65
|
+
* Acquire one model slot, resolving immediately when one is free or the gate
|
|
66
|
+
* is disabled, and buffering FIFO otherwise. The caller's signal cancels the
|
|
67
|
+
* wait: an aborted waiter is dropped from the queue and its promise rejects
|
|
68
|
+
* with an AbortError — it never receives a freed slot.
|
|
69
|
+
* @param signal - optional caller cancellation observed during the wait only.
|
|
70
|
+
* @returns `true` when the caller holds a slot, or `false` when admission is
|
|
71
|
+
* disabled and the call bypasses the budget.
|
|
72
|
+
*/
|
|
73
|
+
acquire(signal) {
|
|
74
|
+
if (!this.enabled)
|
|
75
|
+
return Promise.resolve(false);
|
|
76
|
+
if (this.running < this.capacity) {
|
|
77
|
+
this.running += 1;
|
|
78
|
+
this.acquiredTotal += 1;
|
|
79
|
+
return Promise.resolve(true);
|
|
80
|
+
}
|
|
81
|
+
this.waiting += 1;
|
|
82
|
+
return new Promise((resolve, reject) => {
|
|
83
|
+
const waiter = {
|
|
84
|
+
grant: () => {
|
|
85
|
+
this.waiting -= 1;
|
|
86
|
+
this.acquiredTotal += 1;
|
|
87
|
+
waiter.cleanup();
|
|
88
|
+
resolve(true);
|
|
89
|
+
},
|
|
90
|
+
reject: (error) => {
|
|
91
|
+
const index = this.queue.indexOf(waiter);
|
|
92
|
+
if (index >= 0) {
|
|
93
|
+
this.queue.splice(index, 1);
|
|
94
|
+
this.waiting -= 1;
|
|
95
|
+
}
|
|
96
|
+
waiter.cleanup();
|
|
97
|
+
reject(error);
|
|
98
|
+
},
|
|
99
|
+
cleanup: () => signal?.removeEventListener('abort', onAbort),
|
|
100
|
+
};
|
|
101
|
+
const onAbort = () => { waiter.reject(admissionAborted()); };
|
|
102
|
+
signal?.addEventListener('abort', onAbort, { once: true });
|
|
103
|
+
this.queue.push(waiter);
|
|
104
|
+
});
|
|
105
|
+
}
|
|
106
|
+
/**
|
|
107
|
+
* Release one held slot, granting a waiting acquirer FIFO when one is queued.
|
|
108
|
+
* @throws on a release with nothing held (programmer error — each acquire
|
|
109
|
+
* path must pair with exactly one release).
|
|
110
|
+
*/
|
|
111
|
+
release() {
|
|
112
|
+
if (this.running === 0) {
|
|
113
|
+
throw new Error('model-slot gate released with no held slot');
|
|
114
|
+
}
|
|
115
|
+
this.running -= 1;
|
|
116
|
+
const next = this.queue.shift();
|
|
117
|
+
if (next !== undefined) {
|
|
118
|
+
// The freed slot transfers directly to the head waiter: it consumes the
|
|
119
|
+
// free capacity without another running increment/decrement pair.
|
|
120
|
+
this.running += 1;
|
|
121
|
+
next.grant();
|
|
122
|
+
}
|
|
123
|
+
}
|
|
124
|
+
/**
|
|
125
|
+
* Apply a module-config change (enabled switch / capacity) at runtime.
|
|
126
|
+
* @param config - the new module configuration.
|
|
127
|
+
*/
|
|
128
|
+
reconfigure(config) {
|
|
129
|
+
this.enabled = config.enabled;
|
|
130
|
+
this.capacity = config.capacity;
|
|
131
|
+
}
|
|
132
|
+
/**
|
|
133
|
+
* Transient stats snapshot.
|
|
134
|
+
* @returns the current gate statistics.
|
|
135
|
+
*/
|
|
136
|
+
snapshot() {
|
|
137
|
+
return {
|
|
138
|
+
enabled: this.enabled,
|
|
139
|
+
capacity: this.capacity,
|
|
140
|
+
running: this.running,
|
|
141
|
+
waiting: this.waiting,
|
|
142
|
+
acquiredTotal: this.acquiredTotal,
|
|
143
|
+
};
|
|
144
|
+
}
|
|
145
|
+
}
|
|
146
|
+
/** Process-global gate shared by every mounted instance and derived context. */
|
|
147
|
+
let gate;
|
|
148
|
+
/**
|
|
149
|
+
* Resolve the process-global gate, creating it on first use.
|
|
150
|
+
* @returns the shared process-global gate.
|
|
151
|
+
*/
|
|
152
|
+
export function globalModelSlotGate() {
|
|
153
|
+
gate ??= new ModelSlotGate();
|
|
154
|
+
return gate;
|
|
155
|
+
}
|
|
156
|
+
/** Reset the process-global gate identity. Test-only. */
|
|
157
|
+
export function resetGlobalModelSlotGate() {
|
|
158
|
+
gate = undefined;
|
|
159
|
+
}
|
|
160
|
+
/**
|
|
161
|
+
* Host service exposing the global gate: live stats and runtime capacity
|
|
162
|
+
* controls. Constructed inside {@link apply}; registered with the mounting
|
|
163
|
+
* fiber, so stop/update removes it.
|
|
164
|
+
*/
|
|
165
|
+
export class ModelSlotsService extends Service {
|
|
166
|
+
track;
|
|
167
|
+
constructor(ctx, gate) {
|
|
168
|
+
super(ctx, 'modelSlots');
|
|
169
|
+
this.track = gate;
|
|
170
|
+
}
|
|
171
|
+
/**
|
|
172
|
+
* Transient admission snapshot for operators and the GUI.
|
|
173
|
+
* @returns the current slot-gate stats.
|
|
174
|
+
*/
|
|
175
|
+
stats() {
|
|
176
|
+
return this.track.snapshot();
|
|
177
|
+
}
|
|
178
|
+
/**
|
|
179
|
+
* Toggle admission enforcement (independent of capacity).
|
|
180
|
+
* @param enabled - whether admission is enforced.
|
|
181
|
+
*/
|
|
182
|
+
setEnabled(enabled) {
|
|
183
|
+
this.track.reconfigure({ enabled, capacity: this.track.capacity });
|
|
184
|
+
}
|
|
185
|
+
/**
|
|
186
|
+
* Change the host-wide slot budget; a shrink takes effect as calls drain.
|
|
187
|
+
* @param capacity - the new slot budget (clamped to >= 1).
|
|
188
|
+
*/
|
|
189
|
+
setCapacity(capacity) {
|
|
190
|
+
const clamped = Math.max(1, Math.trunc(capacity));
|
|
191
|
+
this.track.reconfigure({ enabled: this.track.enabled, capacity: clamped });
|
|
192
|
+
}
|
|
193
|
+
/**
|
|
194
|
+
* Wrap one `llm/stream` waterfall tail in the admission gate: acquire a slot
|
|
195
|
+
* before the first chunk is requested and hold it until the stream settles
|
|
196
|
+
* (finish, error, or consumer abort). Aborting while queued rejects the
|
|
197
|
+
* acquire, which surfaces as a thrown AbortError to the stream consumer.
|
|
198
|
+
* @param options - the full model request; `signal` cancels the wait.
|
|
199
|
+
* @param next - the downstream waterfall tail.
|
|
200
|
+
* @returns a stream that begins only after a slot is held.
|
|
201
|
+
*/
|
|
202
|
+
async *wrap(options, next) {
|
|
203
|
+
// A `false` grant means admission is disabled: the call bypasses the
|
|
204
|
+
// budget entirely, so no paired release runs.
|
|
205
|
+
if (!(await this.track.acquire(options.signal))) {
|
|
206
|
+
yield* next();
|
|
207
|
+
return;
|
|
208
|
+
}
|
|
209
|
+
try {
|
|
210
|
+
yield* next();
|
|
211
|
+
}
|
|
212
|
+
finally {
|
|
213
|
+
this.track.release();
|
|
214
|
+
}
|
|
215
|
+
}
|
|
216
|
+
}
|
|
217
|
+
/**
|
|
218
|
+
* Install host-wide model-slot admission: configure the global gate, expose
|
|
219
|
+
* `ctx.modelSlots`, and listen at `llm/stream`. Mount exactly once per host in
|
|
220
|
+
* a shared-row composition (`dsh-base` ships the row); additional mounts share
|
|
221
|
+
* the same global budget and are safe but redundant.
|
|
222
|
+
* @param ctx - the mounting context (assumed to be the host root).
|
|
223
|
+
* @param config - admission switch and slot budget.
|
|
224
|
+
*/
|
|
225
|
+
export function apply(ctx, config = {}) {
|
|
226
|
+
const { enabled, capacity } = Config(config);
|
|
227
|
+
const track = globalModelSlotGate();
|
|
228
|
+
track.reconfigure({ enabled, capacity });
|
|
229
|
+
const service = new ModelSlotsService(ctx, track);
|
|
230
|
+
ctx.on('llm/stream', (options, next) => service.wrap(options, next), { global: true, prepend: true });
|
|
231
|
+
}
|
|
232
|
+
export default apply;
|
|
233
|
+
//# sourceMappingURL=index.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkCG;AAEH,OAAO,EAAW,OAAO,EAAE,MAAM,qBAAqB,CAAA;AACtD,OAAO,CAAC,MAAM,0BAA0B,CAAA;AAGxC,qDAAqD;AACrD,MAAM,CAAC,MAAM,IAAI,GAAG,WAAW,CAAA;AAU/B,MAAM,CAAC,MAAM,MAAM,GAAc,CAAC,CAAC,MAAM,CAAC;IACxC,OAAO,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC;IAClC,QAAQ,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;CAChD,CAAC,CAAA;AAkCF,kFAAkF;AAClF,SAAS,gBAAgB;IACvB,OAAO,MAAM,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,2BAA2B,CAAC,EAAE,EAAE,IAAI,EAAE,YAAY,EAAE,CAAC,CAAA;AACtF,CAAC;AAED;;;GAGG;AACH,MAAM,OAAO,aAAa;IACxB,+CAA+C;IAC/C,OAAO,GAAG,IAAI,CAAA;IACd,sCAAsC;IACtC,QAAQ,GAAG,CAAC,CAAA;IACZ,qDAAqD;IACrD,OAAO,GAAG,CAAC,CAAA;IACX,uDAAuD;IACvD,OAAO,GAAG,CAAC,CAAA;IACX,iDAAiD;IACjD,aAAa,GAAG,CAAC,CAAA;IACA,KAAK,GAAa,EAAE,CAAA;IAErC;;;;;;;;OAQG;IACH,OAAO,CAAC,MAAoB;QAC1B,IAAI,CAAC,IAAI,CAAC,OAAO;YAAE,OAAO,OAAO,CAAC,OAAO,CAAC,KAAK,CAAC,CAAA;QAChD,IAAI,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC,QAAQ,EAAE,CAAC;YACjC,IAAI,CAAC,OAAO,IAAI,CAAC,CAAA;YACjB,IAAI,CAAC,aAAa,IAAI,CAAC,CAAA;YACvB,OAAO,OAAO,CAAC,OAAO,CAAC,IAAI,CAAC,CAAA;QAC9B,CAAC;QACD,IAAI,CAAC,OAAO,IAAI,CAAC,CAAA;QACjB,OAAO,IAAI,OAAO,CAAU,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;YAC9C,MAAM,MAAM,GAAW;gBACrB,KAAK,EAAE,GAAG,EAAE;oBACV,IAAI,CAAC,OAAO,IAAI,CAAC,CAAA;oBACjB,IAAI,CAAC,aAAa,IAAI,CAAC,CAAA;oBACvB,MAAM,CAAC,OAAO,EAAE,CAAA;oBAChB,OAAO,CAAC,IAAI,CAAC,CAAA;gBACf,CAAC;gBACD,MAAM,EAAE,CAAC,KAAY,EAAE,EAAE;oBACvB,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,CAAA;oBACxC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;wBACf,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC,CAAA;wBAC3B,IAAI,CAAC,OAAO,IAAI,CAAC,CAAA;oBACnB,CAAC;oBACD,MAAM,CAAC,OAAO,EAAE,CAAA;oBAChB,MAAM,CAAC,KAAK,CAAC,CAAA;gBACf,CAAC;gBACD,OAAO,EAAE,GAAG,EAAE,CAAC,MAAM,EAAE,mBAAmB,CAAC,OAAO,EAAE,OAAO,CAAC;aAC7D,CAAA;YACD,MAAM,OAAO,GAAG,GAAS,EAAE,GAAG,MAAM,CAAC,MAAM,CAAC,gBAAgB,EAAE,CAAC,CAAA,CAAC,CAAC,CAAA;YACjE,MAAM,EAAE,gBAAgB,CAAC,OAAO,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,CAAA;YAC1D,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,CAAA;QACzB,CAAC,CAAC,CAAA;IACJ,CAAC;IAED;;;;OAIG;IACH,OAAO;QACL,IAAI,IAAI,CAAC,OAAO,KAAK,CAAC,EAAE,CAAC;YACvB,MAAM,IAAI,KAAK,CAAC,4CAA4C,CAAC,CAAA;QAC/D,CAAC;QACD,IAAI,CAAC,OAAO,IAAI,CAAC,CAAA;QACjB,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAA;QAC/B,IAAI,IAAI,KAAK,SAAS,EAAE,CAAC;YACvB,wEAAwE;YACxE,kEAAkE;YAClE,IAAI,CAAC,OAAO,IAAI,CAAC,CAAA;YACjB,IAAI,CAAC,KAAK,EAAE,CAAA;QACd,CAAC;IACH,CAAC;IAED;;;OAGG;IACH,WAAW,CAAC,MAAwB;QAClC,IAAI,CAAC,OAAO,GAAG,MAAM,CAAC,OAAO,CAAA;QAC7B,IAAI,CAAC,QAAQ,GAAG,MAAM,CAAC,QAAQ,CAAA;IACjC,CAAC;IAED;;;OAGG;IACH,QAAQ;QACN,OAAO;YACL,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,QAAQ,EAAE,IAAI,CAAC,QAAQ;YACvB,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,aAAa,EAAE,IAAI,CAAC,aAAa;SAClC,CAAA;IACH,CAAC;CACF;AAED,gFAAgF;AAChF,IAAI,IAA+B,CAAA;AAEnC;;;GAGG;AACH,MAAM,UAAU,mBAAmB;IACjC,IAAI,KAAK,IAAI,aAAa,EAAE,CAAA;IAC5B,OAAO,IAAI,CAAA;AACb,CAAC;AAED,yDAAyD;AACzD,MAAM,UAAU,wBAAwB;IACtC,IAAI,GAAG,SAAS,CAAA;AAClB,CAAC;AAQD;;;;GAIG;AACH,MAAM,OAAO,iBAAkB,SAAQ,OAAO;IAC3B,KAAK,CAAe;IAErC,YAAY,GAAY,EAAE,IAAmB;QAC3C,KAAK,CAAC,GAAG,EAAE,YAAY,CAAC,CAAA;QACxB,IAAI,CAAC,KAAK,GAAG,IAAI,CAAA;IACnB,CAAC;IAED;;;OAGG;IACH,KAAK;QACH,OAAO,IAAI,CAAC,KAAK,CAAC,QAAQ,EAAE,CAAA;IAC9B,CAAC;IAED;;;OAGG;IACH,UAAU,CAAC,OAAgB;QACzB,IAAI,CAAC,KAAK,CAAC,WAAW,CAAC,EAAE,OAAO,EAAE,QAAQ,EAAE,IAAI,CAAC,KAAK,CAAC,QAAQ,EAAE,CAAC,CAAA;IACpE,CAAC;IAED;;;OAGG;IACH,WAAW,CAAC,QAAgB;QAC1B,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAA;QACjD,IAAI,CAAC,KAAK,CAAC,WAAW,CAAC,EAAE,OAAO,EAAE,IAAI,CAAC,KAAK,CAAC,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,CAAC,CAAA;IAC5E,CAAC;IAED;;;;;;;;OAQG;IACH,KAAK,CAAC,CAAC,IAAI,CAAC,OAAwB,EAAE,IAAsC;QAC1E,qEAAqE;QACrE,8CAA8C;QAC9C,IAAI,CAAC,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,EAAE,CAAC;YAChD,KAAK,CAAC,CAAC,IAAI,EAAE,CAAA;YACb,OAAM;QACR,CAAC;QACD,IAAI,CAAC;YACH,KAAK,CAAC,CAAC,IAAI,EAAE,CAAA;QACf,CAAC;gBAAS,CAAC;YACT,IAAI,CAAC,KAAK,CAAC,OAAO,EAAE,CAAA;QACtB,CAAC;IACH,CAAC;CACF;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,KAAK,CAAC,GAAY,EAAE,SAAiB,EAAE;IACrD,MAAM,EAAE,OAAO,EAAE,QAAQ,EAAE,GAAG,MAAM,CAAC,MAAM,CAA2C,CAAA;IACtF,MAAM,KAAK,GAAG,mBAAmB,EAAE,CAAA;IACnC,KAAK,CAAC,WAAW,CAAC,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC,CAAA;IACxC,MAAM,OAAO,GAAG,IAAI,iBAAiB,CAAC,GAAG,EAAE,KAAK,CAAC,CAAA;IACjD,GAAG,CAAC,EAAE,CAAC,YAAY,EAAE,CAAC,OAAO,EAAE,IAAI,EAAE,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,OAAO,EAAE,IAAI,CAAC,EAAE,EAAE,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC,CAAA;AACvG,CAAC;AAED,eAAe,KAAK,CAAA"}
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
/** Package-owned model-slot admission accounting invariants. @module @hy-sde-org/dsh-llm-slots/invariant */
|
|
2
|
+
import type { Context } from '@deepseek-ai/cordis';
|
|
3
|
+
/** Cordis companion plugin name. */
|
|
4
|
+
export declare const name = "llm-slots-invariant";
|
|
5
|
+
/** Service required before the companion can reserve package ownership. */
|
|
6
|
+
export declare const inject: string[];
|
|
7
|
+
/**
|
|
8
|
+
* Register the model-slot invariant companion.
|
|
9
|
+
* @param ctx - Cordis context carrying the invariant service.
|
|
10
|
+
* @returns the installed registration's disposer after setup succeeds.
|
|
11
|
+
*/
|
|
12
|
+
export declare const apply: (ctx: Context) => Promise<() => void>;
|
|
13
|
+
//# sourceMappingURL=invariant.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"invariant.d.ts","sourceRoot":"","sources":["../src/invariant.ts"],"names":[],"mappings":"AAAA,4GAA4G;AAE5G,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AAOlD,oCAAoC;AACpC,eAAO,MAAM,IAAI,wBAAwB,CAAA;AACzC,2EAA2E;AAC3E,eAAO,MAAM,MAAM,UAAiB,CAAA;AA0DpC;;;;GAIG;AACH,eAAO,MAAM,KAAK,GAAI,KAAK,OAAO,KAAG,OAAO,CAAC,MAAM,IAAI,CACU,CAAA"}
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
/** Package-owned model-slot admission accounting invariants. @module @hy-sde-org/dsh-llm-slots/invariant */
|
|
2
|
+
const PACKAGE_NAME = '@hy-sde-org/dsh-llm-slots';
|
|
3
|
+
/** Cordis companion plugin name. */
|
|
4
|
+
export const name = 'llm-slots-invariant';
|
|
5
|
+
/** Service required before the companion can reserve package ownership. */
|
|
6
|
+
export const inject = ['invariants'];
|
|
7
|
+
/** Read live admission stats, or `undefined` when the plugin is not mounted. */
|
|
8
|
+
function readStats(ctx) {
|
|
9
|
+
const slots = ctx.get('modelSlots');
|
|
10
|
+
if (slots === undefined)
|
|
11
|
+
return undefined;
|
|
12
|
+
try {
|
|
13
|
+
return slots.stats();
|
|
14
|
+
}
|
|
15
|
+
catch {
|
|
16
|
+
// A service mid-teardown may throw; absence of stats makes the checks inert.
|
|
17
|
+
return undefined;
|
|
18
|
+
}
|
|
19
|
+
}
|
|
20
|
+
/**
|
|
21
|
+
* Assert the gate's structural accounting around one model call, as far as a
|
|
22
|
+
* single observer can see it without coupling to concurrent streams:
|
|
23
|
+
*
|
|
24
|
+
* - `running` stays within `[0, capacity]` at settlement — a value above
|
|
25
|
+
* capacity means acquisitions leaked past the budget; a negative value means
|
|
26
|
+
* releases outran acquisitions. Cross-stream interleaving cannot break this
|
|
27
|
+
* bound, so the check never races a sibling.
|
|
28
|
+
* - `waiting` never goes negative.
|
|
29
|
+
* - `acquiredTotal` is monotonic across the observed call (it only counts
|
|
30
|
+
* acquisitions and never decreases, regardless of concurrent streams).
|
|
31
|
+
*
|
|
32
|
+
* A missing service (plugin not mounted on the checked root, or mid-teardown)
|
|
33
|
+
* makes every check inert rather than failing.
|
|
34
|
+
*/
|
|
35
|
+
async function* inspected(_options, next, fail, stats) {
|
|
36
|
+
const before = stats();
|
|
37
|
+
try {
|
|
38
|
+
yield* next();
|
|
39
|
+
}
|
|
40
|
+
finally {
|
|
41
|
+
const after = stats();
|
|
42
|
+
if (before === undefined || after === undefined)
|
|
43
|
+
return;
|
|
44
|
+
if (after.running < 0 || after.running > after.capacity) {
|
|
45
|
+
fail(`model-slot accounting broken after one call: running=${after.running} outside [0, ${after.capacity}]`);
|
|
46
|
+
}
|
|
47
|
+
if (after.waiting < 0) {
|
|
48
|
+
fail(`model-slot waiting counter went negative: ${after.waiting}`);
|
|
49
|
+
}
|
|
50
|
+
if (after.acquiredTotal < before.acquiredTotal) {
|
|
51
|
+
fail(`model-slot acquiredTotal decreased: ${after.acquiredTotal} < ${before.acquiredTotal}`);
|
|
52
|
+
}
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
/** Install admission-accounting checks around every model call. */
|
|
56
|
+
const install = (ctx, fail) => {
|
|
57
|
+
ctx.on('llm/stream', (options, next) => inspected(options, next, fail, () => readStats(ctx)), { global: true });
|
|
58
|
+
};
|
|
59
|
+
/**
|
|
60
|
+
* Register the model-slot invariant companion.
|
|
61
|
+
* @param ctx - Cordis context carrying the invariant service.
|
|
62
|
+
* @returns the installed registration's disposer after setup succeeds.
|
|
63
|
+
*/
|
|
64
|
+
export const apply = (ctx) => Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install));
|
|
65
|
+
//# sourceMappingURL=invariant.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"invariant.js","sourceRoot":"","sources":["../src/invariant.ts"],"names":[],"mappings":"AAAA,4GAA4G;AAO5G,MAAM,YAAY,GAAG,2BAA2B,CAAA;AAEhD,oCAAoC;AACpC,MAAM,CAAC,MAAM,IAAI,GAAG,qBAAqB,CAAA;AACzC,2EAA2E;AAC3E,MAAM,CAAC,MAAM,MAAM,GAAG,CAAC,YAAY,CAAC,CAAA;AAEpC,gFAAgF;AAChF,SAAS,SAAS,CAAC,GAAY;IAC7B,MAAM,KAAK,GAAG,GAAG,CAAC,GAAG,CAAC,YAAY,CAAC,CAAA;IACnC,IAAI,KAAK,KAAK,SAAS;QAAE,OAAO,SAAS,CAAA;IACzC,IAAI,CAAC;QACH,OAAO,KAAK,CAAC,KAAK,EAAE,CAAA;IACtB,CAAC;IAAC,MAAM,CAAC;QACP,6EAA6E;QAC7E,OAAO,SAAS,CAAA;IAClB,CAAC;AACH,CAAC;AAED;;;;;;;;;;;;;;GAcG;AACH,KAAK,SAAS,CAAC,CAAC,SAAS,CACvB,QAAyB,EACzB,IAAsC,EACtC,IAAsB,EACtB,KAAwC;IAExC,MAAM,MAAM,GAAG,KAAK,EAAE,CAAA;IACtB,IAAI,CAAC;QACH,KAAK,CAAC,CAAC,IAAI,EAAE,CAAA;IACf,CAAC;YAAS,CAAC;QACT,MAAM,KAAK,GAAG,KAAK,EAAE,CAAA;QACrB,IAAI,MAAM,KAAK,SAAS,IAAI,KAAK,KAAK,SAAS;YAAE,OAAM;QACvD,IAAI,KAAK,CAAC,OAAO,GAAG,CAAC,IAAI,KAAK,CAAC,OAAO,GAAG,KAAK,CAAC,QAAQ,EAAE,CAAC;YACxD,IAAI,CAAC,wDAAwD,KAAK,CAAC,OAAO,gBAAgB,KAAK,CAAC,QAAQ,GAAG,CAAC,CAAA;QAC9G,CAAC;QACD,IAAI,KAAK,CAAC,OAAO,GAAG,CAAC,EAAE,CAAC;YACtB,IAAI,CAAC,6CAA6C,KAAK,CAAC,OAAO,EAAE,CAAC,CAAA;QACpE,CAAC;QACD,IAAI,KAAK,CAAC,aAAa,GAAG,MAAM,CAAC,aAAa,EAAE,CAAC;YAC/C,IAAI,CAAC,uCAAuC,KAAK,CAAC,aAAa,MAAM,MAAM,CAAC,aAAa,EAAE,CAAC,CAAA;QAC9F,CAAC;IACH,CAAC;AACH,CAAC;AAED,mEAAmE;AACnE,MAAM,OAAO,GAAuB,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE;IAChD,GAAG,CAAC,EAAE,CAAC,YAAY,EAAE,CAAC,OAAO,EAAE,IAAI,EAAE,EAAE,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,GAAG,EAAE,CAAC,SAAS,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC,CAAA;AACjH,CAAC,CAAA;AAED;;;;GAIG;AACH,MAAM,CAAC,MAAM,KAAK,GAAG,CAAC,GAAY,EAAuB,EAAE,CACzD,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,UAAU,CAAC,QAAQ,CAAC,YAAY,EAAE,OAAO,CAAC,CAAC,CAAA"}
|
package/package.json
ADDED
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@hy-sde-org/dsh-llm-slots",
|
|
3
|
+
"version": "0.1.2-rc.1",
|
|
4
|
+
"description": "Host-wide model-slot admission control (ctx.modelSlots) decided FIFO at the llm/stream waterfall: a shared budget over every model call so 2-3 concurrent local inference slots stay predictable, as a standalone npm package.",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"main": "./dist/index.js",
|
|
7
|
+
"types": "./dist/index.d.ts",
|
|
8
|
+
"exports": {
|
|
9
|
+
".": {
|
|
10
|
+
"types": "./dist/index.d.ts",
|
|
11
|
+
"default": "./dist/index.js"
|
|
12
|
+
},
|
|
13
|
+
"./invariant": {
|
|
14
|
+
"types": "./dist/invariant.d.ts",
|
|
15
|
+
"default": "./dist/invariant.js"
|
|
16
|
+
},
|
|
17
|
+
"./package.json": "./package.json"
|
|
18
|
+
},
|
|
19
|
+
"files": [
|
|
20
|
+
"dist",
|
|
21
|
+
"README.md",
|
|
22
|
+
"LICENSE"
|
|
23
|
+
],
|
|
24
|
+
"engines": {
|
|
25
|
+
"node": ">=22.19.0"
|
|
26
|
+
},
|
|
27
|
+
"publishConfig": {
|
|
28
|
+
"access": "public"
|
|
29
|
+
},
|
|
30
|
+
"repository": {
|
|
31
|
+
"type": "git",
|
|
32
|
+
"url": "git+https://github.com/hy-sde/dsh-llm-slots.git"
|
|
33
|
+
},
|
|
34
|
+
"license": "MIT",
|
|
35
|
+
"peerDependencies": {
|
|
36
|
+
"@deepseek-ai/cordis": "^4.0.2",
|
|
37
|
+
"@deepseek-ai/dsh-invariants": "^0.1.2-rc.1",
|
|
38
|
+
"@deepseek-ai/dsh-llm": "^0.1.2-rc.1"
|
|
39
|
+
},
|
|
40
|
+
"dependencies": {
|
|
41
|
+
"@deepseek-ai/schemastery": "^3.18.2"
|
|
42
|
+
},
|
|
43
|
+
"devDependencies": {
|
|
44
|
+
"@deepseek-ai/cordis": "^4.0.2",
|
|
45
|
+
"@deepseek-ai/dsh-invariants": "^0.1.2-rc.1",
|
|
46
|
+
"@deepseek-ai/dsh-llm": "^0.1.2-rc.1",
|
|
47
|
+
"@deepseek-ai/schemastery": "^3.18.2",
|
|
48
|
+
"@types/node": "^24.0.0",
|
|
49
|
+
"typescript": "^5.9.3",
|
|
50
|
+
"vitest": "^3.2.7"
|
|
51
|
+
},
|
|
52
|
+
"keywords": [
|
|
53
|
+
"deepseek",
|
|
54
|
+
"agent",
|
|
55
|
+
"harness",
|
|
56
|
+
"dsh-plugin",
|
|
57
|
+
"llm",
|
|
58
|
+
"slots",
|
|
59
|
+
"admission",
|
|
60
|
+
"concurrency",
|
|
61
|
+
"fifo"
|
|
62
|
+
],
|
|
63
|
+
"scripts": {
|
|
64
|
+
"clean": "rm -rf dist",
|
|
65
|
+
"build": "tsc -p tsconfig.build.json",
|
|
66
|
+
"check": "tsc -p tsconfig.json --noEmit",
|
|
67
|
+
"test": "vitest run"
|
|
68
|
+
}
|
|
69
|
+
}
|