@x1a0f3n9/dsh-llm-mock-server 0.1.5-rc.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.i18n.yaml +6 -0
- package/README.md +170 -0
- package/README.zh.md +170 -0
- package/lib/index.js +520 -0
- package/lib/types/bin.d.ts +7 -0
- package/lib/types/cli.d.ts +34 -0
- package/lib/types/index.d.ts +122 -0
- package/package.json +35 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 DeepSeek
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.i18n.yaml
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
|
2
|
+
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
|
3
|
+
# after editing either side, bring the other along and re-record with:
|
|
4
|
+
# pnpm run verify-translation-pairing --write packages/test-support/llm-mock-server/README.md
|
|
5
|
+
README.md: be311d9c83bcd35ad5bd0c003f6e95861b6d6976
|
|
6
|
+
README.zh.md: 7a022131d9ef09d9ef89c91cae418f093f77aca1
|
package/README.md
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "Scriptable OpenAI-compatible fault server for testing LLM adapters and recovery policy without a provider key, for test authors and demos."
|
|
3
|
+
kind: "package-library"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# @x1a0f3n9/dsh-llm-mock-server
|
|
7
|
+
|
|
8
|
+
English | [中文](README.zh.md)
|
|
9
|
+
|
|
10
|
+
## Summary
|
|
11
|
+
|
|
12
|
+
This package gives tests and demos a scriptable OpenAI-compatible HTTP/SSE endpoint, so they can exercise model-provider failures and successes without a provider key. Each accepted `/chat/completions` request consumes the next scripted behavior, including resets, stalls, malformed chunks, rate limits, server errors, completions, and tool calls. Test authors can run it with `pnpm run mock:llm` or call `startMockLlmServer`, which returns captured requests for assertions. Seeded `random` behavior supports reproducible mixed-failure stress runs.
|
|
13
|
+
|
|
14
|
+
## Table of Contents
|
|
15
|
+
|
|
16
|
+
- [Use this package](#use-this-package)
|
|
17
|
+
- [Understand the implementation](#understand-the-implementation)
|
|
18
|
+
- [Further Exploration](#further-exploration)
|
|
19
|
+
- [Model Experience](#model-experience)
|
|
20
|
+
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
|
21
|
+
- [Dev Note](#dev-note)
|
|
22
|
+
|
|
23
|
+
-----
|
|
24
|
+
|
|
25
|
+
<a id="use-this-package"></a>
|
|
26
|
+
## Use this package
|
|
27
|
+
|
|
28
|
+
This package lets a test or demo speak the provider protocol without a provider: start the server, script the wire behaviors you want to exercise, and point a real LLM adapter at its base URL.
|
|
29
|
+
|
|
30
|
+
### Running it standalone
|
|
31
|
+
|
|
32
|
+
Run the source entry from this repository:
|
|
33
|
+
|
|
34
|
+
```sh
|
|
35
|
+
pnpm run mock:llm \
|
|
36
|
+
--port 8000 \
|
|
37
|
+
--api-key mock-key \
|
|
38
|
+
--sequence partial_disconnect,success \
|
|
39
|
+
--partial-text "discard this half"
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Point the shipping DeepSeek adapter at the server; it appends `/chat/completions` to the configured base:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
DEEPSEEK_BASE_URL=http://127.0.0.1:8000/v1 \
|
|
46
|
+
DEEPSEEK_API_KEY=mock-key \
|
|
47
|
+
pnpm dsh --profile headless "test provider recovery"
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
The repository script writes JSONL to stdout: a `ready` record carries the `/v1` base URL and random seed, followed by request/result records that name both the scripted behavior and the concrete behavior selected. The package exposes no installable binary.
|
|
51
|
+
|
|
52
|
+
### Scripting behaviors
|
|
53
|
+
|
|
54
|
+
`--sequence` is a comma-separated FIFO. Exhaustion returns a structured HTTP 500; `--repeat-last` explicitly reuses the last entry.
|
|
55
|
+
|
|
56
|
+
| Behavior | Wire result |
|
|
57
|
+
|---|---|
|
|
58
|
+
| `connection_reset` | Destroy the socket before HTTP headers |
|
|
59
|
+
| `stream_disconnect` | Send SSE headers, then reset before the first event |
|
|
60
|
+
| `partial_disconnect` | Send text deltas, then reset the socket |
|
|
61
|
+
| `stall` | Send SSE headers and remain idle until client/server cancellation |
|
|
62
|
+
| `empty` | Send a valid content-less stop and `[DONE]` |
|
|
63
|
+
| `empty_body` / `stream_eof` / `partial_eof` | End cleanly without the required `[DONE]` boundary |
|
|
64
|
+
| `malformed_json` / `malformed_event` | Send invalid SSE JSON or an invalid provider chunk shape |
|
|
65
|
+
| `rate_limit` / `server_error` / `service_unavailable` | Return retry-oriented 429/500/503 JSON errors |
|
|
66
|
+
| `auth_error` / `invalid_request` / `context_overflow` / `quota_exceeded` | Return terminal or separately recovered provider errors |
|
|
67
|
+
| `success` / `slow_success` / `reasoning_success` | Stream a complete text response, optionally delayed or preceded by reasoning |
|
|
68
|
+
| `tool_call_success` / `max_tokens` | Complete with a tool call or `length` finish |
|
|
69
|
+
| `wrong_content_type` | Send a valid SSE body under `application/json` |
|
|
70
|
+
| `random` | Select a concrete request behavior from weighted seeded randomness |
|
|
71
|
+
|
|
72
|
+
`connection_refused` is CLI-only and must be the first entry. It delays binding a caller-specified nonzero port, so requests during `--listen-delay-ms` receive a real TCP refusal; the remaining entries begin after the listener starts.
|
|
73
|
+
|
|
74
|
+
### Random mode
|
|
75
|
+
|
|
76
|
+
Use a repeating `random` entry for an open-ended mixed run:
|
|
77
|
+
|
|
78
|
+
```sh
|
|
79
|
+
pnpm run mock:llm \
|
|
80
|
+
--port 8000 \
|
|
81
|
+
--sequence random \
|
|
82
|
+
--repeat-last \
|
|
83
|
+
--seed 42 \
|
|
84
|
+
--random-weights 'success=60,slow_success=10,connection_reset=5,stream_disconnect=5,partial_disconnect=10,empty=5,server_error=5'
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Omitting `--seed` generates one and prints it in the `ready` record. `--random-weights` accepts non-negative relative `behavior=weight` entries and requires at least one positive concrete behavior. The exported default is a success-heavy stress profile containing reset, disconnect, partial output, empty completion, stall, 429/5xx, clean truncation, and malformed JSON; it is test pressure, not an estimate of production incident frequency. `connection_refused` is excluded because a bound request handler cannot produce a true refusal. When random weights include `stall`, configure the client under test with a short stream-idle timeout so the scenario terminates promptly.
|
|
88
|
+
|
|
89
|
+
### Timing and content controls
|
|
90
|
+
|
|
91
|
+
The CLI exposes `--success-text`, `--partial-text`, `--reasoning-text`, `--chunk-size`, `--chunk-delay-ms`, `--disconnect-delay-ms`, `--retry-after-ms`, `--request-id`, `--tool-name`, and `--tool-arguments`. Millisecond delays are bounded integers within Node's timer range; `retryAfterMs` must also be positive. The library accepts the same camel-case options. An optional exact `apiKey` validates `Authorization: Bearer <token>`; omission accepts any token.
|
|
92
|
+
|
|
93
|
+
### What can go wrong
|
|
94
|
+
|
|
95
|
+
- **The script runs out** — exhaustion returns a structured HTTP 500; set `--repeat-last` or lengthen the sequence when a run needs more requests.
|
|
96
|
+
- **Random weights without a positive concrete behavior are rejected** — every entry must name an existing behavior and at least one must carry positive weight.
|
|
97
|
+
- **Invalid requests do not consume the script** — wrong methods, paths, bearer tokens, and malformed JSON get ordinary 4xx responses, so a misconfigured client can burn retries without advancing the sequence.
|
|
98
|
+
|
|
99
|
+
-----
|
|
100
|
+
|
|
101
|
+
<a id="understand-the-implementation"></a>
|
|
102
|
+
## Understand the implementation
|
|
103
|
+
|
|
104
|
+
<details>
|
|
105
|
+
<summary>Implementation internals — click to expand</summary>
|
|
106
|
+
|
|
107
|
+
This section explains the design of the server; the observable behavior is fully covered in [Use this package](#use-this-package).
|
|
108
|
+
|
|
109
|
+
### Design
|
|
110
|
+
|
|
111
|
+
The server is built on one rule: each accepted chat-completions request consumes exactly one behavior from an arrival-ordered FIFO cursor, and the server never retries or interprets harness policy. Validation happens before the cursor advances — only a `POST` whose path ends in `/chat/completions`, with a valid bearer token when one is configured and a parseable JSON body, consumes the script; everything else receives an ordinary 4xx. `random` entries resolve at request time through a seeded PRNG over the configured weights, so a run is reproducible from its printed seed.
|
|
112
|
+
|
|
113
|
+
### Source map
|
|
114
|
+
|
|
115
|
+
| File | Role |
|
|
116
|
+
|---|---|
|
|
117
|
+
| [`src/index.ts`](src/index.ts) | `startMockLlmServer`: listener, behavior table, seeded randomness, telemetry, captured request records |
|
|
118
|
+
| [`src/cli.ts`](src/cli.ts) | `--sequence` and timing/content option parsing, JSONL stdout telemetry |
|
|
119
|
+
| [`src/bin.ts`](src/bin.ts) | The `pnpm run mock:llm` source entry |
|
|
120
|
+
| — | No runtime invariant companion is published; this standalone test server owns no Cordis event stream or shared data; its wire behavior and lifecycle are exercised through direct HTTP and assembled-loop tests. |
|
|
121
|
+
|
|
122
|
+
### Wire flow
|
|
123
|
+
|
|
124
|
+
A request enters the handler, is validated, and selects a behavior: a concrete script entry runs directly, `random` draws one, and an exhausted script reports `script_exhausted` as a structured 500. `runBehavior` then executes the wire result — socket destroy, SSE stream, JSON error, or completion — while every request and outcome is recorded in arrival order on the returned handle for test assertions. `close()` stops accepting requests and force-terminates stalled connections.
|
|
125
|
+
|
|
126
|
+
</details>
|
|
127
|
+
|
|
128
|
+
-----
|
|
129
|
+
|
|
130
|
+
<a id="further-exploration"></a>
|
|
131
|
+
## Further Exploration
|
|
132
|
+
|
|
133
|
+
Read these pages when the package-level contract is not enough. They move from the fault server to the adapter contract it exercises and the keyless alternative for recorded success transcripts.
|
|
134
|
+
|
|
135
|
+
- [LLM package](../../llm/llm/README.md) — the provider stream contract and retry policy this server exercises.
|
|
136
|
+
- [llm-replay](../llm-replay/README.md) — the keyless counterpart that replays recorded success transcripts instead of faulting.
|
|
137
|
+
- [Testing policy](../../../docs/testing.md) — the coverage tiers and recovery tests this server serves.
|
|
138
|
+
- [Test-support group map](../README.md) — sibling harnesses and support packages.
|
|
139
|
+
|
|
140
|
+
-----
|
|
141
|
+
|
|
142
|
+
<a id="model-experience"></a>
|
|
143
|
+
## Model Experience
|
|
144
|
+
|
|
145
|
+
None, as this test server substitutes provider wire behavior without invoking a real model.
|
|
146
|
+
|
|
147
|
+
#### KV Cache effect
|
|
148
|
+
|
|
149
|
+
None; requests terminate locally and never reach a provider cache.
|
|
150
|
+
|
|
151
|
+
## Known Limitations and Deferred Work
|
|
152
|
+
|
|
153
|
+
<a id="known-limitations-and-deferred-work"></a>
|
|
154
|
+
|
|
155
|
+
|
|
156
|
+
These limits define when the server needs special care. They are current package constraints, not a task backlog.
|
|
157
|
+
|
|
158
|
+
- **Random weights model test pressure, not production incidence** — callers that want an environment-specific distribution must provide measured weights and record the emitted seed.
|
|
159
|
+
- **Request scripts are arrival-ordered** — concurrent callers share one cursor, so deterministic per-session fault assignment requires separate server instances.
|
|
160
|
+
- **True connection refusal is a listener lifecycle phase** — the CLI delay must overlap the client attempt; request-level random selection can only reset an accepted connection.
|
|
161
|
+
|
|
162
|
+
<a id="dev-note"></a>
|
|
163
|
+
### Dev Note
|
|
164
|
+
|
|
165
|
+
<details>
|
|
166
|
+
<summary>Working context for maintainers — click to expand</summary>
|
|
167
|
+
|
|
168
|
+
None.
|
|
169
|
+
|
|
170
|
+
</details>
|
package/README.zh.md
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "用于在无提供方密钥的情况下测试 LLM(大语言模型)适配器与恢复策略的可通过脚本控制的 OpenAI 兼容故障服务器,面向测试作者与演示。"
|
|
3
|
+
kind: "package-library"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# @x1a0f3n9/dsh-llm-mock-server
|
|
7
|
+
|
|
8
|
+
[English](README.md) | 中文
|
|
9
|
+
|
|
10
|
+
## 概述
|
|
11
|
+
|
|
12
|
+
本包为测试与演示提供可编脚本的 OpenAI 兼容 HTTP/SSE(Server-Sent Events)端点,使其无需提供方密钥即可检验模型提供方的失败与成功。每个已接受的 `/chat/completions` 请求依次消费下一个脚本行为,包括重置、停滞、畸形分片、限流、服务器错误、补全与工具调用。测试作者可以通过 `pnpm run mock:llm` 运行服务器,也可以调用 `startMockLlmServer`,后者会返回捕获的请求供断言使用。带种子的 `random` 行为支持可复现的混合故障压力运行。
|
|
13
|
+
|
|
14
|
+
## 目录
|
|
15
|
+
|
|
16
|
+
- [使用本包](#use-this-package)
|
|
17
|
+
- [理解实现](#understand-the-implementation)
|
|
18
|
+
- [进一步探索](#further-exploration)
|
|
19
|
+
- [模型体验](#model-experience)
|
|
20
|
+
- [已知限制与延期工作](#known-limitations-and-deferred-work)
|
|
21
|
+
- [开发备注](#dev-note)
|
|
22
|
+
|
|
23
|
+
-----
|
|
24
|
+
|
|
25
|
+
<a id="use-this-package"></a>
|
|
26
|
+
## 使用本包
|
|
27
|
+
|
|
28
|
+
本包让测试或演示无需实际提供方即可使用提供方协议进行通信:启动服务器,脚本化你想检验的协议行为,然后把真实 LLM 适配器指向它的 base URL。
|
|
29
|
+
|
|
30
|
+
### 独立运行
|
|
31
|
+
|
|
32
|
+
从本仓库运行源入口:
|
|
33
|
+
|
|
34
|
+
```sh
|
|
35
|
+
pnpm run mock:llm \
|
|
36
|
+
--port 8000 \
|
|
37
|
+
--api-key mock-key \
|
|
38
|
+
--sequence partial_disconnect,success \
|
|
39
|
+
--partial-text "discard this half"
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
将发布的 DeepSeek 适配器指向服务器;它会将 `/chat/completions` 追加到已配置 base:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
DEEPSEEK_BASE_URL=http://127.0.0.1:8000/v1 \
|
|
46
|
+
DEEPSEEK_API_KEY=mock-key \
|
|
47
|
+
pnpm dsh --profile headless "test provider recovery"
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
仓库脚本将 JSONL 写入 stdout:`ready` 记录携带以 `/v1` 结尾的 base URL 与随机种子,后续请求/结果记录同时命名脚本行为与实际选中的具体行为。本包不公开可安装的二进制命令。
|
|
51
|
+
|
|
52
|
+
### 脚本化行为
|
|
53
|
+
|
|
54
|
+
`--sequence` 是逗号分隔的 FIFO。耗尽时返回结构化 HTTP 500;`--repeat-last` 显式重用最后一项。
|
|
55
|
+
|
|
56
|
+
| 行为 | 协议结果 |
|
|
57
|
+
|---|---|
|
|
58
|
+
| `connection_reset` | 在发送 HTTP 标头前销毁 socket |
|
|
59
|
+
| `stream_disconnect` | 发送 SSE 标头,然后在第一个事件前重置连接 |
|
|
60
|
+
| `partial_disconnect` | 发送文本增量,然后重置 socket |
|
|
61
|
+
| `stall` | 发送 SSE 标头,并保持空闲,直到客户端/服务器取消 |
|
|
62
|
+
| `empty` | 发送有效的无内容 stop 和 `[DONE]` |
|
|
63
|
+
| `empty_body` / `stream_eof` / `partial_eof` | 正常结束,但缺少必需的 `[DONE]` 边界 |
|
|
64
|
+
| `malformed_json` / `malformed_event` | 发送无效 SSE JSON 或无效提供方分片形态 |
|
|
65
|
+
| `rate_limit` / `server_error` / `service_unavailable` | 返回面向重试的 429/500/503 JSON 错误 |
|
|
66
|
+
| `auth_error` / `invalid_request` / `context_overflow` / `quota_exceeded` | 返回终止性错误或需要单独恢复的提供方错误 |
|
|
67
|
+
| `success` / `slow_success` / `reasoning_success` | 流式发送完整文本响应,可选延迟或先发送推理(reasoning) |
|
|
68
|
+
| `tool_call_success` / `max_tokens` | 以工具调用或结束原因 `length` 完成 |
|
|
69
|
+
| `wrong_content_type` | 以 `application/json` 内容类型发送有效 SSE 正文 |
|
|
70
|
+
| `random` | 按带权重的种子随机选择具体请求行为 |
|
|
71
|
+
|
|
72
|
+
`connection_refused` 只能在 CLI 中使用,且必须是第一个条目。它会延迟绑定调用方指定的非零端口,因此 `--listen-delay-ms` 期间的请求会收到真实 TCP 拒绝;其余条目在 listener 启动后开始。
|
|
73
|
+
|
|
74
|
+
### 随机模式
|
|
75
|
+
|
|
76
|
+
使用重复 `random` 条目执行开放式混合运行:
|
|
77
|
+
|
|
78
|
+
```sh
|
|
79
|
+
pnpm run mock:llm \
|
|
80
|
+
--port 8000 \
|
|
81
|
+
--sequence random \
|
|
82
|
+
--repeat-last \
|
|
83
|
+
--seed 42 \
|
|
84
|
+
--random-weights 'success=60,slow_success=10,connection_reset=5,stream_disconnect=5,partial_disconnect=10,empty=5,server_error=5'
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
省略 `--seed` 会生成种子,并在 `ready` 记录中打印。`--random-weights` 接受非负的相对 `behavior=weight` 条目,并要求至少一个正权重具体行为。导出默认值是一个成功占主导的压力分布,包含 reset、disconnect、部分输出、空完成、stall、429/5xx、干净截断与格式错误的 JSON;它用于施加测试压力,而非估计生产事故频率。`connection_refused` 被排除,因为已绑定的请求处理器无法产生真实拒绝。随机权重包含 `stall` 时,为待测客户端配置较短的流空闲超时,使场景及时结束。
|
|
88
|
+
|
|
89
|
+
### 时序与内容控制
|
|
90
|
+
|
|
91
|
+
CLI 公开 `--success-text`、`--partial-text`、`--reasoning-text`、`--chunk-size`、`--chunk-delay-ms`、`--disconnect-delay-ms`、`--retry-after-ms`、`--request-id`、`--tool-name` 与 `--tool-arguments`。毫秒延迟是 Node 定时器范围内的有界整数;`retryAfterMs` 还必须为正数。库接受相同的 camel-case 选项。可选的 `apiKey` 会精确验证 `Authorization: Bearer <token>`;省略时接受任何 token。
|
|
92
|
+
|
|
93
|
+
### 可能出什么问题
|
|
94
|
+
|
|
95
|
+
- **脚本耗尽**——耗尽时返回结构化 HTTP 500;当一次运行需要更多请求时设置 `--repeat-last` 或加长序列。
|
|
96
|
+
- **没有正权重具体行为的随机权重会被拒绝**——每个条目都必须命名现有行为,且至少一个条目带正权重。
|
|
97
|
+
- **无效请求不消费脚本**——错误方法、路径、Bearer token 与畸形 JSON 会收到普通 4xx 响应,因此配置错误的客户端可能耗尽重试却不推进序列。
|
|
98
|
+
|
|
99
|
+
-----
|
|
100
|
+
|
|
101
|
+
<a id="understand-the-implementation"></a>
|
|
102
|
+
## 理解实现
|
|
103
|
+
|
|
104
|
+
<details>
|
|
105
|
+
<summary>实现细节——点击展开</summary>
|
|
106
|
+
|
|
107
|
+
本节解释服务器的设计;可观察行为已在[使用本包](#use-this-package)中完整说明。
|
|
108
|
+
|
|
109
|
+
### 设计
|
|
110
|
+
|
|
111
|
+
服务器建立在一个规则之上:每个已接受的 chat-completions 请求从按到达顺序排列的 FIFO 游标消费恰好一个行为,服务器从不重试或解读 harness 策略。校验先于游标推进——只有 `POST` 且路径以 `/chat/completions` 结尾、配置密钥时携带有效 Bearer token、且 JSON 正文可解析的请求才消费脚本;其余请求都收到普通 4xx。`random` 条目在请求时通过带种子的 PRNG 按配置权重解析,因此一次运行可由其打印出的种子复现。
|
|
112
|
+
|
|
113
|
+
### 源码地图
|
|
114
|
+
|
|
115
|
+
| 文件 | 职责 |
|
|
116
|
+
|---|---|
|
|
117
|
+
| [`src/index.ts`](src/index.ts) | `startMockLlmServer`:listener、行为表、种子随机、遥测(telemetry)、捕获的请求记录 |
|
|
118
|
+
| [`src/cli.ts`](src/cli.ts) | `--sequence` 与时序/内容选项解析、JSONL stdout 遥测 |
|
|
119
|
+
| [`src/bin.ts`](src/bin.ts) | `pnpm run mock:llm` 源入口 |
|
|
120
|
+
| — | 不发布运行时不变式伴生组件;该独立测试服务器不拥有 Cordis 事件流或共享数据;其协议行为和生命周期通过直接 HTTP 测试及组装后的循环测试进行检验。 |
|
|
121
|
+
|
|
122
|
+
### 协议流程
|
|
123
|
+
|
|
124
|
+
请求进入处理器、通过校验,然后选择行为:具体脚本条目直接运行,`random` 抽取一个,已耗尽脚本则以结构化 500 报告 `script_exhausted`。随后 `runBehavior` 执行协议结果——销毁 socket、SSE 流、JSON 错误或补全——同时每个请求与结果按到达顺序记录到返回的句柄上,供测试断言。`close()` 停止接受请求并强制终止停滞连接。
|
|
125
|
+
|
|
126
|
+
</details>
|
|
127
|
+
|
|
128
|
+
-----
|
|
129
|
+
|
|
130
|
+
<a id="further-exploration"></a>
|
|
131
|
+
## 进一步探索
|
|
132
|
+
|
|
133
|
+
当包级约定不够用时阅读以下页面。它们从故障服务器逐步进入它所检验的适配器约定,以及用于已记录成功 transcript(文本记录)的无密钥替代方案。
|
|
134
|
+
|
|
135
|
+
- [LLM 包](../../llm/llm/README.zh.md)——本服务器所检验的提供方流约定与重试策略。
|
|
136
|
+
- [llm-replay](../llm-replay/README.zh.md)——回放已记录成功 transcript 而非制造故障的无密钥替代方案。
|
|
137
|
+
- [测试策略](../../../docs/testing.zh.md)——本服务器服务的覆盖层级与恢复测试。
|
|
138
|
+
- [test-support 组地图](../README.zh.md)——兄弟 harness 与支持包。
|
|
139
|
+
|
|
140
|
+
-----
|
|
141
|
+
|
|
142
|
+
<a id="model-experience"></a>
|
|
143
|
+
## 模型体验
|
|
144
|
+
|
|
145
|
+
无。该测试服务器替代提供方协议行为,而不调用真实模型。
|
|
146
|
+
|
|
147
|
+
#### KV Cache 影响
|
|
148
|
+
|
|
149
|
+
无;请求在本地终止,绝不会到达提供方缓存。
|
|
150
|
+
|
|
151
|
+
## 已知限制与延期工作
|
|
152
|
+
|
|
153
|
+
<a id="known-limitations-and-deferred-work"></a>
|
|
154
|
+
|
|
155
|
+
|
|
156
|
+
这些限制说明何时需要对该服务器特别小心。它们是当前包约束,不是任务积压。
|
|
157
|
+
|
|
158
|
+
- **随机权重建模测试压力,而非生产事故频率**——需要环境专用分布的调用方必须提供已测量权重,并记录发出的种子。
|
|
159
|
+
- **请求脚本按到达顺序执行**——并发调用方共享一个游标,因此确定性的每会话故障分配需要独立服务器实例。
|
|
160
|
+
- **真实连接拒绝发生在监听器生命周期阶段**——CLI 延迟必须与客户端尝试重叠;请求级随机选择只能重置已接受的连接。
|
|
161
|
+
|
|
162
|
+
<a id="dev-note"></a>
|
|
163
|
+
### 开发备注
|
|
164
|
+
|
|
165
|
+
<details>
|
|
166
|
+
<summary>维护者的工作上下文——点击展开</summary>
|
|
167
|
+
|
|
168
|
+
无。
|
|
169
|
+
|
|
170
|
+
</details>
|
package/lib/index.js
ADDED
|
@@ -0,0 +1,520 @@
|
|
|
1
|
+
import { createServer } from "node:http";
|
|
2
|
+
import { randomBytes } from "node:crypto";
|
|
3
|
+
import { isIP } from "node:net";
|
|
4
|
+
import { setTimeout } from "node:timers/promises";
|
|
5
|
+
//#region lib/types/index.js
|
|
6
|
+
/**
|
|
7
|
+
* Scriptable OpenAI-compatible HTTP/SSE server for transport, protocol, and
|
|
8
|
+
* semantic-empty LLM recovery tests. Each accepted chat-completions request
|
|
9
|
+
* consumes one behavior; the server never retries or interprets harness policy.
|
|
10
|
+
*
|
|
11
|
+
* @module @x1a0f3n9/dsh-llm-mock-server
|
|
12
|
+
*/
|
|
13
|
+
/** Request-scoped behaviors accepted by {@link startMockLlmServer}. */
|
|
14
|
+
const MOCK_LLM_BEHAVIORS = [
|
|
15
|
+
"connection_reset",
|
|
16
|
+
"stream_disconnect",
|
|
17
|
+
"empty",
|
|
18
|
+
"empty_body",
|
|
19
|
+
"stream_eof",
|
|
20
|
+
"partial_eof",
|
|
21
|
+
"partial_disconnect",
|
|
22
|
+
"stall",
|
|
23
|
+
"malformed_json",
|
|
24
|
+
"malformed_event",
|
|
25
|
+
"wrong_content_type",
|
|
26
|
+
"rate_limit",
|
|
27
|
+
"server_error",
|
|
28
|
+
"service_unavailable",
|
|
29
|
+
"auth_error",
|
|
30
|
+
"invalid_request",
|
|
31
|
+
"context_overflow",
|
|
32
|
+
"quota_exceeded",
|
|
33
|
+
"success",
|
|
34
|
+
"reasoning_success",
|
|
35
|
+
"tool_call_success",
|
|
36
|
+
"max_tokens",
|
|
37
|
+
"slow_success",
|
|
38
|
+
"random"
|
|
39
|
+
];
|
|
40
|
+
/**
|
|
41
|
+
* Default stress profile for `random`. Weights are configurable test pressure,
|
|
42
|
+
* not a claim about production incident frequency.
|
|
43
|
+
*/
|
|
44
|
+
const DEFAULT_MOCK_LLM_RANDOM_WEIGHTS = Object.freeze({
|
|
45
|
+
success: 48,
|
|
46
|
+
slow_success: 10,
|
|
47
|
+
max_tokens: 2,
|
|
48
|
+
connection_reset: 5,
|
|
49
|
+
stream_disconnect: 5,
|
|
50
|
+
partial_disconnect: 10,
|
|
51
|
+
empty: 5,
|
|
52
|
+
stall: 2,
|
|
53
|
+
rate_limit: 5,
|
|
54
|
+
server_error: 4,
|
|
55
|
+
service_unavailable: 2,
|
|
56
|
+
partial_eof: 1,
|
|
57
|
+
malformed_json: 1
|
|
58
|
+
});
|
|
59
|
+
/** Largest millisecond delay accepted by Node timers without truncation. */
|
|
60
|
+
const MAX_MOCK_LLM_TIMER_DELAY_MS = 2147483647;
|
|
61
|
+
const DEFAULT_SUCCESS_TEXT = "mock response recovered";
|
|
62
|
+
const DEFAULT_PARTIAL_TEXT = "discarded partial response";
|
|
63
|
+
const DEFAULT_REASONING_TEXT = "mock reasoning";
|
|
64
|
+
const CONCRETE_BEHAVIORS = new Set(MOCK_LLM_BEHAVIORS.filter((behavior) => behavior !== "random"));
|
|
65
|
+
function boundedInteger(name, value, min, max) {
|
|
66
|
+
if (!Number.isInteger(value) || value < min || value > max) throw new Error(`llm-mock-server: ${name} must be an integer between ${min} and ${max}`);
|
|
67
|
+
return value;
|
|
68
|
+
}
|
|
69
|
+
function resolveOptions(options) {
|
|
70
|
+
const host = options.host ?? "127.0.0.1";
|
|
71
|
+
const port = boundedInteger("port", options.port ?? 0, 0, 65535);
|
|
72
|
+
const chunkSize = boundedInteger("chunkSize", options.chunkSize ?? 8, 1, Number.MAX_SAFE_INTEGER);
|
|
73
|
+
const chunkDelayMs = boundedInteger("chunkDelayMs", options.chunkDelayMs ?? 25, 0, MAX_MOCK_LLM_TIMER_DELAY_MS);
|
|
74
|
+
const disconnectDelayMs = boundedInteger("disconnectDelayMs", options.disconnectDelayMs ?? 10, 0, MAX_MOCK_LLM_TIMER_DELAY_MS);
|
|
75
|
+
const retryAfterMs = boundedInteger("retryAfterMs", options.retryAfterMs ?? 1e3, 1, MAX_MOCK_LLM_TIMER_DELAY_MS);
|
|
76
|
+
const randomSeed = boundedInteger("randomSeed", options.randomSeed ?? randomBytes(4).readUInt32LE(0), 0, 4294967295);
|
|
77
|
+
const successText = options.successText ?? DEFAULT_SUCCESS_TEXT;
|
|
78
|
+
const partialText = options.partialText ?? DEFAULT_PARTIAL_TEXT;
|
|
79
|
+
const reasoningText = options.reasoningText ?? DEFAULT_REASONING_TEXT;
|
|
80
|
+
const toolName = options.toolName ?? "mock_tool";
|
|
81
|
+
const toolArguments = options.toolArguments ?? "{\"value\":\"mock\"}";
|
|
82
|
+
if (host.length === 0) throw new Error("llm-mock-server: host must not be empty");
|
|
83
|
+
if (options.sequence.length === 0) throw new Error("llm-mock-server: sequence must not be empty");
|
|
84
|
+
const lastBehavior = options.sequence.reduce((_previous, behavior) => behavior);
|
|
85
|
+
if (options.apiKey === "") throw new Error("llm-mock-server: apiKey must not be empty");
|
|
86
|
+
if (successText.length === 0) throw new Error("llm-mock-server: successText must not be empty");
|
|
87
|
+
if (partialText.length === 0) throw new Error("llm-mock-server: partialText must not be empty");
|
|
88
|
+
if (reasoningText.length === 0) throw new Error("llm-mock-server: reasoningText must not be empty");
|
|
89
|
+
if (toolName.length === 0) throw new Error("llm-mock-server: toolName must not be empty");
|
|
90
|
+
if (options.requestId === "") throw new Error("llm-mock-server: requestId must not be empty");
|
|
91
|
+
try {
|
|
92
|
+
JSON.parse(toolArguments);
|
|
93
|
+
} catch {
|
|
94
|
+
throw new Error("llm-mock-server: toolArguments must be valid JSON");
|
|
95
|
+
}
|
|
96
|
+
const configuredWeights = options.randomWeights ?? DEFAULT_MOCK_LLM_RANDOM_WEIGHTS;
|
|
97
|
+
const randomWeights = [];
|
|
98
|
+
for (const [behavior, weight] of Object.entries(configuredWeights)) {
|
|
99
|
+
if (!CONCRETE_BEHAVIORS.has(behavior)) throw new Error(`llm-mock-server: randomWeights contains unknown concrete behavior ${JSON.stringify(behavior)}`);
|
|
100
|
+
if (!Number.isFinite(weight) || weight < 0) throw new Error(`llm-mock-server: random weight for ${behavior} must be a non-negative finite number`);
|
|
101
|
+
if (weight > 0) randomWeights.push([behavior, weight]);
|
|
102
|
+
}
|
|
103
|
+
if (randomWeights.length === 0) throw new Error("llm-mock-server: randomWeights must contain at least one positive weight");
|
|
104
|
+
return {
|
|
105
|
+
host,
|
|
106
|
+
port,
|
|
107
|
+
...options.apiKey === void 0 ? {} : { apiKey: options.apiKey },
|
|
108
|
+
sequence: [...options.sequence],
|
|
109
|
+
lastBehavior,
|
|
110
|
+
repeatLast: options.repeatLast ?? false,
|
|
111
|
+
randomSeed,
|
|
112
|
+
randomWeights,
|
|
113
|
+
successText,
|
|
114
|
+
partialText,
|
|
115
|
+
reasoningText,
|
|
116
|
+
chunkSize,
|
|
117
|
+
chunkDelayMs,
|
|
118
|
+
disconnectDelayMs,
|
|
119
|
+
retryAfterMs,
|
|
120
|
+
...options.requestId === void 0 ? {} : { requestId: options.requestId },
|
|
121
|
+
toolName,
|
|
122
|
+
toolArguments,
|
|
123
|
+
...options.onEvent === void 0 ? {} : { onEvent: options.onEvent }
|
|
124
|
+
};
|
|
125
|
+
}
|
|
126
|
+
function emit(options, event) {
|
|
127
|
+
try {
|
|
128
|
+
options.onEvent?.(Object.freeze(event));
|
|
129
|
+
} catch (_telemetryObserverFailure) {}
|
|
130
|
+
}
|
|
131
|
+
async function readJsonBody(request) {
|
|
132
|
+
const chunks = [];
|
|
133
|
+
for await (const chunk of request) chunks.push(Buffer.from(chunk));
|
|
134
|
+
const body = Buffer.concat(chunks).toString("utf8");
|
|
135
|
+
return body.length === 0 ? void 0 : JSON.parse(body);
|
|
136
|
+
}
|
|
137
|
+
function splitText(text, size) {
|
|
138
|
+
const points = Array.from(text);
|
|
139
|
+
const chunks = [];
|
|
140
|
+
for (let index = 0; index < points.length; index += size) chunks.push(points.slice(index, index + size).join(""));
|
|
141
|
+
return chunks;
|
|
142
|
+
}
|
|
143
|
+
function openSse(response, contentType = "text/event-stream; charset=utf-8") {
|
|
144
|
+
response.writeHead(200, {
|
|
145
|
+
"content-type": contentType,
|
|
146
|
+
"cache-control": "no-cache",
|
|
147
|
+
"connection": "keep-alive"
|
|
148
|
+
});
|
|
149
|
+
response.flushHeaders();
|
|
150
|
+
}
|
|
151
|
+
function writeSse(record, response, payload) {
|
|
152
|
+
response.write(`data: ${typeof payload === "string" ? payload : JSON.stringify(payload)}\n\n`);
|
|
153
|
+
record.chunksSent += 1;
|
|
154
|
+
}
|
|
155
|
+
function writeDone(record, response) {
|
|
156
|
+
writeSse(record, response, "[DONE]");
|
|
157
|
+
}
|
|
158
|
+
function finishRecord(options, record, outcome) {
|
|
159
|
+
if (record.outcome !== void 0) return;
|
|
160
|
+
record.outcome = outcome;
|
|
161
|
+
emit(options, {
|
|
162
|
+
type: "result",
|
|
163
|
+
attempt: record.attempt,
|
|
164
|
+
scriptBehavior: record.scriptBehavior,
|
|
165
|
+
behavior: record.behavior,
|
|
166
|
+
outcome,
|
|
167
|
+
chunksSent: record.chunksSent
|
|
168
|
+
});
|
|
169
|
+
}
|
|
170
|
+
function httpError(options, record, response, status, message, code, type = "mock_error") {
|
|
171
|
+
const headers = { "content-type": "application/json" };
|
|
172
|
+
if (record.behavior === "rate_limit") headers["retry-after"] = String(Math.ceil(options.retryAfterMs / 1e3));
|
|
173
|
+
if (options.requestId !== void 0) headers["x-request-id"] = options.requestId;
|
|
174
|
+
response.writeHead(status, headers);
|
|
175
|
+
response.end(JSON.stringify({ error: {
|
|
176
|
+
message,
|
|
177
|
+
type,
|
|
178
|
+
code
|
|
179
|
+
} }));
|
|
180
|
+
finishRecord(options, record, "completed");
|
|
181
|
+
}
|
|
182
|
+
function terminalChunk(reason, outputTokens) {
|
|
183
|
+
return {
|
|
184
|
+
choices: [{
|
|
185
|
+
index: 0,
|
|
186
|
+
delta: { content: "" },
|
|
187
|
+
finish_reason: reason
|
|
188
|
+
}],
|
|
189
|
+
usage: {
|
|
190
|
+
prompt_tokens: 3,
|
|
191
|
+
completion_tokens: outputTokens
|
|
192
|
+
}
|
|
193
|
+
};
|
|
194
|
+
}
|
|
195
|
+
async function pause(milliseconds, response) {
|
|
196
|
+
if (milliseconds === 0) return !response.destroyed;
|
|
197
|
+
const controller = new AbortController();
|
|
198
|
+
const stop = () => {
|
|
199
|
+
controller.abort();
|
|
200
|
+
};
|
|
201
|
+
response.once("close", stop);
|
|
202
|
+
try {
|
|
203
|
+
await setTimeout(milliseconds, void 0, { signal: controller.signal });
|
|
204
|
+
return true;
|
|
205
|
+
} catch (_responseClosed) {
|
|
206
|
+
return false;
|
|
207
|
+
} finally {
|
|
208
|
+
response.off("close", stop);
|
|
209
|
+
}
|
|
210
|
+
}
|
|
211
|
+
async function streamText(options, record, response, text, delayMs) {
|
|
212
|
+
for (const chunk of splitText(text, options.chunkSize)) {
|
|
213
|
+
writeSse(record, response, { choices: [{
|
|
214
|
+
index: 0,
|
|
215
|
+
delta: { content: chunk },
|
|
216
|
+
finish_reason: null
|
|
217
|
+
}] });
|
|
218
|
+
if (!await pause(delayMs, response)) return false;
|
|
219
|
+
}
|
|
220
|
+
return true;
|
|
221
|
+
}
|
|
222
|
+
async function completeText(options, record, response, reason, delayMs) {
|
|
223
|
+
if (!await streamText(options, record, response, options.successText, delayMs)) {
|
|
224
|
+
finishRecord(options, record, "client_closed");
|
|
225
|
+
return;
|
|
226
|
+
}
|
|
227
|
+
writeSse(record, response, terminalChunk(reason, Array.from(options.successText).length));
|
|
228
|
+
writeDone(record, response);
|
|
229
|
+
response.end();
|
|
230
|
+
finishRecord(options, record, "completed");
|
|
231
|
+
}
|
|
232
|
+
async function disconnect(options, record, response) {
|
|
233
|
+
if (!await pause(options.disconnectDelayMs, response)) {
|
|
234
|
+
finishRecord(options, record, "client_closed");
|
|
235
|
+
return;
|
|
236
|
+
}
|
|
237
|
+
finishRecord(options, record, "reset");
|
|
238
|
+
response.destroy();
|
|
239
|
+
}
|
|
240
|
+
function toolCallChunks(options) {
|
|
241
|
+
const midpoint = Math.max(1, Math.floor(options.toolArguments.length / 2));
|
|
242
|
+
return [{ choices: [{
|
|
243
|
+
index: 0,
|
|
244
|
+
delta: { tool_calls: [{
|
|
245
|
+
index: 0,
|
|
246
|
+
id: "mock-call-1",
|
|
247
|
+
type: "function",
|
|
248
|
+
function: {
|
|
249
|
+
name: options.toolName,
|
|
250
|
+
arguments: options.toolArguments.slice(0, midpoint)
|
|
251
|
+
}
|
|
252
|
+
}] },
|
|
253
|
+
finish_reason: null
|
|
254
|
+
}] }, { choices: [{
|
|
255
|
+
index: 0,
|
|
256
|
+
delta: { tool_calls: [{
|
|
257
|
+
index: 0,
|
|
258
|
+
function: { arguments: options.toolArguments.slice(midpoint) }
|
|
259
|
+
}] },
|
|
260
|
+
finish_reason: null
|
|
261
|
+
}] }];
|
|
262
|
+
}
|
|
263
|
+
async function runBehavior(options, record, request, response) {
|
|
264
|
+
switch (record.behavior) {
|
|
265
|
+
case "script_exhausted":
|
|
266
|
+
httpError(options, record, response, 500, "mock script exhausted", "MOCK_SCRIPT_EXHAUSTED");
|
|
267
|
+
return;
|
|
268
|
+
case "connection_reset":
|
|
269
|
+
finishRecord(options, record, "reset");
|
|
270
|
+
request.socket.destroy();
|
|
271
|
+
return;
|
|
272
|
+
case "stream_disconnect":
|
|
273
|
+
openSse(response);
|
|
274
|
+
await disconnect(options, record, response);
|
|
275
|
+
return;
|
|
276
|
+
case "empty":
|
|
277
|
+
openSse(response);
|
|
278
|
+
writeSse(record, response, terminalChunk("stop", 0));
|
|
279
|
+
writeDone(record, response);
|
|
280
|
+
response.end();
|
|
281
|
+
finishRecord(options, record, "completed");
|
|
282
|
+
return;
|
|
283
|
+
case "empty_body":
|
|
284
|
+
openSse(response);
|
|
285
|
+
response.end();
|
|
286
|
+
finishRecord(options, record, "completed");
|
|
287
|
+
return;
|
|
288
|
+
case "stream_eof":
|
|
289
|
+
openSse(response);
|
|
290
|
+
writeSse(record, response, { choices: [{
|
|
291
|
+
index: 0,
|
|
292
|
+
delta: { role: "assistant" },
|
|
293
|
+
finish_reason: null
|
|
294
|
+
}] });
|
|
295
|
+
response.end();
|
|
296
|
+
finishRecord(options, record, "completed");
|
|
297
|
+
return;
|
|
298
|
+
case "partial_eof":
|
|
299
|
+
openSse(response);
|
|
300
|
+
await streamText(options, record, response, options.partialText, 0);
|
|
301
|
+
response.end();
|
|
302
|
+
finishRecord(options, record, "completed");
|
|
303
|
+
return;
|
|
304
|
+
case "partial_disconnect":
|
|
305
|
+
openSse(response);
|
|
306
|
+
if (!await streamText(options, record, response, options.partialText, options.chunkDelayMs)) return;
|
|
307
|
+
await disconnect(options, record, response);
|
|
308
|
+
return;
|
|
309
|
+
case "stall":
|
|
310
|
+
openSse(response);
|
|
311
|
+
finishRecord(options, record, "stalled");
|
|
312
|
+
return;
|
|
313
|
+
case "malformed_json":
|
|
314
|
+
openSse(response);
|
|
315
|
+
writeSse(record, response, "{not-json");
|
|
316
|
+
writeDone(record, response);
|
|
317
|
+
response.end();
|
|
318
|
+
finishRecord(options, record, "completed");
|
|
319
|
+
return;
|
|
320
|
+
case "malformed_event":
|
|
321
|
+
openSse(response);
|
|
322
|
+
writeSse(record, response, { choices: [null] });
|
|
323
|
+
writeDone(record, response);
|
|
324
|
+
response.end();
|
|
325
|
+
finishRecord(options, record, "completed");
|
|
326
|
+
return;
|
|
327
|
+
case "wrong_content_type":
|
|
328
|
+
openSse(response, "application/json");
|
|
329
|
+
await completeText(options, record, response, "stop", 0);
|
|
330
|
+
return;
|
|
331
|
+
case "rate_limit":
|
|
332
|
+
httpError(options, record, response, 429, "mock rate limit", "rate_limit");
|
|
333
|
+
return;
|
|
334
|
+
case "server_error":
|
|
335
|
+
httpError(options, record, response, 500, "mock server error", "server_error");
|
|
336
|
+
return;
|
|
337
|
+
case "service_unavailable":
|
|
338
|
+
httpError(options, record, response, 503, "mock service unavailable", "service_unavailable");
|
|
339
|
+
return;
|
|
340
|
+
case "auth_error":
|
|
341
|
+
httpError(options, record, response, 401, "mock authentication failed", "invalid_api_key");
|
|
342
|
+
return;
|
|
343
|
+
case "invalid_request":
|
|
344
|
+
httpError(options, record, response, 400, "mock invalid request", "invalid_request");
|
|
345
|
+
return;
|
|
346
|
+
case "context_overflow":
|
|
347
|
+
httpError(options, record, response, 400, "mock input exceeds the model context window", "context_length_exceeded", "invalid_request_error");
|
|
348
|
+
return;
|
|
349
|
+
case "quota_exceeded":
|
|
350
|
+
httpError(options, record, response, 429, "mock insufficient quota", "insufficient_quota");
|
|
351
|
+
return;
|
|
352
|
+
case "success":
|
|
353
|
+
openSse(response);
|
|
354
|
+
await completeText(options, record, response, "stop", 0);
|
|
355
|
+
return;
|
|
356
|
+
case "reasoning_success":
|
|
357
|
+
openSse(response);
|
|
358
|
+
for (const chunk of splitText(options.reasoningText, options.chunkSize)) writeSse(record, response, { choices: [{
|
|
359
|
+
index: 0,
|
|
360
|
+
delta: { reasoning_content: chunk },
|
|
361
|
+
finish_reason: null
|
|
362
|
+
}] });
|
|
363
|
+
await completeText(options, record, response, "stop", 0);
|
|
364
|
+
return;
|
|
365
|
+
case "tool_call_success":
|
|
366
|
+
openSse(response);
|
|
367
|
+
for (const chunk of toolCallChunks(options)) writeSse(record, response, chunk);
|
|
368
|
+
writeSse(record, response, terminalChunk("tool_calls", 2));
|
|
369
|
+
writeDone(record, response);
|
|
370
|
+
response.end();
|
|
371
|
+
finishRecord(options, record, "completed");
|
|
372
|
+
return;
|
|
373
|
+
case "max_tokens":
|
|
374
|
+
openSse(response);
|
|
375
|
+
await completeText(options, record, response, "length", 0);
|
|
376
|
+
return;
|
|
377
|
+
case "slow_success":
|
|
378
|
+
openSse(response);
|
|
379
|
+
await completeText(options, record, response, "stop", options.chunkDelayMs);
|
|
380
|
+
return;
|
|
381
|
+
}
|
|
382
|
+
}
|
|
383
|
+
function seededRandom(seed) {
|
|
384
|
+
let state = seed;
|
|
385
|
+
return () => {
|
|
386
|
+
state = state + 1831565813 >>> 0;
|
|
387
|
+
let mixed = state;
|
|
388
|
+
mixed = Math.imul(mixed ^ mixed >>> 15, mixed | 1);
|
|
389
|
+
mixed ^= mixed + Math.imul(mixed ^ mixed >>> 7, mixed | 61);
|
|
390
|
+
return ((mixed ^ mixed >>> 14) >>> 0) / 4294967296;
|
|
391
|
+
};
|
|
392
|
+
}
|
|
393
|
+
function chooseRandomBehavior(weights, random) {
|
|
394
|
+
const total = weights.reduce((sum, entry) => sum + entry[1], 0);
|
|
395
|
+
let draw = random() * total;
|
|
396
|
+
for (const [behavior, weight] of weights) {
|
|
397
|
+
if (draw < weight) return behavior;
|
|
398
|
+
draw -= weight;
|
|
399
|
+
}
|
|
400
|
+
/* v8 ignore next -- seededRandom is strictly less than one; this guards floating-point residue only */
|
|
401
|
+
return weights.at(-1)[0];
|
|
402
|
+
}
|
|
403
|
+
/**
|
|
404
|
+
* Start a local chat-completions server that consumes one configured behavior
|
|
405
|
+
* per accepted request. Only a `POST` path ending in `/chat/completions` consumes the script;
|
|
406
|
+
* invalid routes, methods, authorization, and JSON receive ordinary 4xx
|
|
407
|
+
* responses. Closing the handle terminates stalled connections.
|
|
408
|
+
*
|
|
409
|
+
* @param options - listener, script, response content, timing, and telemetry options.
|
|
410
|
+
* @returns the listening handle after the port is bound.
|
|
411
|
+
*/
|
|
412
|
+
async function startMockLlmServer(options) {
|
|
413
|
+
const resolved = resolveOptions(options);
|
|
414
|
+
const requests = [];
|
|
415
|
+
const random = seededRandom(resolved.randomSeed);
|
|
416
|
+
let cursor = 0;
|
|
417
|
+
const selectBehavior = () => {
|
|
418
|
+
const selected = resolved.sequence[cursor];
|
|
419
|
+
cursor += 1;
|
|
420
|
+
const scriptBehavior = selected ?? (resolved.repeatLast ? resolved.lastBehavior : "script_exhausted");
|
|
421
|
+
return {
|
|
422
|
+
scriptBehavior,
|
|
423
|
+
behavior: scriptBehavior === "random" ? chooseRandomBehavior(resolved.randomWeights, random) : scriptBehavior
|
|
424
|
+
};
|
|
425
|
+
};
|
|
426
|
+
const handle = async (request, response) => {
|
|
427
|
+
/* v8 ignore next -- node:http server requests always carry a URL despite the shared optional type */
|
|
428
|
+
const path = new URL(request.url ?? "/", "http://mock.invalid").pathname;
|
|
429
|
+
if (request.method !== "POST") {
|
|
430
|
+
response.writeHead(405, { allow: "POST" }).end();
|
|
431
|
+
return;
|
|
432
|
+
}
|
|
433
|
+
if (!path.endsWith("/chat/completions")) {
|
|
434
|
+
response.writeHead(404).end();
|
|
435
|
+
return;
|
|
436
|
+
}
|
|
437
|
+
if (resolved.apiKey !== void 0 && request.headers.authorization !== `Bearer ${resolved.apiKey}`) {
|
|
438
|
+
response.writeHead(401, { "content-type": "application/json" });
|
|
439
|
+
response.end(JSON.stringify({ error: {
|
|
440
|
+
message: "invalid mock bearer token",
|
|
441
|
+
code: "invalid_api_key"
|
|
442
|
+
} }));
|
|
443
|
+
return;
|
|
444
|
+
}
|
|
445
|
+
let body;
|
|
446
|
+
try {
|
|
447
|
+
body = await readJsonBody(request);
|
|
448
|
+
} catch {
|
|
449
|
+
response.writeHead(400, { "content-type": "application/json" });
|
|
450
|
+
response.end(JSON.stringify({ error: {
|
|
451
|
+
message: "request body must be valid JSON",
|
|
452
|
+
code: "invalid_json"
|
|
453
|
+
} }));
|
|
454
|
+
return;
|
|
455
|
+
}
|
|
456
|
+
const selected = selectBehavior();
|
|
457
|
+
const record = {
|
|
458
|
+
attempt: requests.length + 1,
|
|
459
|
+
scriptBehavior: selected.scriptBehavior,
|
|
460
|
+
behavior: selected.behavior,
|
|
461
|
+
path,
|
|
462
|
+
headers: { ...request.headers },
|
|
463
|
+
body,
|
|
464
|
+
chunksSent: 0
|
|
465
|
+
};
|
|
466
|
+
requests.push(record);
|
|
467
|
+
response.once("close", () => {
|
|
468
|
+
if (!response.writableFinished && record.outcome === void 0) finishRecord(resolved, record, "client_closed");
|
|
469
|
+
});
|
|
470
|
+
emit(resolved, {
|
|
471
|
+
type: "request",
|
|
472
|
+
attempt: record.attempt,
|
|
473
|
+
scriptBehavior: record.scriptBehavior,
|
|
474
|
+
behavior: record.behavior,
|
|
475
|
+
path
|
|
476
|
+
});
|
|
477
|
+
await runBehavior(resolved, record, request, response);
|
|
478
|
+
};
|
|
479
|
+
const server = createServer((request, response) => {
|
|
480
|
+
/* v8 ignore start -- last-resort containment for Node response failures after validated test inputs */
|
|
481
|
+
handle(request, response).catch((error) => {
|
|
482
|
+
const record = requests.at(-1);
|
|
483
|
+
if (record !== void 0) finishRecord(resolved, record, "server_error");
|
|
484
|
+
if (response.headersSent) {
|
|
485
|
+
response.destroy(error instanceof Error ? error : new Error(String(error)));
|
|
486
|
+
return;
|
|
487
|
+
}
|
|
488
|
+
response.writeHead(500, { "content-type": "application/json" });
|
|
489
|
+
response.end(JSON.stringify({ error: {
|
|
490
|
+
message: "mock server handler failed",
|
|
491
|
+
code: "MOCK_HANDLER_FAILED"
|
|
492
|
+
} }));
|
|
493
|
+
});
|
|
494
|
+
/* v8 ignore stop */
|
|
495
|
+
});
|
|
496
|
+
let closing;
|
|
497
|
+
const close = () => closing ??= new Promise((resolveClose) => {
|
|
498
|
+
server.close(() => {
|
|
499
|
+
resolveClose();
|
|
500
|
+
});
|
|
501
|
+
server.closeAllConnections();
|
|
502
|
+
});
|
|
503
|
+
await new Promise((resolveListen, rejectListen) => {
|
|
504
|
+
server.once("error", rejectListen);
|
|
505
|
+
server.listen(resolved.port, resolved.host, () => {
|
|
506
|
+
server.off("error", rejectListen);
|
|
507
|
+
resolveListen();
|
|
508
|
+
});
|
|
509
|
+
});
|
|
510
|
+
const address = server.address();
|
|
511
|
+
return {
|
|
512
|
+
baseURL: `http://${isIP(resolved.host) === 6 ? `[${resolved.host}]` : resolved.host}:${address.port}`,
|
|
513
|
+
port: address.port,
|
|
514
|
+
randomSeed: resolved.randomSeed,
|
|
515
|
+
requests,
|
|
516
|
+
close
|
|
517
|
+
};
|
|
518
|
+
}
|
|
519
|
+
//#endregion
|
|
520
|
+
export { DEFAULT_MOCK_LLM_RANDOM_WEIGHTS, MAX_MOCK_LLM_TIMER_DELAY_MS, MOCK_LLM_BEHAVIORS, startMockLlmServer };
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Dependency-free CLI parsing for the standalone mock LLM server.
|
|
3
|
+
* @module @x1a0f3n9/dsh-llm-mock-server/cli
|
|
4
|
+
*/
|
|
5
|
+
import type { MockLlmServerOptions } from './index.ts';
|
|
6
|
+
/** Listener lifecycle behavior understood only by the standalone CLI. */
|
|
7
|
+
export declare const CONNECTION_REFUSED_BEHAVIOR = "connection_refused";
|
|
8
|
+
/** Parsed CLI configuration, including a pre-listen unavailable interval. */
|
|
9
|
+
export interface MockLlmCliConfig {
|
|
10
|
+
/** Server options after removing the lifecycle-only `connection_refused` entry. */
|
|
11
|
+
readonly server: MockLlmServerOptions;
|
|
12
|
+
/** Delay before binding the model port; an integer from zero through the Node timer maximum. */
|
|
13
|
+
readonly listenDelayMs: number;
|
|
14
|
+
/** Whether the original sequence requested a true pre-listen refusal phase. */
|
|
15
|
+
readonly startsUnavailable: boolean;
|
|
16
|
+
}
|
|
17
|
+
/** Result of parsing `dsh-llm-mock-server` arguments. */
|
|
18
|
+
export type MockLlmCliParseResult = {
|
|
19
|
+
readonly kind: 'help';
|
|
20
|
+
} | {
|
|
21
|
+
readonly kind: 'run';
|
|
22
|
+
readonly config: MockLlmCliConfig;
|
|
23
|
+
};
|
|
24
|
+
/** Command usage written for `--help` and invalid arguments. */
|
|
25
|
+
export declare const MOCK_LLM_CLI_USAGE = "Usage: dsh-llm-mock-server [options]\n\nRequired:\n --sequence <a,b,...> Ordered behaviors; connection_refused is allowed first\n\nListener:\n --host <host> Default 127.0.0.1\n --port <port> Default 8000; required and nonzero for connection_refused\n --api-key <token> Validate exact Bearer token when present\n --listen-delay-ms <ms> Unavailable interval (default 750 with connection_refused)\n --repeat-last Repeat the final request behavior after exhaustion\n --seed <uint32> Reproduce random selections\n --random-weights <a=n,...> Relative weights for concrete behaviors\n\nResponse:\n --success-text <text>\n --partial-text <text>\n --reasoning-text <text>\n --chunk-size <count>\n --chunk-delay-ms <ms>\n --disconnect-delay-ms <ms>\n --retry-after-ms <ms>\n --request-id <id>\n --tool-name <name>\n --tool-arguments <json>\n\nOther:\n --help\n";
|
|
26
|
+
/**
|
|
27
|
+
* Parse standalone server arguments without starting a process or listener.
|
|
28
|
+
* Tokenizing rides `node:util` `parseArgs` (strict, no positionals); numeric
|
|
29
|
+
* coercion, bounds, and cross-option constraints remain manual below it.
|
|
30
|
+
* @param argv - arguments after the executable name.
|
|
31
|
+
* @returns help or validated run configuration.
|
|
32
|
+
*/
|
|
33
|
+
export declare function parseMockLlmCliArgs(argv: readonly string[]): MockLlmCliParseResult;
|
|
34
|
+
//# sourceMappingURL=cli.d.ts.map
|
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Scriptable OpenAI-compatible HTTP/SSE server for transport, protocol, and
|
|
3
|
+
* semantic-empty LLM recovery tests. Each accepted chat-completions request
|
|
4
|
+
* consumes one behavior; the server never retries or interprets harness policy.
|
|
5
|
+
*
|
|
6
|
+
* @module @x1a0f3n9/dsh-llm-mock-server
|
|
7
|
+
*/
|
|
8
|
+
import type { IncomingHttpHeaders } from 'node:http';
|
|
9
|
+
/** Request-scoped behaviors accepted by {@link startMockLlmServer}. */
|
|
10
|
+
export declare const MOCK_LLM_BEHAVIORS: readonly ["connection_reset", "stream_disconnect", "empty", "empty_body", "stream_eof", "partial_eof", "partial_disconnect", "stall", "malformed_json", "malformed_event", "wrong_content_type", "rate_limit", "server_error", "service_unavailable", "auth_error", "invalid_request", "context_overflow", "quota_exceeded", "success", "reasoning_success", "tool_call_success", "max_tokens", "slow_success", "random"];
|
|
11
|
+
/** One scripted mock behavior name; `random` selects a concrete behavior per request. */
|
|
12
|
+
export type MockLlmBehavior = typeof MOCK_LLM_BEHAVIORS[number];
|
|
13
|
+
/** One concrete request behavior after resolving a `random` script entry. */
|
|
14
|
+
export type ConcreteMockLlmBehavior = Exclude<MockLlmBehavior, 'random'>;
|
|
15
|
+
/** Relative non-negative weights for random request behavior selection. */
|
|
16
|
+
export type MockLlmRandomWeights = Partial<Record<ConcreteMockLlmBehavior, number>>;
|
|
17
|
+
/**
|
|
18
|
+
* Default stress profile for `random`. Weights are configurable test pressure,
|
|
19
|
+
* not a claim about production incident frequency.
|
|
20
|
+
*/
|
|
21
|
+
export declare const DEFAULT_MOCK_LLM_RANDOM_WEIGHTS: Readonly<MockLlmRandomWeights>;
|
|
22
|
+
/** Largest millisecond delay accepted by Node timers without truncation. */
|
|
23
|
+
export declare const MAX_MOCK_LLM_TIMER_DELAY_MS = 2147483647;
|
|
24
|
+
/** How one accepted request ended at the mock boundary. */
|
|
25
|
+
export type MockLlmRequestOutcome = 'completed' | 'reset' | 'stalled' | 'client_closed' | 'server_error';
|
|
26
|
+
/** Immutable telemetry emitted when a request starts or reaches an outcome. */
|
|
27
|
+
export type MockLlmServerEvent = {
|
|
28
|
+
readonly type: 'request';
|
|
29
|
+
readonly attempt: number;
|
|
30
|
+
readonly scriptBehavior: MockLlmBehavior | 'script_exhausted';
|
|
31
|
+
readonly behavior: ConcreteMockLlmBehavior | 'script_exhausted';
|
|
32
|
+
readonly path: string;
|
|
33
|
+
} | {
|
|
34
|
+
readonly type: 'result';
|
|
35
|
+
readonly attempt: number;
|
|
36
|
+
readonly scriptBehavior: MockLlmBehavior | 'script_exhausted';
|
|
37
|
+
readonly behavior: ConcreteMockLlmBehavior | 'script_exhausted';
|
|
38
|
+
readonly outcome: MockLlmRequestOutcome;
|
|
39
|
+
readonly chunksSent: number;
|
|
40
|
+
};
|
|
41
|
+
/** Captured wire request and its final server-side outcome. */
|
|
42
|
+
export interface MockLlmRequestRecord {
|
|
43
|
+
/** One-based accepted chat-completions request number. */
|
|
44
|
+
readonly attempt: number;
|
|
45
|
+
/** Script entry consumed for this request before random resolution. */
|
|
46
|
+
readonly scriptBehavior: MockLlmBehavior | 'script_exhausted';
|
|
47
|
+
/** Concrete behavior selected for this request, or exhaustion after the configured script. */
|
|
48
|
+
readonly behavior: ConcreteMockLlmBehavior | 'script_exhausted';
|
|
49
|
+
/** Original request path, including a `/v1` prefix when the client supplied one. */
|
|
50
|
+
readonly path: string;
|
|
51
|
+
/** Detached request headers. */
|
|
52
|
+
readonly headers: Readonly<IncomingHttpHeaders>;
|
|
53
|
+
/** Parsed JSON request body. */
|
|
54
|
+
readonly body: unknown;
|
|
55
|
+
/** Number of SSE `data:` events handed to Node before the outcome. */
|
|
56
|
+
chunksSent: number;
|
|
57
|
+
/** Final server-side outcome; absent while a stalled request remains open. */
|
|
58
|
+
outcome?: MockLlmRequestOutcome;
|
|
59
|
+
}
|
|
60
|
+
/** Configuration for one mock server instance. */
|
|
61
|
+
export interface MockLlmServerOptions {
|
|
62
|
+
/** Loopback host by default. */
|
|
63
|
+
readonly host?: string;
|
|
64
|
+
/** TCP port; zero requests an OS-assigned port. */
|
|
65
|
+
readonly port?: number;
|
|
66
|
+
/** Optional exact bearer token; omission accepts any authorization header. */
|
|
67
|
+
readonly apiKey?: string;
|
|
68
|
+
/** Ordered request behaviors; exhaustion fails loud unless `repeatLast` is true. */
|
|
69
|
+
readonly sequence: readonly MockLlmBehavior[];
|
|
70
|
+
/** Reuse the final behavior after the sequence is consumed. */
|
|
71
|
+
readonly repeatLast?: boolean;
|
|
72
|
+
/** Optional deterministic unsigned 32-bit seed; omission generates and exposes one. */
|
|
73
|
+
readonly randomSeed?: number;
|
|
74
|
+
/** Relative weights used whenever a script entry is `random`. */
|
|
75
|
+
readonly randomWeights?: Readonly<MockLlmRandomWeights>;
|
|
76
|
+
/** Complete text returned by success-shaped behaviors. */
|
|
77
|
+
readonly successText?: string;
|
|
78
|
+
/** Text emitted before partial EOF/reset behaviors terminate. */
|
|
79
|
+
readonly partialText?: string;
|
|
80
|
+
/** Reasoning text emitted by `reasoning_success`. */
|
|
81
|
+
readonly reasoningText?: string;
|
|
82
|
+
/** Unicode code-point count per text or reasoning SSE delta. */
|
|
83
|
+
readonly chunkSize?: number;
|
|
84
|
+
/** Inter-chunk delay for `slow_success`, in milliseconds. */
|
|
85
|
+
readonly chunkDelayMs?: number;
|
|
86
|
+
/** Delay after headers/deltas before a forced disconnect, in milliseconds. */
|
|
87
|
+
readonly disconnectDelayMs?: number;
|
|
88
|
+
/** Provider retry delay; the wire `Retry-After` value rounds up to whole seconds. */
|
|
89
|
+
readonly retryAfterMs?: number;
|
|
90
|
+
/** Optional provider request id returned on HTTP failures. */
|
|
91
|
+
readonly requestId?: string;
|
|
92
|
+
/** Tool name emitted by `tool_call_success`. */
|
|
93
|
+
readonly toolName?: string;
|
|
94
|
+
/** Raw JSON arguments emitted by `tool_call_success`. */
|
|
95
|
+
readonly toolArguments?: string;
|
|
96
|
+
/** Optional observer for JSONL CLI telemetry; observer failures never affect wire behavior. */
|
|
97
|
+
readonly onEvent?: (event: MockLlmServerEvent) => void;
|
|
98
|
+
}
|
|
99
|
+
/** Running mock server and captured request state. */
|
|
100
|
+
export interface MockLlmServer {
|
|
101
|
+
/** Base URL without `/v1`; both root and `/v1` chat-completions paths are accepted. */
|
|
102
|
+
readonly baseURL: string;
|
|
103
|
+
/** Actual bound port, including an OS-assigned value. */
|
|
104
|
+
readonly port: number;
|
|
105
|
+
/** Seed used for random behavior selection, including the generated default. */
|
|
106
|
+
readonly randomSeed: number;
|
|
107
|
+
/** Live request records in arrival order. */
|
|
108
|
+
readonly requests: readonly MockLlmRequestRecord[];
|
|
109
|
+
/** Stop accepting requests and force-close stalled/streaming connections; idempotent. */
|
|
110
|
+
close(): Promise<void>;
|
|
111
|
+
}
|
|
112
|
+
/**
|
|
113
|
+
* Start a local chat-completions server that consumes one configured behavior
|
|
114
|
+
* per accepted request. Only a `POST` path ending in `/chat/completions` consumes the script;
|
|
115
|
+
* invalid routes, methods, authorization, and JSON receive ordinary 4xx
|
|
116
|
+
* responses. Closing the handle terminates stalled connections.
|
|
117
|
+
*
|
|
118
|
+
* @param options - listener, script, response content, timing, and telemetry options.
|
|
119
|
+
* @returns the listening handle after the port is bound.
|
|
120
|
+
*/
|
|
121
|
+
export declare function startMockLlmServer(options: MockLlmServerOptions): Promise<MockLlmServer>;
|
|
122
|
+
//# sourceMappingURL=index.d.ts.map
|
package/package.json
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@x1a0f3n9/dsh-llm-mock-server",
|
|
3
|
+
"description": "Scriptable OpenAI-compatible HTTP/SSE fault server for LLM recovery tests",
|
|
4
|
+
"version": "0.1.5-rc.3",
|
|
5
|
+
"publishConfig": {
|
|
6
|
+
"access": "public"
|
|
7
|
+
},
|
|
8
|
+
"repository": {
|
|
9
|
+
"type": "git",
|
|
10
|
+
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
|
|
11
|
+
"directory": "packages/test-support/llm-mock-server"
|
|
12
|
+
},
|
|
13
|
+
"type": "module",
|
|
14
|
+
"main": "lib/index.js",
|
|
15
|
+
"types": "lib/types/index.d.ts",
|
|
16
|
+
"exports": {
|
|
17
|
+
".": {
|
|
18
|
+
"types": "./lib/types/index.d.ts",
|
|
19
|
+
"default": "./lib/index.js"
|
|
20
|
+
},
|
|
21
|
+
"./src/*": "./src/*",
|
|
22
|
+
"./package.json": "./package.json"
|
|
23
|
+
},
|
|
24
|
+
"files": [
|
|
25
|
+
"lib/index.js",
|
|
26
|
+
"lib/types/**/*.d.ts"
|
|
27
|
+
],
|
|
28
|
+
"license": "MIT",
|
|
29
|
+
"peerDependencies": {
|
|
30
|
+
"@deepseek-ai/cordis": "^4.0.2"
|
|
31
|
+
},
|
|
32
|
+
"devDependencies": {
|
|
33
|
+
"@deepseek-ai/cordis": "^4.0.2"
|
|
34
|
+
}
|
|
35
|
+
}
|