llm-switcher 1.2.6 → 1.2.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.impeccable/hook.cache.json +1 -0
- package/CHANGELOG.md +57 -0
- package/README.md +74 -4
- package/README.vi.md +75 -4
- package/blindfold/blindfold.mjs +2 -1
- package/docs/TOKEN-OPTIMIZER-INTEROP.md +110 -110
- package/docs/cross-platform.md +3 -1
- package/docs/response-matrix.json +1130 -1130
- package/hook-status.mjs +51 -0
- package/notify-route.ps1 +46 -0
- package/package.json +2 -2
- package/plugin.mjs +204 -0
- package/proxy.mjs +8 -3
- package/service.mjs +1 -1
- package/shim.mjs +28 -2
- package/skills/llm-switcher/SKILL.md +93 -93
- package/state.mjs +41 -5
- package/switch +0 -0
- package/switch.cmd +2 -2
- package/switch.mjs +53 -2
- package/tests/blindfold.test.mjs +1 -1
- package/tests/cdp.mjs +297 -0
- package/tests/dashboard-fixture.mjs +167 -0
- package/tests/dashboard.e2e.test.mjs +786 -0
- package/tests/gateway.e2e.test.mjs +1 -1
- package/tests/helpers.mjs +24 -24
- package/tests/hook-status.test.mjs +119 -0
- package/tests/live-optimizer-interop.mjs +205 -205
- package/tests/plugin.test.mjs +132 -0
- package/tests/real-user-sim.test.mjs +2 -2
- package/tests/service.test.mjs +2 -0
- package/tests/shim.test.mjs +112 -0
- package/tests/state.test.mjs +130 -5
- package/tests/ui.test.mjs +1 -1
- package/ui.html +121 -20
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":1,"sessions":{"1d508082-a9d5-487c-8568-c4ba91785a8d":{"updatedAt":1790431452442,"files":{"C:\\Users\\louis\\tools\\llm-switcher\\ui.html":{"editCount":23,"findings":[],"cleanAcked":true}},"footerShown":true},"673c932f-b92d-4af4-8899-3c8e73082f67":{"updatedAt":1790689606133,"files":{"C:\\Users\\louis\\tools\\llm-switcher\\ui.html":{"editCount":7,"findings":["bounce-easing:0:cubic-bezier(0.2, 0.9, 0.3, 1.2)","bounce-easing:0:cubic-bezier(0.3, 1.4, 0.6, 1)"],"cleanAcked":true}},"footerShown":true}}}
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,62 @@
|
|
|
1
1
|
# Changelog — LLM Switcher
|
|
2
2
|
|
|
3
|
+
## Release 1.2.8
|
|
4
|
+
|
|
5
|
+
- **Launch notice:** A person opened `claude` or `codex`, and nothing on screen said that the switcher took the traffic. The shim now raises a notice at launch. On Windows it is a desktop toast, because both tools can claim the whole screen and hide a printed line. On Linux and macOS it is one line on stderr. The notice names the profile, the host, the model, and the 1M window.
|
|
6
|
+
- **Only a routed tool:** The notice appears only while that tool has a profile. A tool on its official endpoint stays silent. `switch off codex` empties `route-codex.txt`, and the codex shim then says nothing.
|
|
7
|
+
- **New files:** `route-claude.txt` and `route-codex.txt` hold the notice text. The switcher writes each one with the env file of the same tool. `notify-route.ps1` raises the toast, and it needs no PowerShell module.
|
|
8
|
+
- **Tests:** New tests prove the state files and the notice. One test runs the real `.cmd` shim through cmd.exe with a fake `powershell` on PATH. It reads what the shim asked for.
|
|
9
|
+
- **A second notice, inside the tool:** The shim notice needs the shim directory first on `PATH`. A person who keeps their own `claude` wrapper never sees it. `switch plugin install` adds a hook that runs inside each tool instead. It reports three states: routed and the gateway answers, routed and the gateway does not answer, and not routed. The second state used to appear only as a connection error with no cause.
|
|
10
|
+
- **No configuration file is opened:** Claude Code loads `~/.claude/skills/llm-switcher-status/` as a plugin on the next session, with no marketplace and no install step. Codex reads `~/.codex/hooks.json` by itself. `settings.json` and `config.toml` stay closed. An existing `~/.codex/hooks.json` is merged, and a file that does not parse is refused.
|
|
11
|
+
- **The notice is optional:** `switch plugin install` is a choice. Install nothing and the gateway still routes every request.
|
|
12
|
+
|
|
13
|
+
### Found by pressing the dashboard like a person
|
|
14
|
+
|
|
15
|
+
A pass over the running dashboard in a real browser, with real clicks and real keys, never a value
|
|
16
|
+
set through the page's own JavaScript. Each item below has a browser test that reproduces it.
|
|
17
|
+
|
|
18
|
+
- **A narrow window hid the navigation:** At 400 px the sidebar collapsed to its padding, 16 px, and the navigation was drawn outside it. No link could be pressed. The `@media` rule set `height: auto` but kept `overflow-y: auto` from the base rule, and a grid item that scrolls has an automatic minimum size of zero. The rule now resets `overflow`.
|
|
19
|
+
- **A card stayed in the editing state:** After Escape, Cancel or a press on the dark area, the card kept its ring and its button still read "Editing Profile". `closeProfileModal` never cleared the key that the render reads.
|
|
20
|
+
- **The keyboard stayed outside the dialog:** Opening a profile dialog left the focus on the button behind the overlay, so the first Tab walked into the page under the dialog. The dialog now takes the focus when it opens.
|
|
21
|
+
- **A connection result from the profile before:** A new profile opened showing the "HTTP 200 OK" and the latency of the profile opened before it. The three result fields are emptied on every open.
|
|
22
|
+
- **A double-click on Save sent two saves:** One double-click sent two `POST /api/save-profile`. The button is disabled while the request runs, the same way the catalog sync button already was.
|
|
23
|
+
- **A model search with no match looked broken:** Every row was hidden while the headers kept the counts of the full list, and nothing said why the list was empty. The count follows the filter now, and an empty result says so in words.
|
|
24
|
+
- **Tests:** Eight new browser tests, one for each finding. Two more findings from that pass did not reproduce under a real click and are kept as guards rather than reported as bugs.
|
|
25
|
+
|
|
26
|
+
### Linux and Node 18
|
|
27
|
+
|
|
28
|
+
- **Tests on Node 18 and 20:** `engines` accepts Node 18.17 and later, but four tests failed on Node 18 and Node 20. A global `WebSocket` client arrived in Node 22, and `zlib.zstdCompressSync` arrived in Node 22.15. The runtime code already tested for zstd before it used it, so only the tests assumed the newer runtime. The three WebSocket tests and the zstd assertion now skip when the runtime does not hold those APIs.
|
|
29
|
+
- **A zstd capture on an older runtime:** `DECODERS` mapped `zstd` to `zlib.zstdDecompressSync`, which is `undefined` before Node 22.15. The capture then wrote the compressed bytes as UTF-8 noise, and that noise is what the decoder table exists to prevent. The capture now holds `[capture: cannot decode zstd body of N bytes: zstd is not supported by this Node.js version]`.
|
|
30
|
+
- **`LLM_SWITCHER_HOME` and the background service:** The README gives `LLM_SWITCHER_HOME` as the override for the data folder, but `switch service install` did not copy it into the systemd unit or the launchd plist. The service then read a different `config.json` and a different `admin.token` than the command that installed it. The variable now goes into the unit with the other three.
|
|
31
|
+
- **Mode of the data folder:** The Quick Start created the data folder with `mkdir -p`, so a common umask left it readable by the group and by everyone. `state.mjs` applies mode 700 only when it creates the folder itself. The README now uses `mkdir -p -m 700`.
|
|
32
|
+
- **Notes for the Linux service:** The unit is a systemd user service, so it starts at login and not at boot. On a headless host, run `loginctl enable-linger $USER` first. `ExecStart` holds the path of the running Node binary. After you remove that Node version, run `switch service install` again.
|
|
33
|
+
|
|
34
|
+
### The profile switch
|
|
35
|
+
|
|
36
|
+
- **A backup with the API keys on every switch:** `setTargetProfile` and `activateProfile` still wrote the legacy `activeProfile` pointer. `needsMigration` reports any configuration that holds that key, so the next load migrated the file again and wrote another `config.json.bak-*`. Each backup is a full copy of `config.json`, with every provider API key, and the backups stayed after a success. Neither function writes the pointer now. `ensureActiveMap` already folds a legacy pointer into `activeProfiles`, and a load still migrates an old file once. Four switches left three backups before this change and none after it.
|
|
37
|
+
- **A placeholder profile activated without a warning:** With the unedited `config.example.json`, `switch on` printed `[SUCCESS]` and routed to `https://YOUR-ROUTER-HOST/v1`. Only `switch doctor` gave a warning. Activation now uses the same `hasPlaceholder` test as `doctor`, and it names the file to edit. Activation only warns and does not refuse, so a scripted setup still works.
|
|
38
|
+
- **Tests:** New tests cover the service environment keys, `hasPlaceholder`, and a migrated configuration that does not need migration again after a switch. One test pins the same invariant on `deleteProfile`, the third place that can write the legacy pointer.
|
|
39
|
+
|
|
40
|
+
### Ports, shells, and what the package serves
|
|
41
|
+
|
|
42
|
+
- **A second instance had no port for the interceptor:** `--port` and `LLM_SWITCHER_PORT` move the gateway through `resolvePort`, but `computeLaunchState` read the interceptor port from `config.json` alone. `LLM_SWITCHER_BLINDFOLD_PORT` reached a hand-started `blindfold.mjs` and nothing else. A gateway sent to 3457 also landed on the port of the interceptor. The interceptor port now follows the same precedence as the gateway port, and it moves off the gateway port when the two meet. The README names both variables together.
|
|
43
|
+
- **A fish user got a line that fish cannot run:** The PATH hint printed `export PATH="..."`, which is not fish syntax, and it named `~/.profile`, which fish does not read. The hint now names `~/.config/fish/config.fish` and gives `fish_add_path -m`. zsh and bash keep what they had.
|
|
44
|
+
- **`switch on` installs the shims:** The README gave `switch shim install` as a separate step. `switch on` already installs the shims and prints what it installed. The README now says so, and it names the one case that still needs the command: a set `LLM_SWITCHER_STATE_DIR`.
|
|
45
|
+
- **What the package says it serves:** The description and both README taglines gave OpenAI and Gemini as clients. This gateway accepts two clients, Claude Code and Codex, and `toIR` accepts no other input format. OpenAI, Anthropic and Vertex are upstreams. All three texts now say that.
|
|
46
|
+
- **A test that failed on a loaded machine:** The lock test released the lock 400 ms after it started the child, then asserted that the child waited 100 ms or more. A spawn plus an import measured 163 to 541 ms on Node 18, so the child sometimes found the lock gone. The test now counts the 400 ms from the moment the child reports its start, the same way the test below it does.
|
|
47
|
+
|
|
48
|
+
## Release 1.2.7
|
|
49
|
+
|
|
50
|
+
- **Save from the dashboard:** The dashboard sent `tool: "auto"` for a profile that serves both tools. The gateway refuses that value. As a result, a new profile with the default tool, four of the five templates, and a save of any both-tools profile all failed. Now the dashboard saves `tool: null` for such a profile. The Vertex AI template uses `claude`.
|
|
51
|
+
- **New profile with a used key:** A new profile with the key of an existing profile replaced that profile and its API key without a warning. Now the dashboard refuses the save and names the key.
|
|
52
|
+
- **Turn all routes on:** After you turned every route off, "Activate compatible routes" gave only one profile a route. Codex stayed on the official endpoint. Now each tool gets its own profile, the same way as the `target` form of `/api/toggle`.
|
|
53
|
+
- **Failed switch:** After a failed switch, the dashboard kept the state that the person had pressed. Now it reads the state from the gateway again, so the switch, the badges, and the file agree.
|
|
54
|
+
- **Element ids:** The sidebar and the page header both used the id `btn-toggle`. The header button is now `btn-toggle-routes`.
|
|
55
|
+
- **Unsaved changes:** The dashboard never asked before it threw away typed text. Now the close button, Cancel, a press on the dark area, and Escape ask "Discard unsaved changes?" when you typed something. A refresh of the state no longer wipes the text you are typing.
|
|
56
|
+
- **Save with no change:** Every save added `thinkingMode: "auto"` and an empty `model1M` to the profile. Now a save with no edit leaves the profile as it was.
|
|
57
|
+
- **Emptied slots:** A model slot that you emptied, and a 1M box that you cleared, came back after the save. Now the save keeps what the form shows.
|
|
58
|
+
- **Tests:** New browser tests press the real dashboard: routes, profile forms, model slots, connection test, delete, catalog, and request inspector. After each step they check that the page, the gateway, and `config.json` say the same. They need Chromium and a global `WebSocket` (Node 22 or newer). Without them, the tests skip. Set `LLM_SWITCHER_CHROMIUM` to use another browser binary.
|
|
59
|
+
|
|
3
60
|
## Release 1.2.6
|
|
4
61
|
|
|
5
62
|
- **Deactivate in the dashboard:** The Deactivate button did not change the configuration, but the dashboard showed "deactivated". The dashboard sends `deactivate: true` and the profile key in `profile`. The gateway read `deactivate` as the profile key, so no target matched. Now `POST /api/switch` accepts both forms: `{"profile": "x", "deactivate": true}` and `{"deactivate": "x"}`.
|
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
<p align="center">
|
|
4
4
|
<b>Zero-dependency, multi-protocol edge gateway & provider switcher</b><br>
|
|
5
|
-
|
|
5
|
+
Bridge <b>Claude Code</b> and <b>Codex</b> to any upstream LLM API: OpenAI-compatible, Anthropic, or Vertex.<br>
|
|
6
6
|
Full bi-directional protocol conversion, official-model context windows, thinking protocol extraction, and edge message healing.
|
|
7
7
|
</p>
|
|
8
8
|
|
|
@@ -171,7 +171,7 @@ LLM Switcher acts as a transparent man-in-the-middle without ever touching clien
|
|
|
171
171
|
npm install -g llm-switcher
|
|
172
172
|
|
|
173
173
|
# Copy the example configuration into your data folder.
|
|
174
|
-
mkdir -p ~/.llm-switcher
|
|
174
|
+
mkdir -p -m 700 ~/.llm-switcher
|
|
175
175
|
cp "$(npm root -g)/llm-switcher/config.example.json" ~/.llm-switcher/config.json
|
|
176
176
|
```
|
|
177
177
|
|
|
@@ -234,6 +234,10 @@ In practice you never call these files. `switch shim install` puts `~/.llm-switc
|
|
|
234
234
|
and every `claude` and `codex` invocation — including `claude --resume` in a brand-new terminal —
|
|
235
235
|
runs the shim, which injects the variables into that one process.
|
|
236
236
|
|
|
237
|
+
`switch on` installs the shims itself and prints what it installed, so the command above is only
|
|
238
|
+
necessary when `LLM_SWITCHER_STATE_DIR` is set. A shim holds the state directory, and `switch on`
|
|
239
|
+
leaves the install to you in that case, because a temporary state directory outlives no shim.
|
|
240
|
+
|
|
237
241
|
---
|
|
238
242
|
|
|
239
243
|
### Claude Code Setup (Windows)
|
|
@@ -416,6 +420,9 @@ switch service uninstall # Remove background autostart service
|
|
|
416
420
|
switch shim install # Route new Claude and Codex sessions through the gateway
|
|
417
421
|
switch shim status # Verify shims + detect running sessions that bypass the gateway
|
|
418
422
|
switch shim uninstall # Remove the launcher shims
|
|
423
|
+
switch plugin install # Optional: a launch notice inside Claude Code and Codex
|
|
424
|
+
switch plugin status # Report whether that notice is installed
|
|
425
|
+
switch plugin uninstall # Remove that notice
|
|
419
426
|
switch off # Stop everything and restore the official endpoints
|
|
420
427
|
switch off claude # Turn Claude Code off; Codex keeps running
|
|
421
428
|
switch off codex # Turn Codex off; Claude Code keeps running
|
|
@@ -428,7 +435,9 @@ either `"claude"` or `"codex"` (or `null` for a profile that is switched off), s
|
|
|
428
435
|
`switch codex` can never point at the same profile by accident. There is no `openai` or `vertex`
|
|
429
436
|
target any more — the input routes those names stood for are gone.
|
|
430
437
|
|
|
431
|
-
The service runs without your shell. `switch service install` therefore copies `CLAUDE_CONFIG_DIR`, `LLM_SWITCHER_CONFIG`, `LLM_SWITCHER_STATE_DIR` and `LLM_SWITCHER_BLINDFOLD_CERTS` into the systemd unit or the launchd plist when they are set. The Windows task cannot carry them; set them as User environment variables instead. If the installed definition differs from the new one, for example after a hand edit, the old file is kept as `<file>.bak`. On Windows the task is created from an XML definition, so paths with spaces need no extra quoting and the task has no run-time limit. This Windows path is not tested on Windows yet.
|
|
438
|
+
The service runs without your shell. `switch service install` therefore copies `LLM_SWITCHER_HOME`, `CLAUDE_CONFIG_DIR`, `LLM_SWITCHER_CONFIG`, `LLM_SWITCHER_STATE_DIR` and `LLM_SWITCHER_BLINDFOLD_CERTS` into the systemd unit or the launchd plist when they are set. The Windows task cannot carry them; set them as User environment variables instead. If the installed definition differs from the new one, for example after a hand edit, the old file is kept as `<file>.bak`. On Windows the task is created from an XML definition, so paths with spaces need no extra quoting and the task has no run-time limit. This Windows path is not tested on Windows yet.
|
|
439
|
+
|
|
440
|
+
On Linux the unit is a systemd *user* service: it starts when you log in. To start it at boot without a login (a server reached over SSH), run `loginctl enable-linger $USER` once. The unit records the absolute path of the current `node`; if you manage Node with nvm and remove that version, run `switch service install` again.
|
|
432
441
|
|
|
433
442
|
### Resumed sessions & the shim (important)
|
|
434
443
|
|
|
@@ -472,6 +481,66 @@ re-opened from a shell where the shim is on `PATH`.
|
|
|
472
481
|
|
|
473
482
|
---
|
|
474
483
|
|
|
484
|
+
### Knowing that the switcher is on
|
|
485
|
+
|
|
486
|
+
Transparency has a price. The switcher opens neither `settings.json` nor `config.toml`, so no banner
|
|
487
|
+
appears, and you can forget that another provider answers every request. Two notices correct that.
|
|
488
|
+
The first is always on. The second is your choice, and you lose nothing if you skip it.
|
|
489
|
+
|
|
490
|
+
**1. The notice from the shim — always on**
|
|
491
|
+
|
|
492
|
+
When the shim starts a routed tool, it reports where the traffic goes:
|
|
493
|
+
|
|
494
|
+
```
|
|
495
|
+
claude -> intact-claude | intact.louispham.qzz.io | antigravity/gemini-3.8-flash | 1M
|
|
496
|
+
```
|
|
497
|
+
|
|
498
|
+
On Windows this is a desktop toast. Both tools can claim the whole screen, so a printed line
|
|
499
|
+
disappears behind the interface. On Linux and macOS it is one line on stderr. A tool that is off says
|
|
500
|
+
nothing at all, because the notice is read from `route-<tool>.txt`, which is written with that tool's
|
|
501
|
+
env file and is empty exactly when the tool is not routed.
|
|
502
|
+
|
|
503
|
+
This notice needs the shim, so it needs `~/.llm-switcher/bin` first on `PATH`. If you keep your own
|
|
504
|
+
`claude` or `codex` wrapper, the shim never runs and the notice never appears. The second notice
|
|
505
|
+
covers that case.
|
|
506
|
+
|
|
507
|
+
**2. The notice inside the tool — optional**
|
|
508
|
+
|
|
509
|
+
```bash
|
|
510
|
+
switch plugin install # add it
|
|
511
|
+
switch plugin status # report whether it is there
|
|
512
|
+
switch plugin uninstall # remove it
|
|
513
|
+
```
|
|
514
|
+
|
|
515
|
+
This writes one file for each tool, and it opens no configuration file of either tool:
|
|
516
|
+
|
|
517
|
+
| Tool | What is written | Why the tool loads it |
|
|
518
|
+
| --- | --- | --- |
|
|
519
|
+
| Claude Code | `~/.claude/skills/llm-switcher-status/` | A folder with `.claude-plugin/plugin.json` under a skills directory loads as a plugin on the next session. There is no marketplace and no install step. |
|
|
520
|
+
| Codex | `~/.codex/hooks.json` | Codex reads this file by itself. `config.toml` stays closed. |
|
|
521
|
+
|
|
522
|
+
An existing `~/.codex/hooks.json` is merged. Your own hooks stay, and `switch plugin uninstall` takes
|
|
523
|
+
only ours away. If the file does not parse, the install refuses it and changes nothing, because that
|
|
524
|
+
file can hold work that no backup returns.
|
|
525
|
+
|
|
526
|
+
The hook runs inside the tool, so it reports whatever launched the tool. It reports three states:
|
|
527
|
+
|
|
528
|
+
- **Routed, and the gateway answers** — it names the profile, the host, the model and the 1M window.
|
|
529
|
+
- **Routed, and the gateway does not answer** — it warns that the tool cannot reach the provider, it
|
|
530
|
+
names the port, and it names `switch on`. Before this notice, that state appeared only as a
|
|
531
|
+
connection error with no cause.
|
|
532
|
+
- **Not routed** — nothing. A tool on its official endpoint stays silent.
|
|
533
|
+
|
|
534
|
+
Codex asks you once to trust a new hook. If you decline, the hook stays off and nothing else changes.
|
|
535
|
+
|
|
536
|
+
**If you want neither notice**
|
|
537
|
+
|
|
538
|
+
Install no plugin, and keep the shim off `PATH`. The gateway still routes every request, the healer
|
|
539
|
+
still runs, and `switch status` still reports the state. A notice is a reminder. It is never a part
|
|
540
|
+
of the routing.
|
|
541
|
+
|
|
542
|
+
---
|
|
543
|
+
|
|
475
544
|
## Configuration Schema (`config.json`)
|
|
476
545
|
|
|
477
546
|
```jsonc
|
|
@@ -564,7 +633,8 @@ Because of this split, a provider fingerprint is never a rule in this gateway. I
|
|
|
564
633
|
|---|---|
|
|
565
634
|
| `LLM_SWITCHER_HOME=/path` | Use this folder as the data folder (config, admin token, launch files, logs) for an npm install or a checkout. |
|
|
566
635
|
| `LLM_SWITCHER_CONFIG=/path/config.json` | Use a config file outside the data folder (the proxy, `switch` and `mcp.mjs` all honour it). |
|
|
567
|
-
| `--port <n>` / `LLM_SWITCHER_PORT` | Override the listening port (priority: flag > env > `config.port`). |
|
|
636
|
+
| `--port <n>` / `LLM_SWITCHER_PORT` | Override the listening port of the gateway (priority: flag > env > `config.port`). |
|
|
637
|
+
| `LLM_SWITCHER_BLINDFOLD_PORT` | Override the listening port of the interceptor (priority: env > `config.blindfold.port` > 3457). Set it together with `LLM_SWITCHER_PORT` to run a second instance. |
|
|
568
638
|
| `x-llm-profile: <key>` header (alias `x-profile`) or `?profile=<key>` | Route a single request through a specific profile. An unknown key returns HTTP 400 instead of silently falling back. |
|
|
569
639
|
| `profile.thinkingMode` | `auto` (default, for gateways like intact or 9Router): restore stripped thinking, inject a `<think>` guide for non-reasoning models, send `thinking` + `reasoning_effort`. `native` (strict OpenAI APIs): send only `reasoning_effort` when the client asks, never touch the prompt, use `max_completion_tokens`. `off`: never send reasoning parameters. |
|
|
570
640
|
| `profile.endpoints.countTokens` | Override the Anthropic `count_tokens` URL. |
|
package/README.vi.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
<p align="center">
|
|
4
4
|
<b>Cổng ngõ biên (Edge Gateway) chuyển đổi đa giao thức LLM siêu nhẹ, Zero-Dependency</b><br>
|
|
5
|
-
Cầu nối hai chiều giữa <b>Claude Code</b
|
|
5
|
+
Cầu nối hai chiều giữa <b>Claude Code</b> và <b>Codex</b> với mọi nhà cung cấp LLM: OpenAI-compatible, Anthropic hoặc Vertex.<br>
|
|
6
6
|
Chuyển đổi giao thức qua IR, cửa sổ context theo model chính thức, trích xuất thinking blocks và tự chữa lành đồ thị tin nhắn trước khi ra Internet.
|
|
7
7
|
</p>
|
|
8
8
|
|
|
@@ -171,7 +171,7 @@ LLM Switcher hoạt động như một lớp trung gian mạng trong suốt (tra
|
|
|
171
171
|
npm install -g llm-switcher
|
|
172
172
|
|
|
173
173
|
# Chép file cấu hình mẫu vào thư mục dữ liệu.
|
|
174
|
-
mkdir -p ~/.llm-switcher
|
|
174
|
+
mkdir -p -m 700 ~/.llm-switcher
|
|
175
175
|
cp "$(npm root -g)/llm-switcher/config.example.json" ~/.llm-switcher/config.json
|
|
176
176
|
```
|
|
177
177
|
|
|
@@ -236,6 +236,10 @@ Trong thực tế bạn không bao giờ tự gọi các file này. `switch shim
|
|
|
236
236
|
vào `PATH`, và mọi lệnh `claude` hay `codex` — kể cả `claude --resume` trong một terminal hoàn toàn
|
|
237
237
|
mới — đều chạy shim, vốn chỉ tiêm biến vào đúng tiến trình đó.
|
|
238
238
|
|
|
239
|
+
`switch on` tự cài shim và in ra những gì đã cài, nên lệnh trên chỉ cần khi `LLM_SWITCHER_STATE_DIR`
|
|
240
|
+
được đặt. Shim mang theo state directory, và trong trưồng hợp đó `switch on` để việc cài cho bạn,
|
|
241
|
+
vì một state directory tạm thời sẽ biến mất trước shim.
|
|
242
|
+
|
|
239
243
|
---
|
|
240
244
|
|
|
241
245
|
### Cấu hình cho Claude Code (Windows)
|
|
@@ -413,6 +417,9 @@ switch service uninstall # Gỡ bỏ service chạy ngầm
|
|
|
413
417
|
switch shim install # Route phiên Claude và Codex mới qua gateway
|
|
414
418
|
switch shim status # Kiểm tra shim + phát hiện phiên đang chạy ngoài gateway
|
|
415
419
|
switch shim uninstall # Gỡ shim khỏi launcher
|
|
420
|
+
switch plugin install # Tuỳ chọn: nhắc nhở ngay trong Claude Code và Codex
|
|
421
|
+
switch plugin status # Xem nhắc nhở đó đã cài chưa
|
|
422
|
+
switch plugin uninstall # Gỡ nhắc nhở đó
|
|
416
423
|
switch off # Tắt tất cả và quay về endpoint chính thức
|
|
417
424
|
switch off claude # Tắt Claude Code; Codex vẫn chạy tiếp
|
|
418
425
|
switch off codex # Tắt Codex; Claude Code vẫn chạy tiếp
|
|
@@ -426,7 +433,9 @@ cụ: `tool` là `"claude"` hoặc `"codex"` (hoặc `null` khi profile đang t
|
|
|
426
433
|
các route đầu vào mà chúng đại diện đã bị gỡ.
|
|
427
434
|
|
|
428
435
|
|
|
429
|
-
Service không chạy trong shell của bạn. Vì vậy `switch service install` chép `CLAUDE_CONFIG_DIR`, `LLM_SWITCHER_CONFIG`, `LLM_SWITCHER_STATE_DIR` và `LLM_SWITCHER_BLINDFOLD_CERTS` vào unit systemd hoặc plist launchd khi các biến này có giá trị. Task Windows không mang được các biến này; hãy đặt chúng thành biến môi trường User. Nếu file định nghĩa đã cài khác file mới (ví dụ đã sửa tay), file cũ được giữ lại thành `<file>.bak`. Trên Windows, task được tạo từ file XML, nên đường dẫn có dấu cách không cần quote thêm và task không bị giới hạn thời gian chạy. Đường Windows này chưa được test trên Windows.
|
|
436
|
+
Service không chạy trong shell của bạn. Vì vậy `switch service install` chép `LLM_SWITCHER_HOME`, `CLAUDE_CONFIG_DIR`, `LLM_SWITCHER_CONFIG`, `LLM_SWITCHER_STATE_DIR` và `LLM_SWITCHER_BLINDFOLD_CERTS` vào unit systemd hoặc plist launchd khi các biến này có giá trị. Task Windows không mang được các biến này; hãy đặt chúng thành biến môi trường User. Nếu file định nghĩa đã cài khác file mới (ví dụ đã sửa tay), file cũ được giữ lại thành `<file>.bak`. Trên Windows, task được tạo từ file XML, nên đường dẫn có dấu cách không cần quote thêm và task không bị giới hạn thời gian chạy. Đường Windows này chưa được test trên Windows.
|
|
437
|
+
|
|
438
|
+
Trên Linux, unit là service systemd *của user*: nó khởi động khi bạn đăng nhập. Muốn nó chạy ngay khi máy boot mà không cần đăng nhập (ví dụ server truy cập qua SSH), chạy một lần `loginctl enable-linger $USER`. Unit ghi đường dẫn tuyệt đối của `node` hiện tại; nếu bạn quản lý Node bằng nvm và gỡ phiên bản đó, hãy chạy lại `switch service install`.
|
|
430
439
|
### Phiên mở lại (`--resume`) và cơ chế shim — quan trọng
|
|
431
440
|
|
|
432
441
|
`switch on` ghi `env-claude.*` và `env-codex.*`, và **không ghi gì** vào
|
|
@@ -466,6 +475,67 @@ trong `PATH`.
|
|
|
466
475
|
|
|
467
476
|
---
|
|
468
477
|
|
|
478
|
+
### Biết được switcher đang bật
|
|
479
|
+
|
|
480
|
+
Vô hình cũng có giá của nó. Switcher không mở `settings.json` cũng không mở `config.toml`, nên không
|
|
481
|
+
có banner nào hiện lên, và bạn dễ quên rằng mọi request đang do một provider khác trả lời. Có hai lớp
|
|
482
|
+
nhắc nhở để bù chỗ đó. Lớp thứ nhất luôn bật. Lớp thứ hai là tuỳ bạn, không cài cũng không mất gì.
|
|
483
|
+
|
|
484
|
+
**1. Nhắc nhở từ shim — luôn bật**
|
|
485
|
+
|
|
486
|
+
Khi shim mở một tool đang được route, nó cho biết traffic đi đâu:
|
|
487
|
+
|
|
488
|
+
```
|
|
489
|
+
claude -> intact-claude | intact.louispham.qzz.io | antigravity/gemini-3.8-flash | 1M
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
Trên Windows đây là một toast của hệ điều hành. Cả hai tool đều có thể chiếm trọn màn hình, nên một
|
|
493
|
+
dòng chữ in ra sẽ bị giao diện che mất. Trên Linux và macOS thì là một dòng trên stderr. Tool nào
|
|
494
|
+
đang tắt thì im lặng hoàn toàn, vì nội dung nhắc nhở đọc từ `route-<tool>.txt`, file này được ghi
|
|
495
|
+
cùng lượt với file env của chính tool đó và rỗng đúng khi tool đó không được route.
|
|
496
|
+
|
|
497
|
+
Nhắc nhở này cần shim, nên cần `~/.llm-switcher/bin` đứng trước trong `PATH`. Nếu bạn dùng wrapper
|
|
498
|
+
`claude` hoặc `codex` tự viết thì shim không chạy và nhắc nhở này không xuất hiện. Lớp thứ hai lo
|
|
499
|
+
đúng trường hợp đó.
|
|
500
|
+
|
|
501
|
+
**2. Nhắc nhở ngay trong tool — tuỳ chọn**
|
|
502
|
+
|
|
503
|
+
```bash
|
|
504
|
+
switch plugin install # cài
|
|
505
|
+
switch plugin status # xem đã cài chưa
|
|
506
|
+
switch plugin uninstall # gỡ
|
|
507
|
+
```
|
|
508
|
+
|
|
509
|
+
Lệnh này ghi một file cho mỗi tool, và không mở file cấu hình nào của hai tool đó:
|
|
510
|
+
|
|
511
|
+
| Tool | Ghi gì | Vì sao tool tự nạp |
|
|
512
|
+
| --- | --- | --- |
|
|
513
|
+
| Claude Code | `~/.claude/skills/llm-switcher-status/` | Một thư mục có `.claude-plugin/plugin.json` nằm dưới skills directory sẽ được nạp như một plugin ở phiên kế tiếp. Không cần marketplace, không cần bước install. |
|
|
514
|
+
| Codex | `~/.codex/hooks.json` | Codex tự đọc file này. `config.toml` không bị mở. |
|
|
515
|
+
|
|
516
|
+
Nếu `~/.codex/hooks.json` đã có sẵn thì lệnh sẽ **merge**. Hook của bạn được giữ nguyên, và
|
|
517
|
+
`switch plugin uninstall` chỉ lấy đi phần của switcher. Nếu file không parse được thì lệnh từ chối và
|
|
518
|
+
không đổi gì, vì file đó có thể đang giữ công việc mà không bản backup nào lấy lại được.
|
|
519
|
+
|
|
520
|
+
Hook chạy bên trong tool, nên nó báo được bất kể cái gì đã mở tool. Nó báo ba trạng thái:
|
|
521
|
+
|
|
522
|
+
- **Đang route, gateway trả lời** — nêu profile, host, model và cửa sổ 1M.
|
|
523
|
+
- **Đang route, gateway không trả lời** — cảnh báo tool không tới được provider, nêu số cổng, và nêu
|
|
524
|
+
lệnh `switch on`. Trước khi có nhắc nhở này, trạng thái đó chỉ hiện ra dưới dạng một lỗi kết nối
|
|
525
|
+
không rõ nguyên nhân.
|
|
526
|
+
- **Không route** — im lặng. Tool đang dùng endpoint chính thức thì không nói gì.
|
|
527
|
+
|
|
528
|
+
Codex sẽ hỏi bạn một lần để tin cậy hook mới. Nếu bạn từ chối thì hook không chạy và mọi thứ khác
|
|
529
|
+
giữ nguyên.
|
|
530
|
+
|
|
531
|
+
**Nếu bạn không muốn lớp nhắc nhở nào**
|
|
532
|
+
|
|
533
|
+
Đừng cài plugin, và để shim ngoài `PATH`. Gateway vẫn route mọi request, healer vẫn chạy, và
|
|
534
|
+
`switch status` vẫn báo trạng thái. Nhắc nhở chỉ là nhắc nhở. Nó không bao giờ là một phần của việc
|
|
535
|
+
route.
|
|
536
|
+
|
|
537
|
+
---
|
|
538
|
+
|
|
469
539
|
## Cấu trúc Cấu hình (`config.json`)
|
|
470
540
|
|
|
471
541
|
```jsonc
|
|
@@ -559,7 +629,8 @@ Nhờ cách chia này, fingerprint của provider không bao giờ là rule tron
|
|
|
559
629
|
|---|---|
|
|
560
630
|
| `LLM_SWITCHER_HOME=/path` | Dùng thư mục này làm thư mục dữ liệu (config, admin token, file launcher, log) cho cả bản npm lẫn bản checkout. |
|
|
561
631
|
| `LLM_SWITCHER_CONFIG=/path/config.json` | Dùng file cấu hình nằm ngoài thư mục dữ liệu (proxy, `switch` và `mcp.mjs` đều hỗ trợ). |
|
|
562
|
-
| `--port <n>` / `LLM_SWITCHER_PORT` | Ghi đè cổng lắng nghe (ưu tiên: flag > env > `config.port`). |
|
|
632
|
+
| `--port <n>` / `LLM_SWITCHER_PORT` | Ghi đè cổng lắng nghe của gateway (ưu tiên: flag > env > `config.port`). |
|
|
633
|
+
| `LLM_SWITCHER_BLINDFOLD_PORT` | Ghi đè cổng lắng nghe của interceptor (ưu tiên: env > `config.blindfold.port` > 3457). Đảm bảo đặt cùng `LLM_SWITCHER_PORT` khi chạy instance thứ hai. |
|
|
563
634
|
| Header `x-llm-profile: <key>` (tên khác `x-profile`) hoặc `?profile=<key>` | Định tuyến riêng 1 request qua profile chỉ định. Key không tồn tại trả HTTP 400 thay vì âm thầm dùng profile khác. |
|
|
564
635
|
| `profile.thinkingMode` | `auto` (mặc định, cho gateway như intact hoặc 9Router): phục hồi thinking bị xoá, inject hướng dẫn `<think>` cho model không có reasoning, gửi `thinking` + `reasoning_effort`. `native` (API OpenAI nghiêm ngặt): chỉ gửi `reasoning_effort` khi client yêu cầu, không sửa prompt, dùng `max_completion_tokens`. `off`: không bao giờ gửi tham số reasoning. |
|
|
565
636
|
| `profile.endpoints.countTokens` | Ghi đè URL `count_tokens` của Anthropic. |
|
package/blindfold/blindfold.mjs
CHANGED
|
@@ -122,7 +122,8 @@ const DECODERS = {
|
|
|
122
122
|
'x-gzip': zlib.gunzipSync,
|
|
123
123
|
br: zlib.brotliDecompressSync,
|
|
124
124
|
deflate: zlib.inflateSync,
|
|
125
|
-
zstd
|
|
125
|
+
// zstd arrived in Node 22.15; older runtimes report the body as undecodable instead of noise.
|
|
126
|
+
zstd: zlib.zstdDecompressSync || (() => { throw new Error('zstd is not supported by this Node.js version'); })
|
|
126
127
|
};
|
|
127
128
|
|
|
128
129
|
export function decodeBody(buffer, contentEncoding) {
|
|
@@ -1,110 +1,110 @@
|
|
|
1
|
-
# Token Optimizer Interoperability & Failure Mode Report
|
|
2
|
-
|
|
3
|
-
**How LLM Switcher acts as the protective outermost edge gateway for aggressive prompt/token optimizers (Headroom, RTK, Ponytail).**
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
## Executive Summary
|
|
8
|
-
|
|
9
|
-
Third-party prompt optimizers and token compressors — such as **Headroom**, **RTK (Rust Token Killer)**, and **Ponytail** — attempt to reduce LLM input tokens by aggressively pruning message history, truncating command stdout, or forcing extreme prompt brevity.
|
|
10
|
-
|
|
11
|
-
While these tools can reduce raw token counts in simple scenarios, **they frequently break complex agentic coding workflows** by corrupting message graphs, orphaning tool calls, and stripping reasoning parameters. When these pruned payloads hit upstream APIs directly (such as Anthropic, OpenAI, or 9Router), the provider immediately throws fatal `HTTP 400 Bad Request` errors or severely degrades reasoning depth.
|
|
12
|
-
|
|
13
|
-
**LLM Switcher solves this by acting as the outermost edge gatekeeper (`127.0.0.1:3456`).** It intercepts the pruned payload before it leaves your machine, runs its built-in **Healer Engine** to repair message graphs and restore reasoning parameters, unlocks 1M context windows, and safely converts the protocol to your upstream provider.
|
|
14
|
-
|
|
15
|
-
---
|
|
16
|
-
|
|
17
|
-
## Tool Breakdown: What They Do & How They Break Payloads
|
|
18
|
-
|
|
19
|
-
### 1. Headroom (`headroomlabs-ai/headroom`)
|
|
20
|
-
- **Mechanism:** Runs as a local proxy on `:8787` (or wraps CLI agents). Compresses conversation history, RAG chunks, and tool outputs using SmartCrusher (JSON), CodeCompressor (AST), and Kompress-v2-base. Also attempts "effort routing" to dial down thinking budgets.
|
|
21
|
-
- **Critical Failure Points:**
|
|
22
|
-
- **Orphaned `tool_result` blocks:** When pruning historical turns, Headroom often discards the `assistant` turn containing a `tool_use`, while retaining the subsequent `user` turn containing the `tool_result`. Anthropic's API strictly validates tool use IDs and crashes with:
|
|
23
|
-
```
|
|
24
|
-
HTTP 400 invalid_request_error: "tool_use_id 'xxx' does not correspond to any tool_use"
|
|
25
|
-
```
|
|
26
|
-
- **Consecutive `user` turns:** Dropping intermediary assistant turns causes multiple user messages to sit adjacent to each other. Anthropic strictly throws:
|
|
27
|
-
```
|
|
28
|
-
HTTP 400 invalid_request_error: "roles must alternate between 'user' and 'assistant'"
|
|
29
|
-
```
|
|
30
|
-
- **Reasoning Suppression:** Its "effort routing" dials down `thinking.budget_tokens` on routine tool turns. On complex models (Claude Opus, Gemini Flash), this prevents the model from formulating multi-step reasoning before acting.
|
|
31
|
-
|
|
32
|
-
### 2. RTK (`rtk-ai/rtk` - Rust Token Killer)
|
|
33
|
-
- **Mechanism:** A single Rust binary that hooks into shell tool execution (e.g. `PreToolUse` in Claude Code / Cursor) and rewrites CLI commands (`git`, `ls`, `cat`, `grep`, `pytest`) to filter out noise, truncate lines, and inject recall tokens (`[full output: rtk recall xxx]`).
|
|
34
|
-
- **Critical Failure Points:**
|
|
35
|
-
- **Corrupted Structural Data:** When an agent invokes a tool expecting machine-readable JSON or exact AST formatting, RTK's heuristic summaries can alter structural delimiters, causing downstream tool call parsing errors.
|
|
36
|
-
- **Custom Tracking Headers:** RTK and associated tracing proxies inject headers (`x-rtk-*`, `traceparent`, `x-optimizer-id`) that some strict upstream endpoints reject if not cleanly forwarded.
|
|
37
|
-
|
|
38
|
-
### 3. Ponytail (`DietrichGebert/ponytail`)
|
|
39
|
-
- **Mechanism:** A behavioral prompt engineering plugin/ruleset that injects extreme conciseness instructions ("write one line, it works, YAGNI") into agent system prompts across 20+ coding tools.
|
|
40
|
-
- **Critical Failure Points:**
|
|
41
|
-
- **Premature Execution without Reasoning:** By commanding the model to be maximally brief and avoid planning, reasoning models are discouraged from spending thinking tokens. The model outputs untested single-liners that often fail type checks and test suites.
|
|
42
|
-
- **System Prompt Prefix Invalidation:** Injected rules alter the leading system prompt bytes, busting provider prompt caches unless carefully aligned.
|
|
43
|
-
|
|
44
|
-
---
|
|
45
|
-
|
|
46
|
-
## The Healer Engine: How LLM Switcher Protects the Workflow
|
|
47
|
-
|
|
48
|
-
LLM Switcher sits between the optimizer tool and the upstream LLM:
|
|
49
|
-
|
|
50
|
-
```
|
|
51
|
-
[CLI Agent] ──> [Optimizer: Headroom / RTK] ──> [LLM Switcher :3456] ──> [Upstream / 9Router]
|
|
52
|
-
│
|
|
53
|
-
├── 1. Heal Orphaned tool_results
|
|
54
|
-
├── 2. Merge Consecutive Turns
|
|
55
|
-
├── 3. Restore Stripped Thinking
|
|
56
|
-
└── 4. Enforce 1M Context
|
|
57
|
-
```
|
|
58
|
-
|
|
59
|
-
### Protection Matrix: Before vs. After
|
|
60
|
-
|
|
61
|
-
| Scenario | Direct to Upstream (Without Switcher) | Through LLM Switcher (Healer Engine) |
|
|
62
|
-
|---|---|---|
|
|
63
|
-
| **Orphaned `tool_result` turn** | ❌ **HTTP 400 Crash**: `tool_use_id does not correspond to any tool_use` | ✅ **HTTP 200 OK**: Heals orphaned result into contextual text block `[Tool Result (id)]: ...` |
|
|
64
|
-
| **Consecutive `user` turns** | ❌ **HTTP 400 Crash**: `roles must alternate` | ✅ **HTTP 200 OK**: Merges consecutive turns into a single valid turn seamlessly |
|
|
65
|
-
| **Stripped `thinking` parameters** | ⚠️ **Degraded AI**: Reasoning disabled, model outputs shallow single-liners | ✅ **HTTP 200 OK**: Detects reasoning models and automatically restores safe thinking budget |
|
|
66
|
-
| **Orphaned `tool` role in Chat API** | ❌ **HTTP 400 Crash**: `tool role must respond to tool_calls` | ✅ **HTTP 200 OK**: Converts orphaned tool message into user context |
|
|
67
|
-
| **Custom tracking headers** | ⚠️ Connection dropped / unrecognized header warnings | ✅ **HTTP 200 OK**: Cleanly passes through `traceparent`, `x-request-id`, `x-rtk-*` |
|
|
68
|
-
|
|
69
|
-
---
|
|
70
|
-
|
|
71
|
-
## Test Methodology & Verification Suite
|
|
72
|
-
|
|
73
|
-
We created an automated verification suite in [`tests/live-optimizer-interop.mjs`](../tests/live-optimizer-interop.mjs) that systematically replicates the failure modes of each tool:
|
|
74
|
-
|
|
75
|
-
### Running the Test Suite
|
|
76
|
-
```bash
|
|
77
|
-
node tests/live-optimizer-interop.mjs
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
### Live Test Results
|
|
81
|
-
```text
|
|
82
|
-
================================================================
|
|
83
|
-
LLM SWITCHER — TOKEN OPTIMIZER INTEROPERABILITY TEST SUITE
|
|
84
|
-
Simulating failure modes from Headroom, RTK, and Ponytail
|
|
85
|
-
================================================================
|
|
86
|
-
|
|
87
|
-
[TEST] Headroom Simulation: Orphaned tool_result turn... PASS (8840ms)
|
|
88
|
-
↳ Healed orphaned tool_result. Response HTTP 200: "It looks like you've shared a fragment of context ..."
|
|
89
|
-
[TEST] Headroom Simulation: Consecutive User turns (Role alternation violation)... PASS (2683ms)
|
|
90
|
-
↳ Merged consecutive turns seamlessly. Stop reason: end_turn
|
|
91
|
-
[TEST] Thinking Guard: Restoring stripped thinking parameter on reasoning models... PASS (5372ms)
|
|
92
|
-
↳ Automatically restored thinking: thinking_delta=true, signature_delta=true, text_delta=true
|
|
93
|
-
[TEST] RTK Intermediary: Custom headers and traceparent passthrough... PASS (2453ms)
|
|
94
|
-
↳ Headers accepted cleanly with HTTP 200 OK
|
|
95
|
-
[TEST] OpenAI Chat Healer: Orphaned role tool without preceding assistant tool_calls... PASS (1561ms)
|
|
96
|
-
↳ Chat Healer rescued orphaned tool role. Stop reason: length
|
|
97
|
-
|
|
98
|
-
================================================================
|
|
99
|
-
TEST RESULTS: 5 PASSED / 0 FAILED
|
|
100
|
-
================================================================
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
---
|
|
104
|
-
|
|
105
|
-
## Recommended User Setup
|
|
106
|
-
|
|
107
|
-
For developers using prompt optimization tools:
|
|
108
|
-
1. Keep the optimizer installed in your CLI tool as usual.
|
|
109
|
-
2. In the optimizer's configuration (e.g. `headroom.yaml` or RTK upstream settings), set the upstream target URL to **LLM Switcher** (`http://127.0.0.1:3456`).
|
|
110
|
-
3. Enjoy prompt compression savings without worrying about broken conversation graphs, HTTP 400 crashes, or lost reasoning depth.
|
|
1
|
+
# Token Optimizer Interoperability & Failure Mode Report
|
|
2
|
+
|
|
3
|
+
**How LLM Switcher acts as the protective outermost edge gateway for aggressive prompt/token optimizers (Headroom, RTK, Ponytail).**
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Executive Summary
|
|
8
|
+
|
|
9
|
+
Third-party prompt optimizers and token compressors — such as **Headroom**, **RTK (Rust Token Killer)**, and **Ponytail** — attempt to reduce LLM input tokens by aggressively pruning message history, truncating command stdout, or forcing extreme prompt brevity.
|
|
10
|
+
|
|
11
|
+
While these tools can reduce raw token counts in simple scenarios, **they frequently break complex agentic coding workflows** by corrupting message graphs, orphaning tool calls, and stripping reasoning parameters. When these pruned payloads hit upstream APIs directly (such as Anthropic, OpenAI, or 9Router), the provider immediately throws fatal `HTTP 400 Bad Request` errors or severely degrades reasoning depth.
|
|
12
|
+
|
|
13
|
+
**LLM Switcher solves this by acting as the outermost edge gatekeeper (`127.0.0.1:3456`).** It intercepts the pruned payload before it leaves your machine, runs its built-in **Healer Engine** to repair message graphs and restore reasoning parameters, unlocks 1M context windows, and safely converts the protocol to your upstream provider.
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## Tool Breakdown: What They Do & How They Break Payloads
|
|
18
|
+
|
|
19
|
+
### 1. Headroom (`headroomlabs-ai/headroom`)
|
|
20
|
+
- **Mechanism:** Runs as a local proxy on `:8787` (or wraps CLI agents). Compresses conversation history, RAG chunks, and tool outputs using SmartCrusher (JSON), CodeCompressor (AST), and Kompress-v2-base. Also attempts "effort routing" to dial down thinking budgets.
|
|
21
|
+
- **Critical Failure Points:**
|
|
22
|
+
- **Orphaned `tool_result` blocks:** When pruning historical turns, Headroom often discards the `assistant` turn containing a `tool_use`, while retaining the subsequent `user` turn containing the `tool_result`. Anthropic's API strictly validates tool use IDs and crashes with:
|
|
23
|
+
```
|
|
24
|
+
HTTP 400 invalid_request_error: "tool_use_id 'xxx' does not correspond to any tool_use"
|
|
25
|
+
```
|
|
26
|
+
- **Consecutive `user` turns:** Dropping intermediary assistant turns causes multiple user messages to sit adjacent to each other. Anthropic strictly throws:
|
|
27
|
+
```
|
|
28
|
+
HTTP 400 invalid_request_error: "roles must alternate between 'user' and 'assistant'"
|
|
29
|
+
```
|
|
30
|
+
- **Reasoning Suppression:** Its "effort routing" dials down `thinking.budget_tokens` on routine tool turns. On complex models (Claude Opus, Gemini Flash), this prevents the model from formulating multi-step reasoning before acting.
|
|
31
|
+
|
|
32
|
+
### 2. RTK (`rtk-ai/rtk` - Rust Token Killer)
|
|
33
|
+
- **Mechanism:** A single Rust binary that hooks into shell tool execution (e.g. `PreToolUse` in Claude Code / Cursor) and rewrites CLI commands (`git`, `ls`, `cat`, `grep`, `pytest`) to filter out noise, truncate lines, and inject recall tokens (`[full output: rtk recall xxx]`).
|
|
34
|
+
- **Critical Failure Points:**
|
|
35
|
+
- **Corrupted Structural Data:** When an agent invokes a tool expecting machine-readable JSON or exact AST formatting, RTK's heuristic summaries can alter structural delimiters, causing downstream tool call parsing errors.
|
|
36
|
+
- **Custom Tracking Headers:** RTK and associated tracing proxies inject headers (`x-rtk-*`, `traceparent`, `x-optimizer-id`) that some strict upstream endpoints reject if not cleanly forwarded.
|
|
37
|
+
|
|
38
|
+
### 3. Ponytail (`DietrichGebert/ponytail`)
|
|
39
|
+
- **Mechanism:** A behavioral prompt engineering plugin/ruleset that injects extreme conciseness instructions ("write one line, it works, YAGNI") into agent system prompts across 20+ coding tools.
|
|
40
|
+
- **Critical Failure Points:**
|
|
41
|
+
- **Premature Execution without Reasoning:** By commanding the model to be maximally brief and avoid planning, reasoning models are discouraged from spending thinking tokens. The model outputs untested single-liners that often fail type checks and test suites.
|
|
42
|
+
- **System Prompt Prefix Invalidation:** Injected rules alter the leading system prompt bytes, busting provider prompt caches unless carefully aligned.
|
|
43
|
+
|
|
44
|
+
---
|
|
45
|
+
|
|
46
|
+
## The Healer Engine: How LLM Switcher Protects the Workflow
|
|
47
|
+
|
|
48
|
+
LLM Switcher sits between the optimizer tool and the upstream LLM:
|
|
49
|
+
|
|
50
|
+
```
|
|
51
|
+
[CLI Agent] ──> [Optimizer: Headroom / RTK] ──> [LLM Switcher :3456] ──> [Upstream / 9Router]
|
|
52
|
+
│
|
|
53
|
+
├── 1. Heal Orphaned tool_results
|
|
54
|
+
├── 2. Merge Consecutive Turns
|
|
55
|
+
├── 3. Restore Stripped Thinking
|
|
56
|
+
└── 4. Enforce 1M Context
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
### Protection Matrix: Before vs. After
|
|
60
|
+
|
|
61
|
+
| Scenario | Direct to Upstream (Without Switcher) | Through LLM Switcher (Healer Engine) |
|
|
62
|
+
|---|---|---|
|
|
63
|
+
| **Orphaned `tool_result` turn** | ❌ **HTTP 400 Crash**: `tool_use_id does not correspond to any tool_use` | ✅ **HTTP 200 OK**: Heals orphaned result into contextual text block `[Tool Result (id)]: ...` |
|
|
64
|
+
| **Consecutive `user` turns** | ❌ **HTTP 400 Crash**: `roles must alternate` | ✅ **HTTP 200 OK**: Merges consecutive turns into a single valid turn seamlessly |
|
|
65
|
+
| **Stripped `thinking` parameters** | ⚠️ **Degraded AI**: Reasoning disabled, model outputs shallow single-liners | ✅ **HTTP 200 OK**: Detects reasoning models and automatically restores safe thinking budget |
|
|
66
|
+
| **Orphaned `tool` role in Chat API** | ❌ **HTTP 400 Crash**: `tool role must respond to tool_calls` | ✅ **HTTP 200 OK**: Converts orphaned tool message into user context |
|
|
67
|
+
| **Custom tracking headers** | ⚠️ Connection dropped / unrecognized header warnings | ✅ **HTTP 200 OK**: Cleanly passes through `traceparent`, `x-request-id`, `x-rtk-*` |
|
|
68
|
+
|
|
69
|
+
---
|
|
70
|
+
|
|
71
|
+
## Test Methodology & Verification Suite
|
|
72
|
+
|
|
73
|
+
We created an automated verification suite in [`tests/live-optimizer-interop.mjs`](../tests/live-optimizer-interop.mjs) that systematically replicates the failure modes of each tool:
|
|
74
|
+
|
|
75
|
+
### Running the Test Suite
|
|
76
|
+
```bash
|
|
77
|
+
node tests/live-optimizer-interop.mjs
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### Live Test Results
|
|
81
|
+
```text
|
|
82
|
+
================================================================
|
|
83
|
+
LLM SWITCHER — TOKEN OPTIMIZER INTEROPERABILITY TEST SUITE
|
|
84
|
+
Simulating failure modes from Headroom, RTK, and Ponytail
|
|
85
|
+
================================================================
|
|
86
|
+
|
|
87
|
+
[TEST] Headroom Simulation: Orphaned tool_result turn... PASS (8840ms)
|
|
88
|
+
↳ Healed orphaned tool_result. Response HTTP 200: "It looks like you've shared a fragment of context ..."
|
|
89
|
+
[TEST] Headroom Simulation: Consecutive User turns (Role alternation violation)... PASS (2683ms)
|
|
90
|
+
↳ Merged consecutive turns seamlessly. Stop reason: end_turn
|
|
91
|
+
[TEST] Thinking Guard: Restoring stripped thinking parameter on reasoning models... PASS (5372ms)
|
|
92
|
+
↳ Automatically restored thinking: thinking_delta=true, signature_delta=true, text_delta=true
|
|
93
|
+
[TEST] RTK Intermediary: Custom headers and traceparent passthrough... PASS (2453ms)
|
|
94
|
+
↳ Headers accepted cleanly with HTTP 200 OK
|
|
95
|
+
[TEST] OpenAI Chat Healer: Orphaned role tool without preceding assistant tool_calls... PASS (1561ms)
|
|
96
|
+
↳ Chat Healer rescued orphaned tool role. Stop reason: length
|
|
97
|
+
|
|
98
|
+
================================================================
|
|
99
|
+
TEST RESULTS: 5 PASSED / 0 FAILED
|
|
100
|
+
================================================================
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
---
|
|
104
|
+
|
|
105
|
+
## Recommended User Setup
|
|
106
|
+
|
|
107
|
+
For developers using prompt optimization tools:
|
|
108
|
+
1. Keep the optimizer installed in your CLI tool as usual.
|
|
109
|
+
2. In the optimizer's configuration (e.g. `headroom.yaml` or RTK upstream settings), set the upstream target URL to **LLM Switcher** (`http://127.0.0.1:3456`).
|
|
110
|
+
3. Enjoy prompt compression savings without worrying about broken conversation graphs, HTTP 400 crashes, or lost reasoning depth.
|
package/docs/cross-platform.md
CHANGED
|
@@ -141,5 +141,7 @@ Run the test suite:
|
|
|
141
141
|
npm test
|
|
142
142
|
```
|
|
143
143
|
|
|
144
|
-
On Linux and macOS every test runs. On Windows four shim tests skip.
|
|
144
|
+
On Linux and macOS every test runs. On Windows four shim tests skip. On Node older than 22,
|
|
145
|
+
the three WebSocket client tests skip (no global `WebSocket`), and the zstd assertion is
|
|
146
|
+
skipped before Node 22.15. A failure on any
|
|
145
147
|
platform is a real defect; report it with the platform name and the Node version.
|