@gapi/gcli 1.8.263 → 1.8.265
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/local-ai-runners/SKILL.md +11 -1
- package/README-CLI.md +94 -29
- package/README.md +41 -1
- package/docs/changelog.d/2026-11-15-local-ai-runners-one-runner-serves-several-model-servers-through-adapters-oll.md +43 -0
- package/docs/changelog.d/2026-11-17-lambda-spread-lambda-create-update-spread-and-spreadstrict-lambforge.yaml.md +16 -0
- package/docs/features/README.md +1 -0
- package/docs/features/lambda-spread.md +9 -0
- package/docs/features/local-ai-runners.md +55 -5
- package/package.json +1 -1
- package/release/index.js +550 -154
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: local-ai-runners
|
|
3
|
-
description: Work on the gcli local AI runner (runner:start/list/models/delete) - the subscriptions-transport-ws client that bridges a user's Ollama to the Lambforge chat. USE when the runner won't connect, drops requests, or before changing src/commands/runner/**.
|
|
3
|
+
description: Work on the gcli local AI runner (runner:start/list/models/delete) - the subscriptions-transport-ws client that bridges a user's local model servers (Ollama, Strata / OpenAI-compatible, via adapters) to the Lambforge chat. USE when the runner won't connect, drops requests, or before changing src/commands/runner/**.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# gcli local AI runner
|
|
@@ -22,6 +22,16 @@ security, symptom table): `graphql-server/docs/features/local-ai-runners.md`.
|
|
|
22
22
|
idle timeout kills it (spawned agent workers queue on one GPU).
|
|
23
23
|
- Socket close aborts every in-flight Ollama request; a refused token is
|
|
24
24
|
fatal (`RunnerAuthError`), everything else reconnects with backoff.
|
|
25
|
+
- Model servers are adapters (`src/commands/runner/adapters/`): the chat
|
|
26
|
+
format stays OpenAI `/v1/chat/completions`; adapters only discover models
|
|
27
|
+
and forward. The heartbeat's model objects must keep the
|
|
28
|
+
`LocalRunnerModelInput` shape (name, parameterSize, quantization, family,
|
|
29
|
+
capabilities, contextLength, runtimeContextLength) - adding a field there
|
|
30
|
+
breaks the GraphQL input unless graphql-server adds it too. The primary
|
|
31
|
+
(first) backend's model names must stay unprefixed: saved `runner:models`
|
|
32
|
+
selections depend on them.
|
|
33
|
+
- API keys of backends live only in the 0600 `runner.json` or env
|
|
34
|
+
(`$<NAME>_API_KEY`), never in the systemd unit or logs.
|
|
25
35
|
- Global `WebSocket` -> Node >= 22.
|
|
26
36
|
- Daemon (`runner:daemon:*`, `daemon.ts`): systemd USER unit; never put the
|
|
27
37
|
token or other secrets in the unit (it's world-readable-ish config) - it
|
package/README-CLI.md
CHANGED
|
@@ -5,7 +5,7 @@ Options:
|
|
|
5
5
|
-h, --help display help for command
|
|
6
6
|
|
|
7
7
|
Commands:
|
|
8
|
-
login [options]
|
|
8
|
+
login [options] Log in. Without --token: prints a link, opens your browser and waits for you to approve it on lambforge.com
|
|
9
9
|
lambda:list [options] List of all lambdas for project
|
|
10
10
|
lambda:get [options] Get lambda by id
|
|
11
11
|
lambda:create [options]
|
|
@@ -43,6 +43,7 @@ Commands:
|
|
|
43
43
|
cluster:nodes [options] Every node of a cluster: its Hetzner control planes and workers and the machines attached to it
|
|
44
44
|
cluster:pause [options] Pause an attached machine (cordon): nothing new runs there, running work stays
|
|
45
45
|
cluster:resume [options] Let new work run on a paused machine again (uncordon)
|
|
46
|
+
cluster:node-storage [options] Let an attached machine carry volumes on its own disk (it restarts once), or stop with --off
|
|
46
47
|
cluster:reset-machine [options] Bring a detached machine back to maintenance mode to attach it again (wipes its config and data, keeps the boot loader)
|
|
47
48
|
cluster:talosconfig [options] A short-lived talosconfig for one of your Talos clusters (admin of its machines with talosctl) - printed, or saved with -o
|
|
48
49
|
cluster:detach [options] Drain and remove an attached machine from a cluster (reset the machine afterwards)
|
|
@@ -52,6 +53,7 @@ Commands:
|
|
|
52
53
|
cluster:scale [options] Set the number of worker nodes (1-10) of a multi-node cluster - billed per node-hour
|
|
53
54
|
cluster:scale:control-planes [options] Set the number of control-plane nodes (1, 3 or 5)
|
|
54
55
|
cluster:autoscale [options] Change the worker autoscaling policy - only the flags you pass change
|
|
56
|
+
cluster:plan [options] Why pods are pending and what the pods-mode autoscaler would do: a verdict per pod, the nodes it would create, consolidation
|
|
55
57
|
cluster:multi-node [options] Convert a single-node cluster to dedicated worker nodes (required for autoscaling)
|
|
56
58
|
cluster:single-node [options] Convert a multi-node cluster back to a single node (removes the worker nodes)
|
|
57
59
|
cluster:load-balancer [options] Add or remove the cluster's load balancer (billed per hour while enabled). On a virtual cluster this is its dedicated LB - its own IP; the platform subdomain keeps working either way
|
|
@@ -62,7 +64,7 @@ Commands:
|
|
|
62
64
|
plugin:installed [options] List plugins installed for a project (may include the same plugin on several clusters)
|
|
63
65
|
plugin:install [options] Install a marketplace plugin for a project
|
|
64
66
|
plugin:uninstall [options] Uninstall a plugin from a project (--cluster picks which installation, if it's on several)
|
|
65
|
-
runner:start [options] Connect this machine's Ollama to the Lambforge chat as a local AI runner (outbound connection only - works behind NAT,
|
|
67
|
+
runner:start [options] Connect this machine's model servers (Ollama, Strata or any OpenAI-compatible server) to the Lambforge chat as a local AI runner (outbound connection only - works behind NAT, nothing is exposed)
|
|
66
68
|
runner:daemon:install|runner:start:daemon [options] Run the runner in the background as a systemd user service: starts with your session, restarts on failure (Linux)
|
|
67
69
|
runner:daemon:status [options] Show the runner service status (systemctl --user status)
|
|
68
70
|
runner:daemon:start [options] Start the installed runner service, and report whether it stays up
|
|
@@ -94,9 +96,10 @@ Commands:
|
|
|
94
96
|
--- Subcommand: login ---[0m
|
|
95
97
|
Usage: gcli login [options]
|
|
96
98
|
|
|
97
|
-
|
|
99
|
+
Log in. Without --token: prints a link, opens your browser and waits for you to approve it on lambforge.com
|
|
98
100
|
|
|
99
101
|
Options:
|
|
102
|
+
--no-browser Only print the login link, do not open a browser
|
|
100
103
|
-k, --key <key> Specify api key
|
|
101
104
|
-uu, --uploadUrl <key> Specify upload server
|
|
102
105
|
-ci, --ci Initialize directories for CI/CD purposes
|
|
@@ -371,12 +374,17 @@ Usage: gcli build [options]
|
|
|
371
374
|
Build bundle using esbuild https://esbuild.github.io
|
|
372
375
|
|
|
373
376
|
Options:
|
|
374
|
-
-f, --files <files...> File or files to bundle
|
|
375
|
-
|
|
376
|
-
-
|
|
377
|
+
-f, --files <files...> File or files to bundle (default: function.file
|
|
378
|
+
from lambforge.yaml, else index.ts)
|
|
379
|
+
-b, --bundle Bundle code (default: options.bundler.bundle,
|
|
380
|
+
else true)
|
|
381
|
+
-m, --minify Minify code (default: options.bundler.minify,
|
|
382
|
+
else false)
|
|
377
383
|
-p, --platform <char> Platform (default: "node")
|
|
378
|
-
-t, --target <char> Target
|
|
379
|
-
|
|
384
|
+
-t, --target <char> Target (default: options.bundler.target, else
|
|
385
|
+
node24)
|
|
386
|
+
-o, --outfile <char> Outfile name (default: options.bundler.outfile,
|
|
387
|
+
else index.js)
|
|
380
388
|
-e, --external <external...> External libraries
|
|
381
389
|
-h, --help display help for command
|
|
382
390
|
[36m
|
|
@@ -693,6 +701,8 @@ Options:
|
|
|
693
701
|
--reserved Only workloads that tolerate the pool run there
|
|
694
702
|
(default: plain extra capacity)
|
|
695
703
|
--disk <disk> Install disk, e.g. /dev/nvme0n1 (WIPED)
|
|
704
|
+
--storage The machine also carries volumes on its own disk
|
|
705
|
+
(default: lambdas only)
|
|
696
706
|
--yes Do not ask before wiping the disk
|
|
697
707
|
--no-wait Do not wait for the machine to join
|
|
698
708
|
--timeout <minutes> How long to wait for the join (default 10)
|
|
@@ -733,6 +743,22 @@ Options:
|
|
|
733
743
|
-n, --name <name> Machine name (see cluster:nodes)
|
|
734
744
|
-h, --help display help for command
|
|
735
745
|
[36m
|
|
746
|
+
--- Subcommand: cluster:node-storage ---[0m
|
|
747
|
+
Usage: gcli cluster:node-storage [options]
|
|
748
|
+
|
|
749
|
+
Let an attached machine carry volumes on its own disk (it restarts once), or stop with --off
|
|
750
|
+
|
|
751
|
+
Options:
|
|
752
|
+
-p, --project <project>
|
|
753
|
+
-c, --cluster <cluster> Cluster id (see cluster:list)
|
|
754
|
+
--clusterName <clusterName> Cluster name instead of --cluster
|
|
755
|
+
-n, --name <name> Machine name (see cluster:nodes)
|
|
756
|
+
--off Stop carrying volumes (refused while it holds
|
|
757
|
+
one)
|
|
758
|
+
--no-wait Do not wait until the machine is ready
|
|
759
|
+
--timeout <minutes> How long to wait (default 40)
|
|
760
|
+
-h, --help display help for command
|
|
761
|
+
[36m
|
|
736
762
|
--- Subcommand: cluster:reset-machine ---[0m
|
|
737
763
|
Usage: gcli cluster:reset-machine [options]
|
|
738
764
|
|
|
@@ -851,6 +877,31 @@ Options:
|
|
|
851
877
|
--memDown <percent> Scale down below this memory % (1-100)
|
|
852
878
|
--samples <n> Consecutive readings before acting (1-20)
|
|
853
879
|
--cooldown <minutes> Minutes between scale actions (1-1440)
|
|
880
|
+
--profile <profile> conservative | balanced | aggressive presets the
|
|
881
|
+
thresholds above; custom uses them as given
|
|
882
|
+
--serverTypes <types> Comma-separated worker server types in order of
|
|
883
|
+
preference, e.g. cx33,cpx22 (the next is used
|
|
884
|
+
when one is out of stock); "" = the cluster's
|
|
885
|
+
own type
|
|
886
|
+
--step <n> Workers added per scale-up (1-3, custom profile)
|
|
887
|
+
--mode <mode> thresholds (CPU/memory %) | pods (create the
|
|
888
|
+
cheapest nodes that fit unschedulable pods)
|
|
889
|
+
--consolidation pods mode: remove the newest worker when its
|
|
890
|
+
pods fit elsewhere
|
|
891
|
+
--no-consolidation pods mode: never remove workers by itself
|
|
892
|
+
--minNodeAge <minutes> pods mode: never remove a worker younger than
|
|
893
|
+
this (5-1440)
|
|
894
|
+
-h, --help display help for command
|
|
895
|
+
[36m
|
|
896
|
+
--- Subcommand: cluster:plan ---[0m
|
|
897
|
+
Usage: gcli cluster:plan [options]
|
|
898
|
+
|
|
899
|
+
Why pods are pending and what the pods-mode autoscaler would do: a verdict per pod, the nodes it would create, consolidation
|
|
900
|
+
|
|
901
|
+
Options:
|
|
902
|
+
-p, --project <project>
|
|
903
|
+
-c, --cluster <cluster> Cluster id (see cluster:list)
|
|
904
|
+
--clusterName <clusterName> Cluster name instead of --cluster
|
|
854
905
|
-h, --help display help for command
|
|
855
906
|
[36m
|
|
856
907
|
--- Subcommand: cluster:multi-node ---[0m
|
|
@@ -972,18 +1023,25 @@ Options:
|
|
|
972
1023
|
--- Subcommand: runner:start ---[0m
|
|
973
1024
|
Usage: gcli runner:start [options]
|
|
974
1025
|
|
|
975
|
-
Connect this machine's Ollama to the Lambforge chat as a local AI runner (outbound connection only - works behind NAT,
|
|
976
|
-
|
|
977
|
-
Options:
|
|
978
|
-
-n, --name <name>
|
|
979
|
-
|
|
980
|
-
-t, --token <token>
|
|
981
|
-
|
|
982
|
-
--ollama <url>
|
|
983
|
-
|
|
984
|
-
--
|
|
985
|
-
|
|
986
|
-
|
|
1026
|
+
Connect this machine's model servers (Ollama, Strata or any OpenAI-compatible server) to the Lambforge chat as a local AI runner (outbound connection only - works behind NAT, nothing is exposed)
|
|
1027
|
+
|
|
1028
|
+
Options:
|
|
1029
|
+
-n, --name <name> Runner name shown in the chat model picker
|
|
1030
|
+
(default: hostname)
|
|
1031
|
+
-t, --token <token> Runner token from Profile -> Local AI runners
|
|
1032
|
+
(instead of `gcli login`)
|
|
1033
|
+
--ollama <url> Ollama URL (default: $OLLAMA_HOST or
|
|
1034
|
+
http://127.0.0.1:11434)
|
|
1035
|
+
--backend <name=kind:url> Model server to serve from, repeatable; kinds:
|
|
1036
|
+
ollama, openai (Strata, llama-server, LM Studio,
|
|
1037
|
+
vLLM). E.g. --backend
|
|
1038
|
+
strata=openai:http://127.0.0.1:8080. With
|
|
1039
|
+
--ollama, Ollama comes first
|
|
1040
|
+
--backend-key <name=key> API key for a backend, repeatable (default:
|
|
1041
|
+
$<NAME>_API_KEY, e.g. $STRATA_API_KEY)
|
|
1042
|
+
--wsUrl <url> Lambforge subscriptions endpoint override
|
|
1043
|
+
--reset Forget the saved runner and register a new one
|
|
1044
|
+
-h, --help display help for command
|
|
987
1045
|
[36m
|
|
988
1046
|
--- Subcommand: runner:daemon:install ---[0m
|
|
989
1047
|
Usage: gcli runner:daemon:install|runner:start:daemon [options]
|
|
@@ -991,15 +1049,22 @@ Usage: gcli runner:daemon:install|runner:start:daemon [options]
|
|
|
991
1049
|
Run the runner in the background as a systemd user service: starts with your session, restarts on failure (Linux)
|
|
992
1050
|
|
|
993
1051
|
Options:
|
|
994
|
-
-n, --name <name>
|
|
995
|
-
-t, --token <token>
|
|
996
|
-
--ollama <url>
|
|
997
|
-
|
|
998
|
-
--
|
|
999
|
-
|
|
1000
|
-
|
|
1001
|
-
|
|
1002
|
-
|
|
1052
|
+
-n, --name <name> Runner name when registering (default: hostname)
|
|
1053
|
+
-t, --token <token> Runner token for a machine without `gcli login`
|
|
1054
|
+
--ollama <url> Ollama URL (default: $OLLAMA_HOST or
|
|
1055
|
+
http://127.0.0.1:11434)
|
|
1056
|
+
--backend <name=kind:url> Model server to serve from, repeatable; kinds:
|
|
1057
|
+
ollama, openai (Strata, llama-server, LM Studio,
|
|
1058
|
+
vLLM). E.g. --backend
|
|
1059
|
+
strata=openai:http://127.0.0.1:8080. With
|
|
1060
|
+
--ollama, Ollama comes first
|
|
1061
|
+
--backend-key <name=key> API key for a backend, repeatable (default:
|
|
1062
|
+
$<NAME>_API_KEY, e.g. $STRATA_API_KEY)
|
|
1063
|
+
--wsUrl <url> Lambforge subscriptions endpoint override
|
|
1064
|
+
--reset Forget the saved runner and register a new one
|
|
1065
|
+
--service <name> systemd unit name (default: lambforge-runner)
|
|
1066
|
+
--dry-run Print the unit file instead of installing it
|
|
1067
|
+
-h, --help display help for command
|
|
1003
1068
|
[36m
|
|
1004
1069
|
--- Subcommand: runner:daemon:status ---[0m
|
|
1005
1070
|
Usage: gcli runner:daemon:status [options]
|
package/README.md
CHANGED
|
@@ -409,6 +409,8 @@ function:
|
|
|
409
409
|
concurrency: 1
|
|
410
410
|
functionTimeout: 60
|
|
411
411
|
specializationTimeout: 120
|
|
412
|
+
spread: machines # replicas on different machines ('none' = the scheduler decides)
|
|
413
|
+
spreadStrict: false # true = a replica waits rather than share a machine
|
|
412
414
|
|
|
413
415
|
environment:
|
|
414
416
|
name: nodejs
|
|
@@ -432,6 +434,14 @@ options:
|
|
|
432
434
|
external: []
|
|
433
435
|
```
|
|
434
436
|
|
|
437
|
+
**Spread across machines** (newdeploy, plan 33): `spread: machines` puts the
|
|
438
|
+
replicas on different machines of the cluster, as evenly as possible, so a
|
|
439
|
+
machine going down takes one replica, not several (use it with
|
|
440
|
+
`minScale >= 2`). Soft by default: it never blocks a scale-up.
|
|
441
|
+
`spreadStrict: true` keeps a replica waiting rather than sharing a machine.
|
|
442
|
+
Flags: `--spread machines|none`, `--spreadStrict true|false`. Left out = the
|
|
443
|
+
lambda keeps what it has. Changing it restarts the replicas once.
|
|
444
|
+
|
|
435
445
|
The same file can also carry `cluster:`, `rabbitmq:`, `plugin:` and `triggers:` sections, read by `cluster:apply` / `cluster:create`, `rabbitmq:create`, `plugin:install` and `cron:create` respectively (each also accepts its own dedicated `--spec` file, e.g. `cluster.yaml`, if you'd rather keep them separate):
|
|
436
446
|
|
|
437
447
|
```yaml
|
|
@@ -1038,9 +1048,39 @@ running after logout and start it at boot, enable lingering once:
|
|
|
1038
1048
|
`sudo loginctl enable-linger $USER`. Preview the unit with `--dry-run`;
|
|
1039
1049
|
`--service <name>` installs several runners side by side.
|
|
1040
1050
|
|
|
1051
|
+
### Several model servers in one runner (Ollama + Strata, LM Studio, ...)
|
|
1052
|
+
|
|
1053
|
+
One runner can serve models from several local servers at once. Each one is a
|
|
1054
|
+
`--backend <name>=<kind>:<url>` (repeatable). Built-in kinds:
|
|
1055
|
+
|
|
1056
|
+
| Kind | Servers | Models from |
|
|
1057
|
+
|---|---|---|
|
|
1058
|
+
| `ollama` | Ollama | `/api/tags` + `/api/show` + `/api/ps` |
|
|
1059
|
+
| `openai` | any OpenAI-compatible server: Strata, llama.cpp `llama-server`, LM Studio, vLLM | `/v1/models` (+ `/props` for the running context window, when the server has it) |
|
|
1060
|
+
|
|
1061
|
+
```bash
|
|
1062
|
+
# Ollama + Strata in one runner
|
|
1063
|
+
gcli runner:start --ollama http://127.0.0.1:11434 \
|
|
1064
|
+
--backend strata=openai:http://127.0.0.1:8080
|
|
1065
|
+
# API key: --backend-key strata=<key>, or $STRATA_API_KEY ($<NAME>_API_KEY)
|
|
1066
|
+
|
|
1067
|
+
# the same as a service: backends + keys are saved in ~/.graphql-server/runner.json (0600)
|
|
1068
|
+
gcli runner:daemon:install --ollama http://127.0.0.1:11434 \
|
|
1069
|
+
--backend strata=openai:http://127.0.0.1:8080
|
|
1070
|
+
```
|
|
1071
|
+
|
|
1072
|
+
The first backend is the primary: its models keep their names. The others
|
|
1073
|
+
appear as `<name>/<model>` (e.g. `strata/Qwen3.8-Flash-Next`) in the chat
|
|
1074
|
+
picker and in `runner:models --show`. A backend that is down only removes its
|
|
1075
|
+
own models; the rest keep working. Without `--backend` the runner serves one
|
|
1076
|
+
Ollama, exactly as before. Re-running `runner:daemon:install` without
|
|
1077
|
+
`--backend` goes back to Ollama only.
|
|
1078
|
+
|
|
1041
1079
|
Notes:
|
|
1042
1080
|
|
|
1043
|
-
-
|
|
1081
|
+
- Ollama models are only offered if Ollama reports them as chat-capable;
|
|
1082
|
+
OpenAI-compatible servers don't report capabilities, so their models are
|
|
1083
|
+
offered with tools; a model
|
|
1044
1084
|
without the `tools` capability is used without platform tools.
|
|
1045
1085
|
- Local models don't count against your plan's AI token quota.
|
|
1046
1086
|
- Requires Node.js >= 22.
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
## 2026-11-15 - local-ai-runners: one runner serves several model servers through adapters (Ollama + OpenAI-compatible, e.g. Strata)
|
|
2
|
+
|
|
3
|
+
**Feature:** local-ai-runners · **Type:** feat
|
|
4
|
+
|
|
5
|
+
Why: a runner could only reach one Ollama. Models served by Strata (a llama.cpp-based engine) or any other OpenAI-compatible server (llama-server, LM Studio, vLLM) could not reach the Lambforge chat. Pointing `--ollama` at them did not work either: they have no `/api/tags`, `/api/show` or `/api/ps`.
|
|
6
|
+
|
|
7
|
+
- New `src/commands/runner/adapters/`:
|
|
8
|
+
- `ModelAdapter` (`types.ts`): an adapter owns `listModels()` and `forward()`. The format between Lambforge and the runner stays the OpenAI chat API (`POST /v1/chat/completions`). The allow-list is unchanged, so this is a gcli-only change.
|
|
9
|
+
- `HttpModelAdapter` (`base.ts`) handles the URL and the optional bearer key. Its default `forward` passes the request straight through.
|
|
10
|
+
- Built-in adapters:
|
|
11
|
+
- `OllamaAdapter` (`ollama.ts`) wraps the existing `OllamaModels`.
|
|
12
|
+
- `OpenAIAdapter` (`openai.ts`) reads `/v1/models`. The running context window comes from `meta.n_ctx`, llama.cpp's `/props` `n_ctx`, or vLLM's `max_model_len`.
|
|
13
|
+
- Capabilities use Ollama's names. When `/props` answers: `completion`, plus `tools` (`chat_template_caps.supports_tools` / `supports_tool_calls`), plus `vision`. Without `/props`: `[]`. An empty list would work in the backend, but the web app's runner page shows "chat only - no agent tools" for it; that label is what showed up first with Strata.
|
|
14
|
+
- llama.cpp lists a model again under each alias (Strata: `qwen3.8-flash-next-iq3_s` and `iq3_s`). The adapter keeps the first entry only.
|
|
15
|
+
- `registry.ts`: `registerAdapter(kind, factory)`. A new kind of server needs one class and one registration line.
|
|
16
|
+
- `ModelRouter` (`router.ts`):
|
|
17
|
+
- Merges every backend's models into one heartbeat. The first backend is the primary and keeps its model names; the others are shown as `<backend>/<model>`.
|
|
18
|
+
- Routes each request by its `model` and renames the model back before forwarding. The request body is rewritten only when the name differs.
|
|
19
|
+
- Tracks reachability per backend: a backend that is down only removes its own models.
|
|
20
|
+
- `spec.ts` handles `--backend <name>=<kind>:<url>` parsing and `resolveBackends`. Resolution order: `--backend` (`--ollama` is prepended), then `runner.json.backends`, then one Ollama. API keys come from `--backend-key`, then `$<NAME>_API_KEY`, then the saved key.
|
|
21
|
+
- `runner:start` and `runner:daemon:install` gain the repeatable `--backend` and `--backend-key` flags (`index.ts`):
|
|
22
|
+
- `start.ts` logs `Backends: ...`; a set key shows as `[key]`.
|
|
23
|
+
- `daemon-commands.ts` saves the resolved backends, keys included, in the 0600 `runner.json`, and the unit then runs plain `runner:start`. Keys never go into the unit. Re-installing without `--backend` drops the saved backends.
|
|
24
|
+
- `RunnerConnection` (`connection.ts`) takes `backends` or ready-made `adapters`; the old `ollamaUrl` still works. Errors name the backend, e.g. "Could not reach strata (openai) at ...".
|
|
25
|
+
- Backward compatible: without `--backend`, the heartbeat, the requests and the request bodies are byte-for-byte what they were. Existing units (`runner:start --ollama <url>`) behave the same.
|
|
26
|
+
- No graphql-server change is needed: the model objects keep the `LocalRunnerModelInput` fields, and `/` in model names was already valid (e.g. `x/z-image-turbo:fp8`).
|
|
27
|
+
- Docs updated: README ("Several model servers in one runner"), `docs/features/local-ai-runners.md` ("Backends (model adapters)"), and the `local-ai-runners` skill (invariants).
|
|
28
|
+
- **Test:**
|
|
29
|
+
- `npx jest --coverage=false test/runner-adapters.spec.ts test/runner.spec.ts`. The adapter spec covers `--backend` parsing and errors, key precedence, the OpenAI adapter with and without `/props`, the registry with a custom kind, and the router (naming, renaming, prefix fallback, a down backend, duplicate names). `runner.spec.ts` adds end to end through the socket: one heartbeat carrying both backends' models; a `strata/` model sent to the second backend under its own name with the bearer key; Ollama kept serving while the second backend is down.
|
|
30
|
+
- Live: `gcli runner:start --ollama http://127.0.0.1:11434 --backend strata=openai:http://127.0.0.1:8080`, then pick `strata/<model>` in the chat.
|
|
31
|
+
- **Debug:**
|
|
32
|
+
- Startup line `Backends: ...` shows what was resolved.
|
|
33
|
+
- `Cannot reach <backend> (<kind>) at <url> (...)` is logged once per outage per backend; `<backend> at <url>: N model(s)` when it comes back.
|
|
34
|
+
- `-> chat request (<model>) via <backend>` shows the routing.
|
|
35
|
+
- A model missing from the picker: check that its server answers `/v1/models`. A 401 there means the key is missing or wrong (`--backend-key` / `$<NAME>_API_KEY`).
|
|
36
|
+
- A `strata/...` model that reaches the wrong server: the backend name must match the prefix.
|
|
37
|
+
- **Verification:**
|
|
38
|
+
- jest: 20 suites, 169 tests pass. `tsc --noEmit` and eslint are clean.
|
|
39
|
+
- `npm run build`: `runner:start --help` lists the flags. `runner:daemon:install --dry-run --backend strata=openai:http://127.0.0.1:8080` gives `ExecStart=... runner:start` (no key); without `--backend` it is unchanged (`--ollama http://127.0.0.1:11434`).
|
|
40
|
+
- Live, against the real Ollama with the router and both adapters: 3 models listed with capabilities and context, and a chat request through `forward` returned 200. Strata on :8080 was not running (its model was still downloading); it was reported as unreachable while Ollama kept working.
|
|
41
|
+
- Live with Strata (port 18500, with key) through the `lambforge-runner` service: `gcli runner:list` shows `qwen3.8:27b, gpt-oss-128k:latest, gpt-oss:20b, strata/qwen3.8-flash-next-iq3_s`. The Strata model reports `completion, tools, vision` and runtime context 131072.
|
|
42
|
+
- Not run: a chat request to the Strata model through the Lambforge chat.
|
|
43
|
+
- jest after the capabilities and alias fix: 20 suites, 171 tests pass.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
## 2026-11-17 - lambda-spread: lambda:create/update --spread and --spreadStrict, lambforge.yaml spread keys
|
|
2
|
+
|
|
3
|
+
**Feature:** lambda-spread · **Type:** feat
|
|
4
|
+
|
|
5
|
+
- `lambda:create` / `lambda:update` take `--spread machines|none` and
|
|
6
|
+
`--spreadStrict true|false`; `lambforge.yaml` takes
|
|
7
|
+
`function.scaleOptions.spread` / `spreadStrict`. Flags win over the yaml.
|
|
8
|
+
Sent only when set (`helpers/spread.ts` `spreadScaleOptions`), so gcli
|
|
9
|
+
works against an API without the fields and an update without them keeps
|
|
10
|
+
the lambda's current choice. Canonical doc: graphql-server-lambdas
|
|
11
|
+
`docs/features/lambda-spread.md`.
|
|
12
|
+
- **Test:** `npx jest --coverage=false test/lambda-spread.spec.ts`.
|
|
13
|
+
- **Debug:** "--spread must be 'machines' or 'none'" = a typo in the flag or
|
|
14
|
+
yaml. An API older than the feature answers `Field "spread" is not defined
|
|
15
|
+
by type "LambdaScaleInputOptions"`.
|
|
16
|
+
- **Verification:** jest 174/174, tsc clean.
|
package/docs/features/README.md
CHANGED
|
@@ -42,3 +42,4 @@ Outline (drop sections that don't apply):
|
|
|
42
42
|
| `mcp-proxy` | [mcp-proxy.md](mcp-proxy.md) | `gcli mcp:start` - local stdio MCP server forwarding to the platform MCP endpoint; token refreshed per request |
|
|
43
43
|
| `cli-browser-login` | [cli-browser-login.md](cli-browser-login.md) | Slice (canonical: graphql-server): `gcli login` with no token through the browser; `mcp:start` LambforgeLogin |
|
|
44
44
|
| `lambda-site-url` | [lambda-site-url.md](lambda-site-url.md) | Slice (canonical: graphql-server-lambdas): `Site:` line after create/update |
|
|
45
|
+
| `lambda-spread` | [lambda-spread.md](lambda-spread.md) | Slice (canonical: graphql-server-lambdas): replicas across machines (plan 33): `--spread` / `--spreadStrict`, yaml keys |
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
# lambda-spread (gcli slice)
|
|
2
|
+
|
|
3
|
+
Canonical: graphql-server-lambdas `docs/features/lambda-spread.md` (plan 33).
|
|
4
|
+
|
|
5
|
+
- Flags: `--spread machines|none`, `--spreadStrict true|false` on
|
|
6
|
+
`lambda:create` / `lambda:update`.
|
|
7
|
+
- yaml: `function.scaleOptions.spread`, `function.scaleOptions.spreadStrict`.
|
|
8
|
+
- `src/commands/lambda/helpers/spread.ts` - only set values are sent; flags
|
|
9
|
+
win over the yaml; unknown values refused before the call.
|
|
@@ -9,11 +9,11 @@ what lives in gcli.
|
|
|
9
9
|
|
|
10
10
|
| Command | What it does |
|
|
11
11
|
|---|---|
|
|
12
|
-
| `gcli runner:start [--name] [--token] [--ollama] [--wsUrl] [--reset]` | Connects this machine's Ollama to the chat. First run registers a runner with the `gcli login` session and saves it to `~/.graphql-server/runner.json` (0600); `--token` instead uses a token from Profile -> Local AI runners (headless machines); `--reset` forgets the saved runner and registers a new one. Runs until Ctrl+C, reconnecting with backoff (1s -> 30s). |
|
|
12
|
+
| `gcli runner:start [--name] [--token] [--ollama] [--backend name=kind:url]... [--backend-key name=key]... [--wsUrl] [--reset]` | Connects this machine's model servers (Ollama by default; see "Backends") to the chat. First run registers a runner with the `gcli login` session and saves it to `~/.graphql-server/runner.json` (0600); `--token` instead uses a token from Profile -> Local AI runners (headless machines); `--reset` forgets the saved runner and registers a new one. Runs until Ctrl+C, reconnecting with backoff (1s -> 30s). |
|
|
13
13
|
| `gcli runner:list` | Your runners: online, last seen, models |
|
|
14
14
|
| `gcli runner:models [--id] [--show a,b \| --all \| --none]` | Which models appear in the chat picker; no flag = list |
|
|
15
15
|
| `gcli runner:delete [--id]` | Delete a runner and revoke its token (default: the saved one) |
|
|
16
|
-
| `gcli runner:daemon:install` (alias `runner:start:daemon`) `[runner:start options] [--service] [--dry-run]` | Resolve/register the runner now (saves `runner.json`), then install + enable + restart a systemd USER unit
|
|
16
|
+
| `gcli runner:daemon:install` (alias `runner:start:daemon`) `[runner:start options] [--service] [--dry-run]` | Resolve/register the runner now (saves `runner.json`), then install + enable + restart a systemd USER unit (Linux). One Ollama: the unit runs `runner:start --ollama <resolved>`. With `--backend`: the resolved backends (keys included) are saved in `runner.json.backends` and the unit runs plain `runner:start` - keys never go in the unit. Re-installing without `--backend` drops saved backends |
|
|
17
17
|
| `gcli runner:daemon:start` / `:restart` `[--service]` | `systemctl --user start/restart`, then after ~2 s reads `ActiveState`/`ExecMainStatus` (`SystemdUserService.outcome`): running, or an error - exit 78 = token refused (deleted, or registered on another server than `~/.graphql-server/url`) with the `runner:daemon:install --reset` hint. Refuses when the unit is not installed |
|
|
18
18
|
| `gcli runner:daemon:stop` `[--service]` | `systemctl --user stop` (stays enabled: starts again at the next login) |
|
|
19
19
|
| `gcli runner:daemon:status` / `:logs` / `:uninstall` | `systemctl --user status`, `journalctl --user -f`, disable --now + remove the unit |
|
|
@@ -24,6 +24,46 @@ Endpoints: `--wsUrl` > `runner.json` > derived from the `gcli login` URL
|
|
|
24
24
|
Ollama: `--ollama` > `$OLLAMA_HOST` (scheme optional, `0.0.0.0` -> `127.0.0.1`)
|
|
25
25
|
> `http://127.0.0.1:11434`. Requires Node >= 22 (global `WebSocket`).
|
|
26
26
|
|
|
27
|
+
## Backends (model adapters)
|
|
28
|
+
|
|
29
|
+
The wire format between Lambforge and the runner is always the OpenAI chat
|
|
30
|
+
API (`POST /v1/chat/completions`, the only allowed path - unchanged). A
|
|
31
|
+
**backend** is one local model server; its **adapter** owns only what differs
|
|
32
|
+
per server: discovering models (`listModels`) and forwarding a chat request
|
|
33
|
+
(`forward`, a pass-through for both built-ins). The backend (graphql-server)
|
|
34
|
+
is unaware of adapters: it still sees one runner and one model list, with the
|
|
35
|
+
same `LocalRunnerModelInput` fields.
|
|
36
|
+
|
|
37
|
+
- Built-in kinds: `ollama` (`/api/tags`, `/api/show`, `/api/ps`) and `openai`
|
|
38
|
+
(`/v1/models`; `runtimeContextLength` from llama.cpp's `/props` `n_ctx`,
|
|
39
|
+
else vLLM's `max_model_len`; `contextLength` from `meta.n_ctx_train` /
|
|
40
|
+
`context_length` / `max_model_len`; capabilities from `/props`:
|
|
41
|
+
`completion` + `tools` when `chat_template_caps.supports_tools` /
|
|
42
|
+
`supports_tool_calls`, + `vision` when `modalities.vision` or an image
|
|
43
|
+
input modality - no `/props` = `[]`, which the backend treats as
|
|
44
|
+
tool-capable but the web app's runner page labels "chat only"; alias
|
|
45
|
+
entries of one model are dropped, the first wins). Strata's engine is
|
|
46
|
+
llama.cpp-based: the port and the required bearer key are per instance in
|
|
47
|
+
`~/.strata/data/native-instances/<uuid>.json` (`port`, `api_key`) - e.g.
|
|
48
|
+
18500, not the 8080 of its install file.
|
|
49
|
+
- Resolution (`adapters/spec.ts` `resolveBackends`): `--backend` (with
|
|
50
|
+
`--ollama` prepended as `ollama`) > `runner.json.backends` (ignored when
|
|
51
|
+
`--ollama` is given - that's what single-Ollama units run) > one Ollama.
|
|
52
|
+
Keys: `--backend-key` > `$<NAME>_API_KEY` > saved key. Logs show `[key]`,
|
|
53
|
+
never the key.
|
|
54
|
+
- Naming (`adapters/router.ts` `ModelRouter`): the first backend is primary
|
|
55
|
+
and keeps its model names (a plain Ollama runner reports exactly what it
|
|
56
|
+
always did, so saved `runner:models` selections keep working); the others
|
|
57
|
+
are `<backend>/<model>`. `route()` maps a request's `model` back to the
|
|
58
|
+
backend and its own name (body rewritten only then); unknown models go by
|
|
59
|
+
prefix, else to the primary, unchanged. Each backend's reachability is
|
|
60
|
+
tracked and logged separately; a down backend only drops its own models.
|
|
61
|
+
- New kind: implement `ModelAdapter` (or extend `HttpModelAdapter`) in
|
|
62
|
+
`adapters/` and `registerAdapter('<kind>', factory)` in
|
|
63
|
+
`adapters/registry.ts`. An adapter for a server with a different chat API
|
|
64
|
+
overrides `forward` and translates request and stream to/from the OpenAI
|
|
65
|
+
format.
|
|
66
|
+
|
|
27
67
|
## Code
|
|
28
68
|
|
|
29
69
|
`src/commands/runner/`:
|
|
@@ -36,6 +76,13 @@ Ollama: `--ollama` > `$OLLAMA_HOST` (scheme optional, `0.0.0.0` -> `127.0.0.1`)
|
|
|
36
76
|
30s while Ollama keeps a request queued. `FrameSender` numbers frames
|
|
37
77
|
(`seq`), merges consecutive chunks (<= 32KB) and sends batches every 50ms,
|
|
38
78
|
sequentially. `RunnerAuthError` = token refused -> exit 1 with a `--reset` hint.
|
|
79
|
+
- `adapters/` - `types.ts` (`ModelAdapter`, `AdapterSpec`), `base.ts`
|
|
80
|
+
(`HttpModelAdapter`: URL, bearer key, pass-through `forward`),
|
|
81
|
+
`ollama.ts` (`OllamaAdapter` over `OllamaModels`), `openai.ts`
|
|
82
|
+
(`OpenAIAdapter`), `registry.ts`, `router.ts` (`ModelRouter`), `spec.ts`
|
|
83
|
+
(`--backend` parsing, `resolveBackends`). `RunnerConnection` takes
|
|
84
|
+
`backends` (or ready `adapters`; the old `ollamaUrl` still works) and
|
|
85
|
+
delegates heartbeat models and request routing to the router.
|
|
39
86
|
- `ollama.ts` - `ALLOWED_REQUESTS` (only `POST /v1/chat/completions`; the
|
|
40
87
|
backend has its own copy), `OllamaModels` (`/api/tags` + cached
|
|
41
88
|
`/api/show` per digest; drops models without the `completion` capability
|
|
@@ -64,7 +111,8 @@ Ollama: `--ollama` > `$OLLAMA_HOST` (scheme optional, `0.0.0.0` -> `127.0.0.1`)
|
|
|
64
111
|
## How to test
|
|
65
112
|
|
|
66
113
|
```bash
|
|
67
|
-
npx jest test/runner.spec.ts --coverage=false # fake server socket + fake streaming Ollama
|
|
114
|
+
npx jest test/runner.spec.ts --coverage=false # fake server socket + fake streaming Ollama (+ a Strata-like 2nd backend)
|
|
115
|
+
npx jest test/runner-adapters.spec.ts --coverage=false # --backend parsing, keys, OpenAI adapter, registry, router
|
|
68
116
|
npm run build && node release/index.js runner:start --help
|
|
69
117
|
```
|
|
70
118
|
|
|
@@ -78,7 +126,9 @@ that from a feature branch unless you are releasing.
|
|
|
78
126
|
## How to debug
|
|
79
127
|
|
|
80
128
|
Runner log lines: `-> chat request (<model>)`, `<- <status> (<model>) in Ns`,
|
|
81
|
-
`x cancelled`, `x <error>`, `Heartbeat failed: ...`, `Cannot reach
|
|
82
|
-
at
|
|
129
|
+
`x cancelled`, `x <error>`, `Heartbeat failed: ...`, `Cannot reach <backend>
|
|
130
|
+
(<kind>) at <url> (...)` (once per outage, per backend), `<backend> at <url>: N
|
|
131
|
+
model(s)`. With several backends, `-> chat request (<model>) via <backend>`.
|
|
132
|
+
`Backends: ...` at startup shows what was resolved (`[key]` = a key is set). `Disconnected from Lambforge; reconnecting in Ns` = socket closed
|
|
83
133
|
(backend restart, network). For the backend side and the symptom table see
|
|
84
134
|
the canonical doc.
|