@octocodeai/freellama-mcp-server 0.1.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONFIGURATION.md +108 -0
- package/DEVELOPING.md +109 -0
- package/LICENSE-APACHE +202 -0
- package/LICENSE-MIT +18 -0
- package/README.md +261 -243
- package/assets/logo.jpg +0 -0
- package/dist/index.js +2305 -1332
- package/dist/research/bash-agent.js +2435 -0
- package/dist/research/octocode-agent.js +1639 -0
- package/docs/ARCHITECTURE.md +294 -87
- package/docs/CLI.md +171 -53
- package/docs/CPU_GPU_ROUTING.md +69 -23
- package/docs/MODEL_SELECTION.md +126 -3
- package/docs/MONITORING.md +293 -0
- package/docs/PRODUCTION.md +53 -25
- package/docs/README.md +36 -0
- package/docs/SCOPES_AND_WARMING.md +225 -0
- package/docs/TESTING.md +113 -8
- package/native/index.d.ts +186 -150
- package/native/index.js +3 -3
- package/package.json +15 -11
- package/adapters/agent_context.py +0 -663
- package/adapters/agent_transport.py +0 -84
- package/adapters/bash_agent.py +0 -370
- package/adapters/octocode_agent.py +0 -394
- package/docs/ADR_RESOURCE_AWARE_BACKEND_ROUTING.md +0 -55
- package/docs/ECONOMICS.md +0 -105
- package/docs/INDEX.md +0 -15
- package/docs/OLLAMA_SIDECAR.md +0 -76
- package/docs/OLLAMA_SYSTEM_OPTIMIZATION.md +0 -254
- package/docs/PRODUCT_POSITIONING.md +0 -291
package/docs/README.md
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# FreeLlama documentation
|
|
2
|
+
|
|
3
|
+
FreeLlama helps agents offload bounded work to local models through Ollama. The calling agent owns
|
|
4
|
+
the task and verifies the answer; FreeLlama manages model selection and resources around execution.
|
|
5
|
+
Start with the [repository MCP setup](https://github.com/bgauryy/FreeLlama/blob/main/README.md#connect-your-agent-mcp) to run one managed task.
|
|
6
|
+
|
|
7
|
+
## Choose a guide
|
|
8
|
+
|
|
9
|
+
| Goal | Guide |
|
|
10
|
+
|---|---|
|
|
11
|
+
| Connect an agent and choose a tool | [MCP server](https://github.com/bgauryy/FreeLlama/blob/main/packages/mcp/README.md) |
|
|
12
|
+
| Follow the agent's operating workflow | [FreeLlama skill](https://github.com/bgauryy/FreeLlama/blob/main/skills/freellama/SKILL.md) |
|
|
13
|
+
| Look up commands and task controls | [CLI reference](CLI.md) |
|
|
14
|
+
| Understand task and resource ownership | [Architecture](ARCHITECTURE.md) |
|
|
15
|
+
| Choose a model and interpret its metadata | [Model selection](MODEL_SELECTION.md), [model metadata](MODEL_SELECTION.md#model-metadata) |
|
|
16
|
+
| Configure CPU helpers alongside GPU work | [CPU/GPU routing](CPU_GPU_ROUTING.md) |
|
|
17
|
+
| Inspect queues, memory, usage, and live settings | [Monitoring](MONITORING.md) |
|
|
18
|
+
| Retain history, affinity, or residency | [Scopes and warming](SCOPES_AND_WARMING.md) |
|
|
19
|
+
| Understand raw Ollama compatibility | [Ollama sidecar](ARCHITECTURE.md#ollama-sidecar) |
|
|
20
|
+
| Evaluate context offload and its cost | [Token economics](https://github.com/bgauryy/FreeLlama/blob/main/packages/mcp/README.md#token-economics) |
|
|
21
|
+
| Deploy the management layer | [Production runbook](PRODUCTION.md) |
|
|
22
|
+
| Run development and release checks | [Testing](TESTING.md), [release procedure](https://github.com/bgauryy/FreeLlama/blob/main/RELEASE.md) |
|
|
23
|
+
| Embed the routing core | [Rust core](https://github.com/bgauryy/FreeLlama/blob/main/packages/rust-core/README.md) |
|
|
24
|
+
| Use the read-only status page | `/_freellama/ui` on a running `serve` (see [Monitoring](MONITORING.md)) |
|
|
25
|
+
|
|
26
|
+
## Evidence and design
|
|
27
|
+
|
|
28
|
+
| Goal | Source |
|
|
29
|
+
|---|---|
|
|
30
|
+
| Review dated findings and project comparisons | [Findings report](dev/FINDINGS_AND_POSITIONING.md) |
|
|
31
|
+
| Measure inference, device activity, and thermal behavior | [Local performance testing](dev/LOCAL_PERFORMANCE_TESTING.md) |
|
|
32
|
+
| Separate management controls from inference tuning | [Ollama optimization boundary](dev/OLLAMA_SYSTEM_OPTIMIZATION.md) |
|
|
33
|
+
| Understand backend selection decisions | [Resource-routing ADR](dev/ADR_RESOURCE_AWARE_BACKEND_ROUTING.md) |
|
|
34
|
+
|
|
35
|
+
Historical measurements describe one model, machine, and workload. Inspect current state and
|
|
36
|
+
repeat the relevant checks before using them to select models or set resource limits.
|
|
@@ -0,0 +1,225 @@
|
|
|
1
|
+
# Scope history and model warming
|
|
2
|
+
|
|
3
|
+
Use scopes when delegated chat tasks need retained message history, and warming when an installed
|
|
4
|
+
model needs preparation before related work. FreeLlama manages these controls through the same
|
|
5
|
+
admission and memory checks as task execution. The calling agent owns conversation instructions,
|
|
6
|
+
task dependencies, and verification.
|
|
7
|
+
|
|
8
|
+
## Ownership
|
|
9
|
+
|
|
10
|
+
| State | Owner | Lifetime and meaning |
|
|
11
|
+
|---|---|---|
|
|
12
|
+
| Session affinity | FreeLlama `session` | Bounded, idle-expiring model preference; stores no messages or KV. |
|
|
13
|
+
| Scope history | FreeLlama `scope` | Bounded process-local messages and routing defaults, protected by a revision. |
|
|
14
|
+
| Runner residency | Ollama | Controlled by `keep_alive`, loading, eviction, and available resources. |
|
|
15
|
+
| Prefix cache and KV | Ollama and its inference backend | Opportunistic reuse of compatible requests; no scope-owned KV save/restore contract. |
|
|
16
|
+
|
|
17
|
+
Reusing a scope replays its stored messages. Switching scope IDs does not transfer KV tensors.
|
|
18
|
+
Keeping an identical message prefix can preserve cache reuse when the backend supports it; changing
|
|
19
|
+
the model, runner profile, or prefix can invalidate that reuse. A resident model does not imply that
|
|
20
|
+
the requested history is cached. Cache counts remain unknown when Ollama does not report them.
|
|
21
|
+
|
|
22
|
+
## Scope API
|
|
23
|
+
|
|
24
|
+
All HTTP paths below are relative to `/_freellama/v1`. Authentication matches the other control
|
|
25
|
+
routes. Bodies reject unknown fields.
|
|
26
|
+
|
|
27
|
+
| Operation | HTTP | MCP | CLI |
|
|
28
|
+
|---|---|---|---|
|
|
29
|
+
| Create | `POST /scopes` | `scope {action:"create"}` | `scope create` |
|
|
30
|
+
| Inspect metadata | `GET /scopes/:id` | `scope {action:"get",scopeId}` | `scope get --scope-id SCOPE_ID` |
|
|
31
|
+
| Inspect history | `GET /scopes/:id?include_messages=true` | `scope {action:"get",scopeId,includeMessages:true}` | `scope get --scope-id SCOPE_ID --include-messages` |
|
|
32
|
+
| Fork a snapshot | `POST /scopes/:id/fork` | `scope {action:"fork",scopeId,revision}` | `scope fork --scope-id SCOPE_ID --revision REVISION` |
|
|
33
|
+
| Delete | `DELETE /scopes/:id` | `scope {action:"delete",scopeId}` | `scope delete --scope-id SCOPE_ID` |
|
|
34
|
+
|
|
35
|
+
Replace `SCOPE_ID` with the returned scope identifier and `REVISION` with its current revision.
|
|
36
|
+
Deletion returns HTTP `204`. Create and fork return HTTP `201` with metadata; history requires an
|
|
37
|
+
explicit read. CLI create and fork accept a JSON object through `--json`.
|
|
38
|
+
|
|
39
|
+
Create accepts the following optional fields:
|
|
40
|
+
|
|
41
|
+
| HTTP field | MCP field | Meaning |
|
|
42
|
+
|---|---|---|
|
|
43
|
+
| `messages` | `messages` | Initial Ollama messages, preserving fields, content, and order. |
|
|
44
|
+
| `route_defaults` | — | Task-routing defaults for later scoped execution (HTTP/CLI only). |
|
|
45
|
+
| `limits` | — | Per-scope bounds within the server's configured limits (HTTP/CLI only). |
|
|
46
|
+
|
|
47
|
+
Messages require a supported role and can omit content, including for assistant tool calls.
|
|
48
|
+
When present, content must be a string.
|
|
49
|
+
|
|
50
|
+
Routing defaults accept `task`, `objective`, `model`, `required_capabilities`, `context_tokens`,
|
|
51
|
+
`execution_preference`, `min_placement_evidence`, and `min_confidence`. MCP callers set routing per
|
|
52
|
+
`run_task` call instead. Defaults do not grant admission or qualify a model. Explicit task fields
|
|
53
|
+
take precedence, and execution checks current eligibility and resources.
|
|
54
|
+
|
|
55
|
+
The metadata receipt contains `scope_id`, `revision`, `route_defaults`, `limits`, `message_count`,
|
|
56
|
+
`bytes`, `estimated_tokens`, `expires_at`, and `storage:"process_local"`. Full history can include images, thinking, and tool calls; request it only when needed. Counts and estimates
|
|
57
|
+
describe the stored transcript; they are not an exact tokenizer result or runner-memory estimate.
|
|
58
|
+
Token estimates derive from the full serialized message bytes and count both media and extra fields.
|
|
59
|
+
This storage-budget estimate can refuse media-heavy history before inference; it does not measure
|
|
60
|
+
image token cost or determine the model context required to execute the history.
|
|
61
|
+
|
|
62
|
+
Fork requires `revision`; HTTP/CLI forks also accept replacement routing defaults and limit overrides. It creates a
|
|
63
|
+
separate history snapshot with revision `0`; later writes to either scope do not change the other.
|
|
64
|
+
Fork independent branches before parallel execution instead of submitting concurrent writes to one
|
|
65
|
+
scope.
|
|
66
|
+
|
|
67
|
+
## Scoped task execution
|
|
68
|
+
|
|
69
|
+
HTTP `POST /tasks` accepts `scope_id` and `scope_revision` together. MCP `run_task` uses `scopeId`
|
|
70
|
+
and `scopeRevision`; CLI `task` uses `--scope-id` and `--scope-revision`. Send only the new caller
|
|
71
|
+
messages or prompt. FreeLlama prepends the stored transcript and appends the successful assistant
|
|
72
|
+
response to that scope. Embedding tasks and embedding inputs cannot use scopes. MCP previews
|
|
73
|
+
reject scope references; use a separate execution call.
|
|
74
|
+
|
|
75
|
+
The expected revision prevents a stale caller from overwriting another result. A scope permits one
|
|
76
|
+
active append; a competing write fails before inference. A successful task returns a `scope`
|
|
77
|
+
metadata receipt with the next revision. Failed or cancelled tasks do not append history. A scope
|
|
78
|
+
deleted or expired during execution cannot receive the result. History limits also apply to the
|
|
79
|
+
completed assistant message: a response that exceeds them returns an append error with the prior
|
|
80
|
+
revision intact even though inference has run. Inspect the failure before retrying.
|
|
81
|
+
|
|
82
|
+
Deleting a session releases its affinity without deleting scope history or invalidating an
|
|
83
|
+
already-running scope append. Killing a session cancels its associated work; cancelled work does
|
|
84
|
+
not append history.
|
|
85
|
+
|
|
86
|
+
Limits reject an oversized transcript instead of silently deleting earlier instructions. If a task
|
|
87
|
+
cannot fit, narrow its input, create a compact replacement scope, or fork a suitable snapshot. The
|
|
88
|
+
caller owns any semantic summary and its verification. Stored history also contributes to managed
|
|
89
|
+
context sizing; the selected model's context boundary still applies. Scoped image or audio input
|
|
90
|
+
requires explicit `context_tokens` (`contextTokens` through MCP). Its context receipt estimates
|
|
91
|
+
text and reports total multimodal token cost as unknown. Encoded media is retained in history,
|
|
92
|
+
but its bytes are not counted as text tokens. Text plus the finite output reserve must fit the
|
|
93
|
+
requested window; Ollama determines the remaining model-specific fit with truncation and shifting
|
|
94
|
+
disabled. A passing estimate alone does not prove the complete multimodal input fits.
|
|
95
|
+
|
|
96
|
+
Scopes are process-local. Restarting `serve` discards them. Scope identifiers are handles within one
|
|
97
|
+
operator service, not tenant authorization credentials. Apply external tenant isolation where
|
|
98
|
+
required; see [Production deployment](PRODUCTION.md).
|
|
99
|
+
|
|
100
|
+
Scope failures retain a stable code:
|
|
101
|
+
|
|
102
|
+
| Status | Code | Meaning |
|
|
103
|
+
|---|---|---|
|
|
104
|
+
| `404` | `scope_not_found` | Missing, deleted, or expired scope. |
|
|
105
|
+
| `409` | `scope_revision_conflict` | The expected revision differs from the stored revision. |
|
|
106
|
+
| `409` | `scope_busy` | Another append owns the scope; fork for parallel work. |
|
|
107
|
+
| `413` | `scope_history_limit_exceeded` | Complete history exceeds a per-scope bound. |
|
|
108
|
+
| `422` | `scope_reference_invalid` | Scope identifier and expected revision were not supplied together. |
|
|
109
|
+
| `422` | `scope_limit_invalid` | A requested limit exceeds operator caps or is zero. |
|
|
110
|
+
| `422` | `scope_embedding_unsupported` | An embedding task or input attempted to use history. |
|
|
111
|
+
| `429` | `scope_capacity_exceeded` | The store has reached its count or aggregate-byte bound. |
|
|
112
|
+
| `502` | `scope_response_invalid` | A completed valid assistant message was not returned. |
|
|
113
|
+
|
|
114
|
+
A minimal CLI sequence creates a prefix and submits its first turn:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
npx @octocodeai/freellama scope create \
|
|
118
|
+
--json '{"messages":[{"role":"system","content":"Answer concisely."}]}'
|
|
119
|
+
npx @octocodeai/freellama task --scope-id SCOPE_ID --scope-revision 0 \
|
|
120
|
+
"Explain the previous instruction."
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Replace `SCOPE_ID` with the identifier returned by `scope create`; use the returned revision for
|
|
124
|
+
each subsequent task.
|
|
125
|
+
|
|
126
|
+
## Scope configuration
|
|
127
|
+
|
|
128
|
+
The runtime file accepts a `[scopes]` table. Matching `FREELLAMA_SCOPE_*` environment values take
|
|
129
|
+
precedence. Inspect effective values and their sources with
|
|
130
|
+
`config`; reload behavior follows [Runtime configuration](MONITORING.md#change-settings-without-a-restart).
|
|
131
|
+
|
|
132
|
+
| Setting | Environment variable | Default | Meaning |
|
|
133
|
+
|---|---|---:|---|
|
|
134
|
+
| `max_count` | `FREELLAMA_SCOPE_MAX_COUNT` | `128` | Maximum retained scopes. |
|
|
135
|
+
| `max_messages` | `FREELLAMA_SCOPE_MAX_MESSAGES` | `256` | Maximum messages in one scope. |
|
|
136
|
+
| `max_bytes` | `FREELLAMA_SCOPE_MAX_BYTES` | `1048576` | Maximum serialized history bytes in one scope. |
|
|
137
|
+
| `max_estimated_tokens` | `FREELLAMA_SCOPE_MAX_ESTIMATED_TOKENS` | `32768` | Maximum estimated tokens in one scope. |
|
|
138
|
+
| `total_max_bytes` | `FREELLAMA_SCOPE_TOTAL_MAX_BYTES` | `16777216` | Aggregate retained history-byte budget. |
|
|
139
|
+
| `ttl_seconds` | `FREELLAMA_SCOPE_TTL_SECONDS` | `3600` | Idle expiry since create, fork, or successful append; reads do not renew it. |
|
|
140
|
+
|
|
141
|
+
Per-scope HTTP/CLI `limits` accepts `max_messages`, `max_bytes`, `max_estimated_tokens`, and
|
|
142
|
+
`ttl_seconds` and can only lower these bounds within server policy. Finite safety bounds remain operator-owned. Lowering runtime caps
|
|
143
|
+
does not summarize or trim stored transcripts; subsequent writes must satisfy the active caps.
|
|
144
|
+
Expired entries are removed during store operations.
|
|
145
|
+
|
|
146
|
+
## Warm API
|
|
147
|
+
|
|
148
|
+
`POST /warm`, MCP `run_task {warm:true}`, and CLI `warm` load an exact installed model with a matching
|
|
149
|
+
non-embedding task profile. They do not accept prompts or messages, pull a model, or store conversation
|
|
150
|
+
history. The default task profile is `completion`.
|
|
151
|
+
|
|
152
|
+
| HTTP field | MCP field | Meaning |
|
|
153
|
+
|---|---|---|
|
|
154
|
+
| `model` | `model` | Required exact installed tag. |
|
|
155
|
+
| `task` | `task` | Optional capability/profile selection. |
|
|
156
|
+
| `context_tokens` | `contextTokens` | Context profile used for warming. |
|
|
157
|
+
| `execution_preference` | `executionPreference` | Guarded preference within operator assignments. |
|
|
158
|
+
| `min_placement_evidence` | `minPlacementEvidence` | Required configured or observed placement evidence. |
|
|
159
|
+
| `priority` | `priority` | Admission scheduling class. |
|
|
160
|
+
| `keep_alive` | `keepAlive` | Explicit residency control; overrides adaptive selection. |
|
|
161
|
+
| `max_wait_seconds` | `maxWaitSeconds` | Shorter admission/resource wait budget. |
|
|
162
|
+
| `timeout_seconds` | `timeoutSeconds` | Total request deadline, capped by server policy. |
|
|
163
|
+
| `defer` | `defer` | Return a job handle for the same managed operation. |
|
|
164
|
+
|
|
165
|
+
Synchronous warming returns HTTP `200`; deferred warming returns HTTP `202`. Inspect deferred
|
|
166
|
+
work through `task_jobs` or `jobs` using its returned ID. Warming reuses ordinary routing,
|
|
167
|
+
admission, memory-fit checks, transition locking, deadlines, and physical-placement observations.
|
|
168
|
+
A cold warm request can therefore wait or refuse. Read the `warm` receipt alongside
|
|
169
|
+
`execution.observation` and `execution.keep_alive`; a successful load response alone does not prove
|
|
170
|
+
physical residency or placement. Use configured placement for an initial load,
|
|
171
|
+
inspect its receipt, and require observed placement for subsequent work when placement matters.
|
|
172
|
+
|
|
173
|
+
`warm.loaded` reports observed residency independently of CPU/GPU placement: `true` when the exact
|
|
174
|
+
runner is present, `false` when a successful residency query finds it absent, and `null` when that
|
|
175
|
+
query is unavailable. A resident runner can still have a placement mismatch. Managed non-streaming
|
|
176
|
+
chat requires Ollama's `done:true`; an absent or non-boolean completion marker is an upstream error.
|
|
177
|
+
These are separate observations in Ollama's [chat response](https://docs.ollama.com/api/chat) and
|
|
178
|
+
[running-model inventory](https://docs.ollama.com/api/ps).
|
|
179
|
+
|
|
180
|
+
With `keep_alive:"0"`, warming observes placement and then unloads. The `warm` receipt describes
|
|
181
|
+
residency after unload: `loaded:false` on verified unload, with
|
|
182
|
+
`residency_source:"ollama_api_ps_after_unload"`.
|
|
183
|
+
|
|
184
|
+
Warming uses the same truncation and context-shift policy as managed chat. This preserves a
|
|
185
|
+
compatible runner profile instead of forcing a transition solely because warming omitted those
|
|
186
|
+
controls. Use a context/task profile compatible with the following task. Warming a different profile can
|
|
187
|
+
require another runner transition. Warm residency does not reserve a later execution slot. Cache reuse depends on the compatible
|
|
188
|
+
profile, unchanged prefix, and inference backend. See [Managed execution](ARCHITECTURE.md#managed-task-execution).
|
|
189
|
+
|
|
190
|
+
For a bounded warm operation, replace `MODEL_TAG` with an exact installed tag:
|
|
191
|
+
|
|
192
|
+
```bash
|
|
193
|
+
npx @octocodeai/freellama warm --model MODEL_TAG \
|
|
194
|
+
--context-tokens 4096 --keep-alive 60s --timeout-seconds 45 --defer
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
## Adaptive keep-alive
|
|
198
|
+
|
|
199
|
+
When a managed request omits `keep_alive`, FreeLlama selects a finite duration from measured reuse
|
|
200
|
+
and load cost, within the runtime `[warming]` bounds. Explicit per-call values take precedence,
|
|
201
|
+
including immediate unload (`"0"`) and indefinite residency (`"-1"`). Use
|
|
202
|
+
`--keep-alive=-1` in the CLI so the negative value remains attached to its flag. The configured eviction policy and available
|
|
203
|
+
resources still govern whether retaining a runner is useful.
|
|
204
|
+
|
|
205
|
+
Matching environment values override `[warming]` file values.
|
|
206
|
+
|
|
207
|
+
| Setting | Environment variable | Default | Meaning |
|
|
208
|
+
|---|---|---:|---|
|
|
209
|
+
| `min_seconds` | `FREELLAMA_KEEP_ALIVE_MIN_SECONDS` | `15` | Minimum automatic duration. |
|
|
210
|
+
| `max_seconds` | `FREELLAMA_KEEP_ALIVE_MAX_SECONDS` | `900` | Maximum automatic duration. |
|
|
211
|
+
| `base_seconds` | `FREELLAMA_KEEP_ALIVE_BASE_SECONDS` | `60` | Base duration before measured reuse and load cost. |
|
|
212
|
+
| `reuse_gain_seconds` | `FREELLAMA_KEEP_ALIVE_REUSE_GAIN_SECONDS` | `30` | Additional duration per recent use. |
|
|
213
|
+
| `load_multiplier` | `FREELLAMA_KEEP_ALIVE_LOAD_MULTIPLIER` | `2.0` | Multiplier for measured load seconds. |
|
|
214
|
+
| `pressure_factor` | `FREELLAMA_KEEP_ALIVE_PRESSURE_FACTOR` | `0.25` | Duration factor while observed resource pressure holds admission. |
|
|
215
|
+
|
|
216
|
+
Automatic duration starts from `base_seconds + recent_uses × reuse_gain_seconds + load_seconds ×
|
|
217
|
+
load_multiplier`. A pressure hold applies `pressure_factor`; minimum and maximum bounds clamp the
|
|
218
|
+
result, rounded to whole seconds. Recent-use weight decays with the residency tracker; the load
|
|
219
|
+
measurement comes from completed managed requests. These controls tune retention rather than model quality, CPU/GPU assignment, or host-memory
|
|
220
|
+
reserves. Inspect `execution.keep_alive` policy evidence instead of inferring residency from a model name.
|
|
221
|
+
|
|
222
|
+
Cold and warm helper-model measurements demonstrate feasibility on a specific host and request
|
|
223
|
+
profile. They do not establish a universal speedup, justify lowering production reserves, or
|
|
224
|
+
qualify sustained-load behavior. Compare representative cold, warm, scope-switching, and parallel
|
|
225
|
+
work before changing policy; see [Testing](TESTING.md).
|
package/docs/TESTING.md
CHANGED
|
@@ -3,6 +3,10 @@
|
|
|
3
3
|
FreeLlama separates fast deterministic checks from live-system tests. Run the narrowest relevant
|
|
4
4
|
tier during development, then run the full matrix before release.
|
|
5
5
|
|
|
6
|
+
The checks cover the management layer agents use to offload tasks: model eligibility, admission,
|
|
7
|
+
context and runtime controls, resource receipts, and adapter behavior. Physical CPU/GPU activity
|
|
8
|
+
and throughput require the separate hardware procedures linked in this guide.
|
|
9
|
+
|
|
6
10
|
```mermaid
|
|
7
11
|
flowchart LR
|
|
8
12
|
U["Unit and Rust contracts"] --> A["Local-agent context<br/>and action contracts"]
|
|
@@ -15,6 +19,7 @@ flowchart LR
|
|
|
15
19
|
## Install and build
|
|
16
20
|
|
|
17
21
|
```bash
|
|
22
|
+
corepack enable # Yarn version comes from package.json "packageManager"
|
|
18
23
|
yarn install
|
|
19
24
|
yarn build
|
|
20
25
|
```
|
|
@@ -26,14 +31,18 @@ dependency rather than compiling Rust during installation.
|
|
|
26
31
|
|
|
27
32
|
## Run the test tiers
|
|
28
33
|
|
|
34
|
+
The root `yarn build`, `yarn lint`, and `yarn test` run across every package: the Rust crates
|
|
35
|
+
(`rust-core`, `cli`) through Cargo and the JavaScript workspaces (`mcp`, `cli`, `native/*`) through
|
|
36
|
+
Yarn. Use the `:js` or `:rust` variants to run one side only.
|
|
37
|
+
|
|
29
38
|
| Tier | Command | External requirements |
|
|
30
39
|
|---|---|---|
|
|
31
|
-
|
|
|
32
|
-
| JavaScript
|
|
40
|
+
| All package unit tests (JS + Rust) | `yarn test` | Rust toolchain |
|
|
41
|
+
| JavaScript unit | `yarn test:js` | None |
|
|
33
42
|
| Rust contracts | `yarn test:rust` | Rust toolchain |
|
|
34
|
-
| Local-agent contracts | `yarn test:agents` |
|
|
43
|
+
| Local-agent contracts | `yarn test:agents` | None after install; builds the MCP package |
|
|
35
44
|
| TypeScript | `yarn typecheck` | None after install |
|
|
36
|
-
| MCP integration | `yarn test:integration` | Ollama
|
|
45
|
+
| MCP integration | `yarn test:integration` | Ollama at `FREELLAMA_OLLAMA_ENDPOINT` (default `127.0.0.1:11434`) |
|
|
37
46
|
| MCP end to end | `yarn test:e2e` | Ollama, `freellama serve`, and required models |
|
|
38
47
|
| All configured tiers | `yarn test:all` | Requirements of every included tier |
|
|
39
48
|
| Production verification | `yarn verify:production` | Build, formatting, strict Clippy, all test tiers, and package verification |
|
|
@@ -43,6 +52,26 @@ generation, embedding vectors are withheld by default, impossible models are exc
|
|
|
43
52
|
and models with unusable research evidence are refused without tool calls. Tests that require a
|
|
44
53
|
specific unavailable model report a skip reason.
|
|
45
54
|
|
|
55
|
+
Isolated end-to-end services inherit `FREELLAMA_OLLAMA_ENDPOINT`; they use the same selected Ollama
|
|
56
|
+
endpoint as the prerequisite check. A controlled endpoint fixture verifies this wiring without
|
|
57
|
+
inference.
|
|
58
|
+
|
|
59
|
+
Live inference tests also inspect routing readiness and report a skip when the host cannot admit
|
|
60
|
+
the model. These skips do not qualify inference throughput or physical placement. The pull/delete
|
|
61
|
+
lifecycle test is disabled unless the operator explicitly sets `FREELLAMA_TEST_PULL_TAG` to
|
|
62
|
+
`qwen2.5:0.5b` and `FREELLAMA_TEST_PULL_SIZE_BYTES` to the exact reported download size. The test
|
|
63
|
+
rechecks that size and reads every installed-model page before allowing the round trip.
|
|
64
|
+
|
|
65
|
+
## Model metadata contracts
|
|
66
|
+
|
|
67
|
+
Model metadata unit contracts cover README extraction, complete tag pages, family-versus-variant
|
|
68
|
+
features, digest matching, cloud aliases, cache expiry and eviction, response bounds, and lookup
|
|
69
|
+
failures. MCP integration's `model-knowledge.test.ts` exercises the built server with deterministic
|
|
70
|
+
local HTTP fixtures for both Ollama API data and public HTML. It checks optional enrichment,
|
|
71
|
+
assigned-backend inventory, tag pagination, validation, and preservation of benchmark/policy evidence.
|
|
72
|
+
The fixture tests perform no inference, pulls, or deletions. See the
|
|
73
|
+
[metadata reference](MODEL_SELECTION.md#model-metadata) for the public states these tests protect.
|
|
74
|
+
|
|
46
75
|
## Run static Rust checks
|
|
47
76
|
|
|
48
77
|
```bash
|
|
@@ -52,6 +81,9 @@ cargo clippy --all-targets -- -D warnings
|
|
|
52
81
|
|
|
53
82
|
Routing regressions run in the default Rust suite; none are hidden behind `#[ignore]`. Add a
|
|
54
83
|
focused contract test for each repaired behavior and keep it enabled in ordinary release checks.
|
|
84
|
+
Resource contracts cover conservative full host-memory reservations when discrete-GPU telemetry is
|
|
85
|
+
missing, and immediate-unload cleanup after a failed inference response body. The latter also checks
|
|
86
|
+
that uncertain inference is never replayed and ordinary keep-alive requests retain their runner.
|
|
55
87
|
The local-agent tier runs all context fitting, compaction, pagination, repeat-suppression, and
|
|
56
88
|
strict action-shape contracts. It is included in `yarn test:all`, so adapter regressions cannot be
|
|
57
89
|
missed by the root release matrix.
|
|
@@ -71,6 +103,73 @@ two warm samples do not steer `auto`, the third sample on both backends enables
|
|
|
71
103
|
comparison, differences of 10% or less remain noise, and one GPU plus one CPU request overlap even
|
|
72
104
|
when both admission pools contain one unit.
|
|
73
105
|
|
|
106
|
+
`resource_governor_contract` verifies that tasks held for memory remain counted in the bounded
|
|
107
|
+
queue after releasing execution slots, resume when memory recovers, and release their queue entries
|
|
108
|
+
and reservations on cancellation. A separate contract requires released reservations to wake
|
|
109
|
+
waiters before the next telemetry sample. MCP integration checks forward `maxWaitSeconds` for
|
|
110
|
+
single and batch execution and reject it in a decision-only preview.
|
|
111
|
+
Admission contracts also cover expired deadlines, queue-limit reductions during resource-to-slot
|
|
112
|
+
handoff, and cancellation after that handoff. Governor contracts require CPU pressure to recover
|
|
113
|
+
independently while a memory hold still protects cold loads.
|
|
114
|
+
MCP integration also requires native refusals to preserve error codes, retry timing, resource
|
|
115
|
+
assessments, and unload receipts in both text and structured output.
|
|
116
|
+
GET and DELETE refusals retain scope codes and job error details, while successful DELETE accepts an
|
|
117
|
+
empty HTTP 204 body. Unknown top-level tool arguments are rejected before any server request;
|
|
118
|
+
caller-owned message and function-definition extension fields remain preserved.
|
|
119
|
+
|
|
120
|
+
`task_jobs_contract` covers deferred acceptance, metadata privacy, memory recovery, cancellation
|
|
121
|
+
of waiting and active requests, removal of active and completed jobs without affecting other IDs,
|
|
122
|
+
total deadlines during inference, and input-size/batch boundaries.
|
|
123
|
+
Registry unit tests cover active-job limits, terminal eviction, expiry, and result-size limits.
|
|
124
|
+
Cancellation must retain its receipt even when another operation evicts the completed record.
|
|
125
|
+
`scope_warming_contract` covers opt-in history replay, message-field preservation, isolated forks,
|
|
126
|
+
failed/stale append refusal, and payload-free managed warming. The warming unit contract
|
|
127
|
+
exercises bounded reuse, load, and pressure adjustments. For request fields and failure codes, see
|
|
128
|
+
[Scope history and model warming](SCOPES_AND_WARMING.md).
|
|
129
|
+
`warming_receipt_contract` separates observed residency from processor verification, retains unknown
|
|
130
|
+
residency when inspection fails, rejects incomplete chat responses, and verifies unload after failure.
|
|
131
|
+
`resident_placement_contract` distinguishes installed file size from resident runner size across
|
|
132
|
+
fresh discovery, cache refresh, observed routing, and scoped execution. Missing total runner memory
|
|
133
|
+
with positive VRAM stays unknown and cannot qualify observed GPU routing or adaptive feedback.
|
|
134
|
+
The MCP resident-view fixture checks percentages against resident bytes and preserves disk size.
|
|
135
|
+
|
|
136
|
+
Usage-ledger unit contracts cover ordered writes, bounded queue overflow, failed writes, regular-file
|
|
137
|
+
validation, replay, and flush behavior. `monitoring_contract` also checks that a blocked ledger path
|
|
138
|
+
does not hold task completion. Persistence remains best effort, not a zero-loss promise.
|
|
139
|
+
|
|
140
|
+
MCP `serve.test.ts` checks startup permission failure, retry, and child-lifecycle isolation. The
|
|
141
|
+
protocol integration contracts verify that spawn failure leaves the MCP connection usable and that
|
|
142
|
+
fixture-backed library requests fail on unexpected product errors. External catalog availability
|
|
143
|
+
is separate from these deterministic checks. Runtime-view `app-windows.test.ts` checks legitimate
|
|
144
|
+
Windows static paths alongside traversal refusal.
|
|
145
|
+
|
|
146
|
+
Launcher fixtures use `process.execPath` to avoid an extra PATH-resolved shell wrapper. After the
|
|
147
|
+
MCP fixture's real executable-permission refusal, a test-only transport launches that exact fixture
|
|
148
|
+
script through the same Node executable. Real child processes, HTTP listeners, signals, and exit
|
|
149
|
+
tracking still exercise the lifecycle contract. On the October 2, 2026 host, raw process samples
|
|
150
|
+
placed the failed fixture launches in the macOS dynamic loader before JavaScript loaded. This
|
|
151
|
+
fixture correction preserves test and production deadlines; it does not measure production
|
|
152
|
+
startup performance or identify the underlying host loader policy.
|
|
153
|
+
|
|
154
|
+
Request-validation contracts preserve structured JSON errors for invalid task and batch bodies.
|
|
155
|
+
MCP `task-jobs.test.ts` exercises submission, inspection, result retrieval, cancellation, and removal through
|
|
156
|
+
the native binding, plus preview rejection and embedding-vector summaries. Separate footprint and
|
|
157
|
+
process-attribution tests require independent CPU/GPU settings; `smart_placement_contract` requires
|
|
158
|
+
a resident routing preview to agree with execution under a low-memory-only hold.
|
|
159
|
+
|
|
160
|
+
Run the isolated control timing experiments separately from the full suite:
|
|
161
|
+
|
|
162
|
+
```bash
|
|
163
|
+
cargo test --test integration control_timing_contract -- --include-ignored --nocapture
|
|
164
|
+
cargo test --test integration task_jobs_contract::measure_deferred_control_latency -- --ignored --nocapture
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
These fixed-budget experiments use simulated backends to measure status, discovery, queue
|
|
168
|
+
controls, and resource recovery. They emit `CONTROL_TIMING` records with 20 samples, median, and
|
|
169
|
+
95th-percentile latency. They verify both backend inventories and released reservations; they
|
|
170
|
+
do not measure model tokens per second. Process-refresh contracts require updated observations
|
|
171
|
+
after cache expiry and reject old footprint measurements after parallelism changes.
|
|
172
|
+
|
|
74
173
|
## Validate CPU and GPU concurrency
|
|
75
174
|
|
|
76
175
|
Start the two Ollama processes and FreeLlama as described in
|
|
@@ -93,16 +192,19 @@ After rebuilding the release, a fresh concurrent smoke check completed both mana
|
|
|
93
192
|
Qwen, one embedding from Nomic, positive Qwen VRAM, zero Nomic VRAM, and primary Ollama 0.33.2
|
|
94
193
|
through raw passthrough.
|
|
95
194
|
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
195
|
+
The portable hardware acceptance, workload, comparison, and load runners were removed from the
|
|
196
|
+
repository; they remain in git history. Compilation alone does not validate a driver, physical
|
|
197
|
+
placement, shared-memory contention, or OCR quality.
|
|
198
|
+
|
|
199
|
+
The historical CPU/GPU observations above predate the corrected paired-run acceptance gate.
|
|
200
|
+
They remain records of those runs and do not qualify a release.
|
|
99
201
|
|
|
100
202
|
## Verify release packages
|
|
101
203
|
|
|
102
204
|
```bash
|
|
103
205
|
# Collect each target's `freellama` executable and `freellama.<target>.node`
|
|
104
206
|
# under release-artifacts/<target>/, then stage all eight platform packages.
|
|
105
|
-
|
|
207
|
+
node scripts/assemble-release.mjs release-artifacts release
|
|
106
208
|
yarn release:verify:publish
|
|
107
209
|
```
|
|
108
210
|
|
|
@@ -114,6 +216,9 @@ prove registry publication or another hardware class.
|
|
|
114
216
|
|
|
115
217
|
## Test one layer while iterating
|
|
116
218
|
|
|
219
|
+
When investigating a fixture failure, preserve failed and passing receipts together; a passing
|
|
220
|
+
rerun does not qualify production startup timing.
|
|
221
|
+
|
|
117
222
|
```bash
|
|
118
223
|
yarn build:native
|
|
119
224
|
yarn workspace @octocodeai/freellama-mcp-server build
|