@octocodeai/freellama-mcp-server 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONFIGURATION.md +108 -0
- package/DEVELOPING.md +109 -0
- package/README.md +130 -282
- package/dist/index.js +1509 -1411
- package/dist/research/bash-agent.js +2435 -0
- package/dist/research/octocode-agent.js +1639 -0
- package/docs/ARCHITECTURE.md +145 -81
- package/docs/CLI.md +83 -55
- package/docs/CPU_GPU_ROUTING.md +4 -3
- package/docs/MODEL_SELECTION.md +115 -5
- package/docs/MONITORING.md +19 -11
- package/docs/PRODUCTION.md +10 -13
- package/docs/README.md +9 -13
- package/docs/SCOPES_AND_WARMING.md +8 -9
- package/docs/TESTING.md +18 -30
- package/native/index.d.ts +179 -174
- package/native/index.js +3 -3
- package/package.json +11 -10
- package/adapters/agent_context.py +0 -850
- package/adapters/agent_transport.py +0 -215
- package/adapters/bash_agent.py +0 -345
- package/adapters/octocode_agent.py +0 -404
- package/adapters/shell_sandbox.py +0 -487
- package/docs/ECONOMICS.md +0 -110
- package/docs/INDEX.md +0 -21
- package/docs/MODEL_METADATA.md +0 -106
- package/docs/OLLAMA_SIDECAR.md +0 -78
- package/docs/PRODUCT_POSITIONING.md +0 -121
- package/docs/dev/ADR_RESOURCE_AWARE_BACKEND_ROUTING.md +0 -55
- package/docs/dev/FINDINGS_AND_POSITIONING.md +0 -290
- package/docs/dev/LOCAL_PERFORMANCE_TESTING.md +0 -298
- package/docs/dev/OLLAMA_SYSTEM_OPTIMIZATION.md +0 -255
package/docs/MODEL_SELECTION.md
CHANGED
|
@@ -10,8 +10,7 @@ are not compiled into routing. Re-run
|
|
|
10
10
|
`doctor.machine.memory_bytes` is host RAM; a discrete GPU's usable VRAM must be observed separately.
|
|
11
11
|
|
|
12
12
|
For supported features and sourced publisher use-case guidance, inspect
|
|
13
|
-
`models {view:"installed", includeLibrary:true}` through MCP.
|
|
14
|
-
[Ollama metadata reference](MODEL_METADATA.md) explains digest matching, variant capabilities,
|
|
13
|
+
`models {view:"installed", includeLibrary:true}` through MCP. [Model metadata](#model-metadata) explains digest matching, variant capabilities,
|
|
15
14
|
custom-model fallbacks, and how to read family claims without treating them as quality measurements.
|
|
16
15
|
|
|
17
16
|
```mermaid
|
|
@@ -33,10 +32,15 @@ flowchart TD
|
|
|
33
32
|
grounded research, image description, OCR, and summarization. Using one large multimodal model also
|
|
34
33
|
avoided contention between separate heavy text and vision runners.
|
|
35
34
|
|
|
36
|
-
Read the [retained model profile](
|
|
37
|
-
and [delegation measurements](
|
|
35
|
+
Read the [retained model profile](https://github.com/bgauryy/FreeLlama/blob/main/skills/freellama/references/model-profile-qwen3.8-27b-mlx.md)
|
|
36
|
+
and [delegation measurements](https://github.com/bgauryy/FreeLlama/blob/main/skills/freellama/references/task-delegation.md) for scope and limits.
|
|
38
37
|
These examples do not establish that a tag is installed or suitable on your host.
|
|
39
38
|
|
|
39
|
+
Dedicated OCR models answer best to the prompts their model card documents; send them with base64
|
|
40
|
+
`images` in `run_task`. A model that repeats its answer until the output cap can run with
|
|
41
|
+
`stopOnRepetition:true`. See [Route images](https://github.com/bgauryy/FreeLlama/blob/main/packages/mcp/README.md#route-images)
|
|
42
|
+
for the early stop and the `output_check` receipt.
|
|
43
|
+
|
|
40
44
|
`muse-glimmer:30b-mlx` was a credible alternative and answered 29 of 30 questions correctly
|
|
41
45
|
(96.7%) with the Bash adapter, but it was slower and did not add a required capability in this
|
|
42
46
|
workload.
|
|
@@ -61,7 +65,7 @@ keywords, repository search was both faster and more accurate in the measured sa
|
|
|
61
65
|
|
|
62
66
|
## Place helper models on CPU when useful
|
|
63
67
|
|
|
64
|
-
A small embedding or
|
|
68
|
+
A small embedding or helper model can be assigned to a second CPU Ollama process while a large
|
|
65
69
|
generation model remains resident on the primary GPU-capable process. This can overlap independent
|
|
66
70
|
work, but it does not make CPU inference fast. Apple silicon shares memory bandwidth; discrete-GPU
|
|
67
71
|
systems have different transfer and contention costs and need their own benchmark.
|
|
@@ -76,3 +80,109 @@ this machine and workload, not a universal model-placement rule. Follow
|
|
|
76
80
|
`bench-all` measures local performance. A quality benchmark measures correctness. FreeLlama needs
|
|
77
81
|
both the generated task policy and local benchmark data before grading a route medium confidence.
|
|
78
82
|
See the [CLI reference](CLI.md#earn-medium-routing-confidence) for the commands.
|
|
83
|
+
|
|
84
|
+
## Model metadata
|
|
85
|
+
|
|
86
|
+
The MCP `models` tool reads your Ollama instance and can enrich results with Ollama's public model
|
|
87
|
+
library. Supported features and advertised use cases are metadata: they do not change routing
|
|
88
|
+
confidence, and exact-model benchmarks and policy remain the quality evidence. The CLI's `models`
|
|
89
|
+
command reports the core catalog and does not expose these enrichment flags.
|
|
90
|
+
|
|
91
|
+
| Request | Result | External access |
|
|
92
|
+
|---|---|---|
|
|
93
|
+
| `models {view:"installed"}` | Managed inventory with a `knowledge` object for each model | Local backend data only |
|
|
94
|
+
| `models {view:"installed", includeLibrary:true}` | Inventory enriched with public guidance for verified local tags | Fetches model family and tag pages from ollama.com |
|
|
95
|
+
| `models {view:"detail", model:"gemma3:1b"}` | Ollama details, maximum context, and supported-use explanations | Selected Ollama endpoint only |
|
|
96
|
+
| `models {view:"detail", model:"gemma3:1b", includeLibrary:true}` | Details plus public guidance after a tag and digest check | Selected Ollama endpoint and ollama.com |
|
|
97
|
+
| `models {view:"library", query:"coding"}` | Public family search with descriptions and advertised feature chips | ollama.com and optional local inventory/profile reads |
|
|
98
|
+
| `models {view:"library", model:"gemma3"}` | All public tags with size, digest prefix, context, modalities, and cloud status | ollama.com and optional local inventory/profile reads |
|
|
99
|
+
|
|
100
|
+
|
|
101
|
+
| Option | Valid views | Default | Meaning |
|
|
102
|
+
|---|---|---|---|
|
|
103
|
+
| `includeLibrary` | `installed`, `detail` | `false` | Requests public metadata; the default makes no website calls |
|
|
104
|
+
| `includeReadme` | Library step 2, or `detail` with `includeLibrary:true` | `false` | Returns README text, capped at 65,536 characters, with its format and truncation flag |
|
|
105
|
+
| `includeVerbose` | `detail` | `false` | Returns Ollama's license and Modelfile fields independently of README enrichment |
|
|
106
|
+
| `limit`, `cursor` | `raw`, library tag listings; `limit` also applies to library search | Tag/raw page size 20; search limit 10 | Pages large lists; the tag cursor rejects a changed tag list |
|
|
107
|
+
| `ollamaEndpoint` | Direct Ollama views and library inventory reads | Configured Ollama endpoint | Chooses the Ollama instance; enriched managed inventory uses each model's assigned backend |
|
|
108
|
+
|
|
109
|
+
An incompatible option returns an error before any lookup. Installed enrichment omits full READMEs;
|
|
110
|
+
request one model's detail to read its README.
|
|
111
|
+
|
|
112
|
+
### Supported features
|
|
113
|
+
|
|
114
|
+
`knowledge.supportedFeatures` preserves the feature strings from the selected source
|
|
115
|
+
(`supportedFeaturesSource`: `ollama_api` or `freellama_catalog`). Unknown future strings stay visible
|
|
116
|
+
but do not create routing requirements. `knowledge.supportedUseCases` explains each recognized
|
|
117
|
+
feature with `capability`, `use`, `reason`, and `basis:"supported_feature"`;
|
|
118
|
+
`knowledge.quality` stays `"metadata_only"`.
|
|
119
|
+
|
|
120
|
+
| Feature | Supported workflow | What to evaluate separately |
|
|
121
|
+
|---|---|---|
|
|
122
|
+
| `completion` | Text generation | Coding, summarization, reasoning, and instruction-following accuracy |
|
|
123
|
+
| `tools` | Tool calls | Argument correctness, tool selection, and agent-loop reliability |
|
|
124
|
+
| `vision` | Image input | OCR and visual understanding accuracy |
|
|
125
|
+
| `audio` | Audio workflows advertised by the model | Supported audio task and accuracy |
|
|
126
|
+
| `embedding` | Retrieval and similarity vectors | Retrieval quality on your documents |
|
|
127
|
+
| `thinking` | A thinking mode | Reasoning accuracy, latency, and token cost |
|
|
128
|
+
| `insert` | Suffix-based text insertion | Completion accuracy in the target code or text |
|
|
129
|
+
| `image` | Image generation | Output quality and supported controls |
|
|
130
|
+
|
|
131
|
+
|
|
132
|
+
### Public guidance and provenance
|
|
133
|
+
|
|
134
|
+
For a verified installed model, `knowledge.library` contains the source page, tag-page URL,
|
|
135
|
+
`fetchedAt`, `expiresAt`, `cached`, exact public tag metadata, and:
|
|
136
|
+
|
|
137
|
+
| Field | Meaning |
|
|
138
|
+
|---|---|
|
|
139
|
+
| `description` | The public family description |
|
|
140
|
+
| `advertisedFamilyCapabilities` | Family feature chips; these do not extend the local tag's capabilities |
|
|
141
|
+
| `advertisedUseCases` | Recognized coding, tool-calling, image-understanding/OCR, reasoning, embedding, summarization, question-answering, translation, and long-context claims |
|
|
142
|
+
| `advertisedUseCases[].evidence` | A bounded source excerpt, source URL, and fetch timestamp |
|
|
143
|
+
| `usageNotes` | Bounded excerpts about requirements, restrictions, context, memory, languages, or query prefixes |
|
|
144
|
+
| `readmeAvailable` | Whether a README was extracted |
|
|
145
|
+
| `readme` | Requested text with `format` (`markdown` or `text`) and `truncated` |
|
|
146
|
+
|
|
147
|
+
|
|
148
|
+
Advertised uses have `basis:"advertised"`, `scope:"family"`, and `quality:"unmeasured"`. Extraction
|
|
149
|
+
skips fenced code and negative statements; an omitted use means no recognized claim, not absence.
|
|
150
|
+
These labels are not the `run_task` task enum. Local capabilities filter advertised uses: a family
|
|
151
|
+
vision chip cannot make `gemma3:1b` a vision candidate.
|
|
152
|
+
|
|
153
|
+
Installed enrichment requires the exact public tag and a digest-prefix match of at least 12 hex
|
|
154
|
+
characters. Custom names are never mapped by architecture or guessed base model; without a verified
|
|
155
|
+
match, public guidance is withheld. Library step 2 describes public models before installation, so
|
|
156
|
+
its guidance stays family-scoped; each `tags[]` entry carries its own modalities, context, digest
|
|
157
|
+
prefix, and `cloud` flag. Search's `cloudAvailable` badge does not prove local downloads are absent.
|
|
158
|
+
|
|
159
|
+
### Lookup states
|
|
160
|
+
|
|
161
|
+
| `library.status` | Meaning | Next action |
|
|
162
|
+
|---|---|---|
|
|
163
|
+
| `not_requested` | Website enrichment was not requested | Set `includeLibrary:true` if external lookup is useful |
|
|
164
|
+
| `matched` | Installed tag and public digest prefix match | Read the sourced guidance, then evaluate the exact tag |
|
|
165
|
+
| `available` | Public family and tag pages were parsed in library step 2 | Inspect the exact tag before recommending a download |
|
|
166
|
+
| `unverified` | Digest or current backend inventory could not be verified | Recheck local inventory; retain local feature information |
|
|
167
|
+
| `digest_mismatch` | The public tag differs from the installed digest | Treat public guidance as unavailable for this installed model |
|
|
168
|
+
| `tag_not_found` | The parsed public tag list has no exact matching tag | Check the source and custom-model provenance |
|
|
169
|
+
| `unmatched` | The name cannot resolve to an exact public Ollama name | Keep use-case suitability unknown until provenance is verified |
|
|
170
|
+
| `not_found` | A requested public page returned HTTP 404 | Check the source URL or custom name |
|
|
171
|
+
| `unavailable` | Transport, timeout, HTTP refusal, or page-size limit prevented lookup | Retry later; local results remain available |
|
|
172
|
+
| `parse_error` | HTML content or tag rows were not recognized | Inspect the source; do not conclude the model is absent |
|
|
173
|
+
| `remote_model`, `remote_tag` | Remote Ollama fields or a hosted tag identify cloud execution | Select a local download tag for local execution |
|
|
174
|
+
|
|
175
|
+
|
|
176
|
+
`localModel:false` flags detected remote entries; cloud tags never become local trial
|
|
177
|
+
recommendations. This metadata is not processor-placement evidence.
|
|
178
|
+
|
|
179
|
+
### Cache and request bounds
|
|
180
|
+
|
|
181
|
+
Each MCP process caches at most 128 families: successes for one hour, failures for one minute;
|
|
182
|
+
concurrent requests share one lookup and a restart clears the cache. Installed enrichment runs at
|
|
183
|
+
most four family lookups concurrently. Requests use GET, refuse redirects, cap each page at 2 MiB,
|
|
184
|
+
and time out after the smaller of 10 seconds and `FREELLAMA_MCP_FETCH_TIMEOUT_SECONDS`. They disclose
|
|
185
|
+
the requested public family name to ollama.com. Download size does not establish runtime memory
|
|
186
|
+
fit, placement, or quality, and model discovery never authorizes a pull. Sources:
|
|
187
|
+
[model details API](https://docs.ollama.com/api-reference/show-model-details),
|
|
188
|
+
[model-list API](https://docs.ollama.com/api/tags).
|
package/docs/MONITORING.md
CHANGED
|
@@ -38,7 +38,9 @@ token and keeps it only for the browser tab.
|
|
|
38
38
|
whether the model is pinned.
|
|
39
39
|
- **`host`**: available and reserved RAM, whether the pressure gate is holding, and discrete-GPU
|
|
40
40
|
total and free memory (`nvidia-smi` or amdgpu sysfs) when available. Optional `gpu_activity`
|
|
41
|
-
reports driver counters separately from resident memory.
|
|
41
|
+
reports driver counters separately from resident memory. On macOS, `thermal_pressure` is the
|
|
42
|
+
system thermal pressure level (`nominal`, `moderate`, `heavy`, `trapping`, `sleeping`); `heavy`
|
|
43
|
+
or worse holds CPU-assigned work and cold loads. Other platforms report `null`.
|
|
42
44
|
- **`ollama`**: the Ollama server's effective `OLLAMA_*` settings with a source for each — the
|
|
43
45
|
running process's environment (Linux `/proc/<pid>/environ`, macOS `ps eww` for a same-user
|
|
44
46
|
process), `launchctl getenv` for the macOS app, FreeLlama's own environment, or Ollama's default.
|
|
@@ -72,7 +74,8 @@ FreeLlama admission limits remain governed by its runtime configuration.
|
|
|
72
74
|
|
|
73
75
|
`host.gpu_activity` is nullable. When available, it identifies its `source`,
|
|
74
76
|
`scope: "observed_devices_all_processes"`, `window: "driver_defined"`, and individual `devices`.
|
|
75
|
-
Each device has an identity, nullable `busy_percent`, and nullable `memory_busy_percent`.
|
|
77
|
+
Each device has an identity, nullable `busy_percent`, and nullable `memory_busy_percent`. Apple
|
|
78
|
+
devices may add `allocated_memory_bytes`, the driver's allocated memory across all processes.
|
|
76
79
|
An observed zero means the driver reported zero; a missing or malformed counter remains unknown.
|
|
77
80
|
Use `host.sample_age_ms` to check freshness.
|
|
78
81
|
|
|
@@ -129,12 +132,13 @@ FreeLlama waits for safe capacity within a deadline and bounds the retained back
|
|
|
129
132
|
| Managed queue full | `429` | `admission_queue_full` |
|
|
130
133
|
| Raw proxy over its cap after `raw_queue_wait_seconds` | `429` | |
|
|
131
134
|
| Managed wait exceeded `max_queue_wait_seconds` (or the task's `max_wait_seconds`) | `503` | `admission_timeout` |
|
|
132
|
-
|
|
|
135
|
+
| OS pressure, swapping, CPU, thermal, or required telemetry still holds when the waiting budget expires (low available RAM alone only warns; low RAM and swapping never hold an already-resident runner) | `503` | `resource_admission_unavailable` |
|
|
133
136
|
| Backend circuit open | `503` | `upstream_circuit_open` |
|
|
134
137
|
|
|
135
138
|
These capacity refusals carry a `Retry-After` header and `retry_after_seconds` in the body. For a queue it
|
|
136
|
-
is estimated from the queue depth, the
|
|
137
|
-
|
|
139
|
+
is estimated from the queue depth, the lanes that can run at once (current limit divided by the
|
|
140
|
+
task's cost), and the backend's recent average service time, which excludes queue wait (1 to 120
|
|
141
|
+
seconds). A task can ask for a shorter wait with `max_wait_seconds` (`maxWaitSeconds`
|
|
138
142
|
in MCP); the server caps it at the configured maximum.
|
|
139
143
|
|
|
140
144
|
## Circuit breaker
|
|
@@ -166,7 +170,7 @@ costs, and execution receipts use the resolved policy. Health retains its summar
|
|
|
166
170
|
exposes the complete task-kind map under `admission.costs.base_by_task`. Task kinds remain
|
|
167
171
|
caller-declared; these weights are cooperative scheduling policy, not workload classification.
|
|
168
172
|
See the
|
|
169
|
-
[runtime example](
|
|
173
|
+
[runtime example](https://github.com/bgauryy/FreeLlama/blob/main/freellama.runtime.example.toml) for the table syntax.
|
|
170
174
|
|
|
171
175
|
With `adaptive_concurrency = "cpu"` (the default) or `"all"`, each pool's limit moves between 1 and
|
|
172
176
|
its configured ceiling:
|
|
@@ -273,13 +277,17 @@ send `num_ctx`.
|
|
|
273
277
|
|
|
274
278
|
Put any of the settings above in a TOML file and start `serve --runtime-config <file>` (or set
|
|
275
279
|
`FREELLAMA_RUNTIME_CONFIG`). Start from
|
|
276
|
-
[`freellama.runtime.example.toml`](
|
|
280
|
+
[`freellama.runtime.example.toml`](https://github.com/bgauryy/FreeLlama/blob/main/freellama.runtime.example.toml). `serve` checks the file every
|
|
277
281
|
two seconds and applies changes to live admission pools; a waiting task sees the new limit at once.
|
|
278
282
|
|
|
279
|
-
Precedence for every setting, strongest first: CLI flag,
|
|
280
|
-
runtime file, the built-in default.
|
|
281
|
-
|
|
282
|
-
|
|
283
|
+
Precedence for every setting, strongest first: CLI flag, environment variable (where one exists),
|
|
284
|
+
the runtime file, the built-in default. Only these settings have an environment variable:
|
|
285
|
+
`FREELLAMA_MAX_QUEUE_WAIT_SECONDS`, `FREELLAMA_EVICT_IDLE_MODELS`, `FREELLAMA_USAGE_QUEUE_CAPACITY`,
|
|
286
|
+
and the individual `FREELLAMA_SCOPE_*` and `FREELLAMA_KEEP_ALIVE_*` limits. Set every other key in
|
|
287
|
+
the runtime file or with its CLI flag. A flag or variable pins a value, so editing the file cannot
|
|
288
|
+
override it; `freellama config` shows which source won: `cli`, `env`, `file`, a default source
|
|
289
|
+
such as `default` or `ollama_num_parallel`, or `env_invalid` when a set variable failed validation.
|
|
290
|
+
A malformed file or an unknown key is rejected at reload and the previous values stay in force.
|
|
283
291
|
|
|
284
292
|
The default GPU budget follows the Ollama server: 2 units per `OLLAMA_NUM_PARALLEL` slot (one chat
|
|
285
293
|
per decoding slot), read from the running Ollama process where visible.
|
package/docs/PRODUCTION.md
CHANGED
|
@@ -12,9 +12,12 @@ not provide tenant isolation or billing.
|
|
|
12
12
|
|
|
13
13
|
- Install one Ollama CLI and server version. Run `freellama doctor` and resolve any mismatch.
|
|
14
14
|
- Install Node.js 20 or later for MCP and npm distribution use.
|
|
15
|
-
- Use the release artifacts for your operating system, or build with Rust 1.
|
|
15
|
+
- Use the release artifacts for your operating system, or build with Rust 1.88 or later.
|
|
16
16
|
- Preinstall and benchmark exact model tags. Discovery never authorizes a pull.
|
|
17
17
|
- Keep the primary and optional CPU Ollama processes on distinct loopback sockets.
|
|
18
|
+
- In a slim or distroless container, install the CA bundle (`ca-certificates`) or set
|
|
19
|
+
`SSL_CERT_FILE` if any upstream uses `https://`. Without one, FreeLlama starts with an empty trust
|
|
20
|
+
store and warns: plain-HTTP upstreams work, HTTPS upstreams fail their certificate check.
|
|
18
21
|
|
|
19
22
|
After a tag has been published, install that exact checksummed release on macOS or Linux:
|
|
20
23
|
|
|
@@ -31,17 +34,14 @@ published tag:
|
|
|
31
34
|
powershell -ExecutionPolicy Bypass -File .\scripts\install.ps1 -Version VERSION
|
|
32
35
|
```
|
|
33
36
|
|
|
34
|
-
The
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
JavaScript/native checks on macOS and Linux. These workflows do not prove driver compatibility or
|
|
38
|
-
complete the hardware promotion checklist below. Release artifact builds depend on the reusable CI checks.
|
|
39
|
-
The publish step also verifies that a version tag matches the workspace version.
|
|
37
|
+
The repository has no CI or release workflows; they were removed (see git history). Build, verify,
|
|
38
|
+
and publish artifacts with the [release procedure](../RELEASE.md). The publish verifier checks that
|
|
39
|
+
a version tag matches the workspace version.
|
|
40
40
|
Live E2E and hardware qualification remain separate operator checks; verify those receipts before tagging.
|
|
41
41
|
|
|
42
42
|
For manual assembly, build and test every claimed target in a clean
|
|
43
43
|
environment, place each executable and matching addon under `release-artifacts/<target>/`, run
|
|
44
|
-
`
|
|
44
|
+
`node scripts/assemble-release.mjs release-artifacts release` and `yarn release:verify:publish`, then attach
|
|
45
45
|
`release/SHA256SUMS` and binaries to the release. Publish the eight `@octocodeai/freellama-native-*` packages
|
|
46
46
|
first, followed by the exact-version `@octocodeai/freellama` and `@octocodeai/freellama-mcp-server` packages. Configure
|
|
47
47
|
registry credentials in the release environment; never store them in the repository.
|
|
@@ -197,12 +197,9 @@ the Ollama process does not expose to FreeLlama.
|
|
|
197
197
|
Run compile and deterministic test checks from clean checkouts on Windows, Linux, and macOS. Live
|
|
198
198
|
accelerator acceptance is separate because build machines do not necessarily represent production
|
|
199
199
|
GPU drivers or memory topology. Prepare Apple Metal, NVIDIA Linux, AMD Linux, and NVIDIA Windows
|
|
200
|
-
hosts, then run
|
|
200
|
+
hosts, then run a hardware acceptance check directly on each host.
|
|
201
201
|
|
|
202
|
-
|
|
203
|
-
portable runner in `benchmark/hardware/run_validation.py` executes GPU coding and CPU embedding
|
|
204
|
-
concurrently, verifies physical processor receipts, checks admission receipts, and writes a JSON
|
|
205
|
-
artifact. Add `--vision-model` and `--vision-image` for that host's OCR/vision gate.
|
|
202
|
+
The portable hardware runners were removed from the repository; they remain in git history.
|
|
206
203
|
|
|
207
204
|
Do not mark a hardware row validated until its archived receipt says `verdict: accept`. Missing
|
|
208
205
|
hardware is an explicit release gap, not a result that documentation can waive.
|
package/docs/README.md
CHANGED
|
@@ -2,39 +2,35 @@
|
|
|
2
2
|
|
|
3
3
|
FreeLlama helps agents offload bounded work to local models through Ollama. The calling agent owns
|
|
4
4
|
the task and verifies the answer; FreeLlama manages model selection and resources around execution.
|
|
5
|
-
Start with the [repository
|
|
5
|
+
Start with the [repository MCP setup](https://github.com/bgauryy/FreeLlama/blob/main/README.md#connect-your-agent-mcp) to run one managed task.
|
|
6
6
|
|
|
7
7
|
## Choose a guide
|
|
8
8
|
|
|
9
9
|
| Goal | Guide |
|
|
10
10
|
|---|---|
|
|
11
|
-
| Connect an agent and choose a tool | [MCP server](
|
|
12
|
-
| Follow the agent's operating workflow | [FreeLlama skill](
|
|
11
|
+
| Connect an agent and choose a tool | [MCP server](https://github.com/bgauryy/FreeLlama/blob/main/packages/mcp/README.md) |
|
|
12
|
+
| Follow the agent's operating workflow | [FreeLlama skill](https://github.com/bgauryy/FreeLlama/blob/main/skills/freellama/SKILL.md) |
|
|
13
13
|
| Look up commands and task controls | [CLI reference](CLI.md) |
|
|
14
14
|
| Understand task and resource ownership | [Architecture](ARCHITECTURE.md) |
|
|
15
|
-
| Choose a model
|
|
16
|
-
| Interpret capabilities and public model metadata | [Model metadata](MODEL_METADATA.md) |
|
|
15
|
+
| Choose a model and interpret its metadata | [Model selection](MODEL_SELECTION.md), [model metadata](MODEL_SELECTION.md#model-metadata) |
|
|
17
16
|
| Configure CPU helpers alongside GPU work | [CPU/GPU routing](CPU_GPU_ROUTING.md) |
|
|
18
17
|
| Inspect queues, memory, usage, and live settings | [Monitoring](MONITORING.md) |
|
|
19
18
|
| Retain history, affinity, or residency | [Scopes and warming](SCOPES_AND_WARMING.md) |
|
|
20
|
-
| Understand raw Ollama compatibility | [
|
|
21
|
-
| Evaluate context offload and its cost | [Token economics](
|
|
19
|
+
| Understand raw Ollama compatibility | [Ollama sidecar](ARCHITECTURE.md#ollama-sidecar) |
|
|
20
|
+
| Evaluate context offload and its cost | [Token economics](https://github.com/bgauryy/FreeLlama/blob/main/packages/mcp/README.md#token-economics) |
|
|
22
21
|
| Deploy the management layer | [Production runbook](PRODUCTION.md) |
|
|
23
|
-
| Run development and release checks | [Testing](TESTING.md), [release procedure](
|
|
24
|
-
| Embed the routing core | [Rust core](
|
|
25
|
-
| Use the read-only
|
|
22
|
+
| Run development and release checks | [Testing](TESTING.md), [release procedure](https://github.com/bgauryy/FreeLlama/blob/main/RELEASE.md) |
|
|
23
|
+
| Embed the routing core | [Rust core](https://github.com/bgauryy/FreeLlama/blob/main/packages/rust-core/README.md) |
|
|
24
|
+
| Use the read-only status page | `/_freellama/ui` on a running `serve` (see [Monitoring](MONITORING.md)) |
|
|
26
25
|
|
|
27
26
|
## Evidence and design
|
|
28
27
|
|
|
29
28
|
| Goal | Source |
|
|
30
29
|
|---|---|
|
|
31
|
-
| Describe the product and its limits | [Product positioning](PRODUCT_POSITIONING.md) |
|
|
32
30
|
| Review dated findings and project comparisons | [Findings report](dev/FINDINGS_AND_POSITIONING.md) |
|
|
33
31
|
| Measure inference, device activity, and thermal behavior | [Local performance testing](dev/LOCAL_PERFORMANCE_TESTING.md) |
|
|
34
32
|
| Separate management controls from inference tuning | [Ollama optimization boundary](dev/OLLAMA_SYSTEM_OPTIMIZATION.md) |
|
|
35
33
|
| Understand backend selection decisions | [Resource-routing ADR](dev/ADR_RESOURCE_AWARE_BACKEND_ROUTING.md) |
|
|
36
|
-
| Choose a model, adapter, or hardware benchmark | [Benchmark index](../benchmark/README.md) |
|
|
37
|
-
| Inspect research adapter contracts and caveats | [Adapter reference](../benchmark/local/docs/07-adapter-contracts.md) |
|
|
38
34
|
|
|
39
35
|
Historical measurements describe one model, machine, and workload. Inspect current state and
|
|
40
36
|
repeat the relevant checks before using them to select models or set resource limits.
|
|
@@ -41,15 +41,15 @@ Create accepts the following optional fields:
|
|
|
41
41
|
| HTTP field | MCP field | Meaning |
|
|
42
42
|
|---|---|---|
|
|
43
43
|
| `messages` | `messages` | Initial Ollama messages, preserving fields, content, and order. |
|
|
44
|
-
| `route_defaults` |
|
|
45
|
-
| `limits` |
|
|
44
|
+
| `route_defaults` | — | Task-routing defaults for later scoped execution (HTTP/CLI only). |
|
|
45
|
+
| `limits` | — | Per-scope bounds within the server's configured limits (HTTP/CLI only). |
|
|
46
46
|
|
|
47
47
|
Messages require a supported role and can omit content, including for assistant tool calls.
|
|
48
48
|
When present, content must be a string.
|
|
49
49
|
|
|
50
50
|
Routing defaults accept `task`, `objective`, `model`, `required_capabilities`, `context_tokens`,
|
|
51
|
-
`execution_preference`, `min_placement_evidence`, and `min_confidence`. MCP
|
|
52
|
-
|
|
51
|
+
`execution_preference`, `min_placement_evidence`, and `min_confidence`. MCP callers set routing per
|
|
52
|
+
`run_task` call instead. Defaults do not grant admission or qualify a model. Explicit task fields
|
|
53
53
|
take precedence, and execution checks current eligibility and resources.
|
|
54
54
|
|
|
55
55
|
The metadata receipt contains `scope_id`, `revision`, `route_defaults`, `limits`, `message_count`,
|
|
@@ -59,7 +59,7 @@ Token estimates derive from the full serialized message bytes and count both med
|
|
|
59
59
|
This storage-budget estimate can refuse media-heavy history before inference; it does not measure
|
|
60
60
|
image token cost or determine the model context required to execute the history.
|
|
61
61
|
|
|
62
|
-
Fork requires `revision
|
|
62
|
+
Fork requires `revision`; HTTP/CLI forks also accept replacement routing defaults and limit overrides. It creates a
|
|
63
63
|
separate history snapshot with revision `0`; later writes to either scope do not change the other.
|
|
64
64
|
Fork independent branches before parallel execution instead of submitting concurrent writes to one
|
|
65
65
|
scope.
|
|
@@ -138,15 +138,14 @@ precedence. Inspect effective values and their sources with
|
|
|
138
138
|
| `total_max_bytes` | `FREELLAMA_SCOPE_TOTAL_MAX_BYTES` | `16777216` | Aggregate retained history-byte budget. |
|
|
139
139
|
| `ttl_seconds` | `FREELLAMA_SCOPE_TTL_SECONDS` | `3600` | Idle expiry since create, fork, or successful append; reads do not renew it. |
|
|
140
140
|
|
|
141
|
-
Per-scope `limits` accepts `max_messages`, `max_bytes`, `max_estimated_tokens`, and
|
|
142
|
-
|
|
143
|
-
bounds within server policy. Finite safety bounds remain operator-owned. Lowering runtime caps
|
|
141
|
+
Per-scope HTTP/CLI `limits` accepts `max_messages`, `max_bytes`, `max_estimated_tokens`, and
|
|
142
|
+
`ttl_seconds` and can only lower these bounds within server policy. Finite safety bounds remain operator-owned. Lowering runtime caps
|
|
144
143
|
does not summarize or trim stored transcripts; subsequent writes must satisfy the active caps.
|
|
145
144
|
Expired entries are removed during store operations.
|
|
146
145
|
|
|
147
146
|
## Warm API
|
|
148
147
|
|
|
149
|
-
`POST /warm`, MCP `
|
|
148
|
+
`POST /warm`, MCP `run_task {warm:true}`, and CLI `warm` load an exact installed model with a matching
|
|
150
149
|
non-embedding task profile. They do not accept prompts or messages, pull a model, or store conversation
|
|
151
150
|
history. The default task profile is `completion`.
|
|
152
151
|
|
package/docs/TESTING.md
CHANGED
|
@@ -19,6 +19,7 @@ flowchart LR
|
|
|
19
19
|
## Install and build
|
|
20
20
|
|
|
21
21
|
```bash
|
|
22
|
+
corepack enable # Yarn version comes from package.json "packageManager"
|
|
22
23
|
yarn install
|
|
23
24
|
yarn build
|
|
24
25
|
```
|
|
@@ -30,13 +31,16 @@ dependency rather than compiling Rust during installation.
|
|
|
30
31
|
|
|
31
32
|
## Run the test tiers
|
|
32
33
|
|
|
34
|
+
The root `yarn build`, `yarn lint`, and `yarn test` run across every package: the Rust crates
|
|
35
|
+
(`rust-core`, `cli`) through Cargo and the JavaScript workspaces (`mcp`, `cli`, `native/*`) through
|
|
36
|
+
Yarn. Use the `:js` or `:rust` variants to run one side only.
|
|
37
|
+
|
|
33
38
|
| Tier | Command | External requirements |
|
|
34
39
|
|---|---|---|
|
|
35
|
-
|
|
|
36
|
-
| JavaScript
|
|
40
|
+
| All package unit tests (JS + Rust) | `yarn test` | Rust toolchain |
|
|
41
|
+
| JavaScript unit | `yarn test:js` | None |
|
|
37
42
|
| Rust contracts | `yarn test:rust` | Rust toolchain |
|
|
38
|
-
| Local-agent contracts | `yarn test:agents` |
|
|
39
|
-
| Hardware runner contracts | `python3 -m unittest discover -s benchmark/hardware -p 'test_*.py'` | Python 3 standard library; no inference |
|
|
43
|
+
| Local-agent contracts | `yarn test:agents` | None after install; builds the MCP package |
|
|
40
44
|
| TypeScript | `yarn typecheck` | None after install |
|
|
41
45
|
| MCP integration | `yarn test:integration` | Ollama at `FREELLAMA_OLLAMA_ENDPOINT` (default `127.0.0.1:11434`) |
|
|
42
46
|
| MCP end to end | `yarn test:e2e` | Ollama, `freellama serve`, and required models |
|
|
@@ -66,7 +70,7 @@ failures. MCP integration's `model-knowledge.test.ts` exercises the built server
|
|
|
66
70
|
local HTTP fixtures for both Ollama API data and public HTML. It checks optional enrichment,
|
|
67
71
|
assigned-backend inventory, tag pagination, validation, and preservation of benchmark/policy evidence.
|
|
68
72
|
The fixture tests perform no inference, pulls, or deletions. See the
|
|
69
|
-
[metadata reference](
|
|
73
|
+
[metadata reference](MODEL_SELECTION.md#model-metadata) for the public states these tests protect.
|
|
70
74
|
|
|
71
75
|
## Run static Rust checks
|
|
72
76
|
|
|
@@ -156,8 +160,8 @@ a resident routing preview to agree with execution under a low-memory-only hold.
|
|
|
156
160
|
Run the isolated control timing experiments separately from the full suite:
|
|
157
161
|
|
|
158
162
|
```bash
|
|
159
|
-
cargo test --test control_timing_contract -- --include-ignored --nocapture
|
|
160
|
-
cargo test --test task_jobs_contract
|
|
163
|
+
cargo test --test integration control_timing_contract -- --include-ignored --nocapture
|
|
164
|
+
cargo test --test integration task_jobs_contract::measure_deferred_control_latency -- --ignored --nocapture
|
|
161
165
|
```
|
|
162
166
|
|
|
163
167
|
These fixed-budget experiments use simulated backends to measure status, discovery, queue
|
|
@@ -188,32 +192,19 @@ After rebuilding the release, a fresh concurrent smoke check completed both mana
|
|
|
188
192
|
Qwen, one embedding from Nomic, positive Qwen VRAM, zero Nomic VRAM, and primary Ollama 0.33.2
|
|
189
193
|
through raw passthrough.
|
|
190
194
|
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
The runner measures separate sequential and parallel runs of identical workloads across at least
|
|
196
|
-
three pairs, and checks every result before reporting successful-task throughput. It requires a
|
|
197
|
-
configurable median gain (default 5%). Resident GPU bytes establish placement; they do not measure
|
|
198
|
-
device busy percentage, bandwidth or energy efficiency. Sustained throughput and tail latency
|
|
199
|
-
require a representative workload and separate device/thermal observations.
|
|
195
|
+
The portable hardware acceptance, workload, comparison, and load runners were removed from the
|
|
196
|
+
repository; they remain in git history. Compilation alone does not validate a driver, physical
|
|
197
|
+
placement, shared-memory contention, or OCR quality.
|
|
200
198
|
|
|
201
199
|
The historical CPU/GPU observations above predate the corrected paired-run acceptance gate.
|
|
202
|
-
They remain records of those runs
|
|
203
|
-
|
|
204
|
-
For a GPU workload with independent expected answers, use
|
|
205
|
-
`benchmark/hardware/run_workload.py --workload MANIFEST.json --output RECEIPT.json`. It accepts
|
|
206
|
-
distinct generation or vision cases, runs at least three alternating pairs, retains every
|
|
207
|
-
result, and rejects incomplete or incorrect workloads. Its client deadline also bounds a hung
|
|
208
|
-
measurement worker. See the [hardware guide](../benchmark/hardware/README.md) for the manifest
|
|
209
|
-
contract.
|
|
200
|
+
They remain records of those runs and do not qualify a release.
|
|
210
201
|
|
|
211
202
|
## Verify release packages
|
|
212
203
|
|
|
213
204
|
```bash
|
|
214
205
|
# Collect each target's `freellama` executable and `freellama.<target>.node`
|
|
215
206
|
# under release-artifacts/<target>/, then stage all eight platform packages.
|
|
216
|
-
|
|
207
|
+
node scripts/assemble-release.mjs release-artifacts release
|
|
217
208
|
yarn release:verify:publish
|
|
218
209
|
```
|
|
219
210
|
|
|
@@ -225,11 +216,8 @@ prove registry publication or another hardware class.
|
|
|
225
216
|
|
|
226
217
|
## Test one layer while iterating
|
|
227
218
|
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
`yarn test` command passed 152 tests in 1.83 seconds of suite time. Test and production deadlines
|
|
231
|
-
remain unchanged. Preserve failed and passing receipts together; this result does not qualify
|
|
232
|
-
production startup timing.
|
|
219
|
+
When investigating a fixture failure, preserve failed and passing receipts together; a passing
|
|
220
|
+
rerun does not qualify production startup timing.
|
|
233
221
|
|
|
234
222
|
```bash
|
|
235
223
|
yarn build:native
|