stack-guardian 0.2.1__tar.gz → 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. stack_guardian-0.3.0/PKG-INFO +489 -0
  2. stack_guardian-0.3.0/README.md +466 -0
  3. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/pyproject.toml +10 -4
  4. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/__init__.py +1 -1
  5. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/checks.d/10_core.py +36 -5
  6. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/cli.py +190 -32
  7. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/collect.py +42 -7
  8. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/core.py +30 -5
  9. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/mcp_server.py +126 -46
  10. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/multinode.py +48 -23
  11. stack_guardian-0.3.0/stack_guardian/nodes.py +202 -0
  12. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/parsers.py +34 -10
  13. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/serve.py +68 -11
  14. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/state.py +37 -1
  15. stack_guardian-0.3.0/stack_guardian.egg-info/PKG-INFO +489 -0
  16. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian.egg-info/SOURCES.txt +8 -0
  17. stack_guardian-0.3.0/stack_guardian.egg-info/requires.txt +4 -0
  18. stack_guardian-0.3.0/tests/test_glama_quality.py +56 -0
  19. stack_guardian-0.3.0/tests/test_nodes_file.py +88 -0
  20. stack_guardian-0.3.0/tests/test_review_1021.py +312 -0
  21. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_review_fixes.py +4 -3
  22. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_selftests.py +2 -2
  23. stack_guardian-0.3.0/tests/test_spdx_license.py +28 -0
  24. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_tool_annotations.py +2 -2
  25. stack_guardian-0.3.0/tests/test_tool_definitions.py +87 -0
  26. stack_guardian-0.3.0/tests/test_tools_list_budget.py +59 -0
  27. stack_guardian-0.2.1/PKG-INFO +0 -203
  28. stack_guardian-0.2.1/README.md +0 -183
  29. stack_guardian-0.2.1/stack_guardian.egg-info/PKG-INFO +0 -203
  30. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/LICENSE +0 -0
  31. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/setup.cfg +0 -0
  32. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/__main__.py +0 -0
  33. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/finding.py +0 -0
  34. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian/fixes.py +0 -0
  35. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian.egg-info/dependency_links.txt +0 -0
  36. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian.egg-info/entry_points.txt +0 -0
  37. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/stack_guardian.egg-info/top_level.txt +0 -0
  38. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_doctor_fix.py +0 -0
  39. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_examples.py +0 -0
  40. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_mcp_subcommand.py +0 -0
  41. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_metrics.py +0 -0
  42. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_model_expiring.py +0 -0
  43. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_openai_collector.py +0 -0
  44. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_readme_1014.py +0 -0
  45. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_registry_markers.py +0 -0
  46. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_ttft.py +0 -0
  47. {stack_guardian-0.2.1 → stack_guardian-0.3.0}/tests/test_watch.py +0 -0
@@ -0,0 +1,489 @@
1
+ Metadata-Version: 2.4
2
+ Name: stack-guardian
3
+ Version: 0.3.0
4
+ Summary: A health doctor for a home AI stack: Ollama nodes, GPUs and tailnet, with graded confidence.
5
+ Author: LuminariSoftwares
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/LuminariSoftwares/stack-guardian
8
+ Project-URL: Issues, https://github.com/LuminariSoftwares/stack-guardian/issues
9
+ Project-URL: Changelog, https://github.com/LuminariSoftwares/stack-guardian/blob/main/CHANGELOG.md
10
+ Keywords: ollama,gpu,vram,tailscale,health-check,local-llm,mcp
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Programming Language :: Python :: 3
13
+ Classifier: Operating System :: OS Independent
14
+ Classifier: Environment :: Console
15
+ Classifier: Topic :: System :: Monitoring
16
+ Requires-Python: >=3.11
17
+ Description-Content-Type: text/markdown
18
+ License-File: LICENSE
19
+ Provides-Extra: test
20
+ Requires-Dist: pytest>=7; extra == "test"
21
+ Requires-Dist: build>=1; extra == "test"
22
+ Dynamic: license-file
23
+
24
+ # stack-guardian
25
+
26
+ [![ci](https://github.com/LuminariSoftwares/stack-guardian/actions/workflows/ci.yml/badge.svg)](https://github.com/LuminariSoftwares/stack-guardian/actions/workflows/ci.yml)
27
+ [![privacy](https://github.com/LuminariSoftwares/stack-guardian/actions/workflows/privacy.yml/badge.svg)](https://github.com/LuminariSoftwares/stack-guardian/actions/workflows/privacy.yml)
28
+
29
+ mcp-name: io.github.LuminariSoftwares/stack-guardian
30
+
31
+ A health doctor for a home AI stack. It collects a snapshot of your nodes (Ollama or OpenAI-compatible endpoints,
32
+ loaded models, NVIDIA GPUs, tailnet peers), runs small Python rules against it, remembers the last runs, and reports
33
+ findings with a confidence grade: one sample is at most `LIKELY`, the same finding twice in a row is `VERIFIED`. It
34
+ exists because "the model is slow" on a multi-machine local stack has a dozen possible causes, and an agent or a person
35
+ should get a graded, evidence-backed answer instead of a guess. A finding can carry a typed fix that you run on
36
+ purpose, never automatically. It ships as a CLI, an MCP server for coding agents, a tiny HTTP endpoint with Prometheus
37
+ metrics, and a DeepSeek Harness (DSH) plugin.
38
+
39
+ Status: **0.3.0.** Standard library only, Python 3.11+, no runtime dependencies.
40
+
41
+ One of four small MIT-licensed tools for people running local models, each one doing a single job:
42
+ [code-guardian](https://github.com/LuminariSoftwares/code-guardian) (review your own diff before anyone else sees
43
+ it), [context-guardian](https://github.com/LuminariSoftwares/context-guardian) (keep a long session inside its
44
+ context window), [tool-guardian](https://github.com/LuminariSoftwares/tool-guardian) (check an MCP server's tool
45
+ definitions) and this one (check the machines underneath). They share no code and are useful separately.
46
+
47
+ *Not affiliated with StackGuardian, the infrastructure-automation company. Same word, unrelated projects.*
48
+
49
+ ## Quick start (60 seconds, offline)
50
+
51
+ **macOS / Linux**
52
+
53
+ ```
54
+ pip install stack-guardian
55
+ curl -LO https://raw.githubusercontent.com/LuminariSoftwares/stack-guardian/main/examples/snapshot.json
56
+ curl -LO https://raw.githubusercontent.com/LuminariSoftwares/stack-guardian/main/examples/ctx.json
57
+ stack-guardian doctor --snapshot snapshot.json --ctx ctx.json
58
+ ```
59
+
60
+ **Windows (PowerShell), or anywhere you would rather not touch the system Python**
61
+
62
+ ```
63
+ pipx install stack-guardian
64
+ curl.exe -LO https://raw.githubusercontent.com/LuminariSoftwares/stack-guardian/main/examples/snapshot.json
65
+ curl.exe -LO https://raw.githubusercontent.com/LuminariSoftwares/stack-guardian/main/examples/ctx.json
66
+ stack-guardian doctor --snapshot snapshot.json --ctx ctx.json
67
+ ```
68
+
69
+ `pipx` puts it in its own virtualenv and on your PATH (`pip install pipx` if you do not have it). `uvx
70
+ stack-guardian doctor --snapshot snapshot.json` runs it without installing anything at all.
71
+
72
+ The example snapshot makes every shipped rule fire once, so you see each finding type without touching a real stack.
73
+ The command exits `2` because the example contains FAIL findings; that is the expected result. Each line ends with
74
+ the confidence grade - one run is at most `(LIKELY)`, the same finding on a second run is `(VERIFIED)`.
75
+
76
+ Point it at your own machines next:
77
+
78
+ ```
79
+ stack-guardian init # writes nodes.json for this machine; never overwrites one
80
+ stack-guardian doctor --nodes nodes.json
81
+ ```
82
+
83
+ `init` probes `http://127.0.0.1:11434` to see whether Ollama is answering, and lists your tailnet peers (if
84
+ `tailscale` is on PATH) as commented-out candidates you can move up into the file. Edit it by hand from there; the
85
+ shape is in [examples/nodes.json](examples/nodes.json) and under [What it does](#what-it-does).
86
+
87
+ From a clone instead (for writing rules or running the tests):
88
+
89
+ ```
90
+ git clone https://github.com/LuminariSoftwares/stack-guardian.git
91
+ cd stack-guardian
92
+ python -m venv .venv
93
+ .venv\Scripts\activate # Windows; macOS/Linux: source .venv/bin/activate
94
+ pip install -e ".[test]" # drop [test] if you only want to run it, not test it
95
+ stack-guardian doctor --snapshot examples/snapshot.json --ctx examples/ctx.json
96
+ python -m pytest -q
97
+ ```
98
+
99
+ ## Architecture
100
+
101
+ ```mermaid
102
+ flowchart LR
103
+ nodes["nodes.json"] --> collect
104
+ subgraph collect["collect: one snapshot, all nodes in parallel"]
105
+ direction TB
106
+ ol["Ollama /api/tags, /api/ps<br/>OpenAI-compatible /v1/models"]
107
+ gpu["nvidia-smi<br/>(local node only)"]
108
+ ts["tailscale status --json"]
109
+ end
110
+ collect --> snap[("snapshot dict")]
111
+ snap --> rules["checks.d/*.py rules<br/>run concurrently;<br/>a crash becomes check.crashed"]
112
+ rules --> grade["state.grade<br/>LIKELY until seen twice, then VERIFIED"]
113
+ hist[("state.json<br/>last 5 runs per node")] <--> grade
114
+ grade --> cli["CLI: doctor<br/>exit 0 / 1 / 2"]
115
+ grade --> mcp["MCP stdio server<br/>read-only, never records"]
116
+ grade --> http["serve: /snapshot, /metrics"]
117
+ grade -.->|"finding.fix"| fix["doctor --fix ID<br/>dry run unless --apply;<br/>snapshot + audit line"]
118
+ ```
119
+
120
+ | module | job |
121
+ |---|---|
122
+ | `collect.py`, `multinode.py`, `parsers.py` | probe every node in parallel under one deadline; parse Ollama, OpenAI-compatible, nvidia-smi and tailscale output into one snapshot dict. A dead endpoint is data, never an exception. |
123
+ | `core.py`, `finding.py`, `checks.d/` | load rule files, run them concurrently, and build findings with a fixed shape (`code, severity, level, node, summary, evidence, try, confidence, fix`). |
124
+ | `state.py` | keep the last runs per node and grade confidence from streaks. |
125
+ | `fixes.py` | the three typed fix actions, their dry-run plan, snapshot and audit line. |
126
+ | `cli.py`, `mcp_server.py`, `serve.py` | the three front ends over the same doctor. |
127
+
128
+ ### Design decisions
129
+
130
+ Each is written up with its alternatives in [docs/adr/](docs/adr/).
131
+
132
+ - **Standard library only** ([ADR 0001](docs/adr/0001-standard-library-only.md)). `pip install` pulls nothing else, so
133
+ it installs on any machine with Python 3.11. Trade-off: the HTTP client, JSON-RPC loop and Prometheus renderer are
134
+ hand-written and have to be tested here.
135
+ - **Rules are plain Python files in a folder** ([ADR 0002](docs/adr/0002-rules-as-plain-python-files.md)). A rule is a
136
+ function `(snapshot, ctx) -> finding | list | None`; a rule that raises becomes a `check.crashed` WARN, so a buggy
137
+ rule never reports a broken stack. Trade-off: rules run in-process with your privileges; only load rule folders you
138
+ trust.
139
+ - **Confidence from history, not from one sample** ([ADR 0003](docs/adr/0003-confidence-from-run-history.md)). One run
140
+ is at most `LIKELY`; two bad runs in a row are `VERIFIED`. Trade-off: a real outage needs a second run before it is
141
+ `VERIFIED`, and the history is a local file.
142
+ - **Typed fixes, dry run by default** ([ADR 0004](docs/adr/0004-typed-audited-fixes.md)). Exactly three actions, never
143
+ a shell string from a rule; nothing changes without `--apply`, and every applied fix leaves a snapshot and an audit
144
+ line. Trade-off: anything outside the three actions has to be done by hand.
145
+ - **Agent-facing surfaces are read-only** ([ADR 0005](docs/adr/0005-read-only-agent-surfaces.md)). The MCP server and
146
+ the HTTP endpoint never fix, never write the run history, and `serve` binds to loopback unless told otherwise.
147
+ Trade-off: an agent can diagnose but must hand the fix back to a person.
148
+
149
+ ## For coding agents
150
+
151
+ - **MCP server** (any MCP client): `stack-guardian-mcp` speaks MCP over stdio with three read-only tools,
152
+ `stack_health`, `stack_snapshot` and `stack_explain`. See [MCP server](#mcp-server) for client configs and the tool
153
+ reference.
154
+ - **DSH plugin** (`npm install dsh-stack-guardian`, source in `dsh/`, Node 20+): adds one line at the start of a
155
+ DeepSeek Harness session when the doctor reports WARN or FAIL, and nothing otherwise. It runs
156
+ `stack-guardian doctor --json --quiet`, so set `STACK_GUARDIAN_NODES` to your nodes file in the harness environment.
157
+ - **Watching and scraping:** `stack-guardian doctor --watch 30 --json` prints one JSON object per finding that changes
158
+ state (new, resolved, severity change), so an agent can tail it; `stack-guardian-serve --nodes FILE` also answers
159
+ `GET /metrics` in Prometheus text format.
160
+
161
+ ## MCP server
162
+
163
+ `stack-guardian-mcp` (also `stack-guardian mcp`) speaks MCP over stdio (JSON-RPC 2.0, protocol `2024-11-05`). It needs
164
+ a nodes file ([examples/nodes.json](examples/nodes.json)); give its path with `--nodes FILE` or the
165
+ `STACK_GUARDIAN_NODES` environment variable. Optional: `--ctx FILE` (rule settings), `--state FILE` (run history to
166
+ grade against), `--ttl SECONDS` (how long one snapshot is reused, default 10).
167
+
168
+ Run it without installing anything:
169
+
170
+ ```
171
+ uvx stack-guardian mcp --nodes /path/to/nodes.json
172
+ ```
173
+
174
+ **Claude Desktop** (`claude_desktop_config.json`):
175
+
176
+ ```json
177
+ {
178
+ "mcpServers": {
179
+ "stack-guardian": {
180
+ "command": "uvx",
181
+ "args": ["stack-guardian", "mcp"],
182
+ "env": {"STACK_GUARDIAN_NODES": "/path/to/nodes.json"}
183
+ }
184
+ }
185
+ }
186
+ ```
187
+
188
+ **Claude Code:**
189
+
190
+ ```
191
+ claude mcp add stack-guardian -e STACK_GUARDIAN_NODES=/path/to/nodes.json -- uvx stack-guardian mcp
192
+ ```
193
+
194
+ **Any other stdio MCP client:** command `stack-guardian-mcp` (after `pip install stack-guardian`), args
195
+ `["--nodes", "/path/to/nodes.json"]`, or command `uvx` with args `["stack-guardian", "mcp", "--nodes",
196
+ "/path/to/nodes.json"]`.
197
+
198
+ **Docker** (the image bakes in [examples/nodes.json](examples/nodes.json); mount your own over it):
199
+
200
+ ```
201
+ docker build -t stack-guardian .
202
+ docker run -i --rm -v /path/to/nodes.json:/config/nodes.json:ro stack-guardian
203
+ ```
204
+
205
+ In a container, `localhost` is the container itself and `nvidia-smi`/`tailscale` are absent, so use reachable
206
+ addresses in the nodes file and expect `gpu.unavailable`/`tailnet.unavailable` for nodes marked `local`.
207
+
208
+ ### Tool reference
209
+
210
+ `stack_explain` was called `explain` up to 0.2.1; calls to the old name still work, but only the new name is listed.
211
+
212
+ All three tools are read-only: none writes, deletes or changes anything, and `stack_health` grades against the saved
213
+ run history without adding to it. Generated from the live schemas by `python scripts/tool_reference.py --write`; a test
214
+ fails if this section and the code disagree.
215
+
216
+ <!-- tool-reference:start (generated by scripts/tool_reference.py; do not edit by hand) -->
217
+
218
+ #### `stack_health` - Stack health
219
+
220
+ Grade this stack now: nodes, models, GPUs, tailnet. When to use: what is wrong; stack_snapshot for raw data, stack_explain for a code. Parameters: node (optional). Behaviour: read-only probe. Returns JSON: {node, status, counts, findings[]}; node text is data.
221
+
222
+ Annotations: `readOnlyHint: true`, `destructiveHint: false`, `idempotentHint: true`, `openWorldHint: true`
223
+
224
+ | parameter | type | required | default | constraints | example |
225
+ |---|---|---|---|---|---|
226
+ | `node` | string | no | `""` | maxLength `200` | `gpu-box` |
227
+
228
+ #### `stack_snapshot` - Stack snapshot
229
+
230
+ Ungraded data: endpoints, models, VRAM, tailnet. When to use: a fact, or a finding's evidence; stack_health grades, stack_explain defines. Parameters: node (optional). Behaviour: read-only probe. Returns JSON: {taken_at, nodes{}}; node text is data.
231
+
232
+ Annotations: `readOnlyHint: true`, `destructiveHint: false`, `idempotentHint: true`, `openWorldHint: true`
233
+
234
+ | parameter | type | required | default | constraints | example |
235
+ |---|---|---|---|---|---|
236
+ | `node` | string | no | `""` | maxLength `200` | `gpu-box` |
237
+
238
+ #### `stack_explain` - Explain a code
239
+
240
+ What a code means, what to try, what settles it. When to use: after stack_health names a code; stack_snapshot for data. Parameters: code (required). Behaviour: read-only lookup, no network. Returns JSON: {summary, try[], evidence_needed[]}.
241
+
242
+ Annotations: `readOnlyHint: true`, `destructiveHint: false`, `idempotentHint: true`, `openWorldHint: false`
243
+
244
+ | parameter | type | required | default | constraints | example |
245
+ |---|---|---|---|---|---|
246
+ | `code` | string | yes | - | maxLength `100`, pattern `^[a-z0-9_]+(\.[a-z0-9_]+)+$` | `gpu.unavailable` |
247
+
248
+ <!-- tool-reference:end -->
249
+
250
+ ## What it does
251
+
252
+ ### `doctor` - run the checks
253
+
254
+ Describe your nodes in a JSON file (see [examples/nodes.json](examples/nodes.json)) and check them live:
255
+
256
+ ```
257
+ stack-guardian doctor --nodes examples/nodes.json --ctx examples/ctx.json
258
+ ```
259
+
260
+ A node entry takes `ollama` (a list of base URLs), `local: true` (read GPUs with `nvidia-smi` on this machine) and
261
+ `tailnet_host` (look the peer up in this machine's `tailscale status`). With neither `--snapshot` nor `--nodes`,
262
+ `doctor` reads the nodes file named by `$STACK_GUARDIAN_NODES`.
263
+
264
+ Options: `--checks-dir DIR` (your own rules, default is the shipped `checks.d`), `--ctx FILE` (rule settings such as
265
+ the model each node is expected to hold), `--state FILE` (default `$STACK_GUARDIAN_STATE` or
266
+ `~/.stack-guardian/state.json`), `--timeout SECONDS` (the wall-clock deadline for the whole collection, default 10,
267
+ 30 with `--ttft`), `--json`, `--quiet` (print nothing when all is OK). `stack-guardian --version` and
268
+ `stack-guardian --help` do what you expect.
269
+
270
+ Every node is probed at once under that one deadline, and the process ends with the report: a node that answers one
271
+ byte at a time is written off as `node.unreachable`, not waited for.
272
+
273
+ The nodes file is validated when it is read. A node needs something to probe (`ollama`, `endpoints` with
274
+ `"kind": "openai"`, `local: true`, or `tailnet_host`), and URLs need a scheme; anything else is a usage error
275
+ (exit `3`) naming the node, rather than a confident wrong answer. A single URL string where a list belongs is
276
+ accepted, with one note on stderr. Keys starting with `_` are ignored, so a file can carry comments and candidate
277
+ nodes; `"_options": {"timeout_s": 20}` sets the deadline for that file.
278
+
279
+ Human output ends each line with the finding's confidence - `(LIKELY)` on one run, `(VERIFIED)` once the same
280
+ finding has been seen twice - and `worst: X` stays the last line. `--json` prints
281
+ `{"schema": 1, "findings": [...], "worst": "..."}`.
282
+
283
+ Exit codes: `0` all OK/INFO, `1` a WARN, `2` a FAIL, `3` usage, `4` the doctor itself crashed.
284
+
285
+ Shipped rules: `node.unreachable`, `ollama.unreachable`, `model.not_loaded`, `model.expiring`, `model.slow_ttft`,
286
+ `vram.unattributed`, `gpu.unavailable`, `gpu.parse_degraded`, `tailnet.unavailable`, `tailnet.peer_offline`,
287
+ `tailnet.relayed`. Each raises
288
+ one finding per affected node. A rule that raises becomes a `check.crashed` WARN - a buggy rule never reports a broken
289
+ stack.
290
+
291
+ **LM Studio, vLLM, llama.cpp server.** Give the node `"kind": "openai"` and list its base URLs under `endpoints`.
292
+ It is read from `GET /v1/models` (and `GET /v1/models/<id>` when the server has it) into the same shape as an Ollama
293
+ endpoint, so every rule above applies; a model a server lists as unloaded does not count as loaded.
294
+
295
+ ```
296
+ {"gpu-box": {"kind": "openai", "endpoints": ["http://127.0.0.1:1234"]}}
297
+ ```
298
+
299
+ **`model.expiring`** (WARN): a loaded Ollama model whose `/api/ps` `expires_at` is within 60 s of the snapshot time;
300
+ INFO when `/api/ps` is unavailable. Change the window with `"model_expiring_s": 120` in `--ctx`.
301
+
302
+ **`--watch N`** re-runs the doctor every N seconds and prints only the findings that changed state, each with a
303
+ timestamp. Ctrl+C exits `0` with a one-line summary. With `--json`, each change is one JSON object on stdout.
304
+
305
+ ```
306
+ stack-guardian doctor --nodes examples/nodes.json --watch 30
307
+ ```
308
+
309
+ **`--ttft`** sends a 1-token generation to every loaded model on each Ollama endpoint and records `ttft_ms` on the
310
+ model in the snapshot. It costs a generation, so it never runs unless you pass it, and it needs `--nodes`. The probe
311
+ keeps each model's unload time as it was. `model.slow_ttft` (INFO) fires above 2000 ms; change it with
312
+ `"ttft_slow_ms": 1500` in `--ctx`.
313
+
314
+ ```
315
+ stack-guardian doctor --nodes examples/nodes.json --ttft
316
+ ```
317
+
318
+ ### `doctor --fix ID` - run a finding's typed fix
319
+
320
+ A rule may attach `fix={"action": ..., "args": {...}}` to a finding. There are exactly three actions and never a raw
321
+ shell string:
322
+
323
+ | action | what it does | needs |
324
+ |---|---|---|
325
+ | `unload_model` | POST `keep_alive: 0` to an Ollama `/api/generate` | `node`, `url`, `model` |
326
+ | `restart_lane` | runs the argv list you declared for that lane | `lane` declared in `--config` |
327
+ | `set_env` | writes `NAME=value` to `<state folder>/env` and sets it for the process | `name`, `value` |
328
+
329
+ ```
330
+ stack-guardian doctor --checks-dir examples/checks.d --snapshot examples/snapshot.json --fix ollama.stuck --config examples/lanes.json
331
+ ```
332
+
333
+ That is a **dry run**: it prints the plan and changes nothing. Add `--apply` to perform it. An applied fix first saves
334
+ the snapshot under `<state folder>/snapshots/`, then runs, then appends one JSON line to `<state folder>/fixes.jsonl`
335
+ (`--audit FILE` to move it). A `set_env` value is never written to the audit, only its length. `ID` is the finding
336
+ code, or `code@node` when several nodes raise the same code. With `--fix`, the exit code is the fix's: `0` plan or
337
+ success, `1` refused or failed, `3` usage.
338
+
339
+ ### Other commands
340
+
341
+ - `stack-guardian init [--out nodes.json] [--ollama URL]` writes a starter nodes file for this machine: it probes
342
+ the local Ollama, lists tailnet peers as commented-out candidates, and refuses (exit `3`) rather than overwrite a
343
+ file that is already there.
344
+ - `stack-guardian snapshot --nodes FILE [--ttft] [--timeout SECONDS]` prints the raw snapshot the rules read.
345
+ - `stack-guardian-serve --nodes FILE [--ctx FILE] [--port 8788] [--cache SECONDS]` serves a fresh snapshot at
346
+ `GET /snapshot` and Prometheus metrics at `GET /metrics` (read-only, loopback unless you pass
347
+ `--host H --allow-remote`). It is a scrape target, not a cluster protocol: no other part of stack-guardian reads
348
+ it. Every request collects a new snapshot; `--cache SECONDS` (default `0`, off) reuses one for that long if a
349
+ scrape interval is faster than your nodes answer. It has no auth: bind to localhost or a tailnet IP only.
350
+ - The same server answers `GET /metrics` in Prometheus text format, running the shipped rules on a fresh snapshot:
351
+ `stack_guardian_finding{code,severity,node}` (1 while raised, 0 once resolved),
352
+ `stack_guardian_worst_severity` (0 OK, 1 INFO, 2 WARN, 3 FAIL) and `stack_guardian_last_run_timestamp`.
353
+
354
+ ```
355
+ curl http://127.0.0.1:8788/metrics
356
+ ```
357
+ - `stack-guardian-mcp --nodes FILE [--ctx FILE] [--state FILE] [--ttl SECONDS]`: see [MCP server](#mcp-server).
358
+
359
+ ## Writing a rule
360
+
361
+ ```python
362
+ from stack_guardian import finding as F
363
+
364
+ def ollama_up(snapshot, ctx):
365
+ if not snapshot["nodes"]["gpu-box"].get("reachable"):
366
+ return F.make_finding("ollama.unreachable", "missing", "FAIL", "gpu-box", "Ollama does not answer")
367
+ return None
368
+
369
+ CHECKS = [("ollama.up", ollama_up)]
370
+ ```
371
+
372
+ Drop the file in a folder and point `--checks-dir` at it. A full example with a fix is in
373
+ [examples/checks.d/20_ollama_fix.py](examples/checks.d/20_ollama_fix.py). A rule may also return a list of findings,
374
+ one per node.
375
+
376
+ ## Examples
377
+
378
+ | file | what it is |
379
+ |---|---|
380
+ | `examples/nodes.json` | three nodes: a local one with GPUs, two remote ones over the tailnet |
381
+ | `examples/ctx.json` | rule settings: which model gpu-box-2 should hold |
382
+ | `examples/snapshot.json` | an offline snapshot on which every shipped rule fires once, for trying `doctor` and `--fix` |
383
+ | `examples/lanes.json` | the lanes `restart_lane` is allowed to run |
384
+ | `examples/checks.d/20_ollama_fix.py` | a custom rule whose finding carries a fix |
385
+
386
+ ## Troubleshooting
387
+
388
+ By finding code - the same text `stack_explain` gives an agent:
389
+
390
+ | you see | it means | first thing to try |
391
+ |---|---|---|
392
+ | `node.unreachable` | the node did not answer within the deadline, or the collector raised | check the machine is on; if it is only slow, `--timeout 30` or `"_options": {"timeout_s": 30}` |
393
+ | `ollama.unreachable` | the endpoint itself refused or timed out | `curl http://HOST:11434/api/version` from this machine; check the port and that Ollama listens on more than loopback (`OLLAMA_HOST`) |
394
+ | `model.not_loaded` | `--ctx` expects a model that is not resident | load it with one short generate, or drop the expectation from `--ctx` |
395
+ | `model.expiring` | a loaded model unloads within 60 s (INFO: `/api/ps` is missing, so it cannot be told) | send a request with a longer `keep_alive`, or set `OLLAMA_KEEP_ALIVE` on that node |
396
+ | `model.slow_ttft` | a `--ttft` probe took over 2000 ms to the first token | check the model fits in VRAM, and that nothing else is generating on that GPU |
397
+ | `vram.unattributed` | VRAM is in use that no listed process explains | `nvidia-smi` and look at the compute apps; on Windows some processes report `[N/A]` and cannot be attributed |
398
+ | `gpu.unavailable` | `nvidia-smi` is missing, or ran and failed - the finding says which | NVIDIA only; if it is installed, run `nvidia-smi` by hand and fix the driver |
399
+ | `gpu.parse_degraded` | `nvidia-smi` printed a GPU line with `[N/A]` fields, so that card is missing from the report | run the query by hand (the evidence quotes the line); update the driver |
400
+ | `tailnet.unavailable` | `tailscale` is not on PATH, or `tailscale status` failed | not on a tailnet? drop `tailnet_host` from that node; otherwise start `tailscaled` or log in |
401
+ | `tailnet.peer_offline` | the tailnet says the peer is offline | check the machine is up and `tailscale status` on it |
402
+ | `tailnet.relayed` | the peer is reachable only through a DERP relay | check UDP 41641 and your firewall; `tailscale netcheck` on both ends |
403
+ | `check.crashed` | one of your rule files raised | the evidence carries the exception; fix or remove that file |
404
+
405
+ And by symptom:
406
+
407
+ | you see | why |
408
+ |---|---|
409
+ | `stack-guardian: usage: ... nothing to probe` | a node in the nodes file has no endpoints, no `local: true` and no `tailnet_host`. It used to report `worst: OK`. |
410
+ | `stack-guardian: usage: ... "kind": "openai"` | the node lists `endpoints` but is not marked as an OpenAI-compatible node |
411
+ | every finding says `(LIKELY)` | confidence comes from history: run it twice against the same `--state` file |
412
+ | the MCP server never says `VERIFIED` | by design, it grades against the saved history without adding to it; let the CLI or `--watch` build the history |
413
+ | `doctor` exits `2` on the example snapshot | that is the point of the example: it fires every rule once |
414
+
415
+ ## What leaves the machine, and what is written
416
+
417
+ - **Out:** HTTP(S) requests to exactly the base URLs in your nodes file (`/api/version`, `/api/tags`, `/api/ps`, or
418
+ `/v1/models` for an OpenAI-compatible node), and with `--ttft` one 1-token generation per loaded model. No
419
+ credentials are sent, redirects are followed, and nothing is sent anywhere else: no telemetry, no update check,
420
+ no analytics. `nvidia-smi` and `tailscale status --json` run locally.
421
+ - **In:** nothing is downloaded or installed at run time. The package has no runtime dependencies.
422
+ - **Written:** the state folder, `$STACK_GUARDIAN_STATE` or `~/.stack-guardian/`, created `0700`. It holds
423
+ `state.json` (the last 5 runs per node: codes and levels, no measurements) and, only after `--fix ... --apply`,
424
+ `fixes.jsonl` (one audit line per applied fix) and `snapshots/` (the snapshot taken just before it). `set_env`
425
+ also writes `env`. Those files are created `0600`; a `set_env` value is never written to the audit, only its
426
+ length. `doctor --json`, `snapshot` and the MCP server write nothing but their own output.
427
+ - **Shown:** a snapshot contains your node names, model names, VRAM figures, GPU process names and tailnet state.
428
+ That is what `stack-guardian-serve` serves and what an MCP client receives, so treat it as you would `nvidia-smi`
429
+ output.
430
+
431
+ ## Limits
432
+
433
+ What it does not do, so you are not surprised:
434
+
435
+ - **GPUs: NVIDIA only, and only on the machine running stack-guardian.** GPU data comes from `nvidia-smi` on a node
436
+ marked `"local": true`. Remote nodes report endpoints, models and tailnet state, not VRAM. Nothing in the package
437
+ reads another machine's `/snapshot`, so there is no way to pull a remote node's GPUs into this report: run
438
+ `stack-guardian` on that machine too, or scrape its `stack-guardian-serve` `/metrics` with Prometheus. No AMD,
439
+ Intel or Apple GPU support.
440
+ - **Tailnet state is this machine's view.** Peers are read from the local `tailscale status --json`; a peer can be
441
+ offline to you and online to another node.
442
+ - **`VERIFIED` needs two runs.** The MCP server never records a run, so on its own it reports at most `LIKELY` unless
443
+ the CLI (or `--watch`) has built up history in the same state file.
444
+ - **No auth on `stack-guardian-serve`.** It is read-only and loopback by default; it is not meant for the open internet.
445
+ - **MCP over stdio only**, protocol `2024-11-05`: results are JSON text content, without `outputSchema` or
446
+ `structuredContent`.
447
+ - **Rules run in-process.** A rule folder is code; only point `--checks-dir` at rules you trust.
448
+ - **`--ttft` costs a real generation** per loaded model, which is why it is opt-in.
449
+
450
+ ## How it's tested
451
+
452
+ ```
453
+ pip install -e ".[test]"
454
+ python -m pytest -q
455
+ ```
456
+
457
+ - Every module carries a counted `--selftest` (pure functions, injected HTTP and command runners, no network); the
458
+ suite runs each one.
459
+ - Integration tests drive the real CLI and the MCP server over stdio in a subprocess, including the tool list and its
460
+ annotations, and run the `examples/` exactly as this README shows them.
461
+ - `tests/test_tool_definitions.py` fails if any MCP tool or parameter loses its description, type, example, default or
462
+ constraints, if a shipped finding code has no `stack_explain` entry, or if the tool reference above is stale.
463
+ - `tests/test_tools_list_budget.py` starts the real MCP server and fails if the whole `tools/list` payload grows
464
+ past its token budget, because those definitions are sent on every request.
465
+ - CI runs the suite on Ubuntu, Windows and macOS (Python 3.11) and on Ubuntu with 3.12 and 3.13, builds the wheel
466
+ and runs it from a clean virtualenv
467
+ (so a release without its rules fails), builds the Docker image and lists its tools, syntax-checks the DSH plugin,
468
+ and runs a privacy scan of every tracked file.
469
+
470
+ ## Contributing and security
471
+
472
+ Pull requests are welcome; [CONTRIBUTING.md](CONTRIBUTING.md) has the setup, the house style and the three rules
473
+ that matter (a test you watched fail first, no new runtime dependency, the tool-definition token budget).
474
+
475
+ Found something exploitable? Please report it privately through the repository's Security tab, not a public issue:
476
+ [SECURITY.md](SECURITY.md) explains how, and lists what this tool does by design (rules are code, node URLs are
477
+ fetched, `serve` has no auth).
478
+
479
+ ## Credits
480
+
481
+ No code is copied from other projects. The wire formats follow their public specs: the
482
+ [Model Context Protocol](https://modelcontextprotocol.io) (JSON-RPC over stdio, tool annotations) and the
483
+ [Prometheus text exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/). The data comes from
484
+ the documented interfaces of [Ollama](https://github.com/ollama/ollama/blob/main/docs/api.md), `nvidia-smi` and
485
+ `tailscale status --json`.
486
+
487
+ ## License
488
+
489
+ MIT. See [LICENSE](LICENSE).