racklint 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- racklint-0.1.0/LICENSE +21 -0
- racklint-0.1.0/PKG-INFO +342 -0
- racklint-0.1.0/README.md +306 -0
- racklint-0.1.0/pyproject.toml +70 -0
- racklint-0.1.0/requirements-dev.txt +9 -0
- racklint-0.1.0/requirements.txt +8 -0
- racklint-0.1.0/setup.cfg +4 -0
- racklint-0.1.0/src/racklint/__init__.py +66 -0
- racklint-0.1.0/src/racklint/baseline.py +29 -0
- racklint-0.1.0/src/racklint/checks.py +116 -0
- racklint-0.1.0/src/racklint/cli.py +127 -0
- racklint-0.1.0/src/racklint/engine.py +60 -0
- racklint-0.1.0/src/racklint/errors.py +18 -0
- racklint-0.1.0/src/racklint/models.py +131 -0
- racklint-0.1.0/src/racklint/netbox.py +116 -0
- racklint-0.1.0/src/racklint/py.typed +0 -0
- racklint-0.1.0/src/racklint/redfish.py +172 -0
- racklint-0.1.0/src/racklint/render.py +46 -0
- racklint-0.1.0/src/racklint/settings.py +24 -0
- racklint-0.1.0/src/racklint.egg-info/PKG-INFO +342 -0
- racklint-0.1.0/src/racklint.egg-info/SOURCES.txt +33 -0
- racklint-0.1.0/src/racklint.egg-info/dependency_links.txt +1 -0
- racklint-0.1.0/src/racklint.egg-info/entry_points.txt +2 -0
- racklint-0.1.0/src/racklint.egg-info/requires.txt +15 -0
- racklint-0.1.0/src/racklint.egg-info/top_level.txt +1 -0
- racklint-0.1.0/tests/test_api.py +31 -0
- racklint-0.1.0/tests/test_baseline.py +50 -0
- racklint-0.1.0/tests/test_checks.py +144 -0
- racklint-0.1.0/tests/test_cli.py +118 -0
- racklint-0.1.0/tests/test_engine.py +104 -0
- racklint-0.1.0/tests/test_models.py +44 -0
- racklint-0.1.0/tests/test_netbox.py +138 -0
- racklint-0.1.0/tests/test_redfish_client.py +77 -0
- racklint-0.1.0/tests/test_redfish_reader.py +60 -0
- racklint-0.1.0/tests/test_render.py +49 -0
racklint-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Reza Iskandar Achmad
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
racklint-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,342 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: racklint
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Check bare-metal and GPU servers against NetBox before they go into service.
|
|
5
|
+
Author: Reza Iskandar Achmad
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Repository, https://github.com/riskand/racklint
|
|
8
|
+
Keywords: redfish,bmc,netbox,commissioning,gpu,data-centre
|
|
9
|
+
Classifier: Development Status :: 3 - Alpha
|
|
10
|
+
Classifier: Environment :: Console
|
|
11
|
+
Classifier: Intended Audience :: System Administrators
|
|
12
|
+
Classifier: Operating System :: OS Independent
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
15
|
+
Classifier: Topic :: System :: Hardware
|
|
16
|
+
Classifier: Topic :: System :: Systems Administration
|
|
17
|
+
Classifier: Typing :: Typed
|
|
18
|
+
Requires-Python: >=3.12
|
|
19
|
+
Description-Content-Type: text/markdown
|
|
20
|
+
License-File: LICENSE
|
|
21
|
+
Requires-Dist: httpx>=0.28
|
|
22
|
+
Requires-Dist: pydantic>=2.9
|
|
23
|
+
Requires-Dist: pydantic-settings>=2.6
|
|
24
|
+
Requires-Dist: pyyaml>=6.0
|
|
25
|
+
Requires-Dist: rich>=13.9
|
|
26
|
+
Requires-Dist: typer>=0.15
|
|
27
|
+
Provides-Extra: dev
|
|
28
|
+
Requires-Dist: mypy>=1.13; extra == "dev"
|
|
29
|
+
Requires-Dist: pytest>=8.3; extra == "dev"
|
|
30
|
+
Requires-Dist: pytest-asyncio>=0.25; extra == "dev"
|
|
31
|
+
Requires-Dist: pytest-cov>=6.0; extra == "dev"
|
|
32
|
+
Requires-Dist: respx>=0.22; extra == "dev"
|
|
33
|
+
Requires-Dist: ruff>=0.8; extra == "dev"
|
|
34
|
+
Requires-Dist: types-pyyaml>=6.0; extra == "dev"
|
|
35
|
+
Dynamic: license-file
|
|
36
|
+
|
|
37
|
+
# racklint
|
|
38
|
+
|
|
39
|
+
**Check servers against the inventory before they go into service.**
|
|
40
|
+
|
|
41
|
+
racklint is a commissioning validator for bare-metal and GPU clusters. It reads what each server reports through
|
|
42
|
+
its BMC over [Redfish](https://www.dmtf.org/standards/redfish) and compares that with what
|
|
43
|
+
[NetBox](https://netbox.dev) says should be there. Every mismatch is flagged: a wrong GPU count, a wrong serial
|
|
44
|
+
number, firmware not on the approved baseline, a missing NIC.
|
|
45
|
+
|
|
46
|
+
You can use it in two ways: as the `racklint` command, or as a Python library.
|
|
47
|
+
|
|
48
|
+
## Why
|
|
49
|
+
|
|
50
|
+
When a rack is brought up, the question is not "what hardware is in it?" but "is the hardware in it what we
|
|
51
|
+
ordered and planned for?" Existing tools such as [netbox-sync](https://github.com/bb-Ricardo/netbox-sync) and
|
|
52
|
+
[netbox-agent](https://github.com/Solvik/netbox-agent) answer the first question: they read the hardware and
|
|
53
|
+
write it into NetBox, so NetBox ends up describing whatever was found.
|
|
54
|
+
|
|
55
|
+
racklint works the other way. NetBox holds the planned state and stays the source of truth. racklint checks the
|
|
56
|
+
hardware against it, changes nothing, and fails loudly on any difference. That makes it useful as a gate: a
|
|
57
|
+
node only moves to the next stage of bring-up once racklint passes it.
|
|
58
|
+
|
|
59
|
+
It is read-only. It sends only GET requests to NetBox and to the BMCs. The host OS does not need to be running.
|
|
60
|
+
|
|
61
|
+
## Install
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
pip install racklint
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
racklint needs Python 3.12 or later.
|
|
68
|
+
|
|
69
|
+
## Command line
|
|
70
|
+
|
|
71
|
+
### Settings
|
|
72
|
+
|
|
73
|
+
The command reads its settings from `RACKLINT_*` environment variables, or from a `.env` file in the working
|
|
74
|
+
directory.
|
|
75
|
+
|
|
76
|
+
| Setting | Default | Meaning |
|
|
77
|
+
|---|---|---|
|
|
78
|
+
| `RACKLINT_NETBOX_URL` | required | Base URL of NetBox, for example `https://netbox.example.net` |
|
|
79
|
+
| `RACKLINT_NETBOX_TOKEN` | required | NetBox API token |
|
|
80
|
+
| `RACKLINT_BMC_USERNAME` | required | User name for every BMC |
|
|
81
|
+
| `RACKLINT_BMC_PASSWORD` | required | Password for every BMC |
|
|
82
|
+
| `RACKLINT_BMC_SCHEME` | `https` | `https` or `http` |
|
|
83
|
+
| `RACKLINT_BMC_VERIFY_TLS` | `true` | Verify BMC TLS certificates |
|
|
84
|
+
| `RACKLINT_BMC_CA_BUNDLE` | unset | CA bundle file used to verify BMC certificates |
|
|
85
|
+
| `RACKLINT_CONCURRENCY` | `16` | BMCs read at once |
|
|
86
|
+
| `RACKLINT_BMC_CONCURRENCY` | `4` | Requests in flight to one BMC |
|
|
87
|
+
|
|
88
|
+
A token that starts with `nbt_` (NetBox 4.5 and later) is sent as a bearer token. Any other token is sent as a
|
|
89
|
+
legacy `Token`.
|
|
90
|
+
|
|
91
|
+
### Run a check
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
racklint check --rack R01 --baseline baseline.yaml
|
|
95
|
+
racklint check --device gpu-01 --device gpu-02 --baseline baseline.yaml
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
Pass exactly one of `--rack` and `--device`.
|
|
99
|
+
|
|
100
|
+
| Option | Meaning |
|
|
101
|
+
|---|---|
|
|
102
|
+
| `--rack NAME` | Check every device in this NetBox rack |
|
|
103
|
+
| `--site SLUG` | Site slug. Needed only if the rack name exists in several sites |
|
|
104
|
+
| `--device NAME` | Check this device. Repeat for more |
|
|
105
|
+
| `--baseline PATH` | Firmware baseline file. Required |
|
|
106
|
+
| `--format table\|json` | Output format. Default `table` |
|
|
107
|
+
| `--concurrency N` | BMCs to read at once. Overrides `RACKLINT_CONCURRENCY` |
|
|
108
|
+
| `--ca-bundle PATH` | CA bundle for BMC certificates. Overrides `RACKLINT_BMC_CA_BUNDLE` |
|
|
109
|
+
| `--insecure` | Do not verify BMC certificates. Overrides `RACKLINT_BMC_VERIFY_TLS` |
|
|
110
|
+
|
|
111
|
+
The baseline is a YAML file with the approved firmware versions for each NetBox device type. Each `match` is a
|
|
112
|
+
case-sensitive glob on the `Id` of a member of the BMC's firmware inventory. Versions must be quoted strings.
|
|
113
|
+
|
|
114
|
+
```yaml
|
|
115
|
+
device_types:
|
|
116
|
+
gpu-node-h100: # NetBox device-type slug
|
|
117
|
+
- match: "BMC"
|
|
118
|
+
approved: ["1.4.0"]
|
|
119
|
+
- match: "GPU_*"
|
|
120
|
+
approved: ["96.00.89.00.01"]
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
### Exit codes
|
|
124
|
+
|
|
125
|
+
| Code | Meaning |
|
|
126
|
+
|---|---|
|
|
127
|
+
| `0` | Every node passed |
|
|
128
|
+
| `1` | At least one node failed, or its BMC could not be read |
|
|
129
|
+
| `2` | Usage or configuration error: bad arguments, invalid settings or baseline, NetBox unreachable or returning an error, rack or device not found |
|
|
130
|
+
|
|
131
|
+
Use the exit code to gate a pipeline.
|
|
132
|
+
|
|
133
|
+
### Output
|
|
134
|
+
|
|
135
|
+
The table has one row per node and one line per reason, then a summary:
|
|
136
|
+
|
|
137
|
+
```console
|
|
138
|
+
┏━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
|
|
139
|
+
┃ Node ┃ BMC ┃ Status ┃ Reasons ┃
|
|
140
|
+
┡━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
|
|
141
|
+
│ n01 │ 192.0.2.11 │ PASS │ │
|
|
142
|
+
│ n02 │ 192.0.2.12 │ PASS │ │
|
|
143
|
+
│ n03 │ 192.0.2.13 │ FAIL │ gpu: 7 found, 8 planned │
|
|
144
|
+
│ n04 │ 192.0.2.14 │ PASS │ │
|
|
145
|
+
│ n05 │ 192.0.2.15 │ FAIL │ memory: 31 DIMMs (1984 GiB), 32 planned (2048 │
|
|
146
|
+
│ │ │ │ GiB) │
|
|
147
|
+
│ │ │ │ firmware: BMC 1.3.2 not in approved [1.4.0] │
|
|
148
|
+
│ n06 │ 192.0.2.16 │ PASS │ │
|
|
149
|
+
└──────┴────────────┴────────┴─────────────────────────────────────────────────┘
|
|
150
|
+
4 passed, 2 failed, 0 errored
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
A node has status `PASS`, `FAIL` or `ERROR`. `ERROR` means racklint could not read the BMC (a connection
|
|
154
|
+
failure after retries, a timeout, a rejected login or an invalid reply). No checks run for that node, and the
|
|
155
|
+
reason is the error.
|
|
156
|
+
|
|
157
|
+
With `--format json` the output is the report as JSON. This is one failing node:
|
|
158
|
+
|
|
159
|
+
```json
|
|
160
|
+
{
|
|
161
|
+
"schema_version": 1,
|
|
162
|
+
"results": [
|
|
163
|
+
{
|
|
164
|
+
"node": "n03",
|
|
165
|
+
"bmc_address": "192.0.2.13",
|
|
166
|
+
"status": "fail",
|
|
167
|
+
"findings": [
|
|
168
|
+
{
|
|
169
|
+
"check": "gpu",
|
|
170
|
+
"message": "7 found, 8 planned"
|
|
171
|
+
}
|
|
172
|
+
],
|
|
173
|
+
"error": null
|
|
174
|
+
}
|
|
175
|
+
]
|
|
176
|
+
}
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
`status` is `pass`, `fail` or `error`. `error` is set only when the status is `error`.
|
|
180
|
+
|
|
181
|
+
## Python library
|
|
182
|
+
|
|
183
|
+
The names exported from `racklint` (its `__all__`) are the public API. Anything else may change before 1.0. The
|
|
184
|
+
package ships type hints (`py.typed`). The names fall into these groups:
|
|
185
|
+
|
|
186
|
+
| Group | Names |
|
|
187
|
+
|---|---|
|
|
188
|
+
| State | `ExpectedNode`, `PlannedInterface`, `HardwareSpec`, `GpuSpec`, `MemorySpec`, `ActualNode`, `Gpu`, `Dimm`, `NetworkPort`, `FirmwareItem` |
|
|
189
|
+
| Verdict | `Report`, `NodeResult`, `Finding`, `Status` |
|
|
190
|
+
| Running checks | `check_nodes`, `iter_results`, `ActualReader`, `Check`, `default_checks`, `SerialCheck`, `GpuCheck`, `MemoryCheck`, `NicCheck`, `FirmwareCheck` |
|
|
191
|
+
| Sources | `NetBoxSource`, `netbox_client`, `RedfishReader`, `BmcAccess`, `FirmwareBaseline`, `FirmwareRule` |
|
|
192
|
+
| Errors | `RacklintError`, `BaselineError`, `NetBoxError`, `InventoryError`, `RedfishError` |
|
|
193
|
+
|
|
194
|
+
The library is async and does not read `RACKLINT_*` settings: you pass it explicit objects. This example reads
|
|
195
|
+
the plan for rack R01 from NetBox, reads each BMC and prints a verdict per node. It takes its connection details
|
|
196
|
+
from the environment.
|
|
197
|
+
|
|
198
|
+
```python
|
|
199
|
+
import asyncio
|
|
200
|
+
import os
|
|
201
|
+
from pathlib import Path
|
|
202
|
+
|
|
203
|
+
from racklint import (
|
|
204
|
+
BmcAccess,
|
|
205
|
+
FirmwareBaseline,
|
|
206
|
+
NetBoxSource,
|
|
207
|
+
RedfishReader,
|
|
208
|
+
check_nodes,
|
|
209
|
+
default_checks,
|
|
210
|
+
netbox_client,
|
|
211
|
+
)
|
|
212
|
+
|
|
213
|
+
|
|
214
|
+
async def main() -> None:
|
|
215
|
+
async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
|
|
216
|
+
nodes = await NetBoxSource(client).rack("R01")
|
|
217
|
+
|
|
218
|
+
access = BmcAccess(username=os.environ["BMC_USERNAME"], password=os.environ["BMC_PASSWORD"])
|
|
219
|
+
baseline = FirmwareBaseline.load(Path("baseline.yaml"))
|
|
220
|
+
report = await check_nodes(nodes, RedfishReader(access), default_checks(baseline), concurrency=16)
|
|
221
|
+
|
|
222
|
+
for result in report.results:
|
|
223
|
+
print(result.node, result.status, [finding.message for finding in result.findings])
|
|
224
|
+
print("all passed" if report.passed else "not all passed")
|
|
225
|
+
|
|
226
|
+
|
|
227
|
+
asyncio.run(main())
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
`check_nodes` returns a `Report` with the results in the order of `nodes`. To act on each result as soon as it
|
|
231
|
+
is ready, for live progress, loop over `iter_results` instead. It takes the same arguments and yields
|
|
232
|
+
`NodeResult`s in the order they finish:
|
|
233
|
+
|
|
234
|
+
```python
|
|
235
|
+
import asyncio
|
|
236
|
+
import os
|
|
237
|
+
from pathlib import Path
|
|
238
|
+
|
|
239
|
+
from racklint import (
|
|
240
|
+
BmcAccess,
|
|
241
|
+
FirmwareBaseline,
|
|
242
|
+
NetBoxSource,
|
|
243
|
+
RedfishReader,
|
|
244
|
+
default_checks,
|
|
245
|
+
iter_results,
|
|
246
|
+
netbox_client,
|
|
247
|
+
)
|
|
248
|
+
|
|
249
|
+
|
|
250
|
+
async def main() -> None:
|
|
251
|
+
async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
|
|
252
|
+
nodes = await NetBoxSource(client).devices(["gpu-01", "gpu-02"])
|
|
253
|
+
|
|
254
|
+
access = BmcAccess(username=os.environ["BMC_USERNAME"], password=os.environ["BMC_PASSWORD"])
|
|
255
|
+
checks = default_checks(FirmwareBaseline.load(Path("baseline.yaml")))
|
|
256
|
+
async for result in iter_results(nodes, RedfishReader(access), checks, concurrency=16):
|
|
257
|
+
print(result.node, result.status)
|
|
258
|
+
|
|
259
|
+
|
|
260
|
+
asyncio.run(main())
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
`NetBoxSource` raises `NetBoxError` and `FirmwareBaseline.load` raises `BaselineError`; both are subclasses of
|
|
264
|
+
`RacklintError`. A BMC that cannot be read does not raise: its node gets status `error`.
|
|
265
|
+
|
|
266
|
+
A `FirmwareBaseline` can also be built from a dict with `FirmwareBaseline.model_validate`, so an application
|
|
267
|
+
can keep the baseline in its own database.
|
|
268
|
+
|
|
269
|
+
The NetBox source is optional. A NetBox plugin runs inside NetBox, so it can build `ExpectedNode`s from the
|
|
270
|
+
Django ORM instead of the REST API, then call `asyncio.run(check_nodes(...))` from a background job, which is
|
|
271
|
+
synchronous code.
|
|
272
|
+
|
|
273
|
+
## What it checks
|
|
274
|
+
|
|
275
|
+
| Check | Fails when | Example reason |
|
|
276
|
+
|---|---|---|
|
|
277
|
+
| Serial | The device's serial matches the serial of no system on the BMC | `serial: expected SN-R01-03, BMC reports SN-R01-30` |
|
|
278
|
+
| GPU | The GPU count differs, or a GPU's model differs from the planned model | `gpu: 7 found, 8 planned` |
|
|
279
|
+
| Memory | The enabled DIMM count or the total capacity differs | `memory: 31 DIMMs (1984 GiB), 32 planned (2048 GiB)` |
|
|
280
|
+
| NIC | A planned MAC address is not reported, or fewer ports are reported than interfaces are planned | `nic: eth2 (a0:88:c2:00:03:03) not found` |
|
|
281
|
+
| Firmware | The device type has no baseline, a baseline entry matches no firmware item, or a matched item's version is not approved | `firmware: BMC 1.3.2 not in approved [1.4.0]` |
|
|
282
|
+
|
|
283
|
+
A node passes only if every check passes. Firmware items that match no baseline entry are ignored, and so are
|
|
284
|
+
reported ports beyond the plan.
|
|
285
|
+
|
|
286
|
+
If NetBox lacks a value a check needs, the node fails with `not planned in NetBox: <item>`. A device with no
|
|
287
|
+
BMC address fails with that reason alone.
|
|
288
|
+
|
|
289
|
+
### What NetBox must hold
|
|
290
|
+
|
|
291
|
+
| Item | Where racklint reads it |
|
|
292
|
+
|---|---|
|
|
293
|
+
| Serial | The device's `serial` field |
|
|
294
|
+
| BMC address | The device's out-of-band IP (`oob_ip`). The prefix length is dropped |
|
|
295
|
+
| Device type | The device type's slug. It selects the firmware baseline |
|
|
296
|
+
| NICs | The device's physical interfaces that are not management-only. A MAC address is compared if the interface has a primary MAC; without one, racklint only counts the port |
|
|
297
|
+
| GPU and memory | The `racklint` key of the device's config context |
|
|
298
|
+
|
|
299
|
+
Assign the config context to the device type, so every device of that type gets it:
|
|
300
|
+
|
|
301
|
+
```yaml
|
|
302
|
+
racklint:
|
|
303
|
+
gpu: {count: 8, model: "NVIDIA H100 80GB HBM3"}
|
|
304
|
+
memory: {dimm_count: 32, total_gib: 2048}
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
An invalid `racklint` config context is a configuration error and stops the run with exit code `2`.
|
|
308
|
+
|
|
309
|
+
On the BMC side, racklint starts at `/redfish/v1` and follows the links the service returns. It hardcodes no
|
|
310
|
+
vendor paths. It reads systems, processors, memory, network adapters and the firmware inventory. It uses HTTP
|
|
311
|
+
Basic authentication on every request and opens no Redfish sessions. It makes up to three attempts at a
|
|
312
|
+
request that fails with a connection error, a timeout, HTTP 5xx or 429, and uses a connect timeout of 5
|
|
313
|
+
seconds and a read timeout of 15 seconds.
|
|
314
|
+
|
|
315
|
+
## Try it without hardware
|
|
316
|
+
|
|
317
|
+
[`examples/no-hardware/`](examples/no-hardware) is a Docker Compose stack with a real NetBox and six mock BMCs,
|
|
318
|
+
two of them built wrong on purpose. One command checks the rack, and the summary line is
|
|
319
|
+
`4 passed, 2 failed, 0 errored`. See its README for the steps and what is planted.
|
|
320
|
+
|
|
321
|
+
## Limitations
|
|
322
|
+
|
|
323
|
+
- GPUs are found only as Redfish `Processors` with `ProcessorType == "GPU"`. A BMC that exposes GPUs another way
|
|
324
|
+
shows zero GPUs.
|
|
325
|
+
- One set of BMC credentials applies to every node in a run.
|
|
326
|
+
- Read-only by design. racklint reports differences and never fixes them in NetBox or on a BMC.
|
|
327
|
+
|
|
328
|
+
## Related work
|
|
329
|
+
|
|
330
|
+
| Project | What it does |
|
|
331
|
+
|---|---|
|
|
332
|
+
| [bb-Ricardo/netbox-sync](https://github.com/bb-Ricardo/netbox-sync) | Syncs VMware and Redfish sources into NetBox |
|
|
333
|
+
| [Solvik/netbox-agent](https://github.com/Solvik/netbox-agent) | Agent on each server that pushes hardware into NetBox |
|
|
334
|
+
| [bb-Ricardo/check_redfish](https://github.com/bb-Ricardo/check_redfish) | Monitoring plugin for component health over Redfish |
|
|
335
|
+
| OpenStack Ironic, Metal3, MAAS, Warewulf | Full provisioning platforms with BMC inspection |
|
|
336
|
+
|
|
337
|
+
racklint is smaller than any of these and does one thing: it compares the hardware with NetBox's plan and
|
|
338
|
+
reports the differences.
|
|
339
|
+
|
|
340
|
+
## License
|
|
341
|
+
|
|
342
|
+
[MIT](LICENSE)
|
racklint-0.1.0/README.md
ADDED
|
@@ -0,0 +1,306 @@
|
|
|
1
|
+
# racklint
|
|
2
|
+
|
|
3
|
+
**Check servers against the inventory before they go into service.**
|
|
4
|
+
|
|
5
|
+
racklint is a commissioning validator for bare-metal and GPU clusters. It reads what each server reports through
|
|
6
|
+
its BMC over [Redfish](https://www.dmtf.org/standards/redfish) and compares that with what
|
|
7
|
+
[NetBox](https://netbox.dev) says should be there. Every mismatch is flagged: a wrong GPU count, a wrong serial
|
|
8
|
+
number, firmware not on the approved baseline, a missing NIC.
|
|
9
|
+
|
|
10
|
+
You can use it in two ways: as the `racklint` command, or as a Python library.
|
|
11
|
+
|
|
12
|
+
## Why
|
|
13
|
+
|
|
14
|
+
When a rack is brought up, the question is not "what hardware is in it?" but "is the hardware in it what we
|
|
15
|
+
ordered and planned for?" Existing tools such as [netbox-sync](https://github.com/bb-Ricardo/netbox-sync) and
|
|
16
|
+
[netbox-agent](https://github.com/Solvik/netbox-agent) answer the first question: they read the hardware and
|
|
17
|
+
write it into NetBox, so NetBox ends up describing whatever was found.
|
|
18
|
+
|
|
19
|
+
racklint works the other way. NetBox holds the planned state and stays the source of truth. racklint checks the
|
|
20
|
+
hardware against it, changes nothing, and fails loudly on any difference. That makes it useful as a gate: a
|
|
21
|
+
node only moves to the next stage of bring-up once racklint passes it.
|
|
22
|
+
|
|
23
|
+
It is read-only. It sends only GET requests to NetBox and to the BMCs. The host OS does not need to be running.
|
|
24
|
+
|
|
25
|
+
## Install
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
pip install racklint
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
racklint needs Python 3.12 or later.
|
|
32
|
+
|
|
33
|
+
## Command line
|
|
34
|
+
|
|
35
|
+
### Settings
|
|
36
|
+
|
|
37
|
+
The command reads its settings from `RACKLINT_*` environment variables, or from a `.env` file in the working
|
|
38
|
+
directory.
|
|
39
|
+
|
|
40
|
+
| Setting | Default | Meaning |
|
|
41
|
+
|---|---|---|
|
|
42
|
+
| `RACKLINT_NETBOX_URL` | required | Base URL of NetBox, for example `https://netbox.example.net` |
|
|
43
|
+
| `RACKLINT_NETBOX_TOKEN` | required | NetBox API token |
|
|
44
|
+
| `RACKLINT_BMC_USERNAME` | required | User name for every BMC |
|
|
45
|
+
| `RACKLINT_BMC_PASSWORD` | required | Password for every BMC |
|
|
46
|
+
| `RACKLINT_BMC_SCHEME` | `https` | `https` or `http` |
|
|
47
|
+
| `RACKLINT_BMC_VERIFY_TLS` | `true` | Verify BMC TLS certificates |
|
|
48
|
+
| `RACKLINT_BMC_CA_BUNDLE` | unset | CA bundle file used to verify BMC certificates |
|
|
49
|
+
| `RACKLINT_CONCURRENCY` | `16` | BMCs read at once |
|
|
50
|
+
| `RACKLINT_BMC_CONCURRENCY` | `4` | Requests in flight to one BMC |
|
|
51
|
+
|
|
52
|
+
A token that starts with `nbt_` (NetBox 4.5 and later) is sent as a bearer token. Any other token is sent as a
|
|
53
|
+
legacy `Token`.
|
|
54
|
+
|
|
55
|
+
### Run a check
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
racklint check --rack R01 --baseline baseline.yaml
|
|
59
|
+
racklint check --device gpu-01 --device gpu-02 --baseline baseline.yaml
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Pass exactly one of `--rack` and `--device`.
|
|
63
|
+
|
|
64
|
+
| Option | Meaning |
|
|
65
|
+
|---|---|
|
|
66
|
+
| `--rack NAME` | Check every device in this NetBox rack |
|
|
67
|
+
| `--site SLUG` | Site slug. Needed only if the rack name exists in several sites |
|
|
68
|
+
| `--device NAME` | Check this device. Repeat for more |
|
|
69
|
+
| `--baseline PATH` | Firmware baseline file. Required |
|
|
70
|
+
| `--format table\|json` | Output format. Default `table` |
|
|
71
|
+
| `--concurrency N` | BMCs to read at once. Overrides `RACKLINT_CONCURRENCY` |
|
|
72
|
+
| `--ca-bundle PATH` | CA bundle for BMC certificates. Overrides `RACKLINT_BMC_CA_BUNDLE` |
|
|
73
|
+
| `--insecure` | Do not verify BMC certificates. Overrides `RACKLINT_BMC_VERIFY_TLS` |
|
|
74
|
+
|
|
75
|
+
The baseline is a YAML file with the approved firmware versions for each NetBox device type. Each `match` is a
|
|
76
|
+
case-sensitive glob on the `Id` of a member of the BMC's firmware inventory. Versions must be quoted strings.
|
|
77
|
+
|
|
78
|
+
```yaml
|
|
79
|
+
device_types:
|
|
80
|
+
gpu-node-h100: # NetBox device-type slug
|
|
81
|
+
- match: "BMC"
|
|
82
|
+
approved: ["1.4.0"]
|
|
83
|
+
- match: "GPU_*"
|
|
84
|
+
approved: ["96.00.89.00.01"]
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
### Exit codes
|
|
88
|
+
|
|
89
|
+
| Code | Meaning |
|
|
90
|
+
|---|---|
|
|
91
|
+
| `0` | Every node passed |
|
|
92
|
+
| `1` | At least one node failed, or its BMC could not be read |
|
|
93
|
+
| `2` | Usage or configuration error: bad arguments, invalid settings or baseline, NetBox unreachable or returning an error, rack or device not found |
|
|
94
|
+
|
|
95
|
+
Use the exit code to gate a pipeline.
|
|
96
|
+
|
|
97
|
+
### Output
|
|
98
|
+
|
|
99
|
+
The table has one row per node and one line per reason, then a summary:
|
|
100
|
+
|
|
101
|
+
```console
|
|
102
|
+
┏━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
|
|
103
|
+
┃ Node ┃ BMC ┃ Status ┃ Reasons ┃
|
|
104
|
+
┡━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
|
|
105
|
+
│ n01 │ 192.0.2.11 │ PASS │ │
|
|
106
|
+
│ n02 │ 192.0.2.12 │ PASS │ │
|
|
107
|
+
│ n03 │ 192.0.2.13 │ FAIL │ gpu: 7 found, 8 planned │
|
|
108
|
+
│ n04 │ 192.0.2.14 │ PASS │ │
|
|
109
|
+
│ n05 │ 192.0.2.15 │ FAIL │ memory: 31 DIMMs (1984 GiB), 32 planned (2048 │
|
|
110
|
+
│ │ │ │ GiB) │
|
|
111
|
+
│ │ │ │ firmware: BMC 1.3.2 not in approved [1.4.0] │
|
|
112
|
+
│ n06 │ 192.0.2.16 │ PASS │ │
|
|
113
|
+
└──────┴────────────┴────────┴─────────────────────────────────────────────────┘
|
|
114
|
+
4 passed, 2 failed, 0 errored
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
A node has status `PASS`, `FAIL` or `ERROR`. `ERROR` means racklint could not read the BMC (a connection
|
|
118
|
+
failure after retries, a timeout, a rejected login or an invalid reply). No checks run for that node, and the
|
|
119
|
+
reason is the error.
|
|
120
|
+
|
|
121
|
+
With `--format json` the output is the report as JSON. This is one failing node:
|
|
122
|
+
|
|
123
|
+
```json
|
|
124
|
+
{
|
|
125
|
+
"schema_version": 1,
|
|
126
|
+
"results": [
|
|
127
|
+
{
|
|
128
|
+
"node": "n03",
|
|
129
|
+
"bmc_address": "192.0.2.13",
|
|
130
|
+
"status": "fail",
|
|
131
|
+
"findings": [
|
|
132
|
+
{
|
|
133
|
+
"check": "gpu",
|
|
134
|
+
"message": "7 found, 8 planned"
|
|
135
|
+
}
|
|
136
|
+
],
|
|
137
|
+
"error": null
|
|
138
|
+
}
|
|
139
|
+
]
|
|
140
|
+
}
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
`status` is `pass`, `fail` or `error`. `error` is set only when the status is `error`.
|
|
144
|
+
|
|
145
|
+
## Python library
|
|
146
|
+
|
|
147
|
+
The names exported from `racklint` (its `__all__`) are the public API. Anything else may change before 1.0. The
|
|
148
|
+
package ships type hints (`py.typed`). The names fall into these groups:
|
|
149
|
+
|
|
150
|
+
| Group | Names |
|
|
151
|
+
|---|---|
|
|
152
|
+
| State | `ExpectedNode`, `PlannedInterface`, `HardwareSpec`, `GpuSpec`, `MemorySpec`, `ActualNode`, `Gpu`, `Dimm`, `NetworkPort`, `FirmwareItem` |
|
|
153
|
+
| Verdict | `Report`, `NodeResult`, `Finding`, `Status` |
|
|
154
|
+
| Running checks | `check_nodes`, `iter_results`, `ActualReader`, `Check`, `default_checks`, `SerialCheck`, `GpuCheck`, `MemoryCheck`, `NicCheck`, `FirmwareCheck` |
|
|
155
|
+
| Sources | `NetBoxSource`, `netbox_client`, `RedfishReader`, `BmcAccess`, `FirmwareBaseline`, `FirmwareRule` |
|
|
156
|
+
| Errors | `RacklintError`, `BaselineError`, `NetBoxError`, `InventoryError`, `RedfishError` |
|
|
157
|
+
|
|
158
|
+
The library is async and does not read `RACKLINT_*` settings: you pass it explicit objects. This example reads
|
|
159
|
+
the plan for rack R01 from NetBox, reads each BMC and prints a verdict per node. It takes its connection details
|
|
160
|
+
from the environment.
|
|
161
|
+
|
|
162
|
+
```python
|
|
163
|
+
import asyncio
|
|
164
|
+
import os
|
|
165
|
+
from pathlib import Path
|
|
166
|
+
|
|
167
|
+
from racklint import (
|
|
168
|
+
BmcAccess,
|
|
169
|
+
FirmwareBaseline,
|
|
170
|
+
NetBoxSource,
|
|
171
|
+
RedfishReader,
|
|
172
|
+
check_nodes,
|
|
173
|
+
default_checks,
|
|
174
|
+
netbox_client,
|
|
175
|
+
)
|
|
176
|
+
|
|
177
|
+
|
|
178
|
+
async def main() -> None:
|
|
179
|
+
async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
|
|
180
|
+
nodes = await NetBoxSource(client).rack("R01")
|
|
181
|
+
|
|
182
|
+
access = BmcAccess(username=os.environ["BMC_USERNAME"], password=os.environ["BMC_PASSWORD"])
|
|
183
|
+
baseline = FirmwareBaseline.load(Path("baseline.yaml"))
|
|
184
|
+
report = await check_nodes(nodes, RedfishReader(access), default_checks(baseline), concurrency=16)
|
|
185
|
+
|
|
186
|
+
for result in report.results:
|
|
187
|
+
print(result.node, result.status, [finding.message for finding in result.findings])
|
|
188
|
+
print("all passed" if report.passed else "not all passed")
|
|
189
|
+
|
|
190
|
+
|
|
191
|
+
asyncio.run(main())
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
`check_nodes` returns a `Report` with the results in the order of `nodes`. To act on each result as soon as it
|
|
195
|
+
is ready, for live progress, loop over `iter_results` instead. It takes the same arguments and yields
|
|
196
|
+
`NodeResult`s in the order they finish:
|
|
197
|
+
|
|
198
|
+
```python
|
|
199
|
+
import asyncio
|
|
200
|
+
import os
|
|
201
|
+
from pathlib import Path
|
|
202
|
+
|
|
203
|
+
from racklint import (
|
|
204
|
+
BmcAccess,
|
|
205
|
+
FirmwareBaseline,
|
|
206
|
+
NetBoxSource,
|
|
207
|
+
RedfishReader,
|
|
208
|
+
default_checks,
|
|
209
|
+
iter_results,
|
|
210
|
+
netbox_client,
|
|
211
|
+
)
|
|
212
|
+
|
|
213
|
+
|
|
214
|
+
async def main() -> None:
|
|
215
|
+
async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
|
|
216
|
+
nodes = await NetBoxSource(client).devices(["gpu-01", "gpu-02"])
|
|
217
|
+
|
|
218
|
+
access = BmcAccess(username=os.environ["BMC_USERNAME"], password=os.environ["BMC_PASSWORD"])
|
|
219
|
+
checks = default_checks(FirmwareBaseline.load(Path("baseline.yaml")))
|
|
220
|
+
async for result in iter_results(nodes, RedfishReader(access), checks, concurrency=16):
|
|
221
|
+
print(result.node, result.status)
|
|
222
|
+
|
|
223
|
+
|
|
224
|
+
asyncio.run(main())
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
`NetBoxSource` raises `NetBoxError` and `FirmwareBaseline.load` raises `BaselineError`; both are subclasses of
|
|
228
|
+
`RacklintError`. A BMC that cannot be read does not raise: its node gets status `error`.
|
|
229
|
+
|
|
230
|
+
A `FirmwareBaseline` can also be built from a dict with `FirmwareBaseline.model_validate`, so an application
|
|
231
|
+
can keep the baseline in its own database.
|
|
232
|
+
|
|
233
|
+
The NetBox source is optional. A NetBox plugin runs inside NetBox, so it can build `ExpectedNode`s from the
|
|
234
|
+
Django ORM instead of the REST API, then call `asyncio.run(check_nodes(...))` from a background job, which is
|
|
235
|
+
synchronous code.
|
|
236
|
+
|
|
237
|
+
## What it checks
|
|
238
|
+
|
|
239
|
+
| Check | Fails when | Example reason |
|
|
240
|
+
|---|---|---|
|
|
241
|
+
| Serial | The device's serial matches the serial of no system on the BMC | `serial: expected SN-R01-03, BMC reports SN-R01-30` |
|
|
242
|
+
| GPU | The GPU count differs, or a GPU's model differs from the planned model | `gpu: 7 found, 8 planned` |
|
|
243
|
+
| Memory | The enabled DIMM count or the total capacity differs | `memory: 31 DIMMs (1984 GiB), 32 planned (2048 GiB)` |
|
|
244
|
+
| NIC | A planned MAC address is not reported, or fewer ports are reported than interfaces are planned | `nic: eth2 (a0:88:c2:00:03:03) not found` |
|
|
245
|
+
| Firmware | The device type has no baseline, a baseline entry matches no firmware item, or a matched item's version is not approved | `firmware: BMC 1.3.2 not in approved [1.4.0]` |
|
|
246
|
+
|
|
247
|
+
A node passes only if every check passes. Firmware items that match no baseline entry are ignored, and so are
|
|
248
|
+
reported ports beyond the plan.
|
|
249
|
+
|
|
250
|
+
If NetBox lacks a value a check needs, the node fails with `not planned in NetBox: <item>`. A device with no
|
|
251
|
+
BMC address fails with that reason alone.
|
|
252
|
+
|
|
253
|
+
### What NetBox must hold
|
|
254
|
+
|
|
255
|
+
| Item | Where racklint reads it |
|
|
256
|
+
|---|---|
|
|
257
|
+
| Serial | The device's `serial` field |
|
|
258
|
+
| BMC address | The device's out-of-band IP (`oob_ip`). The prefix length is dropped |
|
|
259
|
+
| Device type | The device type's slug. It selects the firmware baseline |
|
|
260
|
+
| NICs | The device's physical interfaces that are not management-only. A MAC address is compared if the interface has a primary MAC; without one, racklint only counts the port |
|
|
261
|
+
| GPU and memory | The `racklint` key of the device's config context |
|
|
262
|
+
|
|
263
|
+
Assign the config context to the device type, so every device of that type gets it:
|
|
264
|
+
|
|
265
|
+
```yaml
|
|
266
|
+
racklint:
|
|
267
|
+
gpu: {count: 8, model: "NVIDIA H100 80GB HBM3"}
|
|
268
|
+
memory: {dimm_count: 32, total_gib: 2048}
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
An invalid `racklint` config context is a configuration error and stops the run with exit code `2`.
|
|
272
|
+
|
|
273
|
+
On the BMC side, racklint starts at `/redfish/v1` and follows the links the service returns. It hardcodes no
|
|
274
|
+
vendor paths. It reads systems, processors, memory, network adapters and the firmware inventory. It uses HTTP
|
|
275
|
+
Basic authentication on every request and opens no Redfish sessions. It makes up to three attempts at a
|
|
276
|
+
request that fails with a connection error, a timeout, HTTP 5xx or 429, and uses a connect timeout of 5
|
|
277
|
+
seconds and a read timeout of 15 seconds.
|
|
278
|
+
|
|
279
|
+
## Try it without hardware
|
|
280
|
+
|
|
281
|
+
[`examples/no-hardware/`](examples/no-hardware) is a Docker Compose stack with a real NetBox and six mock BMCs,
|
|
282
|
+
two of them built wrong on purpose. One command checks the rack, and the summary line is
|
|
283
|
+
`4 passed, 2 failed, 0 errored`. See its README for the steps and what is planted.
|
|
284
|
+
|
|
285
|
+
## Limitations
|
|
286
|
+
|
|
287
|
+
- GPUs are found only as Redfish `Processors` with `ProcessorType == "GPU"`. A BMC that exposes GPUs another way
|
|
288
|
+
shows zero GPUs.
|
|
289
|
+
- One set of BMC credentials applies to every node in a run.
|
|
290
|
+
- Read-only by design. racklint reports differences and never fixes them in NetBox or on a BMC.
|
|
291
|
+
|
|
292
|
+
## Related work
|
|
293
|
+
|
|
294
|
+
| Project | What it does |
|
|
295
|
+
|---|---|
|
|
296
|
+
| [bb-Ricardo/netbox-sync](https://github.com/bb-Ricardo/netbox-sync) | Syncs VMware and Redfish sources into NetBox |
|
|
297
|
+
| [Solvik/netbox-agent](https://github.com/Solvik/netbox-agent) | Agent on each server that pushes hardware into NetBox |
|
|
298
|
+
| [bb-Ricardo/check_redfish](https://github.com/bb-Ricardo/check_redfish) | Monitoring plugin for component health over Redfish |
|
|
299
|
+
| OpenStack Ironic, Metal3, MAAS, Warewulf | Full provisioning platforms with BMC inspection |
|
|
300
|
+
|
|
301
|
+
racklint is smaller than any of these and does one thing: it compares the hardware with NetBox's plan and
|
|
302
|
+
reports the differences.
|
|
303
|
+
|
|
304
|
+
## License
|
|
305
|
+
|
|
306
|
+
[MIT](LICENSE)
|