onyx-site 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- onyx_site-0.1.0/LICENSE +202 -0
- onyx_site-0.1.0/PKG-INFO +245 -0
- onyx_site-0.1.0/README.md +231 -0
- onyx_site-0.1.0/pyproject.toml +54 -0
- onyx_site-0.1.0/pyproject.toml.orig +44 -0
- onyx_site-0.1.0/src/onyx_site/__init__.py +34 -0
- onyx_site-0.1.0/src/onyx_site/auth.py +117 -0
- onyx_site-0.1.0/src/onyx_site/cli.py +400 -0
- onyx_site-0.1.0/src/onyx_site/config.py +294 -0
- onyx_site-0.1.0/src/onyx_site/conformance.py +253 -0
- onyx_site-0.1.0/src/onyx_site/control.py +116 -0
- onyx_site-0.1.0/src/onyx_site/driver.py +247 -0
- onyx_site-0.1.0/src/onyx_site/driverhost.py +419 -0
- onyx_site-0.1.0/src/onyx_site/drivers/__init__.py +9 -0
- onyx_site-0.1.0/src/onyx_site/drivers/local_build.py +140 -0
- onyx_site-0.1.0/src/onyx_site/drivers/local_program.py +218 -0
- onyx_site-0.1.0/src/onyx_site/drivers/placeholder.py +65 -0
- onyx_site-0.1.0/src/onyx_site/drivers/sim_motor.py +173 -0
- onyx_site-0.1.0/src/onyx_site/enrol.py +112 -0
- onyx_site-0.1.0/src/onyx_site/fetch.py +110 -0
- onyx_site-0.1.0/src/onyx_site/fileserver.py +278 -0
- onyx_site-0.1.0/src/onyx_site/journal.py +311 -0
- onyx_site-0.1.0/src/onyx_site/policy.py +76 -0
- onyx_site-0.1.0/src/onyx_site/py.typed +0 -0
- onyx_site-0.1.0/src/onyx_site/service.py +1221 -0
- onyx_site-0.1.0/src/onyx_site/store.py +117 -0
- onyx_site-0.1.0/src/onyx_site/sync.py +262 -0
- onyx_site-0.1.0/src/onyx_site/templates/site.toml +82 -0
- onyx_site-0.1.0/src/onyx_site/worker.py +165 -0
onyx_site-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
|
|
2
|
+
Apache License
|
|
3
|
+
Version 2.0, January 2004
|
|
4
|
+
http://www.apache.org/licenses/
|
|
5
|
+
|
|
6
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
7
|
+
|
|
8
|
+
1. Definitions.
|
|
9
|
+
|
|
10
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
11
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
12
|
+
|
|
13
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
14
|
+
the copyright owner that is granting the License.
|
|
15
|
+
|
|
16
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
17
|
+
other entities that control, are controlled by, or are under common
|
|
18
|
+
control with that entity. For the purposes of this definition,
|
|
19
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
20
|
+
direction or management of such entity, whether by contract or
|
|
21
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
22
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
23
|
+
|
|
24
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
25
|
+
exercising permissions granted by this License.
|
|
26
|
+
|
|
27
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
28
|
+
including but not limited to software source code, documentation
|
|
29
|
+
source, and configuration files.
|
|
30
|
+
|
|
31
|
+
"Object" form shall mean any form resulting from mechanical
|
|
32
|
+
transformation or translation of a Source form, including but
|
|
33
|
+
not limited to compiled object code, generated documentation,
|
|
34
|
+
and conversions to other media types.
|
|
35
|
+
|
|
36
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
37
|
+
Object form, made available under the License, as indicated by a
|
|
38
|
+
copyright notice that is included in or attached to the work
|
|
39
|
+
(an example is provided in the Appendix below).
|
|
40
|
+
|
|
41
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
42
|
+
form, that is based on (or derived from) the Work and for which the
|
|
43
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
44
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
45
|
+
of this License, Derivative Works shall not include works that remain
|
|
46
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
47
|
+
the Work and Derivative Works thereof.
|
|
48
|
+
|
|
49
|
+
"Contribution" shall mean any work of authorship, including
|
|
50
|
+
the original version of the Work and any modifications or additions
|
|
51
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
52
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
53
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
54
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
55
|
+
means any form of electronic, verbal, or written communication sent
|
|
56
|
+
to the Licensor or its representatives, including but not limited to
|
|
57
|
+
communication on electronic mailing lists, source code control systems,
|
|
58
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
59
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
60
|
+
excluding communication that is conspicuously marked or otherwise
|
|
61
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
62
|
+
|
|
63
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
64
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
65
|
+
subsequently incorporated within the Work.
|
|
66
|
+
|
|
67
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
68
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
69
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
70
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
71
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
72
|
+
Work and such Derivative Works in Source or Object form.
|
|
73
|
+
|
|
74
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
75
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
76
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
77
|
+
(except as stated in this section) patent license to make, have made,
|
|
78
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
79
|
+
where such license applies only to those patent claims licensable
|
|
80
|
+
by such Contributor that are necessarily infringed by their
|
|
81
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
82
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
83
|
+
institute patent litigation against any entity (including a
|
|
84
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
85
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
86
|
+
or contributory patent infringement, then any patent licenses
|
|
87
|
+
granted to You under this License for that Work shall terminate
|
|
88
|
+
as of the date such litigation is filed.
|
|
89
|
+
|
|
90
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
91
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
92
|
+
modifications, and in Source or Object form, provided that You
|
|
93
|
+
meet the following conditions:
|
|
94
|
+
|
|
95
|
+
(a) You must give any other recipients of the Work or
|
|
96
|
+
Derivative Works a copy of this License; and
|
|
97
|
+
|
|
98
|
+
(b) You must cause any modified files to carry prominent notices
|
|
99
|
+
stating that You changed the files; and
|
|
100
|
+
|
|
101
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
102
|
+
that You distribute, all copyright, patent, trademark, and
|
|
103
|
+
attribution notices from the Source form of the Work,
|
|
104
|
+
excluding those notices that do not pertain to any part of
|
|
105
|
+
the Derivative Works; and
|
|
106
|
+
|
|
107
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
108
|
+
distribution, then any Derivative Works that You distribute must
|
|
109
|
+
include a readable copy of the attribution notices contained
|
|
110
|
+
within such NOTICE file, excluding those notices that do not
|
|
111
|
+
pertain to any part of the Derivative Works, in at least one
|
|
112
|
+
of the following places: within a NOTICE text file distributed
|
|
113
|
+
as part of the Derivative Works; within the Source form or
|
|
114
|
+
documentation, if provided along with the Derivative Works; or,
|
|
115
|
+
within a display generated by the Derivative Works, if and
|
|
116
|
+
wherever such third-party notices normally appear. The contents
|
|
117
|
+
of the NOTICE file are for informational purposes only and
|
|
118
|
+
do not modify the License. You may add Your own attribution
|
|
119
|
+
notices within Derivative Works that You distribute, alongside
|
|
120
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
121
|
+
that such additional attribution notices cannot be construed
|
|
122
|
+
as modifying the License.
|
|
123
|
+
|
|
124
|
+
You may add Your own copyright statement to Your modifications and
|
|
125
|
+
may provide additional or different license terms and conditions
|
|
126
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
127
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
128
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
129
|
+
the conditions stated in this License.
|
|
130
|
+
|
|
131
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
132
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
133
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
134
|
+
this License, without any additional terms or conditions.
|
|
135
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
136
|
+
the terms of any separate license agreement you may have executed
|
|
137
|
+
with Licensor regarding such Contributions.
|
|
138
|
+
|
|
139
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
140
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
141
|
+
except as required for reasonable and customary use in describing the
|
|
142
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
143
|
+
|
|
144
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
145
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
146
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
147
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
148
|
+
implied, including, without limitation, any warranties or conditions
|
|
149
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
150
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
151
|
+
appropriateness of using or redistributing the Work and assume any
|
|
152
|
+
risks associated with Your exercise of permissions under this License.
|
|
153
|
+
|
|
154
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
155
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
156
|
+
unless required by applicable law (such as deliberate and grossly
|
|
157
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
158
|
+
liable to You for damages, including any direct, indirect, special,
|
|
159
|
+
incidental, or consequential damages of any character arising as a
|
|
160
|
+
result of this License or out of the use or inability to use the
|
|
161
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
162
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
163
|
+
other commercial damages or losses), even if such Contributor
|
|
164
|
+
has been advised of the possibility of such damages.
|
|
165
|
+
|
|
166
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
167
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
168
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
169
|
+
or other liability obligations and/or rights consistent with this
|
|
170
|
+
License. However, in accepting such obligations, You may act only
|
|
171
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
172
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
173
|
+
defend, and hold each Contributor harmless for any liability
|
|
174
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
175
|
+
of your accepting any such warranty or additional liability.
|
|
176
|
+
|
|
177
|
+
END OF TERMS AND CONDITIONS
|
|
178
|
+
|
|
179
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
180
|
+
|
|
181
|
+
To apply the Apache License to your work, attach the following
|
|
182
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
183
|
+
replaced with your own identifying information. (Don't include
|
|
184
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
185
|
+
comment syntax for the file format. We also recommend that a
|
|
186
|
+
file or class name and description of purpose be included on the
|
|
187
|
+
same "printed page" as the copyright notice for easier
|
|
188
|
+
identification within third-party archives.
|
|
189
|
+
|
|
190
|
+
Copyright [yyyy] [name of copyright owner]
|
|
191
|
+
|
|
192
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
193
|
+
you may not use this file except in compliance with the License.
|
|
194
|
+
You may obtain a copy of the License at
|
|
195
|
+
|
|
196
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
197
|
+
|
|
198
|
+
Unless required by applicable law or agreed to in writing, software
|
|
199
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
200
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
201
|
+
See the License for the specific language governing permissions and
|
|
202
|
+
limitations under the License.
|
onyx_site-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,245 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: onyx-site
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Onyx site-runner: runs onyx-app's jobs on a customer's units (builders, sims, rigs)
|
|
5
|
+
Author: Max Palay
|
|
6
|
+
Author-email: Max Palay <maximiliano@onyx-robotics.com>
|
|
7
|
+
License-Expression: Apache-2.0
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Requires-Dist: httpx>=0.28
|
|
10
|
+
Requires-Dist: cryptography>=44
|
|
11
|
+
Requires-Dist: pydantic>=2.10
|
|
12
|
+
Requires-Python: >=3.12
|
|
13
|
+
Description-Content-Type: text/markdown
|
|
14
|
+
|
|
15
|
+
# onyx-site
|
|
16
|
+
|
|
17
|
+
The Onyx **site-runner**: a long-running service on the customer's machine that runs onyx-app's
|
|
18
|
+
jobs on the site's units (builders, simulators, rigs). It is a runner in onyx-app's jobs protocol
|
|
19
|
+
(`typescript/onyx-app/docs/jobs.md`): it calls `POST /api/v1/runners/sync` once a tick, over
|
|
20
|
+
plain HTTPS, with the site's API key. Each sync offers the site's units and reports on its jobs;
|
|
21
|
+
each answer assigns, cancels or forgets jobs. Every assignment is checked against the site's own
|
|
22
|
+
policy, then run by a customer **driver** (each in its own process). Result files stay on the
|
|
23
|
+
site and are served through signed download links. A local SQLite journal means a restart or a
|
|
24
|
+
lost response loses nothing and does nothing twice.
|
|
25
|
+
|
|
26
|
+
**Setting up a site, step by step: [SETUP.md](https://github.com/onyx-robotics/onyx-agent/blob/main/docs/sites/SETUP.md).**
|
|
27
|
+
|
|
28
|
+
It runs as a packaged service, not in a container, because drivers need USB, CAN and serial.
|
|
29
|
+
Its drivers run as the site's user, without a sandbox, and run code from the experiment's commit,
|
|
30
|
+
which agents may have written: give that user only what the site needs, and make a driver that
|
|
31
|
+
moves hardware safe whatever it's asked to do.
|
|
32
|
+
|
|
33
|
+
## Install
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
uv tool install onyx-site # or: pip install --user onyx-site
|
|
37
|
+
# from a checkout of the Onyx repository instead: uv tool install ./python/site-runner
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Production layout: a dedicated `onyx-site` user, the site file at `/etc/onyx-site/site.toml`,
|
|
41
|
+
state in `/var/lib/onyx-site` (mode 0700), `onyx-site run` under systemd (or launchd). The key is
|
|
42
|
+
a 0600 file only the service user reads. Drivers don't get it in their environment, but they run
|
|
43
|
+
as that user, so they could read it.
|
|
44
|
+
|
|
45
|
+
## Enrol
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
onyx-site enrol --server https://app.onyx.dev
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
This is onyx-app's device flow with scope `site`: it prints a link and a code, an org admin
|
|
52
|
+
approves the machine on `/device` and names the site, and the site-runner stores the site's API
|
|
53
|
+
key (an `onyx_sk_…` key with only the `run` scope) in `<state_dir>/token`, mode 0600. The key is only ever sent to the
|
|
54
|
+
server it enrolled with, over verified TLS (`http://` only to localhost), in the `Authorization`
|
|
55
|
+
header, never following redirects. It is never put in an environment variable, the site file or
|
|
56
|
+
a log line. The site-runner replaces it every `rotate_days` (30) through `POST /sites/me/rotate`;
|
|
57
|
+
the old key works ten more minutes, so a lost response costs nothing.
|
|
58
|
+
|
|
59
|
+
## Run
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
onyx-site run # the service
|
|
63
|
+
onyx-site status # units, jobs, and the sync with onyx-app
|
|
64
|
+
onyx-site stop --job <id> # stop a job now, reported cancelled (works offline)
|
|
65
|
+
onyx-site hold --unit motor-3 --reason "belt" # no new jobs on the unit
|
|
66
|
+
onyx-site release --unit motor-3 # a held or quarantined unit
|
|
67
|
+
onyx-site rotate # a new key now
|
|
68
|
+
onyx-site conformance my_drivers.rig:MotorRig --config '{"port": "/dev/ttyACM0"}'
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
`onyx-site init` writes a site file to start from (`src/onyx_site/templates/site.toml`: onyx-app's
|
|
72
|
+
address, the file server (off until it has a certificate), and commented example units on the placeholder driver). By default the
|
|
73
|
+
site file is `~/.config/onyx-site/site.toml` and the state `~/.local/state/onyx-site` (root, or a
|
|
74
|
+
user whose site is already in `/etc/onyx-site`, keeps the system layout below).
|
|
75
|
+
`--site` / `ONYX_SITE_CONFIG` pick the site file; `--state-dir` / `ONYX_SITE_STATE_DIR` the state
|
|
76
|
+
folder. Stop, hold and release talk to the running service over `<state_dir>/control.sock`; with
|
|
77
|
+
the service down, hold and release are written to the journal.
|
|
78
|
+
|
|
79
|
+
## What it reports
|
|
80
|
+
|
|
81
|
+
Each sync carries every unit (`key` = its id in the site file, `type` = its unit type, which must
|
|
82
|
+
be an asset type in the org's registry, `free`, `ok`), every job it holds, and its reports:
|
|
83
|
+
|
|
84
|
+
| What happened | Report |
|
|
85
|
+
| --- | --- |
|
|
86
|
+
| a run started | `started`, with nothing else |
|
|
87
|
+
| it finished | `ended`, `succeeded`, with its props, metrics and files |
|
|
88
|
+
| the driver raised `JobFault` | `failed`, `job` (never retried) |
|
|
89
|
+
| it ran, but its data is missing or malformed (`RunResult(valid=False)`, bad metrics) | `failed`, `invalid`, keeping what it left |
|
|
90
|
+
| onyx-app refused an end's results (say, a metric its registry doesn't know) | the end again without them: `failed`, `invalid`, with onyx-app's reason |
|
|
91
|
+
| the site's policy refused it | `failed`, `refused`; it never started |
|
|
92
|
+
| the driver raised `UnitFault` | `failed`, `unit`; onyx-app holds the unit until someone resumes it |
|
|
93
|
+
| anything around the run (`InfraFault`, a fetch, the site-runner restarting mid-run) | `failed`, `infra` |
|
|
94
|
+
| onyx-app said `cancel`, or `onyx-site stop` | `cancelled` |
|
|
95
|
+
|
|
96
|
+
`unknown` and `lost` are onyx-app's to decide, never the site's. Files are refs
|
|
97
|
+
`{system: "onyx-site", locator: "<job>/<attempt>/<path>", contentHash, sizeBytes}`; onyx-app
|
|
98
|
+
records which site holds them and hands out links. A unit failing its health check is offered
|
|
99
|
+
`ok: false` with the reason: offline after one failure, and checked again every minute until it
|
|
100
|
+
passes; quarantined after two in a row (no more checks, until `onyx-site release`), but only if it
|
|
101
|
+
has been healthy since it was added: one that never has isn't set up yet, and stays offline.
|
|
102
|
+
|
|
103
|
+
A job with retries left stops when no sync has succeeded for `leaseMs` (onyx-app may run it
|
|
104
|
+
elsewhere); one without finishes under local supervision. A key onyx-app no longer accepts (401,
|
|
105
|
+
or 403/404 on sync) stops syncing; jobs in flight finish, and `onyx-site enrol` resumes.
|
|
106
|
+
|
|
107
|
+
## The site file
|
|
108
|
+
|
|
109
|
+
`onyx-site init` writes a commented site file to start from (in a checkout of the Onyx repository, `examples/site.toml` has every option). In short:
|
|
110
|
+
|
|
111
|
+
| Table | What |
|
|
112
|
+
| --- | --- |
|
|
113
|
+
| `[site]` | `server` (onyx-app), `state_dir`, `health_interval_s` (60), `stop_timeout_s` (10), `rotate_days` (30) |
|
|
114
|
+
| `[files]` | the file server: `bind` (`0.0.0.0:7443`), `public_url` (reported to onyx-app as `filesUrl`), `cert`/`key`, or `insecure_http = true` for development |
|
|
115
|
+
| `[retention]` | `days`, `max_gb`: whole jobs are deleted, oldest first |
|
|
116
|
+
| `[policy]` | `unit_types` (default: all below), `max_deadline_s`. onyx-app can't loosen it |
|
|
117
|
+
| `[repos."<name>"]` | what a job's `git` prop may name: `url`, and a read-only `ssh_key` or `token_file`. Nothing else is ever fetched |
|
|
118
|
+
| `[drivers.<name>]` | `ref = "module:Class"` (Python) or `program = "/path"`; `config = {…}` for the driver; `process = "driver"` or `"unit"` (one worker per unit); `python_path` |
|
|
119
|
+
| `[unit_types."<asset type>"]` | `driver`, `fidelity` (sim, hil, hardware, build), `physical` (default: hardware and hil) |
|
|
120
|
+
|
|
121
|
+
Units come from the drivers' `describe()`. The site file is watched: a change re-offers the units
|
|
122
|
+
and restarts changed drivers once their jobs end.
|
|
123
|
+
|
|
124
|
+
## Writing a driver
|
|
125
|
+
|
|
126
|
+
The full guide is [DRIVERS.md](https://github.com/onyx-robotics/onyx-agent/blob/main/docs/sites/DRIVERS.md): what the site-runner sends a driver, when, and what
|
|
127
|
+
becomes of what it returns. In short, a driver is written for **one** resource, and the site-runner
|
|
128
|
+
runs one instance of it per entry in its site-file section's `instances`, each in its own process,
|
|
129
|
+
handing it a job only when the resource is able to run it:
|
|
130
|
+
|
|
131
|
+
```python
|
|
132
|
+
import json
|
|
133
|
+
from pathlib import Path
|
|
134
|
+
from onyx_site import Driver, Unit, Health, RunContext, RunResult, JobFault, UnitFault, InfraFault
|
|
135
|
+
|
|
136
|
+
|
|
137
|
+
class MotorRig(Driver):
|
|
138
|
+
protocol = 1
|
|
139
|
+
version = "1.2.0"
|
|
140
|
+
|
|
141
|
+
def describe(self) -> Unit: # required: what this resource is
|
|
142
|
+
return Unit(self.id, "bench.motor_rig", {"rev": "B"}, exclusive_group="psu-a")
|
|
143
|
+
|
|
144
|
+
def health(self) -> Health: # required; between jobs, after failures
|
|
145
|
+
return Health(True)
|
|
146
|
+
|
|
147
|
+
def run(self, inp: Path, out: Path, ctx: RunContext) -> RunResult: # required
|
|
148
|
+
params = json.loads((inp / "params.json").read_text())
|
|
149
|
+
while not ctx.cancelled:
|
|
150
|
+
ctx.progress(rpm=812.0)
|
|
151
|
+
ctx.log("streaming")
|
|
152
|
+
...
|
|
153
|
+
(out / "recording.csv").write_text(...)
|
|
154
|
+
return RunResult(conditions={"temperature_c": 31.5}, metrics={"bench.final_rpm": 998.0})
|
|
155
|
+
|
|
156
|
+
def stop(self) -> None: ... # required: any time, twice, during run
|
|
157
|
+
def reset(self) -> RunResult: ... # required for physical units
|
|
158
|
+
|
|
159
|
+
def check(self, inp: Path) -> None: ... # optional: raise JobFault to refuse
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
* `self.config` is the instance's config: the section's `config` with its own `instances`
|
|
163
|
+
entry over it, which always has its `id` (also `self.id`).
|
|
164
|
+
* `inp` holds the job's props as `params.json`, and the fetched commit when its props name one
|
|
165
|
+
(`git: {url, commit}`). Write results to `out`: every file there is offered for download.
|
|
166
|
+
* Metric names must be metrics in the org's registry: an org's own pack names them
|
|
167
|
+
`<pack>.<key>`.
|
|
168
|
+
* Raise `JobFault(code, message)` for the job's own fault, `UnitFault` when the unit failed,
|
|
169
|
+
`InfraFault` for everything around it. `RunResult(valid=False)` reports `invalid`.
|
|
170
|
+
* `ctx.cancelled` turns true on stop or at the deadline; `ctx.wait(s)` sleeps but wakes on stop.
|
|
171
|
+
* The site-runner calls `stop()` with its own timeout (`stop_timeout_s`), then kills the
|
|
172
|
+
driver's process. Each Python driver runs in `python -m onyx_site.worker module:Class` and
|
|
173
|
+
speaks JSON lines; `print()` goes to the service log, not the protocol.
|
|
174
|
+
|
|
175
|
+
**Program drivers** (any language) answer subcommands instead, one process per call, for one
|
|
176
|
+
instance: `driver describe`, `driver health`, `driver run --in in/ --out out/ --deadline-s N`,
|
|
177
|
+
`driver stop`, `driver reset`, `driver check --in in/`. Exit code 0 ok, 10 job
|
|
178
|
+
fault, 11 unit fault, 12 infrastructure fault. Stdout lines are JSON (`{"log": …}`,
|
|
179
|
+
`{"progress": {…}}`, `{"result": {…}}`, `{"fault": {"code", "message"}}`) or plain text
|
|
180
|
+
(logged). `describe` prints `{"result": {"version", "calls": ["reset", …], "unit": {id,
|
|
181
|
+
unit_type, labels}}}`; the instance's config arrives as JSON in `ONYX_DRIVER_CONFIG`.
|
|
182
|
+
|
|
183
|
+
Before a driver gets real jobs, run `onyx-site conformance <module:Class | program>` (or
|
|
184
|
+
`onyx-site conformance <name> --from-site`): it runs every call, including stop during a run.
|
|
185
|
+
`--input` and `--long-input` give it a short job and one that runs until stopped.
|
|
186
|
+
|
|
187
|
+
Shipped examples: `onyx_site.drivers.local_build:LocalBuild` (runs the source's `build.sh` on this
|
|
188
|
+
host, as you, so only for code you trust) and
|
|
189
|
+
`onyx_site.drivers.sim_motor:SimMotor` (a simulated motor rig; `metrics_prefix` names its metrics
|
|
190
|
+
for an org's pack), and `onyx_site.drivers.local_program:LocalProgram`: a slot on this computer,
|
|
191
|
+
running a program from each job's commit (`command`), with the job's props in `$ONYX_PARAMS`,
|
|
192
|
+
everything it writes to `$ONYX_OUT` kept, and `$ONYX_OUT/metrics.json` recorded as its metrics.
|
|
193
|
+
It runs the program directly, so it's for code you trust; it's also the shortest complete driver
|
|
194
|
+
to start your own from. **Writing a driver, step by step: [DRIVERS.md](https://github.com/onyx-robotics/onyx-agent/blob/main/docs/sites/DRIVERS.md).**
|
|
195
|
+
|
|
196
|
+
## Running locally against onyx-app
|
|
197
|
+
|
|
198
|
+
1. Start onyx-app (`bun run dev` in `typescript/onyx-app`). The org needs an asset type for the
|
|
199
|
+
units and, for results to be recorded, the metrics the driver reports (a pack).
|
|
200
|
+
2. Enrol, keeping state in the project folder:
|
|
201
|
+
|
|
202
|
+
```bash
|
|
203
|
+
cd python/site-runner
|
|
204
|
+
uv run onyx-site --site examples/site.toml enrol
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
Approve the code on `http://localhost:3000/device` as an admin, and name the site.
|
|
208
|
+
3. Run it: `uv run onyx-site --site examples/site.toml run`. Its units show up as runner units in
|
|
209
|
+
onyx-app.
|
|
210
|
+
4. Submit a job for one of its unit types (`POST /api/v1/jobs`, or `onyx run --on <type>` from
|
|
211
|
+
onyx-app's CLI). `GET /api/v1/jobs/{id}/files` gives a signed link to each file on
|
|
212
|
+
`http://localhost:7443/files/{job}/{attempt}/`.
|
|
213
|
+
|
|
214
|
+
## Develop
|
|
215
|
+
|
|
216
|
+
```bash
|
|
217
|
+
uv sync
|
|
218
|
+
uv run ruff check && uv run ruff format --check && uv run ty check && uv run pytest
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
The tests run the service against a fake onyx-app that speaks the runner protocol: lost
|
|
222
|
+
responses, rejections, cancel and stale, the lease, rotation and revocation, and signed
|
|
223
|
+
downloads.
|
|
224
|
+
|
|
225
|
+
## Layout
|
|
226
|
+
|
|
227
|
+
| Module | |
|
|
228
|
+
| --- | --- |
|
|
229
|
+
| `driver.py` | the driver contract: `Driver`, `Unit`, `Health`, `RunContext`, `RunResult`, faults |
|
|
230
|
+
| `worker.py` | the process a Python driver runs in (JSON lines on stdin/stdout) |
|
|
231
|
+
| `driverhost.py` | starting, calling, killing drivers (Python workers and programs) |
|
|
232
|
+
| `service.py` | units, assignments, runs, reports, health, recovery after restart |
|
|
233
|
+
| `sync.py` | the runner protocol: the sync loop, `/sites/me`, key rotation, backoff |
|
|
234
|
+
| `journal.py` | SQLite: jobs, unacknowledged reports, unit holds |
|
|
235
|
+
| `policy.py` | the site's check of every assignment |
|
|
236
|
+
| `fetch.py` | git mirrors per repository |
|
|
237
|
+
| `store.py` | the site result store and retention |
|
|
238
|
+
| `fileserver.py` | signed download links, range requests |
|
|
239
|
+
| `auth.py`, `enrol.py` | the site's key and the device flow |
|
|
240
|
+
| `control.py`, `cli.py` | local control socket and the `onyx-site` command |
|
|
241
|
+
| `conformance.py` | the driver check |
|
|
242
|
+
|
|
243
|
+
## License
|
|
244
|
+
|
|
245
|
+
Apache-2.0 (see the `LICENSE` file in the package).
|