ydderd-momentum-cli 0.5.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- ydderd_momentum_cli-0.5.0/.gitignore +7 -0
- ydderd_momentum_cli-0.5.0/PKG-INFO +212 -0
- ydderd_momentum_cli-0.5.0/PUBLISHING.md +74 -0
- ydderd_momentum_cli-0.5.0/README.md +190 -0
- ydderd_momentum_cli-0.5.0/packaging/homebrew/momentum-cli.rb +36 -0
- ydderd_momentum_cli-0.5.0/pyproject.toml +42 -0
- ydderd_momentum_cli-0.5.0/scripts/release.sh +84 -0
- ydderd_momentum_cli-0.5.0/src/momentum_cli/__init__.py +34 -0
- ydderd_momentum_cli-0.5.0/src/momentum_cli/__main__.py +5 -0
- ydderd_momentum_cli-0.5.0/src/momentum_cli/_capture.py +236 -0
- ydderd_momentum_cli-0.5.0/src/momentum_cli/cli.py +506 -0
- ydderd_momentum_cli-0.5.0/src/momentum_cli/episodes.py +324 -0
- ydderd_momentum_cli-0.5.0/src/momentum_cli/eval_runs.py +388 -0
- ydderd_momentum_cli-0.5.0/src/momentum_cli/real_trials.py +93 -0
- ydderd_momentum_cli-0.5.0/src/momentum_cli/sdk.py +346 -0
- ydderd_momentum_cli-0.5.0/src/momentum_cli/secrets.py +73 -0
- ydderd_momentum_cli-0.5.0/tests/test_cli.py +261 -0
- ydderd_momentum_cli-0.5.0/tests/test_sdk.py +358 -0
- ydderd_momentum_cli-0.5.0/tests/test_sdk_capture_episodes.py +388 -0
- ydderd_momentum_cli-0.5.0/tests/test_sdk_eval_trials.py +373 -0
- ydderd_momentum_cli-0.5.0/tests/test_sdk_secrets.py +83 -0
- ydderd_momentum_cli-0.5.0/uv.lock +269 -0
|
@@ -0,0 +1,212 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: ydderd-momentum-cli
|
|
3
|
+
Version: 0.5.0
|
|
4
|
+
Summary: Momentum customer CLI + experiment-logging SDK — authenticate, upload field data, and report training runs to your workspace.
|
|
5
|
+
Project-URL: Homepage, https://withflywheel.com
|
|
6
|
+
Project-URL: Repository, https://github.com/ydderd/flywheel
|
|
7
|
+
Author: Momentum
|
|
8
|
+
License: Apache-2.0
|
|
9
|
+
Keywords: cli,ingest,momentum,upload
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Environment :: Console
|
|
12
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Topic :: Utilities
|
|
15
|
+
Requires-Python: >=3.11
|
|
16
|
+
Requires-Dist: boto3>=1.34
|
|
17
|
+
Requires-Dist: httpx>=0.27
|
|
18
|
+
Provides-Extra: dev
|
|
19
|
+
Requires-Dist: pytest>=8.0; extra == 'dev'
|
|
20
|
+
Requires-Dist: ruff>=0.4; extra == 'dev'
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
|
|
23
|
+
# flywheel-cli
|
|
24
|
+
|
|
25
|
+
The Momentum customer CLI + experiment-logging SDK: authenticate and bulk-upload field data
|
|
26
|
+
straight to your workspace's storage bucket, and report training/eval runs from your own compute
|
|
27
|
+
into your workspace's experiment tracker.
|
|
28
|
+
|
|
29
|
+
PyPI distribution: `ydderd-momentum-cli` · Homebrew formula: `momentum-cli` · command: `momentum`.
|
|
30
|
+
(The clean `momentum-cli` PyPI name was taken, so the distribution carries the `ydderd-` prefix;
|
|
31
|
+
the import package `momentum_cli`, the `momentum` command, and the brew name are unaffected.)
|
|
32
|
+
|
|
33
|
+
## Why this is a separate package
|
|
34
|
+
|
|
35
|
+
The CLI talks to the Momentum API purely over HTTP (and to R2 over S3). It shares **no Python
|
|
36
|
+
code** with the backend, so it ships with a tiny dependency set — `httpx` + `boto3` — instead of
|
|
37
|
+
the full server stack (torch, opencv, fastapi, …). That keeps the install small and avoids
|
|
38
|
+
shipping the backend's AGPL detector to customers.
|
|
39
|
+
|
|
40
|
+
## Install
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
brew install ydderd/flywheel/flywheel-cli
|
|
44
|
+
# or:
|
|
45
|
+
pipx install ydderd-momentum-cli
|
|
46
|
+
flywheel --help
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
## Usage
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
momentum auth login # opens a browser; a workspace admin approves
|
|
53
|
+
momentum auth whoami # confirm tenant
|
|
54
|
+
momentum upload ./your-data --scan # bulk upload + trigger ingest
|
|
55
|
+
flywheel ingest status # ingest ledger stats
|
|
56
|
+
momentum eval submit --model hf://lab/pi05-fast --benchmark bench_roboarena@3 # open an eval run, print URL
|
|
57
|
+
flywheel trials log --policy hf://lab/pi05-fast --n 20 --successes 14 --calibration-set pcsk-…
|
|
58
|
+
flywheel trials log --csv trials.csv # bulk floor tallies (per-row partial success)
|
|
59
|
+
flywheel secrets set lab-bucket # store a secret (value from stdin); prints its creds_ref
|
|
60
|
+
flywheel secrets list # secret names + configured (never values)
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
`upload` always writes to your workspace's one raw prefix — there's no target to choose. Whether
|
|
64
|
+
what you uploaded is raw drone video (needs extraction) or already-extracted frames is classified
|
|
65
|
+
server-side once it lands, not by the client beforehand.
|
|
66
|
+
|
|
67
|
+
For headless/CI use, skip the browser with a token minted by a workspace admin:
|
|
68
|
+
`momentum auth login --token <fw_cli_…>`.
|
|
69
|
+
|
|
70
|
+
Config is stored at `~/.flywheel/config.json`. Auth precedence: `MOMENTUM_CLI_TOKEN` env >
|
|
71
|
+
config file.
|
|
72
|
+
|
|
73
|
+
## Experiment-logging SDK
|
|
74
|
+
|
|
75
|
+
Training and eval runs executed on your own compute (Modal, Brev, a lab box) report themselves
|
|
76
|
+
into your workspace's experiment tracker — W&B-style, and safe to leave in production training
|
|
77
|
+
code (a logging failure never raises into the train):
|
|
78
|
+
|
|
79
|
+
```python
|
|
80
|
+
import momentum_cli as momentum
|
|
81
|
+
|
|
82
|
+
run = momentum.init(name="my_sft_run", tags=["sft"], config={"iters": 800, "lr": 2e-4},
|
|
83
|
+
provider="modal")
|
|
84
|
+
run.log({"train/loss": 0.42}, step=100)
|
|
85
|
+
run.finish(status="succeeded", checkpoint_ref="r2://bucket/ckpt", cost_usd=295.26)
|
|
86
|
+
|
|
87
|
+
# later — scoring results and billed cost arrive after the train, so annotation
|
|
88
|
+
# works on finished runs:
|
|
89
|
+
momentum.annotate(run.id, results={"auroc": {"value": 0.61, "ci": [0.55, 0.67]}})
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
### Eval runs (policy context — the CI-integration path)
|
|
93
|
+
|
|
94
|
+
An eval process (a lab rig, Modal, the robot) evaluates `model × benchmark@version` and streams its
|
|
95
|
+
rollouts back. Same never-raise/heartbeat/reattach posture as training runs; rollouts buffer and flush
|
|
96
|
+
in batches, each with a client-generated id so a re-sent batch is idempotent:
|
|
97
|
+
|
|
98
|
+
```python
|
|
99
|
+
ev = momentum.eval_run(benchmark="bench_roboarena@3", model="hf://lab/pi05-fast", seeds=3)
|
|
100
|
+
ev.log_rollout(scenario="scn_pick", seed=0, status="success",
|
|
101
|
+
scorer={"success": True, "task_progress": 1.0}, latency_p50=61.0)
|
|
102
|
+
ev.log_rollout(scenario="scn_pick", seed=1, status="fail", scorer={"success": False})
|
|
103
|
+
ev.finish() # flushes any buffered rollouts first
|
|
104
|
+
ev.annotate(results={"headline": {"value": 0.5, "ci": [0.31, 0.69]}}) # post-hoc scoring
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
`eval_run()` prints the run URL on create; `eval_run(run_id=…)` (or `MOMENTUM_EVAL_RUN_ID`) reattaches
|
|
108
|
+
after a preemption. Runs land in the UI under **Eval runs**.
|
|
109
|
+
|
|
110
|
+
### Real trials (floor tallies → calibration audit)
|
|
111
|
+
|
|
112
|
+
Report real-robot trials of a policy; landing trials that ground a calibration set recomputes that world
|
|
113
|
+
model's τ/ρ trust:
|
|
114
|
+
|
|
115
|
+
```python
|
|
116
|
+
momentum.real_trials.log(policy="hf://lab/pi05-fast", scenario="scn_pick",
|
|
117
|
+
n=20, successes=14, operator="alice", calibration_set="pcsk-…")
|
|
118
|
+
|
|
119
|
+
report = momentum.real_trials.log_csv("trials.csv") # a path or raw CSV text; per-row partial success
|
|
120
|
+
print(report["accepted"], report["rejected"])
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
### Secrets & referenced episodes
|
|
124
|
+
|
|
125
|
+
Register an episode that lives in your own bucket by first storing its credentials in the tenant secret
|
|
126
|
+
store (Fernet-encrypted at rest; the value is never returned by a read), then passing the returned
|
|
127
|
+
`creds_ref`:
|
|
128
|
+
|
|
129
|
+
```python
|
|
130
|
+
ref = momentum.secrets.set("lab-bucket", '{"access_key": "…", "secret_key": "…"}') # → "secret://lab-bucket"
|
|
131
|
+
momentum.episodes.register("s3://lab-corpus/session_042", creds_ref=ref)
|
|
132
|
+
momentum.secrets.list() # {name: {configured: bool}}, incl. provider keys under provider:<name>
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
Auth: `MOMENTUM_API_KEY` env (a `fw_cli_…` token — inject it as a secret in your training
|
|
136
|
+
environment), falling back to the token saved by `momentum auth login`. `MOMENTUM_API_URL`
|
|
137
|
+
overrides the API endpoint. `with momentum.init(...) as run:` (and `momentum.eval_run(...)`) marks the
|
|
138
|
+
run failed (with the exception) if the block raises. Runs land in the workspace UI under
|
|
139
|
+
**Experiments** / **Eval runs**.
|
|
140
|
+
|
|
141
|
+
Release/consumption mechanics (PyPI, git-ref installs, versioning): see `PUBLISHING.md`.
|
|
142
|
+
|
|
143
|
+
## Developer notes
|
|
144
|
+
|
|
145
|
+
These knobs exist for Momentum developers and are intentionally hidden from customer-facing
|
|
146
|
+
help and docs:
|
|
147
|
+
|
|
148
|
+
- **`--api-url <url>` on `momentum auth login`** — persist a non-production API base URL to the
|
|
149
|
+
config (e.g. a local API). Hidden via `argparse.SUPPRESS`.
|
|
150
|
+
- **`MOMENTUM_API_URL` env** — override the API base per-invocation. Takes precedence over the
|
|
151
|
+
config file.
|
|
152
|
+
|
|
153
|
+
Precedence for the API base URL: `MOMENTUM_API_URL` env > `api_url` in config > default
|
|
154
|
+
(`https://flywheeling.fly.dev/api` — swap to a custom domain once one is live).
|
|
155
|
+
|
|
156
|
+
Point the CLI at a local backend during development:
|
|
157
|
+
|
|
158
|
+
```bash
|
|
159
|
+
MOMENTUM_API_URL=http://localhost:8000 momentum auth whoami
|
|
160
|
+
# or persist it:
|
|
161
|
+
momentum auth login --token <fw_cli_…> --api-url http://localhost:8000
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
### Local development
|
|
165
|
+
|
|
166
|
+
```bash
|
|
167
|
+
cd cli
|
|
168
|
+
uv sync
|
|
169
|
+
uv run flywheel --help
|
|
170
|
+
uv run pytest
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
## Releasing (PyPI + Homebrew)
|
|
174
|
+
|
|
175
|
+
PyPI is the source of truth; the Homebrew formula wraps the published PyPI sdist.
|
|
176
|
+
|
|
177
|
+
### 1. Publish to PyPI — via GitHub Actions (Trusted Publishing, no token)
|
|
178
|
+
|
|
179
|
+
The `.github/workflows/publish-cli.yml` workflow builds and publishes over OIDC. Cut a release
|
|
180
|
+
by pushing a namespaced tag from the monorepo default branch:
|
|
181
|
+
|
|
182
|
+
```bash
|
|
183
|
+
git tag cli-v0.1.0 && git push origin cli-v0.1.0
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
The PyPI project is `ydderd-momentum-cli`, published from `ydderd/flywheel` via the `pypi`
|
|
187
|
+
environment. (First publish activates the "pending" Trusted Publisher and creates the project.)
|
|
188
|
+
|
|
189
|
+
### 2. Update the Homebrew tap formula
|
|
190
|
+
|
|
191
|
+
After the PyPI release exists, point `release.sh` at your tap checkout — with `SKIP_PUBLISH=1`
|
|
192
|
+
it skips the upload and only fetches the published sdist's `url`/`sha256`, rewrites the formula,
|
|
193
|
+
and regenerates its Python `resource` blocks:
|
|
194
|
+
|
|
195
|
+
```bash
|
|
196
|
+
SKIP_PUBLISH=1 \
|
|
197
|
+
FORMULA_PATH=/path/to/homebrew-momentum/Formula/flywheel-cli.rb \
|
|
198
|
+
cli/scripts/release.sh
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Then commit + push the tap. Customers install with:
|
|
202
|
+
|
|
203
|
+
```bash
|
|
204
|
+
brew install ydderd/flywheel/flywheel-cli
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
> `release.sh` can also publish to PyPI itself (`UV_PUBLISH_TOKEN=pypi-… cli/scripts/release.sh`)
|
|
208
|
+
> if you prefer a token-based local release over the GitHub Action.
|
|
209
|
+
|
|
210
|
+
Bumping a release: change `version` in `pyproject.toml`, push a new `cli-v*` tag, then re-run
|
|
211
|
+
step 2.
|
|
212
|
+
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# Publishing the CLI + SDK
|
|
2
|
+
|
|
3
|
+
One distribution ships both surfaces: **`ydderd-momentum-cli`** on PyPI provides the `momentum`
|
|
4
|
+
command (auth, upload, ingest) and the experiment-logging SDK (`import momentum_cli as momentum`).
|
|
5
|
+
They share a package because they share everything that matters — token resolution, the API base,
|
|
6
|
+
and the constraint that the client is pure HTTP with no backend imports (see the dependency note
|
|
7
|
+
in `pyproject.toml`). The SDK landed in **0.3.0**.
|
|
8
|
+
|
|
9
|
+
## Consuming without PyPI (git ref)
|
|
10
|
+
|
|
11
|
+
Any environment that can reach the repo can install straight from git — this is how Sentinel
|
|
12
|
+
consumes the SDK until a PyPI release with these changes exists:
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
# pin a sha (preferred for training images — reproducible)
|
|
16
|
+
uv add "ydderd-momentum-cli @ git+https://github.com/ydderd/flywheel@<sha>#subdirectory=cli"
|
|
17
|
+
|
|
18
|
+
# or with pip, e.g. inside a Modal image definition
|
|
19
|
+
pip install "ydderd-momentum-cli @ git+https://github.com/ydderd/flywheel@main#subdirectory=cli"
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Pin a sha or tag, not a branch, in anything that builds a training image: an image rebuild should
|
|
23
|
+
not silently pick up a different SDK.
|
|
24
|
+
|
|
25
|
+
## Publishing to PyPI
|
|
26
|
+
|
|
27
|
+
The primary path is **GitHub Actions Trusted Publishing** (no token on any laptop) — see the
|
|
28
|
+
"Releasing" section of `README.md` for the full detail. The short version:
|
|
29
|
+
|
|
30
|
+
1. **Bump the version** in `cli/pyproject.toml` (single source of truth — the release tooling
|
|
31
|
+
greps it; keep `__init__.py.__version__` in sync by hand). Merge to the default branch.
|
|
32
|
+
2. **Push the namespaced tag** from the default branch:
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
git tag cli-v0.3.0 && git push origin cli-v0.3.0
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
`.github/workflows/publish-cli.yml` builds and publishes to PyPI over OIDC.
|
|
39
|
+
3. **Update the Homebrew formula** once the PyPI release exists:
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
SKIP_PUBLISH=1 cli/scripts/release.sh # rewrites the in-repo formula
|
|
43
|
+
# or FORMULA_PATH=/path/to/homebrew-momentum/Formula/flywheel-cli.rb for the tap checkout
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
4. **Verify:** `pip install "ydderd-momentum-cli==<version>"` in a scratch venv, then
|
|
47
|
+
`python -c "import momentum_cli as fw; print(fw.__version__)"` and `momentum --help`.
|
|
48
|
+
|
|
49
|
+
(`UV_PUBLISH_TOKEN=... cli/scripts/release.sh` still works as a manual fallback that does
|
|
50
|
+
build + publish + formula in one go, but the tag → Actions path is the normal one.)
|
|
51
|
+
|
|
52
|
+
## Versioning
|
|
53
|
+
|
|
54
|
+
- **Minor bump** for new SDK/CLI surface (0.2.x → 0.3.0 added the SDK).
|
|
55
|
+
- **Patch bump** for fixes.
|
|
56
|
+
- The SDK's wire contract is the `/cli/training-runs*` API; the server treats unknown body fields
|
|
57
|
+
as errors (pydantic), so adding SDK parameters generally means a server release first, then the
|
|
58
|
+
SDK release. Old SDKs against a newer server always work — the API only adds optional fields.
|
|
59
|
+
|
|
60
|
+
## The `import flywheel` question (deferred)
|
|
61
|
+
|
|
62
|
+
Instrumentation reads `import momentum_cli as momentum`. Shipping a bare top-level `momentum`
|
|
63
|
+
module was deliberately skipped: the name collides with the backend's internal package (breaks any
|
|
64
|
+
env that installs both, e.g. e2e), and the `momentum` name on PyPI is taken regardless. Revisit
|
|
65
|
+
only if the SDK becomes a customer-facing product in its own right — the options then are a
|
|
66
|
+
separate `momentum-sdk` distribution or negotiating the PyPI name, neither of which changes the
|
|
67
|
+
wire contract or this package's code.
|
|
68
|
+
|
|
69
|
+
## If the dependency set ever bloats
|
|
70
|
+
|
|
71
|
+
The SDK itself needs only `httpx`. `boto3` is here for the CLI's upload path. If a future CLI
|
|
72
|
+
feature drags in anything heavier, split the SDK out as its own minimal distribution at that
|
|
73
|
+
point — not before; two packages to release and version is a real cost and the current set is two
|
|
74
|
+
small deps.
|
|
@@ -0,0 +1,190 @@
|
|
|
1
|
+
# flywheel-cli
|
|
2
|
+
|
|
3
|
+
The Momentum customer CLI + experiment-logging SDK: authenticate and bulk-upload field data
|
|
4
|
+
straight to your workspace's storage bucket, and report training/eval runs from your own compute
|
|
5
|
+
into your workspace's experiment tracker.
|
|
6
|
+
|
|
7
|
+
PyPI distribution: `ydderd-momentum-cli` · Homebrew formula: `momentum-cli` · command: `momentum`.
|
|
8
|
+
(The clean `momentum-cli` PyPI name was taken, so the distribution carries the `ydderd-` prefix;
|
|
9
|
+
the import package `momentum_cli`, the `momentum` command, and the brew name are unaffected.)
|
|
10
|
+
|
|
11
|
+
## Why this is a separate package
|
|
12
|
+
|
|
13
|
+
The CLI talks to the Momentum API purely over HTTP (and to R2 over S3). It shares **no Python
|
|
14
|
+
code** with the backend, so it ships with a tiny dependency set — `httpx` + `boto3` — instead of
|
|
15
|
+
the full server stack (torch, opencv, fastapi, …). That keeps the install small and avoids
|
|
16
|
+
shipping the backend's AGPL detector to customers.
|
|
17
|
+
|
|
18
|
+
## Install
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
brew install ydderd/flywheel/flywheel-cli
|
|
22
|
+
# or:
|
|
23
|
+
pipx install ydderd-momentum-cli
|
|
24
|
+
flywheel --help
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## Usage
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
momentum auth login # opens a browser; a workspace admin approves
|
|
31
|
+
momentum auth whoami # confirm tenant
|
|
32
|
+
momentum upload ./your-data --scan # bulk upload + trigger ingest
|
|
33
|
+
flywheel ingest status # ingest ledger stats
|
|
34
|
+
momentum eval submit --model hf://lab/pi05-fast --benchmark bench_roboarena@3 # open an eval run, print URL
|
|
35
|
+
flywheel trials log --policy hf://lab/pi05-fast --n 20 --successes 14 --calibration-set pcsk-…
|
|
36
|
+
flywheel trials log --csv trials.csv # bulk floor tallies (per-row partial success)
|
|
37
|
+
flywheel secrets set lab-bucket # store a secret (value from stdin); prints its creds_ref
|
|
38
|
+
flywheel secrets list # secret names + configured (never values)
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
`upload` always writes to your workspace's one raw prefix — there's no target to choose. Whether
|
|
42
|
+
what you uploaded is raw drone video (needs extraction) or already-extracted frames is classified
|
|
43
|
+
server-side once it lands, not by the client beforehand.
|
|
44
|
+
|
|
45
|
+
For headless/CI use, skip the browser with a token minted by a workspace admin:
|
|
46
|
+
`momentum auth login --token <fw_cli_…>`.
|
|
47
|
+
|
|
48
|
+
Config is stored at `~/.flywheel/config.json`. Auth precedence: `MOMENTUM_CLI_TOKEN` env >
|
|
49
|
+
config file.
|
|
50
|
+
|
|
51
|
+
## Experiment-logging SDK
|
|
52
|
+
|
|
53
|
+
Training and eval runs executed on your own compute (Modal, Brev, a lab box) report themselves
|
|
54
|
+
into your workspace's experiment tracker — W&B-style, and safe to leave in production training
|
|
55
|
+
code (a logging failure never raises into the train):
|
|
56
|
+
|
|
57
|
+
```python
|
|
58
|
+
import momentum_cli as momentum
|
|
59
|
+
|
|
60
|
+
run = momentum.init(name="my_sft_run", tags=["sft"], config={"iters": 800, "lr": 2e-4},
|
|
61
|
+
provider="modal")
|
|
62
|
+
run.log({"train/loss": 0.42}, step=100)
|
|
63
|
+
run.finish(status="succeeded", checkpoint_ref="r2://bucket/ckpt", cost_usd=295.26)
|
|
64
|
+
|
|
65
|
+
# later — scoring results and billed cost arrive after the train, so annotation
|
|
66
|
+
# works on finished runs:
|
|
67
|
+
momentum.annotate(run.id, results={"auroc": {"value": 0.61, "ci": [0.55, 0.67]}})
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### Eval runs (policy context — the CI-integration path)
|
|
71
|
+
|
|
72
|
+
An eval process (a lab rig, Modal, the robot) evaluates `model × benchmark@version` and streams its
|
|
73
|
+
rollouts back. Same never-raise/heartbeat/reattach posture as training runs; rollouts buffer and flush
|
|
74
|
+
in batches, each with a client-generated id so a re-sent batch is idempotent:
|
|
75
|
+
|
|
76
|
+
```python
|
|
77
|
+
ev = momentum.eval_run(benchmark="bench_roboarena@3", model="hf://lab/pi05-fast", seeds=3)
|
|
78
|
+
ev.log_rollout(scenario="scn_pick", seed=0, status="success",
|
|
79
|
+
scorer={"success": True, "task_progress": 1.0}, latency_p50=61.0)
|
|
80
|
+
ev.log_rollout(scenario="scn_pick", seed=1, status="fail", scorer={"success": False})
|
|
81
|
+
ev.finish() # flushes any buffered rollouts first
|
|
82
|
+
ev.annotate(results={"headline": {"value": 0.5, "ci": [0.31, 0.69]}}) # post-hoc scoring
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
`eval_run()` prints the run URL on create; `eval_run(run_id=…)` (or `MOMENTUM_EVAL_RUN_ID`) reattaches
|
|
86
|
+
after a preemption. Runs land in the UI under **Eval runs**.
|
|
87
|
+
|
|
88
|
+
### Real trials (floor tallies → calibration audit)
|
|
89
|
+
|
|
90
|
+
Report real-robot trials of a policy; landing trials that ground a calibration set recomputes that world
|
|
91
|
+
model's τ/ρ trust:
|
|
92
|
+
|
|
93
|
+
```python
|
|
94
|
+
momentum.real_trials.log(policy="hf://lab/pi05-fast", scenario="scn_pick",
|
|
95
|
+
n=20, successes=14, operator="alice", calibration_set="pcsk-…")
|
|
96
|
+
|
|
97
|
+
report = momentum.real_trials.log_csv("trials.csv") # a path or raw CSV text; per-row partial success
|
|
98
|
+
print(report["accepted"], report["rejected"])
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### Secrets & referenced episodes
|
|
102
|
+
|
|
103
|
+
Register an episode that lives in your own bucket by first storing its credentials in the tenant secret
|
|
104
|
+
store (Fernet-encrypted at rest; the value is never returned by a read), then passing the returned
|
|
105
|
+
`creds_ref`:
|
|
106
|
+
|
|
107
|
+
```python
|
|
108
|
+
ref = momentum.secrets.set("lab-bucket", '{"access_key": "…", "secret_key": "…"}') # → "secret://lab-bucket"
|
|
109
|
+
momentum.episodes.register("s3://lab-corpus/session_042", creds_ref=ref)
|
|
110
|
+
momentum.secrets.list() # {name: {configured: bool}}, incl. provider keys under provider:<name>
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Auth: `MOMENTUM_API_KEY` env (a `fw_cli_…` token — inject it as a secret in your training
|
|
114
|
+
environment), falling back to the token saved by `momentum auth login`. `MOMENTUM_API_URL`
|
|
115
|
+
overrides the API endpoint. `with momentum.init(...) as run:` (and `momentum.eval_run(...)`) marks the
|
|
116
|
+
run failed (with the exception) if the block raises. Runs land in the workspace UI under
|
|
117
|
+
**Experiments** / **Eval runs**.
|
|
118
|
+
|
|
119
|
+
Release/consumption mechanics (PyPI, git-ref installs, versioning): see `PUBLISHING.md`.
|
|
120
|
+
|
|
121
|
+
## Developer notes
|
|
122
|
+
|
|
123
|
+
These knobs exist for Momentum developers and are intentionally hidden from customer-facing
|
|
124
|
+
help and docs:
|
|
125
|
+
|
|
126
|
+
- **`--api-url <url>` on `momentum auth login`** — persist a non-production API base URL to the
|
|
127
|
+
config (e.g. a local API). Hidden via `argparse.SUPPRESS`.
|
|
128
|
+
- **`MOMENTUM_API_URL` env** — override the API base per-invocation. Takes precedence over the
|
|
129
|
+
config file.
|
|
130
|
+
|
|
131
|
+
Precedence for the API base URL: `MOMENTUM_API_URL` env > `api_url` in config > default
|
|
132
|
+
(`https://flywheeling.fly.dev/api` — swap to a custom domain once one is live).
|
|
133
|
+
|
|
134
|
+
Point the CLI at a local backend during development:
|
|
135
|
+
|
|
136
|
+
```bash
|
|
137
|
+
MOMENTUM_API_URL=http://localhost:8000 momentum auth whoami
|
|
138
|
+
# or persist it:
|
|
139
|
+
momentum auth login --token <fw_cli_…> --api-url http://localhost:8000
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### Local development
|
|
143
|
+
|
|
144
|
+
```bash
|
|
145
|
+
cd cli
|
|
146
|
+
uv sync
|
|
147
|
+
uv run flywheel --help
|
|
148
|
+
uv run pytest
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
## Releasing (PyPI + Homebrew)
|
|
152
|
+
|
|
153
|
+
PyPI is the source of truth; the Homebrew formula wraps the published PyPI sdist.
|
|
154
|
+
|
|
155
|
+
### 1. Publish to PyPI — via GitHub Actions (Trusted Publishing, no token)
|
|
156
|
+
|
|
157
|
+
The `.github/workflows/publish-cli.yml` workflow builds and publishes over OIDC. Cut a release
|
|
158
|
+
by pushing a namespaced tag from the monorepo default branch:
|
|
159
|
+
|
|
160
|
+
```bash
|
|
161
|
+
git tag cli-v0.1.0 && git push origin cli-v0.1.0
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
The PyPI project is `ydderd-momentum-cli`, published from `ydderd/flywheel` via the `pypi`
|
|
165
|
+
environment. (First publish activates the "pending" Trusted Publisher and creates the project.)
|
|
166
|
+
|
|
167
|
+
### 2. Update the Homebrew tap formula
|
|
168
|
+
|
|
169
|
+
After the PyPI release exists, point `release.sh` at your tap checkout — with `SKIP_PUBLISH=1`
|
|
170
|
+
it skips the upload and only fetches the published sdist's `url`/`sha256`, rewrites the formula,
|
|
171
|
+
and regenerates its Python `resource` blocks:
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
SKIP_PUBLISH=1 \
|
|
175
|
+
FORMULA_PATH=/path/to/homebrew-momentum/Formula/flywheel-cli.rb \
|
|
176
|
+
cli/scripts/release.sh
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
Then commit + push the tap. Customers install with:
|
|
180
|
+
|
|
181
|
+
```bash
|
|
182
|
+
brew install ydderd/flywheel/flywheel-cli
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
> `release.sh` can also publish to PyPI itself (`UV_PUBLISH_TOKEN=pypi-… cli/scripts/release.sh`)
|
|
186
|
+
> if you prefer a token-based local release over the GitHub Action.
|
|
187
|
+
|
|
188
|
+
Bumping a release: change `version` in `pyproject.toml`, push a new `cli-v*` tag, then re-run
|
|
189
|
+
step 2.
|
|
190
|
+
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Homebrew formula for flywheel-cli.
|
|
2
|
+
#
|
|
3
|
+
# This is the starter formula. It belongs in a *tap* repo — create
|
|
4
|
+
# github.com/ydderd/homebrew-momentum and place this file at Formula/flywheel-cli.rb.
|
|
5
|
+
# Customers then install with:
|
|
6
|
+
#
|
|
7
|
+
# brew install ydderd/flywheel/flywheel-cli
|
|
8
|
+
#
|
|
9
|
+
# The `url`/`sha256` below are filled in by cli/scripts/release.sh after publishing to PyPI,
|
|
10
|
+
# and the `resource` blocks (httpx, boto3, and their transitive deps) are generated by:
|
|
11
|
+
#
|
|
12
|
+
# brew update-python-resources Formula/flywheel-cli.rb
|
|
13
|
+
#
|
|
14
|
+
class MomentumCli < Formula
|
|
15
|
+
include Language::Python::Virtualenv
|
|
16
|
+
|
|
17
|
+
desc "Authenticate and bulk-upload field data to your Momentum workspace"
|
|
18
|
+
homepage "https://withflywheel.com"
|
|
19
|
+
# release.sh replaces the two PLACEHOLDER lines with the PyPI sdist URL + sha256.
|
|
20
|
+
url "PLACEHOLDER_SDIST_URL"
|
|
21
|
+
sha256 "PLACEHOLDER_SHA256"
|
|
22
|
+
license "Apache-2.0"
|
|
23
|
+
|
|
24
|
+
depends_on "python@3.12"
|
|
25
|
+
|
|
26
|
+
# BEGIN RESOURCES — populated by `brew update-python-resources`. Do not edit by hand.
|
|
27
|
+
# END RESOURCES
|
|
28
|
+
|
|
29
|
+
def install
|
|
30
|
+
virtualenv_install_with_resources
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
test do
|
|
34
|
+
assert_match "usage: momentum", shell_output("#{bin}/flywheel --help")
|
|
35
|
+
end
|
|
36
|
+
end
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
[project]
|
|
2
|
+
# PyPI distribution name (`momentum-cli` was taken). The import package stays `momentum_cli`
|
|
3
|
+
# and the installed command stays `momentum`; the Homebrew formula keeps the clean name too.
|
|
4
|
+
name = "ydderd-momentum-cli"
|
|
5
|
+
version = "0.5.0"
|
|
6
|
+
description = "Momentum customer CLI + experiment-logging SDK — authenticate, upload field data, and report training runs to your workspace."
|
|
7
|
+
readme = "README.md"
|
|
8
|
+
requires-python = ">=3.11"
|
|
9
|
+
license = { text = "Apache-2.0" }
|
|
10
|
+
authors = [{ name = "Momentum" }]
|
|
11
|
+
keywords = ["momentum", "cli", "upload", "ingest"]
|
|
12
|
+
classifiers = [
|
|
13
|
+
"Development Status :: 3 - Alpha",
|
|
14
|
+
"Environment :: Console",
|
|
15
|
+
"License :: OSI Approved :: Apache Software License",
|
|
16
|
+
"Programming Language :: Python :: 3",
|
|
17
|
+
"Topic :: Utilities",
|
|
18
|
+
]
|
|
19
|
+
# The CLI talks to the Momentum API purely over HTTP + S3 — it shares no code with the
|
|
20
|
+
# backend, so it ships with a tiny dependency set (no torch/opencv/fastapi).
|
|
21
|
+
dependencies = [
|
|
22
|
+
"httpx>=0.27",
|
|
23
|
+
"boto3>=1.34",
|
|
24
|
+
]
|
|
25
|
+
|
|
26
|
+
[project.optional-dependencies]
|
|
27
|
+
dev = ["pytest>=8.0", "ruff>=0.4"]
|
|
28
|
+
|
|
29
|
+
[project.urls]
|
|
30
|
+
Homepage = "https://withflywheel.com"
|
|
31
|
+
Repository = "https://github.com/ydderd/flywheel"
|
|
32
|
+
|
|
33
|
+
# Package is `momentum-cli` (brew/pypi name); the command is `momentum`.
|
|
34
|
+
[project.scripts]
|
|
35
|
+
momentum = "momentum_cli.cli:main"
|
|
36
|
+
|
|
37
|
+
[build-system]
|
|
38
|
+
requires = ["hatchling"]
|
|
39
|
+
build-backend = "hatchling.build"
|
|
40
|
+
|
|
41
|
+
[tool.hatch.build.targets.wheel]
|
|
42
|
+
packages = ["src/momentum_cli"]
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
#
|
|
3
|
+
# Release momentum-cli: build → publish to PyPI → update the Homebrew formula.
|
|
4
|
+
#
|
|
5
|
+
# Prerequisites:
|
|
6
|
+
# - uv installed
|
|
7
|
+
# - PyPI auth: UV_PUBLISH_TOKEN env (a pypi- API token) or ~/.pypirc
|
|
8
|
+
# - (optional) Homebrew, to auto-generate the formula's Python resource blocks
|
|
9
|
+
#
|
|
10
|
+
# Usage:
|
|
11
|
+
# cli/scripts/release.sh # build, publish, update in-repo formula
|
|
12
|
+
# SKIP_PUBLISH=1 cli/scripts/release.sh # dry-run: build + compute hash from local sdist only
|
|
13
|
+
# FORMULA_PATH=/path/to/homebrew-momentum/Formula/momentum-cli.rb cli/scripts/release.sh
|
|
14
|
+
#
|
|
15
|
+
set -euo pipefail
|
|
16
|
+
|
|
17
|
+
CLI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
18
|
+
cd "$CLI_DIR"
|
|
19
|
+
|
|
20
|
+
PKG="ydderd-momentum-cli" # PyPI distribution name
|
|
21
|
+
DIST_NAME="ydderd_momentum_cli" # build artifact stem (PEP 503 normalized: - and . → _)
|
|
22
|
+
FORMULA_PATH="${FORMULA_PATH:-$CLI_DIR/packaging/homebrew/momentum-cli.rb}"
|
|
23
|
+
# Grep the version (avoids a hard dependency on Python 3.11+ tomllib for this one field).
|
|
24
|
+
VERSION="$(sed -n 's/^version *= *"\(.*\)".*/\1/p' pyproject.toml | head -1)"
|
|
25
|
+
[ -n "$VERSION" ] || { echo "error: could not parse version from pyproject.toml" >&2; exit 1; }
|
|
26
|
+
|
|
27
|
+
echo "==> Releasing $PKG $VERSION"
|
|
28
|
+
|
|
29
|
+
echo "==> Building sdist + wheel"
|
|
30
|
+
rm -rf dist
|
|
31
|
+
uv build
|
|
32
|
+
|
|
33
|
+
SDIST="dist/${DIST_NAME}-${VERSION}.tar.gz"
|
|
34
|
+
[ -f "$SDIST" ] || { echo "error: expected $SDIST after build" >&2; exit 1; }
|
|
35
|
+
LOCAL_SHA="$(shasum -a 256 "$SDIST" | awk '{print $1}')"
|
|
36
|
+
echo " sdist sha256: $LOCAL_SHA"
|
|
37
|
+
|
|
38
|
+
if [ "${SKIP_PUBLISH:-0}" = "1" ]; then
|
|
39
|
+
echo "==> SKIP_PUBLISH=1 — not uploading to PyPI"
|
|
40
|
+
# Fall back to a canonical (non-hashed) PyPI URL so the formula is still updatable in a dry run.
|
|
41
|
+
SDIST_URL="https://files.pythonhosted.org/packages/source/${PKG:0:1}/${PKG}/${DIST_NAME}-${VERSION}.tar.gz"
|
|
42
|
+
SHA256="$LOCAL_SHA"
|
|
43
|
+
else
|
|
44
|
+
echo "==> Publishing to PyPI"
|
|
45
|
+
uv publish
|
|
46
|
+
|
|
47
|
+
echo "==> Waiting for PyPI to register $PKG $VERSION"
|
|
48
|
+
for _ in $(seq 1 30); do
|
|
49
|
+
META="$(curl -fsSL "https://pypi.org/pypi/${PKG}/${VERSION}/json" 2>/dev/null || true)"
|
|
50
|
+
[ -n "$META" ] && break
|
|
51
|
+
sleep 5
|
|
52
|
+
done
|
|
53
|
+
[ -n "$META" ] || { echo "error: $PKG $VERSION did not appear on PyPI" >&2; exit 1; }
|
|
54
|
+
|
|
55
|
+
read -r SDIST_URL SHA256 <<EOF
|
|
56
|
+
$(printf '%s' "$META" | python3 -c "import sys,json; d=json.load(sys.stdin); s=next(u for u in d['urls'] if u['packagetype']=='sdist'); print(s['url'], s['digests']['sha256'])")
|
|
57
|
+
EOF
|
|
58
|
+
if [ "$SHA256" != "$LOCAL_SHA" ]; then
|
|
59
|
+
echo "warning: PyPI sdist sha256 ($SHA256) != local build ($LOCAL_SHA)" >&2
|
|
60
|
+
fi
|
|
61
|
+
fi
|
|
62
|
+
|
|
63
|
+
echo "==> Updating formula: $FORMULA_PATH"
|
|
64
|
+
[ -f "$FORMULA_PATH" ] || { echo "error: formula not found at $FORMULA_PATH" >&2; exit 1; }
|
|
65
|
+
# Replace the url + sha256 lines (works for both PLACEHOLDER and a prior release's values).
|
|
66
|
+
python3 - "$FORMULA_PATH" "$SDIST_URL" "$SHA256" <<'PY'
|
|
67
|
+
import re, sys
|
|
68
|
+
path, url, sha = sys.argv[1:4]
|
|
69
|
+
text = open(path).read()
|
|
70
|
+
text = re.sub(r'^(\s*url\s+").*(")\s*$', rf'\g<1>{url}\g<2>', text, count=1, flags=re.M)
|
|
71
|
+
text = re.sub(r'^(\s*sha256\s+").*(")\s*$', rf'\g<1>{sha}\g<2>', text, count=1, flags=re.M)
|
|
72
|
+
open(path, 'w').write(text)
|
|
73
|
+
PY
|
|
74
|
+
|
|
75
|
+
if command -v brew >/dev/null 2>&1; then
|
|
76
|
+
echo "==> Regenerating Python resource blocks (brew update-python-resources)"
|
|
77
|
+
brew update-python-resources "$FORMULA_PATH" || \
|
|
78
|
+
echo "note: update-python-resources failed — run it manually in your tap checkout"
|
|
79
|
+
else
|
|
80
|
+
echo "note: Homebrew not found — run 'brew update-python-resources $FORMULA_PATH' in your tap"
|
|
81
|
+
fi
|
|
82
|
+
|
|
83
|
+
echo "==> Done. Commit the updated formula to your tap (github.com/ydderd/homebrew-momentum)."
|
|
84
|
+
echo " Customers install with: brew install ydderd/momentum/${PKG}"
|