vigily 0.1.0b1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (33) hide show
  1. vigily-0.1.0b1/.gitignore +16 -0
  2. vigily-0.1.0b1/CHANGELOG.md +261 -0
  3. vigily-0.1.0b1/Cargo.toml +65 -0
  4. vigily-0.1.0b1/LICENSE +21 -0
  5. vigily-0.1.0b1/PKG-INFO +245 -0
  6. vigily-0.1.0b1/README.md +222 -0
  7. vigily-0.1.0b1/crates/vigily-api/Cargo.toml +25 -0
  8. vigily-0.1.0b1/crates/vigily-cli/Cargo.toml +37 -0
  9. vigily-0.1.0b1/crates/vigily-core/Cargo.toml +37 -0
  10. vigily-0.1.0b1/crates/vigily-server/Cargo.toml +64 -0
  11. vigily-0.1.0b1/crates/vigily-server/LICENSE +1 -0
  12. vigily-0.1.0b1/crates/vigily-server/README.md +14 -0
  13. vigily-0.1.0b1/crates/vigily-ui/Cargo.toml +16 -0
  14. vigily-0.1.0b1/pyproject.toml +67 -0
  15. vigily-0.1.0b1/python/tests/conftest.py +149 -0
  16. vigily-0.1.0b1/python/tests/test_autostart.py +111 -0
  17. vigily-0.1.0b1/python/tests/test_evals.py +175 -0
  18. vigily-0.1.0b1/python/tests/test_sdk.py +187 -0
  19. vigily-0.1.0b1/python/tests/test_sender.py +322 -0
  20. vigily-0.1.0b1/python/tests/test_tracing.py +365 -0
  21. vigily-0.1.0b1/python/tests/test_wheel.py +47 -0
  22. vigily-0.1.0b1/python/vigily/__init__.py +693 -0
  23. vigily-0.1.0b1/python/vigily/_config.py +141 -0
  24. vigily-0.1.0b1/python/vigily/_engine.py +245 -0
  25. vigily-0.1.0b1/python/vigily/_http.py +243 -0
  26. vigily-0.1.0b1/python/vigily/_ids.py +43 -0
  27. vigily-0.1.0b1/python/vigily/_sender.py +1252 -0
  28. vigily-0.1.0b1/python/vigily/_spool.py +186 -0
  29. vigily-0.1.0b1/python/vigily/datasets.py +226 -0
  30. vigily-0.1.0b1/python/vigily/evaluation.py +234 -0
  31. vigily-0.1.0b1/python/vigily/evaluators.py +242 -0
  32. vigily-0.1.0b1/python/vigily/py.typed +0 -0
  33. vigily-0.1.0b1/python/vigily/tracing.py +709 -0
@@ -0,0 +1,16 @@
1
+ /target
2
+ **/node_modules/
3
+ crates/vigily-ui/dist/
4
+ *.tsbuildinfo
5
+ __pycache__/
6
+ *.py[cod]
7
+ .venv/
8
+ *.so
9
+ *.pyd
10
+ .pytest_cache/
11
+ .ruff_cache/
12
+ /.vigily/
13
+ /site/
14
+ .DS_Store
15
+ /dist/
16
+ /vigily-data/
@@ -0,0 +1,261 @@
1
+ # Changelog
2
+
3
+ All notable changes to vigily are documented here. The format follows
4
+ [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
5
+ [Semantic Versioning](https://semver.org/).
6
+
7
+ ## [Unreleased]
8
+
9
+ ## [0.1.0-beta.1](https://github.com/sayef/vigily/releases/tag/v0.1.0-beta.1) - 2026-11-15
10
+
11
+ The first published release, as betas (0.1.0-beta.N) before 0.1.0. Earlier builds were never
12
+ released; Changed, Deprecated and Removed describe what differs from them, mainly that the Python
13
+ SDK no longer writes storage in the training process but logs to the engine over HTTP (see
14
+ docs/guides/whats-new.md).
15
+
16
+ ### Added
17
+
18
+ - `vigily-core`: runs, parameters and metric series behind a storage trait, with a kv0 and an
19
+ in-memory backend; an incrementally refreshed catalog with per-series statistics; a run query
20
+ language; min/max-envelope downsampling onto a shared x axis; group aggregation with interpolation.
21
+ - `vigily-server`: a JSON API (meta, runs, run detail and changes, chart data) and the embedded web
22
+ app, answering on localhost only by default.
23
+ - `vigily` command: `up`, `runs` and `optimize`.
24
+ - Python SDK: `vigily.Run` (parameters, `track` with step, epoch, context and timestamp, tags,
25
+ notes, `finish`, context manager, finish at exit) and `vigily.Repo` for reading back.
26
+ - Web app: runs table, metric explorer, run pages, experiments, command palette, keyboard shortcuts,
27
+ light and dark themes.
28
+ - Run `kind` (`training`, `eval`, ...) and `target` reference, filterable as `kind:` and `target.<key>`.
29
+ - `vigily up` optimizes storage every 15 minutes in the background (`--no-optimize`).
30
+ - Runs whose process stopped (no heartbeat for 10 minutes) show as "No heartbeat" and can be marked
31
+ aborted; runs that are not running can be deleted, with an undo window.
32
+ - `examples/demo.py` with 43 synthetic runs, and `bench/scale.py` and `bench/scale.sh` for scale tests.
33
+
34
+ - `vigily-engine`, the server as its own binary (`vigily up` runs it in the foreground), with
35
+ `--state-dir`, `--detach`, `--idle-timeout` and `--check-config`. One engine serves a
36
+ repository on a machine: it holds `engines/<key>.lock` in the state directory and writes its
37
+ address, version and local token to `engines/<key>.json` (0600) while it runs; a second engine
38
+ on the same repository exits with code 3 and names the first one's address. An engine started
39
+ with `--detach` logs to `engines/<key>.log`, takes a free port when its own is taken and stops
40
+ after 30 idle minutes.
41
+ - `/api/v1` routes for clients other than the web app: the runs list pages with `limit` and
42
+ `offset`, answers `matched` and takes `fields=summary`; a run's parameters, series and the points
43
+ of one series (with a time window; NaN and infinities as strings); dataset versions and a
44
+ version's items as NDJSON; an eval run's results; `POST admin/purge` (with
45
+ `legacy_records_present`, and a 409 `tag_blocks_purge` when a storage tag keeps deleted data) and
46
+ `POST admin/optimize`; `POST traces/delete` for several traces, optionally erasing them; `POST
47
+ schedules/{name}/run`, which answers 202 with the run's id at once; and `GET tracing/policy`.
48
+ Admin routes accept only JSON from the server's own origin, like span ingest.
49
+ - A write path for clients over HTTP: `POST /api/v1/runs` creates a run with an id the client
50
+ chooses (creating it again answers the existing run), `POST runs/{id}/resume`, `finish`,
51
+ `results` and `features`, `POST datasets/{name}/versions` (with a required `Idempotency-Key`,
52
+ remembered for 24 hours), `POST flush`, and `POST ingest`: ordered batches of operations with
53
+ points in columns, as JSON or NDJSON, optionally gzip-compressed. A replayed batch is answered
54
+ with its first report; a batch resent after an engine restart stores nothing twice because
55
+ clients assign steps. A finished run's results and feature samples are always in storage before
56
+ its status changes. Write bodies are limited to 32 MiB, ingest in flight to 256 MiB (then 503
57
+ with `Retry-After`), and a write to a deleted run answers `run_deleted`. Any write counts as the
58
+ run's heartbeat (at most one stored every two minutes), and new data shows up without waiting
59
+ for the next catalog refresh.
60
+ - Sign-in. `vigily up` runs in local mode on a loopback address: a random token per repository,
61
+ kept in a private file, opens the API, and the link it prints (`/?token=...`) stores it in an
62
+ HttpOnly, SameSite=Strict cookie. On any other address (and in Docker) it runs in server mode and
63
+ refuses to start without `VIGILY_ADMIN_TOKEN` (or `VIGILY_ADMIN_TOKEN_FILE`) unless given
64
+ `--insecure-no-auth`. Admins create tokens with the scopes `read`, `ingest` and `admin`
65
+ (`/api/v1/auth/tokens`), stored as SHA-256 hashes; the web app shows a sign-in page when needed.
66
+ Cookie-authenticated writes must come from vigily's own origin. Scheduled commands get
67
+ `VIGILY_URL` and a short-lived token; labels record who entered them. The SDK's span exporter
68
+ sends `VIGILY_TOKEN` (or `vigily.tracing.configure(token=...)`).
69
+ - A typed Rust client for `/api/v1` in `vigily-api` (feature `client`), one method per route, with
70
+ structured errors and gzip-compressed large request bodies, and `vigily_api::local::resolve`,
71
+ which finds a repository's engine (`VIGILY_URL`, then its state file) or starts one in the
72
+ background. `GET /api/v1/schedules` also answers each schedule's `retry_delay`.
73
+ - New commands: `vigily status`, `stop`, `open`, `env`, `tokens list|create|revoke` and
74
+ `token rotate` (`POST /api/v1/auth/local-token/rotate`, admin), and global `--url` and `--token`
75
+ options (`VIGILY_URL`, `VIGILY_TOKEN`) to use a team server.
76
+ - `vigily.configure(url, token, project, offline)`, `vigily.flush(timeout)`, `Repo(url=...)`,
77
+ `Repo.results(run_id)` and `vigily.tracing.export(spans)` in the Python SDK; settings also from
78
+ `VIGILY_URL`, `VIGILY_TOKEN`, `VIGILY_PROJECT`, `VIGILY_OFFLINE`, `VIGILY_EXIT_TIMEOUT`,
79
+ `VIGILY_SPOOL_AFTER`, `VIGILY_SPOOL_MAX_MB` and `VIGILY_BUFFER_MB`.
80
+ - An offline spool: batches the SDK cannot send (after two minutes without the engine, at exit, or
81
+ when the engine falls behind) go to gzip-compressed NDJSON files in `spool/<writer>/` under the
82
+ state directory (0600, written atomically), and are sent in order once the engine answers, by the
83
+ next process that reaches it, or with the new `vigily sync`. The spool has a size limit and never
84
+ deletes data.
85
+ - `vigily-engine` platform wheels (`pip install 'vigily[local]'`): the engine binary alone, built by
86
+ maturin (`bindings = "bin"`) for Linux x86_64 and arm64 (manylinux 2_28), Linux x86_64 with musl
87
+ and macOS arm64 and x86_64, stripped, versioned like the workspace; the release workflow builds
88
+ them (and, on pull requests that change how, only builds them) and has a `pypi-engine` job to
89
+ publish them with trusted publishing, from its own `pypi-engine` environment.
90
+ - `bench/ingest.py` measures ingest through the SDK into an engine (track latency, points per
91
+ second per process and engine-wide, engine CPU and memory, time until new points are readable),
92
+ and `bench/compare.sh` compares an engine built from another commit with this checkout's; the
93
+ Benchmarks workflow runs both.
94
+
95
+ - Datasets: content-addressed, immutable versions; `Dataset.create`, `from_jsonl`, `from_csv`, `add`.
96
+ - Evaluations: `evaluate()` with exact match, contains, regex, numeric, JSON schema and custom
97
+ evaluators (including LLM-as-judge as user code); per-item results, human labels, run comparison
98
+ and a JSON audit export.
99
+ - Scheduled evaluations from `vigily.toml`: cron with time zones, no overlap, missed-run policy,
100
+ retries, timeouts, captured output; `vigily schedules [run NAME]`.
101
+ - Drift: behaviour drift (Mann–Whitney U, bootstrap interval, per-item regressions) and data drift
102
+ (PSI, Kolmogorov–Smirnov, Jensen–Shannon, chi-squared) against rolling or pinned baselines.
103
+ - Alerts for drift and failed schedules, with an optional webhook.
104
+ - Web app: Datasets, Drift, Schedules and Alerts pages; Items and Audit tabs on eval runs; Compare.
105
+ - `examples/eval_demo.py`: 30 synthetic nights of two targets with injected drift.
106
+ - Evaluation records in one storage table per kind (results, labels, schedules, alerts, ...) and
107
+ one per dataset, so a dataset version's tag keeps only that dataset. Repositories with the
108
+ earlier shared records table are migrated once on open; the old table is kept for you to delete.
109
+ - `vigily runs purge [--force]`: erases deleted runs from storage and reports what it removed.
110
+ - `vigily up --optimize-every <duration>` and `--optimize-at-start`; the server returns memory to
111
+ the system after each background optimize.
112
+
113
+ - LLM tracing. `vigily.trace(...)` (a context manager, `async with`, and a decorator) records nested
114
+ spans through `contextvars`, with `span.set_usage(...)` and an opt-in `usage_from(response)` that
115
+ reads common client responses without importing them. Spans are exported in batches from a
116
+ background thread and sent on normal exit. The engine ingests them at
117
+ `POST /api/v1/traces/ingest` and accepts OpenTelemetry OTLP/HTTP JSON at `POST /v1/traces`.
118
+ - Cost from a price table in `vigily.toml` (`[tracing.prices]`, per million input, output and
119
+ cached input tokens, in a configured currency), computed when a span arrives. No prices are
120
+ built in.
121
+ - Payload modes for traces, global or per project: `full`, `truncated` (the default, 4 KiB per
122
+ field) or `metadata` (no inputs or outputs); a redaction hook (`vigily.tracing.set_redactor`).
123
+ Traces linked to a run are deleted with it and erased by `vigily runs purge`, which now
124
+ compacts first so deleted traces without a run are erased too; `vigily traces delete [--erase]`.
125
+ Until a trace's project is known the strictest configured policy applies; truncated spans are
126
+ capped at 64 KiB in total and 128 attributes.
127
+ - Purging compacts first inside the storage layer, so every caller erases trace rows moved under a
128
+ run; an interrupted link of a trace to its run is repaired on the next read or ingest; the trace
129
+ page says when a span's payload was limited because its project was not known yet.
130
+ - The ingest endpoints accept only `application/json` and refuse other origins, so a web page
131
+ cannot post spans to a local server.
132
+ - Traces in the web app: a Traces page under Track with filters (run, model, status, time),
133
+ latency, tokens and cost; a trace page with a span waterfall, the selected span's inputs, outputs
134
+ and attributes, and token and cost totals; links from runs and evaluation items.
135
+ - The demo generator logs synthetic support-assistant traces.
136
+
137
+ - A redesigned web app: warm paper neutrals and ink rules, a sidebar grouped Track / Evaluate /
138
+ Workspace with counts, a top bar with breadcrumbs and search, summary figures, dense tables with
139
+ loss sparklines, and a solid ink selection bar. Light and dark themes; a drawer on phones.
140
+ - Compare runs (`/metrics/compare?runs=...`): selected runs keep their letters A, B, C, D, with a
141
+ headline chart labelled at the line ends, a supporting chart and only the settings that differ.
142
+ - Drift suites show the score per run with its baseline band, a strip per score and the items that
143
+ flipped from pass to fail. A Settings page shows the repository and the browser's preferences.
144
+ - The documentation site in the same paper design: a masthead over an ink rule, a home page that
145
+ opens with the point and numbered figures, ruled sections, a margin column with the page contents,
146
+ and a Copy page button.
147
+ - A documentation site (mkdocs-material) and brand assets (mark, wordmark, banners, social
148
+ preview) built reproducibly by `assets/src/build.py`.
149
+ - Documentation: an HTTP API reference with the scope of every route, a guide to running a team
150
+ server (tokens, TLS through a reverse proxy, backups), and a guide to upgrading from the builds
151
+ where the SDK wrote storage itself.
152
+
153
+ ### Changed
154
+
155
+ - The Python SDK is pure Python (one `py3-none-any` wheel of about 60 kB, no dependencies) and talks
156
+ to the engine over `/api/v1` instead of writing storage in the process; the pyo3 extension and
157
+ the `vigily-py` crate are gone, and the package moves to `python/vigily`. It finds the
158
+ repository's engine like the command line, or starts `vigily-engine --detach` (install hint when
159
+ it is missing). `Run()` never waits for the network: run ids (UUIDv7), steps and timestamps are
160
+ assigned in the process, and a background thread sends an ordered log of operations in batches
161
+ (every second, 20 000 points or 4 MiB; points in columns; gzip; `Idempotency-Key` per batch),
162
+ retried with exponential backoff and full jitter, honouring `Retry-After`. `track()` waits up to
163
+ five seconds when the engine falls behind, then the batches go to the spool; nothing is dropped
164
+ without a message. Forked children log through their own sender; at exit the SDK finishes the
165
+ runs it started and sends what is queued for up to 30 seconds. Spans share the sender; the
166
+ redactor and the engine's payload policy (fetched per project) apply before a span is queued or
167
+ spooled. `RunDeletedError` surfaces on the next call after the engine reported the deletion, and
168
+ `Run.finish()` no longer waits for storage (`vigily.flush()` and `Repo` reads do).
169
+ - `vigily` is now a client of the engine: every command but `up` goes through the HTTP API, and the
170
+ binary no longer contains storage code (`vigily-core` and kv0 are out of its dependency tree).
171
+ Without `--url` it uses the repository's local engine, starting one in the background when none
172
+ runs (not for a directory that does not exist). `vigily up` runs `vigily-engine` in the
173
+ foreground with the same options and output. `vigily schedules` lists the engine's schedules
174
+ (`--config` is gone; it is an option of `vigily up`), and `vigily schedules run` starts the run on
175
+ the engine and follows it through its retries, exiting 1 unless it succeeds. Release archives
176
+ and the Docker image contain both `vigily` and `vigily-engine`, and `scripts/install.sh` installs
177
+ both; the `s3`, `gcs` and `azure` features move to the `vigily-server` crate.
178
+ - The HTTP API moves to `/api/v1`, with `GET /api/v1/info` (version, API version, mode and
179
+ capabilities). Errors there are `{"error": {"code", "message", "position"?}}` with a stable code
180
+ (`not_found`, `invalid_argument`, `conflict`, ...). The old `/api/*` paths keep working as
181
+ deprecated aliases with a `Deprecation` header, a `Link` to the successor and the old flat error
182
+ body; they are removed before 0.1.0. OpenTelemetry's `POST /v1/traces` is unchanged. The web app
183
+ and the Python SDK's span exporter use `/api/v1`.
184
+ - Every API route except health, info and `auth/me` needs a token, including the deprecated
185
+ `/api/*` aliases and OTLP's `/v1/traces`. `docker compose up` needs `VIGILY_ADMIN_TOKEN`.
186
+ - The JSON types of the HTTP API (the run model, every route's answer, evaluation, drift and trace
187
+ types) move to a new crate, `vigily-api`, which depends on serde alone; `vigily-core` re-exports
188
+ them. Schedules and suites answer with typed structures instead of ad-hoc JSON. Golden JSON
189
+ files in `crates/vigily-api/tests/fixtures` pin the wire format; the web app sees the same JSON.
190
+ - The engine writes each ingested series batch with kv0's `Run::track_many` (one lock and one
191
+ log append per batch instead of per point): about five times the points per second with eight
192
+ SDK processes, at a tenth of the CPU time. After writes it commits buffered points (at most once
193
+ a second) before refreshing the run list, so new points are readable within about a second.
194
+ Finishing a run commits its points and status together, one storage commit per run instead of
195
+ two.
196
+ - The engine stores every explicit `heartbeat` operation (implicit ones stay at one per two
197
+ minutes), and `POST /api/v1/runs/{id}/resume` flushes the run first, so a run created a moment ago
198
+ can be resumed.
199
+ - The server stops cleanly on SIGTERM as well as Ctrl-C (`docker stop`, service managers): open
200
+ requests get 10 seconds, then open runs, storage and records are flushed.
201
+ - The local token moves to `engines/<key>.token` in the state directory, which defaults to
202
+ `~/Library/Caches/vigily` on macOS (`~/.cache/vigily` elsewhere); a token kept by an earlier
203
+ build under `tokens/` is moved there on the first start.
204
+ - The Docker image runs `vigily-engine --host 0.0.0.0 --mode server` (entry point `vigily-engine`)
205
+ and checks `/api/v1/health`. `compose.yaml` keeps the data in the `vigily` volume only: the
206
+ `VIGILY_DATA`, `VIGILY_UID`, `VIGILY_GID` and `VIGILY_SERVER_REPO` variables are gone, since
207
+ scripts now log over HTTP (`VIGILY_URL` and `VIGILY_TOKEN`) instead of writing to a shared
208
+ directory. It gives the engine 30 seconds to stop, and shows a `trainer` service logging from a
209
+ second container. New guide: running a team server (tokens, TLS, backups).
210
+ - `examples/demo.py` sends its traces over HTTP (`vigily.tracing.export`); their example prices
211
+ moved to `examples/vigily.toml`.
212
+ - Recursive replaces Atkinson Hyperlegible as vigily's typeface, in the app, the docs and the brand
213
+ assets: Sans Linear for text and Mono Linear for run names, values, keys and code, from one
214
+ self-hosted variable file (SIL OFL). Bold is 660 and headings track slightly tighter; the wordmark,
215
+ banners, social card and screenshots are regenerated.
216
+
217
+ ### Deprecated
218
+
219
+ - The `/api/*` routes of earlier builds answer as aliases of their `/api/v1` successors, with
220
+ `Deprecation: true` and a `Link` header, and are removed before 0.1.0. `/v1/traces` (OTLP) stays.
221
+ - An SDK from before the engine/client split still writes storage directly next to a running engine,
222
+ which picks up its runs; upgrade it so that the engine is the only writer.
223
+
224
+ ### Removed
225
+
226
+ - The pyo3 extension and the `vigily-py` crate: the SDK is pure Python (see Changed).
227
+ - `vigily schedules --config`: the configuration is an option of `vigily up` and `vigily-engine`.
228
+ - The `VIGILY_DATA`, `VIGILY_UID`, `VIGILY_GID` and `VIGILY_SERVER_REPO` variables of `compose.yaml`.
229
+
230
+ ### Fixed
231
+
232
+ - The SDK recorded the branch name as `__system__.git.commit`; it now records the commit hash
233
+ (and the branch as `git.branch`).
234
+ - Web app edits (name, tags, notes, archive) on a running run are no longer reverted when the SDK
235
+ finishes it: both sides re-read the run and change only their own fields.
236
+ - `evaluate()` stores results and features before marking the run finished; the drift monitor checks
237
+ a run again when its results change, and one suite's error no longer stops the others.
238
+ - Scheduled commands that leave background processes no longer block completion or defeat the
239
+ timeout; leftover processes are stopped.
240
+ - Schedule runs and their output are pruned (`keep_runs`, `keep_days`, `keep_output`).
241
+ - Behaviour drift pairs items with their own baseline means (Wilcoxon signed-rank) instead of
242
+ pooling repeated baseline runs as independent samples.
243
+ - Linear-time query parsing (filters over 4 KiB are refused), no step overflow after the largest
244
+ step, alert acknowledgements kept during webhook delivery, impossible cron expressions rejected,
245
+ interrupted schedule runs marked as such.
246
+ - Runs table: each loss sparkline spans the run's own steps (still on one scale per experiment), so
247
+ a live run is no longer squeezed into the first pixels by a long finished run; a training run
248
+ without the metric says "No points yet" or "Not logged" instead of loading forever.
249
+ - Drift: a suite whose eval runs recorded no scores explains why instead of showing an endless
250
+ placeholder, and the demo's evaluations are real `evaluate()` runs that form one scored suite.
251
+ - Web app polish: the sidebar footer keeps pre-release versions whole, header facts no longer split
252
+ around inline code, names and states no longer break at hyphens in narrow tables, figure labels
253
+ and notes wrap instead of truncating on phones, and the trace filters fit two per row on phones.
254
+ - Dark mode in the docs site: the code-block copy button and the breadcrumb separator had too
255
+ little contrast, and some Material colours did not follow the brand tokens.
256
+ - The release workflow finds the tag of a release pull request merged with a merge commit
257
+ (release-plz tags the pull request's own commit, the merge's second parent).
258
+ - A schedule run that had just finished could be read without its captured output: the output
259
+ is now stored before the run's final status.
260
+ - `vigily.__version__` reports the PEP 440 form (`0.1.0b1`) in editable installs too, as built
261
+ wheels do.
@@ -0,0 +1,65 @@
1
+ [workspace]
2
+ resolver = "2"
3
+ members = [
4
+ "crates/vigily-api",
5
+ "crates/vigily-core",
6
+ "crates/vigily-server",
7
+ "crates/vigily-ui",
8
+ "crates/vigily-cli",
9
+ ]
10
+
11
+ [workspace.package]
12
+ version = "0.1.0-beta.1"
13
+ edition = "2021"
14
+ rust-version = "1.94.1"
15
+ license = "MIT"
16
+ authors = ["Saiful Islam"]
17
+ repository = "https://github.com/sayef/vigily"
18
+ homepage = "https://github.com/sayef/vigily"
19
+
20
+ [workspace.dependencies]
21
+ vigily-api = { path = "crates/vigily-api", version = "0.1.0-beta.1" }
22
+ vigily-core = { path = "crates/vigily-core", version = "0.1.0-beta.1" }
23
+ vigily-server = { path = "crates/vigily-server", version = "0.1.0-beta.1" }
24
+ vigily-ui = { path = "crates/vigily-ui", version = "0.1.0-beta.1" }
25
+ # Storage: kv0 (Lance tables on local disk or object storage). A git dependency on `main`, pinned
26
+ # to an exact commit by Cargo.lock; `cargo update -p kv0` moves it. To build against a local kv0
27
+ # checkout instead, see "Storage dependency" in CONTRIBUTING.md.
28
+ # `version` is what crates.io uses once kv0 is published; until then Cargo builds from git. A
29
+ # pre-release needs its exact version here: "0.1" does not match 0.1.0-beta.1.
30
+ kv0 = { git = "https://github.com/sayef/kv0", branch = "main", version = "0.1.0-beta.2", default-features = false }
31
+ # Only in tests, to write a shared records table the way earlier builds did; the same versions
32
+ # kv0 uses, so nothing extra is compiled.
33
+ lance = { version = "14.0.2", default-features = false }
34
+ arrow-array = "58"
35
+ anyhow = "1"
36
+ async-trait = "0.1"
37
+ axum = { version = "0.8", default-features = false, features = ["http1", "json", "query", "tokio", "matched-path"] }
38
+ clap = { version = "4", features = ["derive", "env"] }
39
+ parking_lot = "0.12"
40
+ serde = { version = "1", features = ["derive"] }
41
+ serde_json = { version = "1", features = ["float_roundtrip", "preserve_order"] }
42
+ tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "time", "net", "signal", "process", "io-util"] }
43
+ tower = { version = "0.5", default-features = false, features = ["util"] }
44
+ tower-http = { version = "0.6", default-features = false, features = ["compression-gzip", "decompression-gzip"] }
45
+ tracing = "0.1"
46
+ tracing-subscriber = { version = "0.3", default-features = false, features = ["std", "fmt", "env-filter", "ansi"] }
47
+ tempfile = "3"
48
+ sha2 = "0.10"
49
+ chrono = { version = "0.4", default-features = false, features = ["clock", "std"] }
50
+ chrono-tz = "0.10"
51
+ croner = "2"
52
+ toml = "0.8"
53
+ reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] }
54
+
55
+ [profile.release]
56
+ lto = "thin"
57
+ codegen-units = 1
58
+ strip = true
59
+
60
+ # Keep target/ small: no debug info for third-party crates, line tables for ours.
61
+ [profile.dev]
62
+ debug = "line-tables-only"
63
+
64
+ [profile.dev.package."*"]
65
+ debug = false
vigily-0.1.0b1/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Saiful Islam
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,245 @@
1
+ Metadata-Version: 2.5
2
+ Name: vigily
3
+ Version: 0.1.0-beta.1
4
+ Summary: An experiment tracker that keeps watch: log runs, metrics and evaluations from Python
5
+ Project-URL: Homepage, https://github.com/sayef/vigily
6
+ Project-URL: Issues, https://github.com/sayef/vigily/issues
7
+ Project-URL: Changelog, https://github.com/sayef/vigily/blob/main/CHANGELOG.md
8
+ Author: Saiful Islam
9
+ License-Expression: MIT
10
+ License-File: LICENSE
11
+ Keywords: experiment-tracking,hyperparameters,machine-learning,metrics,mlops
12
+ Classifier: Development Status :: 3 - Alpha
13
+ Classifier: Intended Audience :: Science/Research
14
+ Classifier: Operating System :: MacOS
15
+ Classifier: Operating System :: POSIX :: Linux
16
+ Classifier: Programming Language :: Python :: 3 :: Only
17
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
18
+ Classifier: Typing :: Typed
19
+ Requires-Python: >=3.11
20
+ Provides-Extra: local
21
+ Requires-Dist: vigily-engine<0.2,>=0.1.0b1; extra == 'local'
22
+ Description-Content-Type: text/markdown
23
+
24
+ <p align="center">
25
+ <picture>
26
+ <source media="(prefers-color-scheme: dark)" srcset="assets/banner-dark.png">
27
+ <img alt="vigily: keeps watch over your runs, evals and drift" src="assets/banner-light.png" width="620">
28
+ </picture>
29
+ </p>
30
+
31
+ <p align="center">
32
+ <b>An experiment tracker that keeps watch.</b><br>
33
+ Training runs, evaluations, scheduled evals and drift detection in one self-hosted engine.
34
+ </p>
35
+
36
+ <p align="center">
37
+ <a href="https://vigily.pages.dev/">Docs</a> ·
38
+ <a href="https://vigily.pages.dev/getting-started/">Getting started</a> ·
39
+ <a href="https://vigily.pages.dev/concepts/">Concepts</a> ·
40
+ <a href="https://vigily.pages.dev/guides/python-sdk/">Python SDK</a> ·
41
+ <a href="https://vigily.pages.dev/guides/drift/">Drift methods</a> ·
42
+ <a href="CHANGELOG.md">Changelog</a> ·
43
+ <a href="CONTRIBUTING.md">Contributing</a>
44
+ </p>
45
+
46
+ <p align="center">
47
+ <picture>
48
+ <source media="(prefers-color-scheme: dark)" srcset="docs/assets/screenshots/runs-desktop-dark.png">
49
+ <img alt="The runs table: runs with their status, experiment, parameters and metric values" src="docs/assets/screenshots/runs-desktop-light.png" width="860">
50
+ </picture>
51
+ </p>
52
+
53
+ > vigily is in beta: the API may still change before 0.1.0.
54
+
55
+ ## Why vigily
56
+
57
+ Most teams stitch together a tracker for training runs (MLflow, W&B), a tool for LLM and model
58
+ evaluations (Langfuse) and a monitor for drift (Evidently). vigily is one engine that does the three
59
+ jobs together, on storage you already have:
60
+
61
+ - **One engine, no services.** `vigily-engine` (or `vigily up`) serves the web app, an HTTP API, the
62
+ scheduler and the drift monitor, and is the only process that writes storage. No database
63
+ server, no queue, no Node at runtime; the Python SDK is a 60 kB pure-Python package.
64
+ - **Evaluations that keep running.** Versioned, content-addressed datasets; `evaluate()` with built-in
65
+ and custom evaluators; schedules from a config file run them nightly in your time zone.
66
+ - **Drift you can trust.** Paired tests on matched items with effect sizes and the items that
67
+ regressed, against a baseline that never absorbs a regression; input drift by PSI and
68
+ Jensen–Shannon divergence. Every verdict shows its evidence.
69
+ - **LLM traces with tokens and cost.** `vigily.trace(...)` records nested spans (also over OTLP/HTTP
70
+ JSON), prices model calls from your own price table, and stores prompts truncated, in full or not
71
+ at all, per project.
72
+ - **Storage on [kv0](https://github.com/sayef/kv0).** Lance tables in a directory or on S3, GCS or
73
+ Azure; many training jobs can write at once.
74
+ - **Fast with many runs and long curves.** Thousands of runs filter and sort in memory; charts never
75
+ ask for more points than they have pixels.
76
+
77
+ ## Quickstart
78
+
79
+ ### Install with pip
80
+
81
+ Releases publish two packages to PyPI: `vigily`, the Python SDK (pure Python, no dependencies),
82
+ and `vigily-engine`, the engine that stores runs and serves the web app, as a platform wheel for
83
+ Linux and macOS. `vigily[local]` installs both:
84
+
85
+ ```bash
86
+ pip install 'vigily[local]' # the SDK and the engine for this machine
87
+ vigily-engine # serves ./.vigily; open the link it prints (Ctrl-C to stop)
88
+ python train.py # in another shell: logs to that engine
89
+ ```
90
+
91
+ Without a running engine, the SDK starts one in the background for the repository (`.vigily`, or
92
+ `VIGILY_REPO`), which stops after 30 idle minutes. To log to an engine elsewhere, such as a
93
+ [team server](docs/guides/team-server.md), install only the SDK and point it there with a token
94
+ that has the `read` and `ingest` scopes:
95
+
96
+ ```bash
97
+ pip install vigily
98
+ VIGILY_URL=https://vigily.example.com VIGILY_TOKEN=vgy_... python train.py
99
+ ```
100
+
101
+ Until 0.1.0 every release is a beta; pip installs the newest one when no stable release exists.
102
+
103
+ ### Install the command line
104
+
105
+ Each GitHub release also carries prebuilt binaries for Linux (x86_64 and arm64, glibc 2.31 or
106
+ newer) and macOS (Apple silicon and Intel): `vigily`, the command line, and `vigily-engine`. The
107
+ installer picks the right archive, checks its SHA-256 against the release's checksum file and puts
108
+ both in `~/.local/bin`, without sudo:
109
+
110
+ ```bash
111
+ curl -LsSf https://raw.githubusercontent.com/sayef/vigily/main/scripts/install.sh | sh
112
+ ```
113
+
114
+ To read the script before running it, download it first:
115
+
116
+ ```bash
117
+ curl -LsSfO https://raw.githubusercontent.com/sayef/vigily/main/scripts/install.sh
118
+ sh install.sh
119
+ ```
120
+
121
+ `VIGILY_VERSION=v0.1.0-beta.1` installs a given release and `VIGILY_INSTALL_DIR` changes the
122
+ directory. Releases are published as pre-releases, which `latest` does not serve: set
123
+ `VIGILY_VERSION` (or `VIGILY_IMAGE` for Docker) to a beta tag.
124
+
125
+ `vigily up` runs the engine in the foreground and prints a link that signs the browser in;
126
+ `vigily open` opens the web app of an engine that is already running, `vigily runs` lists runs,
127
+ and `vigily tokens create` makes tokens for scripts and people on a shared engine (see the
128
+ [command line guide](docs/guides/cli.md)).
129
+
130
+ ### Build from source
131
+
132
+ Build with Rust, Node 24, [uv](https://docs.astral.sh/uv/) and `protoc`:
133
+
134
+ ```bash
135
+ git clone https://github.com/sayef/vigily && cd vigily
136
+ (cd crates/vigily-ui && npm ci && npm run build) # the engine embeds the web app
137
+ cargo build --release -p vigily-cli -p vigily-server # vigily and vigily-engine
138
+ uv sync # the Python SDK, editable
139
+ export PATH="$PWD/target/release:$PATH" # the SDK starts vigily-engine from PATH
140
+
141
+ uv run --no-sync python examples/demo.py # synthetic training runs
142
+ uv run --no-sync python examples/eval_demo.py # 30 synthetic nights of evaluations
143
+ vigily stop && vigily up --config examples/vigily.toml # open the link it prints
144
+ ```
145
+
146
+ ### Run with Docker
147
+
148
+ Each release publishes a multi-platform image (amd64 and arm64) as `ghcr.io/sayef/vigily`, tagged
149
+ `vX.Y.Z`, `X.Y` and `latest`. `compose.yaml` runs it and serves the web app on
150
+ <http://localhost:7600>, with the repository in a named volume. The container runs in server mode
151
+ and needs an admin token, which you also sign in with:
152
+
153
+ ```bash
154
+ export VIGILY_ADMIN_TOKEN=$(openssl rand -hex 32)
155
+ docker compose up -d
156
+ ```
157
+
158
+ `compose.dev.yaml` builds the image from a checkout instead:
159
+
160
+ ```bash
161
+ docker compose -f compose.yaml -f compose.dev.yaml up -d --build
162
+ ```
163
+
164
+ Scripts log to it over HTTP with a token that has the `read` and `ingest` scopes:
165
+
166
+ ```bash
167
+ docker compose exec vigily sh -c \
168
+ 'vigily --url http://127.0.0.1:7600 --token "$VIGILY_ADMIN_TOKEN" tokens create train --scope read --scope ingest'
169
+ VIGILY_URL=http://localhost:7600 VIGILY_TOKEN=vgy_... python train.py
170
+ ```
171
+
172
+ [Getting started](docs/getting-started.md#run-with-docker) has the details, and
173
+ [Running a team server](docs/guides/team-server.md) covers tokens, TLS and backups.
174
+
175
+ Log a run:
176
+
177
+ ```python
178
+ from vigily import Run
179
+
180
+ with Run(experiment="cifar10", name="resnet18-lr3e-3") as run:
181
+ run["hparams"] = {"lr": 3e-3, "batch_size": 128}
182
+ for step, loss in train():
183
+ run.track(loss, name="loss", step=step, context={"subset": "train"})
184
+ ```
185
+
186
+ Evaluate a model on a versioned dataset:
187
+
188
+ ```python
189
+ from vigily import Dataset, evaluate, evaluators as ev
190
+
191
+ evaluate(
192
+ Dataset("support-intents"),
193
+ target=classify,
194
+ evaluators=[ev.exact_match(), ev.custom("judge", my_judge)],
195
+ target_ref={"model": "intent-clf-v2"},
196
+ experiment="support-intents-nightly",
197
+ )
198
+ ```
199
+
200
+ ## Screenshots
201
+
202
+ <table>
203
+ <tr>
204
+ <td><img src="docs/assets/screenshots/metrics-desktop-light.png" alt="Loss, accuracy and learning rate for sixteen runs, grouped by optimizer"></td>
205
+ <td><img src="docs/assets/screenshots/metrics-aggregate-desktop-dark.png" alt="Validation curves as a mean and min-max band per learning rate"></td>
206
+ </tr>
207
+ <tr>
208
+ <td><img src="docs/assets/screenshots/eval-items-desktop-light.png" alt="Per-item results of an evaluation run with pass and fail verdicts"></td>
209
+ <td><img src="docs/assets/screenshots/compare-desktop-dark.png" alt="Two evaluation runs compared item by item, regressions first"></td>
210
+ </tr>
211
+ <tr>
212
+ <td><img src="docs/assets/screenshots/drift-desktop-light.png" alt="Drift suites with their verdicts against the baseline"></td>
213
+ <td><img src="docs/assets/screenshots/alerts-desktop-dark.png" alt="Alerts for drift and a failed schedule"></td>
214
+ </tr>
215
+ </table>
216
+
217
+ <p align="center">
218
+ <img src="docs/assets/screenshots/runs-phone-light.png" width="260" alt="The runs list on a phone">
219
+ <img src="docs/assets/screenshots/metrics-phone-dark.png" width="260" alt="Charts on a phone in the dark theme">
220
+ </p>
221
+
222
+ ## Documentation
223
+
224
+ The documentation is a mkdocs-material site in [`docs/`](docs); build it with
225
+ `uvx --with mkdocs-material mkdocs serve`.
226
+
227
+ - [Getting started](docs/getting-started.md) and [concepts](docs/concepts.md)
228
+ - Guides: [Python SDK](docs/guides/python-sdk.md), [command line](docs/guides/cli.md),
229
+ [configuration](docs/guides/configuration.md), [scheduled evaluations and security](docs/guides/scheduled-evals.md),
230
+ [drift methods](docs/guides/drift.md), [LLM tracing](docs/guides/tracing.md), [self-hosting and storage](docs/guides/storage.md),
231
+ [the audit trail](docs/guides/audit-trail.md)
232
+ - Design: [brand](brand/DESIGN.md), [architecture](docs/architecture.md), [decisions](DESIGN.md)
233
+ - [FAQ](docs/faq.md)
234
+
235
+ ## Status
236
+
237
+ vigily is new (0.1, unreleased). It runs end to end on the demo and on 2 000 synthetic runs with 4.8
238
+ million points (start-up 0.2 s, every request in about 0.1 s or less). [TODO.md](TODO.md) lists what comes next,
239
+ and [docs/kv0-requests.md](docs/kv0-requests.md) what it needs from its storage layer.
240
+
241
+ Drift verdicts and evaluation scores are evidence for people to review, not automated decisions.
242
+
243
+ ## Licence
244
+
245
+ MIT. See [LICENSE](LICENSE).