vigily 0.1.0b1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- vigily-0.1.0b1/.gitignore +16 -0
- vigily-0.1.0b1/CHANGELOG.md +261 -0
- vigily-0.1.0b1/Cargo.toml +65 -0
- vigily-0.1.0b1/LICENSE +21 -0
- vigily-0.1.0b1/PKG-INFO +245 -0
- vigily-0.1.0b1/README.md +222 -0
- vigily-0.1.0b1/crates/vigily-api/Cargo.toml +25 -0
- vigily-0.1.0b1/crates/vigily-cli/Cargo.toml +37 -0
- vigily-0.1.0b1/crates/vigily-core/Cargo.toml +37 -0
- vigily-0.1.0b1/crates/vigily-server/Cargo.toml +64 -0
- vigily-0.1.0b1/crates/vigily-server/LICENSE +1 -0
- vigily-0.1.0b1/crates/vigily-server/README.md +14 -0
- vigily-0.1.0b1/crates/vigily-ui/Cargo.toml +16 -0
- vigily-0.1.0b1/pyproject.toml +67 -0
- vigily-0.1.0b1/python/tests/conftest.py +149 -0
- vigily-0.1.0b1/python/tests/test_autostart.py +111 -0
- vigily-0.1.0b1/python/tests/test_evals.py +175 -0
- vigily-0.1.0b1/python/tests/test_sdk.py +187 -0
- vigily-0.1.0b1/python/tests/test_sender.py +322 -0
- vigily-0.1.0b1/python/tests/test_tracing.py +365 -0
- vigily-0.1.0b1/python/tests/test_wheel.py +47 -0
- vigily-0.1.0b1/python/vigily/__init__.py +693 -0
- vigily-0.1.0b1/python/vigily/_config.py +141 -0
- vigily-0.1.0b1/python/vigily/_engine.py +245 -0
- vigily-0.1.0b1/python/vigily/_http.py +243 -0
- vigily-0.1.0b1/python/vigily/_ids.py +43 -0
- vigily-0.1.0b1/python/vigily/_sender.py +1252 -0
- vigily-0.1.0b1/python/vigily/_spool.py +186 -0
- vigily-0.1.0b1/python/vigily/datasets.py +226 -0
- vigily-0.1.0b1/python/vigily/evaluation.py +234 -0
- vigily-0.1.0b1/python/vigily/evaluators.py +242 -0
- vigily-0.1.0b1/python/vigily/py.typed +0 -0
- vigily-0.1.0b1/python/vigily/tracing.py +709 -0
|
@@ -0,0 +1,261 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to vigily are documented here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
|
|
5
|
+
[Semantic Versioning](https://semver.org/).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.1.0-beta.1](https://github.com/sayef/vigily/releases/tag/v0.1.0-beta.1) - 2026-11-15
|
|
10
|
+
|
|
11
|
+
The first published release, as betas (0.1.0-beta.N) before 0.1.0. Earlier builds were never
|
|
12
|
+
released; Changed, Deprecated and Removed describe what differs from them, mainly that the Python
|
|
13
|
+
SDK no longer writes storage in the training process but logs to the engine over HTTP (see
|
|
14
|
+
docs/guides/whats-new.md).
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
|
|
18
|
+
- `vigily-core`: runs, parameters and metric series behind a storage trait, with a kv0 and an
|
|
19
|
+
in-memory backend; an incrementally refreshed catalog with per-series statistics; a run query
|
|
20
|
+
language; min/max-envelope downsampling onto a shared x axis; group aggregation with interpolation.
|
|
21
|
+
- `vigily-server`: a JSON API (meta, runs, run detail and changes, chart data) and the embedded web
|
|
22
|
+
app, answering on localhost only by default.
|
|
23
|
+
- `vigily` command: `up`, `runs` and `optimize`.
|
|
24
|
+
- Python SDK: `vigily.Run` (parameters, `track` with step, epoch, context and timestamp, tags,
|
|
25
|
+
notes, `finish`, context manager, finish at exit) and `vigily.Repo` for reading back.
|
|
26
|
+
- Web app: runs table, metric explorer, run pages, experiments, command palette, keyboard shortcuts,
|
|
27
|
+
light and dark themes.
|
|
28
|
+
- Run `kind` (`training`, `eval`, ...) and `target` reference, filterable as `kind:` and `target.<key>`.
|
|
29
|
+
- `vigily up` optimizes storage every 15 minutes in the background (`--no-optimize`).
|
|
30
|
+
- Runs whose process stopped (no heartbeat for 10 minutes) show as "No heartbeat" and can be marked
|
|
31
|
+
aborted; runs that are not running can be deleted, with an undo window.
|
|
32
|
+
- `examples/demo.py` with 43 synthetic runs, and `bench/scale.py` and `bench/scale.sh` for scale tests.
|
|
33
|
+
|
|
34
|
+
- `vigily-engine`, the server as its own binary (`vigily up` runs it in the foreground), with
|
|
35
|
+
`--state-dir`, `--detach`, `--idle-timeout` and `--check-config`. One engine serves a
|
|
36
|
+
repository on a machine: it holds `engines/<key>.lock` in the state directory and writes its
|
|
37
|
+
address, version and local token to `engines/<key>.json` (0600) while it runs; a second engine
|
|
38
|
+
on the same repository exits with code 3 and names the first one's address. An engine started
|
|
39
|
+
with `--detach` logs to `engines/<key>.log`, takes a free port when its own is taken and stops
|
|
40
|
+
after 30 idle minutes.
|
|
41
|
+
- `/api/v1` routes for clients other than the web app: the runs list pages with `limit` and
|
|
42
|
+
`offset`, answers `matched` and takes `fields=summary`; a run's parameters, series and the points
|
|
43
|
+
of one series (with a time window; NaN and infinities as strings); dataset versions and a
|
|
44
|
+
version's items as NDJSON; an eval run's results; `POST admin/purge` (with
|
|
45
|
+
`legacy_records_present`, and a 409 `tag_blocks_purge` when a storage tag keeps deleted data) and
|
|
46
|
+
`POST admin/optimize`; `POST traces/delete` for several traces, optionally erasing them; `POST
|
|
47
|
+
schedules/{name}/run`, which answers 202 with the run's id at once; and `GET tracing/policy`.
|
|
48
|
+
Admin routes accept only JSON from the server's own origin, like span ingest.
|
|
49
|
+
- A write path for clients over HTTP: `POST /api/v1/runs` creates a run with an id the client
|
|
50
|
+
chooses (creating it again answers the existing run), `POST runs/{id}/resume`, `finish`,
|
|
51
|
+
`results` and `features`, `POST datasets/{name}/versions` (with a required `Idempotency-Key`,
|
|
52
|
+
remembered for 24 hours), `POST flush`, and `POST ingest`: ordered batches of operations with
|
|
53
|
+
points in columns, as JSON or NDJSON, optionally gzip-compressed. A replayed batch is answered
|
|
54
|
+
with its first report; a batch resent after an engine restart stores nothing twice because
|
|
55
|
+
clients assign steps. A finished run's results and feature samples are always in storage before
|
|
56
|
+
its status changes. Write bodies are limited to 32 MiB, ingest in flight to 256 MiB (then 503
|
|
57
|
+
with `Retry-After`), and a write to a deleted run answers `run_deleted`. Any write counts as the
|
|
58
|
+
run's heartbeat (at most one stored every two minutes), and new data shows up without waiting
|
|
59
|
+
for the next catalog refresh.
|
|
60
|
+
- Sign-in. `vigily up` runs in local mode on a loopback address: a random token per repository,
|
|
61
|
+
kept in a private file, opens the API, and the link it prints (`/?token=...`) stores it in an
|
|
62
|
+
HttpOnly, SameSite=Strict cookie. On any other address (and in Docker) it runs in server mode and
|
|
63
|
+
refuses to start without `VIGILY_ADMIN_TOKEN` (or `VIGILY_ADMIN_TOKEN_FILE`) unless given
|
|
64
|
+
`--insecure-no-auth`. Admins create tokens with the scopes `read`, `ingest` and `admin`
|
|
65
|
+
(`/api/v1/auth/tokens`), stored as SHA-256 hashes; the web app shows a sign-in page when needed.
|
|
66
|
+
Cookie-authenticated writes must come from vigily's own origin. Scheduled commands get
|
|
67
|
+
`VIGILY_URL` and a short-lived token; labels record who entered them. The SDK's span exporter
|
|
68
|
+
sends `VIGILY_TOKEN` (or `vigily.tracing.configure(token=...)`).
|
|
69
|
+
- A typed Rust client for `/api/v1` in `vigily-api` (feature `client`), one method per route, with
|
|
70
|
+
structured errors and gzip-compressed large request bodies, and `vigily_api::local::resolve`,
|
|
71
|
+
which finds a repository's engine (`VIGILY_URL`, then its state file) or starts one in the
|
|
72
|
+
background. `GET /api/v1/schedules` also answers each schedule's `retry_delay`.
|
|
73
|
+
- New commands: `vigily status`, `stop`, `open`, `env`, `tokens list|create|revoke` and
|
|
74
|
+
`token rotate` (`POST /api/v1/auth/local-token/rotate`, admin), and global `--url` and `--token`
|
|
75
|
+
options (`VIGILY_URL`, `VIGILY_TOKEN`) to use a team server.
|
|
76
|
+
- `vigily.configure(url, token, project, offline)`, `vigily.flush(timeout)`, `Repo(url=...)`,
|
|
77
|
+
`Repo.results(run_id)` and `vigily.tracing.export(spans)` in the Python SDK; settings also from
|
|
78
|
+
`VIGILY_URL`, `VIGILY_TOKEN`, `VIGILY_PROJECT`, `VIGILY_OFFLINE`, `VIGILY_EXIT_TIMEOUT`,
|
|
79
|
+
`VIGILY_SPOOL_AFTER`, `VIGILY_SPOOL_MAX_MB` and `VIGILY_BUFFER_MB`.
|
|
80
|
+
- An offline spool: batches the SDK cannot send (after two minutes without the engine, at exit, or
|
|
81
|
+
when the engine falls behind) go to gzip-compressed NDJSON files in `spool/<writer>/` under the
|
|
82
|
+
state directory (0600, written atomically), and are sent in order once the engine answers, by the
|
|
83
|
+
next process that reaches it, or with the new `vigily sync`. The spool has a size limit and never
|
|
84
|
+
deletes data.
|
|
85
|
+
- `vigily-engine` platform wheels (`pip install 'vigily[local]'`): the engine binary alone, built by
|
|
86
|
+
maturin (`bindings = "bin"`) for Linux x86_64 and arm64 (manylinux 2_28), Linux x86_64 with musl
|
|
87
|
+
and macOS arm64 and x86_64, stripped, versioned like the workspace; the release workflow builds
|
|
88
|
+
them (and, on pull requests that change how, only builds them) and has a `pypi-engine` job to
|
|
89
|
+
publish them with trusted publishing, from its own `pypi-engine` environment.
|
|
90
|
+
- `bench/ingest.py` measures ingest through the SDK into an engine (track latency, points per
|
|
91
|
+
second per process and engine-wide, engine CPU and memory, time until new points are readable),
|
|
92
|
+
and `bench/compare.sh` compares an engine built from another commit with this checkout's; the
|
|
93
|
+
Benchmarks workflow runs both.
|
|
94
|
+
|
|
95
|
+
- Datasets: content-addressed, immutable versions; `Dataset.create`, `from_jsonl`, `from_csv`, `add`.
|
|
96
|
+
- Evaluations: `evaluate()` with exact match, contains, regex, numeric, JSON schema and custom
|
|
97
|
+
evaluators (including LLM-as-judge as user code); per-item results, human labels, run comparison
|
|
98
|
+
and a JSON audit export.
|
|
99
|
+
- Scheduled evaluations from `vigily.toml`: cron with time zones, no overlap, missed-run policy,
|
|
100
|
+
retries, timeouts, captured output; `vigily schedules [run NAME]`.
|
|
101
|
+
- Drift: behaviour drift (Mann–Whitney U, bootstrap interval, per-item regressions) and data drift
|
|
102
|
+
(PSI, Kolmogorov–Smirnov, Jensen–Shannon, chi-squared) against rolling or pinned baselines.
|
|
103
|
+
- Alerts for drift and failed schedules, with an optional webhook.
|
|
104
|
+
- Web app: Datasets, Drift, Schedules and Alerts pages; Items and Audit tabs on eval runs; Compare.
|
|
105
|
+
- `examples/eval_demo.py`: 30 synthetic nights of two targets with injected drift.
|
|
106
|
+
- Evaluation records in one storage table per kind (results, labels, schedules, alerts, ...) and
|
|
107
|
+
one per dataset, so a dataset version's tag keeps only that dataset. Repositories with the
|
|
108
|
+
earlier shared records table are migrated once on open; the old table is kept for you to delete.
|
|
109
|
+
- `vigily runs purge [--force]`: erases deleted runs from storage and reports what it removed.
|
|
110
|
+
- `vigily up --optimize-every <duration>` and `--optimize-at-start`; the server returns memory to
|
|
111
|
+
the system after each background optimize.
|
|
112
|
+
|
|
113
|
+
- LLM tracing. `vigily.trace(...)` (a context manager, `async with`, and a decorator) records nested
|
|
114
|
+
spans through `contextvars`, with `span.set_usage(...)` and an opt-in `usage_from(response)` that
|
|
115
|
+
reads common client responses without importing them. Spans are exported in batches from a
|
|
116
|
+
background thread and sent on normal exit. The engine ingests them at
|
|
117
|
+
`POST /api/v1/traces/ingest` and accepts OpenTelemetry OTLP/HTTP JSON at `POST /v1/traces`.
|
|
118
|
+
- Cost from a price table in `vigily.toml` (`[tracing.prices]`, per million input, output and
|
|
119
|
+
cached input tokens, in a configured currency), computed when a span arrives. No prices are
|
|
120
|
+
built in.
|
|
121
|
+
- Payload modes for traces, global or per project: `full`, `truncated` (the default, 4 KiB per
|
|
122
|
+
field) or `metadata` (no inputs or outputs); a redaction hook (`vigily.tracing.set_redactor`).
|
|
123
|
+
Traces linked to a run are deleted with it and erased by `vigily runs purge`, which now
|
|
124
|
+
compacts first so deleted traces without a run are erased too; `vigily traces delete [--erase]`.
|
|
125
|
+
Until a trace's project is known the strictest configured policy applies; truncated spans are
|
|
126
|
+
capped at 64 KiB in total and 128 attributes.
|
|
127
|
+
- Purging compacts first inside the storage layer, so every caller erases trace rows moved under a
|
|
128
|
+
run; an interrupted link of a trace to its run is repaired on the next read or ingest; the trace
|
|
129
|
+
page says when a span's payload was limited because its project was not known yet.
|
|
130
|
+
- The ingest endpoints accept only `application/json` and refuse other origins, so a web page
|
|
131
|
+
cannot post spans to a local server.
|
|
132
|
+
- Traces in the web app: a Traces page under Track with filters (run, model, status, time),
|
|
133
|
+
latency, tokens and cost; a trace page with a span waterfall, the selected span's inputs, outputs
|
|
134
|
+
and attributes, and token and cost totals; links from runs and evaluation items.
|
|
135
|
+
- The demo generator logs synthetic support-assistant traces.
|
|
136
|
+
|
|
137
|
+
- A redesigned web app: warm paper neutrals and ink rules, a sidebar grouped Track / Evaluate /
|
|
138
|
+
Workspace with counts, a top bar with breadcrumbs and search, summary figures, dense tables with
|
|
139
|
+
loss sparklines, and a solid ink selection bar. Light and dark themes; a drawer on phones.
|
|
140
|
+
- Compare runs (`/metrics/compare?runs=...`): selected runs keep their letters A, B, C, D, with a
|
|
141
|
+
headline chart labelled at the line ends, a supporting chart and only the settings that differ.
|
|
142
|
+
- Drift suites show the score per run with its baseline band, a strip per score and the items that
|
|
143
|
+
flipped from pass to fail. A Settings page shows the repository and the browser's preferences.
|
|
144
|
+
- The documentation site in the same paper design: a masthead over an ink rule, a home page that
|
|
145
|
+
opens with the point and numbered figures, ruled sections, a margin column with the page contents,
|
|
146
|
+
and a Copy page button.
|
|
147
|
+
- A documentation site (mkdocs-material) and brand assets (mark, wordmark, banners, social
|
|
148
|
+
preview) built reproducibly by `assets/src/build.py`.
|
|
149
|
+
- Documentation: an HTTP API reference with the scope of every route, a guide to running a team
|
|
150
|
+
server (tokens, TLS through a reverse proxy, backups), and a guide to upgrading from the builds
|
|
151
|
+
where the SDK wrote storage itself.
|
|
152
|
+
|
|
153
|
+
### Changed
|
|
154
|
+
|
|
155
|
+
- The Python SDK is pure Python (one `py3-none-any` wheel of about 60 kB, no dependencies) and talks
|
|
156
|
+
to the engine over `/api/v1` instead of writing storage in the process; the pyo3 extension and
|
|
157
|
+
the `vigily-py` crate are gone, and the package moves to `python/vigily`. It finds the
|
|
158
|
+
repository's engine like the command line, or starts `vigily-engine --detach` (install hint when
|
|
159
|
+
it is missing). `Run()` never waits for the network: run ids (UUIDv7), steps and timestamps are
|
|
160
|
+
assigned in the process, and a background thread sends an ordered log of operations in batches
|
|
161
|
+
(every second, 20 000 points or 4 MiB; points in columns; gzip; `Idempotency-Key` per batch),
|
|
162
|
+
retried with exponential backoff and full jitter, honouring `Retry-After`. `track()` waits up to
|
|
163
|
+
five seconds when the engine falls behind, then the batches go to the spool; nothing is dropped
|
|
164
|
+
without a message. Forked children log through their own sender; at exit the SDK finishes the
|
|
165
|
+
runs it started and sends what is queued for up to 30 seconds. Spans share the sender; the
|
|
166
|
+
redactor and the engine's payload policy (fetched per project) apply before a span is queued or
|
|
167
|
+
spooled. `RunDeletedError` surfaces on the next call after the engine reported the deletion, and
|
|
168
|
+
`Run.finish()` no longer waits for storage (`vigily.flush()` and `Repo` reads do).
|
|
169
|
+
- `vigily` is now a client of the engine: every command but `up` goes through the HTTP API, and the
|
|
170
|
+
binary no longer contains storage code (`vigily-core` and kv0 are out of its dependency tree).
|
|
171
|
+
Without `--url` it uses the repository's local engine, starting one in the background when none
|
|
172
|
+
runs (not for a directory that does not exist). `vigily up` runs `vigily-engine` in the
|
|
173
|
+
foreground with the same options and output. `vigily schedules` lists the engine's schedules
|
|
174
|
+
(`--config` is gone; it is an option of `vigily up`), and `vigily schedules run` starts the run on
|
|
175
|
+
the engine and follows it through its retries, exiting 1 unless it succeeds. Release archives
|
|
176
|
+
and the Docker image contain both `vigily` and `vigily-engine`, and `scripts/install.sh` installs
|
|
177
|
+
both; the `s3`, `gcs` and `azure` features move to the `vigily-server` crate.
|
|
178
|
+
- The HTTP API moves to `/api/v1`, with `GET /api/v1/info` (version, API version, mode and
|
|
179
|
+
capabilities). Errors there are `{"error": {"code", "message", "position"?}}` with a stable code
|
|
180
|
+
(`not_found`, `invalid_argument`, `conflict`, ...). The old `/api/*` paths keep working as
|
|
181
|
+
deprecated aliases with a `Deprecation` header, a `Link` to the successor and the old flat error
|
|
182
|
+
body; they are removed before 0.1.0. OpenTelemetry's `POST /v1/traces` is unchanged. The web app
|
|
183
|
+
and the Python SDK's span exporter use `/api/v1`.
|
|
184
|
+
- Every API route except health, info and `auth/me` needs a token, including the deprecated
|
|
185
|
+
`/api/*` aliases and OTLP's `/v1/traces`. `docker compose up` needs `VIGILY_ADMIN_TOKEN`.
|
|
186
|
+
- The JSON types of the HTTP API (the run model, every route's answer, evaluation, drift and trace
|
|
187
|
+
types) move to a new crate, `vigily-api`, which depends on serde alone; `vigily-core` re-exports
|
|
188
|
+
them. Schedules and suites answer with typed structures instead of ad-hoc JSON. Golden JSON
|
|
189
|
+
files in `crates/vigily-api/tests/fixtures` pin the wire format; the web app sees the same JSON.
|
|
190
|
+
- The engine writes each ingested series batch with kv0's `Run::track_many` (one lock and one
|
|
191
|
+
log append per batch instead of per point): about five times the points per second with eight
|
|
192
|
+
SDK processes, at a tenth of the CPU time. After writes it commits buffered points (at most once
|
|
193
|
+
a second) before refreshing the run list, so new points are readable within about a second.
|
|
194
|
+
Finishing a run commits its points and status together, one storage commit per run instead of
|
|
195
|
+
two.
|
|
196
|
+
- The engine stores every explicit `heartbeat` operation (implicit ones stay at one per two
|
|
197
|
+
minutes), and `POST /api/v1/runs/{id}/resume` flushes the run first, so a run created a moment ago
|
|
198
|
+
can be resumed.
|
|
199
|
+
- The server stops cleanly on SIGTERM as well as Ctrl-C (`docker stop`, service managers): open
|
|
200
|
+
requests get 10 seconds, then open runs, storage and records are flushed.
|
|
201
|
+
- The local token moves to `engines/<key>.token` in the state directory, which defaults to
|
|
202
|
+
`~/Library/Caches/vigily` on macOS (`~/.cache/vigily` elsewhere); a token kept by an earlier
|
|
203
|
+
build under `tokens/` is moved there on the first start.
|
|
204
|
+
- The Docker image runs `vigily-engine --host 0.0.0.0 --mode server` (entry point `vigily-engine`)
|
|
205
|
+
and checks `/api/v1/health`. `compose.yaml` keeps the data in the `vigily` volume only: the
|
|
206
|
+
`VIGILY_DATA`, `VIGILY_UID`, `VIGILY_GID` and `VIGILY_SERVER_REPO` variables are gone, since
|
|
207
|
+
scripts now log over HTTP (`VIGILY_URL` and `VIGILY_TOKEN`) instead of writing to a shared
|
|
208
|
+
directory. It gives the engine 30 seconds to stop, and shows a `trainer` service logging from a
|
|
209
|
+
second container. New guide: running a team server (tokens, TLS, backups).
|
|
210
|
+
- `examples/demo.py` sends its traces over HTTP (`vigily.tracing.export`); their example prices
|
|
211
|
+
moved to `examples/vigily.toml`.
|
|
212
|
+
- Recursive replaces Atkinson Hyperlegible as vigily's typeface, in the app, the docs and the brand
|
|
213
|
+
assets: Sans Linear for text and Mono Linear for run names, values, keys and code, from one
|
|
214
|
+
self-hosted variable file (SIL OFL). Bold is 660 and headings track slightly tighter; the wordmark,
|
|
215
|
+
banners, social card and screenshots are regenerated.
|
|
216
|
+
|
|
217
|
+
### Deprecated
|
|
218
|
+
|
|
219
|
+
- The `/api/*` routes of earlier builds answer as aliases of their `/api/v1` successors, with
|
|
220
|
+
`Deprecation: true` and a `Link` header, and are removed before 0.1.0. `/v1/traces` (OTLP) stays.
|
|
221
|
+
- An SDK from before the engine/client split still writes storage directly next to a running engine,
|
|
222
|
+
which picks up its runs; upgrade it so that the engine is the only writer.
|
|
223
|
+
|
|
224
|
+
### Removed
|
|
225
|
+
|
|
226
|
+
- The pyo3 extension and the `vigily-py` crate: the SDK is pure Python (see Changed).
|
|
227
|
+
- `vigily schedules --config`: the configuration is an option of `vigily up` and `vigily-engine`.
|
|
228
|
+
- The `VIGILY_DATA`, `VIGILY_UID`, `VIGILY_GID` and `VIGILY_SERVER_REPO` variables of `compose.yaml`.
|
|
229
|
+
|
|
230
|
+
### Fixed
|
|
231
|
+
|
|
232
|
+
- The SDK recorded the branch name as `__system__.git.commit`; it now records the commit hash
|
|
233
|
+
(and the branch as `git.branch`).
|
|
234
|
+
- Web app edits (name, tags, notes, archive) on a running run are no longer reverted when the SDK
|
|
235
|
+
finishes it: both sides re-read the run and change only their own fields.
|
|
236
|
+
- `evaluate()` stores results and features before marking the run finished; the drift monitor checks
|
|
237
|
+
a run again when its results change, and one suite's error no longer stops the others.
|
|
238
|
+
- Scheduled commands that leave background processes no longer block completion or defeat the
|
|
239
|
+
timeout; leftover processes are stopped.
|
|
240
|
+
- Schedule runs and their output are pruned (`keep_runs`, `keep_days`, `keep_output`).
|
|
241
|
+
- Behaviour drift pairs items with their own baseline means (Wilcoxon signed-rank) instead of
|
|
242
|
+
pooling repeated baseline runs as independent samples.
|
|
243
|
+
- Linear-time query parsing (filters over 4 KiB are refused), no step overflow after the largest
|
|
244
|
+
step, alert acknowledgements kept during webhook delivery, impossible cron expressions rejected,
|
|
245
|
+
interrupted schedule runs marked as such.
|
|
246
|
+
- Runs table: each loss sparkline spans the run's own steps (still on one scale per experiment), so
|
|
247
|
+
a live run is no longer squeezed into the first pixels by a long finished run; a training run
|
|
248
|
+
without the metric says "No points yet" or "Not logged" instead of loading forever.
|
|
249
|
+
- Drift: a suite whose eval runs recorded no scores explains why instead of showing an endless
|
|
250
|
+
placeholder, and the demo's evaluations are real `evaluate()` runs that form one scored suite.
|
|
251
|
+
- Web app polish: the sidebar footer keeps pre-release versions whole, header facts no longer split
|
|
252
|
+
around inline code, names and states no longer break at hyphens in narrow tables, figure labels
|
|
253
|
+
and notes wrap instead of truncating on phones, and the trace filters fit two per row on phones.
|
|
254
|
+
- Dark mode in the docs site: the code-block copy button and the breadcrumb separator had too
|
|
255
|
+
little contrast, and some Material colours did not follow the brand tokens.
|
|
256
|
+
- The release workflow finds the tag of a release pull request merged with a merge commit
|
|
257
|
+
(release-plz tags the pull request's own commit, the merge's second parent).
|
|
258
|
+
- A schedule run that had just finished could be read without its captured output: the output
|
|
259
|
+
is now stored before the run's final status.
|
|
260
|
+
- `vigily.__version__` reports the PEP 440 form (`0.1.0b1`) in editable installs too, as built
|
|
261
|
+
wheels do.
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
[workspace]
|
|
2
|
+
resolver = "2"
|
|
3
|
+
members = [
|
|
4
|
+
"crates/vigily-api",
|
|
5
|
+
"crates/vigily-core",
|
|
6
|
+
"crates/vigily-server",
|
|
7
|
+
"crates/vigily-ui",
|
|
8
|
+
"crates/vigily-cli",
|
|
9
|
+
]
|
|
10
|
+
|
|
11
|
+
[workspace.package]
|
|
12
|
+
version = "0.1.0-beta.1"
|
|
13
|
+
edition = "2021"
|
|
14
|
+
rust-version = "1.94.1"
|
|
15
|
+
license = "MIT"
|
|
16
|
+
authors = ["Saiful Islam"]
|
|
17
|
+
repository = "https://github.com/sayef/vigily"
|
|
18
|
+
homepage = "https://github.com/sayef/vigily"
|
|
19
|
+
|
|
20
|
+
[workspace.dependencies]
|
|
21
|
+
vigily-api = { path = "crates/vigily-api", version = "0.1.0-beta.1" }
|
|
22
|
+
vigily-core = { path = "crates/vigily-core", version = "0.1.0-beta.1" }
|
|
23
|
+
vigily-server = { path = "crates/vigily-server", version = "0.1.0-beta.1" }
|
|
24
|
+
vigily-ui = { path = "crates/vigily-ui", version = "0.1.0-beta.1" }
|
|
25
|
+
# Storage: kv0 (Lance tables on local disk or object storage). A git dependency on `main`, pinned
|
|
26
|
+
# to an exact commit by Cargo.lock; `cargo update -p kv0` moves it. To build against a local kv0
|
|
27
|
+
# checkout instead, see "Storage dependency" in CONTRIBUTING.md.
|
|
28
|
+
# `version` is what crates.io uses once kv0 is published; until then Cargo builds from git. A
|
|
29
|
+
# pre-release needs its exact version here: "0.1" does not match 0.1.0-beta.1.
|
|
30
|
+
kv0 = { git = "https://github.com/sayef/kv0", branch = "main", version = "0.1.0-beta.2", default-features = false }
|
|
31
|
+
# Only in tests, to write a shared records table the way earlier builds did; the same versions
|
|
32
|
+
# kv0 uses, so nothing extra is compiled.
|
|
33
|
+
lance = { version = "14.0.2", default-features = false }
|
|
34
|
+
arrow-array = "58"
|
|
35
|
+
anyhow = "1"
|
|
36
|
+
async-trait = "0.1"
|
|
37
|
+
axum = { version = "0.8", default-features = false, features = ["http1", "json", "query", "tokio", "matched-path"] }
|
|
38
|
+
clap = { version = "4", features = ["derive", "env"] }
|
|
39
|
+
parking_lot = "0.12"
|
|
40
|
+
serde = { version = "1", features = ["derive"] }
|
|
41
|
+
serde_json = { version = "1", features = ["float_roundtrip", "preserve_order"] }
|
|
42
|
+
tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "time", "net", "signal", "process", "io-util"] }
|
|
43
|
+
tower = { version = "0.5", default-features = false, features = ["util"] }
|
|
44
|
+
tower-http = { version = "0.6", default-features = false, features = ["compression-gzip", "decompression-gzip"] }
|
|
45
|
+
tracing = "0.1"
|
|
46
|
+
tracing-subscriber = { version = "0.3", default-features = false, features = ["std", "fmt", "env-filter", "ansi"] }
|
|
47
|
+
tempfile = "3"
|
|
48
|
+
sha2 = "0.10"
|
|
49
|
+
chrono = { version = "0.4", default-features = false, features = ["clock", "std"] }
|
|
50
|
+
chrono-tz = "0.10"
|
|
51
|
+
croner = "2"
|
|
52
|
+
toml = "0.8"
|
|
53
|
+
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] }
|
|
54
|
+
|
|
55
|
+
[profile.release]
|
|
56
|
+
lto = "thin"
|
|
57
|
+
codegen-units = 1
|
|
58
|
+
strip = true
|
|
59
|
+
|
|
60
|
+
# Keep target/ small: no debug info for third-party crates, line tables for ours.
|
|
61
|
+
[profile.dev]
|
|
62
|
+
debug = "line-tables-only"
|
|
63
|
+
|
|
64
|
+
[profile.dev.package."*"]
|
|
65
|
+
debug = false
|
vigily-0.1.0b1/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Saiful Islam
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
vigily-0.1.0b1/PKG-INFO
ADDED
|
@@ -0,0 +1,245 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: vigily
|
|
3
|
+
Version: 0.1.0-beta.1
|
|
4
|
+
Summary: An experiment tracker that keeps watch: log runs, metrics and evaluations from Python
|
|
5
|
+
Project-URL: Homepage, https://github.com/sayef/vigily
|
|
6
|
+
Project-URL: Issues, https://github.com/sayef/vigily/issues
|
|
7
|
+
Project-URL: Changelog, https://github.com/sayef/vigily/blob/main/CHANGELOG.md
|
|
8
|
+
Author: Saiful Islam
|
|
9
|
+
License-Expression: MIT
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: experiment-tracking,hyperparameters,machine-learning,metrics,mlops
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Intended Audience :: Science/Research
|
|
14
|
+
Classifier: Operating System :: MacOS
|
|
15
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
16
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
17
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
18
|
+
Classifier: Typing :: Typed
|
|
19
|
+
Requires-Python: >=3.11
|
|
20
|
+
Provides-Extra: local
|
|
21
|
+
Requires-Dist: vigily-engine<0.2,>=0.1.0b1; extra == 'local'
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
|
|
24
|
+
<p align="center">
|
|
25
|
+
<picture>
|
|
26
|
+
<source media="(prefers-color-scheme: dark)" srcset="assets/banner-dark.png">
|
|
27
|
+
<img alt="vigily: keeps watch over your runs, evals and drift" src="assets/banner-light.png" width="620">
|
|
28
|
+
</picture>
|
|
29
|
+
</p>
|
|
30
|
+
|
|
31
|
+
<p align="center">
|
|
32
|
+
<b>An experiment tracker that keeps watch.</b><br>
|
|
33
|
+
Training runs, evaluations, scheduled evals and drift detection in one self-hosted engine.
|
|
34
|
+
</p>
|
|
35
|
+
|
|
36
|
+
<p align="center">
|
|
37
|
+
<a href="https://vigily.pages.dev/">Docs</a> ·
|
|
38
|
+
<a href="https://vigily.pages.dev/getting-started/">Getting started</a> ·
|
|
39
|
+
<a href="https://vigily.pages.dev/concepts/">Concepts</a> ·
|
|
40
|
+
<a href="https://vigily.pages.dev/guides/python-sdk/">Python SDK</a> ·
|
|
41
|
+
<a href="https://vigily.pages.dev/guides/drift/">Drift methods</a> ·
|
|
42
|
+
<a href="CHANGELOG.md">Changelog</a> ·
|
|
43
|
+
<a href="CONTRIBUTING.md">Contributing</a>
|
|
44
|
+
</p>
|
|
45
|
+
|
|
46
|
+
<p align="center">
|
|
47
|
+
<picture>
|
|
48
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/screenshots/runs-desktop-dark.png">
|
|
49
|
+
<img alt="The runs table: runs with their status, experiment, parameters and metric values" src="docs/assets/screenshots/runs-desktop-light.png" width="860">
|
|
50
|
+
</picture>
|
|
51
|
+
</p>
|
|
52
|
+
|
|
53
|
+
> vigily is in beta: the API may still change before 0.1.0.
|
|
54
|
+
|
|
55
|
+
## Why vigily
|
|
56
|
+
|
|
57
|
+
Most teams stitch together a tracker for training runs (MLflow, W&B), a tool for LLM and model
|
|
58
|
+
evaluations (Langfuse) and a monitor for drift (Evidently). vigily is one engine that does the three
|
|
59
|
+
jobs together, on storage you already have:
|
|
60
|
+
|
|
61
|
+
- **One engine, no services.** `vigily-engine` (or `vigily up`) serves the web app, an HTTP API, the
|
|
62
|
+
scheduler and the drift monitor, and is the only process that writes storage. No database
|
|
63
|
+
server, no queue, no Node at runtime; the Python SDK is a 60 kB pure-Python package.
|
|
64
|
+
- **Evaluations that keep running.** Versioned, content-addressed datasets; `evaluate()` with built-in
|
|
65
|
+
and custom evaluators; schedules from a config file run them nightly in your time zone.
|
|
66
|
+
- **Drift you can trust.** Paired tests on matched items with effect sizes and the items that
|
|
67
|
+
regressed, against a baseline that never absorbs a regression; input drift by PSI and
|
|
68
|
+
Jensen–Shannon divergence. Every verdict shows its evidence.
|
|
69
|
+
- **LLM traces with tokens and cost.** `vigily.trace(...)` records nested spans (also over OTLP/HTTP
|
|
70
|
+
JSON), prices model calls from your own price table, and stores prompts truncated, in full or not
|
|
71
|
+
at all, per project.
|
|
72
|
+
- **Storage on [kv0](https://github.com/sayef/kv0).** Lance tables in a directory or on S3, GCS or
|
|
73
|
+
Azure; many training jobs can write at once.
|
|
74
|
+
- **Fast with many runs and long curves.** Thousands of runs filter and sort in memory; charts never
|
|
75
|
+
ask for more points than they have pixels.
|
|
76
|
+
|
|
77
|
+
## Quickstart
|
|
78
|
+
|
|
79
|
+
### Install with pip
|
|
80
|
+
|
|
81
|
+
Releases publish two packages to PyPI: `vigily`, the Python SDK (pure Python, no dependencies),
|
|
82
|
+
and `vigily-engine`, the engine that stores runs and serves the web app, as a platform wheel for
|
|
83
|
+
Linux and macOS. `vigily[local]` installs both:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
pip install 'vigily[local]' # the SDK and the engine for this machine
|
|
87
|
+
vigily-engine # serves ./.vigily; open the link it prints (Ctrl-C to stop)
|
|
88
|
+
python train.py # in another shell: logs to that engine
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Without a running engine, the SDK starts one in the background for the repository (`.vigily`, or
|
|
92
|
+
`VIGILY_REPO`), which stops after 30 idle minutes. To log to an engine elsewhere, such as a
|
|
93
|
+
[team server](docs/guides/team-server.md), install only the SDK and point it there with a token
|
|
94
|
+
that has the `read` and `ingest` scopes:
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
pip install vigily
|
|
98
|
+
VIGILY_URL=https://vigily.example.com VIGILY_TOKEN=vgy_... python train.py
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
Until 0.1.0 every release is a beta; pip installs the newest one when no stable release exists.
|
|
102
|
+
|
|
103
|
+
### Install the command line
|
|
104
|
+
|
|
105
|
+
Each GitHub release also carries prebuilt binaries for Linux (x86_64 and arm64, glibc 2.31 or
|
|
106
|
+
newer) and macOS (Apple silicon and Intel): `vigily`, the command line, and `vigily-engine`. The
|
|
107
|
+
installer picks the right archive, checks its SHA-256 against the release's checksum file and puts
|
|
108
|
+
both in `~/.local/bin`, without sudo:
|
|
109
|
+
|
|
110
|
+
```bash
|
|
111
|
+
curl -LsSf https://raw.githubusercontent.com/sayef/vigily/main/scripts/install.sh | sh
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
To read the script before running it, download it first:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
curl -LsSfO https://raw.githubusercontent.com/sayef/vigily/main/scripts/install.sh
|
|
118
|
+
sh install.sh
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
`VIGILY_VERSION=v0.1.0-beta.1` installs a given release and `VIGILY_INSTALL_DIR` changes the
|
|
122
|
+
directory. Releases are published as pre-releases, which `latest` does not serve: set
|
|
123
|
+
`VIGILY_VERSION` (or `VIGILY_IMAGE` for Docker) to a beta tag.
|
|
124
|
+
|
|
125
|
+
`vigily up` runs the engine in the foreground and prints a link that signs the browser in;
|
|
126
|
+
`vigily open` opens the web app of an engine that is already running, `vigily runs` lists runs,
|
|
127
|
+
and `vigily tokens create` makes tokens for scripts and people on a shared engine (see the
|
|
128
|
+
[command line guide](docs/guides/cli.md)).
|
|
129
|
+
|
|
130
|
+
### Build from source
|
|
131
|
+
|
|
132
|
+
Build with Rust, Node 24, [uv](https://docs.astral.sh/uv/) and `protoc`:
|
|
133
|
+
|
|
134
|
+
```bash
|
|
135
|
+
git clone https://github.com/sayef/vigily && cd vigily
|
|
136
|
+
(cd crates/vigily-ui && npm ci && npm run build) # the engine embeds the web app
|
|
137
|
+
cargo build --release -p vigily-cli -p vigily-server # vigily and vigily-engine
|
|
138
|
+
uv sync # the Python SDK, editable
|
|
139
|
+
export PATH="$PWD/target/release:$PATH" # the SDK starts vigily-engine from PATH
|
|
140
|
+
|
|
141
|
+
uv run --no-sync python examples/demo.py # synthetic training runs
|
|
142
|
+
uv run --no-sync python examples/eval_demo.py # 30 synthetic nights of evaluations
|
|
143
|
+
vigily stop && vigily up --config examples/vigily.toml # open the link it prints
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
### Run with Docker
|
|
147
|
+
|
|
148
|
+
Each release publishes a multi-platform image (amd64 and arm64) as `ghcr.io/sayef/vigily`, tagged
|
|
149
|
+
`vX.Y.Z`, `X.Y` and `latest`. `compose.yaml` runs it and serves the web app on
|
|
150
|
+
<http://localhost:7600>, with the repository in a named volume. The container runs in server mode
|
|
151
|
+
and needs an admin token, which you also sign in with:
|
|
152
|
+
|
|
153
|
+
```bash
|
|
154
|
+
export VIGILY_ADMIN_TOKEN=$(openssl rand -hex 32)
|
|
155
|
+
docker compose up -d
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
`compose.dev.yaml` builds the image from a checkout instead:
|
|
159
|
+
|
|
160
|
+
```bash
|
|
161
|
+
docker compose -f compose.yaml -f compose.dev.yaml up -d --build
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
Scripts log to it over HTTP with a token that has the `read` and `ingest` scopes:
|
|
165
|
+
|
|
166
|
+
```bash
|
|
167
|
+
docker compose exec vigily sh -c \
|
|
168
|
+
'vigily --url http://127.0.0.1:7600 --token "$VIGILY_ADMIN_TOKEN" tokens create train --scope read --scope ingest'
|
|
169
|
+
VIGILY_URL=http://localhost:7600 VIGILY_TOKEN=vgy_... python train.py
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
[Getting started](docs/getting-started.md#run-with-docker) has the details, and
|
|
173
|
+
[Running a team server](docs/guides/team-server.md) covers tokens, TLS and backups.
|
|
174
|
+
|
|
175
|
+
Log a run:
|
|
176
|
+
|
|
177
|
+
```python
|
|
178
|
+
from vigily import Run
|
|
179
|
+
|
|
180
|
+
with Run(experiment="cifar10", name="resnet18-lr3e-3") as run:
|
|
181
|
+
run["hparams"] = {"lr": 3e-3, "batch_size": 128}
|
|
182
|
+
for step, loss in train():
|
|
183
|
+
run.track(loss, name="loss", step=step, context={"subset": "train"})
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
Evaluate a model on a versioned dataset:
|
|
187
|
+
|
|
188
|
+
```python
|
|
189
|
+
from vigily import Dataset, evaluate, evaluators as ev
|
|
190
|
+
|
|
191
|
+
evaluate(
|
|
192
|
+
Dataset("support-intents"),
|
|
193
|
+
target=classify,
|
|
194
|
+
evaluators=[ev.exact_match(), ev.custom("judge", my_judge)],
|
|
195
|
+
target_ref={"model": "intent-clf-v2"},
|
|
196
|
+
experiment="support-intents-nightly",
|
|
197
|
+
)
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
## Screenshots
|
|
201
|
+
|
|
202
|
+
<table>
|
|
203
|
+
<tr>
|
|
204
|
+
<td><img src="docs/assets/screenshots/metrics-desktop-light.png" alt="Loss, accuracy and learning rate for sixteen runs, grouped by optimizer"></td>
|
|
205
|
+
<td><img src="docs/assets/screenshots/metrics-aggregate-desktop-dark.png" alt="Validation curves as a mean and min-max band per learning rate"></td>
|
|
206
|
+
</tr>
|
|
207
|
+
<tr>
|
|
208
|
+
<td><img src="docs/assets/screenshots/eval-items-desktop-light.png" alt="Per-item results of an evaluation run with pass and fail verdicts"></td>
|
|
209
|
+
<td><img src="docs/assets/screenshots/compare-desktop-dark.png" alt="Two evaluation runs compared item by item, regressions first"></td>
|
|
210
|
+
</tr>
|
|
211
|
+
<tr>
|
|
212
|
+
<td><img src="docs/assets/screenshots/drift-desktop-light.png" alt="Drift suites with their verdicts against the baseline"></td>
|
|
213
|
+
<td><img src="docs/assets/screenshots/alerts-desktop-dark.png" alt="Alerts for drift and a failed schedule"></td>
|
|
214
|
+
</tr>
|
|
215
|
+
</table>
|
|
216
|
+
|
|
217
|
+
<p align="center">
|
|
218
|
+
<img src="docs/assets/screenshots/runs-phone-light.png" width="260" alt="The runs list on a phone">
|
|
219
|
+
<img src="docs/assets/screenshots/metrics-phone-dark.png" width="260" alt="Charts on a phone in the dark theme">
|
|
220
|
+
</p>
|
|
221
|
+
|
|
222
|
+
## Documentation
|
|
223
|
+
|
|
224
|
+
The documentation is a mkdocs-material site in [`docs/`](docs); build it with
|
|
225
|
+
`uvx --with mkdocs-material mkdocs serve`.
|
|
226
|
+
|
|
227
|
+
- [Getting started](docs/getting-started.md) and [concepts](docs/concepts.md)
|
|
228
|
+
- Guides: [Python SDK](docs/guides/python-sdk.md), [command line](docs/guides/cli.md),
|
|
229
|
+
[configuration](docs/guides/configuration.md), [scheduled evaluations and security](docs/guides/scheduled-evals.md),
|
|
230
|
+
[drift methods](docs/guides/drift.md), [LLM tracing](docs/guides/tracing.md), [self-hosting and storage](docs/guides/storage.md),
|
|
231
|
+
[the audit trail](docs/guides/audit-trail.md)
|
|
232
|
+
- Design: [brand](brand/DESIGN.md), [architecture](docs/architecture.md), [decisions](DESIGN.md)
|
|
233
|
+
- [FAQ](docs/faq.md)
|
|
234
|
+
|
|
235
|
+
## Status
|
|
236
|
+
|
|
237
|
+
vigily is new (0.1, unreleased). It runs end to end on the demo and on 2 000 synthetic runs with 4.8
|
|
238
|
+
million points (start-up 0.2 s, every request in about 0.1 s or less). [TODO.md](TODO.md) lists what comes next,
|
|
239
|
+
and [docs/kv0-requests.md](docs/kv0-requests.md) what it needs from its storage layer.
|
|
240
|
+
|
|
241
|
+
Drift verdicts and evaluation scores are evidence for people to review, not automated decisions.
|
|
242
|
+
|
|
243
|
+
## Licence
|
|
244
|
+
|
|
245
|
+
MIT. See [LICENSE](LICENSE).
|