jevstiller 0.2.0__tar.gz

jevstiller-0.2.0/.gitignore

@@ -0,0 +1,227 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#   Usually these files are written by a python script from a template
+#   before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+# Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+# uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+# poetry.lock
+# poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+# pdm.lock
+# pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+# pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# Redis
+*.rdb
+*.aof
+*.pid
+
+# RabbitMQ
+mnesia/
+rabbitmq/
+rabbitmq-data/
+
+# ActiveMQ
+activemq-data/
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#   JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#   be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#   and can be added to the global gitignore or merged into this file.  For a more nuclear
+#   option (not recommended) you can uncomment the following to ignore the entire idea folder.
+# .idea/
+
+# Abstra
+#   Abstra is an AI-powered process automation framework.
+#   Ignore directories containing user credentials, local state, and settings.
+#   Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#   Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#   that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#   and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#   you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+# Temporary file for partial code execution
+tempCodeRunnerFile.py
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Streamlit
+.streamlit/secrets.toml
+
+# Jevstiller experiment outputs
+experiments/results/
+experiments/data/
+experiments/cache/
+
+# Jevstiller deployment secrets
+deploy/admin-token.txt
+jevstiller-data/

jevstiller-0.2.0/CHANGELOG.md

@@ -0,0 +1,96 @@
+# Changelog
+
+## 0.2.0 — 2026-09-25
+
+The drop-in Jev proxy. Point `TYPESAFE_BASE_URL` at `jevstiller serve`, keep your services' own Jev keys, and the proxy learns each repeated `choice` question from Jev's answers, then answers it locally within the agreement budget you set, with a permanent audit to Jev and automatic fallback. First release on PyPI (`pip install "jevstiller[server,onnx]"`) and as a container image (`ghcr.io/tomerglick57/jevstiller`).
+
+**Upgrading from 0.1.0:** existing task directories and sample stores are migrated on open, and stored tasks are re-keyed automatically. `RoutingPolicy.disagreement_ub` now means the per-request bound (versions trained before keep their old numbers). Teacher failures are per item (`TeacherError`, or `errors="return"`).
+
+- **Packaging:** published to PyPI by the release workflow (trusted publishing; the built wheel must pass the test suite first), with the README's links pointing to GitHub. The container image is published to `ghcr.io/tomerglick57/jevstiller` for linux/amd64 and linux/arm64, with build provenance and an SBOM. The image installs dependencies at the versions in `uv.lock`, hash-checked. `deploy/smoke_test.py` checks any image end to end (non-root, read-only, no capabilities, offline start, forwarding, admin API, clean stop), and CI runs it on every image build. Dependabot keeps the pinned actions current.
+- **Security audit run 2** (2026-09-25; [docs/security.md](docs/security.md)): 15 findings, all fixed, with regression tests in `tests/test_audit_run2.py`. None exposed keys, other tenants' data or the admin API. Behaviour changes:
+  - **Settings are stricter.** Values of the wrong type stop startup (`store_text = "false"` used to mean *true*). So do an empty secret, secret file or access-control variable (an empty `JEVSTILLER_ACCESS_TOKEN` used to switch the token off), `text_retention_days` / `idle_ttl_days` ≤ 0, a non-finite `key_ttl_s`, and `trust_forwarded_for` entries with host bits (`10.0.0.5/24`; uvicorn ignored them). `store_text = false` under `[engine]` now works (either one saying `false` wins). `JEVSTILLER_<LIST>=none` clears a list from the config file. The startup log names the access controls in effect and whether request text is stored. Credentials in `upstream` are redacted in logs and `jevstiller config`.
+  - **Encoder backpressure can't latch.** One very large request (or one stalled encode) used to switch local answers off for every tenant until restart. The encoder now sees at most 32,768 characters of a text (model encoders read only 256 tokens anyway), and an idle encoder is never "overloaded". Embeddings of texts longer than that change once.
+  - **Admission counts only questions Jev answered.** A caller could create tasks with questions Jev rejects. The request whose answer admits a task is recorded as its first row.
+  - **Forwarding never fails with a 500 or leaks a routed task:** header bytes are forwarded raw (a UTF-8 `User-Agent` used to give a 500 and pin the task in memory until restart); any forwarding failure is a 502; `retry-after` is capped at 1 h.
+  - **No more crash when a store closes under text retention** (SIGSEGV): retention and admin calls hold the task's engine; admin calls don't count as use of the task.
+  - Paths with control characters or an encoded `?`/`#` are rejected (400). They could escape a path prefix in `upstream`.
+  - `/readyz` no longer flaps under concurrent probes.
+  - Training workers start with forkserver on every Python version (they were forked from the server before 3.14).
+  - `jevstiller backup` / `restore` write owner-only files; `jevstiller admin` URL-escapes names and requires the key argument; class names are escaped in `admin status`.
+  - Also: no duplicate `server`/`date` headers; `ci.yml` has read-only permissions; `--build-arg PRELOAD_ENCODER=` works; `uvicorn>=0.31`.
+- **The agreement guarantee is now sound** (found by a prior-art review, 2026-09-25). The threshold rule used to bound `coverage × UB(disagreement | answered)`, treating the estimated coverage as exact. It kept the widest of ~200 thresholds that each passed at the full `δ`, which has no `δ`-level guarantee. And it chose the OOD cutoff on the same calibration rows. Now:
+  - the rate of *answered and disagreeing* requests over all calibration rows is bounded directly (exact Clopper–Pearson);
+  - thresholds from a fixed grid are tested strictest-first, stopping at the first failure (fixed-sequence testing, as in Learn Then Test);
+  - the OOD cutoff comes from leave-one-out scores of the training reference;
+  - shadow judging uses the same loss.
+  - `tests/test_calibrate.py` simulates 300 calibration sets and checks that the chosen threshold breaks the budget in at most ~`δ` of them.
+  - `RoutingPolicy.disagreement_ub` now means that per-request bound. Versions trained before this keep their old numbers.
+- **Connection reuse:** `jevstiller serve` keeps idle connections for 75 s (it was uvicorn's 5 s, the same as httpx's reuse window, and requests in flight were dropped: 37 in 1.15M in the soak). A forwarded request whose pooled connection Jev had just closed is retried once (7 in 1.15M were 502s).
+- README: **How it compares**, covering stuntd, Distil Labs, routers, caches, open Jev-compatible models and the research lineage.
+- **Phase 6 testing found and fixed:**
+  - **Silent drift never recovered.** When Jev changed its answers without changing its model name, a task fell back and then retrained forever on mixed old and new answers; every candidate failed shadow. A confirmed drift (`fallback`) now restarts the training data: training, calibration, shadow and audit read only rows after the event's `since_id`. A replayed silent drift is back to 81% local answers within 2,000 requests and to its old ~96% within ~20,000. The fallback survives a restart; `mode auto` ends it by hand (`fallback_cleared` event). After a teacher change or a drift, the next student waits for the full readiness thresholds on the new data.
+  - **Memory grew without bound when tasks load and unload often** (more active tasks than `max_loaded`): 555 MB after 4 minutes and accelerating. A reference cycle (engine → training executor → the engine's priority method) kept every unloaded engine and its arrays until a full garbage collection. The scheduler and training callbacks now hold engines weakly; an unloaded engine is freed at once (tested with the collector off). In the final 40-minute soak (~18 reloads a second and a drift) memory ended at 182 MB.
+  - **`max_loaded` didn't hold under a stream of loads.** Only the janitor enforced it, one unload at a time, and it skipped every engine with a maintenance pass merely *requested*, which under steady traffic is nearly all of them. `benchmarks/manager.py` ended up with 505–666 tasks loaded against `max_loaded = 50` (2.7 GB). Now a load that pushes past the cap unloads the least recently used idle engine itself, and only a *running* pass or a pending training job keeps an engine loaded.
+  - **glibc heap fragmentation under task churn:** every load starts threads and SQLite connections, spread over up to 8 arenas per core. Resident memory grew ~47 KB per load/unload with Python's own allocations flat. The manager now limits glibc to 2 arenas (unless `MALLOC_ARENA_MAX` is set): 7,365 load/unload cycles held 234 → 300 MB, with serving latency unchanged.
+  - **Progress was lost on every reload:** a shadow's sample count restarted (so an often-unloaded task never got its shadow judged), the "rows since last training" counter restarted (a retrain on every reload once a task had `min_new_samples` rows), and the drift-check counter restarted. Versions now record `trained_to_id` and `shadow_from_id`, and the manager carries the in-memory counters across unloads.
+  - **Training workers outlived a `kill -9` of the server**, with the forkserver and resource tracker: they never see their job queue close. Each worker now exits within a second of the server's death.
+  - A task without a student for its current data (new, or after a teacher change or drift) whose candidate failed retrained every 100 rows. It now waits for 25% more data (at most `min_new_samples`).
+  - **Beyond the encoder's capacity, latency grew without bound** (the load test: 15 s p99 at 256 callers with bge-small on CPU, and throughput halved). Requests that would wait longer than `max_encoder_wait_ms` (200) for the shared encoder are now forwarded to Jev unrouted (`overloaded`); `jevstiller_encoder_wait_seconds` shows the backlog. Also, requests that are only forwarded (questions not admitted as tasks) are no longer encoded at all.
+  - **Forwarding collapsed beyond ~100 concurrent requests** (~75 req/s at 256 callers). The cause was the single httpx client to Jev: its pool's cost grows with connections × waiting requests. Upstream connections are now split over clients of 16 (`UPSTREAM_POOL`), with each request going to the least busy one: 614 req/s at 256 callers. The same limit in the load generator is why `benchmarks/load.py` spreads its connections over processes.
+  - A failure while routing a request from a not-yet-verified key could raise after Jev had already answered; it now forwards Jev's answer unrecorded.
+  - At the default `info` level, httpx logged a line per forwarded request next to the access log; it now logs warnings only.
+  - **A fit read a task's whole history.** Training time and the training worker's memory grew without bound with a busy task's age (a task with 100k teacher answers a day would hold ~4.6 GB after a month). Fits now use the most recent `max_train_samples` (50,000) and `max_calib_samples` (20,000) rows.
+  - A graceful shutdown during a training job left an empty staging directory (removed at the next start anyway); it is now removed.
+  - A full disk logged a stack trace per failed batch. Now one line per store per minute, and the new counter `jevstiller_store_dropped_records_total`. Serving continues either way.
+- **Tests:** `tests/test_chaos.py` (Jev down, Jev slow, a 429 storm, a full disk, and `kill -9` three times during training with restarts), `tests/test_concurrency_many.py` (16 threads over 12 tasks that unload and reload under load; 300 concurrent async clients through the proxy; every row recorded exactly once), and the drift and reload regressions. **Benchmarks:** `benchmarks/soak.py` (steady traffic through a real server with a silent drift at the midpoint, RSS and local share over time), `benchmarks/load.py` (forward-only, local and mixed traffic at fixed concurrency), `experiments/live_proxy.py` (cold start to local answers through the proxy against live Jev).
+- Concurrency: the engine no longer holds a lock across encoding, inference, the teacher call, or the store. 32 callers with a 50 ms teacher went from 19 to ~500 req/s (`benchmarks/concurrency.py`).
+- Training moved off the request path: `Config.training` = `background` (default, a worker thread per task, paced by `maintenance_interval_s`), `inline` (old behaviour; used by tests and experiment replays), or `manual` (call `maintain()` / `train_now()`). New `drain()`; `close()` stops the worker.
+- `train_executor=`: run training in any `concurrent.futures.Executor`, e.g. a shared `ProcessPoolExecutor`. The job (`training.run_fit_job`) reads the samples from a read-only store connection, fits, scores production, and writes the bundle itself; the registry `adopt()`s the finished directory. Only small objects cross the process boundary.
+- `training.train_pool(workers=2, niceness=10)`: a shared, low-priority process pool for `train_executor`. With 5 tasks training at once, serving p99 rose ~12% (vs ~12x with an uncapped 5-worker pool), and the fits finished ~2.5x faster than in threads.
+- `Config.train_threads` (default 2) caps BLAS threads per fit via `threadpoolctl` (new dependency). Uncapped, one fit takes every core and slows serving whichever process it runs in.
+- Sample store: write-behind batching on one writer thread, per-thread SQLite connections; reads see every earlier insert. `flush()`.
+- Registry: atomic, fsynced writes; crash leftovers are cleaned up; unknown versions/states raise `ValueError`.
+- Teacher failures are per item: adapters may return an `Exception` per text; `classify_batch` raises `TeacherError` (with the partial results) or, with `errors="return"`, returns `Result(label=None, error=...)`. Failed items are not recorded. `Status.teacher_errors`.
+- `JevTeacher`: `timeout=` (default 10 s), per-item errors, token usage from `response.usage`, tolerant of a missing request id.
+- Tasks: `instructions` and class descriptions may be any JSON value (text, object, array, or `None` for a name-only class), as in Jev's `criteria`; 2–255 classes; labels validated. Task versions of text tasks are unchanged from 0.1.0.
+- A trained student stores its label order and is reordered on load. The task version ignores class order, so a task declared with its classes in another order used to silently misread a saved model.
+- States: `classify` / `classify_batch` / `evaluate` take text or a JSON object/array. Objects are encoded and stored as canonical JSON (`state_type` column, added to existing stores on open); the teacher receives the object unchanged.
+- Teacher lineage: `TeacherOutput.model` names the model that answered (`JevTeacher`: `jev:<resolved model>`, e.g. `jev-latest` -> `jev:jev-1.13.0`). After `teacher_change_confirm` (20) answers in a row from a new model, the loop starts a new lineage: training, calibration, shadow and audit use only that model's answers; `teacher_change="fallback"` (default) sends everything to the teacher until a new student passes shadow, `"audit"` keeps serving with a raised audit rate. Bundles record `teacher_model`; `Status.teacher_model`; `teacher_changed` event.
+- Rare classes: `status()` / the report say what the first student is waiting for (samples, and which classes are below `min_samples_per_class`). `Config.rare_classes="defer"` trains without waiting; the policy's `deferred_labels` are never answered by the student (`routing_reason="rare_class"`), and calibration accounts for that.
+- **Security audit** (2026-09-24, run 1 against `65fe64b`; details in docs/security.md). Twelve findings fixed, each with a regression test in `tests/test_audit_fixes.py`:
+  - A key is accepted only after a parsed `/v1/systemone` answer, and requests with an unaccepted key are forwarded first and recorded afterwards.
+  - `/healthz`, `/readyz`, `/metrics` and `/jevstiller/*` are local and never forwarded.
+  - The body limit is enforced while streaming; one `Authorization` header only; no dot segments; `X-Forwarded-For` only from `trust_forwarded_for`.
+  - One routing per distinct question, at most `max_questions` (32) per request, and the state encoded once per request. Task caps (`max_tasks`, per-tenant default 1,000).
+  - `secure_delete` plus a WAL truncate for retention, 0700/0600 files, and an HMAC text hash.
+  - Task keys use the full SHA-256 of the question (`Task.fingerprint`); stored tasks are re-keyed automatically on start.
+  - No cookie jar; strict configuration; `key-hash` never creates a salt.
+- **Operations:**
+  - `jevstiller serve --config` reads TOML + `JEVSTILLER_*` + flags (`jevstiller/settings.py`, validated).
+  - Admin API `/jevstiller/v1/*` and `jevstiller admin` (tasks, status, mode, target, train, promote, rollback, delete, delete-tenant, stats; mode/target persisted per task).
+  - Prometheus `/metrics`, `/readyz`, JSON access logs, `jevstiller backup` / `restore`, `jevstiller config`.
+  - Dockerfile (non-root, encoder baked in, runs offline and read-only), docker-compose, Kubernetes manifest.
+  - `docs/deploy.md`, `docs/operations.md`, `docs/security.md`.
+- **Validated against live Jev** (2026-09-24, `jev-1.13.0`; re-run with the corrected calibration on 2026-09-25: 70.7% at 99.45%): Banking77 replay, 70.6% held-out coverage at 99.40% agreement (target 98%), accuracy preserved (78.7% vs Jev's 78.55%). `experiments/jev_profile.py` measures Jev latency and records wire fixtures (`tests/fixtures/jev/`); `tests/test_live.py` has opt-in live tests (`JEVSTILLER_LIVE=1 pytest -m live`) and replays the recorded responses through the proxy in every run.
+- **Docs:** DESIGN.md v3 (what was built, what the live run changed, what is still roadmap), `docs/proxy.md`, `docs/configuration.md`, `docs/benchmarks.md`, `benchmarks/README.md`; CONTRIBUTING and SECURITY updated.
+- **Drop-in Jev proxy** (`jevstiller serve`, `server.create_app`; extra `server`). Speaks `POST /v1/systemone` and passes everything else through. Each `choice` question is routed to its task (tenant, instructions, criteria, requested model). If every question can be answered locally and the caller's key has been accepted by Jev within `key_ttl_s`, the proxy answers in Jev's exact shape: resolved `model`, zero `usage`, a `jvs_` `x-typesafe-request-id`, and `x-jevstiller-source` / `x-jevstiller-detail` headers. Otherwise the whole request goes to Jev with the caller's key, the response is returned unchanged, and the choice answers are recorded as training rows. The proxy never stores keys: it keeps salted hashes (`key-salt`, created 0600), and a 401/403 revokes a key's local answers at once. It remembers per-key 429 `retry-after` and answers 429 itself meanwhile. Upstream timeout is 9 s, then 504; unreachable gives 502; over `max_upstream_inflight` gives 503. Tenancy is `shared` or `per_key`. Any internal error falls back to forwarding. Tested with the unmodified `typesafe-sdk` (sync and async) over real HTTP; adds ~3 ms to a forwarded request.
+- `Jevstiller.route()` / `defer()` / `complete()`: `classify_batch` split so the caller can make the teacher call itself; `TaskManager.route()` / `complete()`. The requested teacher model is part of a task's key.
+- `TaskManager`: many tasks in one process. A task is found by `hash(tenant, question type, Task.version)`, so services asking the same question share one student and any wording change is a new task. Engines load on demand (~4 ms from disk) and are unloaded least-recently-used when `max_loaded` / `max_memory_mb` is exceeded; engines with requests in flight or maintenance running are never unloaded. `Admission` (default: 50 requests within 24 h) keeps one-off questions from creating tasks; `max_tasks_per_tenant`; `idle_ttl_s` deletes unused tasks; `delete()`. One SQLite store per task under `<data_dir>/tasks/<key>/`.
+- `TrainScheduler`: one executor for every task's training. Round-robin across tenants, highest rate of teacher calls first within a tenant, retries with backoff, broken process pools replaced, queued jobs cancellable. With it (or any executor), training is asynchronous: maintenance keeps judging shadows and checking drift while a job waits for a worker. `Jevstiller.training_priority()`, `busy()`, `footprint_bytes()`; `classify_batch(..., teacher=)` for a per-call teacher (the caller's own key).
+- `BatchingEncoder`: one encoder shared by every task and thread; concurrent calls are merged into batches with no added latency when idle.
+- OOD reference capped at `Config.ood_max_ref` (default 5,000, stratified by class; was 50,000 at random). Held-out coverage and agreement unchanged on Banking77 (even at 1,000) and CLINC150; memory and per-request cost drop up to 10x on large tasks.
+- Sample stores record a schema version and skip setup when current (reopening a store: ~0.4 ms).
+- `__version__` comes from package metadata; `Config` and `set_mode` validate with `ValueError`; CI on Python 3.10–3.14.
+
+## 0.1.0 — 2026-09-22
+
+First public release.
+
+- Core loop: route → record → train → shadow → promote → monitor → fall back, with a teacher-parity contract (`target_agreement`) and a permanent audit channel.
+- Teachers: Jev via `typesafe-sdk`, plus synthetic, replay, and cached adapters.
+- Encoders: hashing (no download), PyTorch (HF checkpoints, CUDA), ONNX Runtime (CPU/CUDA); `small`/`base`/`large` tiers.
+- Student: numpy soft-target logistic regression with early stopping; kNN out-of-distribution gate; Clopper–Pearson threshold selection with headroom.
+- Experiment runner for Banking77, CLINC150, AG News, and synthetic data.
+- `store_text=False` for hash-only storage; thread-safe `classify`; `versions()` and `export()`.
+
+Known limits: training runs synchronously inside `classify_batch` when a trigger fires; single process per task directory; live-Jev benchmark pending.