nova-ai-pro 1.2.3.dev76__tar.gz → 1.2.3.dev78__tar.gz

{nova_ai_pro-1.2.3.dev76 → nova_ai_pro-1.2.3.dev78}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: nova-ai-pro
-Version: 1.2.3.dev76
+Version: 1.2.3.dev78
 Summary: NOVA AI — modular AI assistant backend with composable intelligence primitives
 Project-URL: Homepage, https://github.com/Hamza35779/NOVA-AI
 Project-URL: Documentation, https://hamza35779.github.io/NOVA-AI/

{nova_ai_pro-1.2.3.dev76 → nova_ai_pro-1.2.3.dev78}/docs/architecture/learning.md

@@ -610,3 +610,11 @@ Every edit is assigned a tier from a deterministic lookup table:
 | `manual` | LoRA fine-tuning (v2) | Never auto-apply |
 
 See [LLM-guided spec search guide](../user-guide/llm-guided-spec-search.md) for the architecture and the building blocks.
+
+### Self-training (LoRA from traces)
+
+The `lora_finetune` op is now real: `LoraFinetuneApplier` (`src/nova_ai/learning/spec_search/execute/appliers/lora.py`) mines SFT pairs from traces, trains a LoRA adapter, and (with explicit `auto_apply`) activates it. Outside spec-search, the same pipeline runs via the `nova train` CLI, cron schedules, and the auto-trigger — always behind the benchmark gate and pending-review promotion. See [Self-Training](../learning/self-training.md).
+
+### Model proving ground (head-to-head model A/B)
+
+`nova_ai.learning.proving` closes the model-adoption loop: a benchmark synthesized from high-feedback traces (`PersonalBenchmarkSynthesizer`) is run through `EvalRunner` twice — candidate vs incumbent, same seed, same judge (the incumbent itself by default) — and `RunSummary.per_subject` keys accuracy by routing query class (`classify_query`). A verdict requires ≥ 3 scored samples per class per side; adoption additionally requires a ≥ `min_margin` win. Winners land in `~/.nova_ai/learning/proving/policy_map.json` via `nova prove adopt`, and SmartRouter serves them per class only with the opt-in `proving_adoption` router flag. The gauntlet is read-only; adoption is the only mutation and is reversible (`nova prove revert`). See [Model Proving Ground](../learning/proving-ground.md).

nova_ai_pro-1.2.3.dev78/docs/learning/proving-ground.md

@@ -0,0 +1,150 @@
+---
+title: Model Proving Ground
+description: NOVA A/B-tests new models against your real workloads and only uses them where they win
+---
+
+# Model Proving Ground: only adopt models that win on YOUR traces
+
+Pulling a new model today is vibes: a vendor benchmark says it's good, you `ollama pull`, and hope it's better for *your* work. The proving ground replaces hope with measurement. NOVA synthesizes an evaluation set from **your own high-feedback traces**, runs the candidate and your incumbent head-to-head under the **same judge**, and reports a verdict **per query class** (`code` / `math` / `long` / `short` / `general`). Adoption is manual by default and reversible with one command.
+
+**The pitch: pull any new model — NOVA tests it against your real workloads and only uses it where it wins.**
+
+## The gauntlet
+
+```
+   ┌──────────────┐    ┌────────────────────┐    ┌──────────────────┐
+   │ Your traces  │───▶│ Personal benchmark │───▶│ Head-to-head     │
+   │ (feedback    │    │ (query + reference │    │ candidate vs     │
+   │  ≥ 0.7)      │    │  per high rating)  │    │ incumbent        │
+   └──────────────┘    └────────────────────┘    └────────┬─────────┘
+                                                          ▼
+   ┌──────────────────┐    ┌────────────────────┐    ┌──────────────────┐
+   │ Router serves    │◀───│ policy_map.json    │◀───│ Per-class verdict│
+   │ proven models    │    │ (nova prove adopt) │    │ (Δ accuracy per  │
+   │ (opt-in)         │    │                    │    │  class, margin)  │
+   └──────────────────┘    └────────────────────┘    └──────────────────┘
+```
+
+Fairness rules baked in:
+
+- **Same benchmark both sides.** Samples are synthesized once from your traces; both models answer identical questions with identical prompts, temperature 0, and the same seed.
+- **Same judge both sides.** The judge defaults to the *incumbent model itself* (`judge_model = ""`), so no third model gets to play favorites. Fully offline.
+- **Evidence thresholds.** A class gets a verdict only with ≥ 3 scored samples on both sides; adoption additionally requires the candidate to win by at least `min_margin` (default `0.05`).
+
+## Quickstart
+
+```toml
+# ~/.nova_ai/config.toml
+[learning.proving]
+enabled = true
+```
+
+```bash
+# Run the gauntlet (background by default, --foreground to stream the scorecard)
+nova prove run qwen3:8b --incumbent qwen2.5:7b --foreground
+
+# Inspect results
+nova prove status        # latest run + per-class scorecard
+nova prove list          # run history
+
+# Adopt winners per class — this is the only mutation, and it's explicit
+nova prove adopt prove_20260902_101530_ab12cd
+nova prove roster        # what the router would serve per class
+nova prove revert code   # undo an adoption any time
+```
+
+A scorecard looks like:
+
+```
+Per-Query-Class Scorecard
+Class     Candidate   Incumbent   Δ      Winner
+code           100%         60%   +0.40  qwen3:8b
+math            60%         60%   +0.00  —
+long            80%         60%   +0.20  qwen3:8b
+general         60%         80%   -0.20  qwen2.5:7b
+```
+
+`general` shows the incumbent is better — adopting that run takes `code` and `long` only, and your incumbent keeps serving `math` and `general`.
+
+## Serving proven models
+
+Adoption writes `~/.nova_ai/learning/proving/policy_map.json` (`class → model`), but the router does **not** act on it until you opt in:
+
+```toml
+# in config.toml — under the section your engine's router reads
+proving_adoption = true
+```
+
+With `proving_adoption` enabled, SmartRouter classifies the incoming query and serves the proven model for that class when the engine can serve it; anything unservable or unmapped falls back to the normal tier flow. Every failure mode degrades to "keep routing as before" — a corrupt map can never break routing.
+
+## Watch for new models automatically
+
+```bash
+nova prove watch            # report newly pulled models
+nova prove watch --prove    # prove each new model as it appears
+```
+
+The watcher keeps a snapshot at `~/.nova_ai/learning/proving/known_models.json`. A model pulled again after removal counts as new.
+
+To run the check on a schedule, create a scheduler task with `kind="prove"` metadata (the daemon runs it with your cron schedule):
+
+```bash
+nova scheduler create "prove new models" --cron "0 4 * * *" \
+  --metadata '{"kind": "prove"}'
+```
+
+The scheduled task checks for new models and proves them when both `[learning.proving] enabled` **and** `auto_trigger` are true.
+
+## The flywheel with self-training
+
+`nova train deploy --target ollama` creates `nova-tuned-<model>` tags. Those are new models too — the watcher sees them, the gauntlet tests *tuned vs base* per class, and the router adopts tuning only where it actually helped. Closed loop: use NOVA → rate answers → train → prove → route better.
+
+## Safety model
+
+The gauntlet is **read-only** — it burns GPU time, never changes behavior. Adoption is the only mutation, and it's:
+
+1. **Manual by default** (`auto_adopt = false`): winners wait for `nova prove adopt`.
+2. **Margin-gated**: a class is adopted only when the candidate won by ≥ `min_margin`.
+3. **Reversible**: `nova prove revert <class>` removes an entry instantly.
+4. **Off unless opted in at the router too**: even an adopted map is inert until `proving_adoption = true`.
+
+## Config reference
+
+```toml
+[learning.proving]
+enabled = false             # master switch
+auto_trigger = false        # prove automatically when a new model appears
+auto_adopt = false          # adopt winners without `nova prove adopt`
+min_margin = 0.05           # per-class accuracy margin required to adopt
+min_samples = 10            # minimum synthesized benchmark samples per run
+max_samples = 60            # cap on benchmark size (bounds GPU time)
+schedule = ""               # cron expression (reserved for the prove task)
+incumbent = ""              # default opponent; "" → intelligence.default_model
+judge_engine = "local"      # "local" | "cloud" — engine backing the judge
+judge_model = ""            # "" → judge with the incumbent (same judge both sides)
+```
+
+## Artifacts
+
+| Path | Contents |
+|------|----------|
+| `~/.nova_ai/learning/proving/runs.db` | Run history (`nova prove list`) |
+| `~/.nova_ai/learning/proving/policy_map.json` | Adopted per-class routing (`nova prove roster`) |
+| `~/.nova_ai/learning/proving/known_models.json` | Watcher snapshot of seen models |
+| `~/.nova_ai/learning/proving/last_run.log` | Background run output |
+
+## CLI reference
+
+| Command | What it does |
+|---------|-------------|
+| `nova prove run <candidate> [--incumbent <m>] [--foreground] [--adopt]` | Run the gauntlet |
+| `nova prove status` | Latest run + per-class scorecard |
+| `nova prove list [-n N]` | Run history |
+| `nova prove roster` | Current adoption map |
+| `nova prove adopt <run_id> [--class code,math]` | Adopt winners into the map |
+| `nova prove revert <class>` | Remove a class from the map |
+| `nova prove watch [--prove] [--foreground]` | New-model check, optionally prove |
+
+## Requirements
+
+A working engine backend for both models (e.g. Ollama running locally with both models pulled) and enough high-feedback traces to synthesize `min_samples` benchmark samples. Rate your NOVA answers (`nova feedback`) to build the evaluation data — the proving ground is only as good as the traces it learns from.

nova_ai_pro-1.2.3.dev78/docs/learning/self-training.md

@@ -0,0 +1,116 @@
+---
+title: Self-Training
+description: NOVA AI fine-tunes itself on your usage traces — offline, gated, and under your control
+---
+
+# Self-Training: NOVA learns how *you* use it
+
+NOVA's self-training pipeline turns your recorded interaction traces into a **LoRA adapter** for your local model. The loop is fully offline: mine traces → extract supervised pairs → fine-tune → benchmark gate → deploy. No other local-AI platform does this — most ship a runtime; NOVA ships a feedback loop.
+
+## The loop
+
+```
+   ┌────────────┐    ┌──────────────┐    ┌───────────────┐    ┌─────────┐
+   │  Traces    │───▶│  SFT pairs   │───▶│  LoRA train   │───▶│  Gate   │
+   │ (feedback  │    │ (miner, 0.7+ │    │ (rank 16,     │    │(bench Δ)│
+   │  ≥ 0.7)    │    │  quality)    │    │  Qwen default)│    └────┬────┘
+   └────────────┘    └──────────────┘    └───────────────┘         │
+        ▲                                                          ▼
+        │                                              ┌────────────────────┐
+        │                                              │ deploy_targets:    │
+        └────────────── use NOVA, rate answers ◀────── │ adapter / ollama / │
+                                                       │ llamacpp           │
+                                                       └────────────────────┘
+```
+
+## Quickstart
+
+```toml
+# ~/.nova_ai/config.toml
+[learning.training]
+enabled = true
+deploy_targets = ["adapter", "ollama"]
+min_pairs = 50
+```
+
+```bash
+# Export your trace data any time (portable JSONL)
+nova train export-traces --out pairs.jsonl
+
+# Start a training run in the background
+nova train run
+
+# Check progress / inspect results
+nova train status
+nova train list
+```
+
+When a run finishes with `pending_review`, promote it manually:
+
+```bash
+nova train deploy <run-id> --target ollama   # creates nova-tuned-<model>
+```
+
+## Safety model
+
+Weight updates are the most invasive edit class in the learning system — the spec-search planner pins `lora_finetune` at **MANUAL** risk tier. Three layers protect you:
+
+1. **Benchmark gate** — after training, both the base and tuned models are scored on the personal benchmark. If the delta is below `learning.min_improvement` (default `0.02`), the run is marked `rolled_back` and the adapter is never activated or deployed.
+2. **Pending review by default** — unless `auto_apply = true`, a passing run lands in `pending_review`. `active.json` is untouched; promotion requires an explicit `nova train deploy`.
+3. **Auto-trigger is doubly opt-in** — automatic training fires only when *both* `learning.auto_update` and `learning.training.auto_trigger` are true, and scheduled runs (cron) only when `enabled = true`.
+
+## Config reference
+
+```toml
+[learning.training]
+enabled = false            # master switch
+schedule = ""              # cron expression for scheduled (nightly) runs
+auto_trigger = false       # train when enough new qualifying traces accrue
+auto_apply = false         # deploy without manual review (gate still enforced)
+min_pairs = 50             # minimum SFT pairs before a run is worth starting
+max_pairs = 5000           # cap per run (bounds training time)
+deploy_targets = ["adapter", "ollama"]   # subset: adapter, ollama, llamacpp
+ollama_tag_prefix = "nova-tuned"
+llamacpp_gguf_script = ""  # path to llama.cpp convert_hf_to_gguf.py
+```
+
+Deprecated aliases: `learning.training_enabled` / `learning.training_schedule` still work and fold into `[learning.training]` (`enabled` / `schedule`).
+
+Training hyperparameters (LoRA rank, epochs, learning rate, base model) come from the existing `[learning.intelligence.sft]` section — see `SFTConfig` in `src/nova_ai/core/config.py`.
+
+## Deployment targets
+
+| Target | What it produces | Notes |
+|--------|-----------------|-------|
+| `adapter` | The PEFT adapter directory (default) | Always available; needed for the other two |
+| `ollama` | `ollama create nova-tuned-<model>` | Modelfile (`FROM <base>` + `ADAPTER <dir>`) written into the adapter dir; shows up in `ollama list` and NOVA's model discovery |
+| `llamacpp` | Merged model (+ optional GGUF) | `merge_and_unload()` into `~/.nova_ai/learning/training/merged/`; GGUF conversion requires `llamacpp_gguf_script` pointing at llama.cpp's `convert_hf_to_gguf.py` |
+
+Targets are independent — one failing (e.g. no Ollama installed) never aborts the others.
+
+## Triggers
+
+| Trigger | How | Gate |
+|---------|-----|------|
+| Manual | `nova train run [--foreground]` | benchmark gate + pending review |
+| Scheduled | `schedule = "0 3 * * *"` in `[learning.training]`; the daemon's scheduler dispatches a `kind="train"` task | benchmark gate; runs carry the gate even on auto |
+| Auto | `auto_trigger = true` **and** `learning.auto_update = true`; fires when ≥ `min_pairs` new qualifying traces accrue since the last successful run | benchmark gate + pending review |
+
+Background runs spawn a detached process (Windows-safe) writing to `~/.nova_ai/learning/training/last_run.log`; progress is in the run store (`nova train status`).
+
+## Artifacts
+
+| Path | Contents |
+|------|----------|
+| `~/.nova_ai/learning/training/adapters/<run>/` | Trained adapter + `adapter_meta.json` |
+| `~/.nova_ai/learning/training/active.json` | Pointer to the active adapter |
+| `~/.nova_ai/learning/training/runs.db` | Run history (`nova train list`) |
+| `~/.nova_ai/learning/training/merged/<model>/` | Merged models (llamacpp target) |
+
+## Where it plugs into spec-search
+
+The `LoraFinetuneApplier` (`src/nova_ai/learning/spec_search/execute/appliers/lora.py`) implements the `lora_finetune` edit op, so the teacher can propose weight updates when diagnosis shows a skill gap that prompt edits can't fix. In TIERED/MANUAL autonomy it routes to review like any MANUAL-tier edit; with `auto_apply` it trains and activates the adapter directly. The old `LoraStubApplier` remains as the torch-free fallback that refuses the op with a clear message.
+
+## Requirements
+
+Training requires `torch`, `transformers`, and `peft` (install with `pip install torch transformers peft`). Everything else — mining, export, run history, deployment plumbing — works without them; the CLI reports a clear install hint if torch is missing at train time.

{nova_ai_pro-1.2.3.dev76 → nova_ai_pro-1.2.3.dev78}/docs/user-guide/cli.md

@@ -509,3 +509,80 @@ directly from `nova_ai.learning.spec_search.orchestrator` and call
 [`docs/user-guide/llm-guided-spec-search.md`](llm-guided-spec-search.md)
 for the architecture and the building blocks
 (`splits.py`, external corpora, `external_adapter`).
+
+---
+
+## `nova train`
+
+Self-training: fine-tune a local model from your own usage traces. See [Self-Training](../learning/self-training.md) for the full guide and safety model.
+
+### `nova train run`
+
+Mine traces, train a LoRA adapter, and deploy per `[learning.training]` config.
+
+```bash
+nova train run                    # Start in the background
+nova train run --foreground       # Block and stream progress
+nova train run --base-model Qwen/Qwen3-1.7B   # Override the base model
+```
+
+### `nova train status`
+
+Show the latest (or running) training run: status, pairs used, loss, adapter path, deploy results.
+
+### `nova train list`
+
+List recent training runs with trigger, status, and benchmark delta.
+
+### `nova train deploy <run-id>`
+
+Promote and deploy a `pending_review` adapter (the manual review step).
+
+```bash
+nova train deploy train_20260901_030000_a1b2c3 --target adapter
+nova train deploy train_20260901_030000_a1b2c3 --target ollama --target adapter
+```
+
+### `nova train export-traces`
+
+Export mined SFT pairs to JSONL for external training runs.
+
+```bash
+nova train export-traces                      # pairs.jsonl in cwd
+nova train export-traces -o data/pairs.jsonl --min-quality 0.8
+```
+
+---
+
+## `nova prove`
+
+The Model Proving Ground: A/B-test a candidate model against your incumbent on benchmarks synthesized from *your own* traces, per query class. See [Model Proving Ground](../learning/proving-ground.md) for the full guide and safety model.
+
+### `nova prove run <candidate>`
+
+Run the head-to-head gauntlet (background by default).
+
+```bash
+nova prove run qwen3:8b --incumbent qwen2.5:7b --foreground   # stream the scorecard
+nova prove run qwen3:8b --adopt                               # adopt winners immediately
+```
+
+### `nova prove status` / `nova prove list`
+
+Latest run with its per-class scorecard; run history.
+
+### `nova prove roster`
+
+Show the current per-class adoption map (`policy_map.json`).
+
+### `nova prove adopt <run-id> [--class code,math]`
+
+Adopt a completed run's winners into the routing map (manual review step; margin-gated).
+
+### `nova prove revert <class>`
+
+Remove a query class from the adoption map (rollback).
+
+### `nova prove watch [--prove]`
+
+Check for newly pulled models; with `--prove`, run the gauntlet for each.

{nova_ai_pro-1.2.3.dev76 → nova_ai_pro-1.2.3.dev78}/mkdocs.yml

@@ -197,6 +197,8 @@ nav:
       - Benchmarks: user-guide/benchmarks.md
       - Security: user-guide/security.md
       - LLM-guided spec search: user-guide/llm-guided-spec-search.md
+      - Self-Training: learning/self-training.md
+      - Model Proving Ground: learning/proving-ground.md
   - Leaderboard: leaderboard.md
   - Roadmap: development/roadmap.md
   - Development:

{nova_ai_pro-1.2.3.dev76 → nova_ai_pro-1.2.3.dev78}/src/nova_ai/cli/__init__.py

@@ -34,6 +34,7 @@ from nova_ai.cli.model import model
 from nova_ai.cli.operators_cmd import operators
 from nova_ai.cli.optimize_cmd import optimize_group
 from nova_ai.cli.pearl_cmd import pearl
+from nova_ai.cli.prove_cmd import prove
 from nova_ai.cli.quickstart_cmd import quickstart
 from nova_ai.cli.registry_cmd import registry
 from nova_ai.cli.router_cmd import router_cmd
@@ -45,6 +46,7 @@ from nova_ai.cli.serve import serve
 from nova_ai.cli.skill_cmd import skill
 from nova_ai.cli.telemetry_cmd import telemetry
 from nova_ai.cli.tool_cmd import tool
+from nova_ai.cli.train_cmd import train
 from nova_ai.cli.vault_cmd import vault
 from nova_ai.cli.voice_cmd import voice
 from nova_ai.cli.workflow_cmd import workflow
@@ -130,6 +132,8 @@ cli.add_command(feedback_group, "feedback")
 cli.add_command(compose, "compose")
 cli.add_command(gateway, "gateway")
 cli.add_command(tool, "tool")
+cli.add_command(train, "train")
+cli.add_command(prove, "prove")
 cli.add_command(registry, "registry")
 cli.add_command(config, "config")
 cli.add_command(scan, "scan")