vetcheck 0.1.0__tar.gz

vetcheck-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 SuperInstance
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vetcheck-0.1.0/PKG-INFO

@@ -0,0 +1,383 @@
+Metadata-Version: 2.4
+Name: vetcheck
+Version: 0.1.0
+Summary: Model health monitoring as veterinary care — regression tests as physical exams, drift detection as weight monitoring, auto-quarantine on critical health breaches.
+Author: SuperInstance
+License: MIT
+Project-URL: Homepage, https://github.com/SuperInstance/vetcheck
+Project-URL: Repository, https://github.com/SuperInstance/vetcheck
+Keywords: ml,monitoring,model-health,drift-detection,regression-testing,observability
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# Vetcheck: Model Health Monitoring as Veterinary Care
+
+> Working dogs need regular checkups. So do models.
+
+[![Python](https://img.shields.io/python/required-version-toml?toml=pyproject.toml)](https://python.org)
+[![License](https://img.shields.io/github/license/SuperInstance/vetcheck)](LICENSE)
+[![Tests](https://img.shields.io/badge/tests-passing-brightgreen)](tests/)
+
+A model that passed regression tests six months ago might silently degrade over time — distribution drift, fine-tuning side effects, dependency changes. Vetcheck treats model health monitoring like veterinary care because in the Working Animal Architecture paradigm, every model is a working animal that needs ongoing supervision to stay fit for duty. Physical exams catch regressions. Weight monitoring catches output drift. Quarantine isolates failures before they reach production.
+
+## What It Does
+
+Vetcheck provides three core exams and two lifecycle operations. The **physical exam** runs a full regression test suite against the model, checking vital signs: does it still handle edge cases (temperature), are response times within bounds (heart rate), does it handle load without degradation (blood pressure), do function-calling and structured outputs still work (reflexes)? The result is an `ExamResult` with pass/fail per test and an overall `HealthStatus`.
+
+The **weight check** monitors output distribution drift over time. Just as a vet weighs a dog at every visit and flags concerning changes, Vetcheck compares recent output embeddings or distributions to a baseline and flags statistically significant drift. Trends are tracked across checkups — a gradual 5% drift over a month is different from a sudden 20% shift overnight.
+
+When a model fails critically, **quarantine** auto-isolates it: traffic is routed elsewhere, alerts fire, and the model stays isolated until a clean physical exam passes. The **health certificate** is a signed attestation that a model is fit for production — it verifies the last physical exam passed, confirms no active quarantine, checks weight stability, and returns a certificate with an expiry date.
+
+## Install
+
+```bash
+pip install vetcheck
+```
+
+For development:
+
+```bash
+git clone https://github.com/SuperInstance/vetcheck.git
+cd vetcheck
+pip install -e ".[dev]"
+```
+
+## Quick Start
+
+```python
+from vetcheck import VetCheck
+
+vet = VetCheck(model_id="llama-3-70b")
+
+# Run a full physical — executes regression suite
+report = vet.physical_exam()
+print(report.summary())
+# ════════════════════════════════════════════
+#   PHYSICAL EXAM: llama-3-70b
+#   Overall: HEALTHY
+#   Tests: 47 passed, 3 failed, 0 skipped
+# ════════════════════════════════════════════
+
+# Check for weight drift — compares recent outputs to baseline
+drift = vet.weight_check(window=100)
+if drift.significant:
+    print(f"⚠ Drift detected: {drift.magnitude:.1%} from baseline")
+    print(f"  Trend: {drift.trend}")  # "increasing", "decreasing", "stable"
+else:
+    print("✓ Weight stable")
+
+# Quarantine a sick model
+if report.critical_failures > 0:
+    vet.quarantine(reason="Critical regression detected")
+    print("Model quarantined — traffic routed to backup")
+
+# ... after fixes applied ...
+
+# Run physical again to verify recovery
+recovery = vet.physical_exam()
+if recovery.passed:
+    vet.release()
+    print("Model returned to active duty")
+
+# Issue a clean bill of health
+cert = vet.health_certificate()
+print(f"Health certificate issued, expires: {cert.expires_at}")
+print(f"Valid: {cert.is_valid()}")
+```
+
+## Architecture
+
+```
+src/vetcheck/
+├── __init__.py        # VetCheck orchestrator + HealthCertificate
+├── exam.py             # Regression test runner (physical exams)
+│   ├── run_physical_exam()
+│   ├── ExamResult
+│   └── TestCase / TestSuite
+├── drift.py            # Output drift detection (weight monitoring)
+│   ├── check_weight_drift()
+│   ├── DriftReport
+│   └── BaselineDistribution
+└── quarantine.py       # Quarantine and release system
+    ├── quarantine_model()
+    ├── release_model()
+    ├── get_quarantine_status()
+    └── QuarantineStatus
+```
+
+### Health States
+
+```
+HEALTHY ──────▶ UNDER_OBSERVATION ──────▶ QUARANTINED
+   ▲                    │                       │
+   │                    │                       │
+   └────────────────────┘                       │
+   release() (clean exam)                       │
+   ▲                                            │
+   └────────────────────────────────────────────┘
+                  release() (clean exam)
+```
+
+## API Reference
+
+### `VetCheck`
+
+```python
+class VetCheck:
+    def __init__(self, model_id: str, baseline_dir: str = "./baselines")
+
+    # Exams
+    def physical_exam(self, suite: str = "default") -> ExamResult
+    def weight_check(self, window: int = 100) -> DriftReport
+
+    # Lifecycle
+    def quarantine(self, reason: str) -> QuarantineStatus
+    def release(self) -> bool
+    @property
+    def is_quarantined(self) -> bool
+
+    # Certification
+    def health_certificate(self, validity_days: int = 7) -> HealthCertificate
+    @property
+    def health_status(self) -> HealthStatus
+```
+
+### Result Types
+
+```python
+@dataclass
+class ExamResult:
+    model_id: str
+    passed: bool
+    overall_status: HealthStatus
+    tests_passed: int
+    tests_failed: int
+    critical_failures: int
+    duration_seconds: float
+    details: list[TestResult]
+
+@dataclass
+class DriftReport:
+    model_id: str
+    significant: bool        # statistically significant drift
+    magnitude: float         # 0.0-1.0
+    trend: str               # "increasing", "decreasing", "stable"
+    window: int              # samples compared
+    p_value: float           # statistical test result
+
+@dataclass
+class HealthCertificate:
+    model_id: str
+    status: HealthStatus
+    issued_at: float
+    expires_at: float
+    last_exam_passed: bool
+    weight_stable: bool
+    quarantine_active: bool
+    notes: str
+
+    def is_valid(self, now: float | None = None) -> bool
+    def to_dict(self) -> dict
+```
+
+### `HealthStatus`
+
+```python
+class HealthStatus(str, Enum):
+    HEALTHY = "healthy"
+    UNDER_OBSERVATION = "under_observation"
+    QUARANTINED = "quarantined"
+    CRITICAL = "critical"
+```
+
+## The Analogy
+
+| Veterinary Care | Vetcheck |
+|-----------------|----------|
+| Physical exam | Regression test suite (`physical_exam`) |
+| Weight monitoring | Output drift detection (`weight_check`) |
+| Quarantine | Auto-isolation on critical health breach |
+| Health certificate | Clearance for production deployment |
+| Vaccination schedule | Periodic exam scheduling |
+| Breed-specific screening | Model-specific test suites |
+
+## Testing
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+
+# Test exam runner
+pytest tests/test_exam.py -v
+
+# Test drift detection
+pytest tests/test_drift.py -v
+
+# Test quarantine lifecycle
+pytest tests/test_quarantine.py -v
+```
+
+## Philosophy
+
+In animal husbandry, preventative medicine is cheaper than emergency treatment. The same applies to AI systems: catching a regression before it reaches production is dramatically cheaper than responding to an outage. Vetcheck brings the discipline of veterinary preventative care to model operations — regular checkups, early detection, isolation of sick individuals, and structured health certificates before deployment.
+
+This is part of [Working Animal Architecture](https://github.com/SuperInstance/AI-Writings) — specifically the health monitoring layer that keeps the kennel (flux registry) populated with animals fit for duty. A model with an expired health certificate shouldn't be in production, just as a working dog that hasn't seen a vet in a year shouldn't be herding sheep.
+
+## Ecosystem
+
+| Repo | Role |
+|------|------|
+| **[vetcheck](https://github.com/SuperInstance/vetcheck)** | **This repo** — health monitoring |
+| [shepherds-console](https://github.com/SuperInstance/shepherds-console) | Dashboard (displays vetcheck health status) |
+| [breed-registry](https://github.com/SuperInstance/breed-registry) | Breed selection (uses health certificates) |
+| [lineage-tracker](https://github.com/SuperInstance/lineage-tracker) | Lineage (health data enriches lineage records) |
+| [baton](https://github.com/SuperInstance/baton) | Generational handoff (health informs sunset decisions) |
+
+
+
+## Integration Patterns
+
+### With Breed Registry: Pre-Deployment Health Check
+
+```python
+from vetcheck import VetCheck
+from breed_registry import select_breed
+
+# Select the best model for a task
+recommended = select_breed("code_generation")
+print(f"Recommended: {recommended.recommended}")
+
+# Before deploying, verify it's healthy
+vet = VetCheck(model_id=recommended.recommended)
+cert = vet.health_certificate(validity_days=14)
+
+if cert.is_valid():
+    print(f"✓ {recommended.recommended} is certified healthy — deploy")
+else:
+    print(f"✗ {recommended.recommended} failed health check")
+    print(f"  Falling back to alternatives...")
+    for alt in recommended.alternatives:
+        alt_vet = VetCheck(model_id=alt.breed)
+        alt_cert = alt_vet.health_certificate()
+        if alt_cert.is_valid():
+            print(f"  ✓ {alt.breed} is healthy (score: {alt.score:.1f})")
+            break
+```
+
+### With Lineage Tracker: Regression Source Detection
+
+```python
+from vetcheck import VetCheck
+from lineage_tracker import LineageTracker
+
+# Model failed its physical exam
+vet = VetCheck(model_id="prod-model-v7")
+report = vet.physical_exam()
+
+if not report.passed:
+    # Check if this is a known weakness in the lineage
+    tracker = LineageTracker("lineage.json")
+    lineage = tracker.get_lineage("prod-model-v7")
+
+    print("Failed exam. Checking lineage for inherited weaknesses:")
+    for gen in lineage:
+        model = gen.model
+        failed_areas = [t.name for t in report.details if not t.passed]
+        for area in failed_areas:
+            trait_key = area.lower().replace(" ", "_")
+            if model.traits and trait_key in model.traits:
+                print(f"  Gen {gen.generation} {model.name}: {trait_key}={model.traits[trait_key]}")
+
+    # If the weakness is inherited, quarantine and recommend retraining
+    vet.quarantine(reason=f"Inherited regression: {failed_areas}")
+```
+
+### Continuous Monitoring with Scheduled Exams
+
+```python
+from vetcheck import VetCheck
+import schedule  # pip install schedule
+import time
+
+vet = VetCheck(model_id="prod-model-v7", baseline_dir="./baselines")
+
+# Daily weight check (lightweight — compares output distributions)
+schedule.every().day.at("06:00").do(lambda: vet.weight_check(window=500))
+
+# Weekly physical (full regression suite)
+schedule.every().monday.at("02:00").do(lambda: vet.physical_exam())
+
+# Monthly health certificate renewal
+schedule.every(30).days.do(lambda: vet.health_certificate(validity_days=7))
+
+# Auto-quarantine on critical failure
+def full_checkup():
+    report = vet.physical_exam()
+    if report.critical_failures > 0:
+        vet.quarantine(reason=f"Auto-quarantine: {report.critical_failures} critical failures")
+        # Alert the team
+        print(f"🚨 {vet.model_id} QUARANTINED — {report.critical_failures} critical failures")
+    return report
+
+schedule.every().day.at("12:00").do(full_checkup)
+
+# Run the scheduler
+while True:
+    schedule.run_pending()
+    time.sleep(60)
+```
+
+### Fleet Health Dashboard
+
+```python
+from vetcheck import VetCheck
+from breed_registry import BreedMatcher
+
+matcher = BreedMatcher()
+fleet = ["gpt-4", "claude-3", "llama-3", "glm", "qwen"]
+
+print("FLEET HEALTH REPORT")
+print("=" * 60)
+
+for model_id in fleet:
+    vet = VetCheck(model_id=model_id)
+    status = vet.health_status
+    quarantined = "🔒 QUARANTINED" if vet.is_quarantined else "🟢 ACTIVE"
+
+    # Quick weight check
+    drift = vet.weight_check(window=100)
+    drift_indicator = "📉" if drift.significant else "✓"
+
+    print(f"  {model_id:15s} {status.value:20s} {quarantined} drift:{drift_indicator}")
+
+    # Recommend breed-specific action
+    aptitude = matcher.assess(model_id, "reasoning")
+    if status.value == "quarantined":
+        alternatives = matcher.select("reasoning").alternatives[:2]
+        print(f"    → Failover to: {[a.breed for a in alternatives]}")
+```
+
+## Monitoring Philosophy
+
+The veterinary metaphor runs deeper than naming. In real animal husbandry:
+
+- **Preventative care is cheaper than emergency treatment.** A daily weight check catches drift before it becomes a regression. A weekly physical catches regressions before they reach users.
+- **Quarantine is not punishment.** It's protection — for the model (no traffic to handle while recovering) and for the system (no degraded output reaching users). The quarantine is lifted the moment a clean physical passes.
+- **Health certificates expire.** A model that was healthy last month might not be healthy today. Certificate expiry isn't bureaucracy — it's the acknowledgment that model health is dynamic, not static.
+- **Breed-specific screening matters.** A Thoroughbred (GPT-4) needs different tests than a Mustang (Llama-3). Custom test suites per breed catch breed-specific failure modes that generic tests miss.
+- **The vet's authority overrides the registry.** A model with the best breed score should not be in production if its health certificate has expired. The registry says what's best on paper; the vet says what's fit for duty right now.
+
+## License
+
+MIT — see [LICENSE](LICENSE).