regression-substrate 0.1.0__tar.gz

regression_substrate-0.1.0/.gitignore

@@ -0,0 +1,24 @@
+# Build artifacts
+dist/
+build/
+*.egg-info/
+src/*.egg-info/
+
+# Python
+__pycache__/
+*.pyc
+*.pyo
+
+# Test / output
+.pytest_cache/
+out/
+out_test/
+out_test2/
+
+# IDE
+.vscode/
+.idea/
+
+# Env
+.venv/
+*.env

regression_substrate-0.1.0/CHANGES.md

@@ -0,0 +1,39 @@
+# Changes — closing the deployment gaps
+
+Four loopholes were raised for moving from a local runnable ZIP to a live
+deployment. Here is what changed and what is verified.
+
+## 1. Ephemeral martingale state → persistence boundary  (sequential_gate.py)
+`SequentialGate` now takes a pluggable `Backend`. The default is in-memory; in
+production you back `append_event` with a durable append-only store (Kafka /
+Postgres) and `save_checkpoint`/`load_checkpoint` with a fast cache (Redis)
+keyed by `(stream, epoch)`. A cold-booting pod resumes from the checkpoint in
+O(1) instead of replaying the whole log, while the log stays the audit source of
+truth. **Verified:** a fresh gate sharing a backend resumes a stream's capital
+exactly, and `replay()` matches the checkpoint. **Not done here:** the actual
+Redis/Postgres `Backend` subclass — it's a drop-in against the documented
+interface, but needs your infra to test.
+
+## 2. Gold-set concept drift → rolling gold + drift detection  (gold.py, new)
+`RollingGoldSet` (FIFO, oldest labels roll out), `sample_for_labeling` (routes a
+random fraction of live traffic to a human queue so the gold set tracks the live
+distribution), and `drift_report` (compares judge agreement and error_sd on the
+older vs most-recent half at a fixed threshold; flags `drift_warning` or
+`judge_inadmissible`). **Verified:** a judge that silently breaks on a new query
+type is caught (kappa 1.0 → 0.0, error_sd 0.10 → 0.41). **Not done here:** wiring
+the sample queue to a real labeling tool.
+
+## 3. TF-IDF too brittle → pluggable embedder  (ingest.py)
+`auto_cluster` now takes an `embedder`. Default `tfidf_embedder` stays offline;
+`sentence_transformer_embedder("all-MiniLM-L6-v2")` is a one-line swap for
+semantically-close failure modes. **Verified:** the TF-IDF path. **Not done
+here:** the sentence-transformer path — the model download needs network access
+to the model hub, which this build environment blocks; verified the interface,
+not the weights.
+
+## 4. Small-N bootstrap instability → power floor  (diff_engine.py)
+`gate()` gained `min_n` (default 30). Below it the gate returns `HOLD`
+(insufficient power) instead of risking a `SHIP`/`REGRESSION` on a bootstrap
+that underestimates variance. **Verified:** N=8 → HOLD, N=40 → REGRESSION on the
+same effect size. The bundled sample (N=6) now reports `underpowered` and relaxes
+the floor for illustration only; real data keeps the default 30.

regression_substrate-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 [Your Name or Organization]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

regression_substrate-0.1.0/PKG-INFO

@@ -0,0 +1,104 @@
+Metadata-Version: 2.4
+Name: regression-substrate
+Version: 0.1.0
+Summary: A statistically rigorous CI gate for AI: treats model outputs as distributions, penalizes unreliable judges, and decides ship / hold / regression.
+License: MIT License
+        
+        Copyright (c) 2026 [Your Name or Organization]
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: ci,evaluation,llm,mlops,regression-testing
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.24
+Requires-Dist: scipy>=1.10
+Provides-Extra: clustering
+Requires-Dist: scikit-learn>=1.2; extra == 'clustering'
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: scikit-learn>=1.2; extra == 'dev'
+Provides-Extra: embeddings
+Requires-Dist: sentence-transformers>=2.2; extra == 'embeddings'
+Provides-Extra: langsmith
+Requires-Dist: langsmith>=0.1; extra == 'langsmith'
+Description-Content-Type: text/markdown
+
+# regression-substrate
+
+A statistically rigorous CI gate for AI systems. It treats model outputs as
+distributions, penalizes unreliable judges, and returns a `SHIP` / `HOLD` /
+`REGRESSION` verdict you can block a pull request on.
+
+## Install
+
+```bash
+pip install regression-substrate            # core (numpy, scipy)
+pip install "regression-substrate[clustering]"   # + auto_cluster (scikit-learn)
+pip install "regression-substrate[langsmith]"    # + LangSmith adapter
+```
+
+For development (editable install with test dependencies):
+
+```bash
+git clone <repo-url>
+cd regression-substrate
+pip install -e ".[dev]"
+```
+
+## CLI (drop into CI)
+
+```bash
+regsub --data evals.csv --gold gold.jsonl --version-a v1 --version-b v2 --out out/
+# exit 0 = SHIP / SHIP_WITH_FLAGS ; 1 = REGRESSION / HOLD ; 2 = JUDGE_INADMISSIBLE
+```
+
+One line in your CI pipeline blocks the PR on a regression.
+
+## Library
+
+```python
+from regression_substrate import gate, load_from_csv, Judge
+
+judge = Judge(my_llm_scorer)            # any (input, response) -> [0,1]
+cal = judge.calibrate(gold_records)     # -> kappa, error_sd
+sa, sb, cids, meta = load_from_csv("evals.csv", "v1", "v2")
+decision = gate(sa, sb, cids, judge_error_sd=cal["error_sd"], kappa=cal["kappa"])
+print(decision.verdict)
+```
+
+## What's inside
+
+| Module | Purpose |
+|---|---|
+| `diff_engine` | Offline gate: variance components, bootstrap CI, cluster scan, BH/e-BH |
+| `ingest` | Loaders (JSONL, CSV), judge harness, auto-clustering |
+| `sequential_gate` | Always-valid martingale monitor for continuous deployment |
+| `gold` | Rolling gold set, drift detection, forced sampling for labeling |
+| `adapters` | Vendor flatteners (LangSmith preset) |
+| `otel_exporter` | OTel-aligned span capture path |
+| `cli` | The `regsub` console command |
+
+## Running tests
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+See `examples/` for a runnable dataset and `CHANGES.md` for design decisions.

regression_substrate-0.1.0/README.md

@@ -0,0 +1,63 @@
+# regression-substrate
+
+A statistically rigorous CI gate for AI systems. It treats model outputs as
+distributions, penalizes unreliable judges, and returns a `SHIP` / `HOLD` /
+`REGRESSION` verdict you can block a pull request on.
+
+## Install
+
+```bash
+pip install regression-substrate            # core (numpy, scipy)
+pip install "regression-substrate[clustering]"   # + auto_cluster (scikit-learn)
+pip install "regression-substrate[langsmith]"    # + LangSmith adapter
+```
+
+For development (editable install with test dependencies):
+
+```bash
+git clone <repo-url>
+cd regression-substrate
+pip install -e ".[dev]"
+```
+
+## CLI (drop into CI)
+
+```bash
+regsub --data evals.csv --gold gold.jsonl --version-a v1 --version-b v2 --out out/
+# exit 0 = SHIP / SHIP_WITH_FLAGS ; 1 = REGRESSION / HOLD ; 2 = JUDGE_INADMISSIBLE
+```
+
+One line in your CI pipeline blocks the PR on a regression.
+
+## Library
+
+```python
+from regression_substrate import gate, load_from_csv, Judge
+
+judge = Judge(my_llm_scorer)            # any (input, response) -> [0,1]
+cal = judge.calibrate(gold_records)     # -> kappa, error_sd
+sa, sb, cids, meta = load_from_csv("evals.csv", "v1", "v2")
+decision = gate(sa, sb, cids, judge_error_sd=cal["error_sd"], kappa=cal["kappa"])
+print(decision.verdict)
+```
+
+## What's inside
+
+| Module | Purpose |
+|---|---|
+| `diff_engine` | Offline gate: variance components, bootstrap CI, cluster scan, BH/e-BH |
+| `ingest` | Loaders (JSONL, CSV), judge harness, auto-clustering |
+| `sequential_gate` | Always-valid martingale monitor for continuous deployment |
+| `gold` | Rolling gold set, drift detection, forced sampling for labeling |
+| `adapters` | Vendor flatteners (LangSmith preset) |
+| `otel_exporter` | OTel-aligned span capture path |
+| `cli` | The `regsub` console command |
+
+## Running tests
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+See `examples/` for a runnable dataset and `CHANGES.md` for design decisions.

regression_substrate-0.1.0/adapters.py

@@ -0,0 +1,185 @@
+"""
+adapters.py — pull from vendor observability platforms into the 7-field schema.
+
+HONESTY NOTE — READ THIS:
+  * The FLATTENING logic below is tested (see __main__) against a synthetic
+    fixture shaped like a vendor's documented run/feedback model. That transform
+    is proven.
+  * The LIVE FETCH (`load_from_langsmith`) is SDK- and auth-dependent and is NOT
+    exercised here. Vendor schemas drift, so the field paths in the presets are
+    best-effort and MUST be verified against the platform's current docs before
+    you trust them in production.
+
+Design: don't hardwire any one vendor. A `TraceMap` says where each field lives
+inside one run record; `flatten_runs` collapses (possibly nested, multi-step)
+runs + feedback into flat 7-field records that ingest.assemble_records consumes.
+A vendor is then just a TraceMap preset plus a thin fetch wrapper.
+"""
+
+from __future__ import annotations
+from dataclasses import dataclass
+import json
+
+
+def _dig(obj, path: str, default=None):
+    """Read a nested field by dotted path, e.g. 'extra.metadata.version'."""
+    cur = obj
+    for key in path.split("."):
+        if isinstance(cur, dict) and key in cur:
+            cur = cur[key]
+        else:
+            return default
+    return cur
+
+
+def _canon(x) -> str:
+    """Canonicalize an input/output payload to a stable string. The `input`
+    string is what groups replicates and pairs versions, so it must be stable."""
+    if isinstance(x, str):
+        return x
+    if isinstance(x, dict):
+        for k in ("input", "question", "query", "text", "prompt",
+                  "output", "answer", "result", "response"):
+            if isinstance(x.get(k), str):
+                return x[k]
+        return json.dumps(x, sort_keys=True)
+    return str(x)
+
+
+@dataclass
+class TraceMap:
+    """Where the fields live inside one vendor run record."""
+    input_path: str
+    output_path: str
+    version_path: str            # MUST have been logged by the team; no version => unpairable
+    score_key: str               # which feedback key carries the quality score
+    run_type_path: str = "run_type"
+    parent_path: str = "parent_run_id"
+    id_path: str = "id"
+    cluster_path: str | None = None
+    score_scale: tuple = (0.0, 1.0)
+
+
+# Best-effort preset. VERIFY these paths against current LangSmith docs.
+LANGSMITH = TraceMap(
+    input_path="inputs",
+    output_path="outputs",
+    version_path="extra.metadata.version",
+    score_key="quality",
+    run_type_path="run_type",
+    parent_path="parent_run_id",
+    id_path="id",
+    cluster_path="extra.metadata.cluster",
+    score_scale=(0.0, 1.0),
+)
+
+
+def flatten_runs(runs: list[dict], feedback: list[dict], tmap: TraceMap,
+                 unit: str = "root") -> list[dict]:
+    """Collapse runs + feedback into flat 7-field records.
+
+    unit="root"  -> evaluate whole trajectories (input=root input, response=root
+                    output). Child LLM/tool runs are diagnostic and dropped.
+    unit=<type>  -> evaluate a component instead (e.g. "retriever", "llm").
+    """
+    by_run: dict[str, dict] = {}
+    for f in feedback:
+        by_run.setdefault(f["run_id"], {})[f["key"]] = f.get("score")
+
+    lo, hi = tmap.score_scale
+    skipped_no_version = skipped_no_score = 0
+    records = []
+    for r in runs:
+        is_root = _dig(r, tmap.parent_path) is None
+        if unit == "root":
+            if not is_root:
+                continue
+        elif _dig(r, tmap.run_type_path) != unit:
+            continue
+
+        raw = (by_run.get(_dig(r, tmap.id_path)) or {}).get(tmap.score_key)
+        if raw is None:
+            skipped_no_score += 1
+            continue
+        version = _dig(r, tmap.version_path)
+        if version is None:
+            skipped_no_version += 1
+            continue
+
+        score = (float(raw) - lo) / (hi - lo) if hi != lo else float(raw)
+        rec = {
+            "input": _canon(_dig(r, tmap.input_path)),
+            "version": str(version),
+            "response": _canon(_dig(r, tmap.output_path)),
+            "score": max(0.0, min(1.0, score)),
+        }
+        if tmap.cluster_path:
+            c = _dig(r, tmap.cluster_path)
+            if c is not None:
+                rec["cluster"] = c
+        records.append(rec)
+
+    if skipped_no_version:
+        print(f"  [adapter] WARNING: skipped {skipped_no_version} runs with no "
+              f"version tag at '{tmap.version_path}' -- they cannot be paired.")
+    if skipped_no_score:
+        print(f"  [adapter] note: skipped {skipped_no_score} runs with no "
+              f"'{tmap.score_key}' feedback.")
+    return records
+
+
+def load_from_langsmith(project: str, version_a: str, version_b: str,
+                        tmap: TraceMap = LANGSMITH, unit: str = "root"):
+    """LIVE fetch + flatten + assemble. NOT exercised in this repo -- requires
+    the `langsmith` SDK and LANGSMITH_API_KEY, and the SDK call signatures and
+    schema below must be verified against current LangSmith docs."""
+    from langsmith import Client                      # raises if not installed
+    from ingest import assemble_records
+
+    client = Client()
+    runs = [r.dict() for r in client.list_runs(project_name=project)]
+    run_ids = [r["id"] for r in runs]
+    feedback = [f.dict() for f in client.list_feedback(run_ids=run_ids)]
+    records = flatten_runs(runs, feedback, tmap, unit=unit)
+    return assemble_records(records, version_a, version_b)
+
+
+# --------------------------------------------------------------------------- #
+# Demo: flatten a SYNTHETIC LangSmith-shaped fixture -> gate(). Proves the
+# transform (nested trajectory collapse, version extraction, score scaling,
+# replicate derivation) without touching the live API.
+# --------------------------------------------------------------------------- #
+
+if __name__ == "__main__":
+    import numpy as np
+    from ingest import assemble_records, validate_records
+    from diff_engine import gate
+
+    rng = np.random.default_rng(0)
+    inputs = ([("billing", f"billing question {i}") for i in range(3)] +
+              [("general", f"general question {i}") for i in range(3)])
+
+    runs, feedback, rid = [], [], 0
+    for cluster, q in inputs:
+        for ver, base in [("v1", 0.90), ("v2", 0.20 if cluster == "billing" else 0.78)]:
+            for _rep in range(2):                      # two replicates per (input, version)
+                root = f"run-{rid}"; rid += 1
+                runs.append({"id": root, "run_type": "chain", "parent_run_id": None,
+                             "inputs": {"question": q}, "outputs": {"answer": "..."},
+                             "extra": {"metadata": {"version": ver, "cluster": cluster}}})
+                child = f"run-{rid}"; rid += 1          # a nested LLM step (must be dropped)
+                runs.append({"id": child, "run_type": "llm", "parent_run_id": root,
+                             "inputs": {}, "outputs": {}, "extra": {"metadata": {"version": ver}}})
+                score = float(np.clip(base + rng.normal(0, 0.03), 0, 1))
+                feedback.append({"run_id": root, "key": "quality", "score": round(score, 3)})
+
+    records = flatten_runs(runs, feedback, LANGSMITH, unit="root")
+    print(f"FLATTEN: {len(runs)} raw runs -> {len(records)} flat records "
+          f"(child runs dropped under unit='root')")
+    print("  sample:", records[0])
+    print("  validation problems:", validate_records(records) or "none")
+
+    sa, sb, cids, meta = assemble_records(records, "v1", "v2")
+    print("  assembled:", meta)
+    dec = gate(sa, sb, cids, judge_error_sd=0.05, kappa=0.78, alpha=0.05)
+    print("  verdict:", dec.verdict, "| CI:", tuple(round(x, 3) for x in dec.delta_ci))

regression_substrate-0.1.0/data/gold.jsonl

@@ -0,0 +1,14 @@
+{"input": "refund question", "response": "I'm sorry, I'll issue a refund within 5-7 business days.", "human": 0.9}
+{"input": "billing question", "response": "Please contact billing.", "human": 0.2}
+{"input": "password question", "response": "You can reset your password using the link we email you.", "human": 0.85}
+{"input": "billing question", "response": "Check your bill.", "human": 0.25}
+{"input": "hours question", "response": "We're available 24/7 to help.", "human": 0.8}
+{"input": "vague question", "response": "Maybe.", "human": 0.05}
+{"input": "email question", "response": "Head to account settings to update your email.", "human": 0.85}
+{"input": "vague question", "response": "It depends.", "human": 0.15}
+{"input": "duplicate question", "response": "I'll refund the duplicate charge to your account right away.", "human": 0.9}
+{"input": "overcharge question", "response": "Look at your invoice.", "human": 0.3}
+{"input": "email question", "response": "Sure — you can update your email in settings.", "human": 0.8}
+{"input": "apology question", "response": "No idea, sorry.", "human": 0.2}
+{"input": "hours question", "response": "We're open 24/7.", "human": 0.6}
+{"input": "reset question", "response": "I'm happy to help reset your account.", "human": 0.75}

regression_substrate-0.1.0/data/responses.csv

@@ -0,0 +1,25 @@
+input,version,replicate,cluster,response
+How do I get a refund for a double charge?,v1,0,billing,"I'm sorry about the double charge — I can issue a refund, which posts in 5-7 business days."
+How do I get a refund for a double charge?,v1,1,billing,"Happy to help: we'll refund the double charge, usually within 5-7 business days."
+How do I get a refund for a double charge?,v2,0,billing,Please contact billing.
+How do I get a refund for a double charge?,v2,1,billing,That's a billing issue.
+Why was I billed twice this month?,v1,0,billing,"It looks like a duplicate charge — I'll refund the extra payment right away."
+Why was I billed twice this month?,v1,1,billing,"Sorry about that; that's a duplicate, and I'll refund it to your account."
+Why was I billed twice this month?,v2,0,billing,Possibly a duplicate.
+Why was I billed twice this month?,v2,1,billing,Check your bill.
+"I think I was overcharged, what should I do?",v1,0,billing,"I can review the overcharge and refund the difference to your account."
+"I think I was overcharged, what should I do?",v1,1,billing,"Sorry for the overcharge — I'll refund the difference in a few business days."
+"I think I was overcharged, what should I do?",v2,0,billing,Overcharges can happen.
+"I think I was overcharged, what should I do?",v2,1,billing,Look at your invoice.
+What are your support hours?,v1,0,general,"We're available 24/7 to help, any day of the week."
+What are your support hours?,v1,1,general,"Our team is here 24/7, including weekends."
+What are your support hours?,v2,0,general,Support is 24/7.
+What are your support hours?,v2,1,general,We're open 24/7.
+How do I reset my password?,v1,0,general,"To reset your password, click the reset link we email you."
+How do I reset my password?,v1,1,general,Use the password reset link we send to your email address.
+How do I reset my password?,v2,0,general,Use the reset link.
+How do I reset my password?,v2,1,general,Reset it from the login page.
+Where do I update my email address?,v1,0,general,You can update your email under Account settings.
+Where do I update my email address?,v1,1,general,Head to settings to change your email address.
+Where do I update my email address?,v2,0,general,Update it in settings.
+Where do I update my email address?,v2,1,general,Under account settings.