closed-loop-default-detection 0.2.0__tar.gz → 0.3.0__tar.gz

{closed_loop_default_detection-0.2.0/src/closed_loop_default_detection.egg-info → closed_loop_default_detection-0.3.0}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.4
 Name: closed-loop-default-detection
-Version: 0.2.0
+Version: 0.3.0
 Summary: Closed loop that measures selective-labels default detection on synthetic SMB lending cohorts and finds the PD model's operating frontier.
-Author-email: Hossain Pazooki <hossain@pazooki.com>
+Author: Hossain Pazooki
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/hossainpazooki/closed-loop-default-detection
 Project-URL: Repository, https://github.com/hossainpazooki/closed-loop-default-detection
@@ -148,6 +148,35 @@ seed 42, 10% budget, severity 0.6): the lever buys **157 labels for $439,578**
 in dollars a lender can see, on the only lever that buys it rather than reweighting what is
 already identified.
 
+## Pricing the feedback loop (v3)
+
+`v0.3.0` prices the **dynamic** regime, where the model's own approvals create its next
+training set: three arms (treatment = retrain each generation; **frozen** = generation-0
+model deployed forever; **prior** = prior-policy funding), 25 spaced seeds × 3 severities ×
+2 exploration rates × 12 generations = **450 runs**
+(`artifacts/feedback_profit_sweep.csv`), with three pre-registered hypotheses tested as
+within-seed paired contrasts on realized funded-book P&L at severity 0.4, Holm-corrected.
+Every number below recomputes via `python scripts/feedback_sweep_stats.py`. All of it
+consumes **planted, risk-unlinked default timing** — a verified *experiment*, not a
+verified *result*.
+
+| Hypothesis (severity 0.4, gens 1–11) | Median paired deficit | Sign | Verdict |
+|---|---|---|---|
+| H1 feedback accumulation costs money (treatment − frozen, ε=0) | **−0.0028** | 24/25 | **confirmed** (p_holm 2.3e-06, clears the 0.0018 noise floor) |
+| H2 the policy switch costs money (frozen − prior, ε=0) | +0.0119 | 0/25 | **not confirmed — measured opposite in direction** |
+| H3 exploration buys profit back (treatment ε=.05 − ε=0) | −0.0024 | 0/25 | **not confirmed** |
+
+The decomposition v3 exists for came back one-for-three: letting the loop close on its own
+labels measurably costs the book (H1), but at these severities the *switch* to model-based
+funding is profitable relative to the prior policy (H2's predicted sign was wrong — reported
+as measured, not softened), and 5% exploration does not pay for itself in-window (H3; the
+policy-book-only secondary H3a is negative too). The **H4 integrity control** — frozen-arm
+paired difference must equal the explored slice's own P&L *exactly* — holds at ≤1e-10 across
+all 900 checked rows, and the answer to the v2-banked question is re-verified at 25 seeds:
+the declined-ECE alarm fires at generation 1–2 in every ε=0 run (severity 0.4: 25/25 at
+generation 1). Full mechanics in [`docs/how-it-works.md`](docs/how-it-works.md); gates in
+[`docs/validation.md`](docs/validation.md).
+
 ## Install
 
 ```bash

{closed_loop_default_detection-0.2.0 → closed_loop_default_detection-0.3.0}/README.md

@@ -114,6 +114,35 @@ seed 42, 10% budget, severity 0.6): the lever buys **157 labels for $439,578**
 in dollars a lender can see, on the only lever that buys it rather than reweighting what is
 already identified.
 
+## Pricing the feedback loop (v3)
+
+`v0.3.0` prices the **dynamic** regime, where the model's own approvals create its next
+training set: three arms (treatment = retrain each generation; **frozen** = generation-0
+model deployed forever; **prior** = prior-policy funding), 25 spaced seeds × 3 severities ×
+2 exploration rates × 12 generations = **450 runs**
+(`artifacts/feedback_profit_sweep.csv`), with three pre-registered hypotheses tested as
+within-seed paired contrasts on realized funded-book P&L at severity 0.4, Holm-corrected.
+Every number below recomputes via `python scripts/feedback_sweep_stats.py`. All of it
+consumes **planted, risk-unlinked default timing** — a verified *experiment*, not a
+verified *result*.
+
+| Hypothesis (severity 0.4, gens 1–11) | Median paired deficit | Sign | Verdict |
+|---|---|---|---|
+| H1 feedback accumulation costs money (treatment − frozen, ε=0) | **−0.0028** | 24/25 | **confirmed** (p_holm 2.3e-06, clears the 0.0018 noise floor) |
+| H2 the policy switch costs money (frozen − prior, ε=0) | +0.0119 | 0/25 | **not confirmed — measured opposite in direction** |
+| H3 exploration buys profit back (treatment ε=.05 − ε=0) | −0.0024 | 0/25 | **not confirmed** |
+
+The decomposition v3 exists for came back one-for-three: letting the loop close on its own
+labels measurably costs the book (H1), but at these severities the *switch* to model-based
+funding is profitable relative to the prior policy (H2's predicted sign was wrong — reported
+as measured, not softened), and 5% exploration does not pay for itself in-window (H3; the
+policy-book-only secondary H3a is negative too). The **H4 integrity control** — frozen-arm
+paired difference must equal the explored slice's own P&L *exactly* — holds at ≤1e-10 across
+all 900 checked rows, and the answer to the v2-banked question is re-verified at 25 seeds:
+the declined-ECE alarm fires at generation 1–2 in every ε=0 run (severity 0.4: 25/25 at
+generation 1). Full mechanics in [`docs/how-it-works.md`](docs/how-it-works.md); gates in
+[`docs/validation.md`](docs/validation.md).
+
 ## Install
 
 ```bash

{closed_loop_default_detection-0.2.0 → closed_loop_default_detection-0.3.0}/pyproject.toml

@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "closed-loop-default-detection"
-version = "0.2.0"
+version = "0.3.0"
 description = "Closed loop that measures selective-labels default detection on synthetic SMB lending cohorts and finds the PD model's operating frontier."
 readme = "README.md"
 requires-python = ">=3.10"
 license = "MIT"
 license-files = ["LICENSE"]
-authors = [{ name = "Hossain Pazooki", email = "hossain@pazooki.com" }]
+authors = [{ name = "Hossain Pazooki" }]
 keywords = [
     "selective-labels",
     "probability-of-default",

{closed_loop_default_detection-0.2.0 → closed_loop_default_detection-0.3.0}/src/cldd/emp.py

@@ -49,6 +49,7 @@ __all__ = [
     "emp_harness",
     "loss_fractions",
     "realized_profits",
+    "realized_book_profit",
 ]
 
 
@@ -304,6 +305,52 @@ def realized_profits(
     return profit
 
 
+def realized_book_profit(
+    y_true: np.ndarray,
+    default_day: np.ndarray | None,
+    requested_amount: np.ndarray,
+    funded: np.ndarray,
+    economics: LoanEconomics | None = None,
+) -> float | None:
+    """Realized book P&L of the funded rows, normalized by full-cohort exposure.
+
+    Reuses :func:`realized_profits` priced over the FULL cohort, then sums the
+    funded rows -- so day-90 rows take the decision-5 mean-body imputation with
+    the mean computed from the *cohort's* body defaulters (lambda = 1 fallback
+    when the cohort body is empty, see :func:`loss_fractions`). The basis is a
+    planted cohort property, invariant to who funds: pricing on the funded
+    slice's own body mean made book P&L non-additive across slices and broke
+    the H4 arithmetic identity at the pilot gate (spec Rev 2.1 amendment,
+    2026-07-19). Returns ``None`` when timing columns are absent (flat cohorts,
+    ``default_day is None``), mirroring :func:`emp_harness`. **SCM-only, like
+    everything in v3.**
+
+    Units: total realized profit of funded rows / (n_applicants * mean
+    ``requested_amount`` over the FULL cohort, not funded-only) -- profit per
+    applicant as a fraction of cohort mean exposure (v2 decision 6 extended);
+    the full-cohort denominator keeps arms with different funded counts
+    comparable and paired differences well-defined.
+
+    Pure numpy, zero RNG, deterministic.
+    """
+    if default_day is None:
+        return None
+
+    y_true = np.asarray(y_true, dtype=float)
+    default_day = np.asarray(default_day, dtype=float)
+    requested_amount = np.asarray(requested_amount, dtype=float)
+    funded = np.asarray(funded, dtype=bool)
+    n = y_true.shape[0]
+
+    mean_a = float(np.mean(requested_amount)) if n > 0 else 0.0
+    denom = n * mean_a
+    if denom == 0.0:
+        return float("nan")
+
+    profits = realized_profits(y_true, default_day, requested_amount, economics)
+    return float(np.sum(profits[funded])) / denom
+
+
 def emp_harness(
     y_true: np.ndarray,
     scores: np.ndarray,

{closed_loop_default_detection-0.2.0 → closed_loop_default_detection-0.3.0}/src/cldd/feedback.py

@@ -23,6 +23,15 @@ correction (IPW, retrain) is undefined on the declined side.
   labeled-propensity weights — the identification-buying lever from the static
   loop, now acting as a *stabilizer* of the feedback dynamics.
 
+v3 (docs/superpowers/specs/2026-07-14-cldd-v3-design.md) adds two additive flags,
+byte-exact at their defaults, that isolate the trajectory's causes into three
+arms: ``retrain=False`` freezes the generation-0 model forever (isolates
+feedback accumulation — the "frozen" arm), and ``policy_mode="prior"`` funds
+every generation via the cohort's own prior-policy ``approved`` column while
+the model still trains for its metrics (the noise-floor "prior" arm).
+``retrain=True, policy_mode="model"`` (the defaults) is today's "treatment"
+behavior, unchanged.
+
 Per generation we record the planted-truth calibration on the declined (never
 labeled) side, the *observable* funded-book performance, and the observable
 positivity diagnostics — so the harness can show whether the loop's blind spot
@@ -39,7 +48,7 @@ from dataclasses import dataclass, field
 
 import numpy as np
 
-from . import config, eval_default, model_pd
+from . import config, emp, eval_default, model_pd
 from .diagnostics import PositivityDiagnostics, positivity_diagnostics
 from .loop import LeverMetrics, make_generator
 
@@ -64,6 +73,14 @@ class GenerationResult:
     diagnostics: PositivityDiagnostics
     n_explored: int = 0
     explored_defaults: int = 0
+    #: Realized funded-book P&L (v3), per applicant as a fraction of the FULL
+    #: cohort's mean ``requested_amount`` -- see ``emp.realized_book_profit``.
+    #: ``None`` when the cohort carries no planted default timing (flat).
+    book_profit: float | None = None
+    #: Realized P&L of the explored slice alone, same full-cohort denominator
+    #: as ``book_profit``. ``0.0`` when nothing was explored this generation;
+    #: ``None`` under the same no-timing condition as ``book_profit``.
+    explored_profit: float | None = None
 
 
 @dataclass
@@ -94,11 +111,15 @@ class FeedbackLoop:
         approval_rate: float = config.DEFAULT_APPROVAL_RATE,
         seed: int = config.RANDOM_SEED,
         policy_threshold: float = config.POLICY_PD_THRESHOLD,
+        retrain: bool = True,
+        policy_mode: str = "model",
     ) -> None:
         if not 0.0 <= exploration_rate < 1.0:
             raise ValueError(f"exploration_rate must be in [0, 1); got {exploration_rate!r}")
         if n_generations < 1:
             raise ValueError(f"n_generations must be >= 1; got {n_generations!r}")
+        if policy_mode not in ("model", "prior"):
+            raise ValueError(f"policy_mode must be 'model' or 'prior'; got {policy_mode!r}")
         self.selection_severity = selection_severity
         self.n_generations = n_generations
         self.exploration_rate = exploration_rate
@@ -107,6 +128,15 @@ class FeedbackLoop:
         self.approval_rate = approval_rate
         self.seed = seed
         self.policy_threshold = policy_threshold
+        #: ``False`` (v3 "frozen" arm): keep the generation-0 model deployed
+        #: unchanged forever -- no generation >= 1 ever calls
+        #: ``model_pd.train_pd_model`` again. Byte-exact default is ``True``.
+        self.retrain = retrain
+        #: ``"prior"`` (v3 "prior" arm): fund every generation via the
+        #: cohort's own prior-policy ``approved`` column; the model still
+        #: trains each generation (for its metrics) but never decides
+        #: funding. Byte-exact default is ``"model"`` (today's behavior).
+        self.policy_mode = policy_mode
 
     # ------------------------------------------------------------------ #
 
@@ -152,29 +182,54 @@ class FeedbackLoop:
             X = cohort["features"].to_numpy(dtype=float)
             y = cohort["true_default"]
 
-            if model is None:
+            # ``policy`` label semantics are unchanged by ``policy_mode``: it
+            # reflects whether a model has been trained yet (generation 0 is
+            # always "prior"; generation >= 1 is always "model"), even when
+            # policy_mode="prior" means the model isn't the one FUNDING.
+            policy = "prior" if model is None else "model"
+
+            if self.policy_mode == "prior":
+                policy_funded = cohort["approved"]
+            elif model is None:
                 policy_funded = cohort["approved"]
-                policy = "prior"
             else:
                 policy_funded = self._model_policy(model, X)
-                policy = "model"
 
-            explored = self._explore(policy_funded, generation)
+            # Frozen arm: gen-0 training must be eps-invariant (the H4
+            # identity requires the deployed model itself not to depend on
+            # eps), so exploration is suppressed at generation 0 only when
+            # retrain=False. This never fires on the default retrain=True
+            # path -- the EXPLORE_STREAM_FEEDBACK draw and its call site are
+            # untouched there.
+            if not self.retrain and generation == 0:
+                explored = np.zeros_like(policy_funded)
+            else:
+                explored = self._explore(policy_funded, generation)
             funded = policy_funded | explored
 
             # Diagnose the POLICY selection (pre-exploration): that is the
             # selection a deployment would be monitoring.
             diag = positivity_diagnostics(X, policy_funded, random_state=self.seed + generation)
 
-            if self.exploration_rate > 0.0:
-                weights = np.where(explored, 1.0 / self.exploration_rate, 1.0)[funded]
-            else:
-                weights = None
-            model = model_pd.train_pd_model(
-                X[funded], y[funded], sample_weight=weights, random_state=self.seed + generation
-            )
+            # retrain=True (default): train every generation, byte-identical
+            # to pre-v3 behavior. retrain=False (frozen arm): train only once,
+            # at generation 0 (model is None), then reuse it forever.
+            if self.retrain or model is None:
+                if self.exploration_rate > 0.0:
+                    weights = np.where(explored, 1.0 / self.exploration_rate, 1.0)[funded]
+                else:
+                    weights = None
+                model = model_pd.train_pd_model(
+                    X[funded], y[funded], sample_weight=weights, random_state=self.seed + generation
+                )
 
             scored = eval_default.score_pd_detection(model, cohort, self.policy_threshold, funded=funded)
+
+            default_day = cohort.get("days_to_default")
+            requested_amount = cohort["features"]["requested_amount"].to_numpy(dtype=float)
+            book_profit = emp.realized_book_profit(y, default_day, requested_amount, funded)
+            explored_profit = emp.realized_book_profit(y, default_day, requested_amount, explored)
+
             result.generations.append(
                 GenerationResult(
                     generation=generation,
@@ -185,6 +240,8 @@ class FeedbackLoop:
                     diagnostics=diag,
                     n_explored=int(explored.sum()),
                     explored_defaults=int(y[explored].sum()),
+                    book_profit=book_profit,
+                    explored_profit=explored_profit,
                 )
             )
 

{closed_loop_default_detection-0.2.0 → closed_loop_default_detection-0.3.0/src/closed_loop_default_detection.egg-info}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.4
 Name: closed-loop-default-detection
-Version: 0.2.0
+Version: 0.3.0
 Summary: Closed loop that measures selective-labels default detection on synthetic SMB lending cohorts and finds the PD model's operating frontier.
-Author-email: Hossain Pazooki <hossain@pazooki.com>
+Author: Hossain Pazooki
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/hossainpazooki/closed-loop-default-detection
 Project-URL: Repository, https://github.com/hossainpazooki/closed-loop-default-detection
@@ -148,6 +148,35 @@ seed 42, 10% budget, severity 0.6): the lever buys **157 labels for $439,578**
 in dollars a lender can see, on the only lever that buys it rather than reweighting what is
 already identified.
 
+## Pricing the feedback loop (v3)
+
+`v0.3.0` prices the **dynamic** regime, where the model's own approvals create its next
+training set: three arms (treatment = retrain each generation; **frozen** = generation-0
+model deployed forever; **prior** = prior-policy funding), 25 spaced seeds × 3 severities ×
+2 exploration rates × 12 generations = **450 runs**
+(`artifacts/feedback_profit_sweep.csv`), with three pre-registered hypotheses tested as
+within-seed paired contrasts on realized funded-book P&L at severity 0.4, Holm-corrected.
+Every number below recomputes via `python scripts/feedback_sweep_stats.py`. All of it
+consumes **planted, risk-unlinked default timing** — a verified *experiment*, not a
+verified *result*.
+
+| Hypothesis (severity 0.4, gens 1–11) | Median paired deficit | Sign | Verdict |
+|---|---|---|---|
+| H1 feedback accumulation costs money (treatment − frozen, ε=0) | **−0.0028** | 24/25 | **confirmed** (p_holm 2.3e-06, clears the 0.0018 noise floor) |
+| H2 the policy switch costs money (frozen − prior, ε=0) | +0.0119 | 0/25 | **not confirmed — measured opposite in direction** |
+| H3 exploration buys profit back (treatment ε=.05 − ε=0) | −0.0024 | 0/25 | **not confirmed** |
+
+The decomposition v3 exists for came back one-for-three: letting the loop close on its own
+labels measurably costs the book (H1), but at these severities the *switch* to model-based
+funding is profitable relative to the prior policy (H2's predicted sign was wrong — reported
+as measured, not softened), and 5% exploration does not pay for itself in-window (H3; the
+policy-book-only secondary H3a is negative too). The **H4 integrity control** — frozen-arm
+paired difference must equal the explored slice's own P&L *exactly* — holds at ≤1e-10 across
+all 900 checked rows, and the answer to the v2-banked question is re-verified at 25 seeds:
+the declined-ECE alarm fires at generation 1–2 in every ε=0 run (severity 0.4: 25/25 at
+generation 1). Full mechanics in [`docs/how-it-works.md`](docs/how-it-works.md); gates in
+[`docs/validation.md`](docs/validation.md).
+
 ## Install
 
 ```bash

{closed_loop_default_detection-0.2.0 → closed_loop_default_detection-0.3.0}/src/closed_loop_default_detection.egg-info/SOURCES.txt

@@ -21,6 +21,7 @@ src/closed_loop_default_detection.egg-info/SOURCES.txt
 src/closed_loop_default_detection.egg-info/dependency_links.txt
 src/closed_loop_default_detection.egg-info/requires.txt
 src/closed_loop_default_detection.egg-info/top_level.txt
+tests/test_book_profit.py
 tests/test_correctors.py
 tests/test_counterfactual.py
 tests/test_diagnostics.py
@@ -28,6 +29,7 @@ tests/test_emp.py
 tests/test_eval_default.py
 tests/test_exploration.py
 tests/test_feedback.py
+tests/test_feedback_flags.py
 tests/test_fidelity.py
 tests/test_fidelity_report.py
 tests/test_loop.py
@@ -36,4 +38,5 @@ tests/test_model_pd.py
 tests/test_reject_inference.py
 tests/test_scm_gating.py
 tests/test_sklearn_compat.py
+tests/test_sweep_stats.py
 tests/test_synthetic.py