consequence-ledger 0.3.0__tar.gz → 0.3.1__tar.gz

{consequence_ledger-0.3.0 → consequence_ledger-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: consequence-ledger
-Version: 0.3.0
+Version: 0.3.1
 Summary: Measure what AI coding decisions actually cause: randomized routing, delayed outcome labels, causal estimates.
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/sujan-mitta/consequence-ledger
@@ -38,12 +38,14 @@ It is the 90-day MVP from the "AI elephant in the room" research: decision loggi
 Python 3.9+ and git 2.28+. There are no dependencies.
 
 ```bash
-git clone https://github.com/sujan-mitta/consequence-ledger.git
-cd consequence-ledger
-pip install .          # gives you the `ledger` command; or skip this and use `python -m ledger` from the folder
+pip install consequence-ledger     # gives you the `ledger` command
 ledger --version
 ```
 
+New to it? [docs/QUICKSTART.md](docs/QUICKSTART.md) takes you from install to a first report on your own repo in about five minutes,
+and [builder.html](builder.html) is a form that writes the command for you. To work from source instead:
+`git clone https://github.com/sujan-mitta/consequence-ledger.git`, then `pip install .` inside the folder.
+
 ## Quick start
 
 ```bash
@@ -103,7 +105,7 @@ python -m ledger recommend --stratum difficulty         # and what routing by th
 | `estimate` | Naive, IPW, self-normalised IPW, direct method and doubly robust estimates per arm. Bootstrap and influence-function CIs, effective sample size, minimum propensity, and a check that unmerged work is balanced across arms. `--cluster developer` resamples whole clusters when outcomes are correlated. |
 | `breakdown` | Which tool is better for which kind of task (`--stratum difficulty`). Raw and shrunk effects per stratum with Bonferroni-adjusted p-values, so a lucky stratum is not mistaken for a pattern. |
 | `recommend` | What routing each task type to its best tool would be worth. The tool is chosen on one half of your decisions and judged on the other, so the gain is not inflated; it also shows the gain from simply using the best tool. |
-| `report` | A self-contained HTML report (light and dark). |
+| `report` | A self-contained HTML report (light and dark). Includes the `breakdown` and `recommend` tables once you have run them. |
 | `audit` | Per-tool baseline from history, with Wilson CIs and size-adjusted rates. |
 | `demo` | Synthetic end-to-end run that checks labels against an oracle. |
 
@@ -169,7 +171,7 @@ Full details are in [research/FINDINGS.md](research/FINDINGS.md):
 ```
 ledger/          the package: policy (routing), label + squash (outcomes), estimate, audit, plan, status,
                  incidents + sentry + pagerduty, report, demo, db, gitio, cli
-tests/           230 unittest cases: real git repos in temp dirs, local fake servers, estimator Monte Carlo checks
+tests/           236 unittest cases: real git repos in temp dirs, local fake servers, estimator Monte Carlo checks
 tools/           coverage_lite.py, dependency-free line coverage
 docs/            METHODOLOGY.md
 research/        simulate.py (coverage, power, noise, design, peeking studies), results/, FINDINGS.md

{consequence_ledger-0.3.0 → consequence_ledger-0.3.1}/README.md

@@ -11,12 +11,14 @@ It is the 90-day MVP from the "AI elephant in the room" research: decision loggi
 Python 3.9+ and git 2.28+. There are no dependencies.
 
 ```bash
-git clone https://github.com/sujan-mitta/consequence-ledger.git
-cd consequence-ledger
-pip install .          # gives you the `ledger` command; or skip this and use `python -m ledger` from the folder
+pip install consequence-ledger     # gives you the `ledger` command
 ledger --version
 ```
 
+New to it? [docs/QUICKSTART.md](docs/QUICKSTART.md) takes you from install to a first report on your own repo in about five minutes,
+and [builder.html](builder.html) is a form that writes the command for you. To work from source instead:
+`git clone https://github.com/sujan-mitta/consequence-ledger.git`, then `pip install .` inside the folder.
+
 ## Quick start
 
 ```bash
@@ -76,7 +78,7 @@ python -m ledger recommend --stratum difficulty         # and what routing by th
 | `estimate` | Naive, IPW, self-normalised IPW, direct method and doubly robust estimates per arm. Bootstrap and influence-function CIs, effective sample size, minimum propensity, and a check that unmerged work is balanced across arms. `--cluster developer` resamples whole clusters when outcomes are correlated. |
 | `breakdown` | Which tool is better for which kind of task (`--stratum difficulty`). Raw and shrunk effects per stratum with Bonferroni-adjusted p-values, so a lucky stratum is not mistaken for a pattern. |
 | `recommend` | What routing each task type to its best tool would be worth. The tool is chosen on one half of your decisions and judged on the other, so the gain is not inflated; it also shows the gain from simply using the best tool. |
-| `report` | A self-contained HTML report (light and dark). |
+| `report` | A self-contained HTML report (light and dark). Includes the `breakdown` and `recommend` tables once you have run them. |
 | `audit` | Per-tool baseline from history, with Wilson CIs and size-adjusted rates. |
 | `demo` | Synthetic end-to-end run that checks labels against an oracle. |
 
@@ -142,7 +144,7 @@ Full details are in [research/FINDINGS.md](research/FINDINGS.md):
 ```
 ledger/          the package: policy (routing), label + squash (outcomes), estimate, audit, plan, status,
                  incidents + sentry + pagerduty, report, demo, db, gitio, cli
-tests/           230 unittest cases: real git repos in temp dirs, local fake servers, estimator Monte Carlo checks
+tests/           236 unittest cases: real git repos in temp dirs, local fake servers, estimator Monte Carlo checks
 tools/           coverage_lite.py, dependency-free line coverage
 docs/            METHODOLOGY.md
 research/        simulate.py (coverage, power, noise, design, peeking studies), results/, FINDINGS.md

{consequence_ledger-0.3.0 → consequence_ledger-0.3.1}/consequence_ledger.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: consequence-ledger
-Version: 0.3.0
+Version: 0.3.1
 Summary: Measure what AI coding decisions actually cause: randomized routing, delayed outcome labels, causal estimates.
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/sujan-mitta/consequence-ledger
@@ -38,12 +38,14 @@ It is the 90-day MVP from the "AI elephant in the room" research: decision loggi
 Python 3.9+ and git 2.28+. There are no dependencies.
 
 ```bash
-git clone https://github.com/sujan-mitta/consequence-ledger.git
-cd consequence-ledger
-pip install .          # gives you the `ledger` command; or skip this and use `python -m ledger` from the folder
+pip install consequence-ledger     # gives you the `ledger` command
 ledger --version
 ```
 
+New to it? [docs/QUICKSTART.md](docs/QUICKSTART.md) takes you from install to a first report on your own repo in about five minutes,
+and [builder.html](builder.html) is a form that writes the command for you. To work from source instead:
+`git clone https://github.com/sujan-mitta/consequence-ledger.git`, then `pip install .` inside the folder.
+
 ## Quick start
 
 ```bash
@@ -103,7 +105,7 @@ python -m ledger recommend --stratum difficulty         # and what routing by th
 | `estimate` | Naive, IPW, self-normalised IPW, direct method and doubly robust estimates per arm. Bootstrap and influence-function CIs, effective sample size, minimum propensity, and a check that unmerged work is balanced across arms. `--cluster developer` resamples whole clusters when outcomes are correlated. |
 | `breakdown` | Which tool is better for which kind of task (`--stratum difficulty`). Raw and shrunk effects per stratum with Bonferroni-adjusted p-values, so a lucky stratum is not mistaken for a pattern. |
 | `recommend` | What routing each task type to its best tool would be worth. The tool is chosen on one half of your decisions and judged on the other, so the gain is not inflated; it also shows the gain from simply using the best tool. |
-| `report` | A self-contained HTML report (light and dark). |
+| `report` | A self-contained HTML report (light and dark). Includes the `breakdown` and `recommend` tables once you have run them. |
 | `audit` | Per-tool baseline from history, with Wilson CIs and size-adjusted rates. |
 | `demo` | Synthetic end-to-end run that checks labels against an oracle. |
 
@@ -169,7 +171,7 @@ Full details are in [research/FINDINGS.md](research/FINDINGS.md):
 ```
 ledger/          the package: policy (routing), label + squash (outcomes), estimate, audit, plan, status,
                  incidents + sentry + pagerduty, report, demo, db, gitio, cli
-tests/           230 unittest cases: real git repos in temp dirs, local fake servers, estimator Monte Carlo checks
+tests/           236 unittest cases: real git repos in temp dirs, local fake servers, estimator Monte Carlo checks
 tools/           coverage_lite.py, dependency-free line coverage
 docs/            METHODOLOGY.md
 research/        simulate.py (coverage, power, noise, design, peeking studies), results/, FINDINGS.md

{consequence_ledger-0.3.0 → consequence_ledger-0.3.1}/consequence_ledger.egg-info/SOURCES.txt

@@ -41,6 +41,7 @@ tests/test_parallel.py
 tests/test_plan.py
 tests/test_policy.py
 tests/test_release.py
+tests/test_report_segments.py
 tests/test_robust.py
 tests/test_security.py
 tests/test_segments.py

{consequence_ledger-0.3.0 → consequence_ledger-0.3.1}/ledger/__init__.py

@@ -1,2 +1,2 @@
 """Consequence Ledger: measure what AI coding decisions actually cause."""
-__version__ = "0.3.0"
+__version__ = "0.3.1"

{consequence_ledger-0.3.0 → consequence_ledger-0.3.1}/ledger/cli.py

@@ -4,7 +4,8 @@ import datetime as dt
 import json
 import sys
 import tempfile
-from contextlib import closing
+from contextlib import closing, redirect_stdout
+from io import StringIO
 from pathlib import Path
 
 from . import __version__
@@ -298,9 +299,13 @@ def run_report(repo, out=None, primary="dr", title="Consequence Ledger"):
     stats = json.loads((d / "label.json").read_text())
     truth = json.loads((d / "truth.json").read_text()) if (d / "truth.json").exists() else None
     validation = json.loads((d / "validation.json").read_text()) if (d / "validation.json").exists() else None
+    extra = {}
+    for key, name in (("breakdown", "breakdown.json"), ("recommendation", "recommend.json")):
+        if (d / name).exists():
+            extra[key] = json.loads((d / name).read_text())
     out = Path(out or d / "report.html")
     out.write_text(report_mod.render(est, stats, title, primary, truth, {"repo": Path(repo).name},
-                                     validation), encoding="utf-8")
+                                     validation, **extra), encoding="utf-8")
     return out
 
 
@@ -468,6 +473,11 @@ def cmd_demo(args):
     print(f"labeler vs oracle: precision {v['precision']:.2f}, recall {v['recall']:.2f} "
           f"(tp {v['tp']}, fp {v['fp']}, fn {v['fn']})")
     print("DR on oracle labels: " + ", ".join(f"{a} {e['value'] * 100:.1f}%" for a, e in v["oracle_dr"].items()))
+    seg = argparse.Namespace(repo=repo, stratum="stratum", cluster=None, baseline=None, clip=None,
+                             folds=DEFAULT_FOLDS, json=False)
+    with redirect_stdout(StringIO()):
+        cmd_breakdown(seg)
+        cmd_recommend(seg)
     print(f"\nreport: {run_report(repo, title='Consequence Ledger demo')}")
 
 

{consequence_ledger-0.3.0 → consequence_ledger-0.3.1}/ledger/report.py

@@ -241,6 +241,60 @@ def early_card(est, label_stats):
             f'<tbody>{rows}</tbody></table></div><p class="note">{trust}</p></div>')
 
 
+def _pp(x):
+    return f"{x * 100:+.1f}&nbsp;pp"
+
+
+def breakdown_card(bd, baseline, stratum, stale=""):
+    if not bd:
+        return ""
+    parts = []
+    for arm, d in bd.items():
+        rows = "".join(
+            f"<tr><td>{html.escape(str(x['stratum']))}</td><td>{x['n']}</td>"
+            f"<td>{_pp(x['effect'])} <span class=\"ci\">[{_pp(x['raw_lo'])}, {_pp(x['raw_hi'])}]</span></td>"
+            f"<td>{_pp(x['shrunk'])} <span class=\"ci\">[{_pp(x['shrunk_lo'])}, {_pp(x['shrunk_hi'])}]</span></td>"
+            f"<td>{x['p_adjusted']:.3f}</td></tr>"
+            for x in sorted(d["strata"], key=lambda x: x["shrunk"]))
+        verdict = ("At least one stratum differs from the rest, even after allowing for how many were examined."
+                   if d["heterogeneous"] else
+                   "No stratum stands out once you allow for how many were examined.")
+        parts.append(
+            f"<p><b>{html.escape(arm)}</b> against {html.escape(baseline)}: overall {_pp(d['overall'])}, "
+            f"spread between strata {d['tau'] * 100:.1f}&nbsp;pp. {verdict}</p>"
+            '<div class="table"><table><thead><tr><th>Stratum</th><th>Decisions</th><th>Raw effect [95% CI]</th>'
+            '<th>Shrunk effect [95% CI]</th><th>Adjusted p</th></tr></thead>'
+            f"<tbody>{rows}</tbody></table></div>")
+    return (f"<h2>Which tool for which kind of task</h2><div class=\"card\">{''.join(parts)}"
+            f"<p class=\"note\">Effects are changes in the bad-outcome rate, by <code>{html.escape(stratum)}</code>. "
+            "Shrunk effects pull noisy strata toward the overall effect; adjusted p-values allow for looking at "
+            f"every stratum. Negative is better than the baseline.{stale}</p></div>")
+
+
+def recommend_card(rec, stratum, stale=""):
+    if not rec:
+        return ""
+    if rec["worth_it"]:
+        verdict = ("The interval is above zero: shifting routing toward these choices is likely to help. "
+                   "Keep epsilon at 0.2 or more so later estimates stay valid.")
+    elif rec["best_single_lo"] > 0:
+        verdict = ("Any gain comes from one tool being better overall, not from matching tools to task types.")
+    else:
+        verdict = "The interval includes zero: no evidence yet that routing by task type beats what you did."
+    stable = ", ".join(f"{html.escape(str(s))} → {html.escape(a)}" for s, a in sorted(rec["stable_policy"].items()))
+    unstable = ", ".join(html.escape(str(s)) for s in rec["unstable_strata"])
+    return (
+        '<h2>What routing by task type is worth</h2><div class="card">'
+        f"<p>{rec['current_rate'] * 100:.1f}% of decisions went bad. Sending each <code>{html.escape(stratum)}</code> to its "
+        f"best tool, chosen on one half of the decisions and judged on the other, would have changed that by "
+        f"<b>{_pp(-rec['gain'])}</b> <span class=\"ci\">[{_pp(-rec['gain_hi'])}, {_pp(-rec['gain_lo'])}]</span>. "
+        f"Always using the single best tool: <b>{_pp(-rec['best_single_gain'])}</b> "
+        f"<span class=\"ci\">[{_pp(-rec['best_single_hi'])}, {_pp(-rec['best_single_lo'])}]</span>.</p>"
+        + (f"<p>Choices that held in both halves: {stable}.</p>" if stable else "")
+        + (f"<p>Changed with the data (no real preference): {unstable}.</p>" if unstable else "")
+        + f"<p class=\"note\">{verdict}{stale}</p></div>")
+
+
 def incident_rows(links):
     names = {"suspect": "Suspect commit in the export", "files_off": "Files only, not linked (default)",
              "files_latest": "Files only, latest matching deploy", "files_window": "Files only, every match in 14 days"}
@@ -268,7 +322,8 @@ def validation_card(v):
             f'{incident_table}</div>')
 
 
-def render(est, label_stats, title="Consequence Ledger", primary="dr", truth=None, meta=None, validation=None):
+def render(est, label_stats, title="Consequence Ledger", primary="dr", truth=None, meta=None, validation=None,
+           breakdown=None, recommendation=None):
     meta = meta or {}
     as_of = dt.datetime.fromtimestamp(label_stats["as_of"], dt.timezone.utc).strftime("%Y-%m-%d")
     rows = []
@@ -299,6 +354,16 @@ def render(est, label_stats, title="Consequence Ledger", primary="dr", truth=Non
               + ('<span><i class="dot" style="background:var(--truth)"></i>True rate (synthetic demo)</span>' if truth else "")
               + "</div>")
     source = html.escape(meta.get("repo", ""))
+    stale = ""
+    for part in (breakdown, recommendation):
+        if part and part.get("n") not in (None, est["n"]):
+            stale = (f" These tables were computed from {part['n']} decisions, the estimate above from {est['n']}; "
+                     "rerun <code>ledger breakdown</code> and <code>ledger recommend</code> to refresh them.")
+    segments = ""
+    if breakdown:
+        segments += breakdown_card(breakdown["arms"], breakdown["baseline"], breakdown["stratum"], stale)
+    if recommendation:
+        segments += recommend_card(recommendation, recommendation["stratum"], stale)
     clusters = est.get("clusters", est["n"])
     cluster_note = f", resampling {clusters} clusters of related decisions" if clusters < est["n"] else ""
     return f"""<!doctype html><html lang="en"><head><meta charset="utf-8">
@@ -315,6 +380,7 @@ def render(est, label_stats, title="Consequence Ledger", primary="dr", truth=Non
 <div class="card table"><table><thead><tr><th>Arm</th><th>Decisions</th><th>Bad</th><th>Naive</th><th>IPW</th><th>SNIPW</th>
 <th>{NAMES[primary]} [95% CI]</th><th>vs baseline</th><th>ESS</th>{truth_head}</tr></thead><tbody>{''.join(rows)}</tbody></table></div>
 {early_card(est, label_stats)}
+{segments}
 {validation_card(validation)}
 <h2>How to read this</h2>
 <div class="card note">

consequence_ledger-0.3.1/tests/test_report_segments.py

@@ -0,0 +1,73 @@
+"""The report shows the task-type tables, and only honestly."""
+import random
+import unittest
+
+from ledger.estimate import estimate
+from ledger.report import breakdown_card, recommend_card, render
+from ledger.segments import breakdown, recommend
+from test_degenerate import STATS, has_bad_token
+
+
+def make_rows(n=400, seed=3):
+    rng = random.Random(seed)
+    out = []
+    for i in range(n):
+        arm = "A" if rng.random() < 0.5 else "B"
+        stratum = rng.choice(["easy", "hard"])
+        bad = rng.random() < (0.08 if arm == "A" else 0.14)
+        out.append({"arm": arm, "y": int(bad), "p": {"A": 0.5, "B": 0.5}, "s": stratum, "c": f"c{i}"})
+    return out
+
+
+class SegmentsInReportTests(unittest.TestCase):
+    def setUp(self):
+        self.rows = make_rows()
+        self.est = estimate(self.rows, arms=["A", "B"], baseline="A", boot=20)
+        self.bd = {"baseline": "A", "stratum": "difficulty", "n": len(self.rows),
+                   "arms": breakdown(self.rows, ["A", "B"], "A", None, 5)}
+        self.rec = dict(recommend(self.rows, ["A", "B"], "A", None, 5), baseline="A", stratum="difficulty")
+
+    def test_tables_appear_when_given(self):
+        page = render(self.est, STATS, breakdown=self.bd, recommendation=self.rec)
+        self.assertIn("Which tool for which kind of task", page)
+        self.assertIn("What routing by task type is worth", page)
+        self.assertIn("easy", page)
+        self.assertIn("hard", page)
+        self.assertFalse(has_bad_token(page))
+
+    def test_report_without_them_is_unchanged(self):
+        page = render(self.est, STATS)
+        self.assertNotIn("Which tool for which kind of task", page)
+        self.assertNotIn("What routing by task type", page)
+
+    def test_stale_tables_are_flagged(self):
+        old = dict(self.bd, n=len(self.rows) - 50)
+        page = render(self.est, STATS, breakdown=old)
+        self.assertIn("rerun", page)
+        fresh = render(self.est, STATS, breakdown=self.bd)
+        self.assertNotIn("rerun", fresh)
+
+    def test_stratum_and_arm_names_are_escaped(self):
+        bd = {"baseline": "A", "stratum": "<b>x</b>", "n": 1, "arms": {"<i>B</i>": {
+            "overall": 0.0, "tau": 0.0, "heterogeneous": False,
+            "strata": [{"stratum": "<script>alert(1)</script>", "n": 5, "effect": 0.01, "se": 0.02, "raw_lo": -0.03,
+                        "raw_hi": 0.05, "shrunk": 0.0, "shrunk_lo": -0.02, "shrunk_hi": 0.02, "p_adjusted": 1.0}]}}}
+        page = breakdown_card(bd["arms"], "A", bd["stratum"])
+        self.assertNotIn("<script>", page)
+        self.assertNotIn("<i>B</i>", page)
+
+    def test_empty_inputs_render_nothing(self):
+        self.assertEqual(breakdown_card({}, "A", "s"), "")
+        self.assertEqual(recommend_card(None, "s"), "")
+
+    def test_each_verdict_wording(self):
+        base = {"n": 10, "current_rate": 0.1, "gain": 0.0, "gain_lo": -0.01, "gain_hi": 0.01,
+                "best_single_gain": 0.0, "best_single_lo": -0.01, "best_single_hi": 0.01,
+                "stable_policy": {}, "unstable_strata": [], "worth_it": False}
+        self.assertIn("includes zero", recommend_card(base, "s"))
+        self.assertIn("one tool being better overall", recommend_card(dict(base, best_single_lo=0.01), "s"))
+        self.assertIn("likely to help", recommend_card(dict(base, worth_it=True), "s"))
+
+
+if __name__ == "__main__":
+    unittest.main()