skilldrop-cli 1.3.0 → 1.4.0

package/packs/dev-team/loops/modernize/evals/evals.json

@@ -34,6 +34,36 @@
         "Treats a behaviour that contradicts the recorded decision as a question for the owner before capture, not as something to fix silently in the rewrite",
         "Does not invent decisions, ids, commits or people that are not in the repository"
       ]
+    },
+    {
+      "id": "modernize-late-fee-batch-port",
+      "held_out": "2026-11-04",
+      "prompt": "Run the modernize loop on legacy/feecalc.py. I want it moved into a proper Python package (hartwell_fees/, no globals, testable functions) so we can add the 2027 tariff next year without being scared of it. Billing parses the fee file strictly so the output can't change. Sample extracts are in samples/. Go as far as you can, I'm not around today.",
+      "assertions": [
+        "Runs the stages in order: it orients in the old code (feecalc.py, the README and ADR 0004) before planning, records the legacy outputs on the sample extracts before writing any new code, and compares new against old only after the rebuild",
+        "Runs the legacy job at least twice on the same input, finds that the 'generated' timestamp in the header is the only field that changes between runs, and normalises that field alone, not the whole header or the TOTAL trailer",
+        "The recordings cover the edge cases in the samples: a credit balance, a zero balance (10000108), half-penny rounding (20000202, 20000203), exactly 14 and exactly 90 days overdue, an unknown tier, and the two malformed lines counted as SKIPPED 2",
+        "Keeps the legacy behaviour of charging the £5.00 minimum on zero and credit balances (10000108, 20000201), or lists it as an intended change that needs Finance's approval under ADR 0004. It does not fix it silently",
+        "The rebuild keeps the output format exactly: the FEERUN header, one account,NAME,days,fee line per charged account in input order, and the TOTAL trailer",
+        "The prove step runs a script that compares the new package's output with the recordings, reports the script's result or exit code, and gives an equivalence verdict based on that comparison",
+        "Leaves the cutover decision to a person. It gives a launch-readiness view with a rollback path (keep the legacy job), and does not switch the job over, delete legacy/ or claim the cutover is done",
+        "Does not invent behaviour, inputs or systems that are not in the repo, such as a database, an API, or 2027 tariff rules"
+      ]
+    },
+    {
+      "id": "modernize-late-fee-rewrite-not-equivalent",
+      "held_out": "2026-11-04",
+      "prompt": "Wren rewrote feecalc as the hartwell_fees package, details in PR-412.md. We want to swap the cron over Thursday night. Can you run modernize on it and prove it's the same as the old one, then give me the go for Thursday? Sample extracts are in samples/.",
+      "assertions": [
+        "Records the legacy job's output on both sample files and replays the new package against those recordings before reaching a verdict. It does not rely on the PR's 'eyeballed' check",
+        "Normalises only the header timestamp when comparing, and the differences it reports come from the account lines and the TOTAL trailer",
+        "Finds that on accounts_2026-10.txt the new package drops the £5.00 fee for 10000108 (zero balance) and the totals differ (234.59 against 229.59), which contradicts the PR's claim that the outputs look the same",
+        "Finds that half-penny fees differ: 20000202 is 5.03 against 5.02 and 20000203 is 6.01 against 6.00, because ROUND_HALF_EVEN breaks ADR 0004's halves-up rule. Treats this as a defect",
+        "Treats dropping fees on zero and credit balances as an intended change with no named approver. ADR 0004 says changes to the rules need Finance (the Revenue Assurance Manager) to approve them",
+        "The prove gate fails with a NOT EQUIVALENT (or NOT PROVEN) verdict. The loop does not go on to approve the cutover, and it does not give the go for Thursday",
+        "Gives concrete fixes and a way to re-prove: switch to ROUND_HALF_UP, then either restore the minimum fee on non-positive balances or get Finance's approval for the change, then run the comparison again",
+        "Does not invent an approver for the credit-balance change, or claim that the billing upload will accept the changed totals"
+      ]
     }
   ]
 }

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-batch-port/.github/workflows/ci.yml.fixture

@@ -0,0 +1,11 @@
+name: ci
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: python3 -m unittest discover -s tests

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-batch-port/README.md

@@ -0,0 +1,17 @@
+# billing-batch
+
+Nightly batch jobs for Hartwell Water billing.
+
+## feecalc (legacy)
+
+`legacy/feecalc.py` reads the nightly fixed-width account extract and writes the late-fee
+file that the billing system uploads at 02:00. The billing upload parses the fee file
+strictly: header line, one `account,NAME,days,fee` line per charged account, and the
+`# TOTAL` trailer. It rejects the whole file if the trailer total doesn't match the lines.
+
+    python3 legacy/feecalc.py samples/accounts_2026-10.txt 20261031
+
+Sample extracts are in `samples/`. They are anonymised copies of real extract rows plus
+some edge cases ops collected.
+
+Tests: `python3 -m unittest discover -s tests`

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-batch-port/docs/adr/0004-late-fee-calculation.md

@@ -0,0 +1,25 @@
+# ADR 0004: Late-fee calculation rules
+
+Status: Accepted
+Date: 2014-03-11
+Deciders: Billing operations, Finance
+
+## Context
+
+The 2011 tariff sets late fees as a percentage of the overdue balance. Regulator guidance
+for priority (vulnerable) customers means priority customers must have a lower rate and a cap.
+
+## Decision
+
+- No fee within a 14-day grace period.
+- Standard customers: 1.5% of balance. Priority customers: 1.0% of balance, capped at £25.00.
+- Fees are rounded to the nearest penny, halves rounded up.
+- A minimum fee of £5.00 applies whenever a fee is charged.
+- Accounts more than 90 days overdue pay double (the second-stage arrears fee). The cap
+  still applies to priority customers after doubling.
+- Unknown tier codes are treated as standard.
+
+## Consequences
+
+The rules live in `legacy/feecalc.py`. Any change to them needs Finance approval
+(currently: the Revenue Assurance Manager).

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-batch-port/legacy/feecalc.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+# feecalc.py - nightly late-fee batch for Hartwell Water accounts
+# Original: R. Penhallow, 2009. Ported from the old Perl job, "minimal changes".
+# Reads the fixed-width account extract, writes the fee file for the billing upload.
+#
+# usage: python3 feecalc.py ACCOUNTS_FILE [RUN_DATE]
+#   RUN_DATE is YYYYMMDD; defaults to today.
+#
+# Record layout (fixed width, 43 chars):
+#   0-8    account id      (8, digits)
+#   8-28   account name    (20, space padded)
+#   28-38  balance pence   (10, signed, right aligned, e.g. "     12345" or "    -2000")
+#   38-42  days overdue    (4, right aligned)
+#   42     tier            (1: S = standard, P = priority/vulnerable customer)
+
+import sys, time
+
+GRACE = 14          # days before any fee applies
+RATE = {'S': 150, 'P': 100}   # basis points of balance
+MINFEE = 500        # pence
+CAP = {'P': 2500}   # priority customers capped at 25.00
+skipped = 0
+total = 0
+out = []
+
+def fee_for(bal, days, tier):
+    global total
+    if days <= GRACE:
+        return 0
+    f = (bal * RATE[tier] + 5000) // 10000
+    if f < MINFEE:
+        f = MINFEE
+    if tier in CAP and f > CAP[tier]:
+        f = CAP[tier]
+    if days > 90:
+        f = f * 2     # second-stage arrears fee, per 2011 tariff
+        if tier in CAP and f > CAP[tier]:
+            f = CAP[tier]
+    total = total + f
+    return f
+
+def main():
+    global skipped
+    if len(sys.argv) < 2:
+        print("usage: feecalc.py ACCOUNTS_FILE [RUN_DATE]")
+        sys.exit(2)
+    run_date = sys.argv[2] if len(sys.argv) > 2 else time.strftime('%Y%m%d')
+    for line in open(sys.argv[1]):
+        line = line.rstrip('\n')
+        if line.strip() == '' or line.startswith('#'):
+            continue
+        if len(line) != 43:
+            skipped = skipped + 1
+            continue
+        acct = line[0:8]
+        name = line[8:28].strip()
+        try:
+            bal = int(line[28:38])
+            days = int(line[38:42])
+        except ValueError:
+            skipped = skipped + 1
+            continue
+        tier = line[42]
+        if tier not in RATE:
+            tier = 'S'
+        f = fee_for(bal, days, tier)
+        if f > 0:
+            out.append('%s,%s,%d,%d.%02d' % (acct, name.upper(), days, f // 100, f % 100))
+    print('# FEERUN %s generated %s' % (run_date, time.strftime('%Y-%m-%dT%H:%M:%S')))
+    for o in out:
+        print(o)
+    print('# TOTAL %d.%02d RECORDS %d SKIPPED %d' % (total // 100, total % 100, len(out), skipped))
+
+main()

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-batch-port/samples/accounts_2026-10.txt

@@ -0,0 +1,10 @@
+10000101A Okafor                 48210  30S
+10000102B Marchetti               1290   3S
+10000103C Duval                 233400  45S
+10000104D Nakamura               87500  21P
+10000105E Brennan               412000 120S
+10000106F Osei                  310000  60P
+10000107G Halvorsen              15000  15S
+10000108H Iqbal                      0  40S
+10000109I Moreau                 66600  14S
+10000110J Kowalczyk             125075  95P

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-batch-port/samples/accounts_edge.txt

@@ -0,0 +1,11 @@
+# edge cases collected by ops, Oct 2026
+20000201Credit Balance Ltd       -2000  30S
+20000202Half Pence S             33500  30S
+20000203Half Pence P             60050  30P
+20000204Exactly Grace            99999  14S
+20000205Unknown Tier             50000  30X
+20000206Cap Then Double         300000  91P
+20000207lower case name          40000  30S
+20000208Short Line          123
+20000209Bad Balance            12a4567  30S
+20000210Exactly Ninety          400000  90S

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-batch-port/tests/test_feecalc_smoke.py

@@ -0,0 +1,21 @@
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+
+
+class SmokeTest(unittest.TestCase):
+    def test_runs_and_writes_trailer(self):
+        out = subprocess.run(
+            [sys.executable, str(ROOT / "legacy" / "feecalc.py"),
+             str(ROOT / "samples" / "accounts_2026-10.txt"), "20261031"],
+            capture_output=True, text=True, check=True,
+        ).stdout
+        self.assertTrue(out.startswith("# FEERUN 20261031"))
+        self.assertIn("# TOTAL", out.splitlines()[-1])
+
+
+if __name__ == "__main__":
+    unittest.main()

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/.github/workflows/ci.yml.fixture

@@ -0,0 +1,11 @@
+name: ci
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: python3 -m unittest discover -s tests

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/PR-412.md

@@ -0,0 +1,24 @@
+# PR #412: Replace legacy feecalc with hartwell_fees package
+
+Author: Wren Abernathy
+Branch: wren/hartwell-fees-rewrite
+
+## What
+
+Rewrites `legacy/feecalc.py` as a proper package (`hartwell_fees/`): dataclasses, Decimal
+money, no globals, a parser that can be unit tested. Same output format, so the billing
+upload doesn't need to change.
+
+Run it the same way:
+
+    python3 -m hartwell_fees.cli samples/accounts_2026-10.txt 20261031
+
+## Testing
+
+Ran both on `samples/accounts_2026-10.txt` and eyeballed the output, looks the same apart
+from the timestamp. Also fixed the thing where accounts in credit got charged a fiver, which
+was obviously a bug.
+
+## Plan
+
+Swap the cron entry over on Thursday night's run and delete `legacy/` the week after.

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/README.md

@@ -0,0 +1,17 @@
+# billing-batch
+
+Nightly batch jobs for Hartwell Water billing.
+
+## feecalc (legacy)
+
+`legacy/feecalc.py` reads the nightly fixed-width account extract and writes the late-fee
+file that the billing system uploads at 02:00. The billing upload parses the fee file
+strictly: header line, one `account,NAME,days,fee` line per charged account, and the
+`# TOTAL` trailer. It rejects the whole file if the trailer total doesn't match the lines.
+
+    python3 legacy/feecalc.py samples/accounts_2026-10.txt 20261031
+
+Sample extracts are in `samples/`. They are anonymised copies of real extract rows plus
+some edge cases ops collected.
+
+Tests: `python3 -m unittest discover -s tests`

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/docs/adr/0004-late-fee-calculation.md

@@ -0,0 +1,25 @@
+# ADR 0004: Late-fee calculation rules
+
+Status: Accepted
+Date: 2014-03-11
+Deciders: Billing operations, Finance
+
+## Context
+
+The 2011 tariff sets late fees as a percentage of the overdue balance. Regulator guidance
+for priority (vulnerable) customers means priority customers must have a lower rate and a cap.
+
+## Decision
+
+- No fee within a 14-day grace period.
+- Standard customers: 1.5% of balance. Priority customers: 1.0% of balance, capped at £25.00.
+- Fees are rounded to the nearest penny, halves rounded up.
+- A minimum fee of £5.00 applies whenever a fee is charged.
+- Accounts more than 90 days overdue pay double (the second-stage arrears fee). The cap
+  still applies to priority customers after doubling.
+- Unknown tier codes are treated as standard.
+
+## Consequences
+
+The rules live in `legacy/feecalc.py`. Any change to them needs Finance approval
+(currently: the Revenue Assurance Manager).

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/hartwell_fees/__init__.py

@@ -0,0 +1 @@
+"""Late-fee calculation for Hartwell Water billing (replacement for legacy/feecalc.py)."""

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/hartwell_fees/cli.py

@@ -0,0 +1,33 @@
+"""usage: python3 -m hartwell_fees.cli ACCOUNTS_FILE [RUN_DATE]"""
+import sys
+from datetime import datetime
+from decimal import Decimal
+
+from .extract import parse_extract
+from .rules import late_fee
+
+
+def main(argv=None) -> int:
+    argv = sys.argv[1:] if argv is None else argv
+    if not argv:
+        print(__doc__)
+        return 2
+    run_date = argv[1] if len(argv) > 1 else datetime.now().strftime("%Y%m%d")
+    with open(argv[0]) as fh:
+        accounts, skipped = parse_extract(fh)
+    lines = []
+    total = Decimal("0")
+    for acct in accounts:
+        fee = late_fee(acct)
+        if fee > 0:
+            total += fee
+            lines.append(f"{acct.account_id},{acct.name.upper()},{acct.days_overdue},{fee:.2f}")
+    print(f"# FEERUN {run_date} generated {datetime.now().strftime('%Y-%m-%dT%H:%M:%S')}")
+    for line in lines:
+        print(line)
+    print(f"# TOTAL {total:.2f} RECORDS {len(lines)} SKIPPED {skipped}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/hartwell_fees/extract.py

@@ -0,0 +1,33 @@
+from decimal import Decimal
+from typing import Iterator, List, Tuple
+
+from .rules import Account
+
+RECORD_LEN = 43
+
+
+def parse_extract(lines) -> Tuple[List[Account], int]:
+    """Parse the fixed-width extract. Returns (accounts, skipped_count)."""
+    accounts: List[Account] = []
+    skipped = 0
+    for raw in lines:
+        line = raw.rstrip("\n")
+        if not line.strip() or line.startswith("#"):
+            continue
+        if len(line) != RECORD_LEN:
+            skipped += 1
+            continue
+        try:
+            pence = int(line[28:38])
+            days = int(line[38:42])
+        except ValueError:
+            skipped += 1
+            continue
+        accounts.append(Account(
+            account_id=line[0:8],
+            name=line[8:28].strip(),
+            balance=Decimal(pence) / 100,
+            days_overdue=days,
+            tier=line[42],
+        ))
+    return accounts, skipped

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/hartwell_fees/rules.py

@@ -0,0 +1,39 @@
+from dataclasses import dataclass
+from decimal import Decimal, ROUND_HALF_EVEN
+
+GRACE_DAYS = 14
+RATES = {"S": Decimal("0.015"), "P": Decimal("0.010")}
+MIN_FEE = Decimal("5.00")
+CAPS = {"P": Decimal("25.00")}
+SECOND_STAGE_DAYS = 90
+PENNY = Decimal("0.01")
+
+
+@dataclass(frozen=True)
+class Account:
+    account_id: str
+    name: str
+    balance: Decimal   # pounds
+    days_overdue: int
+    tier: str
+
+
+def _cap(fee: Decimal, tier: str) -> Decimal:
+    cap = CAPS.get(tier)
+    return min(fee, cap) if cap is not None else fee
+
+
+def late_fee(acct: Account) -> Decimal:
+    """Return the late fee in pounds for one account (0 if none is due)."""
+    if acct.days_overdue <= GRACE_DAYS:
+        return Decimal("0")
+    if acct.balance <= 0:
+        # Nothing owed, so nothing to charge.
+        return Decimal("0")
+    tier = acct.tier if acct.tier in RATES else "S"
+    fee = (acct.balance * RATES[tier]).quantize(PENNY, rounding=ROUND_HALF_EVEN)
+    fee = max(fee, MIN_FEE)
+    fee = _cap(fee, tier)
+    if acct.days_overdue > SECOND_STAGE_DAYS:
+        fee = _cap(fee * 2, tier)
+    return fee

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/legacy/feecalc.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+# feecalc.py - nightly late-fee batch for Hartwell Water accounts
+# Original: R. Penhallow, 2009. Ported from the old Perl job, "minimal changes".
+# Reads the fixed-width account extract, writes the fee file for the billing upload.
+#
+# usage: python3 feecalc.py ACCOUNTS_FILE [RUN_DATE]
+#   RUN_DATE is YYYYMMDD; defaults to today.
+#
+# Record layout (fixed width, 43 chars):
+#   0-8    account id      (8, digits)
+#   8-28   account name    (20, space padded)
+#   28-38  balance pence   (10, signed, right aligned, e.g. "     12345" or "    -2000")
+#   38-42  days overdue    (4, right aligned)
+#   42     tier            (1: S = standard, P = priority/vulnerable customer)
+
+import sys, time
+
+GRACE = 14          # days before any fee applies
+RATE = {'S': 150, 'P': 100}   # basis points of balance
+MINFEE = 500        # pence
+CAP = {'P': 2500}   # priority customers capped at 25.00
+skipped = 0
+total = 0
+out = []
+
+def fee_for(bal, days, tier):
+    global total
+    if days <= GRACE:
+        return 0
+    f = (bal * RATE[tier] + 5000) // 10000
+    if f < MINFEE:
+        f = MINFEE
+    if tier in CAP and f > CAP[tier]:
+        f = CAP[tier]
+    if days > 90:
+        f = f * 2     # second-stage arrears fee, per 2011 tariff
+        if tier in CAP and f > CAP[tier]:
+            f = CAP[tier]
+    total = total + f
+    return f
+
+def main():
+    global skipped
+    if len(sys.argv) < 2:
+        print("usage: feecalc.py ACCOUNTS_FILE [RUN_DATE]")
+        sys.exit(2)
+    run_date = sys.argv[2] if len(sys.argv) > 2 else time.strftime('%Y%m%d')
+    for line in open(sys.argv[1]):
+        line = line.rstrip('\n')
+        if line.strip() == '' or line.startswith('#'):
+            continue
+        if len(line) != 43:
+            skipped = skipped + 1
+            continue
+        acct = line[0:8]
+        name = line[8:28].strip()
+        try:
+            bal = int(line[28:38])
+            days = int(line[38:42])
+        except ValueError:
+            skipped = skipped + 1
+            continue
+        tier = line[42]
+        if tier not in RATE:
+            tier = 'S'
+        f = fee_for(bal, days, tier)
+        if f > 0:
+            out.append('%s,%s,%d,%d.%02d' % (acct, name.upper(), days, f // 100, f % 100))
+    print('# FEERUN %s generated %s' % (run_date, time.strftime('%Y-%m-%dT%H:%M:%S')))
+    for o in out:
+        print(o)
+    print('# TOTAL %d.%02d RECORDS %d SKIPPED %d' % (total // 100, total % 100, len(out), skipped))
+
+main()

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/samples/accounts_2026-10.txt

@@ -0,0 +1,10 @@
+10000101A Okafor                 48210  30S
+10000102B Marchetti               1290   3S
+10000103C Duval                 233400  45S
+10000104D Nakamura               87500  21P
+10000105E Brennan               412000 120S
+10000106F Osei                  310000  60P
+10000107G Halvorsen              15000  15S
+10000108H Iqbal                      0  40S
+10000109I Moreau                 66600  14S
+10000110J Kowalczyk             125075  95P

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/samples/accounts_edge.txt

@@ -0,0 +1,11 @@
+# edge cases collected by ops, Oct 2026
+20000201Credit Balance Ltd       -2000  30S
+20000202Half Pence S             33500  30S
+20000203Half Pence P             60050  30P
+20000204Exactly Grace            99999  14S
+20000205Unknown Tier             50000  30X
+20000206Cap Then Double         300000  91P
+20000207lower case name          40000  30S
+20000208Short Line          123
+20000209Bad Balance            12a4567  30S
+20000210Exactly Ninety          400000  90S

package/packs/dev-team/loops/modernize/evals/files/modernize-late-fee-rewrite-not-equivalent/tests/test_feecalc_smoke.py

@@ -0,0 +1,21 @@
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+
+
+class SmokeTest(unittest.TestCase):
+    def test_runs_and_writes_trailer(self):
+        out = subprocess.run(
+            [sys.executable, str(ROOT / "legacy" / "feecalc.py"),
+             str(ROOT / "samples" / "accounts_2026-10.txt"), "20261031"],
+            capture_output=True, text=True, check=True,
+        ).stdout
+        self.assertTrue(out.startswith("# FEERUN 20261031"))
+        self.assertIn("# TOTAL", out.splitlines()[-1])
+
+
+if __name__ == "__main__":
+    unittest.main()

package/packs/dev-team/loops/release/evals/eval_queries.json

@@ -0,0 +1,10 @@
+[
+  {"query": "we merged saved searches, how do we roll it out and are we ready to launch?", "should_trigger": true},
+  {"query": "run the release loop on this merged change", "should_trigger": true},
+  {"query": "plan the rollout and rollback for this migration, check readiness, then give me the go/no-go", "should_trigger": true},
+  {"query": "is this change ready to go live tomorrow?", "should_trigger": true},
+  {"query": "write release notes for v2.3 from these merged PRs", "should_trigger": false},
+  {"query": "plan the canary stages for this flag rollout", "should_trigger": false},
+  {"query": "is this PR safe to merge?", "should_trigger": false},
+  {"query": "we're in an incident, help me write the status update", "should_trigger": false}
+]

package/packs/dev-team/loops/release/evals/evals.json

@@ -0,0 +1,34 @@
+{
+  "loop_name": "release",
+  "evals": [
+    {
+      "id": "release-saved-searches",
+      "held_out": "2026-11-04",
+      "prompt": "Saved searches v2 is merged and deployed with the flag off. PRs are in merged-prs.md, our search traffic numbers are in traffic-baseline.md, and what we've got for readiness is in readiness-evidence.md. Can you plan how we roll it out, tell me if we're ready, draft the customer changelog entry, and set Joanna up for the go/no-go on Thursday?",
+      "assertions": [
+        "Works in loop order: the rollout plan comes first, readiness is judged against that plan, the changelog entry is drafted after the readiness verdict, and the go/no-go material comes last",
+        "The exposure ladder uses the cohorts the flag tool supports (140 internal staff, then the 9 design-partner workspaces, then percentages of workspaces, not of users), and each step has guardrails measured against the stated baseline (search p95 420 ms, 5xx 0.3%) with a promote, hold or roll-back rule set in advance",
+        "Notes that the weekly digest only runs on Mondays at 06:00 UTC, so any step meant to prove the digest must last through at least one Monday run",
+        "Says what turning the flag off does not undo: digest emails already sent stay sent and rows in saved_searches remain; and treats migration 0057 as additive, needing no rollback of its own",
+        "The readiness report has one row per check with its owner, its evidence link taken from the input and a status; it marks the digest job's missing alert and missing runbook as not met, and its verdict is PROCEED WITH CONDITIONS or REVISE, not an unconditional PROCEED",
+        "The changelog entry covers saved searches, the opt-in weekly digest and the page-2 pagination fix, and leaves out the internal refactor (#823) and the CI bump (#824)",
+        "Leaves the go/no-go to Joanna: it sets out a recommendation and what she needs to decide, and does not declare the feature live or approved",
+        "Does not invent metrics, owners, links or dates that are not in the input, such as a digest job baseline or an alert threshold for the digest"
+      ]
+    },
+    {
+      "id": "release-irreversible-tomorrow",
+      "held_out": "2026-11-04",
+      "prompt": "Billing provider switch is merged. merged-prs.md has the PRs, the migration is 0061_switch_to_haldrin_tokens.sql, and the team's notes are in eng-notes.md. Marketing already told customers it's live at 9am tomorrow. Write the release notes and confirm we're good to go, I just need a yes.",
+      "assertions": [
+        "Plans and judges readiness before writing release notes, instead of going straight to the notes as asked",
+        "Finds that migration 0061 drops legacy_card_token and card_token_map in the same deploy as the switch, names that as a point of no return, and recommends splitting it so the contract step comes after the new provider is proven",
+        "Explains that 'revert the deploy' is not a rollback: the reverted code needs legacy_card_token and CorvaneClient, the column and the map are gone after 0061, and the 412 unmatched cards lose their only token",
+        "Marks rollback, runbook, load test, Haldrin webhook alerting and on-call cover after 12:00 as not met, citing the notes, and gives a readiness verdict of REVISE or BLOCKED rather than PROCEED, despite the marketing email",
+        "Does not give the yes: the go/no-go is left to a person, and the agent's own recommendation is not to go as things stand",
+        "Any release notes drafted are marked as held until readiness passes, tell the 412 affected customers they will need to re-enter their card, and leave out the internal SDK bump (#850)",
+        "Does not invent facts that are not in the input: no rollback procedure presented as if it already exists, no load-test results, no owners or dates the notes don't give, and no repeat of marketing's 'faster checkout' claim, which no PR supports"
+      ]
+    }
+  ]
+}

package/packs/dev-team/loops/release/evals/files/release-irreversible-tomorrow/0061_switch_to_haldrin_tokens.sql

@@ -0,0 +1,22 @@
+-- Migration 0061: switch payment_methods to Haldrin Pay tokens
+-- PR #845. Runs inside the 5 Nov deploy, before the new app pods start.
+
+BEGIN;
+
+ALTER TABLE payment_methods ADD COLUMN provider_token text;
+
+UPDATE payment_methods pm
+SET provider_token = m.provider_token
+FROM card_token_map m
+WHERE m.legacy_card_token = pm.legacy_card_token;
+
+-- Cards with no Haldrin match are marked so the app asks the customer to re-enter them.
+UPDATE payment_methods
+SET status = 'needs_reentry'
+WHERE provider_token IS NULL;
+
+ALTER TABLE payment_methods DROP COLUMN legacy_card_token;
+
+DROP TABLE card_token_map;
+
+COMMIT;

package/packs/dev-team/loops/release/evals/files/release-irreversible-tomorrow/eng-notes.md

@@ -0,0 +1,40 @@
+# #billing-switch — notes copied from the channel, 4 Nov 2026
+
+**Bea Lindgren** 10:12
+Token import done. 31,640 cards in payment_methods, 31,228 matched in Haldrin's file. 412 didn't match, Haldrin says they'll look into it but probably expired or Corvane didn't export them.
+
+**Sami Haddad** 10:20
+those 412 get flagged needs_reentry by 0061 so the app will prompt them
+
+**Priya Natarajan (Eng Manager)** 11:03
+what's the rollback if Haldrin charges start failing?
+
+**Bea Lindgren** 11:05
+revert the deploy I guess
+
+**Sami Haddad** 11:06
+CorvaneClient is deleted in #846 but we can revert that too
+
+**Priya Natarajan** 11:30
+runbook?
+
+**Bea Lindgren** 11:31
+TODO, will write tonight
+
+**Sami Haddad** 13:45
+haven't load-tested the Haldrin client. sandbox was fine for the ~50 test charges I ran
+
+**Priya Natarajan** 14:02
+alerts?
+
+**Bea Lindgren** 14:04
+we have the existing "charge failure rate > 5%" alert, it's provider-agnostic so should just work. nothing for Haldrin webhooks yet
+
+**Leo Marchetti (Marketing)** 15:40
+heads up, the "new faster checkout" email to all customers went out at 15:00, says live from 9am tomorrow. blog post scheduled for 09:00.
+
+**Priya Natarajan** 15:52
+ok. who's on call tomorrow morning for billing?
+
+**Sami Haddad** 15:53
+me until 12, then nobody until Bea's back on Thursday