creditdecide 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (56) hide show
  1. creditdecide-0.1.0/.gitignore +13 -0
  2. creditdecide-0.1.0/CHANGELOG.md +37 -0
  3. creditdecide-0.1.0/CITATION.cff +22 -0
  4. creditdecide-0.1.0/DATA_LICENSE.md +26 -0
  5. creditdecide-0.1.0/LICENSE +21 -0
  6. creditdecide-0.1.0/PKG-INFO +283 -0
  7. creditdecide-0.1.0/README.md +249 -0
  8. creditdecide-0.1.0/pyproject.toml +63 -0
  9. creditdecide-0.1.0/scripts/build_credit_default.py +85 -0
  10. creditdecide-0.1.0/scripts/check_calibration.py +48 -0
  11. creditdecide-0.1.0/src/creditdecide/__init__.py +40 -0
  12. creditdecide-0.1.0/src/creditdecide/_version.py +1 -0
  13. creditdecide-0.1.0/src/creditdecide/bootstrap.py +160 -0
  14. creditdecide-0.1.0/src/creditdecide/calibration.py +128 -0
  15. creditdecide-0.1.0/src/creditdecide/cli.py +377 -0
  16. creditdecide-0.1.0/src/creditdecide/data/__init__.py +0 -0
  17. creditdecide-0.1.0/src/creditdecide/data/credit_default.csv.gz +0 -0
  18. creditdecide-0.1.0/src/creditdecide/data/credit_default.json +17 -0
  19. creditdecide-0.1.0/src/creditdecide/datasets.py +198 -0
  20. creditdecide-0.1.0/src/creditdecide/decision.py +250 -0
  21. creditdecide-0.1.0/src/creditdecide/fairness.py +98 -0
  22. creditdecide-0.1.0/src/creditdecide/forest.py +27 -0
  23. creditdecide-0.1.0/src/creditdecide/metrics.py +35 -0
  24. creditdecide-0.1.0/src/creditdecide/modelcard.py +219 -0
  25. creditdecide-0.1.0/src/creditdecide/modelio.py +209 -0
  26. creditdecide-0.1.0/src/creditdecide/monitor.py +71 -0
  27. creditdecide-0.1.0/src/creditdecide/pipeline.py +552 -0
  28. creditdecide-0.1.0/src/creditdecide/plotting.py +64 -0
  29. creditdecide-0.1.0/src/creditdecide/preprocess.py +38 -0
  30. creditdecide-0.1.0/src/creditdecide/quality.py +61 -0
  31. creditdecide-0.1.0/src/creditdecide/reasons.py +188 -0
  32. creditdecide-0.1.0/src/creditdecide/sql/__init__.py +203 -0
  33. creditdecide-0.1.0/src/creditdecide/sql/class_counts.sql +5 -0
  34. creditdecide-0.1.0/src/creditdecide/sql/duplicates.sql +14 -0
  35. creditdecide-0.1.0/src/creditdecide/sql/gains.sql +37 -0
  36. creditdecide-0.1.0/src/creditdecide/sql/outcomes.sql +20 -0
  37. creditdecide-0.1.0/src/creditdecide/sql/outcomes_rows.sql +6 -0
  38. creditdecide-0.1.0/src/creditdecide/sql/psi.sql +44 -0
  39. creditdecide-0.1.0/src/creditdecide/sql/tableau_accounts.sql +25 -0
  40. creditdecide-0.1.0/src/creditdecide/sql/tableau_gains.sql +7 -0
  41. creditdecide-0.1.0/src/creditdecide/sql/tableau_models.sql +6 -0
  42. creditdecide-0.1.0/src/creditdecide/sql/tableau_reasons.sql +9 -0
  43. creditdecide-0.1.0/src/creditdecide/sql/undocumented_codes.sql +10 -0
  44. creditdecide-0.1.0/src/creditdecide/tableau.py +133 -0
  45. creditdecide-0.1.0/src/creditdecide/tuning.py +137 -0
  46. creditdecide-0.1.0/tests/conftest.py +55 -0
  47. creditdecide-0.1.0/tests/pandas_reference.py +111 -0
  48. creditdecide-0.1.0/tests/test_bootstrap.py +111 -0
  49. creditdecide-0.1.0/tests/test_calibration_reasons_tuning.py +292 -0
  50. creditdecide-0.1.0/tests/test_cli.py +253 -0
  51. creditdecide-0.1.0/tests/test_data.py +110 -0
  52. creditdecide-0.1.0/tests/test_decision.py +98 -0
  53. creditdecide-0.1.0/tests/test_modelio.py +104 -0
  54. creditdecide-0.1.0/tests/test_models.py +116 -0
  55. creditdecide-0.1.0/tests/test_monitoring.py +98 -0
  56. creditdecide-0.1.0/tests/test_sql.py +170 -0
@@ -0,0 +1,13 @@
1
+ __pycache__/
2
+ *.py[cod]
3
+ *.egg-info/
4
+ build/
5
+ dist/
6
+ .venv/
7
+ .pytest_cache/
8
+ .ruff_cache/
9
+ results/
10
+ *.zip
11
+ *.xls
12
+ .DS_Store
13
+ private/
@@ -0,0 +1,37 @@
1
+ # Changelog
2
+
3
+ ## 0.1.0 (2026-11-07)
4
+
5
+ First public release.
6
+
7
+ - Data-quality report (`creditdecide quality`): duplicate rows and duplicates with conflicting
8
+ labels, counted on all columns and on the model inputs; undocumented category codes; missing
9
+ values; constant columns; class balance.
10
+ - Group-aware stratified 60/20/20 split, so rows with identical model inputs never cross
11
+ splits, with preprocessing fit on the training split only.
12
+ - Models: a majority-class baseline, logistic regression and a random forest. Optional tuning
13
+ (`--tune`) by group cross-validation inside the training split, which adds histogram
14
+ gradient boosting; its folds and candidates are drawn by the package, so they do not depend
15
+ on the scikit-learn version.
16
+ - Decision layer: capacity (budget) and cost policies with thresholds chosen on validation, a
17
+ per-account expected-loss rule (`--decision-rule expected_loss`), a cost sensitivity table,
18
+ and the best one-feature rule as a benchmark at the model's own volume.
19
+ - Probability calibration (identity, sigmoid or isotonic) chosen by cross-validated log loss
20
+ with the one-standard-error rule, with reliability tables and expected vs actual defaults
21
+ and losses.
22
+ - Reason codes for flagged accounts, grouped into business factors, with measured accuracy.
23
+ - 95% bootstrap intervals for the test results (`bootstrap_test.csv`, `--bootstrap N`), with
24
+ comparisons computed on the same redraws.
25
+ - Fairness audit by protected group, with adverse impact ratios and a proxy check.
26
+ - A saved model directory with SHA-256 checksums, a skops type allow-list and a generated model
27
+ card; `predict`, `drift`, and outcomes analysis when labels arrive.
28
+ - A DuckDB SQL data layer (`creditdecide.sql`): the data-quality counts, the gains table, the
29
+ outcomes join and the drift bins, each tested against the pandas version it replaced.
30
+ `creditdecide outcomes` joins saved decisions to labels that arrive later, and
31
+ `creditdecide sql` runs a read-only query on the data.
32
+ - `creditdecide tableau` writes a run's results as CSV tables for a Tableau Public dashboard
33
+ (docs/TABLEAU.md).
34
+ - Random forests are fitted on all cores but predict on one thread (`creditdecide.forest`), so
35
+ reruns match bit for bit.
36
+ - Bundled UCI credit-card default data (CC BY 4.0), CI on Python 3.10, 3.12 and 3.13, a Docker
37
+ image, and a PyPI Trusted Publishing workflow.
@@ -0,0 +1,22 @@
1
+ cff-version: 1.2.0
2
+ message: "If you use this software, please cite it as below."
3
+ title: "creditdecide"
4
+ version: 0.1.0
5
+ date-released: 2026-11-07
6
+ license: MIT
7
+ type: software
8
+ authors:
9
+ - family-names: Steinle
10
+ given-names: Nathan
11
+ orcid: "https://orcid.org/0000-0003-0658-402X"
12
+ references:
13
+ - type: data
14
+ title: "Default of Credit Card Clients"
15
+ authors:
16
+ - family-names: Yeh
17
+ given-names: I-Cheng
18
+ publisher:
19
+ name: "UCI Machine Learning Repository"
20
+ year: 2009
21
+ doi: "10.24432/C55S3H"
22
+ license: CC-BY-4.0
@@ -0,0 +1,26 @@
1
+ # Bundled data
2
+
3
+ `src/creditdecide/data/credit_default.csv.gz`
4
+
5
+ - **Title:** Default of Credit Card Clients
6
+ - **Creator:** I.-C. Yeh
7
+ - **Source:** UCI Machine Learning Repository, https://archive.ics.uci.edu/dataset/350/default+of+credit+card+clients
8
+ - **DOI:** 10.24432/C55S3H
9
+ - **License:** Creative Commons Attribution 4.0 International (CC BY 4.0),
10
+ https://creativecommons.org/licenses/by/4.0/
11
+ - **Reference:** Yeh, I.-C. & Lien, C.-H. (2009). The comparisons of data mining techniques
12
+ for the predictive accuracy of probability of default of credit card clients. *Expert Systems
13
+ with Applications*, 36(2), 2473–2480.
14
+
15
+ ## Changes made
16
+
17
+ Produced from the official archive (SHA-256
18
+ `56c885f84457f6680f8438f02bfcdac9579323d8a94465ee5f26e32baa727602`) by
19
+ `scripts/build_credit_default.py`:
20
+
21
+ 1. the first header row (`X1` … `X23`, `Y`) was dropped, keeping the descriptive column names;
22
+ 2. the column `default payment next month` was renamed `DEFAULT`;
23
+ 3. the table was stored as gzip-compressed CSV.
24
+
25
+ No values were changed, added or removed (30,000 rows, 25 columns). The SHA-256 of the
26
+ bundled file is recorded in `credit_default.json` and checked on every load.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2025-2026 Nathan Steinle
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,283 @@
1
+ Metadata-Version: 2.5
2
+ Name: creditdecide
3
+ Version: 0.1.0
4
+ Summary: From a labeled table to a monitored, explainable decision rule: leakage-safe model comparison, calibrated probabilities, cost-based decisions, reason codes, confidence intervals, drift and fairness checks, with a DuckDB SQL data layer and Tableau-ready exports.
5
+ Project-URL: Source, https://github.com/nsteinle/creditdecide
6
+ Project-URL: Dataset, https://doi.org/10.24432/C55S3H
7
+ Author: Nathan Steinle
8
+ License-Expression: MIT
9
+ License-File: DATA_LICENSE.md
10
+ License-File: LICENSE
11
+ Keywords: calibration,credit risk,data quality,decision rules,duckdb,model monitoring,reason codes,sql,tableau
12
+ Classifier: Intended Audience :: Science/Research
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.10
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Programming Language :: Python :: 3.13
17
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
18
+ Requires-Python: >=3.10
19
+ Requires-Dist: duckdb>=1.1
20
+ Requires-Dist: numpy>=1.24
21
+ Requires-Dist: pandas>=2.0
22
+ Requires-Dist: scikit-learn>=1.4
23
+ Requires-Dist: skops>=0.10
24
+ Provides-Extra: dev
25
+ Requires-Dist: build; extra == 'dev'
26
+ Requires-Dist: matplotlib>=3.7; extra == 'dev'
27
+ Requires-Dist: pytest>=8; extra == 'dev'
28
+ Requires-Dist: ruff>=0.5; extra == 'dev'
29
+ Requires-Dist: twine; extra == 'dev'
30
+ Requires-Dist: xlrd>=2.0; extra == 'dev'
31
+ Provides-Extra: plot
32
+ Requires-Dist: matplotlib>=3.7; extra == 'plot'
33
+ Description-Content-Type: text/markdown
34
+
35
+ # creditdecide
36
+
37
+ creditdecide takes a labeled table and turns it into a decision rule that can be explained and
38
+ monitored. It checks the data quality first, compares models without leaking test data into
39
+ training, benchmarks the best model against the best one-line rule, calibrates the
40
+ probabilities, picks which accounts to act on from a capacity or a cost budget, gives reason
41
+ codes for every flagged account, puts 95% confidence intervals on the test results, and saves a
42
+ checksum-verified model with drift monitoring and a fairness audit. Its data and reporting
43
+ layer (the data-quality checks, the gains table, the outcomes join and the drift bins) is DuckDB
44
+ SQL, tested against the pandas results, and `creditdecide tableau` writes a run's results as
45
+ tables for a Tableau Public dashboard. It ships with a real credit-card default dataset and
46
+ also accepts your own CSV.
47
+
48
+ **The business question on the bundled data:** a card issuer can only act on a limited number
49
+ of accounts each month (a call, a payment plan, a lower limit). Which accounts should it act on,
50
+ how many of next month's defaults will that reach, how much money will it save, is a model
51
+ better than a simple rule, can each decision be explained, and does the rule treat protected
52
+ groups comparably?
53
+
54
+ ## What a run produces (credit-card default, test split of 6,002 clients)
55
+
56
+ The ranges in brackets are 95% bootstrap intervals over the test accounts (see
57
+ [Uncertainty](#uncertainty-in-the-results)).
58
+
59
+ | Question | Answer |
60
+ |---|---|
61
+ | How good is "do nothing"? | Predicting "no default" for everyone is 77.9% accurate, so accuracy is the wrong measure here. I use ranking, capture and cost instead. |
62
+ | Best ranking model? | Random forest, ROC AUC 0.782 [0.768, 0.797], chosen on the validation split. Logistic regression is 0.738, and the random forest's lead of 0.044 [0.033, 0.055] is clear. |
63
+ | With capacity to act on 10% of accounts? | It reaches **31.5% [29.0%, 33.8%] of next month's defaulters at 69.9% precision**, 3.15 times better than picking accounts at random. The validation estimate was 31.3%. |
64
+ | Is the model better than a simple rule there? | **No.** The rule "two or more months late last month" flags 10.9% and reaches 34.1%; the model at the same volume reaches 33.7%. The difference is −0.4 points [−1.7, +0.9], so the two can't be told apart. At 22.6% of accounts the model is ahead by 3.5 points [1.8, 5.2]. |
65
+ | If acting on a good customer costs NT$8,000 and a missed default loses 60% of the balance? | One score cut-off cuts the expected cost by 43.1% against the better of acting on nobody and acting on everybody. A per-account rule that uses each balance cuts it by 63.9%. The gap of 20.8 points [15.2, 27.7] is NT$7.8 million [5.4, 10.3] on the 6,002 test accounts, and the per-account rule saves more in all nine cost scenarios tested. |
66
+ | How many defaults should we plan for? | The probabilities are calibrated: **1,337 defaults expected and 1,332 occurred**, a forecast error of +0.4% [−3.5%, +4.6%]. The loss forecast is less certain, −2.7% [−9.6%, +4.6%], because a few large balances carry much of the loss. |
67
+ | Why was this account flagged? | Up to four reason codes per flagged account, with measured accuracy. Setting the stated reasons to typical values changes the decision for 100% of flagged accounts, and the top reason alone does it for 82%. "Repayment status last month" is the main reason for 83%. |
68
+ | Do we need adverse-action notices? | It depends on the rule. 99.5% of the accounts flagged at the 10% budget are already late, and actions taken because of a delinquency are not adverse action under Regulation B. The per-account cost rule flags **646 customers in good standing**, and a limit cut for them needs specific reasons. |
69
+ | Does hyperparameter tuning help? | Barely. Tuning gains 0.001 to 0.007 in cross-validated AUC, all within one standard deviation, and the tuned gradient boosting model matches the default random forest on test (ROC AUC 0.782 for both). The model is limited by the data, not its settings. |
70
+ | Is anyone treated very differently? | No group large enough to interpret falls below the four-fifths reference. But age was excluded and the inputs still predict "under 25" with ROC AUC 0.82, and under-25s are flagged at 15.1% vs 8.9% to 10.6% for other ages (their default rate is also higher, 27.7%). |
71
+ | Will it keep working? | Every scoring run reports the population stability of each input and of the score. Scoring only clients under 30 is flagged as a moderate shift in credit limits. |
72
+
73
+ The full write-up, with the industry reasoning behind each choice, is in
74
+ [docs/BUSINESS_CASE.md](https://github.com/nsteinle/creditdecide/blob/main/docs/BUSINESS_CASE.md).
75
+ Data findings and the verification log are in
76
+ [docs/FINDINGS.md](https://github.com/nsteinle/creditdecide/blob/main/docs/FINDINGS.md), and
77
+ the generated model card is
78
+ [docs/MODEL_CARD.md](https://github.com/nsteinle/creditdecide/blob/main/docs/MODEL_CARD.md).
79
+
80
+ ## Install
81
+
82
+ ```bash
83
+ pip install creditdecide # library and command line
84
+ pip install "creditdecide[plot]" # adds the figures
85
+ ```
86
+
87
+ Python 3.10 to 3.13. The dependencies are NumPy, pandas, scikit-learn, skops and DuckDB; no
88
+ GPU, database server or internet access is needed at run time.
89
+
90
+ ## Workflow
91
+
92
+ ```bash
93
+ creditdecide quality --dataset credit-default # 1. check the data
94
+ creditdecide run --dataset credit-default --out results/credit --plot # 2. train, decide, explain, export (~10 s)
95
+ creditdecide predict --model-dir results/credit/model \
96
+ --csv results/credit/holdout_test.csv --out results/scored --id-column ID --reasons 4 # 3. score and explain
97
+ creditdecide drift --model-dir results/credit/model --csv new_month.csv # 4. monitor
98
+ ```
99
+
100
+ When the outcomes arrive later, in a separate file, join them to the saved decisions:
101
+
102
+ ```bash
103
+ creditdecide outcomes --model-dir results/credit/model --scores results/scored/scores.csv \
104
+ --labels labels.csv --id-column ID --out results/outcomes.json
105
+ ```
106
+
107
+ For a dashboard, and for questions of your own about the data:
108
+
109
+ ```bash
110
+ creditdecide tableau --run-dir results/credit --scored results/scored --out results/tableau
111
+ creditdecide sql --dataset credit-default \
112
+ --query 'SELECT EDUCATION, COUNT(*) AS n, AVG("DEFAULT") AS default_rate FROM data GROUP BY ALL ORDER BY ALL'
113
+ ```
114
+
115
+ The table is called `data`. Quote the label column as `"DEFAULT"`, because `DEFAULT` is a SQL
116
+ keyword. Queries can read only that table: files and the network are out of reach.
117
+
118
+ Decision policies and rules (binary targets):
119
+
120
+ ```bash
121
+ creditdecide run ... --budget 0.10 # capacity: act on 10%
122
+ creditdecide run ... --cost-fp 8000 --lgd 0.6 --exposure-column BILL_AMT1 # cost: one cut-off
123
+ creditdecide run ... --cost-fp 8000 --lgd 0.6 --exposure-column BILL_AMT1 \
124
+ --decision-rule expected_loss # cost: per account
125
+ creditdecide run ... --cost-fp 1 --cost-fn 5 # cost: constant
126
+ creditdecide run ... --tune # tune by cross-validation (~4 min on 2 cores)
127
+ creditdecide run ... --bootstrap 0 # skip the confidence intervals
128
+ ```
129
+
130
+ Your own data (a label column plus numeric or declared categorical features):
131
+
132
+ ```bash
133
+ creditdecide run --csv loans.csv --target defaulted --positive-label 1 --drop loan_id \
134
+ --categorical region --signed-log balance income --protected age sex --out results/loans
135
+ ```
136
+
137
+ Python:
138
+
139
+ ```python
140
+ from creditdecide import CREDIT_DEFAULT_SPEC, Config, DecisionPolicy, load_credit_default, \
141
+ load_model, run
142
+
143
+ res = run(load_credit_default(), CREDIT_DEFAULT_SPEC, Config(seed=0), DecisionPolicy(budget=0.1))
144
+ res.decision["test"] # capture rate, precision, lift, ...
145
+ res.decision["simple_rule_benchmark"] # the one-line rule the model has to beat
146
+ res.calibration["test_calibrated"] # ECE, Brier, expected vs actual defaults
147
+ res.bootstrap # 95% intervals for the test results
148
+ res.reasons # reason codes for flagged test accounts
149
+ res.save("results/credit") # evaluation outputs and the deployable model
150
+
151
+ model = load_model("results/credit/model") # checksums and allowed types verified
152
+ model.decide(new_df, id_column="ID") # score, calibrated probability, flag
153
+ wide, detail, summary = model.reasons(new_df, id_column="ID") # reason codes, flagged rows
154
+ ```
155
+
156
+ ## Outputs
157
+
158
+ `run` writes metrics for every model and split; `decision.json` (rule, threshold, validation
159
+ and test outcomes, simple-rule benchmark, reason-code accuracy); `bootstrap_test.csv` (95%
160
+ intervals); `gains_test.csv`; `cost_sensitivity.csv` (both rules, cost policies only);
161
+ `calibration_cv.csv` and `calibration_test.csv`; `reasons_test.csv` and
162
+ `reasons_test_detail.csv`; `fairness_test.csv`; `tuning.csv` (with `--tune`); `samples.csv`
163
+ (one row per example); `holdout_test.csv` (the untouched test rows, for trying `predict`); and
164
+ `model/`:
165
+
166
+ | file | contents |
167
+ |---|---|
168
+ | `manifest.json` | input schema, decision rule and threshold, calibration map, reason-code factors and reference profile, test metrics, file checksums, library versions |
169
+ | `pipeline.skops` | preprocessing and scoring model, in skops format (no pickle) |
170
+ | `reference.json` | validation-split distributions for drift monitoring |
171
+ | `MODEL_CARD.md` | purpose, data, performance, decision outcome, probabilities, reason codes, uncertainty, fairness, monitoring, limitations |
172
+
173
+ `predict` writes `scores.csv` (score, calibrated probability, flag) and `drift.csv`. With
174
+ `--reasons` it also writes `reasons.csv`, `reasons_detail.csv` and `reasons_summary.json`, and
175
+ when the file contains the target it writes `outcomes.json` (realized capture, precision and
176
+ calibration compared with the figures at training time). `outcomes` writes the same file from
177
+ a separate labels file, and counts the rows that appear on only one side of the join.
178
+
179
+ `tableau` writes `accounts.csv` (one row per test account: inputs, readable group names, score,
180
+ probability, decision, score decile and top reason), `decision.csv`, `gains.csv`,
181
+ `calibration.csv`, `models.csv`, `intervals.csv`, `reasons.csv`, `fairness.csv`, `drift.csv`
182
+ (with `--scored`) and `tables.json`, which lists each file's columns and row count.
183
+
184
+ ## Design decisions
185
+
186
+ | Practice | Why |
187
+ |---|---|
188
+ | Data-quality report before any modeling | duplicates with conflicting labels, undocumented codes and class imbalance change what a good result looks like |
189
+ | Group-aware 60/20/20 split, with preprocessing fit on the training rows only | no statistic and no duplicate input crosses from the test split into training |
190
+ | Labels used only by the models | a transform that uses the labels leaks the answer into the features |
191
+ | Tuning by group cross-validation inside the training split; model, calibration and threshold chosen on validation; test used once | each split has one job, so the test result is a fair estimate of how the rule would do in use |
192
+ | Baselines and a one-feature rule always reported, at the model's own volume | a model is only worth deploying if it beats what a simple rule already does |
193
+ | Decisions from capacity or explicit costs, with savings measured against the better of two trivial strategies | a score alone doesn't say whom to act on, and beating "do nothing" can still lose to "act on everyone" |
194
+ | Calibration chosen by cross-validated log loss with the one-standard-error rule | a correction is only added when it clearly helps; calibration is monotone, so threshold decisions never change |
195
+ | Reason codes against the population that just passed, grouped into business factors, with measured accuracy | follows Regulation B comment 9(b)(2)-5, and the CFPB expects explanations of complex models to be accurate |
196
+ | Bootstrap intervals on test results, with comparisons made on the same resamples | 6,002 test accounts is one sample, and a difference that is smaller than its noise shouldn't drive a decision |
197
+ | Protected attributes excluded by default but always audited, with a proxy check | in US lending, sex and marital status are prohibited bases and age is restricted (ECOA / Regulation B), and leaving them out does not remove the information |
198
+ | Model files with checksums, a type allow-list and no pickles | safe to load and to hand to someone else |
199
+ | PSI on every input and on the score at each scoring run | the model was fit on one population and is used on later ones |
200
+ | Data-quality checks, gains table, outcomes join and drift bins in DuckDB SQL, each tested against the pandas version it replaced | set-based checks read most clearly as SQL, and the tests show the SQL gives the same numbers |
201
+ | Random forests fit on all cores but predict on one thread | with several threads the trees' predictions are added in whatever order the threads finish, so reruns could differ in the last bit and a reason with no effect could be listed; on one thread every rerun matches bit for bit |
202
+
203
+ ## Uncertainty in the results
204
+
205
+ The test split is one sample of customers, so each test figure comes with a 95% interval. The
206
+ package redraws the 6,002 test accounts with replacement 1,000 times (accounts with identical
207
+ inputs are drawn together), recomputes every figure on each redraw, and reports the middle 95%
208
+ of the results. Comparisons, such as model minus rule or one model minus another, are computed
209
+ on the same redraws, so a difference gets its own interval. The models, threshold and
210
+ calibration are held fixed, so these intervals cover the choice of test customers but not
211
+ retraining. Retraining with a different seed, which also changes the split, moves the random
212
+ forest's test AUC by an amount of the same order (0.775 to 0.782 over two seeds).
213
+
214
+ What the intervals change in the conclusions:
215
+
216
+ - At a 10% budget the model and the simple rule are indistinguishable. Even at the top of the
217
+ interval the model would reach about 12 more defaulters per 6,000 accounts, so the rule is
218
+ the sensible choice at that capacity.
219
+ - At larger volumes and for cost decisions the model's advantage is clear: 3.5 points [1.8,
220
+ 5.2] more defaulters at 22.6% of accounts, and NT$7.8 million [5.4, 10.3] more saved by the
221
+ per-account rule on the test month.
222
+ - For planning, the default count can be quoted to within about 5% and the loss to within
223
+ about 10%.
224
+ - The random forest's lead over logistic regression is clear, since its interval excludes
225
+ zero. The tuning gains (0.004 for the random forest) are much smaller than the interval
226
+ widths.
227
+
228
+ ## The SQL layer
229
+
230
+ The models are Python, and the data around them is SQL. Four parts of the package are DuckDB
231
+ queries, kept as readable `.sql` files in
232
+ [src/creditdecide/sql](https://github.com/nsteinle/creditdecide/tree/main/src/creditdecide/sql):
233
+
234
+ | query | what it does |
235
+ |---|---|
236
+ | `duplicates.sql`, `class_counts.sql`, `undocumented_codes.sql` | the data-quality report: rows that share all inputs, groups whose labels conflict (`GROUP BY ALL`), values outside the documented codes (`ANTI JOIN`) |
237
+ | `gains.sql` | the gains table: ranks the test accounts with `ROW_NUMBER()`, cuts them into deciles and accumulates capture and lift with a window |
238
+ | `outcomes.sql` | the outcomes join: decisions to labels that arrive later, by ID, with a `FULL OUTER JOIN` so that unmatched rows are counted rather than dropped |
239
+ | `psi.sql` | the drift bins: assigns each new value to its reference bin, adds a bin for unseen categories and sums the population stability index |
240
+
241
+ DuckDB runs inside the Python process and reads the pandas DataFrames in place, so there is no
242
+ server and nothing to set up. `tests/test_sql.py` runs each query on the bundled data and
243
+ checks it against the pandas code it replaced: the counts, the gains table and the outcomes
244
+ match exactly, and the PSI values match to 1e-12 (the bins are summed in a different order).
245
+ On these 30,000 rows the queries show correct SQL, not scale; every table here fits in memory
246
+ many times over.
247
+
248
+ ## Tableau Public dashboard
249
+
250
+ `creditdecide tableau` turns a run into flat CSV tables for
251
+ [Tableau Public](https://public.tableau.com), which opens text files but not Parquet. The tables
252
+ are built by the same SQL layer. [docs/TABLEAU.md](https://github.com/nsteinle/creditdecide/blob/main/docs/TABLEAU.md)
253
+ describes a five-panel dashboard built from them: the KPI tiles, the gains chart against random
254
+ selection, the reliability diagram, the reasons for flagged accounts, and the flag rate by
255
+ protected group.
256
+
257
+ ![ROC and precision-recall curves](https://raw.githubusercontent.com/nsteinle/creditdecide/main/docs/figures/roc_pr_credit_default.png)
258
+
259
+ ## Limitations
260
+
261
+ - There is one bundled dataset (Taiwan, 2005, amounts in NT$). The cost figures are
262
+ illustrative assumptions, which is why every cost run reports a sensitivity table, and the
263
+ cost model assumes that acting prevents the whole loss.
264
+ - Reason codes approximate the model by setting one factor at a time to typical values. Their
265
+ accuracy is measured, not guaranteed, and they describe the model, not causes.
266
+ - Calibration is fit on one validation sample and should be rechecked as outcomes arrive.
267
+ - The confidence intervals cover the choice of test customers only, not retraining or changes
268
+ in the population over time.
269
+ - There is no live champion-challenger testing, no real-time serving (batch scoring only) and
270
+ no monotonic constraints on the models.
271
+ - PSI uses decile bins, which can understate shifts in heavy-tailed variables (see
272
+ FINDINGS.md).
273
+ - With `--tune`, the gradient boosting model depends on the scikit-learn version, because
274
+ scikit-learn 1.8 changed it. Runs without `--tune` give the same results on every supported
275
+ version (FINDINGS.md, verification log).
276
+
277
+ ## Data, citation and license
278
+
279
+ The code is under the MIT license (`LICENSE`). The bundled data are Yeh, I.-C. (2009), *Default
280
+ of Credit Card Clients*, UCI Machine Learning Repository,
281
+ [doi:10.24432/C55S3H](https://doi.org/10.24432/C55S3H), CC BY 4.0, with the changes listed in
282
+ `DATA_LICENSE.md`. Citation metadata is in `CITATION.cff`, and the release process is in
283
+ [docs/DEPLOY.md](https://github.com/nsteinle/creditdecide/blob/main/docs/DEPLOY.md).