creditdecide 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- creditdecide-0.1.0/.gitignore +13 -0
- creditdecide-0.1.0/CHANGELOG.md +37 -0
- creditdecide-0.1.0/CITATION.cff +22 -0
- creditdecide-0.1.0/DATA_LICENSE.md +26 -0
- creditdecide-0.1.0/LICENSE +21 -0
- creditdecide-0.1.0/PKG-INFO +283 -0
- creditdecide-0.1.0/README.md +249 -0
- creditdecide-0.1.0/pyproject.toml +63 -0
- creditdecide-0.1.0/scripts/build_credit_default.py +85 -0
- creditdecide-0.1.0/scripts/check_calibration.py +48 -0
- creditdecide-0.1.0/src/creditdecide/__init__.py +40 -0
- creditdecide-0.1.0/src/creditdecide/_version.py +1 -0
- creditdecide-0.1.0/src/creditdecide/bootstrap.py +160 -0
- creditdecide-0.1.0/src/creditdecide/calibration.py +128 -0
- creditdecide-0.1.0/src/creditdecide/cli.py +377 -0
- creditdecide-0.1.0/src/creditdecide/data/__init__.py +0 -0
- creditdecide-0.1.0/src/creditdecide/data/credit_default.csv.gz +0 -0
- creditdecide-0.1.0/src/creditdecide/data/credit_default.json +17 -0
- creditdecide-0.1.0/src/creditdecide/datasets.py +198 -0
- creditdecide-0.1.0/src/creditdecide/decision.py +250 -0
- creditdecide-0.1.0/src/creditdecide/fairness.py +98 -0
- creditdecide-0.1.0/src/creditdecide/forest.py +27 -0
- creditdecide-0.1.0/src/creditdecide/metrics.py +35 -0
- creditdecide-0.1.0/src/creditdecide/modelcard.py +219 -0
- creditdecide-0.1.0/src/creditdecide/modelio.py +209 -0
- creditdecide-0.1.0/src/creditdecide/monitor.py +71 -0
- creditdecide-0.1.0/src/creditdecide/pipeline.py +552 -0
- creditdecide-0.1.0/src/creditdecide/plotting.py +64 -0
- creditdecide-0.1.0/src/creditdecide/preprocess.py +38 -0
- creditdecide-0.1.0/src/creditdecide/quality.py +61 -0
- creditdecide-0.1.0/src/creditdecide/reasons.py +188 -0
- creditdecide-0.1.0/src/creditdecide/sql/__init__.py +203 -0
- creditdecide-0.1.0/src/creditdecide/sql/class_counts.sql +5 -0
- creditdecide-0.1.0/src/creditdecide/sql/duplicates.sql +14 -0
- creditdecide-0.1.0/src/creditdecide/sql/gains.sql +37 -0
- creditdecide-0.1.0/src/creditdecide/sql/outcomes.sql +20 -0
- creditdecide-0.1.0/src/creditdecide/sql/outcomes_rows.sql +6 -0
- creditdecide-0.1.0/src/creditdecide/sql/psi.sql +44 -0
- creditdecide-0.1.0/src/creditdecide/sql/tableau_accounts.sql +25 -0
- creditdecide-0.1.0/src/creditdecide/sql/tableau_gains.sql +7 -0
- creditdecide-0.1.0/src/creditdecide/sql/tableau_models.sql +6 -0
- creditdecide-0.1.0/src/creditdecide/sql/tableau_reasons.sql +9 -0
- creditdecide-0.1.0/src/creditdecide/sql/undocumented_codes.sql +10 -0
- creditdecide-0.1.0/src/creditdecide/tableau.py +133 -0
- creditdecide-0.1.0/src/creditdecide/tuning.py +137 -0
- creditdecide-0.1.0/tests/conftest.py +55 -0
- creditdecide-0.1.0/tests/pandas_reference.py +111 -0
- creditdecide-0.1.0/tests/test_bootstrap.py +111 -0
- creditdecide-0.1.0/tests/test_calibration_reasons_tuning.py +292 -0
- creditdecide-0.1.0/tests/test_cli.py +253 -0
- creditdecide-0.1.0/tests/test_data.py +110 -0
- creditdecide-0.1.0/tests/test_decision.py +98 -0
- creditdecide-0.1.0/tests/test_modelio.py +104 -0
- creditdecide-0.1.0/tests/test_models.py +116 -0
- creditdecide-0.1.0/tests/test_monitoring.py +98 -0
- creditdecide-0.1.0/tests/test_sql.py +170 -0
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0 (2026-11-07)
|
|
4
|
+
|
|
5
|
+
First public release.
|
|
6
|
+
|
|
7
|
+
- Data-quality report (`creditdecide quality`): duplicate rows and duplicates with conflicting
|
|
8
|
+
labels, counted on all columns and on the model inputs; undocumented category codes; missing
|
|
9
|
+
values; constant columns; class balance.
|
|
10
|
+
- Group-aware stratified 60/20/20 split, so rows with identical model inputs never cross
|
|
11
|
+
splits, with preprocessing fit on the training split only.
|
|
12
|
+
- Models: a majority-class baseline, logistic regression and a random forest. Optional tuning
|
|
13
|
+
(`--tune`) by group cross-validation inside the training split, which adds histogram
|
|
14
|
+
gradient boosting; its folds and candidates are drawn by the package, so they do not depend
|
|
15
|
+
on the scikit-learn version.
|
|
16
|
+
- Decision layer: capacity (budget) and cost policies with thresholds chosen on validation, a
|
|
17
|
+
per-account expected-loss rule (`--decision-rule expected_loss`), a cost sensitivity table,
|
|
18
|
+
and the best one-feature rule as a benchmark at the model's own volume.
|
|
19
|
+
- Probability calibration (identity, sigmoid or isotonic) chosen by cross-validated log loss
|
|
20
|
+
with the one-standard-error rule, with reliability tables and expected vs actual defaults
|
|
21
|
+
and losses.
|
|
22
|
+
- Reason codes for flagged accounts, grouped into business factors, with measured accuracy.
|
|
23
|
+
- 95% bootstrap intervals for the test results (`bootstrap_test.csv`, `--bootstrap N`), with
|
|
24
|
+
comparisons computed on the same redraws.
|
|
25
|
+
- Fairness audit by protected group, with adverse impact ratios and a proxy check.
|
|
26
|
+
- A saved model directory with SHA-256 checksums, a skops type allow-list and a generated model
|
|
27
|
+
card; `predict`, `drift`, and outcomes analysis when labels arrive.
|
|
28
|
+
- A DuckDB SQL data layer (`creditdecide.sql`): the data-quality counts, the gains table, the
|
|
29
|
+
outcomes join and the drift bins, each tested against the pandas version it replaced.
|
|
30
|
+
`creditdecide outcomes` joins saved decisions to labels that arrive later, and
|
|
31
|
+
`creditdecide sql` runs a read-only query on the data.
|
|
32
|
+
- `creditdecide tableau` writes a run's results as CSV tables for a Tableau Public dashboard
|
|
33
|
+
(docs/TABLEAU.md).
|
|
34
|
+
- Random forests are fitted on all cores but predict on one thread (`creditdecide.forest`), so
|
|
35
|
+
reruns match bit for bit.
|
|
36
|
+
- Bundled UCI credit-card default data (CC BY 4.0), CI on Python 3.10, 3.12 and 3.13, a Docker
|
|
37
|
+
image, and a PyPI Trusted Publishing workflow.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
cff-version: 1.2.0
|
|
2
|
+
message: "If you use this software, please cite it as below."
|
|
3
|
+
title: "creditdecide"
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
date-released: 2026-11-07
|
|
6
|
+
license: MIT
|
|
7
|
+
type: software
|
|
8
|
+
authors:
|
|
9
|
+
- family-names: Steinle
|
|
10
|
+
given-names: Nathan
|
|
11
|
+
orcid: "https://orcid.org/0000-0003-0658-402X"
|
|
12
|
+
references:
|
|
13
|
+
- type: data
|
|
14
|
+
title: "Default of Credit Card Clients"
|
|
15
|
+
authors:
|
|
16
|
+
- family-names: Yeh
|
|
17
|
+
given-names: I-Cheng
|
|
18
|
+
publisher:
|
|
19
|
+
name: "UCI Machine Learning Repository"
|
|
20
|
+
year: 2009
|
|
21
|
+
doi: "10.24432/C55S3H"
|
|
22
|
+
license: CC-BY-4.0
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Bundled data
|
|
2
|
+
|
|
3
|
+
`src/creditdecide/data/credit_default.csv.gz`
|
|
4
|
+
|
|
5
|
+
- **Title:** Default of Credit Card Clients
|
|
6
|
+
- **Creator:** I.-C. Yeh
|
|
7
|
+
- **Source:** UCI Machine Learning Repository, https://archive.ics.uci.edu/dataset/350/default+of+credit+card+clients
|
|
8
|
+
- **DOI:** 10.24432/C55S3H
|
|
9
|
+
- **License:** Creative Commons Attribution 4.0 International (CC BY 4.0),
|
|
10
|
+
https://creativecommons.org/licenses/by/4.0/
|
|
11
|
+
- **Reference:** Yeh, I.-C. & Lien, C.-H. (2009). The comparisons of data mining techniques
|
|
12
|
+
for the predictive accuracy of probability of default of credit card clients. *Expert Systems
|
|
13
|
+
with Applications*, 36(2), 2473–2480.
|
|
14
|
+
|
|
15
|
+
## Changes made
|
|
16
|
+
|
|
17
|
+
Produced from the official archive (SHA-256
|
|
18
|
+
`56c885f84457f6680f8438f02bfcdac9579323d8a94465ee5f26e32baa727602`) by
|
|
19
|
+
`scripts/build_credit_default.py`:
|
|
20
|
+
|
|
21
|
+
1. the first header row (`X1` … `X23`, `Y`) was dropped, keeping the descriptive column names;
|
|
22
|
+
2. the column `default payment next month` was renamed `DEFAULT`;
|
|
23
|
+
3. the table was stored as gzip-compressed CSV.
|
|
24
|
+
|
|
25
|
+
No values were changed, added or removed (30,000 rows, 25 columns). The SHA-256 of the
|
|
26
|
+
bundled file is recorded in `credit_default.json` and checked on every load.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025-2026 Nathan Steinle
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,283 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: creditdecide
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: From a labeled table to a monitored, explainable decision rule: leakage-safe model comparison, calibrated probabilities, cost-based decisions, reason codes, confidence intervals, drift and fairness checks, with a DuckDB SQL data layer and Tableau-ready exports.
|
|
5
|
+
Project-URL: Source, https://github.com/nsteinle/creditdecide
|
|
6
|
+
Project-URL: Dataset, https://doi.org/10.24432/C55S3H
|
|
7
|
+
Author: Nathan Steinle
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: DATA_LICENSE.md
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: calibration,credit risk,data quality,decision rules,duckdb,model monitoring,reason codes,sql,tableau
|
|
12
|
+
Classifier: Intended Audience :: Science/Research
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
17
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
18
|
+
Requires-Python: >=3.10
|
|
19
|
+
Requires-Dist: duckdb>=1.1
|
|
20
|
+
Requires-Dist: numpy>=1.24
|
|
21
|
+
Requires-Dist: pandas>=2.0
|
|
22
|
+
Requires-Dist: scikit-learn>=1.4
|
|
23
|
+
Requires-Dist: skops>=0.10
|
|
24
|
+
Provides-Extra: dev
|
|
25
|
+
Requires-Dist: build; extra == 'dev'
|
|
26
|
+
Requires-Dist: matplotlib>=3.7; extra == 'dev'
|
|
27
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
28
|
+
Requires-Dist: ruff>=0.5; extra == 'dev'
|
|
29
|
+
Requires-Dist: twine; extra == 'dev'
|
|
30
|
+
Requires-Dist: xlrd>=2.0; extra == 'dev'
|
|
31
|
+
Provides-Extra: plot
|
|
32
|
+
Requires-Dist: matplotlib>=3.7; extra == 'plot'
|
|
33
|
+
Description-Content-Type: text/markdown
|
|
34
|
+
|
|
35
|
+
# creditdecide
|
|
36
|
+
|
|
37
|
+
creditdecide takes a labeled table and turns it into a decision rule that can be explained and
|
|
38
|
+
monitored. It checks the data quality first, compares models without leaking test data into
|
|
39
|
+
training, benchmarks the best model against the best one-line rule, calibrates the
|
|
40
|
+
probabilities, picks which accounts to act on from a capacity or a cost budget, gives reason
|
|
41
|
+
codes for every flagged account, puts 95% confidence intervals on the test results, and saves a
|
|
42
|
+
checksum-verified model with drift monitoring and a fairness audit. Its data and reporting
|
|
43
|
+
layer (the data-quality checks, the gains table, the outcomes join and the drift bins) is DuckDB
|
|
44
|
+
SQL, tested against the pandas results, and `creditdecide tableau` writes a run's results as
|
|
45
|
+
tables for a Tableau Public dashboard. It ships with a real credit-card default dataset and
|
|
46
|
+
also accepts your own CSV.
|
|
47
|
+
|
|
48
|
+
**The business question on the bundled data:** a card issuer can only act on a limited number
|
|
49
|
+
of accounts each month (a call, a payment plan, a lower limit). Which accounts should it act on,
|
|
50
|
+
how many of next month's defaults will that reach, how much money will it save, is a model
|
|
51
|
+
better than a simple rule, can each decision be explained, and does the rule treat protected
|
|
52
|
+
groups comparably?
|
|
53
|
+
|
|
54
|
+
## What a run produces (credit-card default, test split of 6,002 clients)
|
|
55
|
+
|
|
56
|
+
The ranges in brackets are 95% bootstrap intervals over the test accounts (see
|
|
57
|
+
[Uncertainty](#uncertainty-in-the-results)).
|
|
58
|
+
|
|
59
|
+
| Question | Answer |
|
|
60
|
+
|---|---|
|
|
61
|
+
| How good is "do nothing"? | Predicting "no default" for everyone is 77.9% accurate, so accuracy is the wrong measure here. I use ranking, capture and cost instead. |
|
|
62
|
+
| Best ranking model? | Random forest, ROC AUC 0.782 [0.768, 0.797], chosen on the validation split. Logistic regression is 0.738, and the random forest's lead of 0.044 [0.033, 0.055] is clear. |
|
|
63
|
+
| With capacity to act on 10% of accounts? | It reaches **31.5% [29.0%, 33.8%] of next month's defaulters at 69.9% precision**, 3.15 times better than picking accounts at random. The validation estimate was 31.3%. |
|
|
64
|
+
| Is the model better than a simple rule there? | **No.** The rule "two or more months late last month" flags 10.9% and reaches 34.1%; the model at the same volume reaches 33.7%. The difference is −0.4 points [−1.7, +0.9], so the two can't be told apart. At 22.6% of accounts the model is ahead by 3.5 points [1.8, 5.2]. |
|
|
65
|
+
| If acting on a good customer costs NT$8,000 and a missed default loses 60% of the balance? | One score cut-off cuts the expected cost by 43.1% against the better of acting on nobody and acting on everybody. A per-account rule that uses each balance cuts it by 63.9%. The gap of 20.8 points [15.2, 27.7] is NT$7.8 million [5.4, 10.3] on the 6,002 test accounts, and the per-account rule saves more in all nine cost scenarios tested. |
|
|
66
|
+
| How many defaults should we plan for? | The probabilities are calibrated: **1,337 defaults expected and 1,332 occurred**, a forecast error of +0.4% [−3.5%, +4.6%]. The loss forecast is less certain, −2.7% [−9.6%, +4.6%], because a few large balances carry much of the loss. |
|
|
67
|
+
| Why was this account flagged? | Up to four reason codes per flagged account, with measured accuracy. Setting the stated reasons to typical values changes the decision for 100% of flagged accounts, and the top reason alone does it for 82%. "Repayment status last month" is the main reason for 83%. |
|
|
68
|
+
| Do we need adverse-action notices? | It depends on the rule. 99.5% of the accounts flagged at the 10% budget are already late, and actions taken because of a delinquency are not adverse action under Regulation B. The per-account cost rule flags **646 customers in good standing**, and a limit cut for them needs specific reasons. |
|
|
69
|
+
| Does hyperparameter tuning help? | Barely. Tuning gains 0.001 to 0.007 in cross-validated AUC, all within one standard deviation, and the tuned gradient boosting model matches the default random forest on test (ROC AUC 0.782 for both). The model is limited by the data, not its settings. |
|
|
70
|
+
| Is anyone treated very differently? | No group large enough to interpret falls below the four-fifths reference. But age was excluded and the inputs still predict "under 25" with ROC AUC 0.82, and under-25s are flagged at 15.1% vs 8.9% to 10.6% for other ages (their default rate is also higher, 27.7%). |
|
|
71
|
+
| Will it keep working? | Every scoring run reports the population stability of each input and of the score. Scoring only clients under 30 is flagged as a moderate shift in credit limits. |
|
|
72
|
+
|
|
73
|
+
The full write-up, with the industry reasoning behind each choice, is in
|
|
74
|
+
[docs/BUSINESS_CASE.md](https://github.com/nsteinle/creditdecide/blob/main/docs/BUSINESS_CASE.md).
|
|
75
|
+
Data findings and the verification log are in
|
|
76
|
+
[docs/FINDINGS.md](https://github.com/nsteinle/creditdecide/blob/main/docs/FINDINGS.md), and
|
|
77
|
+
the generated model card is
|
|
78
|
+
[docs/MODEL_CARD.md](https://github.com/nsteinle/creditdecide/blob/main/docs/MODEL_CARD.md).
|
|
79
|
+
|
|
80
|
+
## Install
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
pip install creditdecide # library and command line
|
|
84
|
+
pip install "creditdecide[plot]" # adds the figures
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Python 3.10 to 3.13. The dependencies are NumPy, pandas, scikit-learn, skops and DuckDB; no
|
|
88
|
+
GPU, database server or internet access is needed at run time.
|
|
89
|
+
|
|
90
|
+
## Workflow
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
creditdecide quality --dataset credit-default # 1. check the data
|
|
94
|
+
creditdecide run --dataset credit-default --out results/credit --plot # 2. train, decide, explain, export (~10 s)
|
|
95
|
+
creditdecide predict --model-dir results/credit/model \
|
|
96
|
+
--csv results/credit/holdout_test.csv --out results/scored --id-column ID --reasons 4 # 3. score and explain
|
|
97
|
+
creditdecide drift --model-dir results/credit/model --csv new_month.csv # 4. monitor
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
When the outcomes arrive later, in a separate file, join them to the saved decisions:
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
creditdecide outcomes --model-dir results/credit/model --scores results/scored/scores.csv \
|
|
104
|
+
--labels labels.csv --id-column ID --out results/outcomes.json
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
For a dashboard, and for questions of your own about the data:
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
creditdecide tableau --run-dir results/credit --scored results/scored --out results/tableau
|
|
111
|
+
creditdecide sql --dataset credit-default \
|
|
112
|
+
--query 'SELECT EDUCATION, COUNT(*) AS n, AVG("DEFAULT") AS default_rate FROM data GROUP BY ALL ORDER BY ALL'
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
The table is called `data`. Quote the label column as `"DEFAULT"`, because `DEFAULT` is a SQL
|
|
116
|
+
keyword. Queries can read only that table: files and the network are out of reach.
|
|
117
|
+
|
|
118
|
+
Decision policies and rules (binary targets):
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
creditdecide run ... --budget 0.10 # capacity: act on 10%
|
|
122
|
+
creditdecide run ... --cost-fp 8000 --lgd 0.6 --exposure-column BILL_AMT1 # cost: one cut-off
|
|
123
|
+
creditdecide run ... --cost-fp 8000 --lgd 0.6 --exposure-column BILL_AMT1 \
|
|
124
|
+
--decision-rule expected_loss # cost: per account
|
|
125
|
+
creditdecide run ... --cost-fp 1 --cost-fn 5 # cost: constant
|
|
126
|
+
creditdecide run ... --tune # tune by cross-validation (~4 min on 2 cores)
|
|
127
|
+
creditdecide run ... --bootstrap 0 # skip the confidence intervals
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Your own data (a label column plus numeric or declared categorical features):
|
|
131
|
+
|
|
132
|
+
```bash
|
|
133
|
+
creditdecide run --csv loans.csv --target defaulted --positive-label 1 --drop loan_id \
|
|
134
|
+
--categorical region --signed-log balance income --protected age sex --out results/loans
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Python:
|
|
138
|
+
|
|
139
|
+
```python
|
|
140
|
+
from creditdecide import CREDIT_DEFAULT_SPEC, Config, DecisionPolicy, load_credit_default, \
|
|
141
|
+
load_model, run
|
|
142
|
+
|
|
143
|
+
res = run(load_credit_default(), CREDIT_DEFAULT_SPEC, Config(seed=0), DecisionPolicy(budget=0.1))
|
|
144
|
+
res.decision["test"] # capture rate, precision, lift, ...
|
|
145
|
+
res.decision["simple_rule_benchmark"] # the one-line rule the model has to beat
|
|
146
|
+
res.calibration["test_calibrated"] # ECE, Brier, expected vs actual defaults
|
|
147
|
+
res.bootstrap # 95% intervals for the test results
|
|
148
|
+
res.reasons # reason codes for flagged test accounts
|
|
149
|
+
res.save("results/credit") # evaluation outputs and the deployable model
|
|
150
|
+
|
|
151
|
+
model = load_model("results/credit/model") # checksums and allowed types verified
|
|
152
|
+
model.decide(new_df, id_column="ID") # score, calibrated probability, flag
|
|
153
|
+
wide, detail, summary = model.reasons(new_df, id_column="ID") # reason codes, flagged rows
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
## Outputs
|
|
157
|
+
|
|
158
|
+
`run` writes metrics for every model and split; `decision.json` (rule, threshold, validation
|
|
159
|
+
and test outcomes, simple-rule benchmark, reason-code accuracy); `bootstrap_test.csv` (95%
|
|
160
|
+
intervals); `gains_test.csv`; `cost_sensitivity.csv` (both rules, cost policies only);
|
|
161
|
+
`calibration_cv.csv` and `calibration_test.csv`; `reasons_test.csv` and
|
|
162
|
+
`reasons_test_detail.csv`; `fairness_test.csv`; `tuning.csv` (with `--tune`); `samples.csv`
|
|
163
|
+
(one row per example); `holdout_test.csv` (the untouched test rows, for trying `predict`); and
|
|
164
|
+
`model/`:
|
|
165
|
+
|
|
166
|
+
| file | contents |
|
|
167
|
+
|---|---|
|
|
168
|
+
| `manifest.json` | input schema, decision rule and threshold, calibration map, reason-code factors and reference profile, test metrics, file checksums, library versions |
|
|
169
|
+
| `pipeline.skops` | preprocessing and scoring model, in skops format (no pickle) |
|
|
170
|
+
| `reference.json` | validation-split distributions for drift monitoring |
|
|
171
|
+
| `MODEL_CARD.md` | purpose, data, performance, decision outcome, probabilities, reason codes, uncertainty, fairness, monitoring, limitations |
|
|
172
|
+
|
|
173
|
+
`predict` writes `scores.csv` (score, calibrated probability, flag) and `drift.csv`. With
|
|
174
|
+
`--reasons` it also writes `reasons.csv`, `reasons_detail.csv` and `reasons_summary.json`, and
|
|
175
|
+
when the file contains the target it writes `outcomes.json` (realized capture, precision and
|
|
176
|
+
calibration compared with the figures at training time). `outcomes` writes the same file from
|
|
177
|
+
a separate labels file, and counts the rows that appear on only one side of the join.
|
|
178
|
+
|
|
179
|
+
`tableau` writes `accounts.csv` (one row per test account: inputs, readable group names, score,
|
|
180
|
+
probability, decision, score decile and top reason), `decision.csv`, `gains.csv`,
|
|
181
|
+
`calibration.csv`, `models.csv`, `intervals.csv`, `reasons.csv`, `fairness.csv`, `drift.csv`
|
|
182
|
+
(with `--scored`) and `tables.json`, which lists each file's columns and row count.
|
|
183
|
+
|
|
184
|
+
## Design decisions
|
|
185
|
+
|
|
186
|
+
| Practice | Why |
|
|
187
|
+
|---|---|
|
|
188
|
+
| Data-quality report before any modeling | duplicates with conflicting labels, undocumented codes and class imbalance change what a good result looks like |
|
|
189
|
+
| Group-aware 60/20/20 split, with preprocessing fit on the training rows only | no statistic and no duplicate input crosses from the test split into training |
|
|
190
|
+
| Labels used only by the models | a transform that uses the labels leaks the answer into the features |
|
|
191
|
+
| Tuning by group cross-validation inside the training split; model, calibration and threshold chosen on validation; test used once | each split has one job, so the test result is a fair estimate of how the rule would do in use |
|
|
192
|
+
| Baselines and a one-feature rule always reported, at the model's own volume | a model is only worth deploying if it beats what a simple rule already does |
|
|
193
|
+
| Decisions from capacity or explicit costs, with savings measured against the better of two trivial strategies | a score alone doesn't say whom to act on, and beating "do nothing" can still lose to "act on everyone" |
|
|
194
|
+
| Calibration chosen by cross-validated log loss with the one-standard-error rule | a correction is only added when it clearly helps; calibration is monotone, so threshold decisions never change |
|
|
195
|
+
| Reason codes against the population that just passed, grouped into business factors, with measured accuracy | follows Regulation B comment 9(b)(2)-5, and the CFPB expects explanations of complex models to be accurate |
|
|
196
|
+
| Bootstrap intervals on test results, with comparisons made on the same resamples | 6,002 test accounts is one sample, and a difference that is smaller than its noise shouldn't drive a decision |
|
|
197
|
+
| Protected attributes excluded by default but always audited, with a proxy check | in US lending, sex and marital status are prohibited bases and age is restricted (ECOA / Regulation B), and leaving them out does not remove the information |
|
|
198
|
+
| Model files with checksums, a type allow-list and no pickles | safe to load and to hand to someone else |
|
|
199
|
+
| PSI on every input and on the score at each scoring run | the model was fit on one population and is used on later ones |
|
|
200
|
+
| Data-quality checks, gains table, outcomes join and drift bins in DuckDB SQL, each tested against the pandas version it replaced | set-based checks read most clearly as SQL, and the tests show the SQL gives the same numbers |
|
|
201
|
+
| Random forests fit on all cores but predict on one thread | with several threads the trees' predictions are added in whatever order the threads finish, so reruns could differ in the last bit and a reason with no effect could be listed; on one thread every rerun matches bit for bit |
|
|
202
|
+
|
|
203
|
+
## Uncertainty in the results
|
|
204
|
+
|
|
205
|
+
The test split is one sample of customers, so each test figure comes with a 95% interval. The
|
|
206
|
+
package redraws the 6,002 test accounts with replacement 1,000 times (accounts with identical
|
|
207
|
+
inputs are drawn together), recomputes every figure on each redraw, and reports the middle 95%
|
|
208
|
+
of the results. Comparisons, such as model minus rule or one model minus another, are computed
|
|
209
|
+
on the same redraws, so a difference gets its own interval. The models, threshold and
|
|
210
|
+
calibration are held fixed, so these intervals cover the choice of test customers but not
|
|
211
|
+
retraining. Retraining with a different seed, which also changes the split, moves the random
|
|
212
|
+
forest's test AUC by an amount of the same order (0.775 to 0.782 over two seeds).
|
|
213
|
+
|
|
214
|
+
What the intervals change in the conclusions:
|
|
215
|
+
|
|
216
|
+
- At a 10% budget the model and the simple rule are indistinguishable. Even at the top of the
|
|
217
|
+
interval the model would reach about 12 more defaulters per 6,000 accounts, so the rule is
|
|
218
|
+
the sensible choice at that capacity.
|
|
219
|
+
- At larger volumes and for cost decisions the model's advantage is clear: 3.5 points [1.8,
|
|
220
|
+
5.2] more defaulters at 22.6% of accounts, and NT$7.8 million [5.4, 10.3] more saved by the
|
|
221
|
+
per-account rule on the test month.
|
|
222
|
+
- For planning, the default count can be quoted to within about 5% and the loss to within
|
|
223
|
+
about 10%.
|
|
224
|
+
- The random forest's lead over logistic regression is clear, since its interval excludes
|
|
225
|
+
zero. The tuning gains (0.004 for the random forest) are much smaller than the interval
|
|
226
|
+
widths.
|
|
227
|
+
|
|
228
|
+
## The SQL layer
|
|
229
|
+
|
|
230
|
+
The models are Python, and the data around them is SQL. Four parts of the package are DuckDB
|
|
231
|
+
queries, kept as readable `.sql` files in
|
|
232
|
+
[src/creditdecide/sql](https://github.com/nsteinle/creditdecide/tree/main/src/creditdecide/sql):
|
|
233
|
+
|
|
234
|
+
| query | what it does |
|
|
235
|
+
|---|---|
|
|
236
|
+
| `duplicates.sql`, `class_counts.sql`, `undocumented_codes.sql` | the data-quality report: rows that share all inputs, groups whose labels conflict (`GROUP BY ALL`), values outside the documented codes (`ANTI JOIN`) |
|
|
237
|
+
| `gains.sql` | the gains table: ranks the test accounts with `ROW_NUMBER()`, cuts them into deciles and accumulates capture and lift with a window |
|
|
238
|
+
| `outcomes.sql` | the outcomes join: decisions to labels that arrive later, by ID, with a `FULL OUTER JOIN` so that unmatched rows are counted rather than dropped |
|
|
239
|
+
| `psi.sql` | the drift bins: assigns each new value to its reference bin, adds a bin for unseen categories and sums the population stability index |
|
|
240
|
+
|
|
241
|
+
DuckDB runs inside the Python process and reads the pandas DataFrames in place, so there is no
|
|
242
|
+
server and nothing to set up. `tests/test_sql.py` runs each query on the bundled data and
|
|
243
|
+
checks it against the pandas code it replaced: the counts, the gains table and the outcomes
|
|
244
|
+
match exactly, and the PSI values match to 1e-12 (the bins are summed in a different order).
|
|
245
|
+
On these 30,000 rows the queries show correct SQL, not scale; every table here fits in memory
|
|
246
|
+
many times over.
|
|
247
|
+
|
|
248
|
+
## Tableau Public dashboard
|
|
249
|
+
|
|
250
|
+
`creditdecide tableau` turns a run into flat CSV tables for
|
|
251
|
+
[Tableau Public](https://public.tableau.com), which opens text files but not Parquet. The tables
|
|
252
|
+
are built by the same SQL layer. [docs/TABLEAU.md](https://github.com/nsteinle/creditdecide/blob/main/docs/TABLEAU.md)
|
|
253
|
+
describes a five-panel dashboard built from them: the KPI tiles, the gains chart against random
|
|
254
|
+
selection, the reliability diagram, the reasons for flagged accounts, and the flag rate by
|
|
255
|
+
protected group.
|
|
256
|
+
|
|
257
|
+

|
|
258
|
+
|
|
259
|
+
## Limitations
|
|
260
|
+
|
|
261
|
+
- There is one bundled dataset (Taiwan, 2005, amounts in NT$). The cost figures are
|
|
262
|
+
illustrative assumptions, which is why every cost run reports a sensitivity table, and the
|
|
263
|
+
cost model assumes that acting prevents the whole loss.
|
|
264
|
+
- Reason codes approximate the model by setting one factor at a time to typical values. Their
|
|
265
|
+
accuracy is measured, not guaranteed, and they describe the model, not causes.
|
|
266
|
+
- Calibration is fit on one validation sample and should be rechecked as outcomes arrive.
|
|
267
|
+
- The confidence intervals cover the choice of test customers only, not retraining or changes
|
|
268
|
+
in the population over time.
|
|
269
|
+
- There is no live champion-challenger testing, no real-time serving (batch scoring only) and
|
|
270
|
+
no monotonic constraints on the models.
|
|
271
|
+
- PSI uses decile bins, which can understate shifts in heavy-tailed variables (see
|
|
272
|
+
FINDINGS.md).
|
|
273
|
+
- With `--tune`, the gradient boosting model depends on the scikit-learn version, because
|
|
274
|
+
scikit-learn 1.8 changed it. Runs without `--tune` give the same results on every supported
|
|
275
|
+
version (FINDINGS.md, verification log).
|
|
276
|
+
|
|
277
|
+
## Data, citation and license
|
|
278
|
+
|
|
279
|
+
The code is under the MIT license (`LICENSE`). The bundled data are Yeh, I.-C. (2009), *Default
|
|
280
|
+
of Credit Card Clients*, UCI Machine Learning Repository,
|
|
281
|
+
[doi:10.24432/C55S3H](https://doi.org/10.24432/C55S3H), CC BY 4.0, with the changes listed in
|
|
282
|
+
`DATA_LICENSE.md`. Citation metadata is in `CITATION.cff`, and the release process is in
|
|
283
|
+
[docs/DEPLOY.md](https://github.com/nsteinle/creditdecide/blob/main/docs/DEPLOY.md).
|