feature-pruning 0.1.0__tar.gz

feature_pruning-0.1.0/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Vrukshya Org
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

feature_pruning-0.1.0/PKG-INFO

@@ -0,0 +1,330 @@
+Metadata-Version: 2.4
+Name: feature_pruning
+Version: 0.1.0
+Summary: High-performance PySpark library for scalable feature reduction and Information Value (IV) pruning in Credit Risk Scorecards
+Author-email: Vrukshya <vrukshyaai@gmail.com>
+Maintainer-email: Vrukshya <vrukshyaai@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 Vrukshya Org
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+Project-URL: Homepage, https://github.com/vrukshya/feature_pruning
+Project-URL: Documentation, https://github.com/vrukshya/feature_pruning#readme
+Project-URL: Repository, https://github.com/vrukshya/feature_pruning
+Project-URL: Bug Tracker, https://github.com/vrukshya/feature_pruning/issues
+Keywords: credit-risk,credit-scoring,scorecard,feature-reduction,feature-pruning,feature-selection,information-value,woe,pyspark,fintech,machine-learning
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Financial and Insurance Industry
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Office/Business :: Financial
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: pandas>=1.3.0
+Requires-Dist: pyspark>=3.1.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: flake8>=6.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Dynamic: license-file
+
+# feature_pruning
+
+[![Python Version](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue.svg)](https://pypi.org/project/feature-pruning/)
+[![PySpark](https://img.shields.io/badge/PySpark-%3E%3D3.1.0-orange.svg)](https://spark.apache.org/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
+**High-Performance Distributed Feature Reduction Engine for Credit Risk Scorecards.**
+
+`feature_pruning` is an enterprise-grade Python library built natively on Apache Spark for selecting and pruning variables in credit risk scorecard development (Probability of Default / Basel II/III / IFRS 9 / Retail Scorecards).
+
+It automates the transition from thousands of raw credit bureau, transaction, and demographic attributes down to an optimal, highly predictive, non-collinear feature set ready for Weight of Evidence (WoE) binning and Logistic Regression.
+
+---
+
+## The Credit Risk Scorecard Challenge
+
+Building regulatory-compliant credit scorecards presents unique data engineering and modeling hurdles:
+
+- **Audit & Governance (SR 11-7 / Basel / IFRS 9)**: Model risk management (MRM) and regulatory auditors require an explicit justification for every discarded or retained variable.
+- **Extreme Multicollinearity**: Credit bureau tables often contain dozens of collinear metrics (e.g., `num_inquiries_3m`, `num_inquiries_6m`, `num_inquiries_12m`). In standard logistic regression scorecards, collinearity causes unstable coefficients and counter-intuitive sign reversals.
+- **Predictive Quality**: Features must meet minimum **Information Value (IV)** standards while preserving mandatory business or regulatory key indicators.
+- **Big Data Scale**: Modern credit datasets often span millions of accounts and thousands of features. Single-machine libraries (pandas/scikit-learn) crash with `OutOfMemory` errors when computing quantile cuts and pairwise correlations.
+
+`feature_pruning` solves these challenges by running distributed quantile binning, IV evaluation, and correlation pruning entirely within **Apache Spark**.
+
+---
+
+## Pipeline Architecture
+
+```
+                           Raw Spark DataFrame (Millions of Rows, 1000s of Features)
+                                                   │
+                                                   ▼
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Stage 1: Feature Classification                                                                        │
+│ • Detects Numeric vs. Categorical vs. Datetime columns                                                 │
+│ • Flags & filters high-cardinality strings (> max_cat_levels)                                          │
+│ • Protects mandatory features and excludes requested columns                                          │
+└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
+                                                   │
+                                                   ▼
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Stage 2: Distributed Precision Downcasting                                                             │
+│ • Downcasts DoubleType & DecimalType to 32-bit FloatType                                              │
+│ • Cuts executor memory consumption by ~50% during matrix aggregation                                  │
+└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
+                                                   │
+                                                   ▼
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Stage 3: High-Throughput Information Value (IV) Computation                                            │
+│ • Computes continuous quantiles and distributed binning via Spark mapInPandas                         │
+│ • Aggregates Goods/Bads and calculates IV for numeric and categorical variables                        │
+│ • Filters out variables with IV < iv_threshold                                                         │
+└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
+                                                   │
+                                                   ▼
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Stage 4: IV-Prioritized Correlation Pruning                                                            │
+│ • Computes Pearson correlation matrix via PySpark VectorAssembler + Correlation                        │
+│ • Between collinear pairs (r >= corr_threshold), retains the feature with higher IV                    │
+│ • Dynamic threshold relaxation ensures minimum required feature count is met                          │
+└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
+                                                   │
+                                                   ▼
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Stage 5: Full Audit Reporting & Data Delivery                                                          │
+│ • Generates comprehensive audit DataFrame (is_selected, iv, correlated_with, exclusion_reason)         │
+│ • Outputs pruned Spark DataFrame ready for Weight of Evidence (WoE) binning                           │
+└────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Key Features
+
+- ⚡ **Native PySpark Scalability**: Distributed quantile histograms and correlation matrices executed directly on cluster workers via Spark ML and `mapInPandas`.
+- 📊 **IV-Driven Pruning**: When two features are collinear, the pipeline drops the weaker predictor and retains the variable with higher Information Value.
+- 🛡️ **Mandatory Variable Protection**: Ensure business-critical variables (e.g., debt-to-income, credit bureau score) are never removed, regardless of their statistical properties.
+- 📋 **Regulatory Audit Trail**: Automatically produces a full governance table explaining why every single column was accepted or eliminated (e.g., `low_iv (0.012)`, `correlated (r=0.962)`, `high_cardinality (>200)`, or `datetime_column`).
+- 🔄 **Dynamic Threshold Relaxation**: Automatically adjusts correlation thresholds if filtering becomes overly aggressive, keeping feature counts within target bounds (`expected_final_min`, `expected_final_max`).
+
+---
+
+## Installation
+
+### From PyPI
+```bash
+pip install feature-pruning
+```
+
+### For Local Development
+```bash
+git clone https://github.com/vrukshya/feature_pruning.git
+cd feature_pruning
+pip install -e ".[dev]"
+```
+
+### In Databricks / Cloud Notebooks
+In your Databricks notebook cell:
+```python
+%pip install feature-pruning
+```
+Or add `feature-pruning` to your Databricks cluster libraries.
+
+---
+
+## Quickstart
+
+```python
+from pyspark.sql import SparkSession
+from feature_pruning import FeatureSelectionPipeline
+
+# 1. Initialize Spark session (or use active session in Databricks/EMR)
+spark = SparkSession.builder.appName("CreditRiskScorecard").getOrCreate()
+
+# 2. Load credit training data
+df = spark.table("risk_catalog.credit_data.application_train")
+
+# 3. Configure the feature selection pipeline
+pipeline = FeatureSelectionPipeline(
+    df=df,
+    target_col="default_flag",              # 0 = Good loan, 1 = Default / Bad loan
+    mandatory_features=["bureau_score", "dti_ratio"],  # Keep regardless of correlation
+    exclude_features=["application_id", "ssn_hash"],   # Exclude identifiers
+    iv_threshold=0.03,                      # Industry baseline: IV >= 0.03
+    corr_threshold=0.95,                    # Multicollinearity cutoff
+    expected_final_min=20,
+    expected_final_max=150,
+    verbose=True,
+)
+
+# 4. Execute pipeline
+selected_spark_df, audit_report = pipeline.run()
+
+# 5. Review results
+print("Selected features count:", len(pipeline.get_selected_columns()))
+print("Selected columns:", pipeline.get_selected_columns())
+
+# 6. Inspect audit report
+print(audit_report.head(20))
+```
+
+---
+
+## Inspection & Diagnostic Methods
+
+After calling `pipeline.run()`, several diagnostic helpers allow inspection of the feature reduction decisions:
+
+```python
+# 1. Retrieve list of final selected feature names
+selected_features = pipeline.get_selected_columns()
+
+# 2. Inspect Information Value ranking for all evaluated features
+iv_summary = pipeline.get_iv_summary()
+print(iv_summary.head(10))
+
+# 3. Inspect which features were pruned due to correlation and their collinear counterpart
+correlation_drops = pipeline.get_correlation_drops()
+print(correlation_drops.head(10))
+
+# 4. Full audit report with exact exclusion reasons
+print(pipeline.feature_report_)
+```
+
+### Sample Audit Report Output
+
+| feature | dtype | iv | correlated_with | correlation_value | is_selected | exclusion_reason |
+|---|---|---|---|---|---|---|
+| `bureau_score` | numeric | 0.4521 | None | None | True | |
+| `utilization_rate`| numeric | 0.3180 | None | None | True | |
+| `num_inquiries_6m` | numeric | 0.1420 | None | None | True | |
+| `num_inquiries_3m` | numeric | 0.1210 | `num_inquiries_6m`| 0.965 | False | correlated (r=0.965) |
+| `employer_name` | categorical | None | None | None | False | high_cardinality (>200) |
+| `postal_code_raw`| numeric | 0.0120 | None | None | False | low_iv (0.0120) |
+| `application_date`| datetime | None | None | None | False | datetime_column |
+
+---
+
+## API Reference
+
+### `FeatureSelectionPipeline`
+
+```python
+FeatureSelectionPipeline(
+    df: SparkDataFrame,
+    target_col: str,
+    mandatory_features: Optional[List[str]] = None,
+    exclude_features: Optional[List[str]] = None,
+    iv_threshold: float = 0.03,
+    iv_n_bins: int = 100,
+    iv_sample_rows: int = 1_000_000,
+    corr_threshold: float = 0.95,
+    corr_sample_rows: int = 500_000,
+    max_corr_features: int = 5000,
+    max_cat_levels: int = 200,
+    expected_final_min: int = 200,
+    expected_final_max: int = 1000,
+    downcast: bool = True,
+    verbose: bool = True,
+)
+```
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `df` | `SparkDataFrame` | *Required* | Input Spark DataFrame containing raw candidate variables and target. |
+| `target_col` | `str` | *Required* | Name of the binary target (0 = Good, 1 = Bad). Case-insensitive. |
+| `mandatory_features` | `List[str]` | `None` | Columns guaranteed to be retained regardless of IV or correlation. |
+| `exclude_features` | `List[str]` | `None` | Columns explicitly excluded from candidate pool (IDs, timestamps). |
+| `iv_threshold` | `float` | `0.03` | Minimum Information Value (IV) required for feature inclusion. |
+| `iv_n_bins` | `int` | `100` | Quantile bin resolution for continuous variable histogram computation. |
+| `iv_sample_rows` | `int` | `1,000,000` | Sample ceiling for computing quantile thresholds on continuous columns. |
+| `corr_threshold` | `float` | `0.95` | Pearson correlation ceiling. Collinear variable with lower IV is pruned. |
+| `corr_sample_rows` | `int` | `500,000` | Sample ceiling for Pearson correlation matrix calculation. |
+| `max_corr_features` | `int` | `5000` | Upper limit of top IV features fed into correlation assembler. |
+| `max_cat_levels` | `int` | `200` | Maximum unique levels allowed before categorical is dropped as high-cardinality. |
+| `expected_final_min` | `int` | `200` | Minimum retained count. Relaxes `corr_threshold` if pruned too aggressively. |
+| `expected_final_max` | `int` | `1000` | Maximum cap on final retained variables. |
+| `downcast` | `bool` | `True` | Downcasts Decimal/Double columns to Float32 to optimize Spark worker memory. |
+| `verbose` | `bool` | `True` | Prints stage progress and execution timing logs. |
+
+---
+
+## Credit Risk Scorecard Rules of Thumb
+
+### Information Value (IV) Benchmarks
+
+In credit risk modeling (Siddiqi, 2005), Information Value serves as the primary metric for filtering out uninformative signals:
+
+| Information Value (IV) | Predictive Power | Action in Scorecard Development |
+|---|---|---|
+| **< 0.02** | Unpredictive | **Drop**: Adds noise and degrees of freedom without signal. |
+| **0.02 – 0.10** | Weak Predictor | **Evaluate**: May be retained if part of a key credit policy dimension. |
+| **0.10 – 0.30** | Medium Predictor | **Keep**: Core candidate for scorecard inclusion. |
+| **0.30 – 0.50** | Strong Predictor | **Keep**: High diagnostic quality feature. |
+| **> 0.50** | Suspicious / Too Good | **Investigate**: Often indicative of target leakage or operational bias. |
+
+### Correlation Thresholds
+
+- Standard practice sets the correlation threshold between `0.80` and `0.95`.
+- Setting `corr_threshold=0.95` catches near-duplicate metrics (e.g., balance in dollars vs. balance in thousands).
+- Setting `corr_threshold=0.85` produces a tighter, more orthogonal set of features that prevents variance inflation in final logistic regression models.
+
+---
+
+## Performance & Spark Optimization
+
+- **Quantile Binning (`mapInPandas`)**: Rather than running expensive full-dataset sorting on each column, `feature_pruning` samples continuous variables to establish robust quantile boundaries, then computes distributed frequency histograms across partitions in a single pass.
+- **Spark Storage Management**: Intermediate working DataFrames are cached at `StorageLevel.MEMORY_AND_DISK` and unpersisted automatically at the conclusion of report generation.
+- **Vector Correlation**: Correlation is calculated using Spark ML's native distributed linear algebra (`Correlation.corr`), supporting thousands of features simultaneously.
+
+---
+
+## Contributing
+
+Contributions, bug reports, and feature requests are welcome!
+Please feel free to submit a pull request or open an issue on GitHub.
+
+1. Fork the Project
+2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
+3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
+4. Push to the Branch (`git push origin feature/AmazingFeature`)
+5. Open a Pull Request
+
+---
+
+## License
+
+Distributed under the MIT License. See [LICENSE.txt](LICENSE.txt) for more details.

feature_pruning-0.1.0/README.md

@@ -0,0 +1,269 @@
+# feature_pruning
+
+[![Python Version](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue.svg)](https://pypi.org/project/feature-pruning/)
+[![PySpark](https://img.shields.io/badge/PySpark-%3E%3D3.1.0-orange.svg)](https://spark.apache.org/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
+**High-Performance Distributed Feature Reduction Engine for Credit Risk Scorecards.**
+
+`feature_pruning` is an enterprise-grade Python library built natively on Apache Spark for selecting and pruning variables in credit risk scorecard development (Probability of Default / Basel II/III / IFRS 9 / Retail Scorecards).
+
+It automates the transition from thousands of raw credit bureau, transaction, and demographic attributes down to an optimal, highly predictive, non-collinear feature set ready for Weight of Evidence (WoE) binning and Logistic Regression.
+
+---
+
+## The Credit Risk Scorecard Challenge
+
+Building regulatory-compliant credit scorecards presents unique data engineering and modeling hurdles:
+
+- **Audit & Governance (SR 11-7 / Basel / IFRS 9)**: Model risk management (MRM) and regulatory auditors require an explicit justification for every discarded or retained variable.
+- **Extreme Multicollinearity**: Credit bureau tables often contain dozens of collinear metrics (e.g., `num_inquiries_3m`, `num_inquiries_6m`, `num_inquiries_12m`). In standard logistic regression scorecards, collinearity causes unstable coefficients and counter-intuitive sign reversals.
+- **Predictive Quality**: Features must meet minimum **Information Value (IV)** standards while preserving mandatory business or regulatory key indicators.
+- **Big Data Scale**: Modern credit datasets often span millions of accounts and thousands of features. Single-machine libraries (pandas/scikit-learn) crash with `OutOfMemory` errors when computing quantile cuts and pairwise correlations.
+
+`feature_pruning` solves these challenges by running distributed quantile binning, IV evaluation, and correlation pruning entirely within **Apache Spark**.
+
+---
+
+## Pipeline Architecture
+
+```
+                           Raw Spark DataFrame (Millions of Rows, 1000s of Features)
+                                                   │
+                                                   ▼
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Stage 1: Feature Classification                                                                        │
+│ • Detects Numeric vs. Categorical vs. Datetime columns                                                 │
+│ • Flags & filters high-cardinality strings (> max_cat_levels)                                          │
+│ • Protects mandatory features and excludes requested columns                                          │
+└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
+                                                   │
+                                                   ▼
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Stage 2: Distributed Precision Downcasting                                                             │
+│ • Downcasts DoubleType & DecimalType to 32-bit FloatType                                              │
+│ • Cuts executor memory consumption by ~50% during matrix aggregation                                  │
+└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
+                                                   │
+                                                   ▼
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Stage 3: High-Throughput Information Value (IV) Computation                                            │
+│ • Computes continuous quantiles and distributed binning via Spark mapInPandas                         │
+│ • Aggregates Goods/Bads and calculates IV for numeric and categorical variables                        │
+│ • Filters out variables with IV < iv_threshold                                                         │
+└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
+                                                   │
+                                                   ▼
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Stage 4: IV-Prioritized Correlation Pruning                                                            │
+│ • Computes Pearson correlation matrix via PySpark VectorAssembler + Correlation                        │
+│ • Between collinear pairs (r >= corr_threshold), retains the feature with higher IV                    │
+│ • Dynamic threshold relaxation ensures minimum required feature count is met                          │
+└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
+                                                   │
+                                                   ▼
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Stage 5: Full Audit Reporting & Data Delivery                                                          │
+│ • Generates comprehensive audit DataFrame (is_selected, iv, correlated_with, exclusion_reason)         │
+│ • Outputs pruned Spark DataFrame ready for Weight of Evidence (WoE) binning                           │
+└────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Key Features
+
+- ⚡ **Native PySpark Scalability**: Distributed quantile histograms and correlation matrices executed directly on cluster workers via Spark ML and `mapInPandas`.
+- 📊 **IV-Driven Pruning**: When two features are collinear, the pipeline drops the weaker predictor and retains the variable with higher Information Value.
+- 🛡️ **Mandatory Variable Protection**: Ensure business-critical variables (e.g., debt-to-income, credit bureau score) are never removed, regardless of their statistical properties.
+- 📋 **Regulatory Audit Trail**: Automatically produces a full governance table explaining why every single column was accepted or eliminated (e.g., `low_iv (0.012)`, `correlated (r=0.962)`, `high_cardinality (>200)`, or `datetime_column`).
+- 🔄 **Dynamic Threshold Relaxation**: Automatically adjusts correlation thresholds if filtering becomes overly aggressive, keeping feature counts within target bounds (`expected_final_min`, `expected_final_max`).
+
+---
+
+## Installation
+
+### From PyPI
+```bash
+pip install feature-pruning
+```
+
+### For Local Development
+```bash
+git clone https://github.com/vrukshya/feature_pruning.git
+cd feature_pruning
+pip install -e ".[dev]"
+```
+
+### In Databricks / Cloud Notebooks
+In your Databricks notebook cell:
+```python
+%pip install feature-pruning
+```
+Or add `feature-pruning` to your Databricks cluster libraries.
+
+---
+
+## Quickstart
+
+```python
+from pyspark.sql import SparkSession
+from feature_pruning import FeatureSelectionPipeline
+
+# 1. Initialize Spark session (or use active session in Databricks/EMR)
+spark = SparkSession.builder.appName("CreditRiskScorecard").getOrCreate()
+
+# 2. Load credit training data
+df = spark.table("risk_catalog.credit_data.application_train")
+
+# 3. Configure the feature selection pipeline
+pipeline = FeatureSelectionPipeline(
+    df=df,
+    target_col="default_flag",              # 0 = Good loan, 1 = Default / Bad loan
+    mandatory_features=["bureau_score", "dti_ratio"],  # Keep regardless of correlation
+    exclude_features=["application_id", "ssn_hash"],   # Exclude identifiers
+    iv_threshold=0.03,                      # Industry baseline: IV >= 0.03
+    corr_threshold=0.95,                    # Multicollinearity cutoff
+    expected_final_min=20,
+    expected_final_max=150,
+    verbose=True,
+)
+
+# 4. Execute pipeline
+selected_spark_df, audit_report = pipeline.run()
+
+# 5. Review results
+print("Selected features count:", len(pipeline.get_selected_columns()))
+print("Selected columns:", pipeline.get_selected_columns())
+
+# 6. Inspect audit report
+print(audit_report.head(20))
+```
+
+---
+
+## Inspection & Diagnostic Methods
+
+After calling `pipeline.run()`, several diagnostic helpers allow inspection of the feature reduction decisions:
+
+```python
+# 1. Retrieve list of final selected feature names
+selected_features = pipeline.get_selected_columns()
+
+# 2. Inspect Information Value ranking for all evaluated features
+iv_summary = pipeline.get_iv_summary()
+print(iv_summary.head(10))
+
+# 3. Inspect which features were pruned due to correlation and their collinear counterpart
+correlation_drops = pipeline.get_correlation_drops()
+print(correlation_drops.head(10))
+
+# 4. Full audit report with exact exclusion reasons
+print(pipeline.feature_report_)
+```
+
+### Sample Audit Report Output
+
+| feature | dtype | iv | correlated_with | correlation_value | is_selected | exclusion_reason |
+|---|---|---|---|---|---|---|
+| `bureau_score` | numeric | 0.4521 | None | None | True | |
+| `utilization_rate`| numeric | 0.3180 | None | None | True | |
+| `num_inquiries_6m` | numeric | 0.1420 | None | None | True | |
+| `num_inquiries_3m` | numeric | 0.1210 | `num_inquiries_6m`| 0.965 | False | correlated (r=0.965) |
+| `employer_name` | categorical | None | None | None | False | high_cardinality (>200) |
+| `postal_code_raw`| numeric | 0.0120 | None | None | False | low_iv (0.0120) |
+| `application_date`| datetime | None | None | None | False | datetime_column |
+
+---
+
+## API Reference
+
+### `FeatureSelectionPipeline`
+
+```python
+FeatureSelectionPipeline(
+    df: SparkDataFrame,
+    target_col: str,
+    mandatory_features: Optional[List[str]] = None,
+    exclude_features: Optional[List[str]] = None,
+    iv_threshold: float = 0.03,
+    iv_n_bins: int = 100,
+    iv_sample_rows: int = 1_000_000,
+    corr_threshold: float = 0.95,
+    corr_sample_rows: int = 500_000,
+    max_corr_features: int = 5000,
+    max_cat_levels: int = 200,
+    expected_final_min: int = 200,
+    expected_final_max: int = 1000,
+    downcast: bool = True,
+    verbose: bool = True,
+)
+```
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `df` | `SparkDataFrame` | *Required* | Input Spark DataFrame containing raw candidate variables and target. |
+| `target_col` | `str` | *Required* | Name of the binary target (0 = Good, 1 = Bad). Case-insensitive. |
+| `mandatory_features` | `List[str]` | `None` | Columns guaranteed to be retained regardless of IV or correlation. |
+| `exclude_features` | `List[str]` | `None` | Columns explicitly excluded from candidate pool (IDs, timestamps). |
+| `iv_threshold` | `float` | `0.03` | Minimum Information Value (IV) required for feature inclusion. |
+| `iv_n_bins` | `int` | `100` | Quantile bin resolution for continuous variable histogram computation. |
+| `iv_sample_rows` | `int` | `1,000,000` | Sample ceiling for computing quantile thresholds on continuous columns. |
+| `corr_threshold` | `float` | `0.95` | Pearson correlation ceiling. Collinear variable with lower IV is pruned. |
+| `corr_sample_rows` | `int` | `500,000` | Sample ceiling for Pearson correlation matrix calculation. |
+| `max_corr_features` | `int` | `5000` | Upper limit of top IV features fed into correlation assembler. |
+| `max_cat_levels` | `int` | `200` | Maximum unique levels allowed before categorical is dropped as high-cardinality. |
+| `expected_final_min` | `int` | `200` | Minimum retained count. Relaxes `corr_threshold` if pruned too aggressively. |
+| `expected_final_max` | `int` | `1000` | Maximum cap on final retained variables. |
+| `downcast` | `bool` | `True` | Downcasts Decimal/Double columns to Float32 to optimize Spark worker memory. |
+| `verbose` | `bool` | `True` | Prints stage progress and execution timing logs. |
+
+---
+
+## Credit Risk Scorecard Rules of Thumb
+
+### Information Value (IV) Benchmarks
+
+In credit risk modeling (Siddiqi, 2005), Information Value serves as the primary metric for filtering out uninformative signals:
+
+| Information Value (IV) | Predictive Power | Action in Scorecard Development |
+|---|---|---|
+| **< 0.02** | Unpredictive | **Drop**: Adds noise and degrees of freedom without signal. |
+| **0.02 – 0.10** | Weak Predictor | **Evaluate**: May be retained if part of a key credit policy dimension. |
+| **0.10 – 0.30** | Medium Predictor | **Keep**: Core candidate for scorecard inclusion. |
+| **0.30 – 0.50** | Strong Predictor | **Keep**: High diagnostic quality feature. |
+| **> 0.50** | Suspicious / Too Good | **Investigate**: Often indicative of target leakage or operational bias. |
+
+### Correlation Thresholds
+
+- Standard practice sets the correlation threshold between `0.80` and `0.95`.
+- Setting `corr_threshold=0.95` catches near-duplicate metrics (e.g., balance in dollars vs. balance in thousands).
+- Setting `corr_threshold=0.85` produces a tighter, more orthogonal set of features that prevents variance inflation in final logistic regression models.
+
+---
+
+## Performance & Spark Optimization
+
+- **Quantile Binning (`mapInPandas`)**: Rather than running expensive full-dataset sorting on each column, `feature_pruning` samples continuous variables to establish robust quantile boundaries, then computes distributed frequency histograms across partitions in a single pass.
+- **Spark Storage Management**: Intermediate working DataFrames are cached at `StorageLevel.MEMORY_AND_DISK` and unpersisted automatically at the conclusion of report generation.
+- **Vector Correlation**: Correlation is calculated using Spark ML's native distributed linear algebra (`Correlation.corr`), supporting thousands of features simultaneously.
+
+---
+
+## Contributing
+
+Contributions, bug reports, and feature requests are welcome!
+Please feel free to submit a pull request or open an issue on GitHub.
+
+1. Fork the Project
+2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
+3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
+4. Push to the Branch (`git push origin feature/AmazingFeature`)
+5. Open a Pull Request
+
+---
+
+## License
+
+Distributed under the MIT License. See [LICENSE.txt](LICENSE.txt) for more details.

feature_pruning-0.1.0/feature_pruning/__init__.py

@@ -0,0 +1,39 @@
+"""
+Feature Pruning: High-Performance Feature Reduction for Credit Risk Scorecards
+=============================================================================
+
+`feature_pruning` is an enterprise-grade, distributed feature reduction library
+specifically optimized for developing regulatory-compliant credit risk scorecards
+(such as Basel II/III, IFRS 9, and retail scorecard models).
+
+Key Capabilities:
+-----------------
+- Automated variable classification (numeric, categorical, datetime, high-cardinality).
+- Memory-efficient precision downcasting for massive distributed datasets.
+- Fast distributed Information Value (IV) calculation via quantile binning.
+- IV-prioritized Pearson correlation matrix pruning to eliminate multicollinearity.
+- Transparent audit trail tracking all feature inclusion and exclusion decisions.
+
+Quick Example:
+--------------
+>>> from feature_pruning import FeatureSelectionPipeline
+>>> pipeline = FeatureSelectionPipeline(
+...     df=spark_df,
+...     target_col="default_flag",
+...     mandatory_features=["bureau_score", "annual_income"],
+...     exclude_features=["applicant_id"],
+...     iv_threshold=0.03,
+...     corr_threshold=0.95,
+... )
+>>> pruned_spark_df, audit_report = pipeline.run()
+"""
+
+from .core import FeatureSelectionPipeline
+
+__version__ = "0.1.0"
+__author__ = "Vrukshya"
+__all__ = [
+    "FeatureSelectionPipeline",
+    "__version__",
+    "__author__",
+]