kmds-featurization 0.1.0__tar.gz

kmds_featurization-0.1.0/PKG-INFO

@@ -0,0 +1,178 @@
+Metadata-Version: 2.4
+Name: kmds-featurization
+Version: 0.1.0
+Summary: KMDS Featurization Service and Orchestration Pipeline
+License-Expression: LicenseRef-Proprietary
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: pyyaml
+Requires-Dist: scikit-learn
+Requires-Dist: pgeocode
+Requires-Dist: category-encoders
+Requires-Dist: kneed
+Requires-Dist: matplotlib
+Requires-Dist: tabulate>=0.10.0
+
+# KMDS Featurization
+
+This repository provides a configurable, stage-based featurization engine for SBA modeling workflows.
+
+The design goal is simple:
+- keep stage logic understandable and composable
+- keep orchestration/configuration centralized
+- keep modeling flow leakage-safe (fit on train only, reuse on val/active)
+
+## What This Produces
+
+The pipeline writes two CSV outputs:
+- featurized_data.csv: consolidated engineered dataset (modeled + active partitions)
+- model_ready_numeric_data.csv: numeric model-ready export from the final stage output
+
+Additional diagnostic artifact:
+- feature_selection_knee_curve.png: ranked feature-importance knee plot saved in the featurization output directory
+
+For the current SBA flow, the model-ready dataset is:
+- numeric/bool only
+- train-fitted feature-selected
+- schema-aligned across train/val/active
+- persisted with index=False (no index artifact column)
+
+## Core Concepts
+
+- Anchor index: record_id
+- Stage contract: method(context, stage_cfg) -> DataFrame
+- Waterfall behavior: each stage can shrink survivor rows by index
+- Horizontal feature assembly: stage outputs are concatenated by index
+- Controlled index expansion: only stages marked allow_new_indices may re-introduce rows
+
+## Pipeline Layout (Current Hybrid Design)
+
+Front section (feature assembly):
+1. record_id_definition
+2. borrower_geo_coding
+3. prepare_categorical_data
+4. prepare_numerical_data
+5. merge_categorical_and_numerical
+6. merge_with_borrower_geo
+
+Merge stage design:
+- package component: src/tabular/merge_ops.py
+- user wrappers: featurization_scripts/featurization.py
+- merge key: record_id index
+
+Leakage-safe modeling section:
+7. low_count_featurization_of_cat_vars
+8. hierarchical_low_count_var_encoding
+9. loan_status_recoding
+10. filter_modeling_universe
+11. stratified_train_val_split
+12. target_encode_categorical_vars
+13. harmonize_and_project_feature_space
+14. merge_modeled_and_active_partitions
+
+Current encoding rule:
+- if both raw and rarity-corrected categorical variants exist (x and x_rcs), only x_rcs is target-encoded
+
+## Tree-Based Feature Selection
+
+Feature selection runs in harmonize_and_project_feature_space using train rows only.
+
+Supported selector modes:
+- threshold
+- tree_ensemble
+
+Supported tree models:
+- gbm
+- random_forest
+- xgboost (optional dependency)
+
+All selector choices are config-driven via featurizer_config.yaml and surfaced through PathCoordinator (no stage-level hardcoded constants).
+
+Feature-count tuning for kneedle mode:
+- FEATURE_SELECTION_TOP_K_MODE: kneedle
+- FEATURE_SELECTION_TOP_K_MIN_RATIO: conservative default floor, e.g. 0.5
+- FEATURE_SELECTION_MIN_FEATURE_COUNT: hard floor for retained features
+- FEATURE_SELECTION_TARGET_FEATURE_COUNT: explicit count override when the curve is too aggressive
+- FEATURE_SELECTION_REQUIRE_KNEEDLE: fail loudly if the knee cannot be determined
+
+## Repository Organization
+
+- src/featurization/core: orchestration, configuration bootstrap, path resolution
+- src/featurization/transforms: reusable transformation primitives
+- src/tabular: reusable tabular feature modules (encoding, splitting, feature space)
+- src/tabular/merge_ops.py: reusable index-aligned tabular merge helper
+- tests: package-level smoke and behavior checks
+- documents: architecture and configuration references
+
+## Package Component Buckets
+
+The tabular package modules are intentionally split into two modeling buckets:
+
+- Row-selection components:
+	- src/tabular/modeling_filter.py
+	- src/tabular/train_val_split.py
+	- Purpose: decide which records participate in training and how records are partitioned.
+
+- Column-selection components:
+	- src/tabular/feature_space.py
+	- src/tabular/target_encoding.py
+	- src/tabular/low_count_cat_var_encoding.py
+	- src/tabular/hierarchical_low_count_var_encoding.py
+	- Purpose: decide which feature columns are engineered, selected, encoded, and projected.
+
+- Assembly components:
+	- src/tabular/merge_ops.py
+	- Purpose: index-aligned horizontal composition of prepared payloads.
+
+## CLI
+
+Initialize config:
+
+```bash
+featurization-cli init \
+  --working-dir /path/to/workspace \
+  --metadata-file sba_loans_metadata_table.csv \
+  --data-file sba_loans_user_cleaned.csv
+```
+
+Run pipeline:
+
+```bash
+featurization-cli run --working-dir /path/to/workspace
+```
+
+Run smoke test in this repo:
+
+```bash
+pytest -q tests/test_sba_pipeline.py
+```
+
+## How To Extend Safely
+
+1. Add reusable logic in src/tabular first whenever possible.
+2. Keep stage wrappers in workspace featurization_scripts/featurization.py thin and explicit.
+3. Add new tunables to:
+	- featurizer_config.yaml
+	- src/featurization/core/path_coordinator.py
+	- src/featurization/core/featurization_init.py
+4. Preserve leakage rules:
+	- fit artifacts on train only
+	- transform val/active using train-fitted artifacts
+5. Validate with tests after each change.
+
+## Recommended Read Order
+
+1. documents/sba_pipeline_featurization.md
+2. documents/config_blueprint.md
+3. documents/path_coordinator_function.md
+4. src/featurization/core/sequential_pipeline_runner.py
+5. src/tabular/feature_space.py

kmds_featurization-0.1.0/README.md

@@ -0,0 +1,154 @@
+# KMDS Featurization
+
+This repository provides a configurable, stage-based featurization engine for SBA modeling workflows.
+
+The design goal is simple:
+- keep stage logic understandable and composable
+- keep orchestration/configuration centralized
+- keep modeling flow leakage-safe (fit on train only, reuse on val/active)
+
+## What This Produces
+
+The pipeline writes two CSV outputs:
+- featurized_data.csv: consolidated engineered dataset (modeled + active partitions)
+- model_ready_numeric_data.csv: numeric model-ready export from the final stage output
+
+Additional diagnostic artifact:
+- feature_selection_knee_curve.png: ranked feature-importance knee plot saved in the featurization output directory
+
+For the current SBA flow, the model-ready dataset is:
+- numeric/bool only
+- train-fitted feature-selected
+- schema-aligned across train/val/active
+- persisted with index=False (no index artifact column)
+
+## Core Concepts
+
+- Anchor index: record_id
+- Stage contract: method(context, stage_cfg) -> DataFrame
+- Waterfall behavior: each stage can shrink survivor rows by index
+- Horizontal feature assembly: stage outputs are concatenated by index
+- Controlled index expansion: only stages marked allow_new_indices may re-introduce rows
+
+## Pipeline Layout (Current Hybrid Design)
+
+Front section (feature assembly):
+1. record_id_definition
+2. borrower_geo_coding
+3. prepare_categorical_data
+4. prepare_numerical_data
+5. merge_categorical_and_numerical
+6. merge_with_borrower_geo
+
+Merge stage design:
+- package component: src/tabular/merge_ops.py
+- user wrappers: featurization_scripts/featurization.py
+- merge key: record_id index
+
+Leakage-safe modeling section:
+7. low_count_featurization_of_cat_vars
+8. hierarchical_low_count_var_encoding
+9. loan_status_recoding
+10. filter_modeling_universe
+11. stratified_train_val_split
+12. target_encode_categorical_vars
+13. harmonize_and_project_feature_space
+14. merge_modeled_and_active_partitions
+
+Current encoding rule:
+- if both raw and rarity-corrected categorical variants exist (x and x_rcs), only x_rcs is target-encoded
+
+## Tree-Based Feature Selection
+
+Feature selection runs in harmonize_and_project_feature_space using train rows only.
+
+Supported selector modes:
+- threshold
+- tree_ensemble
+
+Supported tree models:
+- gbm
+- random_forest
+- xgboost (optional dependency)
+
+All selector choices are config-driven via featurizer_config.yaml and surfaced through PathCoordinator (no stage-level hardcoded constants).
+
+Feature-count tuning for kneedle mode:
+- FEATURE_SELECTION_TOP_K_MODE: kneedle
+- FEATURE_SELECTION_TOP_K_MIN_RATIO: conservative default floor, e.g. 0.5
+- FEATURE_SELECTION_MIN_FEATURE_COUNT: hard floor for retained features
+- FEATURE_SELECTION_TARGET_FEATURE_COUNT: explicit count override when the curve is too aggressive
+- FEATURE_SELECTION_REQUIRE_KNEEDLE: fail loudly if the knee cannot be determined
+
+## Repository Organization
+
+- src/featurization/core: orchestration, configuration bootstrap, path resolution
+- src/featurization/transforms: reusable transformation primitives
+- src/tabular: reusable tabular feature modules (encoding, splitting, feature space)
+- src/tabular/merge_ops.py: reusable index-aligned tabular merge helper
+- tests: package-level smoke and behavior checks
+- documents: architecture and configuration references
+
+## Package Component Buckets
+
+The tabular package modules are intentionally split into two modeling buckets:
+
+- Row-selection components:
+	- src/tabular/modeling_filter.py
+	- src/tabular/train_val_split.py
+	- Purpose: decide which records participate in training and how records are partitioned.
+
+- Column-selection components:
+	- src/tabular/feature_space.py
+	- src/tabular/target_encoding.py
+	- src/tabular/low_count_cat_var_encoding.py
+	- src/tabular/hierarchical_low_count_var_encoding.py
+	- Purpose: decide which feature columns are engineered, selected, encoded, and projected.
+
+- Assembly components:
+	- src/tabular/merge_ops.py
+	- Purpose: index-aligned horizontal composition of prepared payloads.
+
+## CLI
+
+Initialize config:
+
+```bash
+featurization-cli init \
+  --working-dir /path/to/workspace \
+  --metadata-file sba_loans_metadata_table.csv \
+  --data-file sba_loans_user_cleaned.csv
+```
+
+Run pipeline:
+
+```bash
+featurization-cli run --working-dir /path/to/workspace
+```
+
+Run smoke test in this repo:
+
+```bash
+pytest -q tests/test_sba_pipeline.py
+```
+
+## How To Extend Safely
+
+1. Add reusable logic in src/tabular first whenever possible.
+2. Keep stage wrappers in workspace featurization_scripts/featurization.py thin and explicit.
+3. Add new tunables to:
+	- featurizer_config.yaml
+	- src/featurization/core/path_coordinator.py
+	- src/featurization/core/featurization_init.py
+4. Preserve leakage rules:
+	- fit artifacts on train only
+	- transform val/active using train-fitted artifacts
+5. Validate with tests after each change.
+
+## Recommended Read Order
+
+1. documents/sba_pipeline_featurization.md
+2. documents/config_blueprint.md
+3. documents/path_coordinator_function.md
+4. src/featurization/core/sequential_pipeline_runner.py
+5. src/tabular/feature_space.py

kmds_featurization-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "kmds-featurization"
+version = "0.1.0"
+description = "KMDS Featurization Service and Orchestration Pipeline"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "LicenseRef-Proprietary"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Operating System :: OS Independent",
+]
+dependencies = [
+    "pandas",
+    "numpy",
+    "pyyaml",
+    "scikit-learn",
+    "pgeocode",
+    "category-encoders",
+    "kneed",
+    "matplotlib",
+    "tabulate>=0.10.0",
+]
+
+[project.scripts]
+featurization-cli = "featurization.cli:main"
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]

kmds_featurization-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

kmds_featurization-0.1.0/src/featurization/__init__.py

@@ -0,0 +1 @@
+# KMDS Tabular Utilities Package

kmds_featurization-0.1.0/src/featurization/cli.py

@@ -0,0 +1,131 @@
+import argparse
+import sys
+import os
+import yaml
+from featurization.core.sequential_pipeline_runner import PipelineRunner
+from featurization.core.featurization_init import initialize_config
+from featurization.core.path_coordinator import PathCoordinator
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Featurization Service Shell - kmds-data-helper Ecosystem"
+    )
+    subparsers = parser.add_subparsers(dest="command", help="Available commands")
+
+    # --- Init Command ---
+    init_parser = subparsers.add_parser("init", help="Initialize a featurization workspace configuration.")
+    init_parser.add_argument(
+        "--working-dir", 
+        required=True, 
+        help="Path to the workflow directory."
+    )
+    init_parser.add_argument(
+        "--metadata-file", 
+        required=True, 
+        help="The dd-parser-cleaner summary CSV filename."
+    )
+    init_parser.add_argument(
+        "--data-file", 
+        required=True, 
+        help="The cleaned data CSV filename."
+    )
+    init_parser.add_argument(
+        "--structural-type",
+        choices=["cross-sectional", "longitudinal", "panel"],
+        default="cross-sectional",
+        help="The structural type of the dataset (cross-sectional, longitudinal, or panel)."
+    )
+
+    # --- Add-Stage Command ---
+    add_parser = subparsers.add_parser("add-stage", help="Add a custom stage to the pipeline.")
+    add_parser.add_argument("--name", required=True, help="Name of the stage.")
+    add_parser.add_argument("--entity", required=True, help="Entity category in metadata (e.g., geographical).")
+    add_parser.add_argument("--sub-filter", required=True, help="Specific sub-filter group (e.g., Borrower).")
+    add_parser.add_argument("--config", default="featurizer_config.yaml", help="Config filename.")
+
+    # --- Check Command ---
+    check_parser = subparsers.add_parser("check", help="Check if a stage is configured.")
+    check_parser.add_argument("--name", required=True, help="Stage name to verify.")
+    check_parser.add_argument("--config", default="featurizer_config.yaml", help="Config filename.")
+
+    # --- Run Command ---
+    run_parser = subparsers.add_parser("run", help="Execute the featurization pipeline.")
+    run_parser.add_argument(
+        "--working-dir", 
+        type=str, 
+        required=True, 
+        help="Path to the active project working directory containing configurations and datasets."
+    )
+    run_parser.add_argument(
+        "--target-col", 
+        type=str, 
+        default="target", 
+        help="Name of the machine learning target prediction column."
+    )
+    run_parser.add_argument(
+        "--config", 
+        type=str, 
+        default="featurizer_config.yaml", 
+        help="Path to the environment config layout file."
+    )
+
+    args = parser.parse_args()
+    
+    if not args.command:
+        parser.print_help()
+        sys.exit(0)
+
+    try:
+        if args.command == "init":
+            if not os.path.isdir(args.working_dir):
+                print(f"❌ Error: Provided working directory does not exist: {args.working_dir}")
+                sys.exit(1)
+            initialize_config(
+                args.working_dir, 
+                args.metadata_file, 
+                args.data_file, 
+                args.structural_type
+            )
+            print("✅ Initialization Complete.")
+            return
+
+        if args.command == "check":
+            if not os.path.exists(args.config):
+                print(f"❌ Config file '{args.config}' not found.")
+                sys.exit(1)
+            with open(args.config, "r") as f:
+                config = yaml.safe_load(f)
+            working_dir = config.get("working_dir", os.getcwd())
+            resolver = PathCoordinator(working_dir, config)
+            
+            if resolver.is_stage_configured(args.name):
+                print(f"✅ Stage '{args.name}' is configured in the pipeline.")
+            else:
+                print(f"❌ Stage '{args.name}' is NOT configured.")
+                sys.exit(1)
+            return
+
+        if args.command == "run":
+            # Load configuration
+            config_path = os.path.join(args.working_dir, args.config)
+            if not os.path.exists(config_path):
+                config_path = args.config
+                
+            if not os.path.exists(config_path):
+                print(f"❌ Error: Config file not found at {config_path}")
+                sys.exit(1)
+
+            with open(config_path, "r") as f:
+                config = yaml.safe_load(f)
+
+            # Initialize the runner and execute accumulation
+            runner = PipelineRunner(working_dir=args.working_dir, config=config)
+            runner.accumulate_stages()
+            # runner.accumulate_stages() now handles persistence internally via PathCoordinator
+            return
+    except Exception as e:
+        print(f"💥 Pipeline Execution Failed: {str(e)}")
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()

kmds_featurization-0.1.0/src/featurization/core/__init__.py

@@ -0,0 +1 @@
+# KMDS Core Infrastructure Package

kmds_featurization-0.1.0/src/featurization/core/data_loader.py

@@ -0,0 +1,37 @@
+import os
+import pandas as pd
+from featurization.utils import load_kmds_metadata
+
+class KMDSDataLoader:
+    """
+    Package Component: Centralized data loading module.
+    Handles lazy-loading of KMDS datasets and metadata to avoid redundant I/O.
+    """
+    def __init__(self, resolver):
+        self.resolver = resolver
+        self._data = None
+        self._metadata = None
+
+    @property
+    def data(self) -> pd.DataFrame:
+        """Lazy-loads the primary cleaned dataset."""
+        if self._data is None:
+            path = self.resolver.featurization_input_path
+            if not os.path.exists(path):
+                raise FileNotFoundError(f"Source data not found at: {path}")
+            print(f"📥 [Data Loader] Reading cleaned data: {os.path.basename(path)}")
+            self._data = pd.read_csv(path, low_memory=False)
+        return self._data
+
+    @property
+    def metadata(self) -> pd.DataFrame:
+        """Lazy-loads the KMDS metadata (data dictionary)."""
+        if self._metadata is None:
+            path = self.resolver.metadata_path
+            if not os.path.exists(path):
+                print(f"⚠️ [Data Loader] Metadata missing at {path}. Returning empty frame.")
+                self._metadata = pd.DataFrame()
+            else:
+                print(f"📥 [Data Loader] Reading metadata: {os.path.basename(path)}")
+                self._metadata = load_kmds_metadata(path)
+        return self._metadata

kmds_featurization-0.1.0/src/featurization/core/featurization_init.py

@@ -0,0 +1,110 @@
+import os
+import yaml
+
+def initialize_config(working_dir: str, metadata_file: str, data_file: str, structural_type: str = "cross-sectional", config_name: str = "featurizer_config.yaml"):
+    """
+    Bootstraps a new featurization workspace by persisting key project anchors 
+    from dd-parser-cleaner into the featurizer_config.yaml.
+    """
+    abs_working_dir = os.path.abspath(working_dir)
+    config_path = os.path.join(abs_working_dir, config_name)
+
+    # 1. Load existing config if available (check target path or current directory)
+    existing_config = {}
+    search_path = config_path if os.path.exists(config_path) else config_name
+    if os.path.exists(search_path):
+        with open(search_path, "r") as f:
+            existing_config = yaml.safe_load(f) or {}
+
+    # 2. Define final config, preferring existing values for non-anchor settings
+    final_config = {
+        "working_dir": abs_working_dir,
+        "pipeline": existing_config.get("pipeline", []), # New pipeline structure
+        "metadata_file": metadata_file,
+        "dd_cleaner_output_dir": existing_config.get("dd_cleaner_output_dir", "dd_cleaner"),
+        "featurization_input_data": existing_config.get("featurization_input_data", data_file),
+        "featurization_output_dir": existing_config.get("featurization_output_dir", "featurization"),
+        "quarantine_dir": existing_config.get("quarantine_dir", "featurization/quarantine"),
+        "featurized_data_file": existing_config.get("featurized_data_file", "featurized_data.csv"),
+        "model_ready_data_file": existing_config.get("model_ready_data_file", "model_ready_numeric_data.csv"),
+        "feat_doc_directory": existing_config.get("feat_doc_directory", "featurization_docs"),
+        "entity_assignment_output": existing_config.get("entity_assignment_output", "entity_assignments.md"),
+        "script_dir": existing_config.get("script_dir", "featurization_scripts"), # New script directory key
+        "script_name": existing_config.get("script_name", "featurization.py"),   # New script file name key
+        "country_code": existing_config.get("country_code", "us"),
+        "structural_type": structural_type,
+        "VALIDATION_SIZE": existing_config.get("VALIDATION_SIZE", 0.2),
+        "FEATURE_SELECTION_MIN_NON_NULL_RATE": existing_config.get(
+            "FEATURE_SELECTION_MIN_NON_NULL_RATE", 0.01
+        ),
+        "FEATURE_SELECTION_METHOD": existing_config.get("FEATURE_SELECTION_METHOD", "tree_ensemble"),
+        "FEATURE_SELECTION_TOP_K": existing_config.get("FEATURE_SELECTION_TOP_K", 50),
+        "FEATURE_SELECTION_TOP_K_MODE": existing_config.get("FEATURE_SELECTION_TOP_K_MODE", "fixed"),
+        "FEATURE_SELECTION_TOP_K_MIN": existing_config.get("FEATURE_SELECTION_TOP_K_MIN", 1),
+        "FEATURE_SELECTION_TOP_K_MIN_RATIO": existing_config.get(
+            "FEATURE_SELECTION_TOP_K_MIN_RATIO", 0.0
+        ),
+        "FEATURE_SELECTION_MIN_FEATURE_COUNT": existing_config.get(
+            "FEATURE_SELECTION_MIN_FEATURE_COUNT", 0
+        ),
+        "FEATURE_SELECTION_TOP_K_MAX": existing_config.get("FEATURE_SELECTION_TOP_K_MAX", 0),
+        "FEATURE_SELECTION_TARGET_FEATURE_COUNT": existing_config.get(
+            "FEATURE_SELECTION_TARGET_FEATURE_COUNT", 0
+        ),
+        "FEATURE_SELECTION_KNEEDLE_SENSITIVITY": existing_config.get(
+            "FEATURE_SELECTION_KNEEDLE_SENSITIVITY", 1.0
+        ),
+        "FEATURE_SELECTION_KNEEDLE_CURVE": existing_config.get(
+            "FEATURE_SELECTION_KNEEDLE_CURVE", "convex"
+        ),
+        "FEATURE_SELECTION_KNEEDLE_DIRECTION": existing_config.get(
+            "FEATURE_SELECTION_KNEEDLE_DIRECTION", "decreasing"
+        ),
+        "FEATURE_SELECTION_REQUIRE_KNEEDLE": existing_config.get(
+            "FEATURE_SELECTION_REQUIRE_KNEEDLE", False
+        ),
+        "FEATURE_SELECTION_IMPORTANCE_FLOOR": existing_config.get(
+            "FEATURE_SELECTION_IMPORTANCE_FLOOR", 0.0
+        ),
+        "FEATURE_SELECTION_TREE_MODEL": existing_config.get("FEATURE_SELECTION_TREE_MODEL", "gbm"),
+        "FEATURE_SELECTION_TREE_N_ESTIMATORS": existing_config.get(
+            "FEATURE_SELECTION_TREE_N_ESTIMATORS", 200
+        ),
+        "FEATURE_SELECTION_TREE_LEARNING_RATE": existing_config.get(
+            "FEATURE_SELECTION_TREE_LEARNING_RATE", 0.05
+        ),
+        "FEATURE_SELECTION_TREE_MAX_DEPTH": existing_config.get("FEATURE_SELECTION_TREE_MAX_DEPTH", 3),
+        "FEATURE_SELECTION_TREE_SUBSAMPLE": existing_config.get(
+            "FEATURE_SELECTION_TREE_SUBSAMPLE", 0.8
+        ),
+        "FEATURE_SELECTION_TREE_RANDOM_STATE": existing_config.get(
+            "FEATURE_SELECTION_TREE_RANDOM_STATE", 42
+        ),
+        "MODEL_READY_NUMERIC_ONLY": existing_config.get("MODEL_READY_NUMERIC_ONLY", True)
+    }
+
+    # 3. Preserve any custom stage definitions or extra settings from the existing config
+    for key, value in existing_config.items():
+        if key not in final_config:
+            final_config[key] = value
+    
+    # 4. Ensure standard directory structure exists relative to finalized working_dir
+    required_dirs = [
+        "data",
+        "documents",
+        "notebooks",
+        "featurization_scripts",
+        os.path.join("data", final_config["dd_cleaner_output_dir"]),
+        os.path.join("data", final_config["featurization_output_dir"]),
+        os.path.join("data", final_config["quarantine_dir"]),
+        os.path.join("documents", final_config["feat_doc_directory"])
+    ]
+    for folder in required_dirs:
+        os.makedirs(os.path.join(abs_working_dir, folder), exist_ok=True)
+    
+    with open(config_path, "w") as f:
+        yaml.safe_dump(final_config, f, default_flow_style=False, sort_keys=False)
+    
+    print(f"✨ Workspace Initialized: {config_path}")
+    print(f"   - Metadata Anchor: {metadata_file}")
+    print(f"   - Cleaned Data Anchor: {data_file}")