upgini 1.1.127__tar.gz → 1.1.129__tar.gz

{upgini-1.1.127/src/upgini.egg-info → upgini-1.1.129}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: upgini
-Version: 1.1.127
+Version: 1.1.129
 Summary: Intelligent data search & enrichment for Machine Learning
 Home-page: https://upgini.com/
 Author: Upgini Developers
@@ -66,7 +66,7 @@ License-File: LICENSE
 ## 🚀 Awesome features
 ⭐️ Automatically find only relevant features that *give accuracy improvement for ML model*. Not just correlated with target variable, what 9 out of 10 cases gives zero accuracy improvement  
 ⭐️ Data source optimizations for ML tasks to *"squeeze" maximum information for models* out of the source data: automated feature generation with Large Language Models' data augmentation, RNNs, GraphNN; multiple data source ensembling  
-⭐️ Iterative search with *automatic search key augmentation* from all connected sources. If you are missing any search keys in your search request, Upgini will try to find those keys to broaden the search across all available data sources.   
+⭐️ *Automatic search key augmentation* from all connected sources. If you do not have all search keys in your search request, such as postal/zip code, Upgini will try to add those keys based on the provided set of search keys. This will broaden the search across all available data sources   
 ⭐️ Calculate *accuracy metrics and uplifts* after enrichment existing ML model with external features   
 ⭐️ Check the stability of accuracy gain from external data on out-of-time intervals and verification datasets. Mitigate risks of unstable external data dependencies in ML pipeline   
 ⭐️ Easy to use - single request to enrich training dataset with [*all of the keys at once*](#-search-key-types-we-support-more-to-come):  

{upgini-1.1.127 → upgini-1.1.129}/README.md

@@ -36,7 +36,7 @@
 ## 🚀 Awesome features
 ⭐️ Automatically find only relevant features that *give accuracy improvement for ML model*. Not just correlated with target variable, what 9 out of 10 cases gives zero accuracy improvement  
 ⭐️ Data source optimizations for ML tasks to *"squeeze" maximum information for models* out of the source data: automated feature generation with Large Language Models' data augmentation, RNNs, GraphNN; multiple data source ensembling  
-⭐️ Iterative search with *automatic search key augmentation* from all connected sources. If you are missing any search keys in your search request, Upgini will try to find those keys to broaden the search across all available data sources.   
+⭐️ *Automatic search key augmentation* from all connected sources. If you do not have all search keys in your search request, such as postal/zip code, Upgini will try to add those keys based on the provided set of search keys. This will broaden the search across all available data sources   
 ⭐️ Calculate *accuracy metrics and uplifts* after enrichment existing ML model with external features   
 ⭐️ Check the stability of accuracy gain from external data on out-of-time intervals and verification datasets. Mitigate risks of unstable external data dependencies in ML pipeline   
 ⭐️ Easy to use - single request to enrich training dataset with [*all of the keys at once*](#-search-key-types-we-support-more-to-come):  

{upgini-1.1.127 → upgini-1.1.129}/setup.py

@@ -35,7 +35,7 @@ def send_log(msg: str):
 
 
 here = Path(__file__).parent.resolve()
-version = "1.1.127"
+version = "1.1.129"
 try:
     send_log(f"Start setup PyLib version {version}")
     setup(

{upgini-1.1.127 → upgini-1.1.129}/src/upgini/features_enricher.py

@@ -53,7 +53,6 @@ from upgini.utils.datetime_utils import DateTimeSearchKeyConverter, is_time_seri
 from upgini.utils.display_utils import (
     display_html_dataframe,
     do_without_pandas_limits,
-    ipython_available,
 )
 from upgini.utils.email_utils import EmailSearchKeyConverter, EmailSearchKeyDetector
 from upgini.utils.features_validator import FeaturesValidator
@@ -576,13 +575,13 @@ class FeaturesEnricher(TransformerMixin):
             finally:
                 self.logger.info(f"Transform elapsed time: {time.time() - start_time}")
 
-            if self.country_added and COUNTRY in result.columns:
-                result = result.drop(columns=COUNTRY)
+            if self.country_added:
+                result = drop_existing_columns(result, COUNTRY)
 
             if keep_input:
                 return result
             else:
-                return result.drop(columns=[c for c in X.columns if c in result.columns])
+                return drop_existing_columns(result, X.columns)
 
     def calculate_metrics(
         self,
@@ -1002,7 +1001,7 @@ class FeaturesEnricher(TransformerMixin):
             self.logger.info("Cached enriched dataset found - use it")
             X_sampled, y_sampled, enriched_X, eval_set_sampled_dict, search_keys = self.__cached_sampled_datasets
             if exclude_features_sources:
-                enriched_X = enriched_X.drop(columns=[c for c in exclude_features_sources if c in enriched_X.columns])
+                enriched_X = drop_existing_columns(enriched_X, exclude_features_sources)
         elif len(self.feature_importances_) == 0:
             self.logger.info("No external features selected. So use only input datasets for metrics calculation")
             X_sampled, search_keys = self._extend_x(validated_X)
@@ -1024,7 +1023,7 @@ class FeaturesEnricher(TransformerMixin):
                 self._search_task.get_all_initial_raw_features(trace_id, metrics_calculation=True),
             )
 
-            enriched_X = enriched_Xy.drop(columns=TARGET)
+            enriched_X = drop_existing_columns(enriched_Xy, TARGET)
             x_columns = [
                 c for c in validated_X.columns.to_list() + self.fit_generated_features if c in enriched_X.columns
             ]
@@ -1042,7 +1041,7 @@ class FeaturesEnricher(TransformerMixin):
                     )
 
                 for idx in range(len(eval_set)):
-                    enriched_eval_X = enriched_eval_sets[idx + 1].drop(columns=TARGET)
+                    enriched_eval_X = drop_existing_columns(enriched_eval_sets[idx + 1], TARGET)
                     eval_X_sampled = enriched_eval_sets[idx + 1][x_columns].copy()
                     eval_y_sampled = enriched_eval_sets[idx + 1][TARGET].copy()
                     eval_set_sampled_dict[idx] = (eval_X_sampled, enriched_eval_X, eval_y_sampled)
@@ -1160,6 +1159,11 @@ class FeaturesEnricher(TransformerMixin):
         fitting_X = X_sorted[client_features].copy()
         fitting_enriched_X = enriched_X_sorted[client_features + existing_filtered_enriched_features].copy()
 
+        # Detect and drop high cardinality columns in train
+        columns_with_high_cardinality = FeaturesValidator.find_high_cardinality(fitting_X)
+        fitting_X = drop_existing_columns(fitting_X, columns_with_high_cardinality)
+        fitting_enriched_X = drop_existing_columns(fitting_enriched_X, columns_with_high_cardinality)
+
         fitting_eval_set_dict = dict()
         for idx, eval_tuple in eval_set_sampled_dict.items():
             eval_X_sampled, enriched_eval_X, eval_y_sampled = eval_tuple
@@ -1171,6 +1175,11 @@ class FeaturesEnricher(TransformerMixin):
             fitting_enriched_eval_X = enriched_eval_X_sorted[
                 client_features + existing_filtered_enriched_features
             ].copy()
+
+            # Drop high cardinality columns in eval set
+            fitting_eval_X = drop_existing_columns(fitting_eval_X, columns_with_high_cardinality)
+            fitting_enriched_eval_X = drop_existing_columns(fitting_enriched_eval_X, columns_with_high_cardinality)
+
             fitting_eval_set_dict[idx] = (
                 fitting_eval_X,
                 eval_y_sorted,
@@ -2027,7 +2036,7 @@ class FeaturesEnricher(TransformerMixin):
                 self.feature_importances_.append(round_shap_value(feature_meta.shap_value))
 
             internal_provider = feature_meta.data_provider or ""
-            if feature_meta.data_provider and ipython_available():
+            if feature_meta.data_provider:
                 provider = (
                     f"<a href='{feature_meta.data_provider_link}' "
                     "target='_blank' rel='noopener noreferrer'>"
@@ -2037,7 +2046,7 @@ class FeaturesEnricher(TransformerMixin):
                 provider = internal_provider
 
             internal_source = feature_meta.data_source or ""
-            if feature_meta.data_source and ipython_available():
+            if feature_meta.data_source:
                 source = (
                     f"<a href='{feature_meta.data_source_link}' "
                     "target='_blank' rel='noopener noreferrer'>"
@@ -2047,7 +2056,7 @@ class FeaturesEnricher(TransformerMixin):
                 source = internal_source
 
             internal_feature_name = feature_meta.name
-            if feature_meta.doc_link and ipython_available():
+            if feature_meta.doc_link:
                 feature_name = (
                     f"<a href='{feature_meta.doc_link}' "
                     "target='_blank' rel='noopener noreferrer'>"
@@ -2247,7 +2256,7 @@ class FeaturesEnricher(TransformerMixin):
                 self.warning_counter.increment()
         except (ImportError, NameError):
             print(msg)
-            print(self.features_info.head(60))
+            print(self._internal_features_info)
 
     def __validate_importance_threshold(self, importance_threshold: Optional[float]) -> float:
         try:
@@ -2487,3 +2496,14 @@ def drop_duplicates(df: Union[pd.DataFrame, np.ndarray]) -> pd.DataFrame:
 
 def hash_row(row) -> int:
     return zlib.crc32(str(row).encode())
+
+
+def drop_existing_columns(df: pd.DataFrame, columns_to_drop: Union[List[str], str]) -> pd.DataFrame:
+    if isinstance(columns_to_drop, str):
+        columns_to_drop = [columns_to_drop] if columns_to_drop in df.columns else []
+    elif hasattr(columns_to_drop, "__iter__"):
+        columns_to_drop = [c for c in columns_to_drop if c in df.columns]
+    else:
+        return df
+
+    return df.drop(columns=columns_to_drop)

{upgini-1.1.127 → upgini-1.1.129}/src/upgini/metrics.py

@@ -1,4 +1,5 @@
 import logging
+from copy import deepcopy
 from typing import Callable, List, Optional, Tuple, Union
 
 import numpy as np
@@ -6,7 +7,7 @@ import pandas as pd
 from catboost import CatBoostClassifier, CatBoostRegressor
 from lightgbm import LGBMClassifier, LGBMRegressor
 from numpy import log1p
-from pandas.api.types import is_numeric_dtype, is_string_dtype, is_integer_dtype
+from pandas.api.types import is_numeric_dtype
 from sklearn.metrics import SCORERS, check_scoring, get_scorer, make_scorer
 from sklearn.metrics._regression import (
     _check_reg_targets,
@@ -14,7 +15,6 @@ from sklearn.metrics._regression import (
     mean_squared_error,
 )
 from sklearn.model_selection import BaseCrossValidator, cross_validate
-from copy import deepcopy
 
 from upgini.errors import ValidationError
 from upgini.metadata import ModelTaskType
@@ -82,15 +82,6 @@ class EstimatorWrapper:
             else:
                 X[c] = X[c].astype(str)
 
-        # Remove high cardinality columns
-        row_count = X.shape[0]
-        columns_cardinality = [
-            i
-            for i in X
-            if (is_string_dtype(X[i]) or is_integer_dtype(X[i])) and (X[i].nunique() / row_count >= 0.9)
-        ]
-        X = X.drop(columns=columns_cardinality)
-
         if not isinstance(y, pd.Series):
             raise Exception(bundle.get("metrics_unsupported_target_type").format(type(y)))
 
@@ -98,7 +89,7 @@ class EstimatorWrapper:
         joined = joined[joined[y.name].notna()]
         joined = joined.reset_index(drop=True)
         X = joined.drop(columns=y.name)
-        y = joined[y.name].values
+        y = np.array(list(joined[y.name].values))
         return X, y, {}
 
     def cross_val_predict(self, X: pd.DataFrame, y: np.ndarray):
@@ -167,9 +158,9 @@ class EstimatorWrapper:
             kwargs["estimator"] = estimator_copy
             if isinstance(estimator, CatBoostClassifier) or isinstance(estimator, CatBoostRegressor):
                 if cat_features is not None:
-                    estimator_copy.set_params(cat_features=[
-                        X.columns.get_loc(cat_feature) for cat_feature in cat_features
-                    ])
+                    estimator_copy.set_params(
+                        cat_features=[X.columns.get_loc(cat_feature) for cat_feature in cat_features]
+                    )
                 estimator = CatBoostWrapper(**kwargs)
             else:
                 try:
@@ -312,11 +303,19 @@ def _get_scorer(target_type: ModelTaskType, scoring: Union[Callable, str, None])
             supported_metrics = set(SCORERS.keys())
             neg_metrics = [m[4:] for m in supported_metrics if m.startswith("neg_")]
             supported_metrics.update(neg_metrics)
-            supported_metrics.update([
-                "mean_squared_log_error", "MSLE", "msle",
-                "root_mean_squared_log_error", "RMSLE", "rmsle",
-                "root_mean_squared_error", "RMSE", "rmse"
-            ])
+            supported_metrics.update(
+                [
+                    "mean_squared_log_error",
+                    "MSLE",
+                    "msle",
+                    "root_mean_squared_log_error",
+                    "RMSLE",
+                    "rmsle",
+                    "root_mean_squared_error",
+                    "RMSE",
+                    "rmse",
+                ]
+            )
             raise ValidationError(bundle.get("metrics_invalid_scoring").format(scoring, sorted(supported_metrics)))
     elif hasattr(scoring, "__name__"):
         metric_name = scoring.__name__

{upgini-1.1.127 → upgini-1.1.129}/src/upgini/utils/features_validator.py

@@ -3,7 +3,7 @@ from logging import Logger
 from typing import List, Optional
 
 import pandas as pd
-from pandas.api.types import is_object_dtype  # , is_integer_dtype, is_string_dtype
+from pandas.api.types import is_object_dtype, is_integer_dtype, is_string_dtype
 from upgini.resource_bundle import bundle
 from upgini.utils.warning_counter import WarningCounter
 
@@ -53,3 +53,13 @@ class FeaturesValidator:
             self.logger.warning(msg)
 
         return empty_or_constant_features
+
+    @staticmethod
+    def find_high_cardinality(df: pd.DataFrame):
+        # Remove high cardinality columns
+        row_count = df.shape[0]
+        return [
+            i
+            for i in df
+            if (is_string_dtype(df[i]) or is_integer_dtype(df[i])) and (df[i].nunique() / row_count >= 0.9)
+        ]

{upgini-1.1.127 → upgini-1.1.129/src/upgini.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: upgini
-Version: 1.1.127
+Version: 1.1.129
 Summary: Intelligent data search & enrichment for Machine Learning
 Home-page: https://upgini.com/
 Author: Upgini Developers
@@ -66,7 +66,7 @@ License-File: LICENSE
 ## 🚀 Awesome features
 ⭐️ Automatically find only relevant features that *give accuracy improvement for ML model*. Not just correlated with target variable, what 9 out of 10 cases gives zero accuracy improvement  
 ⭐️ Data source optimizations for ML tasks to *"squeeze" maximum information for models* out of the source data: automated feature generation with Large Language Models' data augmentation, RNNs, GraphNN; multiple data source ensembling  
-⭐️ Iterative search with *automatic search key augmentation* from all connected sources. If you are missing any search keys in your search request, Upgini will try to find those keys to broaden the search across all available data sources.   
+⭐️ *Automatic search key augmentation* from all connected sources. If you do not have all search keys in your search request, such as postal/zip code, Upgini will try to add those keys based on the provided set of search keys. This will broaden the search across all available data sources   
 ⭐️ Calculate *accuracy metrics and uplifts* after enrichment existing ML model with external features   
 ⭐️ Check the stability of accuracy gain from external data on out-of-time intervals and verification datasets. Mitigate risks of unstable external data dependencies in ML pipeline   
 ⭐️ Easy to use - single request to enrich training dataset with [*all of the keys at once*](#-search-key-types-we-support-more-to-come):