teradataml 20.0.0.2__py3-none-any.whl → 20.0.0.4__py3-none-any.whl

teradataml/README.md

@@ -16,6 +16,306 @@ Copyright 2024, Teradata. All Rights Reserved.
 * [License](#license)
 
 ## Release Notes:
+
+#### teradataml 20.00.00.04
+* ##### New Features/Functionality
+    * ###### teradataml OTF Support:
+      * This release has enabled the support for accessing OTF data from teradataml. 
+      * User can now create a teradataml DataFrame on OTF table, allowing user to use teradataml functions.
+        * Example usage below:
+          * Creation of view on OTF/datalake table is not supported. Hence, user has to set `configure.temp_object_type` to `VT` using below-mentioned statement.
+            ```configure.temp_object_type = "VT"```
+          * User needs to provide additional information about datalake while creating the DataFrame. There are two approaches to provide datalake information
+            * Approach 1: Using `in_schema()`
+              ```
+              >>> from teradataml.dataframe.dataframe import in_schema
+              # Create an in_schema object to privide additional information about datalake.
+              >>> in_schema_tbl = in_schema(schema_name="datalake_db",
+              ...                           table_name="datalake_table_name",
+              ...                           datalake_name="datalake")
+              >>> otf_df = DataFrame(in_schema_tbl)
+                ```
+            * Approach 2: Using `DataFrame.from_table()`
+              ```
+               >>> otf_df = DataFrame.from_table(table_name = "datalake_table_name",
+               ...                               schema_name="datalake_db",
+               ...                               datalake_name="datalake")
+              ```
+          * Once this DataFrame is created, users can use any DataFrame method or analytics features/functionality from teradataml with it. Visit Limitations and considerations section in _Teradata Python Package User Guide_ to check the supportability.
+            *  Note: All further operations create volatile tables in local database.
+               ```
+                >>> new_df = otf_df.assign(new_col=otf_df.existing_col*2)
+               ```
+  * ###### teradataml: DataFrame
+    * Introduced a new feature 'Exploratory Data Analysis UI' (EDA-UI), which enhances
+      the user experience of teradataml with Jupyter notebook. EDA-UI is displayed by default 
+      when a teradataml DataFrame is printed in the Jupyter notebook.
+    * User can control the EDA-UI using a new configuration option `display.enable_ui`.
+      It can be disabled by setting `display.enable_ui` to False.
+    * New Function
+      * `get_output()` is added to get the result of Analytic function when executed from EDA UI.
+
+  * ###### OpensourceML
+    * `td_lightgbm` - A teradataml OpenSourceML module
+      * `deploy()` - User can now deploy the models created by lightgbm `Booster` and `sklearn` modules. Deploying the model stores the model in Vantage for future use with `td_lightgbm`.
+        * `td_lightgbm.deploy()` - Deploy the lightgbm `Booster` or any `scikit-learn` model trained outside Vantage.
+        * `td_lightgbm.train().deploy()` - Deploys the lightgbm `Booster` object trained within Vantage.
+        * `td_lightgbm.<sklearn_class>().deploy()` - Deploys lightgbm's sklearn class object created/trained within Vantage.
+      * `load()` - User can load the deployed models back in the current session. This allows user to use the lightgbm functions with the `td_lightgbm` module.
+        * `td_lightgbm.load()` - Load the deployed model in the current session.
+
+  * ###### FeatureStore
+    * New function `FeatureStore.delete()` is added to drop the Feature Store and corresponding repo from Vantage. 
+
+  * ###### Database Utility
+    * `db_python_version_diff()` - Identifies the Python interpreter major version difference between the interpreter installed on Vantage vs interpreter on the local user environment.
+    * `db_python_package_version_diff()` - Identifies the Python package version difference between the packages installed on Vantage vs the local user environment.
+
+  * ###### BYOM Function
+    * `ONNXEmbeddings()` - Calculate embeddings values in Vantage using an embeddings model that has been created outside Vantage and stored in ONNX format.
+
+  * ###### teradataml Options
+      * Configuration Options
+        * `configure.temp_object_type` - Allows user to choose between creating volatile tables or views for teradataml internal use. By default, teradataml internally creates the views for some of the operations. Now, with new configuration option, user can opt to create Volatile tables instead of views. This provides greater flexibility for users who lack the necessary permissions to create view or need to create views on tables without WITH GRANT permissions.
+      * Display Options
+        * `display.enable_ui` - Specifies whether to display exploratory data analysis UI when DataFrame is printed. By default, this option is enabled (True), allowing exploratory data analysis UI to be displayed. When set to False, exploratory data analysis UI is hidden. 
+
+* ##### Updates
+  * ###### teradataml: DataFrame function
+    * `describe()`
+      * New argument added: `pivot`.
+      * When argument `pivot` is set to False, Non-numeric columns are no longer supported for generating statistics. 
+        Use `CategoricalSummary` and `ColumnSummary`.
+    * `fillna()` - Accepts new argument `partition_column` to partition the data and impute null values accordingly.
+    * Optimised performance for `DataFrame.plot()`.
+      * `DataFrame.plot()` will not regenerate the image when run more than once with same arguments.
+    * `DataFrame.from_table()`: New argument `datalake_name` added to accept datalake name while creating DataFrame on datalake table.
+
+  * ###### teradataml: DataFrame Utilities
+    * `in_schema()`: New argument `datalake_name` added to accept datalake name.
+
+  * ###### Table Operator 
+    * `Apply()` no longer looks at authentication token by default. Authentication token is now required only if user want to update backend Open Analytics Framework service.
+  
+  * ###### Hyper Parameter Tuner
+    * `GridSearch()` and `RandomSearch()` now displays a message to refer to `get_error_log()` api when model training fails in HPT.
+
+  * ###### teradataml Options
+    * Configuration Options
+      * `configure.indb_install_location`
+        Determines the installation location of the In-DB Python package based on the installed RPM version.
+  
+  * ###### teradataml Context Creation
+    * `create_context()` - Enables user to create connection using either parameters set in environment or config file, in addition to previous method. Newly added options help users to hide the sensitive data from the script.
+
+  * ###### Open Analytics Framework
+    * Enhanced the `create_env()` to display a message when an invalid base_env is passed, informing users that the default base_env is being used.
+
+  * ###### OpensourceML
+    * Raises a TeradataMlException, if the Python interpreter major version is different between the Vantage Python environment and the local user environment.
+    * Displays a warning, if specific Python package versions are different between the Vantage Python environment and the local user environment. 
+
+  * ###### Database Utility
+    * `db_list_tables()`: New argument `datalake_name` added to accept datalake name to list tables from.
+    * `db_drop_table()`:
+      * New argument `datalake_name` added to accept datalake name to drop tables from.
+      * New argument `purge` added to specify whether to use `PURGE ALL` or `NO PURGE` clause while dropping table.
+
+* ##### Bug Fixes
+  * `td_lightgbm` OpensourceML module: In multi model case, `td_lightgbm.Dataset().add_features_from()` function should add features of one partition in first Dataset to features of the same partition in second Dataset. This is not the case before and this function fails. Fixed this now.
+  * Fixed a minor bug in the `Shap()` and converted argument `training_method` to required argument.
+  * Fixed PCA-related warnings in `AutoML`.
+  * `AutoML` no longer fails when data with all categorical columns are provided.
+  * Fixed `AutoML` issue with upsampling method.
+  * Excluded the identifier column from outlier processing in `AutoML`.
+  * `DataFrame.set_index()` no longer modifies the original DataFrame's index when argument `append` is used.
+  * `concat()` function now supports the DataFrame with column name starts with digit or contains special characters or contains reserved keywords.
+  * `create_env()` proceeds to install other files even if current file installation fails.
+  * Corrected the error message being raised in `create_env()` when authentication is not set.
+  * Added missing argument `charset` for Vantage Analytic Library functions.
+  * New argument `seed` is added to `AutoML`, `AutoRegressor` and `AutoClassifier` to ensure consistency on result.
+  * Analytic functions now work even if name of columns for underlying tables is non-ascii characters.
+
+#### teradataml 20.00.00.03
+
+* teradataml no longer supports setting the `auth_token` using `set_config_params()`. Users should use `set_auth_token()` to set the token. 
+
+* ##### New Features/Functionality
+  * ###### teradataml: DataFrame
+    * New Function
+      * `alias()` - Creates a DataFrame with alias name.
+    * New Properties
+      * `db_object_name` - Get the underlying database object name, on which DataFrame is created.
+
+  * ###### teradataml: GeoDataFrame
+    * New Function
+      * `alias()` - Creates a GeoDataFrame with alias name.
+
+  * ###### teradataml: DataFrameColumn a.k.a. ColumnExpression
+    * _Arithmetic Functions_
+      * `DataFrameColumn.isnan()` - Function evaluates expression to determine if the floating-point
+                                    argument is a NaN (Not-a-Number) value.
+      * `DataFrameColumn.isinf()` - Function evaluates expression to determine if the floating-point
+                                    argument is an infinite number.
+      * `DataFrameColumn.isfinite()` - Function evaluates expression to determine if it is a finite
+                                       floating value.
+
+  * ###### FeatureStore - handles feature management within the Vantage environment
+    * FeatureStore Components
+      * Feature - Represents a feature which is used in ML Modeling. 
+      * Entity - Represents the columns which serves as uniqueness for the data used in ML Modeling. 
+      * DataSource - Represents the source of Data.
+      * FeatureGroup - Collection of Feature, Entity and DataSource.
+        * Methods
+          * `apply()` - Adds Feature, Entity, DataSource to a FeatureGroup.
+          * `from_DataFrame()` - Creates a FeatureGroup from teradataml DataFrame.
+          * `from_query()` - Creates a FeatureGroup using a SQL query.
+          * `remove()` - Removes Feature, Entity, or DataSource from a FeatureGroup.
+          * `reset_labels()` - Removes the labels assigned to the FeatureGroup, that are set using `set_labels()`.
+          * `set_labels()` - Sets the Features as labels for a FeatureGroup.
+        * Properties
+          * `features` - Get the features of a FeatureGroup.
+          * `labels` - Get the labels of FeatureGroup.
+    * FeatureStore 
+      * Methods
+        * `apply()` - Adds Feature, Entity, DataSource, FeatureGroup to FeatureStore.
+        * `archive_data_source()` - Archives a specified DataSource from a FeatureStore.
+        * `archive_entity()` - Archives a specified Entity from a FeatureStore.
+        * `archive_feature()` - Archives a specified Feature from a FeatureStore.
+        * `archive_feature_group()` - Archives a specified FeatureGroup from a FeatureStore. Method archives underlying Feature, Entity, DataSource also.
+        * `delete_data_source()` - Deletes an archived DataSource.
+        * `delete_entity()` - Deletes an archived Entity.
+        * `delete_feature()` - Deletes an archived Feature.
+        * `delete_feature_group()` - Deletes an archived FeatureGroup. 
+        * `get_data_source()` - Get the DataSources associated with FeatureStore.
+        * `get_dataset()` - Get the teradataml DataFrame based on Features, Entities and DataSource from FeatureGroup.
+        * `get_entity()` - Get the Entity associated with FeatureStore.
+        * `get_feature()` - Get the Feature associated with FeatureStore.
+        * `get_feature_group()` - Get the FeatureGroup associated with FeatureStore.
+        * `list_data_sources()` - List DataSources.
+        * `list_entities()` - List Entities.
+        * `list_feature_groups()` - List FeatureGroups.
+        * `list_features()` - List Features.
+        * `list_repos()` - List available repos which are configured for FeatureStore. 
+        * `repair()` - Repairs the underlying FeatureStore schema on database.  
+        * `set_features_active()` - Marks the Features as active.
+        * `set_features_inactive()` - Marks the Features as inactive.
+        * `setup()` - Setup the FeatureStore for a repo.
+      * Property
+        * `repo` - Property for FeatureStore repo.
+        * `grant` - Property to Grant access on FeatureStore to user.
+        * `revoke` - Property to Revoke access on FeatureStore from user.
+
+  * ###### teradataml: Table Operator Functions
+    * `Image2Matrix()` - Converts an image into a matrix.
+  
+  * ###### teradataml: SQLE Engine Analytic Functions
+    * New Analytics Database Analytic Functions:
+      * `CFilter()`
+      * `NaiveBayes()`
+      * `TDNaiveBayesPredict()`
+      * `Shap()`
+      * `SMOTE()`
+
+    * ###### teradataml: Unbounded Array Framework (UAF) Functions
+      * New Unbounded Array Framework(UAF) Functions:
+        * `CopyArt()`
+
+  * ###### General functions
+    * Vantage File Management Functions
+      * `list_files()` - List the installed files in Database.
+
+  * ###### OpensourceML: LightGBM
+    * teradataml adds support for lightGBM package through `OpensourceML` (`OpenML`) feature.
+      The following functionality is added in the current release:
+      * `td_lightgbm` - Interface object to run lightgbm functions and classes through Teradata Vantage.
+      Example usage below:
+        ```
+        from teradataml import td_lightgbm, DataFrame
+
+        df_train = DataFrame("multi_model_classification")
+
+        feature_columns = ["col1", "col2", "col3", "col4"]
+        label_columns = ["label"]
+        part_columns = ["partition_column_1", "partition_column_2"]
+
+        df_x = df_train.select(feature_columns)
+        df_y = df_train.select(label_columns)
+
+        # Dataset creation.
+        # Single model case.
+        obj_s = td_lightgbm.Dataset(df_x, df_y, silent=True, free_raw_data=False)
+
+        # Multi model case.
+        obj_m = td_lightgbm.Dataset(df_x, df_y, free_raw_data=False, partition_columns=part_columns)
+        obj_m_v = td_lightgbm.Dataset(df_x, df_y, free_raw_data=False, partition_columns=part_columns)
+
+        ## Model training.
+        # Single model case.
+        opt = td_lightgbm.train(params={}, train_set = obj_s, num_boost_round=30)
+
+        opt.predict(data=df_x, num_iteration=20, pred_contrib=True)
+
+        # Multi model case.
+        opt = td_lightgbm.train(params={}, train_set = obj_m, num_boost_round=30,
+                                callbacks=[td_lightgbm.record_evaluation(rec)],
+                                valid_sets=[obj_m_v, obj_m_v])
+        
+        # Passing `label` argument to get it returned in output DataFrame.
+        opt.predict(data=df_x, label=df_y, num_iteration=20)
+
+        ```
+      * Added support for accessing scikit-learn APIs using exposed inteface object `td_lightgbm`.
+    
+    Refer Teradata Python Package User Guide for more details of this feature, arguments, usage, examples and supportability in Vantage.
+
+  * ###### teradataml: Functions
+    * `register()` - Registers a user defined function (UDF).
+    * `call_udf()` - Calls a registered user defined function (UDF) and returns ColumnExpression.
+    * `list_udfs()` - List all the UDFs registered using 'register()' function.
+    * `deregister()` - Deregisters a user defined function (UDF).
+
+  * ###### teradataml: Options
+    * Configuration Options
+      * `table_operator` - Specifies the name of table operator.
+
+* ##### Updates
+  * ###### General functions
+    * `set_auth_token()` - Added `base_url` parameter which accepts the CCP url. 
+                           'ues_url' will be deprecated in future and users
+                           will need to specify 'base_url' instead.
+
+  * ###### teradataml: DataFrame function
+     * `join()`
+       * Now supports compound ColumExpression having more than one binary operator in `on` argument.
+       * Now supports ColumExpression containing FunctionExpression(s) in `on` argument.
+       * self-join now expects aliased DataFrame in `other` argument.
+
+  * ###### teradataml: GeoDataFrame function
+     * `join()`
+       * Now supports compound ColumExpression having more than one binary operator in `on` argument.
+       * Now supports ColumExpression containing FunctionExpression(s) in `on` argument.
+       * self-join now expects aliased DataFrame in `other` argument.
+
+  * ###### teradataml: Unbounded Array Framework (UAF) Functions
+    * `SAX()` - Default value added for `window_size` and `output_frequency`.
+    * `DickeyFuller()`
+      * Supports TDAnalyticResult as input.
+      * Default value added for `max_lags`.
+      * Removed parameter `drift_trend_formula`.
+      * Updated permitted values for `algorithm`.
+
+  * ##### teradataml: AutoML
+    * `AutoML`, `AutoRegressor` and `AutoClassifier`
+      * Now supports DECIMAL datatype as input.
+  
+  * ##### teradataml: SQLE Engine Analytic Functions
+    * `TextParser()`
+      * Argument name `covert_to_lowercase` changed to `convert_to_lowercase`.
+
+* ##### Bug Fixes
+  * `db_list_tables()` now returns correct results when '%' is used.
+
 #### teradataml 20.00.00.02
 
 * teradataml will no longer be supported with SQLAlchemy < 2.0.
@@ -83,6 +383,10 @@ Copyright 2024, Teradata. All Rights Reserved.
       * Following arguments will be deprecated in the future:
         * `ues_url`
         * `auth_token`
+  
+  * #### teradata DataFrame
+    * `to_pandas()` - Function returns the pandas dataframe with Decimal columns types as float instead of object.
+                      If user want datatype to be object, set argument `coerce_float` to False.
 
   * ###### Database Utility
       * `list_td_reserved_keywords()` - Accepts a list of strings as argument.
@@ -102,7 +406,7 @@ Copyright 2024, Teradata. All Rights Reserved.
 * ##### Bug Fixes
   * KNN `predict()` function can now predict on test data which does not contain target column.
   * Metrics functions are supported on the Lake system.
-  * The following OpensourceML functions from different sklearn modules are fixed.
+  * The following OpensourceML functions from different sklearn modules in single model case are fixed.
     * `sklearn.ensemble`:
       * ExtraTreesClassifier - `apply()`
       * ExtraTreesRegressor - `apply()`
@@ -115,12 +419,21 @@ Copyright 2024, Teradata. All Rights Reserved.
       * Nystroem - `transform()`, `fit_transform()`
       * PolynomialCountSketch - `transform()`, `fit_transform()`
       * RBFSampler - `transform()`, `fit_transform()`
-    * `sklearn.neighbours`:
+    * `sklearn.neighbors`:
       * KNeighborsTransformer - `transform()`, `fit_transform()`
       * RadiusNeighborsTransformer - `transform()`, `fit_transform()`
     * `sklearn.preprocessing`:
       * KernelCenterer - `transform()`
       * OneHotEncoder - `transform()`, `inverse_transform()`
+  * The following OpensourceML functions from different sklearn modules in multi model case are fixed.
+    * `sklearn.feature_selection`:
+      * SelectFpr - `transform()`, `fit_transform()`, `inverse_transform()`
+      * SelectFdr - `transform()`, `fit_transform()`, `inverse_transform()`
+      * SelectFromModel - `transform()`, `fit_transform()`, `inverse_transform()`
+      * SelectFwe - `transform()`, `fit_transform()`, `inverse_transform()`
+      * RFECV - `transform()`, `fit_transform()`, `inverse_transform()`
+    * `sklearn.clustering`:
+      * Birch - `transform()`, `fit_transform()`
   * OpensourceML returns teradataml objects for model attributes and functions instead of sklearn
     objects so that the user can perform further operations like `score()`, `predict()` etc on top
     of the returned objects.

teradataml/__init__.py

@@ -71,3 +71,7 @@ session_queryband.configure_queryband_parameters(app_name="TDML", app_version=__
 
 # Import functions.
 from teradataml.dataframe.functions import *
+
+# Import FeatureStore and VectorStore
+from teradataml.store import *
+

teradataml/_version.py

@@ -8,4 +8,4 @@
 # 
 # ################################################################## 
 
-version = "20.00.00.02"
+version = "20.00.00.04"

teradataml/analytics/analytic_function_executor.py

@@ -28,8 +28,8 @@ from teradataml.common.messages import Messages, MessageCodes
 from teradataml.common.wrapper_utils import AnalyticsWrapperUtils
 from teradataml.common.utils import UtilFuncs
 from teradataml.context.context import _get_context_temp_databasename
-from teradataml.dataframe.dataframe import in_schema
-from teradataml.dbutils.dbutils import _create_table, db_drop_table
+from teradataml.dataframe.dataframe import in_schema, DataFrame
+from teradataml.dbutils.dbutils import _create_table, db_drop_table, list_td_reserved_keywords
 from teradatasqlalchemy.types import *
 from teradataml.table_operators.table_operator_query_generator import TableOperatorQueryGenerator
 from teradataml.telemetry_utils.queryband import collect_queryband
@@ -343,6 +343,17 @@ class _AnlyticFunctionExecutor:
                 self._func_output_args.append(temp_table_name)
                 self._function_output_table_map[lang_name] = temp_table_name
 
+    def _get_column_name_from_feature(self, obj):
+        # Extract the associated column name from Feature.
+        from teradataml.store.feature_store.feature_store import Feature
+        if isinstance(obj, Feature):
+            return obj.column_name
+
+        if isinstance(obj, list):
+            return [self._get_column_name_from_feature(col) for col in obj]
+
+        return obj
+
     def _process_other_argument(self, **kwargs):
         """
         DESCRIPTION:
@@ -439,6 +450,9 @@ class _AnlyticFunctionExecutor:
 
             self._validate_analytic_function_argument(arg_name, arg_value, argument)
 
+            # Extract column names if it is a Feature.
+            arg_value = self._get_column_name_from_feature(arg_value)
+
             # Perform the checks which are specific to argument(_AnlyFuncArgument) type.
             # Check lower bound and upper bound for number type of arguments.
             if isinstance(arg_value, (int, float)):
@@ -468,12 +482,21 @@ class _AnlyticFunctionExecutor:
 
                 # Validate column is existed or not in the table.
                 _Validators._validate_dataframe_has_argument_columns(
-                    arg_value, arg_name, dataframe, target_table_argument_name)
+                    arg_value, arg_name, dataframe, target_table_argument_name, case_insensitive=True)
 
                 # Append square brackets for column range when function
                 # does not require special case handler.
                 arg_value = self._spl_func_obj._add_square_bracket(arg_value)
 
+                # Check if there are columns with non-ASCII characters.
+                if UtilFuncs._is_ascii(arg_value):
+                    arg_value = UtilFuncs._teradata_quote_arg(arg_value, "\"", False)
+                # Handling special case for Teradata reserved keywords or column names with spaces.
+                # If argument is a string or list of strings, then add quotes to the string.
+                elif arg_name not in ["partition_columns"] and (\
+                    UtilFuncs._contains_space(arg_value) or list_td_reserved_keywords(arg_value)):
+                    arg_value = UtilFuncs._teradata_quote_arg(arg_value, "\"", False) 
+
             # SequenceInputBy arguments require special processing.
             if 500 <= argument.get_r_order_number() <= 510:
                 quoted_value = UtilFuncs._teradata_collapse_arglist(arg_value, "")
@@ -535,6 +558,17 @@ class _AnlyticFunctionExecutor:
             return repr_string
         self._dyn_cls_data_members["__repr__"] = print_result
 
+        def copy(self, **args):
+            """ Function to copy the ART to another table."""
+            from teradataml import CopyArt
+            params = {
+                "data": self.result,
+                "database_name": args.get("database_name", None),
+                "table_name": args.get("table_name", None),
+                "map_name": args.get("map_name", None),
+                "persist": args.get("persist", False)}
+            return CopyArt(**params)
+
         query = self.sqlmr_query
         build_time = None if self.__build_time is None else round(self.__build_time, 2)
 
@@ -544,6 +578,7 @@ class _AnlyticFunctionExecutor:
         # To list attributes using dict()
         self._dyn_cls_data_members["__dict__"] = self._dyn_cls_data_members
         self._dyn_cls_data_members["_mlresults"] = self._mlresults
+        self._dyn_cls_data_members["copy"] = copy
 
         # Dynamic class creation with In-DB function name.
         indb_class = type(self.func_name, (object,), self._dyn_cls_data_members)
@@ -685,10 +720,17 @@ class _AnlyticFunctionExecutor:
         kwargs.update(kwargs.pop("generic_arguments", {}))
 
         # Add all arguments to dynamic class as data members.
+        global_volatile = False
+        if configure.temp_object_type == TeradataConstants.TERADATA_VOLATILE_TABLE:
+            global_volatile = True
 
         start_time = time.time()
         persist = kwargs.get("persist", False)
-        volatile = kwargs.get("volatile", False)
+        # Use global volatile only when persist argument is False. If persist argument
+        # is True, then volatile can't be used whether it is global volatile or normal 
+        # volatile. If it is normal volatile, then it will raise 
+        # `CANNOT_USE_TOGETHER_WITH` error below.
+        volatile = kwargs.get("volatile", global_volatile if not persist else False)
         display_table_name = kwargs.get("display_table_name", True)
 
         # Validate local_order_column argument type and values.
@@ -700,6 +742,14 @@ class _AnlyticFunctionExecutor:
             raise TeradataMlException(
                 Messages.get_message(MessageCodes.CANNOT_USE_TOGETHER_WITH, "persist", "volatile"),
                 MessageCodes.CANNOT_USE_TOGETHER_WITH)
+        
+        # If function is VectorDistance and largereference_input is set to True,
+        # then set data_partition_column to PartitionKind.DIMENSION and
+        # reference_data_partition_column to PartitionKind.ANY .
+        if self.func_name == "VectorDistance" and \
+            kwargs.get("largereference_input", False):
+            kwargs['target_data_partition_column'] = PartitionKind.DIMENSION
+            kwargs['reference_data_partition_column'] = PartitionKind.ANY
 
         self._dyn_cls_data_members.update(kwargs)
         
@@ -721,6 +771,11 @@ class _AnlyticFunctionExecutor:
         if self.func_name in ['GLM', 'TDGLMPredict'] and \
             any(key in kwargs for key in ['data_partition_column', 'data_hash_column', 'local_order_data']):
             skip_output_arg_processing = True
+        elif self.func_name in ['CopyArt']:
+            # CopyArt function take care of persisting the result table internally
+            # through 'permanent_table' argument.
+            persist = False
+            volatile = False
 
         if not skip_output_arg_processing:
             self._process_output_argument(**kwargs)
@@ -994,7 +1049,8 @@ class _SQLEFunctionExecutor(_AnlyticFunctionExecutor):
                     _Validators._validate_dataframe_has_argument_columns(arg_value,
                                                                          arg,
                                                                          input_table_arg_value,
-                                                                         input_table_arg
+                                                                         input_table_arg,
+                                                                         case_insensitive=True
                                                                          )
 
             order_column_arg_value = UtilFuncs._teradata_collapse_arglist(order_column_arg_value, "\"")
@@ -1446,7 +1502,8 @@ class _TableOperatorExecutor(_SQLEFunctionExecutor):
         _Validators._validate_dataframe_has_argument_columns(hash_column_value,
                                                              hash_column_arg,
                                                              input_table_arg_value,
-                                                             input_table_arg
+                                                             input_table_arg,
+                                                             case_insensitive=True
                                                              )
         
         # Hash and order by can be used together as long as is_local_order = True.
@@ -2180,6 +2237,31 @@ class _StoredProcedureExecutor(_UAFFunctionExecutor):
             self._func_other_args['database_name'] = UtilFuncs._teradata_quote_arg(schema_name, "\'", False)
             self._func_other_args['table_name'] = UtilFuncs._teradata_quote_arg(table_name, "\'", False)
 
+        # 'CopyArt' function requires 'SRC_DATABASENMAE' and 'SRC_TABLENAME' as input arguments.
+        # Extract the database and table name from the 'data' argument and add them to the
+        # '_func_other_args' dictionary.
+        if self.func_name == "CopyArt":
+            data = kwargs.get('data', None)
+            argument_info = ["data", data, False, (DataFrame), True]
+            # 'data' is a required argument for 'CopyArt' function to get the source table name and database name.
+            _Validators._validate_missing_required_arguments([argument_info])
+            # 'data' should be a DataFrame.
+            _Validators._validate_function_arguments([argument_info])
+
+            # Add the 'SRC_DATABASENMAE' and 'SRC_TABLENAME' to the '_func_other_args' dictionary.
+            self._func_other_args["SRC_DATABASENMAE"] = "'{0}'".format(UtilFuncs._extract_db_name(data._table_name))
+            self._func_other_args["SRC_TABLENAME"] = "'{0}'".format(UtilFuncs._extract_table_name(data._table_name))
+
+            # Setting permanent_table to True if 'persist' is set to True, else False.
+            kwargs['permanent_table'] = 'True' if kwargs.get('persist', False) else 'False'
+
+            # Setting 'map_name' to empty string if not provided.
+            if kwargs.get('map_name', None) is None:
+                kwargs['map_name'] = ""
+
+            # CopyArt does not take 'data' as input argument.
+            kwargs.pop('data')
+
         for argument in self._metadata.arguments:
             sql_name = argument.get_name()
             lang_name = argument.get_lang_name()
@@ -2236,8 +2318,13 @@ class _StoredProcedureExecutor(_UAFFunctionExecutor):
           Internal function to process the function output.
         """
         for lang_name, table_name in self._function_output_table_map.items():
-            out_table_name = UtilFuncs._extract_table_name(table_name)
-            out_db_name = UtilFuncs._extract_db_name(table_name)
+            # For 'CopyArt' function, the result should be the destination table name and database name provided as input.
+            if self.func_name == "CopyArt":
+                out_table_name = kwargs.get('table_name')
+                out_db_name = kwargs.get('database_name')
+            else:
+                out_table_name = UtilFuncs._extract_table_name(table_name)
+                out_db_name = UtilFuncs._extract_db_name(table_name)
             df = self._awu._create_data_set_object(
                 df_input=out_table_name, database_name=out_db_name, source_type="table")
             self._dyn_cls_data_members[lang_name] = df

teradataml/analytics/byom/__init__.py

@@ -4,7 +4,7 @@ from teradataml.analytics.byom.PMMLPredict import PMMLPredict
 from teradataml.analytics.meta_class import _AnalyticFunction
 from teradataml.analytics.meta_class import _common_init, _common_dir
 
-_byom_functions = ['H2OPredict', 'PMMLPredict', 'ONNXPredict', 'DataikuPredict', 'DataRobotPredict']
+_byom_functions = ['H2OPredict', 'PMMLPredict', 'ONNXPredict', 'DataikuPredict', 'DataRobotPredict', 'ONNXEmbeddings']
 
 for func in _byom_functions:
     globals()[func] = type("{}".format(func), (_AnalyticFunction,),

teradataml/analytics/json_parser/metadata.py

@@ -1179,10 +1179,19 @@ class _AnlyFuncMetadata:
             # from teradataml.data.docs.<function_type>.<doc_dir_with_version_info>.<func_name>
             # import <func_name>
             func_module = __import__(("teradataml.data.docs.{}.{}.{}".
-                                      format(function_type, doc_dir, self.func_name)),
-                                     fromlist=[self.func_name])
-            return getattr(func_module, self.func_name).__doc__
+                                    format(function_type, doc_dir, self.func_name)),
+                                    fromlist=[self.func_name])
+            return getattr(func_module, self.func_name).__doc__   
         except:
+            # For db_version 20.00, if function type is sqle, then check for docs_17_20 directory.
+            if version_dir == '20.00' and function_type == 'sqle':
+                try:
+                    func_module = __import__(("teradataml.data.docs.{}.{}.{}".
+                                             format(function_type, "docs_17_20", self.func_name)),
+                                             fromlist=[self.func_name])
+                    return getattr(func_module, self.func_name).__doc__
+                except:
+                    pass
             return ("Refer to Teradata Package for Python Function Reference guide for "
                     "Documentation. Reference guide can be found at: https://docs.teradata.com ."
                     "Refer to the section with Database version: {}".format(self.__database_version))

teradataml/analytics/json_parser/utils.py

@@ -54,7 +54,6 @@ def _get_json_data_from_tdml_repo():
     # both versions are matched, then the json store has data available so no need
     # to parse again.
     if configure.database_version != _JsonStore.version:
-
         # Json store version is different from database version. So, json's should
         # be parsed again. Before parsing the json, first clean the json store.
         _JsonStore.clean()
@@ -171,9 +170,15 @@ def __get_json_files_directory():
         if func_info.value["lowest_version"]:
             # Check if current function type is allowed on connected Vantage version or not.
             if func_info.value["func_type"] in func_type_json_version.keys():
+                # If function type is SQLE and db_version is 20.00, then add 17.20 JSON directory.
+                if func_type_json_version[func_info.value["func_type"]] == '20.00' and  \
+                   func_info.value["func_type"] == 'sqle':
+                    yield [UtilFuncs._get_data_directory(dir_name="jsons", func_type=func_info,
+                                                         version='17.20'),
+                                                         func_info.name]
                 yield [UtilFuncs._get_data_directory(dir_name="jsons", func_type=func_info,
                                                      version=func_type_json_version[func_info.value["func_type"]]),
-                       func_info.name]
+                                                     func_info.name] 
         else:
             yield [UtilFuncs._get_data_directory(dir_name="jsons", func_type=func_info), func_info.name]
 

teradataml/analytics/sqle/__init__.py

@@ -71,6 +71,7 @@ _sqle_functions = ['ANOVA',
                   'Silhouette',
                   'SimpleImputeFit',
                   'SimpleImputeTransform',
+                  'SMOTE',
                   'StrApply',
                   'StringSimilarity',
                   'SVM',
@@ -94,7 +95,10 @@ _sqle_functions = ['ANOVA',
                   'WordEmbeddings',
                   'XGBoost',
                   'XGBoostPredict',
-                  'ZTest'
+                  'ZTest',
+                  'HNSW',
+                  'HNSWPredict',
+                  'HNSWSummary',
                   ]
 
 for func in _sqle_functions:

teradataml/analytics/table_operator/__init__.py

@@ -1,7 +1,7 @@
 from teradataml.analytics.meta_class import _AnalyticFunction
 from teradataml.analytics.meta_class import _common_init, _common_dir
 
-_nos_functions = ['ReadNOS', 'WriteNOS']
+_nos_functions = ['ReadNOS', 'WriteNOS', 'Image2Matrix']
 
 for func in _nos_functions:
     globals()[func] = type("{}".format(func), (_AnalyticFunction,),