goe-framework 1.0.6__tar.gz → 1.0.6.dev0__tar.gz

{goe_framework-1.0.6/src/goe_framework.egg-info → goe_framework-1.0.6.dev0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: goe-framework
-Version: 1.0.6
+Version: 1.0.6.dev0
 Maintainer-email: Neil Johnson <neiljohnson@google.com>
 Project-URL: Homepage, https://github.com/gluent/goe
 Project-URL: Issues, https://github.com/gluent/goe/issues
@@ -77,9 +77,9 @@ At present GOE is a command line tool. Alongside installing the Python package w
 # Offload Home
 In addition to the GOE software we need a supporting directory tree called the Offload Home. This is identified using the `OFFLOAD_HOME` environment variable. In this directory we keep configuration files, logs and the GOE software if you choose not to run scripts directly out of the cloned repo. Offload Home will also typically contain a Python virtual environment into which the GOE package and dependencies will be installed, you can run these out of the repository directory but, for separation of duties purposes, may choose to keep the source code away from users of the tool.
 
-# Installation from a package
+# Installation From a Package
 
-## Installing the GOE Python package
+## Installing the GOE Python Package
 1) Copy the package to the target host, this may not be the same host the repository was cloned to.
 2) Create the Offload Home directory, for example:
 ```
@@ -98,12 +98,17 @@ python3 -m venv .venv && . .venv/bin/activate
 5) Install the GOE Python package:
 ```
 cd $OFFLOAD_HOME
-GOE_WHEEL="goe_framework-$(cat version | tr 'A-Z-' 'a-z.')0-py3-none-any.whl"
+python3 -m pip install goe-framework==<goe-version>
+```
+or use the bundled wheel file:
+```
+cd $OFFLOAD_HOME
+GOE_WHEEL="goe_framework-<goe-version>-py3-none-any.whl"
 python3 -m pip install lib/${GOE_WHEEL}
 ```
 
-## Configuration file
-Create `offload.env` in the Offload Home, this file contains the necessary configuration specific to your environment:
+## Configuration File
+Create `offload.env` in the Offload Home. This file contains the necessary configuration specific to your environment:
 ```
 cp ${OFFLOAD_HOME}/conf/oracle-bigquery-offload.env.template ${OFFLOAD_HOME}/conf/offload.env
 vi ${OFFLOAD_HOME}/conf/offload.env
@@ -129,7 +134,7 @@ If using Dataproc Batches to provide Spark:
 - GOOGLE_DATAPROC_REGION
 - GOOGLE_DATAPROC_BATCHES_SUBNET
 
-## Install database objects
+## Install Database Objects
 To install supporting database objects you need access to a database admin account that can create users, grant them system privileges and create objects in the schemas created. SYSTEM has been used in the example below but this is *not* a necessity:
 ```
 cd ${OFFLOAD_HOME}/setup
@@ -143,11 +148,11 @@ alter user goe_adm identified by ...;
 alter user goe_app identified by ...;
 ```
 
-# Building a custom package
+# Building a Custom Package
 
-If you want to test with the latest commits that have not yet been included in a Github release you can build a custom package from the repository.
+If you want to test with the latest commits that have not yet been included in a GitHub release you can build a custom package from the repository.
 
-## Prepare the host/cloned repository
+## Prepare the Host/Cloned Repository
 Debian prerequisites:
 ```
 sudo apt-get -y install make python3-venv
@@ -166,13 +171,13 @@ curl -fL https://github.com/coursier/coursier/releases/latest/download/cs-x86_64
 . ~/.bash_profile
 ```
 
-## Make a GOE package
+## Make a GOE Package
 To create a package which contains all required artifacts for running GOE commands use the `make` target below:
 ```
 make clean && make package
 ```
 
-# Install for development
+# Install for Development
 To create a Python virtual environment and install all required dependencies into the repository directory:
 ```
 make clean && make install-dev
@@ -187,7 +192,10 @@ Note only the Python dependencies for Oracle and BigQuery are installed by defau
 make install-dev-extras
 ```
 
-# Running commands
+# Supporting Infrastructure
+GOE requires access to cloud storage and Spark. For a Google Cloud installation this is described here: [Google Cloud Platform Setup for a BigQuery Target](docs/gcp_setup.md)
+
+# Running Commands
 Activate the GOE Python virtual environment:
 ```
 source ./.venv/bin/activate

{goe_framework-1.0.6 → goe_framework-1.0.6.dev0}/README.md

@@ -7,9 +7,9 @@ At present GOE is a command line tool. Alongside installing the Python package w
 # Offload Home
 In addition to the GOE software we need a supporting directory tree called the Offload Home. This is identified using the `OFFLOAD_HOME` environment variable. In this directory we keep configuration files, logs and the GOE software if you choose not to run scripts directly out of the cloned repo. Offload Home will also typically contain a Python virtual environment into which the GOE package and dependencies will be installed, you can run these out of the repository directory but, for separation of duties purposes, may choose to keep the source code away from users of the tool.
 
-# Installation from a package
+# Installation From a Package
 
-## Installing the GOE Python package
+## Installing the GOE Python Package
 1) Copy the package to the target host, this may not be the same host the repository was cloned to.
 2) Create the Offload Home directory, for example:
 ```
@@ -28,12 +28,17 @@ python3 -m venv .venv && . .venv/bin/activate
 5) Install the GOE Python package:
 ```
 cd $OFFLOAD_HOME
-GOE_WHEEL="goe_framework-$(cat version | tr 'A-Z-' 'a-z.')0-py3-none-any.whl"
+python3 -m pip install goe-framework==<goe-version>
+```
+or use the bundled wheel file:
+```
+cd $OFFLOAD_HOME
+GOE_WHEEL="goe_framework-<goe-version>-py3-none-any.whl"
 python3 -m pip install lib/${GOE_WHEEL}
 ```
 
-## Configuration file
-Create `offload.env` in the Offload Home, this file contains the necessary configuration specific to your environment:
+## Configuration File
+Create `offload.env` in the Offload Home. This file contains the necessary configuration specific to your environment:
 ```
 cp ${OFFLOAD_HOME}/conf/oracle-bigquery-offload.env.template ${OFFLOAD_HOME}/conf/offload.env
 vi ${OFFLOAD_HOME}/conf/offload.env
@@ -59,7 +64,7 @@ If using Dataproc Batches to provide Spark:
 - GOOGLE_DATAPROC_REGION
 - GOOGLE_DATAPROC_BATCHES_SUBNET
 
-## Install database objects
+## Install Database Objects
 To install supporting database objects you need access to a database admin account that can create users, grant them system privileges and create objects in the schemas created. SYSTEM has been used in the example below but this is *not* a necessity:
 ```
 cd ${OFFLOAD_HOME}/setup
@@ -73,11 +78,11 @@ alter user goe_adm identified by ...;
 alter user goe_app identified by ...;
 ```
 
-# Building a custom package
+# Building a Custom Package
 
-If you want to test with the latest commits that have not yet been included in a Github release you can build a custom package from the repository.
+If you want to test with the latest commits that have not yet been included in a GitHub release you can build a custom package from the repository.
 
-## Prepare the host/cloned repository
+## Prepare the Host/Cloned Repository
 Debian prerequisites:
 ```
 sudo apt-get -y install make python3-venv
@@ -96,13 +101,13 @@ curl -fL https://github.com/coursier/coursier/releases/latest/download/cs-x86_64
 . ~/.bash_profile
 ```
 
-## Make a GOE package
+## Make a GOE Package
 To create a package which contains all required artifacts for running GOE commands use the `make` target below:
 ```
 make clean && make package
 ```
 
-# Install for development
+# Install for Development
 To create a Python virtual environment and install all required dependencies into the repository directory:
 ```
 make clean && make install-dev
@@ -117,7 +122,10 @@ Note only the Python dependencies for Oracle and BigQuery are installed by defau
 make install-dev-extras
 ```
 
-# Running commands
+# Supporting Infrastructure
+GOE requires access to cloud storage and Spark. For a Google Cloud installation this is described here: [Google Cloud Platform Setup for a BigQuery Target](docs/gcp_setup.md)
+
+# Running Commands
 Activate the GOE Python virtual environment:
 ```
 source ./.venv/bin/activate

{goe_framework-1.0.6 → goe_framework-1.0.6.dev0}/pyproject.toml

@@ -14,7 +14,7 @@
 
 [project]
 name = "goe-framework"
-version = "1.0.6"
+version = "1.0.6.dev0"
 readme = "README.md"
 requires-python = ">=3.7"
 maintainers = [

{goe_framework-1.0.6 → goe_framework-1.0.6.dev0}/src/goe/config/config_validation_functions.py

@@ -22,13 +22,6 @@ import random
 import re
 
 from goe.config import orchestration_defaults
-from goe.data_governance.hadoop_data_governance_constants import (
-    SUPPORTED_DATA_GOVERNANCE_BACKENDS,
-)
-from goe.data_governance.hadoop_data_governance import (
-    is_valid_data_governance_tag,
-    is_valid_data_governance_auto_property,
-)
 from goe.filesystem.goe_dfs import (
     AZURE_OFFLOAD_FS_SCHEMES,
     OFFLOAD_FS_SCHEME_MAPRFS,
@@ -188,51 +181,6 @@ def normalise_backend_session_parameters(opts):
         opts.backend_session_parameters = json.loads(opts.backend_session_parameters)
 
 
-def normalise_data_governance_config(options):
-    if options.data_governance_api_url:
-        if not options.data_governance_backend:
-            raise OrchestrationConfigException(
-                "DATA_GOVERNANCE_BACKEND is required when connecting to a Data Governance API"
-            )
-        if options.data_governance_backend not in SUPPORTED_DATA_GOVERNANCE_BACKENDS:
-            raise OrchestrationConfigException(
-                "Invalid DATA_GOVERNANCE_BACKEND %s. Valid values are: %s"
-                % (options.data_governance_backend, SUPPORTED_DATA_GOVERNANCE_BACKENDS)
-            )
-        if (
-            options.data_governance_backend == "navigator"
-            and not options.cloudera_navigator_hive_source_id
-        ):
-            raise OrchestrationConfigException(
-                "--cloudera-navigator-hive-source-id is required for Cloudera Navigator integration"
-            )
-
-        tag_list = (
-            options.data_governance_auto_tags_csv.split(",")
-            if options.data_governance_auto_tags_csv
-            else []
-        )
-        invalid_auto_tags = [_ for _ in tag_list if not is_valid_data_governance_tag(_)]
-        if invalid_auto_tags:
-            # Using env var name for exception because the command line option is hidden
-            raise OrchestrationConfigException(
-                "Invalid values for DATA_GOVERNANCE_AUTO_TAGS: %s" % invalid_auto_tags
-            )
-
-        auto_properties = (
-            options.data_governance_auto_properties_csv.split(",")
-            if options.data_governance_auto_properties_csv
-            else []
-        )
-        invalid_auto_props = [
-            _ for _ in auto_properties if not is_valid_data_governance_auto_property(_)
-        ]
-        # TODO nj@2020-01-31 It feels like we should add the lines below but need to investigate/test before we do
-        # if invalid_auto_props:
-        # raise OrchestrationConfigException('Invalid values for DATA_GOVERNANCE_AUTO_PROPERTIES: %s'
-        #                                    % invalid_auto_props)
-
-
 def normalise_filesystem_options(options, exc_cls=OrchestrationConfigException):
     options.offload_fs_scheme = (
         options.offload_fs_scheme.lower()

{goe_framework-1.0.6 → goe_framework-1.0.6.dev0}/src/goe/config/orchestration_config.py

@@ -28,7 +28,6 @@ from goe.config.config_validation_functions import (
     check_offload_fs_scheme_supported_in_backend,
     normalise_backend_session_parameters,
     normalise_backend_options,
-    normalise_data_governance_config,
     normalise_db_prefix_and_paths,
     normalise_filesystem_options,
     normalise_listener_options,
@@ -67,16 +66,9 @@ EXPECTED_CONFIG_ARGS = [
     "bigquery_dataset_location",
     "bigquery_dataset_project",
     "ca_cert",
-    "cloudera_navigator_hive_source_id",
     "db_name_pattern",
     "db_name_prefix",
     "db_type",
-    "data_governance_api_url",
-    "data_governance_api_user",
-    "data_governance_api_pass",
-    "data_governance_backend",
-    "data_governance_auto_tags_csv",
-    "data_governance_auto_properties_csv",
     "dev_log_level",
     "error_on_token",
     "frontend_odbc_driver_name",
@@ -311,7 +303,6 @@ class OrchestrationConfig:
         self._do_not_connect = do_not_connect
 
         normalise_backend_session_parameters(self)
-        normalise_data_governance_config(self)
         normalise_filesystem_options(self)
         try:
             normalise_backend_options(self)
@@ -382,34 +373,6 @@ class OrchestrationConfig:
             ca_cert=config_dict.get(
                 "ca_cert", orchestration_defaults.ca_cert_default()
             ),
-            cloudera_navigator_hive_source_id=config_dict.get(
-                "cloudera_navigator_hive_source_id",
-                orchestration_defaults.cloudera_navigator_hive_source_id_default(),
-            ),
-            data_governance_api_url=config_dict.get(
-                "data_governance_api_url",
-                orchestration_defaults.data_governance_api_url_default(),
-            ),
-            data_governance_api_user=config_dict.get(
-                "data_governance_api_user",
-                orchestration_defaults.data_governance_api_user_default(),
-            ),
-            data_governance_api_pass=config_dict.get(
-                "data_governance_api_pass",
-                orchestration_defaults.data_governance_api_pass_default(),
-            ),
-            data_governance_backend=config_dict.get(
-                "data_governance_backend",
-                orchestration_defaults.data_governance_api_backend_default(),
-            ),
-            data_governance_auto_tags_csv=config_dict.get(
-                "data_governance_auto_tags_csv",
-                orchestration_defaults.data_governance_auto_tags_default(),
-            ),
-            data_governance_auto_properties_csv=config_dict.get(
-                "data_governance_auto_properties_csv",
-                orchestration_defaults.data_governance_auto_properties_default(),
-            ),
             db_name_pattern=config_dict.get(
                 "db_name_pattern", orchestration_defaults.db_name_pattern_default()
             ),

{goe_framework-1.0.6 → goe_framework-1.0.6.dev0}/src/goe/config/orchestration_defaults.py

@@ -901,45 +901,6 @@ def sqoop_queue_name_default() -> Optional[str]:
     return os.environ.get("SQOOP_QUEUE_NAME")
 
 
-# DATA GOVERNANCE DEFAULTS
-
-
-def data_governance_api_url_default():
-    return os.environ.get("DATA_GOVERNANCE_API_URL")
-
-
-def data_governance_api_user_default():
-    return os.environ.get("DATA_GOVERNANCE_API_USER")
-
-
-def data_governance_api_pass_default():
-    return os.environ.get("DATA_GOVERNANCE_API_PASS")
-
-
-def data_governance_api_backend_default():
-    return os.environ.get("DATA_GOVERNANCE_BACKEND")
-
-
-def cloudera_navigator_hive_source_id_default():
-    return os.environ.get("CLOUDERA_NAVIGATOR_HIVE_SOURCE_ID")
-
-
-def data_governance_auto_tags_default():
-    return os.environ.get("DATA_GOVERNANCE_AUTO_TAGS")
-
-
-def data_governance_custom_tags_default():
-    return os.environ.get("DATA_GOVERNANCE_CUSTOM_TAGS")
-
-
-def data_governance_auto_properties_default():
-    return os.environ.get("DATA_GOVERNANCE_AUTO_PROPERTIES")
-
-
-def data_governance_custom_properties_default():
-    return os.environ.get("DATA_GOVERNANCE_CUSTOM_PROPERTIES")
-
-
 ###########################################################################
 # GOE LISTENER DEFAULTS
 ###########################################################################

{goe_framework-1.0.6 → goe_framework-1.0.6.dev0}/src/goe/goe.py

@@ -137,11 +137,6 @@ from goe.persistence.orchestration_metadata import (
     INCREMENTAL_PREDICATE_TYPE_RANGE_AND_PREDICATE,
 )
 
-from goe.data_governance.hadoop_data_governance import (
-    get_hadoop_data_governance_client_from_options,
-    is_valid_data_governance_tag,
-)
-
 from goe.util.goe_log_fh import GOELogFileHandle
 from goe.util.misc_functions import (
     all_int_chars,
@@ -185,8 +180,6 @@ EXPECTED_OFFLOAD_ARGS = [
     "compress_load_table",
     "compute_load_table_stats",
     "create_backend_db",
-    "data_governance_custom_tags_csv",
-    "data_governance_custom_properties",
     "data_sample_parallelism",
     "data_sample_pct",
     "date_columns_csv",
@@ -1105,7 +1098,6 @@ def normalise_options(options, normalise_owner_table=True):
     normalise_offload_predicate_options(options)
     normalise_datatype_control_options(options)
     normalise_stats_options(options, options.target)
-    normalise_data_governance_options(options)
 
 
 def verify_json_option(option_name, option_value):
@@ -1136,25 +1128,7 @@ def verify_json_option(option_name, option_value):
             )
 
 
-def normalise_data_governance_options(options):
-    tag_list = (
-        options.data_governance_custom_tags_csv.split(",")
-        if options.data_governance_custom_tags_csv
-        else []
-    )
-    invalid_custom_tags = [_ for _ in tag_list if not is_valid_data_governance_tag(_)]
-    if invalid_custom_tags:
-        raise OffloadOptionError(
-            "Invalid values for --data-governance-custom-tags: %s" % invalid_custom_tags
-        )
-
-    verify_json_option(
-        "--data-governance-custom-properties", options.data_governance_custom_properties
-    )
-
-
 def version():
-    """Note that this function is modified in the top level Makefile"""
     with open(
         os.path.join(os.environ.get("OFFLOAD_HOME"), "version_build")
     ) as version_file:
@@ -1169,7 +1143,7 @@ def comp_ver_check(frontend_api):
 
 def version_abort(check_version, frontend_api):
     match, v_goe, v_ora = comp_ver_check(frontend_api)
-    if check_version and not match and "-DEV" not in v_goe:
+    if check_version and not match and ".dev" not in v_goe:
         return True, v_goe, v_ora
     else:
         return False, v_goe, v_ora
@@ -2035,17 +2009,17 @@ class BaseOperation(object):
 
     def validate_sort_columns(
         self,
-        rdbms_column_names,
-        messages,
-        offload_options,
+        offload_source_table: OffloadSourceTableInterface,
+        messages: OffloadMessages,
+        offload_options: "OrchestrationConfig",
         backend_cols,
         hybrid_metadata,
         backend_api=None,
-        metadata_refresh=False,
     ):
         """Default sort_columns for storage index benefit if not specified by the user.
-        sort_columns_csv: The incoming option string which can be a CSV list of column names or the special token
-            SORT_COLUMNS_NO_CHANGE which identifies the user has not asked for a change.
+        sort_columns_csv: The incoming option string which can be a CSV list of column names or:
+            - the special token SORT_COLUMNS_NO_CHANGE which identifies the user has not asked for a change.
+            - the special token SORT_COLUMNS_NONE which identifies the user has not asked for no sort columns.
         sort_columns: A Python list of column names defined by the user.
         backend_cols: A standalone parameter because this function may be used on tables that do not yet exist.
         """
@@ -2083,10 +2057,9 @@ class BaseOperation(object):
             self.sort_columns = sort_columns_csv_to_sort_columns(
                 self.sort_columns_csv,
                 hybrid_metadata,
-                rdbms_column_names,
+                offload_source_table,
                 backend_cols,
                 backend_api,
-                metadata_refresh,
                 messages,
             )
         finally:
@@ -2225,8 +2198,6 @@ class OffloadOperation(BaseOperation):
             compress_load_table=options.compress_load_table,
             compute_load_table_stats=options.compute_load_table_stats,
             create_backend_db=options.create_backend_db,
-            data_governance_custom_tags_csv=options.data_governance_custom_tags_csv,
-            data_governance_custom_properties=options.data_governance_custom_properties,
             data_sample_parallelism=options.data_sample_parallelism,
             data_sample_pct=options.data_sample_pct,
             date_columns_csv=options.date_columns_csv,
@@ -2347,14 +2318,6 @@ class OffloadOperation(BaseOperation):
             create_backend_db=operation_dict.get(
                 "create_backend_db", orchestration_defaults.create_backend_db_default()
             ),
-            data_governance_custom_tags_csv=operation_dict.get(
-                "data_governance_custom_tags_csv",
-                orchestration_defaults.data_governance_custom_tags_default(),
-            ),
-            data_governance_custom_properties=operation_dict.get(
-                "data_governance_custom_properties",
-                orchestration_defaults.data_governance_custom_properties_default(),
-            ),
             data_sample_parallelism=operation_dict.get(
                 "data_sample_parallelism",
                 orchestration_defaults.data_sample_parallelism_default(),
@@ -2657,7 +2620,7 @@ def offload_operation_logic(
                 )
 
     offload_operation.validate_sort_columns(
-        offload_source_table.get_column_names(),
+        offload_source_table,
         messages,
         offload_options,
         offload_target_table.get_columns(),
@@ -2841,13 +2804,6 @@ def offload_table(
     ):
         return False
 
-    data_gov_client = get_data_gov_client(
-        offload_options,
-        messages,
-        offload_operation,
-        rdbms_schema=offload_source_table.owner,
-        source_rdbms_object_name=offload_source_table.table_name,
-    )
     offload_operation.offload_transport_method = choose_offload_transport_method(
         offload_operation, offload_source_table, offload_options, messages
     )
@@ -2859,7 +2815,6 @@ def offload_table(
     offload_target_table.refresh_operational_settings(
         offload_operation,
         rdbms_columns=offload_source_table.columns,
-        data_gov_client=data_gov_client,
     )
 
     if offload_operation.ddl_file:
@@ -2921,7 +2876,6 @@ def offload_table(
         offload_options,
         source_data_client,
         messages,
-        data_gov_client,
     )
     messages.log(
         "%s: %s"
@@ -3016,33 +2970,6 @@ def get_synthetic_partition_cols(backend_cols):
     return [col for col in backend_cols if is_synthetic_partition_column(col.name)]
 
 
-def get_data_gov_client(
-    options,
-    messages,
-    execute,
-    rdbms_schema=None,
-    source_rdbms_object_name=None,
-    target_rdbms_object_name=None,
-):
-    if options.data_governance_api_url:
-        data_gov_client = get_hadoop_data_governance_client_from_options(
-            options, messages, dry_run=bool(not execute)
-        )
-        data_gov_client.healthcheck_api()
-        data_gov_client.cache_property_values(
-            auto_tags_csv=options.data_governance_auto_tags_csv,
-            custom_tags_csv=options.data_governance_custom_tags_csv,
-            auto_properties_csv=options.data_governance_auto_properties_csv,
-            custom_properties=options.data_governance_custom_properties,
-            rdbms_name=get_db_unique_name(options),
-            rdbms_schema=rdbms_schema,
-            source_rdbms_object_name=source_rdbms_object_name,
-            target_rdbms_object_name=target_rdbms_object_name,
-        )
-        return data_gov_client
-    return None
-
-
 def check_posint(option, opt, value):
     """Type checker for a new "posint" type to define
     and enforce positive integer option values. Used
@@ -3204,21 +3131,6 @@ def get_oracle_options(opt):
     return opt
 
 
-def get_data_governance_options(opt):
-    opt.add_option(
-        "--data-governance-custom-properties",
-        dest="data_governance_custom_properties",
-        default=orchestration_defaults.data_governance_custom_properties_default(),
-        help=SUPPRESS_HELP,
-    )
-    opt.add_option(
-        "--data-governance-custom-tags",
-        dest="data_governance_custom_tags_csv",
-        default=orchestration_defaults.data_governance_custom_tags_default(),
-        help=SUPPRESS_HELP,
-    )
-
-
 def get_options(usage=None, operation_name=None):
     opt = get_common_options(usage)
 
@@ -3450,7 +3362,7 @@ def get_options(usage=None, operation_name=None):
         "--sort-columns",
         dest="sort_columns_csv",
         default=orchestration_defaults.sort_columns_default(),
-        help="CSV list of sort/cluster columns to use when storing data in a backend table",
+        help=f'CSV list of sort/cluster columns to use when storing data in a backend table or "{offload_constants.SORT_COLUMNS_NONE}" to force no sort columns',
     )
     opt.add_option(
         "--offload-distribute-enabled",
@@ -3500,8 +3412,6 @@ def get_options(usage=None, operation_name=None):
         help=SUPPRESS_HELP,
     )
 
-    get_data_governance_options(opt)
-
     return opt
 
 

{goe_framework-1.0.6 → goe_framework-1.0.6.dev0}/src/goe/listener/schemas/orchestration.py

@@ -205,22 +205,6 @@ class OffloadOptions(BaseSchema):
         description="Create backend database",
         cli=("--create-backend-db"),
     )
-    # TODO: Deprecated
-    data_governance_custom_properties: Optional[Json] = Field(
-        default=None,
-        title="Data governance custom properties",
-        description="Data governance custom properties",
-        cli=("--data-governance-custom-properties"),
-        no_api=True,
-    )
-    # TODO: Deprecated
-    data_governance_custom_tags_csv: Optional[List[str]] = Field(
-        default=[],
-        title="Data governance custom tags",
-        description="Data governance custom tags",
-        cli=("--data-governance-custom-tags"),
-        no_api=True,
-    )
     data_sample_parallelism: Optional[int] = Field(
         default=0,
         title="Data sample parallelism",

{goe_framework-1.0.6 → goe_framework-1.0.6.dev0}/src/goe/offload/backend_api.py

@@ -1973,6 +1973,10 @@ FROM   %(db)s.%(table)s%(where_clause)s%(group_by)s%(order_by)s""" % {
     def create_database_supported(self):
         return self.is_capability_supported(CAPABILITY_CREATE_DB)
 
+    def default_sort_columns_to_primary_key(self) -> bool:
+        """Does this backend suit clustering on primary key if there's no user specified alternative."""
+        return False
+
     def drop_column_supported(self):
         """Note that there is an Impala override for this"""
         return self.is_capability_supported(CAPABILITY_DROP_COLUMN)