streamsets 6.6.0rc1__tar.gz → 6.6.2.dev0__tar.gz

{streamsets-6.6.0rc1 → streamsets-6.6.2.dev0}/HISTORY.rst

@@ -3,6 +3,32 @@
 
 History
 =======
+6.6.1 (June 2025)
+----------------------
+* Fixed an issue that caused an error to be raised when calling the method :py:meth:`streamsets.sdk.ControlHub.delete_deployment`.
+
+* Bug fixes and improvements
+
+6.6.0 (June 2025)
+-----------------
+* Added support for interacting with network access rules
+
+    * Added the ability to modify whether network access rules are enabled for your organization using the :py:attr:`streamsets.sdk.sch_models.Organization.ip_auth_enabled` attribute.
+
+    * Added the ability to modify network access rules for your organization using the :py:meth:`streamsets.sdk.sch_models.Organization.add_ip_auth_ruleset` and :py:meth:`streamsets.sdk.sch_models.Organization.remove_ip_auth_ruleset`.
+
+    * Added the ability to view the current network access rules using the :py:attr:`streamsets.sdk.sch_models.Organization.ip_auth_rules` attribute.
+
+* Added functionality to check the possible values of a stage configuration by using the :py:meth:`streamsets.sdk.sch_models.PipelineBuilder.get_stage_configuration_options` method.
+
+* Added support for interacting with Projects
+
+    * Added SDK support for projects feature by being able to switch between different projects using the :py:attr:`streamsets.sdk.ControlHub.current_project` attribute.
+
+    * Added SDK support to list and modify projects by using the :py:attr:`streamsets.sdk.ControlHub.projects` attribute and the methods :py:meth:`streamsets.sdk.ControlHub.add_project` and :py:meth:`streamsets.sdk.ControlHub.delete_project`.
+
+* Bug fixes and improvements.
+
 6.5.0 (November 2024)
 -----------------
 * The :py:class:`streamsets.sdk.sch_models.Step` class now supports Finish Conditions and has a new attribute ``name`` which will allow retrieving and setting the name of a step.

streamsets-6.6.2.dev0/PKG-INFO

@@ -0,0 +1,66 @@
+Metadata-Version: 2.1
+Name: streamsets
+Version: 6.6.2.dev0
+Summary: A Python SDK for StreamSets
+Author: StreamSets Inc
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Programming Language :: Python :: 3.4
+Classifier: Programming Language :: Python :: 3.5
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.4
+Description-Content-Type: text/x-rst
+Requires-Dist: dpath==1.5.0
+Requires-Dist: inflection
+Requires-Dist: PyYAML
+Requires-Dist: requests
+
+=========================
+StreamSets SDK for Python
+=========================
+
+|Version| |Python|
+
+.. |Python| image:: https://shields.io/pypi/pyversions/streamsets.svg?style=flat
+    :target: https://pypi.python.org/pypi/streamsets/
+    :alt: Python Versions
+.. |Version| image:: http://img.shields.io/pypi/v/streamsets.svg?style=flat
+    :target: https://pypi.python.org/pypi/streamsets/
+    :alt: Package Version
+
+The StreamSets SDK allows users to interact with StreamSets products programmatically using Python, including Control Hub, Data Collector, Transformer, and Transformer for Snowflake.
+
+Notices
+-------
+Versions 4.0.0 and higher of the StreamSets SDK for Python support the `StreamSets Platform <https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/GettingStarted/GettingStarted_title.html>`_.
+Versions below 4.0.0 support our `Legacy products <https://docs.streamsets.com/portal/controlhub/latest/help/controlhub/UserGuide/GettingStarted/DPM.html#concept_l45_qwf_xw>`_.
+Please verify that the version of the SDK you're using is compatible with the StreamSets products you plan to interact with.
+
+Documentation
+-------------
+.. _documentation:
+
+For documentation on the StreamSets SDK for Platform, refer to our `Platform SDK documentation <https://docs.streamsets.com/platform-sdk/latest/index.html>`_.
+For documentation on the StreamSets SDK for Legacy products, refer to our `Legacy SDK documentation <https://docs.streamsets.com/sdk/latest/index.html>`_.
+
+More Resources
+--------------
+You can find training and additional resources on the `StreamSets Academy <https://academy.streamsets.com/learn/courses/45/streamsets-sdk-for-python>`_.
+
+You can also find a wealth of knowledge on the `StreamSets Community Forums <https://community.streamsets.com>`_ including product updates, workshops, events and webinars, and more!
+
+Getting Help
+------------
+Running into issues while using the StreamSets SDK?
+Get assistance from `StreamSets Support <https://www.ibm.com/mysupport>`_!
+
+Release Notes
+-------------
+Release notes for the Platform SDK can be found in the `Platform SDK release notes <https://docs.streamsets.com/platform-sdk/latest/welcome/release_notes.html>`_.
+
+Release notes for the Legacy SDK can be found in the `Legacy SDK release notes <https://docs.streamsets.com/sdk/latest/history.html>`_.

{streamsets-6.6.0rc1 → streamsets-6.6.2.dev0}/README.rst

@@ -23,7 +23,6 @@ Documentation
 -------------
 .. _documentation:
 
-You can find the latest, most updated documentation on our `documentation portal <https://docs.streamsets.com/>`_, including how to get started with the StreamSets SDK for Python.
 For documentation on the StreamSets SDK for Platform, refer to our `Platform SDK documentation <https://docs.streamsets.com/platform-sdk/latest/index.html>`_.
 For documentation on the StreamSets SDK for Legacy products, refer to our `Legacy SDK documentation <https://docs.streamsets.com/sdk/latest/index.html>`_.
 
@@ -36,10 +35,10 @@ You can also find a wealth of knowledge on the `StreamSets Community Forums <htt
 Getting Help
 ------------
 Running into issues while using the StreamSets SDK?
-Get assistance from `StreamSets Support <https://streamsets.com/support/>`_!
+Get assistance from `StreamSets Support <https://www.ibm.com/mysupport>`_!
 
 Release Notes
 -------------
-Release notes for the Platform SDK can be found in the `Platform SDK release notes <https://docs.streamsets.com/platform-sdk/latest/learn/release_notes.html>`_.
+Release notes for the Platform SDK can be found in the `Platform SDK release notes <https://docs.streamsets.com/platform-sdk/latest/welcome/release_notes.html>`_.
 
 Release notes for the Legacy SDK can be found in the `Legacy SDK release notes <https://docs.streamsets.com/sdk/latest/history.html>`_.

{streamsets-6.6.0rc1 → streamsets-6.6.2.dev0}/docs/build/creating_pipelines.rst

@@ -12,6 +12,7 @@ for pipelines.
 
 Instantiating a Pipeline Builder
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. _instantiate_builder:
 
 In the UI, a pipeline can be created and modified from the Pipelines section as seen below:
 

{streamsets-6.6.0rc1 → streamsets-6.6.2.dev0}/docs/build/editing_pipelines.rst

@@ -216,7 +216,8 @@ Setting the configuration value is as simple as directly setting the value in-me
 .. note::
   The same workflow can be followed to access/edit configuration values of :py:class:`streamsets.sdk.sch_models.Pipeline` objects
 
-Some configuration values have a predefined set of options to choose from. You can get a list of these using the :py:meth:`streamsets.sdk.sch.PipelineBuilder.get_stage_configuration_options` method, as shown below:
+Some configuration values have a predefined set of options to choose from.
+You can get a list of these using the :py:meth:`streamsets.sdk.sch_models.PipelineBuilder.get_stage_configuration_options` method, as shown below:
 
 .. code-block:: python
 
@@ -235,6 +236,10 @@ Some configuration values have a predefined set of options to choose from. You c
 The list will match the values shown in the `JSON Content` UI dropdown.
 
 .. image:: ../_static/images/build/value_chooser__JSON_content.png
+
+.. tip::
+    Refer to the :ref:`Instantiating a Pipeline Builder<instantiate_builder>` section for creating a Pipeline Builder instance.
+
 |
 |
 Once you have edited your :py:class:`streamsets.sdk.sch_models.Pipeline` or :py:class:`streamsets.sdk.sch_models.SchSdcStage`, the changes must be published to Control Hub.

{streamsets-6.6.0rc1 → streamsets-6.6.2.dev0}/docs/usage/set_up/connections/connections.rst

@@ -59,7 +59,7 @@ The parameters you'll need to provide for the Connection are ``title``, ``connec
 
 .. tip::
   The available options for the ``connection_type`` in the :py:meth:`streamsets.sdk.sch_models.ConnectionBuilder.build` method are:
-  ``'STREAMSETS_AWS_EMR_CLUSTER'``, ``'STREAMSETS_MYSQL'``, ``'STREAMSETS_SNOWFLAKE'``, ``'STREAMSETS_COAP_CLIENT'``, ``'STREAMSETS_OPC_UA_CLIENT'``, ``'STREAMSETS_GOOGLE_PUB_SUB'``, ``'STREAMSETS_MQTT'``, ``'STREAMSETS_POSTGRES'``, ``'STREAMSETS_GOOGLE_CLOUD_STORAGE'``, ``'STREAMSETS_AWS_REDSHIFT'``, ``'STREAMSETS_GOOGLE_BIG_QUERY'``, ``'STREAMSETS_ORACLE'``, ``'STREAMSETS_AWS_S3'``, ``'STREAMSETS_REMOTE_FILE'``, ``'STREAMSETS_SQLSERVER'``, ``'STREAMSETS_AWS_SQS'``, ``'STREAMSETS_SNOWPIPE'``, and ``'STREAMSETS_JDBC'``.
+  ``'STREAMSETS_MYSQL'``, ``'STREAMSETS_AZURE_SYNAPSE'``, ``'STREAMSETS_ORCHESTRATOR'``, ``'STREAMSETS_INFLUX2'``, ``'STREAMSETS_DATABRICKS_DELTA_LAKE'``, ``'STREAMSETS_HIVE'``, ``'STREAMSETS_GOOGLE_PUB_SUB'``, ``'STREAMSETS_BLOB_STORAGE'``, ``'STREAMSETS_MQTT'``, ``'STREAMSETS_MONGODB'``, ``'STREAMSETS_POSTGRES'``, ``'STREAMSETS_GOOGLE_CLOUD_STORAGE'``, ``'STREAMSETS_CASSANDRA'``, ``'STREAMSETS_JMS'``, ``'STREAMSETS_ELASTICSEARCH'``, ``'STREAMSETS_ORACLE'``, ``'STREAMSETS_DB2'``, ``'STREAMSETS_ADLS_GEN2'``, ``'STREAMSETS_TERADATA'``, ``'STREAMSETS_WEBCLIENT'``, ``'STREAMSETS_AEROSPIKE'``, ``'STREAMSETS_SNOWPIPE'``, ``'STREAMSETS_AWS_EMR_CLUSTER'``, ``'STREAMSETS_SNOWFLAKE'``, ``'STREAMSETS_SPLUNK'``, ``'STREAMSETS_COAP_CLIENT'``, ``'STREAMSETS_OPC_UA_CLIENT'``, ``'STREAMSETS_COUCHBASE'``, ``'STREAMSETS_MONGODB_ATLAS'``, ``'STREAMSETS_WATSONXDATA'``, ``'STREAMSETS_AWS_KINESIS_STREAM'``, ``'STREAMSETS_AWS_KINESIS_FIREHOSE'``, ``'STREAMSETS_KUDU'``, ``'STREAMSETS_PULSAR'``, ``'STREAMSETS_AWS_REDSHIFT'``, ``'STREAMSETS_GOOGLE_BIG_QUERY'``, ``'STREAMSETS_REMOTE_FILE'``, ``'STREAMSETS_AWS_S3'``, ``'STREAMSETS_KAFKA'``, ``'STREAMSETS_SQLSERVER'``, ``'STREAMSETS_AWS_SQS'``, ``'STREAMSETS_AWS_EMR_SERVERLESS'``, ``'STREAMSETS_CONNX'``, ``'STREAMSETS_RABBITMQ'``, ``'STREAMSETS_SALESFORCE'``, ``'STREAMSETS_REDIS'``, ``'STREAMSETS_JDBC'``.
 
 Please refer to the `StreamSets Platform documentation <https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/Connections/Connections_title.html>`_ for details on the various connection types as well as the corresponding configuration properties.
 

{streamsets-6.6.0rc1 → streamsets-6.6.2.dev0}/pyproject.toml

@@ -4,7 +4,7 @@ authors = [
     {name = "StreamSets Inc"}
 ]
 description = "A Python SDK for StreamSets"
-readme = {file = "README.txt", content-type = "text/x-rst"}
+readme = {file = "README.rst", content-type = "text/x-rst"}
 requires-python = ">=3.4"
 classifiers=[
     'Development Status :: 5 - Production/Stable',

{streamsets-6.6.0rc1 → streamsets-6.6.2.dev0}/requirements.txt

@@ -7,4 +7,4 @@ Cython==0.29.30
 inflection==0.3.1
 javaproperties==0.4.0
 PyYAML==5.4.1
-requests==2.32.2
+requests==2.32.4

streamsets-6.6.2.dev0/streamsets/sdk/__version__.py

@@ -0,0 +1,5 @@
+#  IBM Confidential
+#  PID 5900-BAF
+#  Copyright StreamSets Inc., an IBM Company 2025
+
+__version__ = "6.6.2dev0"

{streamsets-6.6.0rc1 → streamsets-6.6.2.dev0}/streamsets/sdk/sch.py

@@ -4243,7 +4243,7 @@ class ControlHub:
         delete_deployment_exception = None
         for deployment in deployments:
             try:
-                self.api_client.wait_for_deployment_to_be_deleted(deployment_id=deployment.id)
+                self.api_client.wait_for_deployment_to_be_deleted(deployment_id=deployment.deployment_id)
             except Exception as ex:
                 # just log the exceptions and ultimately raise the last one.
                 logger.debug(ex)

{streamsets-6.6.0rc1 → streamsets-6.6.2.dev0}/streamsets/sdk/sch_models.py

@@ -70,7 +70,7 @@ ModelCollectionResults = collections.namedtuple('ModelCollectionResults', ['resu
 CollectionModelResults = collections.namedtuple(
     'CollectionModelResults', ['results', 'kwargs', 'class_type', 'class_kwargs']
 )
-NameMap = namedtuple('NameMap', ['label', 'fieldName', 'name'])
+NameMap = namedtuple('NameMap', ['label', 'fieldName', 'name', 'sdk'])
 ChooseOption = namedtuple('ChooseOption', ['UI_VALUE', 'SDK_VALUE'])
 
 MILLIS_IN_HOUR = 3600000
@@ -404,8 +404,8 @@ class CollectionModel:
         if kwargs.get('offset') and not (isinstance(kwargs.get('offset'), int) and kwargs.get('offset') >= 0):
             raise ValueError("`offset` should be an integer with value of at least 0")
 
+        is_api_paginable = lambda rsp: all(i in rsp for i in ['data', 'totalCount', 'offset', 'len'])
         all_ids = set()
-        previous_length = len(self)
         requested_length = kwargs.get('len', float('inf'))
         if requested_length == -1:
             requested_length = float('inf')
@@ -416,12 +416,15 @@ class CollectionModel:
         kwargs_without_pagination_params = {
             k: v for k, v in kwargs.items() if k not in CollectionModel.PAGINATION_PARAMS
         }
+        logger.debug('Fetching items with offset=%d and len=%d', current_offset, page_length)
         response, current_new_kwargs, class_type, class_kwargs = self._get_all_results_from_api(
             offset=current_offset, len=page_length, **kwargs_without_pagination_params
         )
+        previous_length = response['totalCount'] if is_api_paginable(response) else len(self)
+
         # If an API is paginated, the result set we need to page through will be contained in the response's 'data'
         # attribute. If there is no 'data' attribute, the response itself is the result set (unpaginated API).
-        if 'data' in response:
+        if is_api_paginable(response):
             current_results = SeekableList(class_type(item, **class_kwargs) for item in response['data'])
         else:
             current_results = SeekableList(class_type(item, **class_kwargs) for item in response)
@@ -432,17 +435,18 @@ class CollectionModel:
             # ControlHub api endpoints)
             current_results = current_results.get_all(**current_new_kwargs)
             for result in current_results:
-                if len(all_ids) == requested_length:
-                    # We fetched specified number of items so, return
-                    return
                 # This check is to avoid duplicates especially since we are doing the
                 # if current_length < previous_length check below to handle deleted entities.
                 item_id = getattr(result, self._id_attr)
                 if item_id not in all_ids:
                     all_ids.add(item_id)
                     yield result
+
+            if len(all_ids) >= requested_length:
+                return
+
             current_offset += page_length
-            current_length = len(self)
+            current_length = response['totalCount'] if is_api_paginable(response) else len(self)
             # If the total number of items decreased, reduce the offset by the difference to make sure we return all the
             # items. If duplicates occur in the process as described in step 5 of workflow in the docstring are handled
             # above by checking for the id in all_ids.
@@ -2775,22 +2779,26 @@ class PipelineBuilder(SdcPipelineBuilder):
 
         # add simple field config
         for field_config in raw_stage.definition['configDefinitions']:
+            sdk_name, a = get_attribute(field_config)
             names.append(
                 NameMap(
                     label=field_config['label'],
                     fieldName=field_config['fieldName'],
                     name=field_config['name'],
+                    sdk=sdk_name,
                 )
             )
 
         # add services fields config (with full name <service_name>.<field_name>) - this name is probably invisible for the end user
         for service in raw_stage.definition['services']:
             for field_config in service_definitions[service['service']]['configDefinitions']:
+                sdk_name, a = get_attribute(field_config)
                 names.append(
                     NameMap(
                         label=field_config['label'],
                         fieldName=field_config['fieldName'],
                         name=field_config['name'],
+                        sdk=sdk_name,
                     )
                 )
 
@@ -2804,7 +2812,7 @@ class PipelineBuilder(SdcPipelineBuilder):
         Args:
             config_name (:obj:`str`): Name of the configuration field.
             stage_name (:obj:`str`): Name of the stage type (eg. com_streamsets_pipeline_stage_devtest_rawdata_RawDataDSource)
-            config_name_type (:obj:`str`): Type of config_name value. Available options: ['full_name', 'field_name', 'label']. Default ``label``.
+            config_name_type (:obj:`str`): Type of config_name value. Available options: ['full_name', 'field_name', 'label', 'sdk']. Default ``label``.
             stage_name_type (:obj:`str`): Type of stage_name value. Available options: ['label', 'name']. Default ``label``.
 
         Returns:
@@ -2814,7 +2822,7 @@ class PipelineBuilder(SdcPipelineBuilder):
             ValueError: If incorrect value of config_name_type.
             ValueError: If config_name value is not in configuration fields list for stage.
         """
-        ALLOWED_CONFIG_NAME_TYPES = {'full_name': 'name', 'field_name': 'fieldName', 'label': 'label'}
+        ALLOWED_CONFIG_NAME_TYPES = {'full_name': 'name', 'field_name': 'fieldName', 'label': 'label', 'sdk': 'sdk'}
 
         if not isinstance(config_name_type, str) or config_name_type not in ALLOWED_CONFIG_NAME_TYPES.keys():
             raise ValueError(
@@ -3624,6 +3632,150 @@ class Pipeline(BaseModel):
 
         return stage_list
 
+    @property
+    def start_event(self):
+        """Get the Start Event of the Pipeline."""
+        library_definitions = self._library_definitions
+        stage = self._pipeline_definition.get('startEventStages', [{}])[0]
+
+        if not stage:
+            return None
+
+        stage_definition = next(
+            (stage_def for stage_def in library_definitions['stages'] if stage_def['name'] == stage['stageName']),
+            {},
+        )
+
+        stage_class = self._get_stage_class(stage_definition, library_definitions)
+        stage_object = stage_class(stage=stage, pipeline=self)
+
+        return stage_object
+
+    @property
+    def stop_event(self):
+        """Get the Stop Event of the Pipeline."""
+        library_definitions = self._library_definitions
+        stage = self._pipeline_definition.get('stopEventStages', [{}])[0]
+
+        if not stage:
+            return None
+
+        stage_definition = next(
+            (stage_def for stage_def in library_definitions['stages'] if stage_def['name'] == stage['stageName']),
+            {},
+        )
+
+        stage_class = self._get_stage_class(stage_definition, library_definitions)
+        stage_object = stage_class(stage=stage, pipeline=self)
+
+        return stage_object
+
+    @property
+    def test_origin(self):
+        """Get the Test Origin of the Pipeline."""
+        library_definitions = self._library_definitions
+        stage = self._pipeline_definition.get('testOriginStage', [{}])
+
+        if not stage:
+            return None
+
+        stage_definition = next(
+            (stage_def for stage_def in library_definitions['stages'] if stage_def['name'] == stage['stageName']),
+            {},
+        )
+
+        stage_class = self._get_stage_class(stage_definition, library_definitions)
+        stage_object = stage_class(stage=stage, pipeline=self)
+
+        return stage_object
+
+    def set_start_event(self, label=None, name=None, library=None):
+        """Set the Start Event of the Pipeline."""
+
+        # If _library_definitions is not populated, we call the internal _library_definitions property to perform
+        # lazy-loading of the underlying data
+        if not self._data_internal['libraryDefinitions']:
+            self._library_definitions
+
+        try:
+            builder = self._get_builder()
+        # ValueError gets raised by the Engines class when trying to find an inaccessible Engine
+        # However, sometimes ControlHub finds the Inaccessible Engine instance but we then error out later
+        # downstream of the get_pipeline_builder method and throws an InternalServerError
+        except (ValueError, InternalServerError) as e:
+            raise ValueError(
+                'Editing pipeline {} is not supported when the engine is not accessible,'
+                ' please check if the engine is running and try again.'.format(self.pipeline_id)
+            ) from e
+
+        stage = builder.add_start_event_stage(label=label, name=name, library=library)
+
+        # Pass the newly created stage into the Pipeline
+        self._pipeline_definition['startEventStages'] = builder._pipeline[builder._config_key]['startEventStages']
+
+        # Set the value in the Pipeline configuration
+        self.configuration.start_event = self._builder._stage_to_configuration_name(stage._data)
+
+        return stage
+
+    def set_stop_event(self, label=None, name=None, library=None):
+        """Set the Stop Event of the Pipeline."""
+
+        # If _library_definitions is not populated, we call the internal _library_definitions property to perform
+        # lazy-loading of the underlying data
+        if not self._data_internal['libraryDefinitions']:
+            self._library_definitions
+
+        try:
+            builder = self._get_builder()
+        # ValueError gets raised by the Engines class when trying to find an inaccessible Engine
+        # However, sometimes ControlHub finds the Inaccessible Engine instance but we then error out later
+        # downstream of the get_pipeline_builder method and throws an InternalServerError
+        except (ValueError, InternalServerError) as e:
+            raise ValueError(
+                'Editing pipeline {} is not supported when the engine is not accessible,'
+                ' please check if the engine is running and try again.'.format(self.pipeline_id)
+            ) from e
+
+        stage = builder.add_stop_event_stage(label=label, name=name, library=library)
+
+        # Pass the newly created stage into the Pipeline
+        self._pipeline_definition['stopEventStages'] = builder._pipeline[builder._config_key]['stopEventStages']
+
+        # Set the value in the Pipeline configuration
+        self.configuration.stop_event = self._builder._stage_to_configuration_name(stage._data)
+
+        return stage
+
+    def set_test_origin(self, label=None, name=None, library=None):
+        """Set the Test Origin of the Pipeline."""
+
+        # If _library_definitions is not populated, we call the internal _library_definitions property to perform
+        # lazy-loading of the underlying data
+        if not self._data_internal['libraryDefinitions']:
+            self._library_definitions
+
+        try:
+            builder = self._get_builder()
+        # ValueError gets raised by the Engines class when trying to find an inaccessible Engine
+        # However, sometimes ControlHub finds the Inaccessible Engine instance but we then error out later
+        # downstream of the get_pipeline_builder method and throws an InternalServerError
+        except (ValueError, InternalServerError) as e:
+            raise ValueError(
+                'Editing pipeline {} is not supported when the engine is not accessible,'
+                ' please check if the engine is running and try again.'.format(self.pipeline_id)
+            ) from e
+
+        stage = builder.add_test_origin_stage(label=label, name=name, library=library)
+
+        # Pass the newly created stage into the Pipeline
+        self._pipeline_definition['testOriginStage'] = builder._pipeline[builder._config_key]['testOriginStage']
+
+        # Set the value in Pipeline configuration
+        self.configuration.test_origin = self._builder._stage_to_configuration_name(stage._data)
+
+        return stage
+
     @property
     def error_stage(self):
         """Get the error stage of the Pipeline."""
@@ -8933,7 +9085,7 @@ class Engines(CollectionModel):
                     kwargs_defaults['engine_type'] = response['executorType']
                 kwargs_instance = MutableKwargs(kwargs_defaults, kwargs)
                 response = [response]
-            except (requests.exceptions.HTTPError, requests.JSONDecodeError):
+            except (requests.exceptions.HTTPError, requests.exceptions.JSONDecodeError):
                 raise ValueError('Engine (id={}) not found'.format(id))
 
         else:
@@ -12183,11 +12335,19 @@ class ConnectionBuilder:
         Args:
             title (:obj:`str`): Connection title.
             connection_type (:obj:`str`): Type of connection. The options are 'STREAMSETS_AWS_EMR_CLUSTER',
-                'STREAMSETS_MYSQL', 'STREAMSETS_SNOWFLAKE', 'STREAMSETS_COAP_CLIENT', 'STREAMSETS_OPC_UA_CLIENT',
-                'STREAMSETS_GOOGLE_PUB_SUB', 'STREAMSETS_MQTT', 'STREAMSETS_POSTGRES',
-                'STREAMSETS_GOOGLE_CLOUD_STORAGE', 'STREAMSETS_AWS_REDSHIFT', 'STREAMSETS_GOOGLE_BIG_QUERY',
-                'STREAMSETS_ORACLE', 'STREAMSETS_AWS_S3', 'STREAMSETS_REMOTE_FILE', 'STREAMSETS_SQLSERVER',
-                'STREAMSETS_AWS_SQS', 'STREAMSETS_SNOWPIPE', and 'STREAMSETS_JDBC'.
+                'STREAMSETS_MYSQL', 'STREAMSETS_AZURE_SYNAPSE', 'STREAMSETS_ORCHESTRATOR', 'STREAMSETS_INFLUX2',
+                'STREAMSETS_DATABRICKS_DELTA_LAKE', 'STREAMSETS_HIVE', 'STREAMSETS_GOOGLE_PUB_SUB',
+                'STREAMSETS_BLOB_STORAGE', 'STREAMSETS_MQTT', 'STREAMSETS_MONGODB', 'STREAMSETS_POSTGRES',
+                'STREAMSETS_GOOGLE_CLOUD_STORAGE', 'STREAMSETS_CASSANDRA', 'STREAMSETS_JMS', 'STREAMSETS_ELASTICSEARCH',
+                'STREAMSETS_ORACLE', 'STREAMSETS_DB2', 'STREAMSETS_ADLS_GEN2', 'STREAMSETS_TERADATA',
+                'STREAMSETS_WEBCLIENT', 'STREAMSETS_AEROSPIKE', 'STREAMSETS_SNOWPIPE', 'STREAMSETS_AWS_EMR_CLUSTER',
+                'STREAMSETS_SNOWFLAKE', 'STREAMSETS_SPLUNK', 'STREAMSETS_COAP_CLIENT', 'STREAMSETS_OPC_UA_CLIENT',
+                'STREAMSETS_COUCHBASE', 'STREAMSETS_MONGODB_ATLAS', 'STREAMSETS_WATSONXDATA',
+                'STREAMSETS_AWS_KINESIS_STREAM', 'STREAMSETS_AWS_KINESIS_FIREHOSE', 'STREAMSETS_KUDU',
+                'STREAMSETS_PULSAR', 'STREAMSETS_AWS_REDSHIFT', 'STREAMSETS_GOOGLE_BIG_QUERY', 'STREAMSETS_REMOTE_FILE',
+                'STREAMSETS_AWS_S3', 'STREAMSETS_KAFKA', 'STREAMSETS_SQLSERVER', 'STREAMSETS_AWS_SQS',
+                'STREAMSETS_AWS_EMR_SERVERLESS', 'STREAMSETS_CONNX', 'STREAMSETS_RABBITMQ', 'STREAMSETS_SALESFORCE',
+                'STREAMSETS_REDIS' and 'STREAMSETS_JDBC
             authoring_data_collector (:obj:`streamsets.sdk.sch.DataCollector`): Authoring Data Collector.
             tags (:obj:`list`, optional): List of tags (strings). Default: ``None``.
 

{streamsets-6.6.0rc1 → streamsets-6.6.2.dev0}/streamsets/sdk/sdc_models.py

@@ -21,8 +21,8 @@ from .constants import (
 from .exceptions import ServiceDefinitionNotFound
 from .models import Configuration, _StageWithPredicates
 from .utils import (
-    SeekableList, determine_fragment_label, format_log, get_attribute, get_color_icon_from_stage_definition,
-    get_open_output_lanes, get_params, pipeline_json_encoder,
+    SeekableList, determine_fragment_label, format_log, get_accepted_labels_libraries_and_names, get_attribute,
+    get_color_icon_from_stage_definition, get_open_output_lanes, get_params, pipeline_json_encoder,
 )
 
 # fmt: on
@@ -1062,11 +1062,28 @@ class PipelineBuilder:
         Returns:
             An instance of :py:class:`streamsets.sdk.sdc_models.Stage`.
         """
+        accepted_labels, accepted_libraries, accepted_java_names = get_accepted_labels_libraries_and_names(
+            self, 'startEventStage'
+        )
+
+        if label and label not in accepted_labels:
+            raise ValueError('Stage with label {} is not a supported start event stage'.format(label))
+        if name and name not in accepted_java_names:
+            raise ValueError('Stage with name {} is not a supported start event stage'.format(name))
+        if library and library not in accepted_libraries:
+            raise ValueError('Stage with library {} is not a supported start event stage'.format(library))
+
         stage_instance, stage_definition = next(
             (stage.instance, stage.definition)
             for stage in self._get_stage_data(label=label, name=name, library=library)
             if stage.definition.get('pipelineLifecycleStage') is True
         )
+
+        # We need instanceName in the form <instance_name>_StartEventStage and not <instance_name>_01
+        instance_name_split_list = stage_instance['instanceName'].split('_')
+        instance_name_split_list[-1] = 'StartEventStage'
+        stage_instance['instanceName'] = '_'.join(instance_name_split_list)
+
         self._pipeline[self._config_key]['startEventStages'] = [stage_instance]
         self._set_pipeline_configuration('startEventStage', self._stage_to_configuration_name(stage_instance))
         return self._all_stages.get(stage_instance['stageName'], Stage)(
@@ -1092,11 +1109,28 @@ class PipelineBuilder:
         Returns:
             An instance of :py:class:`streamsets.sdk.sdc_models.Stage`.
         """
+        accepted_labels, accepted_libraries, accepted_java_names = get_accepted_labels_libraries_and_names(
+            self, 'stopEventStage'
+        )
+
+        if label and label not in accepted_labels:
+            raise ValueError('Stage with label {} is not a supported stop event stage'.format(label))
+        if name and name not in accepted_java_names:
+            raise ValueError('Stage with name {} is not a supported stop event stage'.format(name))
+        if library and library not in accepted_libraries:
+            raise ValueError('Stage with library {} is not a supported stop event stage'.format(library))
+
         stage_instance, stage_definition = next(
             (stage.instance, stage.definition)
             for stage in self._get_stage_data(label=label, name=name, library=library)
             if stage.definition.get('pipelineLifecycleStage') is True
         )
+
+        # We need instanceName in the form <instance_name>_StartEventStage and not <instance_name>_01
+        instance_name_split_list = stage_instance['instanceName'].split('_')
+        instance_name_split_list[-1] = 'StopEventStage'
+        stage_instance['instanceName'] = '_'.join(instance_name_split_list)
+
         self._pipeline[self._config_key]['stopEventStages'] = [stage_instance]
         self._set_pipeline_configuration('stopEventStage', self._stage_to_configuration_name(stage_instance))
         return self._all_stages.get(stage_instance['stageName'], Stage)(
@@ -1121,12 +1155,30 @@ class PipelineBuilder:
 
         Returns:
             An instance of :py:class:`streamsets.sdk.sdc_models.Stage`.
+
         """
+        accepted_labels, accepted_libraries, accepted_java_names = get_accepted_labels_libraries_and_names(
+            self, 'testOriginStage'
+        )
+
+        if label and label not in accepted_labels:
+            raise ValueError('Stage with label {} is not a supported test origin stage'.format(label))
+        if name and name not in accepted_java_names:
+            raise ValueError('Stage with name {} is not a supported test origin stage'.format(name))
+        if library and library not in accepted_libraries:
+            raise ValueError('Stage with library {} is not a supported test origin stage'.format(library))
+
         stage_instance, stage_definition = next(
             (stage.instance, stage.definition)
             for stage in self._get_stage_data(label=label, name=name, library=library)
             if stage.definition.get('type') == 'SOURCE'
         )
+
+        # We need instanceName in the form <instance_name>_StartEventStage and not <instance_name>_01
+        instance_name_split_list = stage_instance['instanceName'].split('_')
+        instance_name_split_list[-1] = 'TestOriginStage'
+        stage_instance['instanceName'] = '_'.join(instance_name_split_list)
+
         self._pipeline[self._config_key]['testOriginStage'] = stage_instance
         self._set_pipeline_configuration('testOriginStage', self._stage_to_configuration_name(stage_instance))
         return self._all_stages.get(stage_instance['stageName'], Stage)(