e6data-python-connector 2.3.12rc31__tar.gz → 2.3.13__tar.gz

{e6data_python_connector-2.3.12rc31/e6data_python_connector.egg-info → e6data_python_connector-2.3.13}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: e6data-python-connector
-Version: 2.3.12rc31
+Version: 2.3.13
 Summary: Client for the e6data distributed SQL Engine.
 Home-page: https://github.com/e6x-labs/e6data-python-connector
 Author: e6data, Inc.
@@ -39,7 +39,7 @@ Dynamic: summary
 
 # e6data Python Connector
 
-![version](https://img.shields.io/badge/version-2.3.12rc31-blue.svg)
+![version](https://img.shields.io/badge/version-2.3.13-blue.svg)
 
 ## Introduction
 
@@ -54,6 +54,10 @@ yum install python3-devel gcc-c++ -y
 # Ubuntu/Debian dependencies
 apt install python3-dev g++ -y
 
+# Windows dependencies
+# Install Visual C++ Build Tools from:
+# https://visualstudio.microsoft.com/visual-cpp-build-tools/
+# Select the "Desktop Development with C++" option during installation.
 
 # Pip dependencies
 pip install wheel
@@ -122,6 +126,7 @@ The `Connection` class supports the following parameters:
 | `catalog` | str | No | None | Catalog name |
 | `cluster_name` | str | No | None | Name of the cluster for cluster-specific operations |
 | `secure` | bool | No | False | Enable SSL/TLS for secure connections |
+| `ssl_cert` | str/bytes | No | None | Path to CA certificate (PEM) or certificate bytes for HTTPS connections |
 | `auto_resume` | bool | No | True | Automatically resume cluster if suspended |
 | `grpc_options` | dict | No | None | Additional gRPC configuration options |
 | `debug` | bool | No | False | Enable debug logging for troubleshooting |
@@ -159,6 +164,72 @@ conn = Connection(
 )
 ```
 
+#### HTTPS Connection with HAProxy
+
+When connecting through HAProxy with HTTPS, you can provide a custom CA certificate for secure connections. The `ssl_cert` parameter accepts either a file path to a PEM certificate or the certificate content as bytes.
+
+**Using a CA certificate file path:**
+
+```python
+conn = Connection(
+    host=host,
+    port=443,
+    username=username,
+    password=password,
+    database=database,
+    secure=True,
+    ssl_cert='/path/to/ca-cert.pem'  # Path to your CA certificate
+)
+```
+
+**Reading certificate content as bytes:**
+
+```python
+# Read certificate file and pass as bytes
+with open('/path/to/ca-cert.pem', 'rb') as cert_file:
+    cert_data = cert_file.read()
+
+conn = Connection(
+    host=host,
+    port=443,
+    username=username,
+    password=password,
+    database=database,
+    secure=True,
+    ssl_cert=cert_data  # Certificate content as bytes
+)
+```
+
+**Using system CA bundle for publicly signed certificates:**
+
+```python
+# When ssl_cert is None, system default CA bundle is used
+conn = Connection(
+    host=host,
+    port=443,
+    username=username,
+    password=password,
+    database=database,
+    secure=True  # Uses system CA bundle by default
+)
+```
+
+**Connection pooling with custom CA certificate:**
+
+```python
+pool = ConnectionPool(
+    min_size=2,
+    max_size=10,
+    host=host,
+    port=443,
+    username=username,
+    password=password,
+    database=database,
+    secure=True,
+    ssl_cert='/path/to/ca-cert.pem'  # Custom CA certificate for pool connections
+)
+```
+
 #### Handling Missing System Dependencies
 
 The e6data connector uses the `fastbinary` module (from Apache Thrift) for optimal performance when deserializing data. This module requires system-level dependencies (`python3-devel` and `gcc-c++`) to be installed.

{e6data_python_connector-2.3.12rc31 → e6data_python_connector-2.3.13}/README.md

@@ -1,6 +1,6 @@
 # e6data Python Connector
 
-![version](https://img.shields.io/badge/version-2.3.12rc31-blue.svg)
+![version](https://img.shields.io/badge/version-2.3.13-blue.svg)
 
 ## Introduction
 
@@ -15,6 +15,10 @@ yum install python3-devel gcc-c++ -y
 # Ubuntu/Debian dependencies
 apt install python3-dev g++ -y
 
+# Windows dependencies
+# Install Visual C++ Build Tools from:
+# https://visualstudio.microsoft.com/visual-cpp-build-tools/
+# Select the "Desktop Development with C++" option during installation.
 
 # Pip dependencies
 pip install wheel
@@ -83,6 +87,7 @@ The `Connection` class supports the following parameters:
 | `catalog` | str | No | None | Catalog name |
 | `cluster_name` | str | No | None | Name of the cluster for cluster-specific operations |
 | `secure` | bool | No | False | Enable SSL/TLS for secure connections |
+| `ssl_cert` | str/bytes | No | None | Path to CA certificate (PEM) or certificate bytes for HTTPS connections |
 | `auto_resume` | bool | No | True | Automatically resume cluster if suspended |
 | `grpc_options` | dict | No | None | Additional gRPC configuration options |
 | `debug` | bool | No | False | Enable debug logging for troubleshooting |
@@ -120,6 +125,72 @@ conn = Connection(
 )
 ```
 
+#### HTTPS Connection with HAProxy
+
+When connecting through HAProxy with HTTPS, you can provide a custom CA certificate for secure connections. The `ssl_cert` parameter accepts either a file path to a PEM certificate or the certificate content as bytes.
+
+**Using a CA certificate file path:**
+
+```python
+conn = Connection(
+    host=host,
+    port=443,
+    username=username,
+    password=password,
+    database=database,
+    secure=True,
+    ssl_cert='/path/to/ca-cert.pem'  # Path to your CA certificate
+)
+```
+
+**Reading certificate content as bytes:**
+
+```python
+# Read certificate file and pass as bytes
+with open('/path/to/ca-cert.pem', 'rb') as cert_file:
+    cert_data = cert_file.read()
+
+conn = Connection(
+    host=host,
+    port=443,
+    username=username,
+    password=password,
+    database=database,
+    secure=True,
+    ssl_cert=cert_data  # Certificate content as bytes
+)
+```
+
+**Using system CA bundle for publicly signed certificates:**
+
+```python
+# When ssl_cert is None, system default CA bundle is used
+conn = Connection(
+    host=host,
+    port=443,
+    username=username,
+    password=password,
+    database=database,
+    secure=True  # Uses system CA bundle by default
+)
+```
+
+**Connection pooling with custom CA certificate:**
+
+```python
+pool = ConnectionPool(
+    min_size=2,
+    max_size=10,
+    host=host,
+    port=443,
+    username=username,
+    password=password,
+    database=database,
+    secure=True,
+    ssl_cert='/path/to/ca-cert.pem'  # Custom CA certificate for pool connections
+)
+```
+
 #### Handling Missing System Dependencies
 
 The e6data connector uses the `fastbinary` module (from Apache Thrift) for optimal performance when deserializing data. This module requires system-level dependencies (`python3-devel` and `gcc-c++`) to be installed.

{e6data_python_connector-2.3.12rc31 → e6data_python_connector-2.3.13}/e6data_python_connector/cluster_manager.py

@@ -9,27 +9,9 @@ import multiprocessing
 
 logger = logging.getLogger(__name__)
 
+from e6data_python_connector.common import get_ssl_credentials
 from e6data_python_connector.strategy import _get_active_strategy, _set_active_strategy, _set_pending_strategy, \
-    _get_grpc_header as _get_strategy_header
-
-
-def _get_grpc_header(engine_ip=None, cluster=None, strategy=None):
-    """
-    Generate gRPC metadata headers for the request.
-
-    This function creates a list of metadata headers to be used in gRPC requests.
-    It includes optional headers for the engine IP, cluster UUID, and deployment strategy.
-
-    Args:
-        engine_ip (str, optional): The IP address of the engine. Defaults to None.
-        cluster (str, optional): The UUID of the cluster. Defaults to None.
-        strategy (str, optional): The deployment strategy (blue/green). Defaults to None.
-
-    Returns:
-        list: A list of tuples representing the gRPC metadata headers.
-    """
-    # Use the strategy module's implementation
-    return _get_strategy_header(engine_ip=engine_ip, cluster=cluster, strategy=strategy)
+    _get_grpc_header
 
 
 class _StatusLock:
@@ -140,7 +122,7 @@ class ClusterManager:
     """
 
     def __init__(self, host: str, port: int, user: str, password: str, secure_channel: bool = False, timeout=60 * 5,
-                 cluster_uuid=None, grpc_options=None, debug=False):
+                 cluster_uuid=None, grpc_options=None, debug=False, ssl_cert=None):
         """
         Initializes a new instance of the ClusterManager class.
 
@@ -156,6 +138,8 @@ class ClusterManager:
             cluster_uuid (str, optional): The unique identifier for the target cluster;
                 defaults to None.
             debug (bool, optional): Enable debug logging; defaults to False.
+            ssl_cert (str or bytes, optional): Path to CA certificate file (PEM format) or
+                certificate content as bytes for secure connections; defaults to None.
         """
 
         self._host = host
@@ -169,6 +153,7 @@ class ClusterManager:
         if grpc_options is None:
             self._grpc_options = dict()
         self._debug = debug
+        self._ssl_cert = ssl_cert
 
     @property
     def _get_connection(self):
@@ -184,7 +169,7 @@ class ClusterManager:
             self._channel = grpc.secure_channel(
                 target='{}:{}'.format(self._host, self._port),
                 options=self._grpc_options,
-                credentials=grpc.ssl_channel_credentials()
+                credentials=get_ssl_credentials(self._ssl_cert)
             )
         else:
             self._channel = grpc.insecure_channel(

{e6data_python_connector-2.3.12rc31 → e6data_python_connector-2.3.13}/e6data_python_connector/common.py

@@ -15,6 +15,7 @@ from builtins import int
 from builtins import object
 from builtins import str
 
+import grpc
 from future.utils import with_metaclass
 from past.builtins import basestring
 
@@ -267,3 +268,46 @@ class UniversalSet(object):
 
     def __contains__(self, item):
         return True
+
+
+def get_ssl_credentials(ssl_cert):
+    """
+    Get SSL credentials for secure gRPC channel.
+
+    Handles three scenarios:
+    1. ssl_cert is a string (file path): Read the PEM certificate from the file
+    2. ssl_cert is bytes: Use the certificate content directly
+    3. ssl_cert is None: Use system default CA bundle
+
+    Parameters:
+        ssl_cert (str or bytes or None): SSL certificate as file path, bytes, or None
+
+    Returns:
+        grpc.ChannelCredentials: SSL credentials for gRPC channel
+
+    Raises:
+        FileNotFoundError: If ssl_cert is a file path but the file doesn't exist
+        IOError: If ssl_cert file cannot be read
+    """
+    if ssl_cert is None:
+        # Use system default CA bundle
+        return grpc.ssl_channel_credentials()
+    elif isinstance(ssl_cert, str):
+        # ssl_cert is a file path - read the certificate from file
+        try:
+            with open(ssl_cert, 'rb') as cert_file:
+                root_ca_cert = cert_file.read()
+            return grpc.ssl_channel_credentials(root_certificates=root_ca_cert)
+        except FileNotFoundError:
+            _logger.error("SSL certificate file not found: {}".format(ssl_cert))
+            raise
+        except IOError as e:
+            _logger.error("Failed to read SSL certificate file {}: {}".format(ssl_cert, e))
+            raise
+    elif isinstance(ssl_cert, bytes):
+        # ssl_cert is certificate content as bytes
+        return grpc.ssl_channel_credentials(root_certificates=ssl_cert)
+    else:
+        # Invalid type - fall back to system default CA bundle with warning
+        _logger.warning("Invalid ssl_cert type: {}. Using system default CA bundle.".format(type(ssl_cert)))
+        return grpc.ssl_channel_credentials()

e6data_python_connector-2.3.13/e6data_python_connector/constants.py

@@ -0,0 +1,56 @@
+import pytz
+
+# Type definitions
+PRIMITIVE_TYPES = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 15, 16, 17, 18, 19, 20, 21}
+COMPLEX_TYPES = {10, 11, 12, 13, 14}
+COLLECTION_TYPES = {10, 11}
+TYPE_NAMES = {
+    0: "BOOLEAN",
+    1: "TINYINT",
+    2: "SMALLINT",
+    3: "INT",
+    4: "BIGINT",
+    5: "FLOAT",
+    6: "DOUBLE",
+    7: "STRING",
+    8: "TIMESTAMP",
+    9: "BINARY",
+    10: "ARRAY",
+    11: "MAP",
+    12: "STRUCT",
+    13: "UNIONTYPE",
+    15: "DECIMAL",
+    16: "NULL",
+    17: "DATE",
+    18: "VARCHAR",
+    19: "CHAR",
+    20: "INTERVAL_YEAR_MONTH",
+    21: "INTERVAL_DAY_TIME",
+}
+CHARACTER_MAXIMUM_LENGTH = "characterMaximumLength"
+PRECISION = "precision"
+SCALE = "scale"
+ZONE = pytz.timezone('UTC')
+
+# Retry and timeout constants
+MAX_RETRY_ATTEMPTS = 5
+RETRY_SLEEP_SECONDS = 0.2
+STRATEGY_CACHE_TIMEOUT_SECONDS = 300  # 5 minutes
+DEFAULT_GRPC_PREPARE_TIMEOUT_SECONDS = 600  # 10 minutes
+DEFAULT_AUTO_RESUME_TIMEOUT_SECONDS = 300  # 5 minutes
+CLUSTER_STATUS_CHECK_SLEEP_SECONDS = 5
+LOCK_TIMEOUT_MS = 500
+
+# Connection pool constants
+POOL_GET_TIMEOUT_SECONDS = 0.1
+POOL_RETRY_SLEEP_SECONDS = 0.1
+
+# Blue-green deployment strategy constants
+STRATEGY_BLUE = 'blue'
+STRATEGY_GREEN = 'green'
+VALID_STRATEGIES = {STRATEGY_BLUE, STRATEGY_GREEN}
+
+# gRPC error codes and messages
+GRPC_ERROR_STRATEGY_MISMATCH = 'status: 456'
+GRPC_ERROR_SERVICE_UNAVAILABLE = 'status: 503'
+GRPC_ERROR_ACCESS_DENIED = 'Access denied'

{e6data_python_connector-2.3.12rc31 → e6data_python_connector-2.3.13}/e6data_python_connector/datainputstream.py

@@ -10,7 +10,13 @@ from thrift.transport import TTransport
 
 from e6data_python_connector.e6x_vector.ttypes import Chunk, Vector, VectorType
 from e6data_python_connector.constants import ZONE
-from e6data_python_connector.date_time_utils import floor_div, floor_mod, timezone_from_offset
+from e6data_python_connector.date_time_utils import (
+    floor_div,
+    floor_mod,
+    timezone_from_offset,
+    format_iso_date_from_epoch_micros,
+    format_iso_datetime_from_epoch_micros,
+)
 
 # Try to import fastbinary - it's optional but provides better performance
 _fastbinary_available = False
@@ -463,16 +469,11 @@ def read_values_from_array(query_columns_description: list, dis: DataInputStream
             if dtype == "LONG":
                 value_array.append(dis.read_long())
             elif dtype == "DATE":
-                epoch_seconds = floor_div(dis.read_long(), 1000_000)
-                date = datetime.fromtimestamp(epoch_seconds, ZONE)
-                value_array.append(date.strftime("%Y-%m-%d"))
+                value_array.append(format_iso_date_from_epoch_micros(dis.read_long()))
             elif dtype == "DATETIME":
-                epoch_micros = dis.read_long()
-                epoch_seconds = floor_div(epoch_micros, 1000_000)
-                micros_of_the_day = floor_mod(epoch_micros, 1000_000)
-                date_time = datetime.fromtimestamp(epoch_seconds, ZONE)
-                date_time = date_time + timedelta(microseconds=micros_of_the_day)
-                value_array.append(date_time.strftime("%Y-%m-%d %H:%M:%S"))
+                value_array.append(format_iso_datetime_from_epoch_micros(
+                    dis.read_long(), tz=ZONE, separator=' ',
+                    include_millis=False, include_offset=False))
             elif dtype == "STRING" or dtype == "ARRAY" or dtype == "MAP" or dtype == "STRUCT":
                 value_array.append(dis.read_utf().decode())
             elif dtype == "INT":
@@ -560,27 +561,38 @@ def get_column_from_chunk(vector: Vector) -> list:
                 value_array.append(vector.data.int64Data.data[
                                        row] if not vector.isConstantVector else vector.data.numericConstantData.data)
         elif d_type == VectorType.DATE:
+            # Use the JDBC-parity formatter so years > 9999 emit "+YYYYY-MM-DD"
+            # instead of raising. Per-row try/except keeps a single bad value
+            # from truncating the column (which previously cascaded to an
+            # IndexError in read_rows_from_chunk).
             for row in range(vector.size):
                 if get_null(vector, row):
                     value_array.append(None)
                     continue
-                epoch_seconds = floor_div(vector.data.dateData.data[
-                                              row] if not vector.isConstantVector else vector.data.dateConstantData.data,
-                                          1000_000)
-                date = datetime.fromtimestamp(epoch_seconds, zone)
-                value_array.append(date.strftime("%Y-%m-%d"))
+                try:
+                    raw = (vector.data.dateData.data[row]
+                           if not vector.isConstantVector
+                           else vector.data.dateConstantData.data)
+                    value_array.append(format_iso_date_from_epoch_micros(raw))
+                except Exception as e:
+                    _logger.error("Failed to parse DATE row=%s: %s", row, e)
+                    value_array.append('Failed to parse.')
         elif d_type == VectorType.DATETIME:
+            # Default formatter knobs reproduce the prior datetime.isoformat
+            # output ("YYYY-MM-DDTHH:MM:SS.sss+HH:MM"). See DATE branch above
+            # for the rationale on per-row try/except.
             for row in range(vector.size):
                 if get_null(vector, row):
                     value_array.append(None)
                     continue
-                epoch_micros = vector.data.timeData.data[
-                    row] if not vector.isConstantVector else vector.data.timeConstantData.data
-                epoch_seconds = floor_div(epoch_micros, 1000_000)
-                micros_of_the_day = floor_mod(epoch_micros, 1000_000)
-                date_time = datetime.fromtimestamp(epoch_seconds, zone)
-                date_time = date_time + timedelta(microseconds=micros_of_the_day)
-                value_array.append(date_time.isoformat(timespec='milliseconds'))
+                try:
+                    epoch_micros = (vector.data.timeData.data[row]
+                                    if not vector.isConstantVector
+                                    else vector.data.timeConstantData.data)
+                    value_array.append(format_iso_datetime_from_epoch_micros(epoch_micros, tz=zone))
+                except Exception as e:
+                    _logger.error("Failed to parse DATETIME row=%s: %s", row, e)
+                    value_array.append('Failed to parse.')
         elif d_type == VectorType.STRING or d_type == VectorType.ARRAY or d_type == VectorType.MAP or d_type == VectorType.STRUCT:
             for row in range(vector.size):
                 if get_null(vector, row):
@@ -627,22 +639,27 @@ def get_column_from_chunk(vector: Vector) -> list:
             for row in range(vector.size):
                 value_array.append(None)
         elif d_type == VectorType.TIMESTAMP_TZ:
+            # row_zone is scoped to the row (not the outer `zone`) so a
+            # per-row zone resolution doesn't leak into later iterations.
             for row in range(vector.size):
                 if get_null(vector, row):
                     value_array.append(None)
                     continue
-                epoch_micros = vector.data.timeData.data[
-                    row] if not vector.isConstantVector else vector.data.timeConstantData.data
-                if ((vector.isConstantVector and vector.data.timeConstantData.zoneData is not None) or
-                        (not vector.isConstantVector and vector.data.timeData.zoneData is not None)):
-                    zone_id = vector.data.timeData.zoneData[
-                        row] if not vector.isConstantVector else vector.data.timeConstantData.zoneData
-                    zone = timezone_from_offset(zone_id)
-                epoch_seconds = floor_div(epoch_micros, 1000_000)
-                micros_of_the_day = floor_mod(epoch_micros, 1000_000)
-                date_time = datetime.fromtimestamp(epoch_seconds, zone)
-                date_time = date_time + timedelta(microseconds=micros_of_the_day)
-                value_array.append(date_time.isoformat(timespec='milliseconds'))
+                try:
+                    epoch_micros = (vector.data.timeData.data[row]
+                                    if not vector.isConstantVector
+                                    else vector.data.timeConstantData.data)
+                    row_zone = zone
+                    if ((vector.isConstantVector and vector.data.timeConstantData.zoneData is not None) or
+                            (not vector.isConstantVector and vector.data.timeData.zoneData is not None)):
+                        zone_id = (vector.data.timeData.zoneData[row]
+                                   if not vector.isConstantVector
+                                   else vector.data.timeConstantData.zoneData)
+                        row_zone = timezone_from_offset(zone_id)
+                    value_array.append(format_iso_datetime_from_epoch_micros(epoch_micros, tz=row_zone))
+                except Exception as e:
+                    _logger.error("Failed to parse TIMESTAMP_TZ row=%s: %s", row, e)
+                    value_array.append('Failed to parse.')
         elif d_type == VectorType.DECIMAL128:
             # Handle both constant and non-constant vectors following Java implementation
             if vector.isConstantVector:
@@ -679,6 +696,12 @@ def get_column_from_chunk(vector: Vector) -> list:
         else:
             value_array.append(None)
     except Exception as e:
-        _logger.error(e)
-        value_array.append('Failed to parse.')
+        # Safety net: if anything escapes the per-row try/excepts above (or
+        # comes from a branch without one), pad value_array to vector.size so
+        # read_rows_from_chunk's columns[colIndex][rowIndex] access can never
+        # IndexError on a short column.
+        _logger.error("get_column_from_chunk failed (vectorType=%s, parsed=%s/%s): %s",
+                      d_type, len(value_array), vector.size, e)
+        while len(value_array) < vector.size:
+            value_array.append('Failed to parse.')
     return value_array