snowflake-connector-python 3.7.0__tar.gz → 3.8.0__tar.gz

{snowflake-connector-python-3.7.0 → snowflake-connector-python-3.8.0}/DESCRIPTION.md

@@ -8,6 +8,35 @@ Source code is also available at: https://github.com/snowflakedb/snowflake-conne
 
 # Release Notes
 
+- v3.8.0(April 04,2024)
+
+  - Improved `externalbrowser` auth in containerized environments
+    - Instruct browser to not fetch `/favicon` on success page
+    - Simple retry strategy on empty socket.recv
+    - Add `SNOWFLAKE_AUTH_SOCKET_REUSE_PORT` flag (usage: `SNOWFLAKE_AUTH_SOCKET_REUSE_PORT=true`) to set the underlying socket's `SO_REUSEPORT` flag (described in the [socket man page](https://man7.org/linux/man-pages/man7/socket.7.html))
+      - Useful when the randomized port used in the localhost callback url is being followed before the container engine completes port forwarding to host
+      - Statically map a port between your host and container and allow that port to be reused in rapid succession with:
+         `SF_AUTH_SOCKET_PORT=3037 SNOWFLAKE_AUTH_SOCKET_REUSE_PORT=true poetry run python somescript.py`
+    - Add `SNOWFLAKE_AUTH_SOCKET_MSG_DONTWAIT` flag (usage: `SNOWFLAKE_AUTH_SOCKET_MSG_DONTWAIT=true`) to make a non-blocking socket.recv call and retry on Error
+      - Consider using this if running in a containerized environment and externalbrowser auth frequently hangs while waiting for callback
+      - NOTE: this has not been tested extensively, but has been shown to improve the experience when using WSL
+  - Added support for parsing structured type information in schema queries.
+  - Bumped platformdirs from >=2.6.0,<4.0.0 to >=2.6.0,<5.0.0
+  - Updated diagnostics to use system$allowlist instead of system$whitelist.
+  - Updated `write_pandas` to skip TABLE IF NOT EXISTS in truncate mode.
+  - Improved cleanup logic for connection to rely on interpreter shutdown instead of the `__del__` method.
+  - Updated the logging level from INFO to DEBUG when logging the executed query using `SnowflakeCursor.execute`.
+  - Fixed a bug that the truncated password in log is not masked.
+
+- v3.7.1(February 21, 2024)
+
+  - Bumped pandas dependency from >=1.0.0,<2.2.0 to >=1.0.0,<3.0.0.
+  - Bumped cryptography dependency from <42.0.0,>=3.1.0 to >=3.1.0,<43.0.0.
+  - Bumped pyOpenSSL dependency from >=16.2.0,<24.0.0 to >=16.2.0,<25.0.0.
+  - Fixed a memory leak in decimal data conversion.
+  - Fixed a bug where `write_pandas` wasn't truncating the target table.
+  - Bumped keyring dependency lower bound to 23.1.0 to address security vulnerability.
+
 - v3.7.0(January 25,2024)
 
   - Added a new boolean parameter `force_return_table` to `SnowflakeCursor.fetch_arrow_all` to force returning `pyarrow.Table` in case of zero rows.
@@ -17,6 +46,7 @@ Source code is also available at: https://github.com/snowflakedb/snowflake-conne
   - Fixed PyArrow Table type hinting
   - Added support for connecting using an existing connection via the session and master token.
   - Added support for connecting to Snowflake by authenticating with multiple SAML IDP using external browser.
+  - Added support for structured types (OBJECT, MAP, ARRAY) to nanoarrow converters.
   - Fixed compilation issue due to missing cstdint header on gcc13.
   - Improved config permissions warning message.
 
@@ -52,6 +82,7 @@ Source code is also available at: https://github.com/snowflakedb/snowflake-conne
   - Added the `socket_timeout` argument to `snowflake.connector.connect` specifying socket read and connect timeout.
   - Fixed `login_timeout` and `network_timeout` behaviour. Retries of login and network requests are now properly halted after these timeouts expire.
   - Fixed bug for issue https://github.com/urllib3/urllib3/issues/1878 in vendored `urllib`.
+  - Add User-Agent header for diagnostic report for tracking.
 
 - v3.3.1(October 16,2023)
 

{snowflake-connector-python-3.7.0 → snowflake-connector-python-3.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: snowflake-connector-python
-Version: 3.7.0
+Version: 3.8.0
 Summary: Snowflake Connector for Python
 Home-page: https://www.snowflake.com/
 Author: Snowflake, Inc
@@ -40,8 +40,8 @@ License-File: LICENSE.txt
 License-File: NOTICE
 Requires-Dist: asn1crypto<2.0.0,>0.24.0
 Requires-Dist: cffi<2.0.0,>=1.9
-Requires-Dist: cryptography<42.0.0,>=3.1.0
-Requires-Dist: pyOpenSSL<24.0.0,>=16.2.0
+Requires-Dist: cryptography<43.0.0,>=3.1.0
+Requires-Dist: pyOpenSSL<25.0.0,>=16.2.0
 Requires-Dist: pyjwt<3.0.0
 Requires-Dist: pytz
 Requires-Dist: requests<3.0.0
@@ -54,7 +54,7 @@ Requires-Dist: certifi>=2017.4.17
 Requires-Dist: typing_extensions<5,>=4.3
 Requires-Dist: filelock<4,>=3.5
 Requires-Dist: sortedcontainers>=2.4.0
-Requires-Dist: platformdirs<4.0.0,>=2.6.0
+Requires-Dist: platformdirs<5.0.0,>=2.6.0
 Requires-Dist: tomlkit
 Provides-Extra: development
 Requires-Dist: Cython; extra == "development"
@@ -70,10 +70,10 @@ Requires-Dist: pytest-timeout; extra == "development"
 Requires-Dist: pytest-xdist; extra == "development"
 Requires-Dist: pytzdata; extra == "development"
 Provides-Extra: pandas
-Requires-Dist: pandas<2.2.0,>=1.0.0; extra == "pandas"
+Requires-Dist: pandas<3.0.0,>=1.0.0; extra == "pandas"
 Requires-Dist: pyarrow; extra == "pandas"
 Provides-Extra: secure-local-storage
-Requires-Dist: keyring!=16.1.0,<25.0.0; extra == "secure-local-storage"
+Requires-Dist: keyring<25.0.0,>=23.1.0; extra == "secure-local-storage"
 
 This package includes the Snowflake Connector for Python, which conforms to the Python DB API 2.0 specification:
 https://www.python.org/dev/peps/pep-0249/
@@ -85,6 +85,35 @@ Source code is also available at: https://github.com/snowflakedb/snowflake-conne
 
 # Release Notes
 
+- v3.8.0(April 04,2024)
+
+  - Improved `externalbrowser` auth in containerized environments
+    - Instruct browser to not fetch `/favicon` on success page
+    - Simple retry strategy on empty socket.recv
+    - Add `SNOWFLAKE_AUTH_SOCKET_REUSE_PORT` flag (usage: `SNOWFLAKE_AUTH_SOCKET_REUSE_PORT=true`) to set the underlying socket's `SO_REUSEPORT` flag (described in the [socket man page](https://man7.org/linux/man-pages/man7/socket.7.html))
+      - Useful when the randomized port used in the localhost callback url is being followed before the container engine completes port forwarding to host
+      - Statically map a port between your host and container and allow that port to be reused in rapid succession with:
+         `SF_AUTH_SOCKET_PORT=3037 SNOWFLAKE_AUTH_SOCKET_REUSE_PORT=true poetry run python somescript.py`
+    - Add `SNOWFLAKE_AUTH_SOCKET_MSG_DONTWAIT` flag (usage: `SNOWFLAKE_AUTH_SOCKET_MSG_DONTWAIT=true`) to make a non-blocking socket.recv call and retry on Error
+      - Consider using this if running in a containerized environment and externalbrowser auth frequently hangs while waiting for callback
+      - NOTE: this has not been tested extensively, but has been shown to improve the experience when using WSL
+  - Added support for parsing structured type information in schema queries.
+  - Bumped platformdirs from >=2.6.0,<4.0.0 to >=2.6.0,<5.0.0
+  - Updated diagnostics to use system$allowlist instead of system$whitelist.
+  - Updated `write_pandas` to skip TABLE IF NOT EXISTS in truncate mode.
+  - Improved cleanup logic for connection to rely on interpreter shutdown instead of the `__del__` method.
+  - Updated the logging level from INFO to DEBUG when logging the executed query using `SnowflakeCursor.execute`.
+  - Fixed a bug that the truncated password in log is not masked.
+
+- v3.7.1(February 21, 2024)
+
+  - Bumped pandas dependency from >=1.0.0,<2.2.0 to >=1.0.0,<3.0.0.
+  - Bumped cryptography dependency from <42.0.0,>=3.1.0 to >=3.1.0,<43.0.0.
+  - Bumped pyOpenSSL dependency from >=16.2.0,<24.0.0 to >=16.2.0,<25.0.0.
+  - Fixed a memory leak in decimal data conversion.
+  - Fixed a bug where `write_pandas` wasn't truncating the target table.
+  - Bumped keyring dependency lower bound to 23.1.0 to address security vulnerability.
+
 - v3.7.0(January 25,2024)
 
   - Added a new boolean parameter `force_return_table` to `SnowflakeCursor.fetch_arrow_all` to force returning `pyarrow.Table` in case of zero rows.
@@ -94,6 +123,7 @@ Source code is also available at: https://github.com/snowflakedb/snowflake-conne
   - Fixed PyArrow Table type hinting
   - Added support for connecting using an existing connection via the session and master token.
   - Added support for connecting to Snowflake by authenticating with multiple SAML IDP using external browser.
+  - Added support for structured types (OBJECT, MAP, ARRAY) to nanoarrow converters.
   - Fixed compilation issue due to missing cstdint header on gcc13.
   - Improved config permissions warning message.
 
@@ -129,6 +159,7 @@ Source code is also available at: https://github.com/snowflakedb/snowflake-conne
   - Added the `socket_timeout` argument to `snowflake.connector.connect` specifying socket read and connect timeout.
   - Fixed `login_timeout` and `network_timeout` behaviour. Retries of login and network requests are now properly halted after these timeouts expire.
   - Fixed bug for issue https://github.com/urllib3/urllib3/issues/1878 in vendored `urllib`.
+  - Add User-Agent header for diagnostic report for tracking.
 
 - v3.3.1(October 16,2023)
 

{snowflake-connector-python-3.7.0 → snowflake-connector-python-3.8.0}/setup.cfg

@@ -45,8 +45,8 @@ packages = find_namespace:
 install_requires = 
 	asn1crypto>0.24.0,<2.0.0
 	cffi>=1.9,<2.0.0
-	cryptography>=3.1.0,<42.0.0
-	pyOpenSSL>=16.2.0,<24.0.0
+	cryptography>=3.1.0,<43.0.0
+	pyOpenSSL>=16.2.0,<25.0.0
 	pyjwt<3.0.0
 	pytz
 	requests<3.0.0
@@ -59,7 +59,7 @@ install_requires =
 	typing_extensions>=4.3,<5
 	filelock>=3.5,<4
 	sortedcontainers>=2.4.0
-	platformdirs>=2.6.0,<4.0.0
+	platformdirs>=2.6.0,<5.0.0
 	tomlkit
 include_package_data = True
 package_dir = 
@@ -93,10 +93,10 @@ development =
 	pytest-xdist
 	pytzdata
 pandas = 
-	pandas>=1.0.0,<2.2.0
+	pandas>=1.0.0,<3.0.0
 	pyarrow
 secure-local-storage = 
-	keyring!=16.1.0,<25.0.0
+	keyring>=23.1.0,<25.0.0
 
 [egg_info]
 tag_build = 

{snowflake-connector-python-3.7.0 → snowflake-connector-python-3.8.0}/setup.py

@@ -94,6 +94,7 @@ if _ABLE_TO_COMPILE_EXTENSIONS and not SNOWFLAKE_DISABLE_COMPILE_ARROW_EXTENSION
                         *((file,) if isinstance(file, str) else file)
                     )
                     for file in {
+                        "ArrayConverter.cpp",
                         "BinaryConverter.cpp",
                         "BooleanConverter.cpp",
                         "CArrowChunkIterator.cpp",
@@ -104,6 +105,8 @@ if _ABLE_TO_COMPILE_EXTENSIONS and not SNOWFLAKE_DISABLE_COMPILE_ARROW_EXTENSION
                         "FixedSizeListConverter.cpp",
                         "FloatConverter.cpp",
                         "IntConverter.cpp",
+                        "MapConverter.cpp",
+                        "ObjectConverter.cpp",
                         "SnowflakeType.cpp",
                         "StringConverter.cpp",
                         "TimeConverter.cpp",

{snowflake-connector-python-3.7.0 → snowflake-connector-python-3.8.0}/src/snowflake/connector/auth/webbrowser.py

@@ -10,13 +10,14 @@ import json
 import logging
 import os
 import secrets
+import select
 import socket
 import time
 import webbrowser
 from types import ModuleType
 from typing import TYPE_CHECKING, Any
 
-from ..compat import parse_qs, urlencode, urlparse, urlsplit
+from ..compat import IS_WINDOWS, parse_qs, urlencode, urlparse, urlsplit
 from ..constants import (
     HTTP_HEADER_ACCEPT,
     HTTP_HEADER_CONTENT_TYPE,
@@ -116,6 +117,15 @@ class AuthByWebBrowser(AuthByPlugin):
         logger.debug("authenticating by Web Browser")
 
         socket_connection = self._socket(socket.AF_INET, socket.SOCK_STREAM)
+
+        if os.getenv("SNOWFLAKE_AUTH_SOCKET_REUSE_PORT", "False").lower() == "true":
+            if IS_WINDOWS:
+                logger.warning(
+                    "Configuration SNOWFLAKE_AUTH_SOCKET_REUSE_PORT is not available in Windows. Ignoring."
+                )
+            else:
+                socket_connection.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
+
         try:
             try:
                 socket_connection.bind(
@@ -205,14 +215,68 @@ class AuthByWebBrowser(AuthByPlugin):
     def _receive_saml_token(self, conn: SnowflakeConnection, socket_connection) -> None:
         """Receives SAML token from web browser."""
         while True:
-            socket_client, _ = socket_connection.accept()
             try:
-                # Receive the data in small chunks and retransmit it
-                data = socket_client.recv(BUF_SIZE).decode("utf-8").split("\r\n")
+                attempts = 0
+                raw_data = bytearray()
+                socket_client = None
+                max_attempts = 15
+
+                msg_dont_wait = (
+                    os.getenv("SNOWFLAKE_AUTH_SOCKET_MSG_DONTWAIT", "false").lower()
+                    == "true"
+                )
+                if IS_WINDOWS:
+                    if msg_dont_wait:
+                        logger.warning(
+                            "Configuration SNOWFLAKE_AUTH_SOCKET_MSG_DONTWAIT is not available in Windows. Ignoring."
+                        )
+                    msg_dont_wait = False
+
+                # when running in a containerized environment, socket_client.recv ocassionally returns an empty byte array
+                #   an immediate successive call to socket_client.recv gets the actual data
+                while len(raw_data) == 0 and attempts < max_attempts:
+                    attempts += 1
+                    read_sockets, _write_sockets, _exception_sockets = select.select(
+                        [socket_connection], [], []
+                    )
+
+                    if read_sockets[0] is not None:
+                        # Receive the data in small chunks and retransmit it
+                        socket_client, _ = socket_connection.accept()
+
+                        try:
+                            if msg_dont_wait:
+                                # WSL containerized environment sometimes causes socket_client.recv to hang indefinetly
+                                #   To avoid this, passing the socket.MSG_DONTWAIT flag which raises BlockingIOError if
+                                #   operation would block
+                                logger.debug(
+                                    "Calling socket_client.recv with MSG_DONTWAIT flag due to SNOWFLAKE_AUTH_SOCKET_MSG_DONTWAIT env var"
+                                )
+                                raw_data = socket_client.recv(
+                                    BUF_SIZE, socket.MSG_DONTWAIT
+                                )
+                            else:
+                                raw_data = socket_client.recv(BUF_SIZE)
+
+                        except BlockingIOError:
+                            logger.debug(
+                                "BlockingIOError raised from socket.recv while attempting to retrieve callback token request"
+                            )
+                            if attempts < max_attempts:
+                                sleep_time = 0.25
+                                logger.debug(
+                                    f"Waiting {sleep_time} seconds before trying again"
+                                )
+                                time.sleep(sleep_time)
+                            else:
+                                logger.debug("Exceeded retry count")
+
+                data = raw_data.decode("utf-8").split("\r\n")
 
                 if not self._process_options(data, socket_client):
                     self._process_receive_saml_token(conn, data, socket_client)
                     break
+
             finally:
                 socket_client.shutdown(socket.SHUT_RDWR)
                 socket_client.close()
@@ -282,6 +346,7 @@ class AuthByWebBrowser(AuthByPlugin):
         else:
             msg = f"""
 <!DOCTYPE html><html><head><meta charset="UTF-8"/>
+<link rel="icon" href="data:,">
 <title>SAML Response for Snowflake</title></head>
 <body>
 Your identity was confirmed and propagated to Snowflake {self._application}.

{snowflake-connector-python-3.7.0 → snowflake-connector-python-3.8.0}/src/snowflake/connector/connection.py

@@ -5,6 +5,7 @@
 
 from __future__ import annotations
 
+import atexit
 import logging
 import os
 import pathlib
@@ -16,6 +17,7 @@ import warnings
 import weakref
 from concurrent.futures import as_completed
 from concurrent.futures.thread import ThreadPoolExecutor
+from contextlib import suppress
 from difflib import get_close_matches
 from functools import partial
 from io import StringIO
@@ -124,8 +126,10 @@ def DefaultConverterClass() -> type:
 
 def _get_private_bytes_from_file(
     private_key_file: str | bytes | os.PathLike[str] | os.PathLike[bytes],
-    private_key_file_pwd: bytes | None = None,
+    private_key_file_pwd: bytes | str | None = None,
 ) -> bytes:
+    if private_key_file_pwd is not None and isinstance(private_key_file_pwd, str):
+        private_key_file_pwd = private_key_file_pwd.encode("utf-8")
     with open(private_key_file, "rb") as key:
         private_key = serialization.load_pem_private_key(
             key.read(),
@@ -178,7 +182,7 @@ DEFAULT_CONFIGURATION: dict[str, tuple[Any, type | tuple[type, ...]]] = {
     "passcode": (None, (type(None), str)),  # Snowflake MFA
     "private_key": (None, (type(None), str, RSAPrivateKey)),
     "private_key_file": (None, (type(None), str)),
-    "private_key_file_pwd": (None, (type(None), str)),
+    "private_key_file_pwd": (None, (type(None), str, bytes)),
     "token": (None, (type(None), str)),  # OAuth or JWT Token
     "authenticator": (DEFAULT_AUTHENTICATOR, (type(None), str)),
     "mfa_callback": (None, (type(None), Callable)),
@@ -235,7 +239,11 @@ DEFAULT_CONFIGURATION: dict[str, tuple[Any, type | tuple[type, ...]]] = {
     "connection_diag_whitelist_path": (
         None,
         (type(None), str),
-    ),  # Path to connection diag whitelist json
+    ),  # Path to connection diag whitelist json - Deprecated remove in future
+    "connection_diag_allowlist_path": (
+        None,
+        (type(None), str),
+    ),  # Path to connection diag allowlist json
     "log_imported_packages_in_telemetry": (
         True,
         bool,
@@ -343,7 +351,8 @@ class SnowflakeConnection:
         enable_stage_s3_privatelink_for_us_east_1: when true, clients use regional s3 url to upload files.
         enable_connection_diag: when true, clients will generate a connectivity diagnostic report.
         connection_diag_log_path: path to location to create diag report with enable_connection_diag.
-        connection_diag_whitelist_path: path to a whitelist.json file to test with enable_connection_diag.
+        connection_diag_whitelist_path: path to a whitelist.json file to test with enable_connection_diag - deprecated remove in future
+        connection_diag_allowlist_path: path to a allowlist.json file to test with enable_connection_diag.
         json_result_force_utf8_decoding: When true, json result will be decoded in utf-8,
           when false, the encoding of the content is auto-detected. Default value is false.
           This parameter is only effective when the result format is JSON.
@@ -432,12 +441,8 @@ class SnowflakeConnection:
 
         # get the imported modules from sys.modules
         self._log_telemetry_imported_packages()
-
-    def __del__(self) -> None:  # pragma: no cover
-        try:
-            self.close(retry=False)
-        except Exception:
-            pass
+        # check SNOW-1218851 for long term improvement plan to refactor ocsp code
+        atexit.register(self._close_at_exit)
 
     @property
     def insecure_mode(self) -> bool:
@@ -655,8 +660,23 @@ class SnowflakeConnection:
 
     @property
     def connection_diag_whitelist_path(self):
+        """
+        Old version of ``connection_diag_allowlist_path``.
+        This used to be the original name, but snowflake backend
+        deprecated whitelist for allowlist. This name will be
+        deprecated in the future.
+        """
+        warnings.warn(
+            "connection_diag_whitelist_path has been deprecated, use connection_diag_allowlist_path instead",
+            DeprecationWarning,
+            stacklevel=2,
+        )
         return self._connection_diag_whitelist_path
 
+    @property
+    def connection_diag_allowlist_path(self):
+        return self._connection_diag_allowlist_path
+
     @arrow_number_to_decimal.setter
     def arrow_number_to_decimal_setter(self, value: bool) -> None:
         self._arrow_number_to_decimal = value
@@ -689,7 +709,9 @@ class SnowflakeConnection:
                 account=self.account,
                 host=self.host,
                 connection_diag_log_path=self.connection_diag_log_path,
-                connection_diag_whitelist_path=self.connection_diag_whitelist_path,
+                connection_diag_allowlist_path=self.connection_diag_allowlist_path
+                if self.connection_diag_allowlist_path is not None
+                else self.connection_diag_whitelist_path,
                 proxy_host=self.proxy_host,
                 proxy_port=self.proxy_port,
                 proxy_user=self.proxy_user,
@@ -720,6 +742,8 @@ class SnowflakeConnection:
 
     def close(self, retry: bool = True) -> None:
         """Closes the connection."""
+        # unregister to dereference connection object as it's already closed after the execution
+        atexit.unregister(self._close_at_exit)
         try:
             if not self.rest:
                 logger.debug("Rest object has been destroyed, cannot close session")
@@ -1747,6 +1771,10 @@ class SnowflakeConnection:
             )  # Prevent KeyError when multiple threads try to remove the same query id
             self._done_async_sfqids[sf_qid] = None
 
+    def _close_at_exit(self):
+        with suppress(Exception):
+            self.close(retry=False)
+
     def get_query_status(self, sf_qid: str) -> QueryStatus:
         """Retrieves the status of query with sf_qid.