snowflake-sqlalchemy 1.7.1__tar.gz → 1.7.2__tar.gz

{snowflake_sqlalchemy-1.7.1 → snowflake_sqlalchemy-1.7.2}/DESCRIPTION.md

@@ -9,12 +9,19 @@ Source code is also available at:
 
 # Release Notes
 
+- v1.7.2(December 18, 2024)
+  - Fix quoting of `_` as column name
+  - Fix index columns was not being reflected
+  - Fix index reflection cache not working
+  - Add support for structured OBJECT datatype
+  - Add support for structured ARRAY datatype
+
 - v1.7.1(December 02, 2024)
   - Add support for partition by to copy into <location>
   - Fix BOOLEAN type not found in snowdialect
+  - Add support for autocommit Isolation Level
 
 - v1.7.0(November 21, 2024)
-
   - Add support for dynamic tables and required options
   - Add support for hybrid tables
   - Fixed SAWarning when registering functions with existing name in default namespace

{snowflake_sqlalchemy-1.7.1 → snowflake_sqlalchemy-1.7.2}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.4
 Name: snowflake-sqlalchemy
-Version: 1.7.1
+Version: 1.7.2
 Summary: Snowflake SQLAlchemy Dialect
 Project-URL: Changelog, https://github.com/snowflakedb/snowflake-sqlalchemy/blob/main/DESCRIPTION.md
 Project-URL: Documentation, https://docs.snowflake.com/en/user-guide/sqlalchemy.html
@@ -8,7 +8,8 @@ Project-URL: Homepage, https://www.snowflake.com/
 Project-URL: Issues, https://github.com/snowflakedb/snowflake-sqlalchemy/issues
 Project-URL: Source, https://github.com/snowflakedb/snowflake-sqlalchemy
 Author-email: "Snowflake Inc." <triage-snowpark-python-api-dl@snowflake.com>
-License: Apache-2.0
+License-Expression: Apache-2.0
+License-File: LICENSE.txt
 Keywords: Snowflake,analytics,cloud,database,db,warehouse
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console
@@ -45,6 +46,7 @@ Requires-Dist: pytest-cov; extra == 'development'
 Requires-Dist: pytest-rerunfailures; extra == 'development'
 Requires-Dist: pytest-timeout; extra == 'development'
 Requires-Dist: pytz; extra == 'development'
+Requires-Dist: setuptools; extra == 'development'
 Requires-Dist: syrupy==4.6.1; extra == 'development'
 Provides-Extra: pandas
 Requires-Dist: snowflake-connector-python[pandas]; extra == 'pandas'
@@ -419,6 +421,79 @@ data_object  = json.loads(row[1])
 data_array   = json.loads(row[2])
 ```
 
+### Structured Data Types Support
+
+This module defines custom SQLAlchemy types for Snowflake structured data, specifically for **Iceberg tables**.
+The types —**MAP**, **OBJECT**, and **ARRAY**— allow you to store complex data structures in your SQLAlchemy models.
+For detailed information, refer to the Snowflake [Structured data types](https://docs.snowflake.com/en/sql-reference/data-types-structured) documentation.
+
+---
+
+#### MAP
+
+The `MAP` type represents a collection of key-value pairs, where each key and value can have different types.
+
+- **Key Type**: The type of the keys (e.g., `TEXT`, `NUMBER`).
+- **Value Type**: The type of the values (e.g., `TEXT`, `NUMBER`).
+- **Not Null**: Whether `NULL` values are allowed (default is `False`).
+
+*Example Usage*
+
+```python
+IcebergTable(
+    table_name,
+    metadata,
+    Column("id", Integer, primary_key=True),
+    Column("map_col", MAP(NUMBER(10, 0), TEXT(16777216))),
+    external_volume="external_volume",
+    base_location="base_location",
+)
+```
+
+#### OBJECT
+
+The `OBJECT` type represents a semi-structured object with named fields. Each field can have a specific type, and you can also specify whether each field is nullable.
+
+- **Items Types**: A dictionary of field names and their types. The type can optionally include a nullable flag (`True` for not nullable, `False` for nullable, default is `False`).
+
+*Example Usage*
+
+```python
+IcebergTable(
+    table_name,
+    metadata,
+    Column("id", Integer, primary_key=True),
+    Column(
+        "object_col",
+        OBJECT(key1=(TEXT(16777216), False), key2=(NUMBER(10, 0), False)),
+        OBJECT(key1=TEXT(16777216), key2=NUMBER(10, 0)), # Without nullable flag
+    ),
+    external_volume="external_volume",
+    base_location="base_location",
+)
+```
+
+#### ARRAY
+
+The `ARRAY` type represents an ordered list of values, where each element has the same type. The type of the elements is defined when creating the array.
+
+- **Value Type**: The type of the elements in the array (e.g., `TEXT`, `NUMBER`).
+- **Not Null**: Whether `NULL` values are allowed (default is `False`).
+
+*Example Usage*
+
+```python
+IcebergTable(
+    table_name,
+    metadata,
+    Column("id", Integer, primary_key=True),
+    Column("array_col", ARRAY(TEXT(16777216))),
+    external_volume="external_volume",
+    base_location="base_location",
+)
+```
+
+
 ### CLUSTER BY Support
 
 Snowflake SQLAchemy supports the `CLUSTER BY` parameter for tables. For information about the parameter, see :doc:`/sql-reference/sql/create-table`.

{snowflake_sqlalchemy-1.7.1 → snowflake_sqlalchemy-1.7.2}/README.md

@@ -367,6 +367,79 @@ data_object  = json.loads(row[1])
 data_array   = json.loads(row[2])
 ```
 
+### Structured Data Types Support
+
+This module defines custom SQLAlchemy types for Snowflake structured data, specifically for **Iceberg tables**.
+The types —**MAP**, **OBJECT**, and **ARRAY**— allow you to store complex data structures in your SQLAlchemy models.
+For detailed information, refer to the Snowflake [Structured data types](https://docs.snowflake.com/en/sql-reference/data-types-structured) documentation.
+
+---
+
+#### MAP
+
+The `MAP` type represents a collection of key-value pairs, where each key and value can have different types.
+
+- **Key Type**: The type of the keys (e.g., `TEXT`, `NUMBER`).
+- **Value Type**: The type of the values (e.g., `TEXT`, `NUMBER`).
+- **Not Null**: Whether `NULL` values are allowed (default is `False`).
+
+*Example Usage*
+
+```python
+IcebergTable(
+    table_name,
+    metadata,
+    Column("id", Integer, primary_key=True),
+    Column("map_col", MAP(NUMBER(10, 0), TEXT(16777216))),
+    external_volume="external_volume",
+    base_location="base_location",
+)
+```
+
+#### OBJECT
+
+The `OBJECT` type represents a semi-structured object with named fields. Each field can have a specific type, and you can also specify whether each field is nullable.
+
+- **Items Types**: A dictionary of field names and their types. The type can optionally include a nullable flag (`True` for not nullable, `False` for nullable, default is `False`).
+
+*Example Usage*
+
+```python
+IcebergTable(
+    table_name,
+    metadata,
+    Column("id", Integer, primary_key=True),
+    Column(
+        "object_col",
+        OBJECT(key1=(TEXT(16777216), False), key2=(NUMBER(10, 0), False)),
+        OBJECT(key1=TEXT(16777216), key2=NUMBER(10, 0)), # Without nullable flag
+    ),
+    external_volume="external_volume",
+    base_location="base_location",
+)
+```
+
+#### ARRAY
+
+The `ARRAY` type represents an ordered list of values, where each element has the same type. The type of the elements is defined when creating the array.
+
+- **Value Type**: The type of the elements in the array (e.g., `TEXT`, `NUMBER`).
+- **Not Null**: Whether `NULL` values are allowed (default is `False`).
+
+*Example Usage*
+
+```python
+IcebergTable(
+    table_name,
+    metadata,
+    Column("id", Integer, primary_key=True),
+    Column("array_col", ARRAY(TEXT(16777216))),
+    external_volume="external_volume",
+    base_location="base_location",
+)
+```
+
+
 ### CLUSTER BY Support
 
 Snowflake SQLAchemy supports the `CLUSTER BY` parameter for tables. For information about the parameter, see :doc:`/sql-reference/sql/create-table`.

{snowflake_sqlalchemy-1.7.1 → snowflake_sqlalchemy-1.7.2}/pyproject.toml

@@ -47,6 +47,7 @@ path = "src/snowflake/sqlalchemy/version.py"
 development = [
   "pre-commit",
   "pytest",
+  "setuptools",
   "pytest-cov",
   "pytest-timeout",
   "pytest-rerunfailures",
@@ -74,6 +75,8 @@ exclude = ["/.github"]
 packages = ["src/snowflake"]
 
 [tool.hatch.envs.default]
+path = ".venv"
+type = "virtual"
 extra-dependencies = ["SQLAlchemy>=1.4.19,<2.1.0"]
 features = ["development", "pandas"]
 python = "3.8"

{snowflake_sqlalchemy-1.7.1 → snowflake_sqlalchemy-1.7.2}/src/snowflake/sqlalchemy/base.py

@@ -5,6 +5,7 @@
 import itertools
 import operator
 import re
+import string
 from typing import List
 
 from sqlalchemy import exc as sa_exc
@@ -114,7 +115,8 @@ RESERVED_WORDS = frozenset(
 AUTOCOMMIT_REGEXP = re.compile(
     r"\s*(?:UPDATE|INSERT|DELETE|MERGE|COPY)", re.I | re.UNICODE
 )
-
+# used for quoting identifiers ie. table names, column names, etc.
+ILLEGAL_INITIAL_CHARACTERS = frozenset({d for d in string.digits}.union({"_", "$"}))
 
 """
 Overwrite methods to handle Snowflake BCR change:
@@ -439,6 +441,7 @@ class SnowflakeORMSelectCompileState(context.ORMSelectCompileState):
 
 class SnowflakeIdentifierPreparer(compiler.IdentifierPreparer):
     reserved_words = {x.lower() for x in RESERVED_WORDS}
+    illegal_initial_characters = ILLEGAL_INITIAL_CHARACTERS
 
     def __init__(self, dialect, **kw):
         quote = '"'
@@ -1093,10 +1096,24 @@ class SnowflakeTypeCompiler(compiler.GenericTypeCompiler):
         )
 
     def visit_ARRAY(self, type_, **kw):
-        return "ARRAY"
+        if type_.is_semi_structured:
+            return "ARRAY"
+        not_null = f" {NOT_NULL}" if type_.not_null else ""
+        return f"ARRAY({type_.value_type.compile()}{not_null})"
 
     def visit_OBJECT(self, type_, **kw):
-        return "OBJECT"
+        if type_.is_semi_structured:
+            return "OBJECT"
+        else:
+            contents = []
+            for key in type_.items_types:
+
+                row_text = f"{key} {type_.items_types[key][0].compile()}"
+                # Type and not null is specified
+                if len(type_.items_types[key]) > 1:
+                    row_text += f"{' NOT NULL' if type_.items_types[key][1] else ''}"
+                contents.append(row_text)
+            return "OBJECT" if contents == [] else f"OBJECT({', '.join(contents)})"
 
     def visit_BLOB(self, type_, **kw):
         return "BINARY"

{snowflake_sqlalchemy-1.7.1 → snowflake_sqlalchemy-1.7.2}/src/snowflake/sqlalchemy/custom_types.py

@@ -1,9 +1,11 @@
 #
 # Copyright (c) 2012-2023 Snowflake Computing Inc. All rights reserved.
 #
+from typing import Optional, Tuple, Union
 
 import sqlalchemy.types as sqltypes
 import sqlalchemy.util as util
+from sqlalchemy.types import TypeEngine
 
 TEXT = sqltypes.VARCHAR
 CHARACTER = sqltypes.CHAR
@@ -38,7 +40,8 @@ class VARIANT(SnowflakeType):
 
 
 class StructuredType(SnowflakeType):
-    def __init__(self):
+    def __init__(self, is_semi_structured: bool = False):
+        self.is_semi_structured = is_semi_structured
         super().__init__()
 
 
@@ -57,13 +60,40 @@ class MAP(StructuredType):
         super().__init__()
 
 
-class OBJECT(SnowflakeType):
+class OBJECT(StructuredType):
     __visit_name__ = "OBJECT"
 
+    def __init__(self, **items_types: Union[TypeEngine, Tuple[TypeEngine, bool]]):
+        for key, value in items_types.items():
+            if not isinstance(value, tuple):
+                items_types[key] = (value, False)
 
-class ARRAY(SnowflakeType):
+        self.items_types = items_types
+        self.is_semi_structured = len(items_types) == 0
+        super().__init__()
+
+    def __repr__(self):
+        quote_char = "'"
+        return "OBJECT(%s)" % ", ".join(
+            [
+                f"{repr(key).strip(quote_char)}={repr(value)}"
+                for key, value in self.items_types.items()
+            ]
+        )
+
+
+class ARRAY(StructuredType):
     __visit_name__ = "ARRAY"
 
+    def __init__(
+        self,
+        value_type: Optional[sqltypes.TypeEngine] = None,
+        not_null: bool = False,
+    ):
+        self.value_type = value_type
+        self.not_null = not_null
+        super().__init__(is_semi_structured=value_type is None)
+
 
 class TIMESTAMP_TZ(SnowflakeType):
     __visit_name__ = "TIMESTAMP_TZ"

{snowflake_sqlalchemy-1.7.1 → snowflake_sqlalchemy-1.7.2}/src/snowflake/sqlalchemy/parser/custom_type_parser.py

@@ -1,5 +1,6 @@
 #
 # Copyright (c) 2012-2023 Snowflake Computing Inc. All rights reserved.
+from typing import List
 
 import sqlalchemy.types as sqltypes
 from sqlalchemy.sql.type_api import TypeEngine
@@ -48,11 +49,10 @@ ischema_names = {
     "DECIMAL": DECIMAL,
     "DOUBLE": DOUBLE,
     "FIXED": DECIMAL,
-    "FLOAT": FLOAT,  # Snowflake FLOAT datatype doesn't has parameters
+    "FLOAT": FLOAT,  # Snowflake FLOAT datatype doesn't have parameters
     "INT": INTEGER,
     "INTEGER": INTEGER,
     "NUMBER": _CUSTOM_DECIMAL,
-    # 'OBJECT': ?
     "REAL": REAL,
     "BYTEINT": SMALLINT,
     "SMALLINT": SMALLINT,
@@ -74,19 +74,22 @@ ischema_names = {
     "GEOMETRY": GEOMETRY,
 }
 
+NOT_NULL_STR = "NOT NULL"
 
-def extract_parameters(text: str) -> list:
+
+def tokenize_parameters(text: str, character_for_strip=",") -> list:
     """
     Extracts parameters from a comma-separated string, handling parentheses.
 
     :param text: A string with comma-separated parameters, which may include parentheses.
 
+    :param character_for_strip: A character to strip the text.
+
     :return: A list of parameters as strings.
 
     :example:
         For input `"a, (b, c), d"`, the output is `['a', '(b, c)', 'd']`.
     """
-
     output_parameters = []
     parameter = ""
     open_parenthesis = 0
@@ -97,9 +100,9 @@ def extract_parameters(text: str) -> list:
         elif c == ")":
             open_parenthesis -= 1
 
-        if open_parenthesis > 0 or c != ",":
+        if open_parenthesis > 0 or c != character_for_strip:
             parameter += c
-        elif c == ",":
+        elif c == character_for_strip:
             output_parameters.append(parameter.strip(" "))
             parameter = ""
     if parameter != "":
@@ -107,6 +110,21 @@ def extract_parameters(text: str) -> list:
     return output_parameters
 
 
+def parse_index_columns(columns: str) -> List[str]:
+    """
+    Parses a string with a list of columns for an index.
+
+    :param columns: A string with a list of columns for an index, which may include parentheses.
+    :param compiler: A SQLAlchemy compiler.
+
+    :return: A list of columns as strings.
+
+    :example:
+        For input `"[A, B, C]"`, the output is `['A', 'B', 'C']`.
+    """
+    return [column.strip() for column in columns.strip("[]").split(",")]
+
+
 def parse_type(type_text: str) -> TypeEngine:
     """
     Parses a type definition string and returns the corresponding SQLAlchemy type.
@@ -122,14 +140,17 @@ def parse_type(type_text: str) -> TypeEngine:
         parse_type("VARCHAR(255)")
         String(length=255)
     """
+
     index = type_text.find("(")
     type_name = type_text[:index] if index != -1 else type_text
+
     parameters = (
-        extract_parameters(type_text[index + 1 : -1]) if type_name != type_text else []
+        tokenize_parameters(type_text[index + 1 : -1]) if type_name != type_text else []
     )
 
     col_type_class = ischema_names.get(type_name, None)
     col_type_kw = {}
+
     if col_type_class is None:
         col_type_class = NullType
     else:
@@ -139,6 +160,10 @@ def parse_type(type_text: str) -> TypeEngine:
             col_type_kw = __parse_type_with_length_parameters(parameters)
         elif issubclass(col_type_class, MAP):
             col_type_kw = __parse_map_type_parameters(parameters)
+        elif issubclass(col_type_class, OBJECT):
+            col_type_kw = __parse_object_type_parameters(parameters)
+        elif issubclass(col_type_class, ARRAY):
+            col_type_kw = __parse_nullable_parameter(parameters)
         if col_type_kw is None:
             col_type_class = NullType
             col_type_kw = {}
@@ -146,33 +171,63 @@ def parse_type(type_text: str) -> TypeEngine:
     return col_type_class(**col_type_kw)
 
 
-def __parse_map_type_parameters(parameters):
-    if len(parameters) != 2:
+def __parse_object_type_parameters(parameters):
+    object_rows = {}
+    not_null_parts = NOT_NULL_STR.split(" ")
+    for parameter in parameters:
+        parameter_parts = tokenize_parameters(parameter, " ")
+        if len(parameter_parts) >= 2:
+            key = parameter_parts[0]
+            value_type = parse_type(parameter_parts[1])
+            if isinstance(value_type, NullType):
+                return None
+            not_null = (
+                len(parameter_parts) == 4
+                and parameter_parts[2] == not_null_parts[0]
+                and parameter_parts[3] == not_null_parts[1]
+            )
+            object_rows[key] = (value_type, not_null)
+    return object_rows
+
+
+def __parse_nullable_parameter(parameters):
+    if len(parameters) < 1:
+        return {}
+    elif len(parameters) > 1:
         return None
-
-    key_type_str = parameters[0]
-    value_type_str = parameters[1]
-    not_null_str = "NOT NULL"
-    not_null = False
+    parameter_str = parameters[0]
+    is_not_null = False
     if (
-        len(value_type_str) >= len(not_null_str)
-        and value_type_str[-len(not_null_str) :] == not_null_str
+        len(parameter_str) >= len(NOT_NULL_STR)
+        and parameter_str[-len(NOT_NULL_STR) :] == NOT_NULL_STR
     ):
-        not_null = True
-        value_type_str = value_type_str[: -len(not_null_str) - 1]
+        is_not_null = True
+        parameter_str = parameter_str[: -len(NOT_NULL_STR) - 1]
 
-    key_type: TypeEngine = parse_type(key_type_str)
-    value_type: TypeEngine = parse_type(value_type_str)
-    if isinstance(key_type, NullType) or isinstance(value_type, NullType):
+    value_type: TypeEngine = parse_type(parameter_str)
+    if isinstance(value_type, NullType):
         return None
 
     return {
-        "key_type": key_type,
         "value_type": value_type,
-        "not_null": not_null,
+        "not_null": is_not_null,
     }
 
 
+def __parse_map_type_parameters(parameters):
+    if len(parameters) != 2:
+        return None
+
+    key_type_str = parameters[0]
+    value_type_str = parameters[1]
+    key_type: TypeEngine = parse_type(key_type_str)
+    value_type = __parse_nullable_parameter([value_type_str])
+    if isinstance(value_type, NullType) or isinstance(key_type, NullType):
+        return None
+
+    return {"key_type": key_type, **value_type}
+
+
 def __parse_type_with_length_parameters(parameters):
     return (
         {"length": int(parameters[0])}