databricks-sql-connector 3.7.0__tar.gz → 4.0.0__tar.gz

{databricks_sql_connector-3.7.0 → databricks_sql_connector-4.0.0}/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Release History
 
+# 4.0.0 (2025-01-19)
+
+- Split the connector into two separate packages: `databricks-sql-connector` and `databricks-sqlalchemy`. The `databricks-sql-connector` package contains the core functionality of the connector, while the `databricks-sqlalchemy` package contains the SQLAlchemy dialect for the connector. 
+- Pyarrow dependency is now optional in `databricks-sql-connector`. Users needing arrow are supposed to explicitly install pyarrow
+
+# 3.7.1 (2025-01-07)
+
+- Relaxed the number of Http retry attempts (databricks/databricks-sql-python#486 by @jprakash-db)
+
 # 3.7.0 (2024-12-23)
 
 - Fix: Incorrect number of rows fetched in inline results when fetching results with FETCH_NEXT orientation (databricks/databricks-sql-python#479 by @jprakash-db)

{databricks_sql_connector-3.7.0 → databricks_sql_connector-4.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: databricks-sql-connector
-Version: 3.7.0
+Version: 4.0.0
 Summary: Databricks SQL Connector for Python
 License: Apache-2.0
 Author: Databricks
@@ -13,18 +13,15 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
-Provides-Extra: alembic
-Provides-Extra: sqlalchemy
-Requires-Dist: alembic (>=1.0.11,<2.0.0) ; extra == "alembic"
+Provides-Extra: pyarrow
 Requires-Dist: lz4 (>=4.0.2,<5.0.0)
-Requires-Dist: numpy (>=1.16.6) ; python_version >= "3.8" and python_version < "3.11"
-Requires-Dist: numpy (>=1.23.4) ; python_version >= "3.11"
+Requires-Dist: numpy (>=1.16.6,<2.0.0) ; python_version >= "3.8" and python_version < "3.11"
+Requires-Dist: numpy (>=1.23.4,<2.0.0) ; python_version >= "3.11"
 Requires-Dist: oauthlib (>=3.1.0,<4.0.0)
 Requires-Dist: openpyxl (>=3.0.10,<4.0.0)
 Requires-Dist: pandas (>=1.2.5,<2.3.0) ; python_version >= "3.8"
-Requires-Dist: pyarrow (>=14.0.1)
+Requires-Dist: pyarrow (>=14.0.1) ; extra == "pyarrow"
 Requires-Dist: requests (>=2.18.1,<3.0.0)
-Requires-Dist: sqlalchemy (>=2.0.21) ; extra == "sqlalchemy" or extra == "alembic"
 Requires-Dist: thrift (>=0.16.0,<0.21.0)
 Requires-Dist: urllib3 (>=1.26)
 Project-URL: Bug Tracker, https://github.com/databricks/databricks-sql-python/issues
@@ -36,9 +33,9 @@ Description-Content-Type: text/markdown
 [![PyPI](https://img.shields.io/pypi/v/databricks-sql-connector?style=flat-square)](https://pypi.org/project/databricks-sql-connector/)
 [![Downloads](https://pepy.tech/badge/databricks-sql-connector)](https://pepy.tech/project/databricks-sql-connector)
 
-The Databricks SQL Connector for Python allows you to develop Python applications that connect to Databricks clusters and SQL warehouses. It is a Thrift-based client with no dependencies on ODBC or JDBC. It conforms to the [Python DB API 2.0 specification](https://www.python.org/dev/peps/pep-0249/) and exposes a [SQLAlchemy](https://www.sqlalchemy.org/) dialect for use with tools like `pandas` and `alembic` which use SQLAlchemy to execute DDL. Use `pip install databricks-sql-connector[sqlalchemy]` to install with SQLAlchemy's dependencies. `pip install databricks-sql-connector[alembic]` will install alembic's dependencies.
+The Databricks SQL Connector for Python allows you to develop Python applications that connect to Databricks clusters and SQL warehouses. It is a Thrift-based client with no dependencies on ODBC or JDBC. It conforms to the [Python DB API 2.0 specification](https://www.python.org/dev/peps/pep-0249/).
 
-This connector uses Arrow as the data-exchange format, and supports APIs to directly fetch Arrow tables. Arrow tables are wrapped in the `ArrowQueue` class to provide a natural API to get several rows at a time.
+This connector uses Arrow as the data-exchange format, and supports APIs (e.g. `fetchmany_arrow`) to directly fetch Arrow tables. Arrow tables are wrapped in the `ArrowQueue` class to provide a natural API to get several rows at a time. [PyArrow](https://arrow.apache.org/docs/python/index.html) is required to enable this and use these APIs, you can install it via  `pip install pyarrow` or `pip install databricks-sql-connector[pyarrow]`.
 
 You are welcome to file an issue here for general use cases. You can also contact Databricks Support [here](help.databricks.com).
 
@@ -55,7 +52,12 @@ For the latest documentation, see
 
 ## Quickstart
 
-Install the library with `pip install databricks-sql-connector`
+### Installing the core library
+Install using `pip install databricks-sql-connector`
+
+### Installing the core library with PyArrow
+Install using `pip install databricks-sql-connector[pyarrow]`
+
 
 ```bash
 export DATABRICKS_HOST=********.databricks.com
@@ -93,6 +95,18 @@ or to a Databricks Runtime interactive cluster (e.g. /sql/protocolv1/o/123456789
 > to authenticate the target Databricks user account and needs to open the browser for authentication. So it 
 > can only run on the user's machine.
 
+## SQLAlchemy
+Starting from `databricks-sql-connector` version 4.0.0 SQLAlchemy support has been extracted to a new library `databricks-sqlalchemy`.
+
+- Github repository [databricks-sqlalchemy github](https://github.com/databricks/databricks-sqlalchemy)
+- PyPI [databricks-sqlalchemy pypi](https://pypi.org/project/databricks-sqlalchemy/)
+
+### Quick SQLAlchemy guide
+Users can now choose between using the SQLAlchemy v1 or SQLAlchemy v2 dialects with the connector core
+
+- Install the latest SQLAlchemy v1 using `pip install databricks-sqlalchemy~=1.0`
+- Install SQLAlchemy v2 using `pip install databricks-sqlalchemy`
+
 
 ## Contributing
 

{databricks_sql_connector-3.7.0 → databricks_sql_connector-4.0.0}/README.md

@@ -3,9 +3,9 @@
 [![PyPI](https://img.shields.io/pypi/v/databricks-sql-connector?style=flat-square)](https://pypi.org/project/databricks-sql-connector/)
 [![Downloads](https://pepy.tech/badge/databricks-sql-connector)](https://pepy.tech/project/databricks-sql-connector)
 
-The Databricks SQL Connector for Python allows you to develop Python applications that connect to Databricks clusters and SQL warehouses. It is a Thrift-based client with no dependencies on ODBC or JDBC. It conforms to the [Python DB API 2.0 specification](https://www.python.org/dev/peps/pep-0249/) and exposes a [SQLAlchemy](https://www.sqlalchemy.org/) dialect for use with tools like `pandas` and `alembic` which use SQLAlchemy to execute DDL. Use `pip install databricks-sql-connector[sqlalchemy]` to install with SQLAlchemy's dependencies. `pip install databricks-sql-connector[alembic]` will install alembic's dependencies.
+The Databricks SQL Connector for Python allows you to develop Python applications that connect to Databricks clusters and SQL warehouses. It is a Thrift-based client with no dependencies on ODBC or JDBC. It conforms to the [Python DB API 2.0 specification](https://www.python.org/dev/peps/pep-0249/).
 
-This connector uses Arrow as the data-exchange format, and supports APIs to directly fetch Arrow tables. Arrow tables are wrapped in the `ArrowQueue` class to provide a natural API to get several rows at a time.
+This connector uses Arrow as the data-exchange format, and supports APIs (e.g. `fetchmany_arrow`) to directly fetch Arrow tables. Arrow tables are wrapped in the `ArrowQueue` class to provide a natural API to get several rows at a time. [PyArrow](https://arrow.apache.org/docs/python/index.html) is required to enable this and use these APIs, you can install it via  `pip install pyarrow` or `pip install databricks-sql-connector[pyarrow]`.
 
 You are welcome to file an issue here for general use cases. You can also contact Databricks Support [here](help.databricks.com).
 
@@ -22,7 +22,12 @@ For the latest documentation, see
 
 ## Quickstart
 
-Install the library with `pip install databricks-sql-connector`
+### Installing the core library
+Install using `pip install databricks-sql-connector`
+
+### Installing the core library with PyArrow
+Install using `pip install databricks-sql-connector[pyarrow]`
+
 
 ```bash
 export DATABRICKS_HOST=********.databricks.com
@@ -60,6 +65,18 @@ or to a Databricks Runtime interactive cluster (e.g. /sql/protocolv1/o/123456789
 > to authenticate the target Databricks user account and needs to open the browser for authentication. So it 
 > can only run on the user's machine.
 
+## SQLAlchemy
+Starting from `databricks-sql-connector` version 4.0.0 SQLAlchemy support has been extracted to a new library `databricks-sqlalchemy`.
+
+- Github repository [databricks-sqlalchemy github](https://github.com/databricks/databricks-sqlalchemy)
+- PyPI [databricks-sqlalchemy pypi](https://pypi.org/project/databricks-sqlalchemy/)
+
+### Quick SQLAlchemy guide
+Users can now choose between using the SQLAlchemy v1 or SQLAlchemy v2 dialects with the connector core
+
+- Install the latest SQLAlchemy v1 using `pip install databricks-sqlalchemy~=1.0`
+- Install SQLAlchemy v2 using `pip install databricks-sqlalchemy`
+
 
 ## Contributing
 

{databricks_sql_connector-3.7.0 → databricks_sql_connector-4.0.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "databricks-sql-connector"
-version = "3.7.0"
+version = "4.0.0"
 description = "Databricks SQL Connector for Python"
 authors = ["Databricks <databricks-sql-connector-maintainers@databricks.com>"]
 license = "Apache-2.0"
@@ -14,23 +14,19 @@ thrift = ">=0.16.0,<0.21.0"
 pandas = [
     { version = ">=1.2.5,<2.3.0", python = ">=3.8" }
 ]
-pyarrow = ">=14.0.1"
-
 lz4 = "^4.0.2"
 requests = "^2.18.1"
 oauthlib = "^3.1.0"
 numpy = [
-    { version = ">=1.16.6", python = ">=3.8,<3.11" },
-    { version = ">=1.23.4", python = ">=3.11" },
+    { version = "^1.16.6", python = ">=3.8,<3.11" },
+    { version = "^1.23.4", python = ">=3.11" },
 ]
-sqlalchemy = { version = ">=2.0.21", optional = true }
 openpyxl = "^3.0.10"
-alembic = { version = "^1.0.11", optional = true }
 urllib3 = ">=1.26"
+pyarrow = { version = ">=14.0.1", optional=true }
 
 [tool.poetry.extras]
-sqlalchemy = ["sqlalchemy"]
-alembic = ["sqlalchemy", "alembic"]
+pyarrow = ["pyarrow"]
 
 [tool.poetry.dev-dependencies]
 pytest = "^7.1.2"
@@ -43,9 +39,6 @@ pytest-dotenv = "^0.5.2"
 "Homepage" = "https://github.com/databricks/databricks-sql-python"
 "Bug Tracker" = "https://github.com/databricks/databricks-sql-python/issues"
 
-[tool.poetry.plugins."sqlalchemy.dialects"]
-"databricks" = "databricks.sqlalchemy:DatabricksDialect"
-
 [build-system]
 requires = ["poetry-core>=1.0.0"]
 build-backend = "poetry.core.masonry.api"
@@ -62,5 +55,5 @@ markers = {"reviewed" = "Test case has been reviewed by Databricks"}
 minversion = "6.0"
 log_cli = "false"
 log_cli_level = "INFO"
-testpaths = ["tests", "src/databricks/sqlalchemy/test_local"]
-env_files = ["test.env"]
+testpaths = ["tests"]
+env_files = ["test.env"]

{databricks_sql_connector-3.7.0 → databricks_sql_connector-4.0.0}/src/databricks/sql/__init__.py

@@ -68,7 +68,7 @@ DATETIME = DBAPITypeObject("timestamp")
 DATE = DBAPITypeObject("date")
 ROWID = DBAPITypeObject()
 
-__version__ = "3.7.0"
+__version__ = "4.0.0"
 USER_AGENT_NAME = "PyDatabricksSqlConnector"
 
 # These two functions are pyhive legacy

{databricks_sql_connector-3.7.0 → databricks_sql_connector-4.0.0}/src/databricks/sql/client.py

@@ -54,6 +54,13 @@ from databricks.sql.thrift_api.TCLIService.ttypes import (
 
 logger = logging.getLogger(__name__)
 
+if pyarrow is None:
+    logger.warning(
+        "[WARN] pyarrow is not installed by default since databricks-sql-connector 4.0.0,"
+        "any arrow specific api (e.g. fetchmany_arrow) and cloud fetch will be disabled."
+        "If you need these features, please run pip install pyarrow or pip install databricks-sql-connector[pyarrow] to install"
+    )
+
 DEFAULT_RESULT_BUFFER_SIZE_BYTES = 104857600
 DEFAULT_ARRAY_SIZE = 100000
 

{databricks_sql_connector-3.7.0 → databricks_sql_connector-4.0.0}/src/databricks/sql/thrift_backend.py

@@ -67,7 +67,7 @@ DEFAULT_SOCKET_TIMEOUT = float(900)
 _retry_policy = {  # (type, default, min, max)
     "_retry_delay_min": (float, 1, 0.1, 60),
     "_retry_delay_max": (float, 30, 5, 3600),
-    "_retry_stop_after_attempts_count": (int, 5, 1, 60),
+    "_retry_stop_after_attempts_count": (int, 30, 1, 60),
     "_retry_stop_after_attempts_duration": (float, 900, 1, 86400),
     "_retry_delay_default": (float, 5, 1, 60),
 }

databricks_sql_connector-3.7.0/src/databricks/sqlalchemy/README.sqlalchemy.md

@@ -1,203 +0,0 @@
-## Databricks dialect for SQLALchemy 2.0
-
-The Databricks dialect for SQLAlchemy serves as bridge between [SQLAlchemy](https://www.sqlalchemy.org/) and the Databricks SQL Python driver. The dialect is included with `databricks-sql-connector==3.0.0` and above. A working example demonstrating usage can be found in `examples/sqlalchemy.py`.
-
-## Usage with SQLAlchemy <= 2.0
-A SQLAlchemy 1.4 compatible dialect was first released in connector [version 2.4](https://github.com/databricks/databricks-sql-python/releases/tag/v2.4.0). Support for SQLAlchemy 1.4 was dropped from the dialect as part of `databricks-sql-connector==3.0.0`. To continue using the dialect with SQLAlchemy 1.x, you can use `databricks-sql-connector^2.4.0`.
-
-
-## Installation
-
-To install the dialect and its dependencies:
-
-```shell
-pip install databricks-sql-connector[sqlalchemy]
-```
-
-If you also plan to use `alembic` you can alternatively run:
-
-```shell
-pip install databricks-sql-connector[alembic]
-```
-
-## Connection String
-
-Every SQLAlchemy application that connects to a database needs to use an [Engine](https://docs.sqlalchemy.org/en/20/tutorial/engine.html#tutorial-engine), which you can create by passing a connection string to `create_engine`. The connection string must include these components:
-
-1. Host
-2. HTTP Path for a compute resource
-3. API access token
-4. Initial catalog for the connection
-5. Initial schema for the connection
-
-**Note: Our dialect is built and tested on workspaces with Unity Catalog enabled. Support for the `hive_metastore` catalog is untested.**
-
-For example:
-
-```python
-import os
-from sqlalchemy import create_engine
-
-host = os.getenv("DATABRICKS_SERVER_HOSTNAME")
-http_path = os.getenv("DATABRICKS_HTTP_PATH")
-access_token = os.getenv("DATABRICKS_TOKEN")
-catalog = os.getenv("DATABRICKS_CATALOG")
-schema = os.getenv("DATABRICKS_SCHEMA")
-
-engine = create_engine(
-    f"databricks://token:{access_token}@{host}?http_path={http_path}&catalog={catalog}&schema={schema}"
-    )
-```
-
-## Types
-
-The [SQLAlchemy type hierarchy](https://docs.sqlalchemy.org/en/20/core/type_basics.html) contains backend-agnostic type implementations (represented in CamelCase) and backend-specific types (represented in UPPERCASE). The majority of SQLAlchemy's [CamelCase](https://docs.sqlalchemy.org/en/20/core/type_basics.html#the-camelcase-datatypes) types are supported. This means that a SQLAlchemy application using these types should "just work" with Databricks.
-
-|SQLAlchemy Type|Databricks SQL Type|
-|-|-|
-[`BigInteger`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.BigInteger)| [`BIGINT`](https://docs.databricks.com/en/sql/language-manual/data-types/bigint-type.html)
-[`LargeBinary`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.LargeBinary)| (not supported)|
-[`Boolean`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Boolean)| [`BOOLEAN`](https://docs.databricks.com/en/sql/language-manual/data-types/boolean-type.html)
-[`Date`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Date)| [`DATE`](https://docs.databricks.com/en/sql/language-manual/data-types/date-type.html)
-[`DateTime`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.DateTime)| [`TIMESTAMP_NTZ`](https://docs.databricks.com/en/sql/language-manual/data-types/timestamp-ntz-type.html)|
-[`Double`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Double)| [`DOUBLE`](https://docs.databricks.com/en/sql/language-manual/data-types/double-type.html)
-[`Enum`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Enum)| (not supported)|
-[`Float`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Float)| [`FLOAT`](https://docs.databricks.com/en/sql/language-manual/data-types/float-type.html)
-[`Integer`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Integer)| [`INT`](https://docs.databricks.com/en/sql/language-manual/data-types/int-type.html)
-[`Numeric`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Numeric)| [`DECIMAL`](https://docs.databricks.com/en/sql/language-manual/data-types/decimal-type.html)|
-[`PickleType`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.PickleType)| (not supported)|
-[`SmallInteger`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.SmallInteger)| [`SMALLINT`](https://docs.databricks.com/en/sql/language-manual/data-types/smallint-type.html)
-[`String`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.String)| [`STRING`](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html)|
-[`Text`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Text)| [`STRING`](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html)|
-[`Time`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Time)| [`STRING`](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html)|
-[`Unicode`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Unicode)| [`STRING`](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html)|
-[`UnicodeText`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.UnicodeText)| [`STRING`](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html)|
-[`Uuid`](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.Uuid)| [`STRING`](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html)
-
-In addition, the dialect exposes three UPPERCASE SQLAlchemy types which are specific to Databricks:
-
-- [`databricks.sqlalchemy.TINYINT`](https://docs.databricks.com/en/sql/language-manual/data-types/tinyint-type.html)
-- [`databricks.sqlalchemy.TIMESTAMP`](https://docs.databricks.com/en/sql/language-manual/data-types/timestamp-type.html)
-- [`databricks.sqlalchemy.TIMESTAMP_NTZ`](https://docs.databricks.com/en/sql/language-manual/data-types/timestamp-ntz-type.html)
-
-
-### `LargeBinary()` and `PickleType()`
-
-Databricks Runtime doesn't currently support binding of binary values in SQL queries, which is a pre-requisite for this functionality in SQLAlchemy.
-
-## `Enum()` and `CHECK` constraints
-
-Support for `CHECK` constraints is not implemented in this dialect. Support is planned for a future release.
-
-SQLAlchemy's `Enum()` type depends on `CHECK` constraints and is therefore not yet supported.
-
-### `DateTime()`, `TIMESTAMP_NTZ()`, and `TIMESTAMP()`
-
-Databricks Runtime provides two datetime-like types: `TIMESTAMP` which is always timezone-aware and `TIMESTAMP_NTZ` which is timezone agnostic. Both types can be imported from `databricks.sqlalchemy` and used in your models.
-
-The SQLAlchemy documentation indicates that `DateTime()` is not timezone-aware by default. So our dialect maps this type to `TIMESTAMP_NTZ()`. In practice, you should never need to use `TIMESTAMP_NTZ()` directly. Just use `DateTime()`.
-
-If you need your field to be timezone-aware, you can import `TIMESTAMP()` and use it instead.
-
-_Note that SQLAlchemy documentation suggests that you can declare a `DateTime()` with `timezone=True` on supported backends. However, if you do this with the Databricks dialect, the `timezone` argument will be ignored._
-
-```python
-from sqlalchemy import DateTime
-from databricks.sqlalchemy import TIMESTAMP
-
-class SomeModel(Base):
-    some_date_without_timezone  = DateTime()
-    some_date_with_timezone     = TIMESTAMP()
-```
-
-### `String()`, `Text()`, `Unicode()`, and `UnicodeText()`
-
-Databricks Runtime doesn't support length limitations for `STRING` fields. Therefore `String()` or `String(1)` or `String(255)` will all produce identical DDL. Since `Text()`, `Unicode()`, `UnicodeText()` all use the same underlying type in Databricks SQL, they will generate equivalent DDL.
-
-### `Time()`
-
-Databricks Runtime doesn't have a native time-like data type. To implement this type in SQLAlchemy, our dialect stores SQLAlchemy `Time()` values in a `STRING` field. Unlike `DateTime` above, this type can optionally support timezone awareness (since the dialect is in complete control of the strings that we write to the Delta table).
-
-```python
-from sqlalchemy import Time
-
-class SomeModel(Base):
-    time_tz     = Time(timezone=True)
-    time_ntz    = Time()
-```
-
-
-# Usage Notes
-
-## `Identity()` and `autoincrement`
-
-Identity and generated value support is currently limited in this dialect.
-
-When defining models, SQLAlchemy types can accept an [`autoincrement`](https://docs.sqlalchemy.org/en/20/core/metadata.html#sqlalchemy.schema.Column.params.autoincrement) argument. In our dialect, this argument is currently ignored. To create an auto-incrementing field in your model you can pass in an explicit [`Identity()`](https://docs.sqlalchemy.org/en/20/core/defaults.html#identity-ddl) instead.
-
-Furthermore, in Databricks Runtime, only `BIGINT` fields can be configured to auto-increment. So in SQLAlchemy, you must use the `BigInteger()` type.
-
-```python
-from sqlalchemy import Identity, String
-
-class SomeModel(Base):
-    id      = BigInteger(Identity())
-    value   = String()
-```
-
-When calling `Base.metadata.create_all()`, the executed DDL will include `GENERATED ALWAYS AS IDENTITY` for the `id` column. This is useful when using SQLAlchemy to generate tables. However, as of this writing, `Identity()` constructs are not captured when SQLAlchemy reflects a table's metadata (support for this is planned).
-
-## Parameters
-
-`databricks-sql-connector` supports two approaches to parameterizing SQL queries: native and inline. Our SQLAlchemy 2.0 dialect always uses the native approach and is therefore limited to DBR 14.2 and above. If you are writing parameterized queries to be executed by SQLAlchemy, you must use the "named" paramstyle (`:param`). Read more about parameterization in `docs/parameters.md`.
-
-## Usage with pandas
-
-Use [`pandas.DataFrame.to_sql`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html) and [`pandas.read_sql`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sql.html#pandas.read_sql) to write and read from Databricks SQL. These methods both accept a SQLAlchemy connection to interact with Databricks.
-
-### Read from Databricks SQL into pandas
-```python
-from sqlalchemy import create_engine
-import pandas as pd
-
-engine = create_engine("databricks://token:dapi***@***.cloud.databricks.com?http_path=***&catalog=main&schema=test")
-with engine.connect() as conn:
-    # This will read the contents of `main.test.some_table`
-    df = pd.read_sql("some_table", conn)
-```
-
-### Write to Databricks SQL from pandas
-
-```python
-from sqlalchemy import create_engine
-import pandas as pd
-
-engine = create_engine("databricks://token:dapi***@***.cloud.databricks.com?http_path=***&catalog=main&schema=test")
-squares = [(i, i * i) for i in range(100)]
-df = pd.DataFrame(data=squares,columns=['x','x_squared'])
-
-with engine.connect() as conn:
-    # This will write the contents of `df` to `main.test.squares`
-    df.to_sql('squares',conn)
-```
-
-## [`PrimaryKey()`](https://docs.sqlalchemy.org/en/20/core/constraints.html#sqlalchemy.schema.PrimaryKeyConstraint) and [`ForeignKey()`](https://docs.sqlalchemy.org/en/20/core/constraints.html#defining-foreign-keys)
-
-Unity Catalog workspaces in Databricks support PRIMARY KEY and FOREIGN KEY constraints. _Note that Databricks Runtime does not enforce the integrity of FOREIGN KEY constraints_. You can establish a primary key by setting `primary_key=True` when defining a column.
-
-When building `ForeignKey` or `ForeignKeyConstraint` objects, you must specify a `name` for the constraint.
-
-If your model definition requires a self-referential FOREIGN KEY constraint, you must include `use_alter=True` when defining the relationship.
-
-```python
-from sqlalchemy import Table, Column, ForeignKey, BigInteger, String
-
-users = Table(
-    "users",
-    metadata_obj,
-    Column("id", BigInteger, primary_key=True),
-    Column("name", String(), nullable=False),
-    Column("email", String()),
-    Column("manager_id", ForeignKey("users.id", name="fk_users_manager_id_x_users_id", use_alter=True))
-)
-```

databricks_sql_connector-3.7.0/src/databricks/sqlalchemy/README.tests.md

@@ -1,44 +0,0 @@
-## SQLAlchemy Dialect Compliance Test Suite with Databricks
-
-The contents of the `test/` directory follow the SQLAlchemy developers' [guidance] for running the reusable dialect compliance test suite. Since not every test in the suite is applicable to every dialect, two options are provided to skip tests:
-
-- Any test can be skipped by subclassing its parent class, re-declaring the test-case and adding a `pytest.mark.skip` directive.
-- Any test that is decorated with a `@requires` decorator can be skipped by marking the indicated requirement as `.closed()` in `requirements.py`
-
-We prefer to skip test cases directly with the first method wherever possible. We only mark requirements as `closed()` if there is no easier option to avoid a test failure. This principally occurs in test cases where the same test in the suite is parametrized, and some parameter combinations are conditionally skipped depending on `requirements.py`. If we skip the entire test method, then we skip _all_ permutations, not just the combinations we don't support.
-
-## Regression, Unsupported, and Future test cases
-
-We maintain three files of test cases that we import from the SQLAlchemy source code:
-
-* **`_regression.py`** contains all the tests cases with tests that we expect to pass for our dialect. Each one is marked with `pytest.mark.reiewed` to indicate that we've evaluated it for relevance. This file only contains base class declarations.
-* **`_unsupported.py`** contains test cases that fail because of missing features in Databricks. We mark them as skipped with a `SkipReason` enumeration. If Databricks comes to support these features, those test or entire classes can be moved to `_regression.py`.
-* **`_future.py`** contains test cases that fail because of missing features in the dialect itself, but which _are_ supported by Databricks generally. We mark them as skipped with a `FutureFeature` enumeration. These are features that have not been prioritised or that do not violate our acceptance criteria. All of these test cases will eventually move to either `_regression.py`.
-
-In some cases, only certain tests in class should be skipped with a `SkipReason` or `FutureFeature` justification. In those cases, we import the class into `_regression.py`, then import it from there into one or both of `_future.py` and `_unsupported.py`. If a class needs to be "touched" by regression, unsupported, and future, the class will be imported in that order. If an entire class should be skipped, then we do not import it into `_regression.py` at all.
-
-We maintain `_extra.py` with test cases that depend on SQLAlchemy's reusable dialect test fixtures but which are specific to Databricks (e.g TinyIntegerTest).
-
-## Running the reusable dialect tests
-
-```
-poetry shell
-cd src/databricks/sqlalchemy/test
-python -m pytest test_suite.py --dburi \
-  "databricks://token:$access_token@$host?http_path=$http_path&catalog=$catalog&schema=$schema"
-```
-
-Whatever schema you pass in the `dburi` argument should be empty. Some tests also require the presence of an empty schema named `test_schema`. Note that we plan to implement our own `provision.py` which SQLAlchemy can automatically use to create an empty schema for testing. But for now this is a manual process.
-
-You can run only reviewed tests by appending `-m "reviewed"` to the test runner invocation.
-
-You can run only the unreviewed tests by appending `-m "not reviewed"` instead.
-
-Note that because these tests depend on SQLAlchemy's custom pytest plugin, they are not discoverable by IDE-based test runners like VSCode or PyCharm and must be invoked from a CLI.
-
-## Running local unit and e2e tests
-
-Apart from the SQLAlchemy reusable suite, we maintain our own unit and e2e tests under the `test_local/` directory. These can be invoked from a VSCode or Pycharm since they don't depend on a custom pytest plugin. Due to pytest's lookup order, the `pytest.ini` which is required for running the reusable dialect tests, also conflicts with VSCode and Pycharm's default pytest implementation and overrides the settings in `pyproject.toml`. So to run these tests, you can delete or rename `pytest.ini`.
-
-
-[guidance]: "https://github.com/sqlalchemy/sqlalchemy/blob/rel_2_0_22/README.dialects.rst"

databricks_sql_connector-3.7.0/src/databricks/sqlalchemy/__init__.py

@@ -1,4 +0,0 @@
-from databricks.sqlalchemy.base import DatabricksDialect
-from databricks.sqlalchemy._types import TINYINT, TIMESTAMP, TIMESTAMP_NTZ
-
-__all__ = ["TINYINT", "TIMESTAMP", "TIMESTAMP_NTZ"]

databricks_sql_connector-3.7.0/src/databricks/sqlalchemy/_ddl.py

@@ -1,100 +0,0 @@
-import re
-from sqlalchemy.sql import compiler, sqltypes
-import logging
-
-logger = logging.getLogger(__name__)
-
-
-class DatabricksIdentifierPreparer(compiler.IdentifierPreparer):
-    """https://docs.databricks.com/en/sql/language-manual/sql-ref-identifiers.html"""
-
-    legal_characters = re.compile(r"^[A-Z0-9_]+$", re.I)
-
-    def __init__(self, dialect):
-        super().__init__(dialect, initial_quote="`")
-
-
-class DatabricksDDLCompiler(compiler.DDLCompiler):
-    def post_create_table(self, table):
-        post = [" USING DELTA"]
-        if table.comment:
-            comment = self.sql_compiler.render_literal_value(
-                table.comment, sqltypes.String()
-            )
-            post.append("COMMENT " + comment)
-
-        post.append("TBLPROPERTIES('delta.feature.allowColumnDefaults' = 'enabled')")
-        return "\n".join(post)
-
-    def visit_unique_constraint(self, constraint, **kw):
-        logger.warning("Databricks does not support unique constraints")
-        pass
-
-    def visit_check_constraint(self, constraint, **kw):
-        logger.warning("This dialect does not support check constraints")
-        pass
-
-    def visit_identity_column(self, identity, **kw):
-        """When configuring an Identity() with Databricks, only the always option is supported.
-        All other options are ignored.
-
-        Note: IDENTITY columns must always be defined as BIGINT. An exception will be raised if INT is used.
-
-        https://www.databricks.com/blog/2022/08/08/identity-columns-to-generate-surrogate-keys-are-now-available-in-a-lakehouse-near-you.html
-        """
-        text = "GENERATED %s AS IDENTITY" % (
-            "ALWAYS" if identity.always else "BY DEFAULT",
-        )
-        return text
-
-    def visit_set_column_comment(self, create, **kw):
-        return "ALTER TABLE %s ALTER COLUMN %s COMMENT %s" % (
-            self.preparer.format_table(create.element.table),
-            self.preparer.format_column(create.element),
-            self.sql_compiler.render_literal_value(
-                create.element.comment, sqltypes.String()
-            ),
-        )
-
-    def visit_drop_column_comment(self, create, **kw):
-        return "ALTER TABLE %s ALTER COLUMN %s COMMENT ''" % (
-            self.preparer.format_table(create.element.table),
-            self.preparer.format_column(create.element),
-        )
-
-    def get_column_specification(self, column, **kwargs):
-        """
-        Emit a log message if a user attempts to set autoincrement=True on a column.
-        See comments in test_suite.py. We may implement implicit IDENTITY using this
-        feature in the future, similar to the Microsoft SQL Server dialect.
-        """
-        if column is column.table._autoincrement_column or column.autoincrement is True:
-            logger.warning(
-                "Databricks dialect ignores SQLAlchemy's autoincrement semantics. Use explicit Identity() instead."
-            )
-
-        colspec = super().get_column_specification(column, **kwargs)
-        if column.comment is not None:
-            literal = self.sql_compiler.render_literal_value(
-                column.comment, sqltypes.STRINGTYPE
-            )
-            colspec += " COMMENT " + literal
-
-        return colspec
-
-
-class DatabricksStatementCompiler(compiler.SQLCompiler):
-    def limit_clause(self, select, **kw):
-        """Identical to the default implementation of SQLCompiler.limit_clause except it writes LIMIT ALL instead of LIMIT -1,
-        since Databricks SQL doesn't support the latter.
-
-        https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-qry-select-limit.html
-        """
-        text = ""
-        if select._limit_clause is not None:
-            text += "\n LIMIT " + self.process(select._limit_clause, **kw)
-        if select._offset_clause is not None:
-            if select._limit_clause is None:
-                text += "\n LIMIT ALL"
-            text += " OFFSET " + self.process(select._offset_clause, **kw)
-        return text