vastdb 2.0.2__tar.gz → 2.0.5__tar.gz

{vastdb-2.0.2 → vastdb-2.0.5}/CHANGELOG.md

@@ -4,6 +4,29 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [2.0.5] (2025-01-08)
+[2.0.4]: https://github.com/vast-data/vastdb_sdk/compare/v2.0.4...v2.0.5
+
+### Added
+- end-user impersonation for adbc connection configurable via Session
+
+
+## [2.0.4] (2025-01-06)
+[2.0.3]: https://github.com/vast-data/vastdb_sdk/compare/v2.0.3...v2.0.4
+
+### Changed
+- Fix race condition in select (get data) mechanism
+
+## [2.0.3] (2025-11-25)
+[2.0.2]: https://github.com/vast-data/vastdb_sdk/compare/v2.0.2...v2.0.3
+
+### Added
+- Support for vastdb's adbc driver
+  - Session now gets an Optional[AdbcDriver]
+  - AdbcDriver is created out of a local path to a driver `.so` (shared library) or a url
+- Vector Search capability
+  - `Table.vector_search` - Approximate Top-N closest vectors
+
 ## [2.0.2] (2025-10-22)
 [2.0.2]: https://github.com/vast-data/vastdb_sdk/compare/v2.0.1...v2.0.2
 

{vastdb-2.0.2 → vastdb-2.0.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: vastdb
-Version: 2.0.2
+Version: 2.0.5
 Summary: VAST Data SDK
 Home-page: https://github.com/vast-data/vastdb_sdk
 Author: VAST DATA
@@ -29,6 +29,7 @@ Requires-Dist: numpy
 Requires-Dist: requests
 Requires-Dist: xmltodict
 Requires-Dist: backoff==2.2.1
+Requires-Dist: adbc_driver_manager
 Dynamic: author
 Dynamic: author-email
 Dynamic: classifier

{vastdb-2.0.2 → vastdb-2.0.5}/README.md

@@ -19,16 +19,12 @@ For technical details about VAST Database architecture, see the [whitepaper](htt
 ### Requirements
 
 - Linux client with Python 3.10 - 3.13, and network access to the VAST Cluster
+- VAST Cluster release `5.0.0-sp10` or later
+  - If your VAST Cluster is running an older release, please contact customer.support@vastdata.com.
 - [Virtual IP pool configured with DNS service](https://support.vastdata.com/s/topic/0TOV40000000FThOAM/configuring-network-access-v50)
 - [S3 access & secret keys on the VAST cluster](https://support.vastdata.com/s/article/UUID-4d2e7e23-b2fb-7900-d98f-96c31a499626)
 - [Tabular identity policy with the proper permissions](https://support.vastdata.com/s/article/UUID-14322b60-d6a2-89ac-3df0-3dfbb6974182)
 
-### Required VAST Cluster release
-
-VAST DB Python SDK requires VAST Cluster release `5.0.0-sp10` or later.
-
-If your VAST Cluster is running an older release, please contact customer.support@vastdata.com.
-
 ### Installation
 
 ```bash
@@ -80,61 +76,81 @@ with session.transaction() as tx:
     # the transaction is automatically committed when exiting the context
 ```
 
-For configuration examples, see [here](docs/config.md).
+Note: the transaction must be remain open while the returned [pyarrow.RecordBatchReader](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatchReader.html) generator is being used.
 
 The list of supported data types can be found [here](docs/types.md).
 
-Note: the transaction must be remain open while the returned [pyarrow.RecordBatchReader](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatchReader.html) generator is being used.
+## Features
 
-## Use Cases
+### Select Performance
 
-### Filters and Projections
+The `Table.select()` method accepts a [QueryConfig](vastdb/config.py) object that modifies how the select is fulfilled.
 
-The SDK supports predicate and projection pushdown:
+The most important setting is the `data_endpoints` parameter that, when set, will allow the SDK to parallelize the select across multiple CNodes. Without this, only the CNode specified in the `connect()` will service the select.
 
 ```python
-    from ibis import _
+from vastdb.config import QueryConfig
 
-    # SELECT c1 FROM t WHERE (c2 > 2) AND (c3 IS NULL)
-    table.select(columns=['c1'],
-                 predicate=(_.c2 > 2) & _.c3.isnull())
+# load default configuration values
+cfg = QueryConfig()
 
-    # SELECT c2, c3 FROM t WHERE (c2 BETWEEN 0 AND 1) OR (c2 > 10)
-    table.select(columns=['c2', 'c3'],
-                 predicate=(_.c2.between(0, 1) | (_.c2 > 10))
+# set data_endpoints to CNode VIPs
+cfg.data_endpoints = [
+    "http://172.19.196.1",
+    "http://172.19.196.2",
+    "http://172.19.196.3",
+    "http://172.19.196.4",
+]
 
-    # SELECT * FROM t WHERE c3 LIKE '%substring%'
-    table.select(predicate=_.c3.contains('substring'))
+table.select(columns=['c1'], predicate=(_.c2 > 2), config=cfg)
 ```
 
-See [here for more details](docs/predicate.md).
+If using DNS with either TTL=0 or multi-response per the [Best Practice on Load Balancing CNodes](https://support.vastdata.com/s/document-item?bundleId=z-kb-articles-publications-prod&topicId=6058049537.html&_LANG=enus), passing in the same DNS name equal to the number of VIPs is a decent proxy.
+
+```python
+cfg.data_endpoints = ["http://vip-pool.v123-xy.VastENG.lab"] * 16  # assuming 16 VIPs in the pool
+```
 
-### Import a single Parquet file via S3 protocol
+### Filters and Projections
 
-You can efficiently create tables from Parquet files (without copying them via the client):
+The SDK supports predicate and projection pushdown using Ibis:
 
 ```python
-    with tempfile.NamedTemporaryFile() as f:
-        pa.parquet.write_table(arrow_table, f.name)
-        s3.put_object(Bucket='bucket-name', Key='staging/file.parquet', Body=f)
+from ibis import _
 
-    schema = tx.bucket('bucket-name').schema('schema-name')
-    table = util.create_table_from_files(
-        schema=schema, table_name='imported-table',
-        parquet_files=['/bucket-name/staging/file.parquet'])
+# SELECT c1 FROM t WHERE (c2 > 2) AND (c3 IS NULL)
+table.select(columns=['c1'],
+             predicate=(_.c2 > 2) & _.c3.isnull())
+
+# SELECT c2, c3 FROM t WHERE (c2 BETWEEN 0 AND 1) OR (c2 > 10)
+table.select(columns=['c2', 'c3'],
+             predicate=(_.c2.between(0, 1) | (_.c2 > 10))
+
+# SELECT * FROM t WHERE c3 LIKE '%substring%'
+table.select(predicate=_.c3.contains('substring'))
 ```
 
-### Import multiple Parquet files concurrently via S3 protocol
+See the [Predicate pushdown support document](docs/predicate.md) for more information on constructing predicates using Ibis.
 
-Import multiple files concurrently into a table (by using multiple CNodes' cores):
+### Import Parquet files via S3 protocol
+
+You can efficiently create tables from Parquet files that already exist in an S3 bucket on VAST without copying them via the client. If more than one file is included in `parquet_files` they will be loaded concurrently.
 
 ```python
-    schema = tx.bucket('bucket-name').schema('schema-name')
+with session.transaction() as tx:
+    schema = tx.bucket('database-name').schema('schema-name')
     table = util.create_table_from_files(
-        schema=schema, table_name='large-imported-table',
-        parquet_files=[f'/bucket-name/staging/file{i}.parquet' for i in range(10)])
+        schema=schema, table_name='imported-table',
+        parquet_files=['/bucket-name/staging/file.parquet'])
 ```
 
+If the table already exists, you can use the `table.import_files()` method to add more data to the table from Parquet files that already exist in an S3 bucket on VAST.
+
+```python
+with session.transaction() as tx:
+    table = tx.bucket('database-name').schema('schema-name').table('table-name')
+    table.import_files(["/bucket-name/staging/file2.parquet"])
+```
 
 ### Semi-sorted Projections
 
@@ -158,9 +174,9 @@ batches = snaps[0].schema('schema-name').table('table-name').select()
 
 ## Interactive and Non-Interactive Workflows
 
-Interactive `Table` created via the `Schema` object (`tx.bucket('..').schema('..').table('..')`) loads metadata and stats eagrly allowing for interactive development.
+A `Table` created via the `Schema` object (`tx.bucket('..').schema('..').table('..')`) loads metadata and stats eagerly allowing for interactive development. Each object (bucket, schema, table) requires one or more round-trips to the server and `.table()` will fetch the full table schema.
 
-For more efficient use-cases where the extra round-trips to the server hurt use the Non-Interactive workflow.
+It's generally more efficient to use the `TableMetadata` interface that allows for both lazy loading of the schemas as it's needed, as well as allowing reusing the metadata across transactions.
 
 ```python
 # load the table schema & stats into an object we can use across transactions
@@ -172,7 +188,7 @@ with session.transaction() as tx:
 table_md = TableMetadata(TableRef("bucket-name", "schema-name", "table-name"),
                          arrow_schema=<some-arrow-schema>)
 
-# now we can reuse without the overhead of reloading the schema and stats,
+# now we can reuse it without the overhead of reloading the schema and stats,
 # such as for inserts:
 with session.transaction() as tx:
     table = tx.table_from_metadata(table_md)
@@ -188,6 +204,15 @@ with session.transaction() as tx:
     print(results)
 ```
 
+Some table operations, like `table.import_files()`, does not require the client to know the table schema, and using the `TableMetadata` interface will bypass fetching the schema entirely.
+
+```python
+table_md = TableMetadata(TableRef("bucket-name", "schema-name", "table-name"))
+
+with session.transaction() as tx:
+    table = tx.table_from_metadata(table_md)
+    table.import_files(["/bucket-name/staging/file2.parquet"])
+```
 
 ## Post-processing
 
@@ -248,9 +273,6 @@ with session.transaction() as tx:
     print(distinct_elements)
 ```
 
-
-
-
 ## More Information
 
 See these blog posts for more examples:
@@ -259,5 +281,3 @@ See these blog posts for more examples:
 - https://vastdata.com/blog/the-vast-catalog-in-action-part-2
 
 See also the [full Vast DB Python SDK documentation](https://vastdb-sdk.readthedocs.io/en/latest/)
-
-

{vastdb-2.0.2 → vastdb-2.0.5}/requirements.txt

@@ -9,3 +9,4 @@ numpy
 requests
 xmltodict
 backoff==2.2.1
+adbc_driver_manager

{vastdb-2.0.2 → vastdb-2.0.5}/setup.py

@@ -29,7 +29,7 @@ setup(
     name='vastdb',
     python_requires='>=3.10.0',
     description='VAST Data SDK',
-    version='2.0.2' + suffix,
+    version='2.0.5' + suffix,
     url='https://github.com/vast-data/vastdb_sdk',
     author='VAST DATA',
     author_email='hello@vastdata.com',

vastdb-2.0.5/vastdb/_adbc.py

@@ -0,0 +1,205 @@
+import hashlib
+import logging
+import os
+import urllib.request
+from typing import Optional
+
+import pyarrow as pa
+import sqlglot
+from adbc_driver_manager.dbapi import Connection, Cursor, connect
+from sqlglot import exp
+
+from vastdb._internal import VectorIndex
+from vastdb._table_interface import IbisPredicate
+from vastdb.table_metadata import TableRef
+
+log = logging.getLogger(__name__)
+
+
+TXID_OVERRIDE_PROPERTY: str = "vast.db.external_txid"
+END_USER_PROPERTY: str = "vast.db.end_user"
+VAST_DIST_ALIAS = "vast_pysdk_vector_dist"
+DEFAULT_ADBC_DRIVER_CACHE_DIR: str = "~/.vast/adbc_drivers_cache"
+DEFAULT_ADBC_DRIVER_CACHE_BY_URL_DIR: str = f"{DEFAULT_ADBC_DRIVER_CACHE_DIR}/by_url"
+
+
+class LocalAdbcDriverNotFound(Exception):
+    """LocalAdbcDriverNotFound."""
+
+    pass
+
+
+class RemoteAdbcDriverDownloadFailed(Exception):
+    """RemoteAdbcDriverDownloadFailed."""
+
+    pass
+
+
+class AdbcDriver:
+    _local_path: str
+
+    def __init__(self, local_path: str):
+        self._local_path = local_path
+
+    @staticmethod
+    def from_local_path(local_path: str) -> "AdbcDriver":
+        """AdbcDriver from a local_path to shared-library."""
+        if not os.path.exists(local_path):
+            raise LocalAdbcDriverNotFound(local_path)
+
+        return AdbcDriver(local_path)
+
+    @staticmethod
+    def from_url(url: str) -> "AdbcDriver":
+        """AdbcDriver to be downloaded by url to shared-library (uses cache if exists)."""
+        expected_local_path = AdbcDriver._url_to_local_path(url)
+
+        if os.path.exists(expected_local_path):
+            return AdbcDriver(expected_local_path)
+
+        AdbcDriver._download_driver(url, expected_local_path)
+        return AdbcDriver(expected_local_path)
+
+    @staticmethod
+    def _url_to_local_path(url: str) -> str:
+        url_hash = hashlib.sha256(url.encode("utf-8")).hexdigest()
+        return os.path.join(DEFAULT_ADBC_DRIVER_CACHE_BY_URL_DIR, url_hash)
+
+    @staticmethod
+    def _download_driver(url: str, target_path: str):
+        os.makedirs(os.path.dirname(target_path), exist_ok=True)
+
+        try:
+            log.info(f"Downloading ADBC driver from {url} to {target_path}...")
+            urllib.request.urlretrieve(url, target_path)
+            log.info(f"Successfully downloaded driver to {target_path}.")
+        except Exception as e:
+            raise RemoteAdbcDriverDownloadFailed(
+                f"Failed to download ADBC driver from {url}: {e}"
+            ) from e
+
+    @property
+    def local_path(self) -> str:
+        return self._local_path
+
+
+def _get_adbc_connection(
+    adbc_driver_path: str,
+    endpoint: str,
+    access_key: str,
+    secret_key: str,
+    txid: int,
+    end_user: Optional[str],
+) -> Connection:
+    """Get an adbc connection in transaction."""
+    conn_kwargs = {TXID_OVERRIDE_PROPERTY: str(txid)}
+    if end_user is not None:
+        conn_kwargs[END_USER_PROPERTY] = end_user
+
+    return connect(
+        driver=adbc_driver_path,
+        db_kwargs={
+            "vast.db.endpoint": endpoint,
+            "vast.db.access_key": access_key,
+            "vast.db.secret_key": secret_key,
+        },
+        conn_kwargs=conn_kwargs,
+    )
+
+
+def _remove_table_qualification_from_columns(expression: exp.Expression):
+    """Goes over all columns which are fully qualified with "t0" table reference (ibis default table qualification for unbound tables.
+
+    Note: use only if one table is involved - if two tables exist in the expression columns might become ambiguous.
+    """
+    for col in expression.find_all(exp.Column):
+        col.set("table", None)
+    return expression
+
+
+def _ibis_to_qe_predicates(predicate: IbisPredicate) -> str:
+    ibis_sql = predicate.to_sql()
+    parsed = sqlglot.parse_one(ibis_sql)
+
+    # currently there is a single table
+    # removing the
+    without_table_qualification = _remove_table_qualification_from_columns(
+        parsed.expressions[0].this
+    )
+
+    return without_table_qualification.sql()
+
+
+def _vector_search_sql(
+    query_vector: list[float],
+    vector_index: VectorIndex,
+    table_ref: TableRef,
+    columns: list[str],
+    limit: int,
+    predicate: Optional[IbisPredicate] = None,
+) -> str:
+    query_vector_dim = len(query_vector)
+
+    query_vector_literal = f"{query_vector}::FLOAT[{query_vector_dim}]"
+    dist_func = f"{vector_index.sql_distance_function}({vector_index.column}::FLOAT[{query_vector_dim}], {query_vector_literal})"
+    dist_alias = f"{dist_func} as {VAST_DIST_ALIAS}"
+
+    projection_str = ",".join(columns + [dist_alias])
+
+    if predicate is not None:
+        where = f"WHERE {_ibis_to_qe_predicates(predicate)}"
+    else:
+        where = ""
+
+    return f"""
+            SELECT {projection_str}
+            FROM {table_ref.query_engine_full_path}
+            {where}
+            ORDER BY {VAST_DIST_ALIAS}
+            LIMIT {limit}"""
+
+
+class AdbcConnection:
+    def __init__(
+        self,
+        adbc_driver: AdbcDriver,
+        endpoint: str,
+        access_key: str,
+        secret_key: str,
+        txid: int,
+        end_user: Optional[str] = None,
+    ):
+        self._adbc_conn = _get_adbc_connection(
+            adbc_driver.local_path, endpoint, access_key, secret_key, txid, end_user
+        )
+
+        self._cursor = self._adbc_conn.cursor()
+
+    @property
+    def cursor(self) -> Cursor:
+        return self._cursor
+
+    def close(self):
+        self._cursor.close()
+
+    def vector_search(
+        self,
+        query_vector: list[float],
+        vector_index: VectorIndex,
+        table_ref: TableRef,
+        columns: list[str],
+        limit: int,
+        predicate: Optional[IbisPredicate] = None,
+    ) -> pa.RecordBatchReader:
+        """Top-n on vector-column."""
+        sql = _vector_search_sql(
+            query_vector=query_vector,
+            vector_index=vector_index,
+            table_ref=table_ref,
+            columns=columns,
+            limit=limit,
+            predicate=predicate,
+        )
+
+        self._cursor.execute(sql)
+        return self._cursor.fetch_record_batch()