vastdb 0.1.6__tar.gz → 0.1.8__tar.gz

{vastdb-0.1.6 → vastdb-0.1.8}/CHANGELOG.md

@@ -4,6 +4,39 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [0.1.8] (2024-06-26)
+[0.1.8]: https://github.com/vast-data/vastdb_sdk/compare/v0.1.7...v0.1.8
+
+ - Allow retries also during query response parsing
+ - Don't retrieve stats during Table.select() if not needed
+ - Log successful server-side file import
+
+## [0.1.7] (2024-06-03)
+[0.1.7]: https://github.com/vast-data/vastdb_sdk/compare/v0.1.6...v0.1.7
+
+## Added
+ - Retry idempotent requests in case of retriable errors
+ - Allow setting `Table.select()` RPC queue priority
+ - Send SDK version via Tabular RPC
+ - Support external row IDs allocation
+ - Document required VAST release
+ - Add a helper function for defining a range of VIPs
+
+## Changed
+ - Update `ibis-framework` dependency to 9.0.0
+ - Refactor internal RPC client to support retries
+ - Refactor request invocation and result handling
+
+## Fixed
+ - Sort record batch according to internal row ID column for updates/deletes
+ - Don't use `ListBuckets` RPC to probe for server's version
+ - Don't fail on an empty batch
+ - Don't use 'us-east-1' region name
+
+## Removed
+ - Remove unused internal methods
+
+
 ## [0.1.6] (2024-05-23)
 [0.1.6]: https://github.com/vast-data/vastdb_sdk/compare/v0.1.5...v0.1.6
 

{vastdb-0.1.6/vastdb.egg-info → vastdb-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: vastdb
-Version: 0.1.6
+Version: 0.1.8
 Summary: VAST Data SDK
 Home-page: https://github.com/vast-data/vastdb_sdk
 Author: VAST DATA
@@ -21,7 +21,7 @@ License-File: LICENSE
 Requires-Dist: aws-requests-auth
 Requires-Dist: boto3
 Requires-Dist: flatbuffers
-Requires-Dist: ibis-framework==8.0.0
+Requires-Dist: ibis-framework==9.0.0
 Requires-Dist: pyarrow
 Requires-Dist: requests
 Requires-Dist: xmltodict

{vastdb-0.1.6 → vastdb-0.1.8}/README.md

@@ -63,12 +63,16 @@ with session.transaction() as tx:
     ])
     table.insert(arrow_table)
 
-    result = pa.Table.from_batches(table.select())  # SELECT * FROM t
+    # run `SELECT * FROM t`
+    reader = table.select()  # return a `pyarrow.RecordBatchReader`
+    result = reader.read_all()  # build an PyArrow Table from the `pyarrow.RecordBatch` objects read from VAST
     assert result == arrow_table
 
     # the transaction is automatically committed when exiting the context
 ```
 
+Note: the transaction must be remain open while the returned [pyarrow.RecordBatchReader](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatchReader.html) generator is being used.
+
 ## Filters and projections
 
 Our SDK supports predicate and projection pushdown:
@@ -82,7 +86,7 @@ Our SDK supports predicate and projection pushdown:
 
     # SELECT c2, c3 FROM t WHERE (c2 BETWEEN 0 AND 1) OR (c2 > 10)
     table.select(columns=['c2', 'c3'],
-                 predicate=(_.c2.between(0, 1) | (table['c2'] > 10))
+                 predicate=(_.c2.between(0, 1) | (_.c2 > 10))
 
     # SELECT * FROM t WHERE c3 LIKE '%substring%'
     table.select(predicate=_.c3.contains('substring'))
@@ -129,7 +133,7 @@ with contextlib.closing(pa.parquet.ParquetWriter('/path/to/file.parquet', batche
         writer.write_batch(batch)
 ```
 
-### DuckDB
+### DuckDB integration
 
 We can use [DuckDB](https://duckdb.org/docs/guides/python/sql_on_arrow.html) to post-process the resulting stream of [PyArrow record batches](https://arrow.apache.org/docs/python/data.html#record-batches):
 
@@ -139,10 +143,14 @@ from ibis import _
 import duckdb
 conn = duckdb.connect()
 
-batches = table.select(columns=['c1'], predicate=(_.c2 > 2))
-print(conn.execute("SELECT sum(c1) FROM batches").arrow())
+with session.transaction() as tx:
+    table = tx.bucket("bucket-name").schema("schema-name").table("table-name")
+    batches = table.select(columns=['c1'], predicate=(_.c2 > 2))
+    print(conn.execute("SELECT sum(c1) FROM batches").arrow())
 ```
 
+Note: the transaction must be active while DuckDB query is executing and fetching results using the Python SDK.
+
 ## Semi-sorted projections
 
 We can create, list and delete [available semi-sorted projections](https://support.vastdata.com/s/article/UUID-e4ca42ab-d15b-6b72-bd6b-f3c77b455de4):
@@ -168,7 +176,7 @@ batches = snaps[0].schema('schema-name').table('table-name').select()
 [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database) can be queried as a regular table:
 
 ```python
-table = pa.Table.from_batches(tx.catalog.select(['element_type']))
+table = tx.catalog().select(['element_type']).read_all()
 df = table.to_pandas()
 
 total_elements = len(df)
@@ -186,3 +194,10 @@ See the following blog posts for more examples:
 
 - https://vastdata.com/blog/the-vast-catalog-in-action-part-1
 - https://vastdata.com/blog/the-vast-catalog-in-action-part-2
+
+
+## Required VAST release
+
+Currently, VAST DB Python SDK requires `5.0.0-sp10` or later VAST release.
+
+If the cluster is running an older VAST release, please contact customer.support@vastdata.com for more details.

{vastdb-0.1.6 → vastdb-0.1.8}/requirements.txt

@@ -1,7 +1,7 @@
 aws-requests-auth
 boto3
 flatbuffers
-ibis-framework==8.0.0
+ibis-framework==9.0.0
 pyarrow
 requests
 xmltodict

{vastdb-0.1.6 → vastdb-0.1.8}/setup.py

@@ -29,7 +29,7 @@ setup(
     name='vastdb',
     python_requires='>=3.9.0',
     description='VAST Data SDK',
-    version='0.1.6' + suffix,
+    version='0.1.8' + suffix,
     url='https://github.com/vast-data/vastdb_sdk',
     author='VAST DATA',
     author_email='hello@vastdata.com',

{vastdb-0.1.6 → vastdb-0.1.8}/vastdb/__init__.py

@@ -1,6 +1,9 @@
 """VAST Database Python SDK."""
 
 import functools
+import importlib.metadata
+
+__version__ = importlib.metadata.distribution(__package__).version
 
 from . import session