vastdb 1.1.0__tar.gz → 1.1.1__tar.gz

{vastdb-1.1.0 → vastdb-1.1.1}/CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [1.1.1] (2024-08-11)
+[1.1.1]: https://github.com/vast-data/vastdb_sdk/compare/v1.1.0...v1.1.1
+
+### Fixed
+ - Combine record batches when inserting `pyarrow.Table`
+
+### Changed
+ - Don't require `boto3` for checking bucket existence
+ - Improve documentation
+
+
 ## [1.1.0] (2024-07-25)
 [1.1.0]: https://github.com/vast-data/vastdb_sdk/compare/v1.0.0...v1.1.0
 

{vastdb-1.1.0 → vastdb-1.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: vastdb
-Version: 1.1.0
+Version: 1.1.1
 Summary: VAST Data SDK
 Home-page: https://github.com/vast-data/vastdb_sdk
 Author: VAST DATA
@@ -19,7 +19,6 @@ Requires-Python: >=3.9.0
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: aws-requests-auth
-Requires-Dist: boto3
 Requires-Dist: flatbuffers
 Requires-Dist: ibis-framework==9.0.0
 Requires-Dist: pyarrow

{vastdb-1.1.0 → vastdb-1.1.1}/README.md

@@ -4,24 +4,26 @@
 
 ## Introduction
 
-`vastdb` is a Python-based SDK designed for interacting with [VAST Database](https://vastdata.com/database) & [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database), enabling schema and table management, efficient ingest, query and modification of columnar data. For more details, see [our whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).
+`vastdb` is a Python-based SDK designed for interacting with a [VAST Database](https://vastdata.com/database) and the [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database), enabling schema and table management, efficient ingest, query, and modification of columnar data. 
+
+For more details about the VAST Database, see [this whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).
 
 [![vastdb](docs/vastdb.png)](https://vastdata.com/database)
 
-## Getting started
+## Getting Started
 
 ### Requirements
 
-- Linux client with Python 3.9+ and a network access to the VAST cluster
+- Linux client with Python 3.9 or later, and network access to the VAST Cluster
 - [Virtual IP pool configured with DNS service](https://support.vastdata.com/s/topic/0TOV40000000FThOAM/configuring-network-access-v50)
-- [S3 access & secret keys on VAST cluster](https://support.vastdata.com/s/article/UUID-4d2e7e23-b2fb-7900-d98f-96c31a499626)
+- [S3 access & secret keys on the VAST cluster](https://support.vastdata.com/s/article/UUID-4d2e7e23-b2fb-7900-d98f-96c31a499626)
 - [Tabular identity policy with the proper permissions](https://support.vastdata.com/s/article/UUID-14322b60-d6a2-89ac-3df0-3dfbb6974182)
 
-### Required VAST release
+### Required VAST Cluster release
 
-Currently, VAST DB Python SDK requires `5.0.0-sp10` or later VAST release.
+VAST DB Python SDK requires VAST Cluster release `5.0.0-sp10` or later.
 
-If the cluster is running an older VAST release, please contact customer.support@vastdata.com for more details.
+If your VAST Cluster is running an older release, please contact customer.support@vastdata.com.
 
 ### Installation
 
@@ -29,11 +31,11 @@ If the cluster is running an older VAST release, please contact customer.support
 pip install vastdb
 ```
 
-Also, see [our release notes](CHANGELOG.md).
+See the [Release Notes](CHANGELOG.md) for the SDK.
 
-### Quickstart
+### Quick Start
 
-Creating schemas and tables + basic inserts and selects:
+Create schemas and tables, basic inserts, and selects:
 
 ```python
 import pyarrow as pa
@@ -74,11 +76,15 @@ with session.transaction() as tx:
     # the transaction is automatically committed when exiting the context
 ```
 
+For configuration examples, see [here](docs/config.md).
+
 Note: the transaction must be remain open while the returned [pyarrow.RecordBatchReader](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatchReader.html) generator is being used.
 
-## Filters and projections
+## Use Cases
+
+### Filters and Projections
 
-Our SDK supports predicate and projection pushdown:
+The SDK supports predicate and projection pushdown:
 
 ```python
     from ibis import _
@@ -97,9 +103,9 @@ Our SDK supports predicate and projection pushdown:
 
 See [here for more details](docs/predicate.md).
 
-## Import a single Parquet file via S3 protocol
+### Import a single Parquet file via S3 protocol
 
-It is possible to efficiently create a table from a Parquet file (without copying it via the client):
+You can efficiently create tables from Parquet files (without copying them via the client):
 
 ```python
     with tempfile.NamedTemporaryFile() as f:
@@ -112,9 +118,9 @@ It is possible to efficiently create a table from a Parquet file (without copyin
         parquet_files=['/bucket-name/staging/file.parquet'])
 ```
 
-## Import multiple Parquet files concurrently via S3 protocol
+### Import multiple Parquet files concurrently via S3 protocol
 
-We can import multiple files concurrently into a table (by utilizing multiple CNodes' cores):
+Import multiple files concurrently into a table (by using multiple CNodes' cores):
 
 ```python
     schema = tx.bucket('bucket-name').schema('schema-name')
@@ -123,6 +129,27 @@ We can import multiple files concurrently into a table (by utilizing multiple CN
         parquet_files=[f'/bucket-name/staging/file{i}.parquet' for i in range(10)])
 ```
 
+
+### Semi-sorted Projections
+
+Create, list and delete [available semi-sorted projections](https://support.vastdata.com/s/article/UUID-e4ca42ab-d15b-6b72-bd6b-f3c77b455de4):
+
+```python
+p = table.create_projection('proj', sorted=['c3'], unsorted=['c1'])
+print(table.projections())
+print(p.get_stats())
+p.drop()
+```
+
+### Snapshots
+
+You can access the VAST Database using [snapshots](https://vastdata.com/blog/bringing-snapshots-to-vasts-element-store):
+
+```python
+snaps = bucket.list_snapshots()
+batches = snaps[0].schema('schema-name').table('table-name').select()
+```
+
 ## Post-processing
 
 ### Export
@@ -136,9 +163,9 @@ with contextlib.closing(pa.parquet.ParquetWriter('/path/to/file.parquet', batche
         writer.write_batch(batch)
 ```
 
-### DuckDB integration
+### DuckDB Integration
 
-We can use [DuckDB](https://duckdb.org/docs/guides/python/sql_on_arrow.html) to post-process the resulting stream of [PyArrow record batches](https://arrow.apache.org/docs/python/data.html#record-batches):
+Use [DuckDB](https://duckdb.org/docs/guides/python/sql_on_arrow.html) to post-process the resulting stream of [PyArrow record batches](https://arrow.apache.org/docs/python/data.html#record-batches):
 
 ```python
 from ibis import _
@@ -152,31 +179,11 @@ with session.transaction() as tx:
     print(conn.execute("SELECT sum(c1) FROM batches").arrow())
 ```
 
-Note: the transaction must be active while DuckDB query is executing and fetching results using the Python SDK.
-
-## Semi-sorted projections
-
-We can create, list and delete [available semi-sorted projections](https://support.vastdata.com/s/article/UUID-e4ca42ab-d15b-6b72-bd6b-f3c77b455de4):
-
-```python
-p = table.create_projection('proj', sorted=['c3'], unsorted=['c1'])
-print(table.projections())
-print(p.get_stats())
-p.drop()
-```
-
-## Snapshots
-
-It is possible to use [snapshots](https://vastdata.com/blog/bringing-snapshots-to-vasts-element-store) for accessing the Database:
-
-```python
-snaps = bucket.list_snapshots()
-batches = snaps[0].schema('schema-name').table('table-name').select()
-```
+Note: the transaction must be active while the DuckDB query is executing and fetching results using the Python SDK.
 
 ## VAST Catalog
 
-[VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database) can be queried as a regular table:
+The [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database) can be queried as a regular table:
 
 ```python
 import pyarrow as pa
@@ -201,8 +208,11 @@ with session.transaction() as tx:
     print("Distinct element types on the system:")
     print(distinct_elements)
 ```
+## More Information
 
-See the following blog posts for more examples:
+See these blog posts for more examples:
 
 - https://vastdata.com/blog/the-vast-catalog-in-action-part-1
 - https://vastdata.com/blog/the-vast-catalog-in-action-part-2
+
+See also the [full Vast DB Python SDK documentation](https://vastdb-sdk.readthedocs.io/en/v1.1.0/)

{vastdb-1.1.0 → vastdb-1.1.1}/requirements.txt

@@ -1,5 +1,4 @@
 aws-requests-auth
-boto3
 flatbuffers
 ibis-framework==9.0.0
 pyarrow

{vastdb-1.1.0 → vastdb-1.1.1}/setup.py

@@ -29,7 +29,7 @@ setup(
     name='vastdb',
     python_requires='>=3.9.0',
     description='VAST Data SDK',
-    version='1.1.0' + suffix,
+    version='1.1.1' + suffix,
     url='https://github.com/vast-data/vastdb_sdk',
     author='VAST DATA',
     author_email='hello@vastdata.com',

{vastdb-1.1.0 → vastdb-1.1.1}/vastdb/_internal.py

@@ -1298,6 +1298,15 @@ class VastdbApi:
 
         return columns, next_key, is_truncated, count
 
+    def head_bucket(self, bucket_name):
+        """
+        Reimplemented, instead of depending on boto3 for checking the existence of a bucket.
+        https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html
+        """
+        return self._request(
+            method="HEAD",
+            url=self._url(bucket=bucket_name))
+
     def begin_transaction(self, client_tags=[], expected_retvals=[]):
         """
         POST /?transaction HTTP/1.1

{vastdb-1.1.0 → vastdb-1.1.1}/vastdb/bench/test_sample.py

@@ -109,7 +109,7 @@ def load_batch(bucket, session_kwargs, offset, limit):
 
 def test_ingest(test_bucket_name, session_kwargs, tabular_endpoint_urls, num_workers, perf_metrics_db):
     session = vastdb.connect(**session_kwargs)
-    metrics_table = metrics.Table(perf_metrics_db, "ingest")
+    metrics_table = metrics.Table(perf_metrics_db, "test_ingest")
 
     with session.transaction() as tx:
         b = tx.bucket(test_bucket_name)
@@ -201,9 +201,11 @@ def run_query(session_kwargs, i, bucket_name, endpoint_url):
                 nbytes=data, rows=rows, cols=len(cols),
                 pid=pid, tid=tid, sdk_version=sdk_version))
 
+    return metrics_rows
+
 
 def test_scan(test_bucket_name, session, num_workers, session_kwargs, tabular_endpoint_urls, perf_metrics_db):
-    metrics_table = metrics.Table(perf_metrics_db, "query")
+    metrics_table = metrics.Table(perf_metrics_db, "test_scan")
 
     log.info("starting %d workers, endpoints=%s", num_workers, tabular_endpoint_urls)
     with ProcessPoolExecutor(max_workers=num_workers) as executor:

{vastdb-1.1.0 → vastdb-1.1.1}/vastdb/conftest.py

@@ -102,4 +102,4 @@ def table_name(request):
 
 @pytest.fixture(scope="function")
 def perf_metrics_db(crater_path):
-    return sqlite3.connect(f"{crater_path}/metrics.sqlite")
+    return sqlite3.connect(f"{crater_path}/metrics.sqlite" if crater_path else ":memory:")

{vastdb-1.1.0 → vastdb-1.1.1}/vastdb/session.py

@@ -23,8 +23,6 @@ class Session:
                  timeout=None,
                  backoff_config: Optional["BackoffConfig"] = None):
         """Connect to a VAST Database endpoint, using specified credentials."""
-        import boto3
-
         from . import _internal, features
 
         if access is None:
@@ -42,10 +40,6 @@ class Session:
             timeout=timeout,
             backoff_config=backoff_config)
         self.features = features.Features(self.api.vast_version)
-        self.s3 = boto3.client('s3',
-            aws_access_key_id=access,
-            aws_secret_access_key=secret,
-            endpoint_url=endpoint)
 
     def __repr__(self):
         """Don't show the secret key."""

{vastdb-1.1.0 → vastdb-1.1.1}/vastdb/table.py

@@ -439,7 +439,7 @@ class Table:
             self.update(rows=column_record_batch, columns=columns_name_chunk)
         return row_ids
 
-    def insert(self, rows: pa.RecordBatch):
+    def insert(self, rows: Union[pa.RecordBatch, pa.Table]):
         """Insert a RecordBatch into this table."""
         if self._imports_table:
             raise errors.NotSupportedCommand(self.bucket.name, self.schema.name, self.name)

{vastdb-1.1.0 → vastdb-1.1.1}/vastdb/tests/test_tables.py

@@ -81,6 +81,19 @@ def test_insert_wide_row(session, clean_bucket_name):
         assert actual == expected
 
 
+def test_multi_batch_table(session, clean_bucket_name):
+    columns = pa.schema([pa.field('s', pa.utf8())])
+    expected = pa.Table.from_batches([
+        pa.record_batch(schema=columns, data=[['a']]),
+        pa.record_batch(schema=columns, data=[['b']]),
+        pa.record_batch(schema=columns, data=[['c']]),
+    ])
+
+    with prepare_data(session, clean_bucket_name, 's', 't', expected) as t:
+        actual = t.select().read_all()
+        assert actual == expected
+
+
 def test_insert_empty(session, clean_bucket_name):
     columns = pa.schema([('a', pa.int8()), ('b', pa.float32())])
     data = [[None] * 5, [None] * 5]

{vastdb-1.1.0 → vastdb-1.1.1}/vastdb/transaction.py

@@ -10,8 +10,6 @@ import logging
 from dataclasses import dataclass
 from typing import TYPE_CHECKING, Iterable, Optional
 
-import botocore
-
 from . import bucket, errors, schema, session
 
 if TYPE_CHECKING:
@@ -64,12 +62,10 @@ class Transaction:
     def bucket(self, name: str) -> "Bucket":
         """Return a VAST Bucket, if exists."""
         try:
-            self._rpc.s3.head_bucket(Bucket=name)
-        except botocore.exceptions.ClientError as e:
-            log.warning("res: %s", e.response)
-            if e.response['Error']['Code'] == '404':
-                raise errors.MissingBucket(name) from e
-            raise
+            self._rpc.api.head_bucket(name)
+        except errors.NotFound as e:
+            raise errors.MissingBucket(name) from e
+
         return bucket.Bucket(name, self)
 
     def catalog_snapshots(self) -> Iterable["Bucket"]:

{vastdb-1.1.0 → vastdb-1.1.1}/vastdb/util.py

@@ -114,6 +114,11 @@ def iter_serialized_slices(batch: Union[pa.RecordBatch, pa.Table], max_rows_per_
 
 def serialize_record_batch(batch: Union[pa.RecordBatch, pa.Table]):
     """Serialize a RecordBatch using Arrow IPC format."""
+    if isinstance(batch, pa.Table):
+        if len(batch.to_batches()) > 1:
+            # the server expects a single RecordBatch per request
+            batch = batch.combine_chunks()
+
     sink = pa.BufferOutputStream()
     with pa.ipc.new_stream(sink, batch.schema) as writer:
         writer.write(batch)

{vastdb-1.1.0 → vastdb-1.1.1}/vastdb.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: vastdb
-Version: 1.1.0
+Version: 1.1.1
 Summary: VAST Data SDK
 Home-page: https://github.com/vast-data/vastdb_sdk
 Author: VAST DATA
@@ -19,7 +19,6 @@ Requires-Python: >=3.9.0
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: aws-requests-auth
-Requires-Dist: boto3
 Requires-Dist: flatbuffers
 Requires-Dist: ibis-framework==9.0.0
 Requires-Dist: pyarrow

{vastdb-1.1.0 → vastdb-1.1.1}/vastdb.egg-info/requires.txt

@@ -1,5 +1,4 @@
 aws-requests-auth
-boto3
 flatbuffers
 ibis-framework==9.0.0
 pyarrow