vastdb 1.0.0__tar.gz → 1.1.1__tar.gz

{vastdb-1.0.0 → vastdb-1.1.1}/CHANGELOG.md

@@ -4,6 +4,27 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [1.1.1] (2024-08-11)
+[1.1.1]: https://github.com/vast-data/vastdb_sdk/compare/v1.1.0...v1.1.1
+
+### Fixed
+ - Combine record batches when inserting `pyarrow.Table`
+
+### Changed
+ - Don't require `boto3` for checking bucket existence
+ - Improve documentation
+
+
+## [1.1.0] (2024-07-25)
+[1.1.0]: https://github.com/vast-data/vastdb_sdk/compare/v1.0.0...v1.1.0
+
+## Fixed
+ - Close internal sessions in `select` and `import`
+
+## Added
+ - VastDB/Parquet performance benchmark harness
+ - Support predicate pushdown over subfields
+
 ## [1.0.0] (2024-07-21)
 [1.0.0]: https://github.com/vast-data/vastdb_sdk/compare/v0.1.11...v1.0.0
 

{vastdb-1.0.0 → vastdb-1.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: vastdb
-Version: 1.0.0
+Version: 1.1.1
 Summary: VAST Data SDK
 Home-page: https://github.com/vast-data/vastdb_sdk
 Author: VAST DATA
@@ -19,7 +19,6 @@ Requires-Python: >=3.9.0
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: aws-requests-auth
-Requires-Dist: boto3
 Requires-Dist: flatbuffers
 Requires-Dist: ibis-framework==9.0.0
 Requires-Dist: pyarrow

{vastdb-1.0.0 → vastdb-1.1.1}/README.md

@@ -4,24 +4,26 @@
 
 ## Introduction
 
-`vastdb` is a Python-based SDK designed for interacting with [VAST Database](https://vastdata.com/database) & [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database), enabling schema and table management, efficient ingest, query and modification of columnar data. For more details, see [our whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).
+`vastdb` is a Python-based SDK designed for interacting with a [VAST Database](https://vastdata.com/database) and the [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database), enabling schema and table management, efficient ingest, query, and modification of columnar data. 
+
+For more details about the VAST Database, see [this whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).
 
 [![vastdb](docs/vastdb.png)](https://vastdata.com/database)
 
-## Getting started
+## Getting Started
 
 ### Requirements
 
-- Linux client with Python 3.9+ and a network access to the VAST cluster
+- Linux client with Python 3.9 or later, and network access to the VAST Cluster
 - [Virtual IP pool configured with DNS service](https://support.vastdata.com/s/topic/0TOV40000000FThOAM/configuring-network-access-v50)
-- [S3 access & secret keys on VAST cluster](https://support.vastdata.com/s/article/UUID-4d2e7e23-b2fb-7900-d98f-96c31a499626)
+- [S3 access & secret keys on the VAST cluster](https://support.vastdata.com/s/article/UUID-4d2e7e23-b2fb-7900-d98f-96c31a499626)
 - [Tabular identity policy with the proper permissions](https://support.vastdata.com/s/article/UUID-14322b60-d6a2-89ac-3df0-3dfbb6974182)
 
-### Required VAST release
+### Required VAST Cluster release
 
-Currently, VAST DB Python SDK requires `5.0.0-sp10` or later VAST release.
+VAST DB Python SDK requires VAST Cluster release `5.0.0-sp10` or later.
 
-If the cluster is running an older VAST release, please contact customer.support@vastdata.com for more details.
+If your VAST Cluster is running an older release, please contact customer.support@vastdata.com.
 
 ### Installation
 
@@ -29,11 +31,11 @@ If the cluster is running an older VAST release, please contact customer.support
 pip install vastdb
 ```
 
-Also, see [our release notes](CHANGELOG.md).
+See the [Release Notes](CHANGELOG.md) for the SDK.
 
-### Quickstart
+### Quick Start
 
-Creating schemas and tables + basic inserts and selects:
+Create schemas and tables, basic inserts, and selects:
 
 ```python
 import pyarrow as pa
@@ -74,11 +76,15 @@ with session.transaction() as tx:
     # the transaction is automatically committed when exiting the context
 ```
 
+For configuration examples, see [here](docs/config.md).
+
 Note: the transaction must be remain open while the returned [pyarrow.RecordBatchReader](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatchReader.html) generator is being used.
 
-## Filters and projections
+## Use Cases
+
+### Filters and Projections
 
-Our SDK supports predicate and projection pushdown:
+The SDK supports predicate and projection pushdown:
 
 ```python
     from ibis import _
@@ -97,9 +103,9 @@ Our SDK supports predicate and projection pushdown:
 
 See [here for more details](docs/predicate.md).
 
-## Import a single Parquet file via S3 protocol
+### Import a single Parquet file via S3 protocol
 
-It is possible to efficiently create a table from a Parquet file (without copying it via the client):
+You can efficiently create tables from Parquet files (without copying them via the client):
 
 ```python
     with tempfile.NamedTemporaryFile() as f:
@@ -112,9 +118,9 @@ It is possible to efficiently create a table from a Parquet file (without copyin
         parquet_files=['/bucket-name/staging/file.parquet'])
 ```
 
-## Import multiple Parquet files concurrently via S3 protocol
+### Import multiple Parquet files concurrently via S3 protocol
 
-We can import multiple files concurrently into a table (by utilizing multiple CNodes' cores):
+Import multiple files concurrently into a table (by using multiple CNodes' cores):
 
 ```python
     schema = tx.bucket('bucket-name').schema('schema-name')
@@ -123,6 +129,27 @@ We can import multiple files concurrently into a table (by utilizing multiple CN
         parquet_files=[f'/bucket-name/staging/file{i}.parquet' for i in range(10)])
 ```
 
+
+### Semi-sorted Projections
+
+Create, list and delete [available semi-sorted projections](https://support.vastdata.com/s/article/UUID-e4ca42ab-d15b-6b72-bd6b-f3c77b455de4):
+
+```python
+p = table.create_projection('proj', sorted=['c3'], unsorted=['c1'])
+print(table.projections())
+print(p.get_stats())
+p.drop()
+```
+
+### Snapshots
+
+You can access the VAST Database using [snapshots](https://vastdata.com/blog/bringing-snapshots-to-vasts-element-store):
+
+```python
+snaps = bucket.list_snapshots()
+batches = snaps[0].schema('schema-name').table('table-name').select()
+```
+
 ## Post-processing
 
 ### Export
@@ -136,9 +163,9 @@ with contextlib.closing(pa.parquet.ParquetWriter('/path/to/file.parquet', batche
         writer.write_batch(batch)
 ```
 
-### DuckDB integration
+### DuckDB Integration
 
-We can use [DuckDB](https://duckdb.org/docs/guides/python/sql_on_arrow.html) to post-process the resulting stream of [PyArrow record batches](https://arrow.apache.org/docs/python/data.html#record-batches):
+Use [DuckDB](https://duckdb.org/docs/guides/python/sql_on_arrow.html) to post-process the resulting stream of [PyArrow record batches](https://arrow.apache.org/docs/python/data.html#record-batches):
 
 ```python
 from ibis import _
@@ -152,31 +179,11 @@ with session.transaction() as tx:
     print(conn.execute("SELECT sum(c1) FROM batches").arrow())
 ```
 
-Note: the transaction must be active while DuckDB query is executing and fetching results using the Python SDK.
-
-## Semi-sorted projections
-
-We can create, list and delete [available semi-sorted projections](https://support.vastdata.com/s/article/UUID-e4ca42ab-d15b-6b72-bd6b-f3c77b455de4):
-
-```python
-p = table.create_projection('proj', sorted=['c3'], unsorted=['c1'])
-print(table.projections())
-print(p.get_stats())
-p.drop()
-```
-
-## Snapshots
-
-It is possible to use [snapshots](https://vastdata.com/blog/bringing-snapshots-to-vasts-element-store) for accessing the Database:
-
-```python
-snaps = bucket.list_snapshots()
-batches = snaps[0].schema('schema-name').table('table-name').select()
-```
+Note: the transaction must be active while the DuckDB query is executing and fetching results using the Python SDK.
 
 ## VAST Catalog
 
-[VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database) can be queried as a regular table:
+The [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database) can be queried as a regular table:
 
 ```python
 import pyarrow as pa
@@ -201,8 +208,11 @@ with session.transaction() as tx:
     print("Distinct element types on the system:")
     print(distinct_elements)
 ```
+## More Information
 
-See the following blog posts for more examples:
+See these blog posts for more examples:
 
 - https://vastdata.com/blog/the-vast-catalog-in-action-part-1
 - https://vastdata.com/blog/the-vast-catalog-in-action-part-2
+
+See also the [full Vast DB Python SDK documentation](https://vastdb-sdk.readthedocs.io/en/v1.1.0/)

{vastdb-1.0.0 → vastdb-1.1.1}/requirements.txt

@@ -1,5 +1,4 @@
 aws-requests-auth
-boto3
 flatbuffers
 ibis-framework==9.0.0
 pyarrow

{vastdb-1.0.0 → vastdb-1.1.1}/setup.py

@@ -29,7 +29,7 @@ setup(
     name='vastdb',
     python_requires='>=3.9.0',
     description='VAST Data SDK',
-    version='1.0.0' + suffix,
+    version='1.1.1' + suffix,
     url='https://github.com/vast-data/vastdb_sdk',
     author='VAST DATA',
     author_email='hello@vastdata.com',

{vastdb-1.0.0 → vastdb-1.1.1}/vastdb/_internal.py

@@ -35,6 +35,7 @@ from ibis.expr.operations.logical import (
 )
 from ibis.expr.operations.relations import Field
 from ibis.expr.operations.strings import StringContains
+from ibis.expr.operations.structs import StructField
 
 import vast_flatbuf.org.apache.arrow.computeir.flatbuf.BinaryLiteral as fb_binary_lit
 import vast_flatbuf.org.apache.arrow.computeir.flatbuf.BooleanLiteral as fb_bool_lit
@@ -182,7 +183,7 @@ class Predicate:
                 _logger.debug('OR args: %s op %s', or_args, op)
                 inner_offsets = []
 
-                prev_field_name = None
+                prev_field_path = None
                 for inner_op in or_args:
                     _logger.debug('inner_op %s', inner_op)
                     op_type = type(inner_op)
@@ -216,28 +217,38 @@ class Predicate:
                             if not isinstance(literal, Literal):
                                 raise NotImplementedError(self.expr)
 
+                    field_path = []
+                    while isinstance(column, StructField):
+                        column, subfield_name = column.args
+                        field_path.append(subfield_name)
+
                     if not isinstance(column, Field):
                         raise NotImplementedError(self.expr)
 
-                    field_name = column.name
-                    if prev_field_name is None:
-                        prev_field_name = field_name
-                    elif prev_field_name != field_name:
+                    field_path.append(column.name)
+                    field_path.reverse()  # first entry should be the top-level column name
+
+                    if prev_field_path is None:
+                        prev_field_path = field_path
+                    elif prev_field_path != field_path:
                         raise NotImplementedError(self.expr)
 
-                    node = self.nodes_map[field_name]
+                    nodes_map = self.nodes_map
+                    for name in field_path:
+                        node = nodes_map[name]
+                        nodes_map = node.children_map
+
                     # TODO: support predicate pushdown for leaf nodes (ORION-160338)
                     if node.children:
                         raise NotImplementedError(node.field)  # no predicate pushdown for nested columns
                     column_offset = self.build_column(position=node.index)
-                    field = self.schema.field(field_name)
                     for literal in literals:
                         args_offsets = [column_offset]
                         if literal is not None:
-                            args_offsets.append(self.build_literal(field=field, value=literal.value))
+                            args_offsets.append(self.build_literal(field=node.field, value=literal.value))
                         if builder_func == self.build_between:
-                            args_offsets.append(self.build_literal(field=field, value=lower.value))
-                            args_offsets.append(self.build_literal(field=field, value=upper.value))
+                            args_offsets.append(self.build_literal(field=node.field, value=lower.value))
+                            args_offsets.append(self.build_literal(field=node.field, value=upper.value))
 
                         inner_offsets.append(builder_func(*args_offsets))
 
@@ -572,6 +583,8 @@ class FieldNode:
         else:
             self.children = []  # for non-nested types
 
+        self.children_map = {c.field.name: c for c in self.children}
+
     def _iter_to_root(self) -> Iterator['FieldNode']:
         yield self
         if self.parent is not None:
@@ -811,7 +824,16 @@ class VastdbApi:
         _logger.critical(msg)
         raise NotImplementedError(msg)
 
+    def __enter__(self):
+        """Allow using this session as a context manager."""
+        return self
+
+    def __exit__(self, *args):
+        """Make sure that the connections closed."""
+        self._session.close()
+
     def with_endpoint(self, endpoint):
+        """Open a new session for targeting a specific endpoint."""
         return VastdbApi(endpoint=endpoint,
             access_key=self.access_key,
             secret_key=self.secret_key,
@@ -1276,6 +1298,15 @@ class VastdbApi:
 
         return columns, next_key, is_truncated, count
 
+    def head_bucket(self, bucket_name):
+        """
+        Reimplemented, instead of depending on boto3 for checking the existence of a bucket.
+        https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html
+        """
+        return self._request(
+            method="HEAD",
+            url=self._url(bucket=bucket_name))
+
     def begin_transaction(self, client_tags=[], expected_retvals=[]):
         """
         POST /?transaction HTTP/1.1

vastdb-1.1.1/vastdb/bench/perf_bench/bench_repo/mega_combo.py

@@ -0,0 +1,87 @@
+from typing import List, Optional
+
+from vastdb.bench.perf_bench.common.constants import (
+    LOCAL_FS_DS_PATH,
+    NFS_DS_PATH,  # noqa: F401
+    S3_DS_PATH,  # noqa: F401
+    ParquetCompression,
+    VastConnDetails,
+)
+from vastdb.bench.perf_bench.dataset.schemas import DEFAULT_BARS_COLUMNS
+from vastdb.bench.perf_bench.orchestrate.scenario import BenchScenario
+from vastdb.bench.perf_bench.orchestrate.scenario_generator import (
+    generate_perf_bench_scenarios,
+)
+from vastdb.bench.perf_bench.query.query import QueryBackend
+
+
+def build_scenarios(
+    base_key: str,
+    conn_details: Optional[VastConnDetails] = None,
+) -> List[BenchScenario]:
+    return generate_perf_bench_scenarios(
+        base_key=base_key,
+        conn_details=conn_details or VastConnDetails(),
+        query_backends=[
+            QueryBackend.pyarrow,
+            # QueryBackend.vastdb,
+        ],
+        columns_choices=(DEFAULT_BARS_COLUMNS,),
+        universe_choices=(
+            "Single",
+            "Tiny",
+            "SmallSeq",
+            "Medium",
+            "Medium2",
+            "Large",
+        ),
+        num_bdays=[
+            1,  # 1d
+            5,  # 1w
+            # 22,  # 1m
+            65,  # 3m
+            # 130,  # 6m
+            252,  # 1y
+        ],
+
+        # Arrow-specific options
+        fs_path_choices=[
+            # NFS_DS_PATH,
+            LOCAL_FS_DS_PATH,
+            # S3_DS_PATH,
+        ],
+        rowgroup_size_choices=[  # make sure you have previously generated the respective datasets
+            # 64 * 1024,
+            # 128 * 1024,
+            256 * 1024,
+            # 512 * 1024,
+            # DEFAULT_ROW_GROUP_SIZE,
+            # int(1.5 * 1024 * 1024),
+        ],
+        compression_choices=[
+            ParquetCompression.LZ4,
+        ],
+        arrow_batching_spec_choices=[
+            # {"batch_size": 2*2**16, "batch_readahead": 16, "fragment_readahead": 4},
+            # {"batch_size": 6*2**16, "batch_readahead": 12, "fragment_readahead": 4},
+            # DEFAULT_ARROW_KWARGS,
+            {"batch_size": 16 * 2 ** 16, "batch_readahead": 16, "fragment_readahead": 4},
+            # {"batch_size": 24 * 2 ** 16, "batch_readahead": 12, "fragment_readahead": 4},
+            # {"batch_size": 32*2**16, "batch_readahead": 12, "fragment_readahead": 4},
+            # {"batch_size": 64*2**16, "batch_readahead": 12, "fragment_readahead": 4},
+            # {"batch_size": 128*2**16, "batch_readahead": 12, "fragment_readahead": 4},
+        ],
+
+        # VastDB-specific options
+        vdb_num_sub_splits_choices=(
+            # 1,  # Default
+            # 4,
+            8,
+            # 16,
+        ),
+        vdb_num_row_groups_per_sub_split_choices=(
+            # 1,
+            # 4,
+            8,  # Default
+        ),
+    )

vastdb-1.1.1/vastdb/bench/perf_bench/cli.py

@@ -0,0 +1,225 @@
+from pathlib import Path
+from typing import Annotated, List, Optional
+
+import pandas as pd
+import typer
+
+from vastdb.bench.perf_bench.common.constants import (
+    DEFAULT_END_T,
+    DEFAULT_RESULTS_DIR,
+    DEFAULT_START_T,
+    DFAULT_PARQUET_COMPRESSION,
+    LOCAL_FS_DS_PATH,
+    LogLevel,
+    ParquetCompression,
+)
+from vastdb.bench.perf_bench.common.log_utils import (
+    get_logger,
+    set_log_file,
+    set_log_level,
+)
+from vastdb.bench.perf_bench.common.utils import getenv_flag, load_module_from_path
+from vastdb.bench.perf_bench.dataset.generate_secmaster import (
+    SM_PATH,
+    generate_secmaster,
+)
+from vastdb.bench.perf_bench.dataset.generate_stocks_dataset import (
+    generate_concurrent_synthetic_stock_1m_bars,
+)
+from vastdb.bench.perf_bench.orchestrate.scenario import BenchScenario
+from vastdb.bench.perf_bench.run import run_scenarios
+
+app = typer.Typer(pretty_exceptions_enable=getenv_flag("TYPER_PRETTY_EXCEPTIONS"))
+
+_MY_DIR = Path(__file__).parent
+
+
+# noinspection PyUnusedLocal
+@app.callback()
+def cli_common(
+    ctx: typer.Context,
+    verbose: Annotated[
+        bool,
+        typer.Option(
+            "--verbose",
+            is_flag=True,
+        ),
+    ] = False,
+    log_level: Annotated[
+        Optional[LogLevel],
+        typer.Option(
+            "--log-level",
+            case_sensitive=False,
+        ),
+    ] = None,
+    log_file: Annotated[
+        Optional[Path],
+        typer.Option(
+            "--log-file",
+            writable=True,
+            file_okay=True,
+            dir_okay=False,
+            resolve_path=True,
+        ),
+    ] = None,
+):
+    if verbose:
+        log_level = LogLevel.DEBUG
+    if log_level:
+        set_log_level(log_level)
+    if log_file:
+        set_log_file(log_file)
+    get_logger(__name__).info("CLI common setup done.")
+
+
+def _positive_int(value: str) -> int:
+    i_value = int(value)
+    if i_value <= 0:
+        raise typer.BadParameter(f"Must be a positive integer: {value}.")
+    return i_value
+
+
+# noinspection PyUnusedLocal
+@app.command()
+def run_bench(
+    ctx: typer.Context,
+    bench_name: Annotated[
+        str,
+        typer.Option(
+            "--bench-name",
+        ),
+    ],
+    parallelism: Annotated[
+        List[int],
+        typer.Option(
+            "--parallelism",
+            callback=lambda par: [_positive_int(p) for p in par],
+        ),
+    ],
+    runs_per_bench: Annotated[
+        int,
+        typer.Option(
+            "--runs-per-bench",
+            callback=_positive_int,
+        ),
+    ] = 3,
+    bench_generator_path: Annotated[
+        Path,
+        typer.Option(
+            "--bench-generator-path",
+            readable=True,
+            file_okay=True,
+            dir_okay=False,
+            resolve_path=True,
+        ),
+    ] = _MY_DIR / "bench_repo" / "mega_combo.py",
+    results_base_dir: Annotated[
+        Path,
+        typer.Option(
+            "--log-file",
+            writable=True,
+            file_okay=False,
+            dir_okay=True,
+            resolve_path=True,
+        ),
+    ] = DEFAULT_RESULTS_DIR,
+):
+    if not (bench_name := bench_name.strip()):
+        raise typer.BadParameter("Bench name must be non-empty.")
+
+    mod = load_module_from_path(bench_generator_path)
+    scenarios: List[BenchScenario] = mod.build_scenarios(base_key=bench_name)
+    for para in parallelism:
+        run_scenarios(
+            scenarios=scenarios,
+            runs_per_bench=runs_per_bench,
+            parallelism=para,
+            results_base_dir=str(results_base_dir),
+        )
+
+
+# noinspection PyUnusedLocal
+@app.command()
+def build_secmaster(
+    ctx: typer.Context,
+):
+    generate_secmaster()
+
+
+# noinspection PyUnusedLocal
+@app.command()
+def build_dataset(
+    ctx: typer.Context,
+    start_date: Annotated[
+        str,
+        typer.Option(
+            "--start-date",
+            help="Start date for the dataset.",
+            callback=lambda d: pd.Timestamp(d).normalize(),
+        ),
+    ] = DEFAULT_START_T.strftime("%Y%m%d"),
+    end_date: Annotated[
+        str,
+        typer.Option(
+            "--end-date",
+            help="Start date for the dataset.",
+            callback=lambda d: pd.Timestamp(d).normalize(),
+        ),
+    ] = DEFAULT_END_T.strftime("%Y%m%d"),
+    output_dir: Annotated[
+        Path,
+        typer.Option(
+            "--output-dir",
+            writable=True,
+            file_okay=False,
+            dir_okay=True,
+            resolve_path=True,
+        ),
+    ] = LOCAL_FS_DS_PATH,
+    parallelism: Annotated[
+        int,
+        typer.Option(
+            "--parallelism",
+            callback=_positive_int,
+        ),
+    ] = 6,
+    row_group_size: Annotated[
+        int,
+        typer.Option(
+            "--row-group-size",
+            callback=_positive_int,
+            help=(
+                "Row group size for the dataset, some common values are: 64 * 1024, 128 * 1024, 256"
+                " * 1024, 512 * 1024,1024 * 1024, 1.5 * 1024 * 1024."
+            ),
+        ),
+    ] = 256 * 1024,
+    compression: Annotated[
+        ParquetCompression,
+        typer.Option(
+            "--compression",
+            help="Parquet compression algorithm.",
+        ),
+    ] = DFAULT_PARQUET_COMPRESSION,
+):
+    if row_group_size < 1024:
+        raise typer.BadParameter("Row group size must be at least 1024.")
+    if parallelism < 1:
+        raise typer.BadParameter("Parallelism must be at least 1.")
+    if start_date > end_date:
+        raise typer.BadParameter("Start date must be before the end date.")
+    if not SM_PATH.is_file():
+        generate_secmaster()
+    generate_concurrent_synthetic_stock_1m_bars(
+        from_t=start_date,
+        to_t=end_date,
+        output_dir=output_dir,
+        num_workers=parallelism,
+        row_group_size=row_group_size,
+        compression=compression,
+    )
+
+
+if __name__ == "__main__":
+    # Set the metadata only if we execute the main (not on just importing this module)
+    app()