vastdb 0.1.11__tar.gz → 1.1.0__tar.gz

{vastdb-0.1.11 → vastdb-1.1.0}/CHANGELOG.md

@@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [1.1.0] (2024-07-25)
+[1.1.0]: https://github.com/vast-data/vastdb_sdk/compare/v1.0.0...v1.1.0
+
+## Fixed
+ - Close internal sessions in `select` and `import`
+
+## Added
+ - VastDB/Parquet performance benchmark harness
+ - Support predicate pushdown over subfields
+
+## [1.0.0] (2024-07-21)
+[1.0.0]: https://github.com/vast-data/vastdb_sdk/compare/v0.1.11...v1.0.0
+
+## Added
+ - Add CLA (`CONTRIBUTING.md`)
+ - Support setting HTTP timeouts
+
 ## [0.1.11] (2024-07-17)
 [0.1.11]: https://github.com/vast-data/vastdb_sdk/compare/v0.1.10...v0.1.11
 

vastdb-1.1.0/CONTRIBUTING.md

@@ -0,0 +1,33 @@
+# VAST DB SDK Contribution License Agreement
+
+Thank you for your interest in VAST DB SDK project (“Project”). You hereby accept and agree to the following terms and conditions (“Agreement”) for Your present and future Contributions (as defined below) submitted to the Project or as part of. Except for the license granted herein to the Project originator – Vast Data Ltd. together with any of its affiliates, assignee, or successor (“Vast Data” or “us” or “we”), You reserve all right, title, and interest in and to Your Contributions.
+
+## Definitions.
+
+"You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with Vast Data. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single contributor. For the purposes of this definition, "control" means (i) the power, directly or indirectly, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+"Contribution" shall mean the code, documentation, or any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to the Project (or to Vast Data), inclusion in, or documentation of, the Project (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Project or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
+
+## Licenses
+
+Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to Vast Data and at Vast Data’ discretion, its affiliates, a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, sell, and distribute Your Contributions and such derivative works.
+
+Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to Vast Data and, at Vast Data’ discretion, its affiliates a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by the combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes a direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
+
+## Representations
+
+You represent that You are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes Your Contributions, You represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to Vast Data, or that your employer has executed a separate written agreement with Vast Data.
+
+You represent that each of Your Contributions is Your original creation (see below for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware, and which are associated with any part of Your Contributions.
+
+You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. 
+
+UNLESS REQUIRED BY APPLICABLE LAW, AGREED TO IN WRITING, OR AS STATED IN THIS CLA, YOU PROVIDE YOUR CONTRIBUTION ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
+
+Should You wish to submit work that is not Your original creation, You may submit it to Vast Data separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]". Vast Data shall have the right to determine at its discretion whether or not to use such Contribution.
+
+Code of Conduct
+
+In Your Contribution and/or Work or otherwise in your participation in the Project, you agree to: (i) not use or include any defamatory, illegal, derogatory, abusive, harassing, destructive, malicious, or otherwise, illegitimate content, code or language; or (ii) publish others’ private or personal information without their explicit permission (“Code of Conduct”) Vast Data’ will be entitled but not obligated to enforce the Code of Conduct and may take appropriate and fair action in response to any behaviour that is reasonably deemed by us to be inappropriate, threatening, offensive, or harmful. Further Vast Data will have the right but not the obligation to remove, edit, or reject any comments, code, issues, and other Contributions or Works submitted hereunder including without limitation by reason of nonalignment with the Code of Conduct. 
+
+You agree to notify Vast Data of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect.

{vastdb-0.1.11/vastdb.egg-info → vastdb-1.1.0}/PKG-INFO

@@ -1,12 +1,12 @@
 Metadata-Version: 2.1
 Name: vastdb
-Version: 0.1.11
+Version: 1.1.0
 Summary: VAST Data SDK
 Home-page: https://github.com/vast-data/vastdb_sdk
 Author: VAST DATA
 Author-email: hello@vastdata.com
 License: Copyright (C) VAST Data Ltd.
-Classifier: Development Status :: 4 - Beta
+Classifier: Development Status :: 5 - Production/Stable
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.9

{vastdb-0.1.11 → vastdb-1.1.0}/README.md

@@ -2,8 +2,6 @@
 
 # VAST DB Python SDK
 
-🚧 Please note that this package in a pre-release stage. Until version 1.x is officially released, the API should be considered unstable.
-
 ## Introduction
 
 `vastdb` is a Python-based SDK designed for interacting with [VAST Database](https://vastdata.com/database) & [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database), enabling schema and table management, efficient ingest, query and modification of columnar data. For more details, see [our whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).

{vastdb-0.1.11 → vastdb-1.1.0}/setup.py

@@ -29,7 +29,7 @@ setup(
     name='vastdb',
     python_requires='>=3.9.0',
     description='VAST Data SDK',
-    version='0.1.11' + suffix,
+    version='1.1.0' + suffix,
     url='https://github.com/vast-data/vastdb_sdk',
     author='VAST DATA',
     author_email='hello@vastdata.com',
@@ -39,7 +39,7 @@ setup(
     long_description=long_description,
     long_description_content_type='text/markdown',
     classifiers=[
-        'Development Status :: 4 - Beta',
+        'Development Status :: 5 - Production/Stable',
         'License :: OSI Approved :: Apache Software License',
         'Programming Language :: Python :: 3',
         'Programming Language :: Python :: 3.9',

{vastdb-0.1.11 → vastdb-1.1.0}/vastdb/_internal.py

@@ -35,6 +35,7 @@ from ibis.expr.operations.logical import (
 )
 from ibis.expr.operations.relations import Field
 from ibis.expr.operations.strings import StringContains
+from ibis.expr.operations.structs import StructField
 
 import vast_flatbuf.org.apache.arrow.computeir.flatbuf.BinaryLiteral as fb_binary_lit
 import vast_flatbuf.org.apache.arrow.computeir.flatbuf.BooleanLiteral as fb_bool_lit
@@ -182,7 +183,7 @@ class Predicate:
                 _logger.debug('OR args: %s op %s', or_args, op)
                 inner_offsets = []
 
-                prev_field_name = None
+                prev_field_path = None
                 for inner_op in or_args:
                     _logger.debug('inner_op %s', inner_op)
                     op_type = type(inner_op)
@@ -216,28 +217,38 @@ class Predicate:
                             if not isinstance(literal, Literal):
                                 raise NotImplementedError(self.expr)
 
+                    field_path = []
+                    while isinstance(column, StructField):
+                        column, subfield_name = column.args
+                        field_path.append(subfield_name)
+
                     if not isinstance(column, Field):
                         raise NotImplementedError(self.expr)
 
-                    field_name = column.name
-                    if prev_field_name is None:
-                        prev_field_name = field_name
-                    elif prev_field_name != field_name:
+                    field_path.append(column.name)
+                    field_path.reverse()  # first entry should be the top-level column name
+
+                    if prev_field_path is None:
+                        prev_field_path = field_path
+                    elif prev_field_path != field_path:
                         raise NotImplementedError(self.expr)
 
-                    node = self.nodes_map[field_name]
+                    nodes_map = self.nodes_map
+                    for name in field_path:
+                        node = nodes_map[name]
+                        nodes_map = node.children_map
+
                     # TODO: support predicate pushdown for leaf nodes (ORION-160338)
                     if node.children:
                         raise NotImplementedError(node.field)  # no predicate pushdown for nested columns
                     column_offset = self.build_column(position=node.index)
-                    field = self.schema.field(field_name)
                     for literal in literals:
                         args_offsets = [column_offset]
                         if literal is not None:
-                            args_offsets.append(self.build_literal(field=field, value=literal.value))
+                            args_offsets.append(self.build_literal(field=node.field, value=literal.value))
                         if builder_func == self.build_between:
-                            args_offsets.append(self.build_literal(field=field, value=lower.value))
-                            args_offsets.append(self.build_literal(field=field, value=upper.value))
+                            args_offsets.append(self.build_literal(field=node.field, value=lower.value))
+                            args_offsets.append(self.build_literal(field=node.field, value=upper.value))
 
                         inner_offsets.append(builder_func(*args_offsets))
 
@@ -572,6 +583,8 @@ class FieldNode:
         else:
             self.children = []  # for non-nested types
 
+        self.children_map = {c.field.name: c for c in self.children}
+
     def _iter_to_root(self) -> Iterator['FieldNode']:
         yield self
         if self.parent is not None:
@@ -748,7 +761,6 @@ class VastdbApi:
 
     def __init__(self, endpoint, access_key, secret_key,
             *,
-            auth_type=AuthType.SIGV4,
             ssl_verify=True,
             timeout=None,
             backoff_config: Optional[BackoffConfig] = None):
@@ -766,15 +778,15 @@ class VastdbApi:
         self._session.verify = ssl_verify
         self._session.headers['user-agent'] = self.client_sdk_version
 
-        backoff_config = backoff_config or BackoffConfig()
+        self.backoff_config = backoff_config or BackoffConfig()
         self._backoff_decorator = backoff.on_exception(
-            wait_gen=backoff_config.wait_gen,
+            wait_gen=self.backoff_config.wait_gen,
             exception=_RETRIABLE_EXCEPTIONS,
             giveup=_backoff_giveup,
-            max_tries=backoff_config.max_tries,
-            max_time=backoff_config.max_time,
-            max_value=backoff_config.max_value,  # passed to `backoff_config.wait_gen`
-            backoff_log_level=backoff_config.backoff_log_level)
+            max_tries=self.backoff_config.max_tries,
+            max_time=self.backoff_config.max_time,
+            max_value=self.backoff_config.max_value,  # passed to `self.backoff_config.wait_gen`
+            backoff_log_level=self.backoff_config.backoff_log_level)
         self._request = self._backoff_decorator(self._single_request)
 
         if url.port in {80, 443, None}:
@@ -812,6 +824,23 @@ class VastdbApi:
         _logger.critical(msg)
         raise NotImplementedError(msg)
 
+    def __enter__(self):
+        """Allow using this session as a context manager."""
+        return self
+
+    def __exit__(self, *args):
+        """Make sure that the connections closed."""
+        self._session.close()
+
+    def with_endpoint(self, endpoint):
+        """Open a new session for targeting a specific endpoint."""
+        return VastdbApi(endpoint=endpoint,
+            access_key=self.access_key,
+            secret_key=self.secret_key,
+            ssl_verify=self._session.verify,
+            timeout=self.timeout,
+            backoff_config=self.backoff_config)
+
     def _single_request(self, *, method, url, skip_status_check=False, **kwargs):
         _logger.debug("Sending request: %s %s %s timeout=%s", method, url, kwargs, self.timeout)
         try:

vastdb-1.1.0/vastdb/bench/perf_bench/bench_repo/mega_combo.py

@@ -0,0 +1,87 @@
+from typing import List, Optional
+
+from vastdb.bench.perf_bench.common.constants import (
+    LOCAL_FS_DS_PATH,
+    NFS_DS_PATH,  # noqa: F401
+    S3_DS_PATH,  # noqa: F401
+    ParquetCompression,
+    VastConnDetails,
+)
+from vastdb.bench.perf_bench.dataset.schemas import DEFAULT_BARS_COLUMNS
+from vastdb.bench.perf_bench.orchestrate.scenario import BenchScenario
+from vastdb.bench.perf_bench.orchestrate.scenario_generator import (
+    generate_perf_bench_scenarios,
+)
+from vastdb.bench.perf_bench.query.query import QueryBackend
+
+
+def build_scenarios(
+    base_key: str,
+    conn_details: Optional[VastConnDetails] = None,
+) -> List[BenchScenario]:
+    return generate_perf_bench_scenarios(
+        base_key=base_key,
+        conn_details=conn_details or VastConnDetails(),
+        query_backends=[
+            QueryBackend.pyarrow,
+            # QueryBackend.vastdb,
+        ],
+        columns_choices=(DEFAULT_BARS_COLUMNS,),
+        universe_choices=(
+            "Single",
+            "Tiny",
+            "SmallSeq",
+            "Medium",
+            "Medium2",
+            "Large",
+        ),
+        num_bdays=[
+            1,  # 1d
+            5,  # 1w
+            # 22,  # 1m
+            65,  # 3m
+            # 130,  # 6m
+            252,  # 1y
+        ],
+
+        # Arrow-specific options
+        fs_path_choices=[
+            # NFS_DS_PATH,
+            LOCAL_FS_DS_PATH,
+            # S3_DS_PATH,
+        ],
+        rowgroup_size_choices=[  # make sure you have previously generated the respective datasets
+            # 64 * 1024,
+            # 128 * 1024,
+            256 * 1024,
+            # 512 * 1024,
+            # DEFAULT_ROW_GROUP_SIZE,
+            # int(1.5 * 1024 * 1024),
+        ],
+        compression_choices=[
+            ParquetCompression.LZ4,
+        ],
+        arrow_batching_spec_choices=[
+            # {"batch_size": 2*2**16, "batch_readahead": 16, "fragment_readahead": 4},
+            # {"batch_size": 6*2**16, "batch_readahead": 12, "fragment_readahead": 4},
+            # DEFAULT_ARROW_KWARGS,
+            {"batch_size": 16 * 2 ** 16, "batch_readahead": 16, "fragment_readahead": 4},
+            # {"batch_size": 24 * 2 ** 16, "batch_readahead": 12, "fragment_readahead": 4},
+            # {"batch_size": 32*2**16, "batch_readahead": 12, "fragment_readahead": 4},
+            # {"batch_size": 64*2**16, "batch_readahead": 12, "fragment_readahead": 4},
+            # {"batch_size": 128*2**16, "batch_readahead": 12, "fragment_readahead": 4},
+        ],
+
+        # VastDB-specific options
+        vdb_num_sub_splits_choices=(
+            # 1,  # Default
+            # 4,
+            8,
+            # 16,
+        ),
+        vdb_num_row_groups_per_sub_split_choices=(
+            # 1,
+            # 4,
+            8,  # Default
+        ),
+    )

vastdb-1.1.0/vastdb/bench/perf_bench/cli.py

@@ -0,0 +1,225 @@
+from pathlib import Path
+from typing import Annotated, List, Optional
+
+import pandas as pd
+import typer
+
+from vastdb.bench.perf_bench.common.constants import (
+    DEFAULT_END_T,
+    DEFAULT_RESULTS_DIR,
+    DEFAULT_START_T,
+    DFAULT_PARQUET_COMPRESSION,
+    LOCAL_FS_DS_PATH,
+    LogLevel,
+    ParquetCompression,
+)
+from vastdb.bench.perf_bench.common.log_utils import (
+    get_logger,
+    set_log_file,
+    set_log_level,
+)
+from vastdb.bench.perf_bench.common.utils import getenv_flag, load_module_from_path
+from vastdb.bench.perf_bench.dataset.generate_secmaster import (
+    SM_PATH,
+    generate_secmaster,
+)
+from vastdb.bench.perf_bench.dataset.generate_stocks_dataset import (
+    generate_concurrent_synthetic_stock_1m_bars,
+)
+from vastdb.bench.perf_bench.orchestrate.scenario import BenchScenario
+from vastdb.bench.perf_bench.run import run_scenarios
+
+app = typer.Typer(pretty_exceptions_enable=getenv_flag("TYPER_PRETTY_EXCEPTIONS"))
+
+_MY_DIR = Path(__file__).parent
+
+
+# noinspection PyUnusedLocal
+@app.callback()
+def cli_common(
+    ctx: typer.Context,
+    verbose: Annotated[
+        bool,
+        typer.Option(
+            "--verbose",
+            is_flag=True,
+        ),
+    ] = False,
+    log_level: Annotated[
+        Optional[LogLevel],
+        typer.Option(
+            "--log-level",
+            case_sensitive=False,
+        ),
+    ] = None,
+    log_file: Annotated[
+        Optional[Path],
+        typer.Option(
+            "--log-file",
+            writable=True,
+            file_okay=True,
+            dir_okay=False,
+            resolve_path=True,
+        ),
+    ] = None,
+):
+    if verbose:
+        log_level = LogLevel.DEBUG
+    if log_level:
+        set_log_level(log_level)
+    if log_file:
+        set_log_file(log_file)
+    get_logger(__name__).info("CLI common setup done.")
+
+
+def _positive_int(value: str) -> int:
+    i_value = int(value)
+    if i_value <= 0:
+        raise typer.BadParameter(f"Must be a positive integer: {value}.")
+    return i_value
+
+
+# noinspection PyUnusedLocal
+@app.command()
+def run_bench(
+    ctx: typer.Context,
+    bench_name: Annotated[
+        str,
+        typer.Option(
+            "--bench-name",
+        ),
+    ],
+    parallelism: Annotated[
+        List[int],
+        typer.Option(
+            "--parallelism",
+            callback=lambda par: [_positive_int(p) for p in par],
+        ),
+    ],
+    runs_per_bench: Annotated[
+        int,
+        typer.Option(
+            "--runs-per-bench",
+            callback=_positive_int,
+        ),
+    ] = 3,
+    bench_generator_path: Annotated[
+        Path,
+        typer.Option(
+            "--bench-generator-path",
+            readable=True,
+            file_okay=True,
+            dir_okay=False,
+            resolve_path=True,
+        ),
+    ] = _MY_DIR / "bench_repo" / "mega_combo.py",
+    results_base_dir: Annotated[
+        Path,
+        typer.Option(
+            "--log-file",
+            writable=True,
+            file_okay=False,
+            dir_okay=True,
+            resolve_path=True,
+        ),
+    ] = DEFAULT_RESULTS_DIR,
+):
+    if not (bench_name := bench_name.strip()):
+        raise typer.BadParameter("Bench name must be non-empty.")
+
+    mod = load_module_from_path(bench_generator_path)
+    scenarios: List[BenchScenario] = mod.build_scenarios(base_key=bench_name)
+    for para in parallelism:
+        run_scenarios(
+            scenarios=scenarios,
+            runs_per_bench=runs_per_bench,
+            parallelism=para,
+            results_base_dir=str(results_base_dir),
+        )
+
+
+# noinspection PyUnusedLocal
+@app.command()
+def build_secmaster(
+    ctx: typer.Context,
+):
+    generate_secmaster()
+
+
+# noinspection PyUnusedLocal
+@app.command()
+def build_dataset(
+    ctx: typer.Context,
+    start_date: Annotated[
+        str,
+        typer.Option(
+            "--start-date",
+            help="Start date for the dataset.",
+            callback=lambda d: pd.Timestamp(d).normalize(),
+        ),
+    ] = DEFAULT_START_T.strftime("%Y%m%d"),
+    end_date: Annotated[
+        str,
+        typer.Option(
+            "--end-date",
+            help="Start date for the dataset.",
+            callback=lambda d: pd.Timestamp(d).normalize(),
+        ),
+    ] = DEFAULT_END_T.strftime("%Y%m%d"),
+    output_dir: Annotated[
+        Path,
+        typer.Option(
+            "--output-dir",
+            writable=True,
+            file_okay=False,
+            dir_okay=True,
+            resolve_path=True,
+        ),
+    ] = LOCAL_FS_DS_PATH,
+    parallelism: Annotated[
+        int,
+        typer.Option(
+            "--parallelism",
+            callback=_positive_int,
+        ),
+    ] = 6,
+    row_group_size: Annotated[
+        int,
+        typer.Option(
+            "--row-group-size",
+            callback=_positive_int,
+            help=(
+                "Row group size for the dataset, some common values are: 64 * 1024, 128 * 1024, 256"
+                " * 1024, 512 * 1024,1024 * 1024, 1.5 * 1024 * 1024."
+            ),
+        ),
+    ] = 256 * 1024,
+    compression: Annotated[
+        ParquetCompression,
+        typer.Option(
+            "--compression",
+            help="Parquet compression algorithm.",
+        ),
+    ] = DFAULT_PARQUET_COMPRESSION,
+):
+    if row_group_size < 1024:
+        raise typer.BadParameter("Row group size must be at least 1024.")
+    if parallelism < 1:
+        raise typer.BadParameter("Parallelism must be at least 1.")
+    if start_date > end_date:
+        raise typer.BadParameter("Start date must be before the end date.")
+    if not SM_PATH.is_file():
+        generate_secmaster()
+    generate_concurrent_synthetic_stock_1m_bars(
+        from_t=start_date,
+        to_t=end_date,
+        output_dir=output_dir,
+        num_workers=parallelism,
+        row_group_size=row_group_size,
+        compression=compression,
+    )
+
+
+if __name__ == "__main__":
+    # Set the metadata only if we execute the main (not on just importing this module)
+    app()

vastdb-1.1.0/vastdb/bench/perf_bench/common/constants.py

@@ -0,0 +1,96 @@
+import logging
+import os
+from dataclasses import dataclass
+from pathlib import Path
+
+import pandas as pd
+from urllib3.util import parse_url
+
+from vastdb.bench.perf_bench.common.types import StrEnum
+
+_MY_DIR = Path(__file__).parent
+
+# VastDB details
+VASTDB_ENDPOINT = os.getenv("VASTDB_ENDPOINT", "")
+VASTDB_BUCKET_NAME = os.getenv("VASTDB_BUCKET_NAME", "")
+VASTDB_TEST_SCHEMA_NAME = os.getenv("VASTDB_TEST_SCHEMA_NAME", "")
+VASTDB_TEST_TABLE_NAME = os.getenv("VASTDB_TEST_TABLE_NAME", "")
+
+# Regular S3 details
+S3_BUCKET_NAME = "my-s3-bucket"
+DEFAULT_S3_SSL_PORT = int(os.getenv("DEFAULT_S3_SSL_PORT", 443))
+if DEFAULT_S3_ENDPOINT_URL := os.getenv("AWS_S3_ENDPOINT_URL", ""):
+    _parsed = parse_url(DEFAULT_S3_ENDPOINT_URL)
+    DEFAULT_S3_HOST = str(_parsed.host)
+    DEFAULT_S3_PORT = int(_parsed.port or 80)
+else:
+    DEFAULT_S3_HOST = os.getenv("DEFAULT_S3_HOST", "1.1.1.1")
+    DEFAULT_S3_PORT = int(os.getenv("DEFAULT_S3_PORT", 80))
+    DEFAULT_S3_ENDPOINT_URL = f"http://{DEFAULT_S3_HOST}:{DEFAULT_S3_PORT}"
+
+# Paths
+DEFAULT_RESULTS_DIR = Path(__file__).parent.parent / "benchmark_results"
+NFS_DS_PATH = f"/mnt/data/{VASTDB_BUCKET_NAME}/data"
+LOCAL_FS_DS_PATH = _MY_DIR.parent / "dataset" / "test_dataset"
+S3_DS_PATH = f"s3://{VASTDB_BUCKET_NAME}/data"
+
+# Access keys
+DEFAULT_ACCESS_KEY = os.getenv("AWS_ACCESS_KEY_ID", "some_access_key")
+DEFAULT_SECRET_KEY = os.getenv("AWS_SECRET_ACCESS_KEY", "some_secret_key")
+
+# Dataset start/end dates
+DEFAULT_START_T = pd.Timestamp("20180101")
+DEFAULT_END_T = pd.Timestamp("20200201")
+
+# Arrow related constants
+DEFAULT_ROW_GROUP_SIZE = 1024 * 1024
+DEFAULT_ARROW_KWARGS = {
+    "batch_size": (DEFAULT_ARROW_BATCH_SIZE := 786432),
+    "batch_readahead": (DEFAULT_ARROW_BATCH_READAHEAD := 12),
+    "fragment_readahead": (DEFAULT_ARROW_BATCH_FRAGMENT_READAHEAD := 4),
+}
+
+
+class ParquetCompression(StrEnum):
+    NONE = "NONE"
+    SNAPPY = "SNAPPY"
+    GZIP = "GZIP"
+    # LZO = "LZO"
+    BROTLI = "BROTLI"
+    LZ4 = "LZ4"
+    ZSTD = "ZSTD"
+
+
+DFAULT_PARQUET_COMPRESSION = ParquetCompression.LZ4
+
+
+class LogLevel(StrEnum):
+    CRITICAL = "CRITICAL"
+    FATAL = "FATAL"
+    ERROR = "ERROR"
+    WARN = "WARNING"
+    WARNING = "WARNING"
+    INFO = "INFO"
+    DEBUG = "DEBUG"
+    NOTSET = "NOTSET"
+
+    def to_int(self) -> int:
+        # noinspection PyUnresolvedReferences,PyProtectedMember
+        return logging._nameToLevel[self]  # noqa: SLF001
+
+
+@dataclass(frozen=True)
+class VastConnDetails:
+    """VAST Config."""
+
+    access: str = DEFAULT_ACCESS_KEY
+    secret: str = DEFAULT_SECRET_KEY
+    vastdb_bucket: str = VASTDB_BUCKET_NAME
+    vastdb_endpoint: str = VASTDB_ENDPOINT
+    vastdb_ssl_verify: bool = True
+    vastdb_schema: str = VASTDB_TEST_SCHEMA_NAME
+    vastdb_table: str = VASTDB_TEST_TABLE_NAME
+    s3_host: str = DEFAULT_S3_HOST
+    s3_bucket: str = VASTDB_BUCKET_NAME
+    s3_port: int = DEFAULT_S3_PORT
+    s3_ssl_port: int = DEFAULT_S3_SSL_PORT