vastdb 0.0.5.3__tar.gz → 0.1.1__tar.gz

vastdb-0.1.1/CHANGELOG.md

@@ -0,0 +1,95 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+
+## [0.1.1] (2024-04-15)
+[0.1.1]: https://github.com/vast-data/vastdb_sdk/compare/v0.1.0...v0.1.1
+
+### Added
+ - Support date & timestamp predicate pushdown [#25]
+
+### Fixed
+ - Add missing `requirements.txt` file to released tarball [#47]
+
+## [0.1.0] (2024-04-11)
+[0.1.0]: https://github.com/vast-data/vastdb_sdk/compare/v0.0.5.3...v0.1.0
+
+### Added
+- Transactions are implemented using [context managers](https://docs.python.org/3/library/stdtypes.html#typecontextmanager)
+- Database entities are derived from a specific transaction
+- Predicate pushdown is using [ibis](https://ibis-project.org/)-like expressions
+- Public SDK [test suite](vastdb/tests/)
+- Internal CI with Python 3.9 - 3.12
+- Using [`ruff`](https://github.com/astral-sh/ruff) for linting and code formatting
+- Improved [`README.md`](README.md) with new API examples
+
+### Removed
+
+- Old SDK classes and methods are deprecated
+
+## [0.0.5.3] (2024-03-24)
+[0.0.5.3]: https://github.com/vast-data/vastdb_sdk/compare/v0.0.5.2...v0.0.5.3
+
+### Added
+- Allow changing default max list_columns page size [#43]
+
+### Fixed
+- Fixing insert function to use the correct tx while listing columns [#42]
+
+## [0.0.5.2] (2024-03-19)
+[0.0.5.2]: https://github.com/vast-data/vastdb_sdk/compare/v0.0.5.1...v0.0.5.2
+
+### Added
+- Support retrieving row IDs via `VastdbApi.query()` [#31]
+
+### Fixed
+- Fixed several functions that didn't use the correct txid [#30]
+- Fix graceful cancellation for `VastdbApi.query_iterator()` [#33]
+
+### Removed
+- Remove `vast_protobuf` package and `protobuf` dependency [#29]
+
+
+## [0.0.5.1] (2024-03-01)
+[0.0.5.1]: https://github.com/vast-data/vastdb_sdk/compare/v0.0.5.0...v0.0.5.1
+
+### Fixed
+- Support latest Python (3.11+) by not requiring too old dependencies. [#16]
+
+
+## [0.0.5.0] (2024-02-25)
+[0.0.5.0]: https://github.com/vast-data/vastdb_sdk/compare/v0.0.4.0...v0.0.5.0
+
+### Added
+- Created a CHANGELOG
+
+### Fixed
+- Wait until all files are processed during import
+- Sync with internal `api.py` code
+
+
+## [0.0.4.0] (2024-02-14)
+[0.0.4.0]: https://github.com/vast-data/vastdb_sdk/compare/v0.0.2...v0.0.4.0
+
+### Added
+- Allow creating a table using the schema of an existing Parquet file via `create_table_from_parquet_schema()`
+- Allow inserting RecordBatch into DB via `insert()`
+
+### Changed
+- Rename `vastdb_api` module to `api`
+
+### Fixed
+- Fix `concurrent.futures` import error
+
+
+[#16]: https://github.com/vast-data/vastdb_sdk/pull/16
+[#25]: https://github.com/vast-data/vastdb_sdk/pull/25
+[#29]: https://github.com/vast-data/vastdb_sdk/pull/29
+[#30]: https://github.com/vast-data/vastdb_sdk/pull/30
+[#31]: https://github.com/vast-data/vastdb_sdk/pull/31
+[#33]: https://github.com/vast-data/vastdb_sdk/pull/33
+[#42]: https://github.com/vast-data/vastdb_sdk/pull/42
+[#43]: https://github.com/vast-data/vastdb_sdk/pull/43
+[#47]: https://github.com/vast-data/vastdb_sdk/pull/47

vastdb-0.1.1/MANIFEST.in

@@ -0,0 +1 @@
+include *.md LICENSE requirements.txt

vastdb-0.1.1/PKG-INFO

@@ -0,0 +1,31 @@
+Metadata-Version: 2.1
+Name: vastdb
+Version: 0.1.1
+Summary: VAST Data SDK
+Home-page: https://github.com/vast-data/vastdb_sdk
+Author: VAST DATA
+Author-email: hello@vastdata.com
+License: Copyright (C) VAST Data Ltd.
+Platform: UNKNOWN
+Classifier: Development Status :: 4 - Beta
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Topic :: Database :: Front-Ends
+Requires-Python: >=3.9.0
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+
+`vastdb` is a Python-based SDK designed for interacting
+with [VAST Database](https://vastdata.com/database)
+and [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database),
+enabling schema and table management, efficient ingest, query and modification of columnar data.
+
+For more details, see [our whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).
+
+

vastdb-0.1.1/README.md

@@ -0,0 +1,181 @@
+[![version](https://img.shields.io/pypi/v/vastdb.svg)](https://pypi.org/project/vastdb)
+
+# VAST DB Python SDK
+
+🚧 Please note that this package in a pre-release stage. Until version 1.x is officially released, the API should be considered unstable.
+
+## Introduction
+
+`vastdb` is a Python-based SDK designed for interacting with [VAST Database](https://vastdata.com/database) & [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database), enabling schema and table management, efficient ingest, query and modification of columnar data. For more details, see [our whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).
+
+[![vastdb](docs/vastdb.png)](https://vastdata.com/database)
+
+## Getting started
+
+### Requirements
+
+- Linux client with Python 3.9+ and a network access to the VAST cluster
+- [Virtual IP pool configured with DNS service](https://support.vastdata.com/s/topic/0TOV40000000FThOAM/configuring-network-access-v50)
+- [S3 access & secret keys on VAST cluster](https://support.vastdata.com/s/article/UUID-4d2e7e23-b2fb-7900-d98f-96c31a499626)
+- [Tabular identity policy with the proper permissions](https://support.vastdata.com/s/article/UUID-14322b60-d6a2-89ac-3df0-3dfbb6974182)
+
+
+### Installation
+
+```bash
+pip install vastdb
+```
+
+Also, see [our release notes](CHANGELOG.md).
+
+### Quickstart
+
+Creating schemas and tables + basic inserts and selects:
+
+```python
+import pyarrow as pa
+import vastdb
+
+session = vastdb.connect(
+    endpoint='http://vip-pool.v123-xy.VastENG.lab',
+    access=AWS_ACCESS_KEY_ID,
+    secret=AWS_SECRET_ACCESS_KEY)
+
+with session.transaction() as tx:
+    bucket = tx.bucket("bucket-name")
+
+    schema = bucket.create_schema("schema-name")
+    print(bucket.schemas())
+
+    columns = pa.schema([
+        ('c1', pa.int16()),
+        ('c2', pa.float32()),
+        ('c3', pa.utf8()),
+    ])
+    table = schema.create_table("table-name", columns)
+    print(schema.tables())
+    print(table.columns())
+
+    arrow_table = pa.table(schema=columns, data=[
+        [111, 222, 333],
+        [0.5, 1.5, 2.5],
+        ['a', 'bb', 'ccc'],
+    ])
+    table.insert(arrow_table)
+
+    result = pa.Table.from_batches(table.select())  # SELECT * FROM t
+    assert result == arrow_table
+
+    # the transaction is automatically committed when exiting the context
+```
+
+## Filters and projections
+
+Our SDK supports predicate and projection pushdown:
+
+```python
+    # SELECT c1 FROM t WHERE (c2 > 2) AND (c3 IS NULL)
+    table.select(columns=['c1'],
+                 predicate=(table['c2'] > 2) & table['c3'].isnull())
+
+    # SELECT c2, c3 FROM t WHERE (c2 BETWEEN 0 AND 1) OR (c2 > 10)
+    table.select(columns=['c2', 'c3'],
+                 predicate=table['c2'].between(0, 1) | (table['c2'] > 10))
+
+    # SELECT * FROM t WHERE c3 LIKE '%substring%'
+    table.select(predicate=table['c3'].contains('substring'))
+```
+
+## Import a single Parquet file via S3 protocol
+
+It is possible to efficiently create a table from a Parquet file (without copying it via the client):
+
+```python
+    with tempfile.NamedTemporaryFile() as f:
+        pa.parquet.write_table(arrow_table, f.name)
+        s3.put_object(Bucket='bucket-name', Key='staging/file.parquet', Body=f)
+
+    schema = tx.bucket('bucket-name').schema('schema-name')
+    table = util.create_table_from_files(
+        schema=schema, table_name='imported-table',
+        parquet_files=['/bucket-name/staging/file.parquet'])
+```
+
+## Import multiple Parquet files concurrently via S3 protocol
+
+We can import multiple files concurrently into a table (by utilizing multiple CNodes' cores):
+
+```python
+    schema = tx.bucket('bucket-name').schema('schema-name')
+    table = util.create_table_from_files(
+        schema=schema, table_name='large-imported-table',
+        parquet_files=[f'/bucket-name/staging/file{i}.parquet' for i in range(10)])
+```
+
+## Post-processing
+
+### Export
+
+`Table.select()` returns a stream of [PyArrow record batches](https://arrow.apache.org/docs/python/data.html#record-batches), which can be directly exported into a Parquet file:
+
+```python
+batches = table.select()
+with contextlib.closing(pa.parquet.ParquetWriter('/path/to/file.parquet', batches.schema)) as writer:
+    for batch in table_batches:
+        writer.write_batch(batch)
+```
+
+### DuckDB
+
+We can use [DuckDB](https://duckdb.org/docs/guides/python/sql_on_arrow.html) to post-process the resulting stream of [PyArrow record batches](https://arrow.apache.org/docs/python/data.html#record-batches):
+
+```python
+import duckdb
+conn = duckdb.connect()
+
+batches = table.select(columns=['c1'], predicate=(table['c2'] > 2))
+print(conn.execute("SELECT sum(c1) FROM batches").arrow())
+```
+
+## Semi-sorted projections
+
+We can create, list and delete [available semi-sorted projections](https://support.vastdata.com/s/article/UUID-e4ca42ab-d15b-6b72-bd6b-f3c77b455de4):
+
+```python
+p = table.create_projection('proj', sorted=['c3'], unsorted=['c1'])
+print(table.projections())
+print(p.get_stats())
+p.drop()
+```
+
+## Snapshots
+
+It is possible to list [available snapshots](https://vastdata.com/blog/bringing-snapshots-to-vasts-element-store):
+
+```python
+print(bucket.list_snapshots())
+```
+
+## VAST Catalog
+
+[VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database) can be queried as a regular table:
+
+```python
+table = pa.Table.from_batches(tx.catalog.select(['element_type']))
+df = table.to_pandas()
+
+total_elements = len(df)
+print(f"Total elements in the catalog: {total_elements}")
+
+file_count = (df['element_type'] == 'FILE').sum()
+print(f"Number of files/objects: {file_count}")
+
+distinct_elements = df['element_type'].unique()
+print("Distinct element types on the system:")
+print(distinct_elements)
+```
+
+See the following blog posts for more examples:
+
+- https://vastdata.com/blog/the-vast-catalog-in-action-part-1
+- https://vastdata.com/blog/the-vast-catalog-in-action-part-2

vastdb-0.1.1/requirements.txt

@@ -0,0 +1,7 @@
+aws-requests-auth
+boto3
+flatbuffers
+ibis-framework
+pyarrow
+requests
+xmltodict

vastdb-0.1.1/setup.py

@@ -0,0 +1,49 @@
+import os
+
+from setuptools import find_packages, setup
+
+long_description = """
+`vastdb` is a Python-based SDK designed for interacting
+with [VAST Database](https://vastdata.com/database)
+and [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database),
+enabling schema and table management, efficient ingest, query and modification of columnar data.
+
+For more details, see [our whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).
+"""
+
+def _get_version_suffix():
+    import subprocess
+
+    commit = subprocess.check_output(["git", "rev-parse", "HEAD"])
+    print(f"Git commit: {commit}")
+    return f".dev1+vast.{commit.decode()[:16]}"
+
+suffix = ''
+if os.environ.get('VASTDB_APPEND_VERSION_SUFFIX'):
+    suffix = _get_version_suffix()
+
+setup(
+    name='vastdb',
+    python_requires='>=3.9.0',
+    description='VAST Data SDK',
+    version='0.1.1' + suffix,
+    url='https://github.com/vast-data/vastdb_sdk',
+    author='VAST DATA',
+    author_email='hello@vastdata.com',
+    license='Copyright (C) VAST Data Ltd.',
+    packages=find_packages(),
+    install_requires=open('requirements.txt').read().strip().split('\n'),
+    long_description=long_description,
+    long_description_content_type='text/markdown',
+    classifiers=[
+        'Development Status :: 4 - Beta',
+        'License :: OSI Approved :: Apache Software License',
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
+        'Programming Language :: Python :: 3.11',
+        'Programming Language :: Python :: 3.12',
+        'Topic :: Database',
+        'Topic :: Database :: Front-Ends',
+    ],
+)

{vastdb-0.0.5.3 → vastdb-0.1.1}/vast_flatbuf/tabular/GetTableStatsResponse.py

@@ -45,7 +45,39 @@ class GetTableStatsResponse(object):
             return bool(self._tab.Get(flatbuffers.number_types.BoolFlags, o + self._tab.Pos))
         return False
 
-def Start(builder): builder.StartObject(3)
+    # GetTableStatsResponse
+    def AddressType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(10))
+        if o != 0:
+            return self._tab.String(o + self._tab.Pos)
+        return None
+
+    # GetTableStatsResponse
+    def Vips(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(12))
+        if o != 0:
+            x = self._tab.Vector(o)
+            x += flatbuffers.number_types.UOffsetTFlags.py_type(j) * 4
+            x = self._tab.Indirect(x)
+            from vast_flatbuf.tabular.VipRange import VipRange
+            obj = VipRange()
+            obj.Init(self._tab.Bytes, x)
+            return obj
+        return None
+
+    # GetTableStatsResponse
+    def VipsLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(12))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+    # GetTableStatsResponse
+    def VipsIsNone(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(12))
+        return o == 0
+
+def Start(builder): builder.StartObject(5)
 def GetTableStatsResponseStart(builder):
     """This method is deprecated. Please switch to Start."""
     return Start(builder)
@@ -61,6 +93,18 @@ def AddIsExternalRowidAlloc(builder, isExternalRowidAlloc): builder.PrependBoolS
 def GetTableStatsResponseAddIsExternalRowidAlloc(builder, isExternalRowidAlloc):
     """This method is deprecated. Please switch to AddIsExternalRowidAlloc."""
     return AddIsExternalRowidAlloc(builder, isExternalRowidAlloc)
+def AddAddressType(builder, addressType): builder.PrependUOffsetTRelativeSlot(3, flatbuffers.number_types.UOffsetTFlags.py_type(addressType), 0)
+def GetTableStatsResponseAddAddressType(builder, addressType):
+    """This method is deprecated. Please switch to AddAddressType."""
+    return AddAddressType(builder, addressType)
+def AddVips(builder, vips): builder.PrependUOffsetTRelativeSlot(4, flatbuffers.number_types.UOffsetTFlags.py_type(vips), 0)
+def GetTableStatsResponseAddVips(builder, vips):
+    """This method is deprecated. Please switch to AddVips."""
+    return AddVips(builder, vips)
+def StartVipsVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def GetTableStatsResponseStartVipsVector(builder, numElems):
+    """This method is deprecated. Please switch to Start."""
+    return StartVipsVector(builder, numElems)
 def End(builder): return builder.EndObject()
 def GetTableStatsResponseEnd(builder):
     """This method is deprecated. Please switch to End."""

vastdb-0.1.1/vast_flatbuf/tabular/VipRange.py

@@ -0,0 +1,56 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: tabular
+
+import flatbuffers
+from flatbuffers.compat import import_numpy
+np = import_numpy()
+
+class VipRange(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAs(cls, buf, offset=0):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = VipRange()
+        x.Init(buf, n + offset)
+        return x
+
+    @classmethod
+    def GetRootAsVipRange(cls, buf, offset=0):
+        """This method is deprecated. Please switch to GetRootAs."""
+        return cls.GetRootAs(buf, offset)
+    # VipRange
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # VipRange
+    def StartAddress(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.String(o + self._tab.Pos)
+        return None
+
+    # VipRange
+    def AddressCount(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint16Flags, o + self._tab.Pos)
+        return 0
+
+def Start(builder): builder.StartObject(2)
+def VipRangeStart(builder):
+    """This method is deprecated. Please switch to Start."""
+    return Start(builder)
+def AddStartAddress(builder, startAddress): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(startAddress), 0)
+def VipRangeAddStartAddress(builder, startAddress):
+    """This method is deprecated. Please switch to AddStartAddress."""
+    return AddStartAddress(builder, startAddress)
+def AddAddressCount(builder, addressCount): builder.PrependUint16Slot(1, addressCount, 0)
+def VipRangeAddAddressCount(builder, addressCount):
+    """This method is deprecated. Please switch to AddAddressCount."""
+    return AddAddressCount(builder, addressCount)
+def End(builder): return builder.EndObject()
+def VipRangeEnd(builder):
+    """This method is deprecated. Please switch to End."""
+    return End(builder)

vastdb-0.1.1/vastdb/__init__.py

@@ -0,0 +1,7 @@
+"""VAST Database Python SDK."""
+
+from . import session
+
+# A helper function, useful as a short-hand for Session c-tor: `session = vastdb.connect(...)`
+connect = session.Session
+connect.__name__ = 'connect'

vastdb-0.1.1/vastdb/bench/test_perf.py

@@ -0,0 +1,29 @@
+import logging
+import time
+
+import pyarrow as pa
+import pytest
+
+from vastdb import util
+from vastdb.table import ImportConfig, QueryConfig
+
+log = logging.getLogger(__name__)
+
+
+@pytest.mark.benchmark
+def test_bench(session, clean_bucket_name, parquets_path, crater_path):
+    files = [str(parquets_path/f) for f in (parquets_path.glob('**/*.pq'))]
+
+    with session.transaction() as tx:
+        b = tx.bucket(clean_bucket_name)
+        s = b.create_schema('s1')
+        t = util.create_table_from_files(s, 't1', files, config=ImportConfig(import_concurrency=8))
+        config = QueryConfig(num_splits=8, num_sub_splits=4)
+        s = time.time()
+        pa_table = pa.Table.from_batches(t.select(columns=['sid'], predicate=t['sid'] == 10033007, config=config))
+        e = time.time()
+        log.info("'SELECT sid from TABLE WHERE sid = 10033007' returned in %s seconds.", e-s)
+        if crater_path:
+            with open(f'{crater_path}/bench_results', 'a') as f:
+                f.write(f"'SELECT sid FROM TABLE WHERE sid = 10033007' returned in {e-s} seconds")
+        assert pa_table.num_rows == 255_075

vastdb-0.1.1/vastdb/bucket.py

@@ -0,0 +1,85 @@
+"""VAST Database bucket.
+
+VAST S3 buckets can be used to create Database schemas and tables.
+It is possible to list and access VAST snapshots generated over a bucket.
+"""
+
+import logging
+from dataclasses import dataclass
+
+from . import errors, schema, transaction
+
+log = logging.getLogger(__name__)
+
+
+@dataclass
+class Snapshot:
+    """VAST bucket-level snapshot."""
+
+    name: str
+    bucket: "Bucket"
+
+
+@dataclass
+class Bucket:
+    """VAST bucket."""
+
+    name: str
+    tx: "transaction.Transaction"
+
+    def create_schema(self, path: str, fail_if_exists=True) -> "schema.Schema":
+        """Create a new schema (a container of tables) under this bucket."""
+        if current := self.schema(path, fail_if_missing=False):
+            if fail_if_exists:
+                raise errors.SchemaExists(self.name, path)
+            else:
+                return current
+        self.tx._rpc.api.create_schema(self.name, path, txid=self.tx.txid)
+        log.info("Created schema: %s", path)
+        return self.schema(path)
+
+    def schema(self, path: str, fail_if_missing=True) -> "schema.Schema":
+        """Get a specific schema (a container of tables) under this bucket."""
+        s = self.schemas(path)
+        log.debug("schema: %s", s)
+        if not s:
+            if fail_if_missing:
+                raise errors.MissingSchema(self.name, path)
+            else:
+                return None
+        assert len(s) == 1, f"Expected to receive only a single schema, but got: {len(s)}. ({s})"
+        log.debug("Found schema: %s", s[0].name)
+        return s[0]
+
+    def schemas(self, name: str = None) -> ["schema.Schema"]:
+        """List bucket's schemas."""
+        schemas = []
+        next_key = 0
+        exact_match = bool(name)
+        log.debug("list schemas param: schema=%s, exact_match=%s", name, exact_match)
+        while True:
+            bucket_name, curr_schemas, next_key, is_truncated, _ = \
+                self.tx._rpc.api.list_schemas(bucket=self.name, next_key=next_key, txid=self.tx.txid,
+                                               name_prefix=name, exact_match=exact_match)
+            if not curr_schemas:
+                break
+            schemas.extend(curr_schemas)
+            if not is_truncated:
+                break
+
+        return [schema.Schema(name=name, bucket=self) for name, *_ in schemas]
+
+    def snapshots(self) -> [Snapshot]:
+        """List bucket's snapshots."""
+        snapshots = []
+        next_key = 0
+        while True:
+            curr_snapshots, is_truncated, next_key = \
+                self.tx._rpc.api.list_snapshots(bucket=self.name, next_token=next_key)
+            if not curr_snapshots:
+                break
+            snapshots.extend(curr_snapshots)
+            if not is_truncated:
+                break
+
+        return [Snapshot(name=snapshot, bucket=self) for snapshot in snapshots]

vastdb-0.1.1/vastdb/conftest.py

@@ -0,0 +1,60 @@
+import os
+from pathlib import Path
+
+import boto3
+import pytest
+
+import vastdb
+
+
+def pytest_addoption(parser):
+    parser.addoption("--tabular-bucket-name", help="Name of the S3 bucket with Tabular enabled", default="vastdb")
+    parser.addoption("--tabular-access-key", help="Access key with Tabular permissions (AWS_ACCESS_KEY_ID)", default=os.environ.get("AWS_ACCESS_KEY_ID", None))
+    parser.addoption("--tabular-secret-key", help="Secret key with Tabular permissions (AWS_SECRET_ACCESS_KEY)", default=os.environ.get("AWS_SECRET_ACCESS_KEY", None))
+    parser.addoption("--tabular-endpoint-url", help="Tabular server endpoint", default="http://localhost:9090")
+    parser.addoption("--data-path", help="Data files location", default=None)
+    parser.addoption("--crater-path", help="Save benchmark results in a dedicated location", default=None)
+
+
+@pytest.fixture(scope="session")
+def session(request):
+    return vastdb.connect(
+        access=request.config.getoption("--tabular-access-key"),
+        secret=request.config.getoption("--tabular-secret-key"),
+        endpoint=request.config.getoption("--tabular-endpoint-url"),
+    )
+
+
+@pytest.fixture(scope="session")
+def test_bucket_name(request):
+    return request.config.getoption("--tabular-bucket-name")
+
+
+@pytest.fixture(scope="function")
+def clean_bucket_name(request, test_bucket_name, session):
+    with session.transaction() as tx:
+        b = tx.bucket(test_bucket_name)
+        for s in b.schemas():
+            for t in s.tables():
+                t.drop()
+            s.drop()
+    return test_bucket_name
+
+
+@pytest.fixture(scope="session")
+def s3(request):
+    return boto3.client(
+        's3',
+        aws_access_key_id=request.config.getoption("--tabular-access-key"),
+        aws_secret_access_key=request.config.getoption("--tabular-secret-key"),
+        endpoint_url=request.config.getoption("--tabular-endpoint-url"))
+
+
+@pytest.fixture(scope="function")
+def parquets_path(request):
+    return Path(request.config.getoption("--data-path"))
+
+
+@pytest.fixture(scope="function")
+def crater_path(request):
+    return request.config.getoption("--crater-path")