vastdb 0.0.5.3__tar.gz → 0.1.0__tar.gz

vastdb-0.1.0/PKG-INFO

@@ -0,0 +1,31 @@
+Metadata-Version: 2.1
+Name: vastdb
+Version: 0.1.0
+Summary: VAST Data SDK
+Home-page: https://github.com/vast-data/vastdb_sdk
+Author: VAST DATA
+Author-email: hello@vastdata.com
+License: Copyright (C) VAST Data Ltd.
+Platform: UNKNOWN
+Classifier: Development Status :: 4 - Beta
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Topic :: Database :: Front-Ends
+Requires-Python: >=3.9.0
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+
+`vastdb` is a Python-based SDK designed for interacting
+with [VAST Database](https://vastdata.com/database)
+and [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database),
+enabling schema and table management, efficient ingest, query and modification of columnar data.
+
+For more details, see [our whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).
+
+

vastdb-0.1.0/README.md

@@ -0,0 +1,181 @@
+[![version](https://img.shields.io/pypi/v/vastdb.svg)](https://pypi.org/project/vastdb)
+
+# VAST DB Python SDK
+
+🚧 Please note that this package in a pre-release stage. Until version 1.x is officially released, the API should be considered unstable.
+
+## Introduction
+
+`vastdb` is a Python-based SDK designed for interacting with [VAST Database](https://vastdata.com/database) & [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database), enabling schema and table management, efficient ingest, query and modification of columnar data. For more details, see [our whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).
+
+[![vastdb](docs/vastdb.png)](https://vastdata.com/database)
+
+## Getting started
+
+### Requirements
+
+- Linux client with Python 3.9+ and a network access to the VAST cluster
+- [Virtual IP pool configured with DNS service](https://support.vastdata.com/s/topic/0TOV40000000FThOAM/configuring-network-access-v50)
+- [S3 access & secret keys on VAST cluster](https://support.vastdata.com/s/article/UUID-4d2e7e23-b2fb-7900-d98f-96c31a499626)
+- [Tabular identity policy with the proper permissions](https://support.vastdata.com/s/article/UUID-14322b60-d6a2-89ac-3df0-3dfbb6974182)
+
+
+### Installation
+
+```bash
+pip install vastdb
+```
+
+Also, see [our release notes](CHANGELOG.md).
+
+### Quickstart
+
+Creating schemas and tables + basic inserts and selects:
+
+```python
+import pyarrow as pa
+import vastdb
+
+session = vastdb.connect(
+    endpoint='http://vip-pool.v123-xy.VastENG.lab',
+    access=AWS_ACCESS_KEY_ID,
+    secret=AWS_SECRET_ACCESS_KEY)
+
+with session.transaction() as tx:
+    bucket = tx.bucket("bucket-name")
+
+    schema = bucket.create_schema("schema-name")
+    print(bucket.schemas())
+
+    columns = pa.schema([
+        ('c1', pa.int16()),
+        ('c2', pa.float32()),
+        ('c3', pa.utf8()),
+    ])
+    table = schema.create_table("table-name", columns)
+    print(schema.tables())
+    print(table.columns())
+
+    arrow_table = pa.table(schema=columns, data=[
+        [111, 222, 333],
+        [0.5, 1.5, 2.5],
+        ['a', 'bb', 'ccc'],
+    ])
+    table.insert(arrow_table)
+
+    result = pa.Table.from_batches(table.select())  # SELECT * FROM t
+    assert result == arrow_table
+
+    # the transaction is automatically committed when exiting the context
+```
+
+## Filters and projections
+
+Our SDK supports predicate and projection pushdown:
+
+```python
+    # SELECT c1 FROM t WHERE (c2 > 2) AND (c3 IS NULL)
+    table.select(columns=['c1'],
+                 predicate=(table['c2'] > 2) & table['c3'].isnull())
+
+    # SELECT c2, c3 FROM t WHERE (c2 BETWEEN 0 AND 1) OR (c2 > 10)
+    table.select(columns=['c2', 'c3'],
+                 predicate=table['c2'].between(0, 1) | (table['c2'] > 10))
+
+    # SELECT * FROM t WHERE c3 LIKE '%substring%'
+    table.select(predicate=table['c3'].contains('substring'))
+```
+
+## Import a single Parquet file via S3 protocol
+
+It is possible to efficiently create a table from a Parquet file (without copying it via the client):
+
+```python
+    with tempfile.NamedTemporaryFile() as f:
+        pa.parquet.write_table(arrow_table, f.name)
+        s3.put_object(Bucket='bucket-name', Key='staging/file.parquet', Body=f)
+
+    schema = tx.bucket('bucket-name').schema('schema-name')
+    table = util.create_table_from_files(
+        schema=schema, table_name='imported-table',
+        parquet_files=['/bucket-name/staging/file.parquet'])
+```
+
+## Import multiple Parquet files concurrently via S3 protocol
+
+We can import multiple files concurrently into a table (by utilizing multiple CNodes' cores):
+
+```python
+    schema = tx.bucket('bucket-name').schema('schema-name')
+    table = util.create_table_from_files(
+        schema=schema, table_name='large-imported-table',
+        parquet_files=[f'/bucket-name/staging/file{i}.parquet' for i in range(10)])
+```
+
+## Post-processing
+
+### Export
+
+`Table.select()` returns a stream of [PyArrow record batches](https://arrow.apache.org/docs/python/data.html#record-batches), which can be directly exported into a Parquet file:
+
+```python
+batches = table.select()
+with contextlib.closing(pa.parquet.ParquetWriter('/path/to/file.parquet', batches.schema)) as writer:
+    for batch in table_batches:
+        writer.write_batch(batch)
+```
+
+### DuckDB
+
+We can use [DuckDB](https://duckdb.org/docs/guides/python/sql_on_arrow.html) to post-process the resulting stream of [PyArrow record batches](https://arrow.apache.org/docs/python/data.html#record-batches):
+
+```python
+import duckdb
+conn = duckdb.connect()
+
+batches = table.select(columns=['c1'], predicate=(table['c2'] > 2))
+print(conn.execute("SELECT sum(c1) FROM batches").arrow())
+```
+
+## Semi-sorted projections
+
+We can create, list and delete [available semi-sorted projections](https://support.vastdata.com/s/article/UUID-e4ca42ab-d15b-6b72-bd6b-f3c77b455de4):
+
+```python
+p = table.create_projection('proj', sorted=['c3'], unsorted=['c1'])
+print(table.projections())
+print(p.get_stats())
+p.drop()
+```
+
+## Snapshots
+
+It is possible to list [available snapshots](https://vastdata.com/blog/bringing-snapshots-to-vasts-element-store):
+
+```python
+print(bucket.list_snapshots())
+```
+
+## VAST Catalog
+
+[VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database) can be queried as a regular table:
+
+```python
+table = pa.Table.from_batches(tx.catalog.select(['element_type']))
+df = table.to_pandas()
+
+total_elements = len(df)
+print(f"Total elements in the catalog: {total_elements}")
+
+file_count = (df['element_type'] == 'FILE').sum()
+print(f"Number of files/objects: {file_count}")
+
+distinct_elements = df['element_type'].unique()
+print("Distinct element types on the system:")
+print(distinct_elements)
+```
+
+See the following blog posts for more examples:
+
+- https://vastdata.com/blog/the-vast-catalog-in-action-part-1
+- https://vastdata.com/blog/the-vast-catalog-in-action-part-2

vastdb-0.1.0/setup.py

@@ -0,0 +1,48 @@
+import os
+from setuptools import setup, find_packages
+
+long_description = """
+`vastdb` is a Python-based SDK designed for interacting
+with [VAST Database](https://vastdata.com/database)
+and [VAST Catalog](https://vastdata.com/blog/vast-catalog-treat-your-file-system-like-a-database),
+enabling schema and table management, efficient ingest, query and modification of columnar data.
+
+For more details, see [our whitepaper](https://vastdata.com/whitepaper/#TheVASTDataBase).
+"""
+
+def _get_version_suffix():
+    import subprocess
+
+    commit = subprocess.check_output(["git", "rev-parse", "HEAD"])
+    print(f"Git commit: {commit}")
+    return f".dev1+vast.{commit.decode()[:16]}"
+
+suffix = ''
+if os.environ.get('VASTDB_APPEND_VERSION_SUFFIX'):
+    suffix = _get_version_suffix()
+
+setup(
+    name='vastdb',
+    python_requires='>=3.9.0',
+    description='VAST Data SDK',
+    version='0.1.0' + suffix,
+    url='https://github.com/vast-data/vastdb_sdk',
+    author='VAST DATA',
+    author_email='hello@vastdata.com',
+    license='Copyright (C) VAST Data Ltd.',
+    packages=find_packages(),
+    install_requires=open('requirements.txt').read().strip().split('\n'),
+    long_description=long_description,
+    long_description_content_type='text/markdown',
+    classifiers=[
+        'Development Status :: 4 - Beta',
+        'License :: OSI Approved :: Apache Software License',
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
+        'Programming Language :: Python :: 3.11',
+        'Programming Language :: Python :: 3.12',
+        'Topic :: Database',
+        'Topic :: Database :: Front-Ends',
+    ],
+)

{vastdb-0.0.5.3 → vastdb-0.1.0}/vast_flatbuf/tabular/GetTableStatsResponse.py

@@ -45,7 +45,39 @@ class GetTableStatsResponse(object):
             return bool(self._tab.Get(flatbuffers.number_types.BoolFlags, o + self._tab.Pos))
         return False
 
-def Start(builder): builder.StartObject(3)
+    # GetTableStatsResponse
+    def AddressType(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(10))
+        if o != 0:
+            return self._tab.String(o + self._tab.Pos)
+        return None
+
+    # GetTableStatsResponse
+    def Vips(self, j):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(12))
+        if o != 0:
+            x = self._tab.Vector(o)
+            x += flatbuffers.number_types.UOffsetTFlags.py_type(j) * 4
+            x = self._tab.Indirect(x)
+            from vast_flatbuf.tabular.VipRange import VipRange
+            obj = VipRange()
+            obj.Init(self._tab.Bytes, x)
+            return obj
+        return None
+
+    # GetTableStatsResponse
+    def VipsLength(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(12))
+        if o != 0:
+            return self._tab.VectorLen(o)
+        return 0
+
+    # GetTableStatsResponse
+    def VipsIsNone(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(12))
+        return o == 0
+
+def Start(builder): builder.StartObject(5)
 def GetTableStatsResponseStart(builder):
     """This method is deprecated. Please switch to Start."""
     return Start(builder)
@@ -61,6 +93,18 @@ def AddIsExternalRowidAlloc(builder, isExternalRowidAlloc): builder.PrependBoolS
 def GetTableStatsResponseAddIsExternalRowidAlloc(builder, isExternalRowidAlloc):
     """This method is deprecated. Please switch to AddIsExternalRowidAlloc."""
     return AddIsExternalRowidAlloc(builder, isExternalRowidAlloc)
+def AddAddressType(builder, addressType): builder.PrependUOffsetTRelativeSlot(3, flatbuffers.number_types.UOffsetTFlags.py_type(addressType), 0)
+def GetTableStatsResponseAddAddressType(builder, addressType):
+    """This method is deprecated. Please switch to AddAddressType."""
+    return AddAddressType(builder, addressType)
+def AddVips(builder, vips): builder.PrependUOffsetTRelativeSlot(4, flatbuffers.number_types.UOffsetTFlags.py_type(vips), 0)
+def GetTableStatsResponseAddVips(builder, vips):
+    """This method is deprecated. Please switch to AddVips."""
+    return AddVips(builder, vips)
+def StartVipsVector(builder, numElems): return builder.StartVector(4, numElems, 4)
+def GetTableStatsResponseStartVipsVector(builder, numElems):
+    """This method is deprecated. Please switch to Start."""
+    return StartVipsVector(builder, numElems)
 def End(builder): return builder.EndObject()
 def GetTableStatsResponseEnd(builder):
     """This method is deprecated. Please switch to End."""

vastdb-0.1.0/vast_flatbuf/tabular/VipRange.py

@@ -0,0 +1,56 @@
+# automatically generated by the FlatBuffers compiler, do not modify
+
+# namespace: tabular
+
+import flatbuffers
+from flatbuffers.compat import import_numpy
+np = import_numpy()
+
+class VipRange(object):
+    __slots__ = ['_tab']
+
+    @classmethod
+    def GetRootAs(cls, buf, offset=0):
+        n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset)
+        x = VipRange()
+        x.Init(buf, n + offset)
+        return x
+
+    @classmethod
+    def GetRootAsVipRange(cls, buf, offset=0):
+        """This method is deprecated. Please switch to GetRootAs."""
+        return cls.GetRootAs(buf, offset)
+    # VipRange
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # VipRange
+    def StartAddress(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.String(o + self._tab.Pos)
+        return None
+
+    # VipRange
+    def AddressCount(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint16Flags, o + self._tab.Pos)
+        return 0
+
+def Start(builder): builder.StartObject(2)
+def VipRangeStart(builder):
+    """This method is deprecated. Please switch to Start."""
+    return Start(builder)
+def AddStartAddress(builder, startAddress): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(startAddress), 0)
+def VipRangeAddStartAddress(builder, startAddress):
+    """This method is deprecated. Please switch to AddStartAddress."""
+    return AddStartAddress(builder, startAddress)
+def AddAddressCount(builder, addressCount): builder.PrependUint16Slot(1, addressCount, 0)
+def VipRangeAddAddressCount(builder, addressCount):
+    """This method is deprecated. Please switch to AddAddressCount."""
+    return AddAddressCount(builder, addressCount)
+def End(builder): return builder.EndObject()
+def VipRangeEnd(builder):
+    """This method is deprecated. Please switch to End."""
+    return End(builder)

vastdb-0.1.0/vastdb/__init__.py

@@ -0,0 +1,7 @@
+"""VAST Database Python SDK."""
+
+from . import session
+
+# A helper function, useful as a short-hand for Session c-tor: `session = vastdb.connect(...)`
+connect = session.Session
+connect.__name__ = 'connect'

vastdb-0.1.0/vastdb/bucket.py

@@ -0,0 +1,77 @@
+"""VAST Database bucket.
+
+VAST S3 buckets can be used to create Database schemas and tables.
+It is possible to list and access VAST snapshots generated over a bucket.
+"""
+
+from . import errors, schema, transaction
+
+from dataclasses import dataclass
+import logging
+
+log = logging.getLogger(__name__)
+
+
+@dataclass
+class Snapshot:
+    """VAST bucket-level snapshot."""
+
+    name: str
+    bucket: "Bucket"
+
+
+@dataclass
+class Bucket:
+    """VAST bucket."""
+
+    name: str
+    tx: "transaction.Transaction"
+
+    def create_schema(self, path: str) -> "schema.Schema":
+        """Create a new schema (a container of tables) under this bucket."""
+        self.tx._rpc.api.create_schema(self.name, path, txid=self.tx.txid)
+        log.info("Created schema: %s", path)
+        return self.schema(path)
+
+    def schema(self, path: str) -> "schema.Schema":
+        """Get a specific schema (a container of tables) under this bucket."""
+        s = self.schemas(path)
+        log.debug("schema: %s", s)
+        if not s:
+            raise errors.MissingSchema(self.name, path)
+        assert len(s) == 1, f"Expected to receive only a single schema, but got: {len(s)}. ({s})"
+        log.debug("Found schema: %s", s[0].name)
+        return s[0]
+
+    def schemas(self, name: str = None) -> ["schema.Schema"]:
+        """List bucket's schemas."""
+        schemas = []
+        next_key = 0
+        exact_match = bool(name)
+        log.debug("list schemas param: schema=%s, exact_match=%s", name, exact_match)
+        while True:
+            bucket_name, curr_schemas, next_key, is_truncated, _ = \
+                self.tx._rpc.api.list_schemas(bucket=self.name, next_key=next_key, txid=self.tx.txid,
+                                               name_prefix=name, exact_match=exact_match)
+            if not curr_schemas:
+                break
+            schemas.extend(curr_schemas)
+            if not is_truncated:
+                break
+
+        return [schema.Schema(name=name, bucket=self) for name, *_ in schemas]
+
+    def snapshots(self) -> [Snapshot]:
+        """List bucket's snapshots."""
+        snapshots = []
+        next_key = 0
+        while True:
+            curr_snapshots, is_truncated, next_key = \
+                self.tx._rpc.api.list_snapshots(bucket=self.name, next_token=next_key)
+            if not curr_snapshots:
+                break
+            snapshots.extend(curr_snapshots)
+            if not is_truncated:
+                break
+
+        return [Snapshot(name=snapshot, bucket=self) for snapshot in snapshots]

vastdb-0.1.0/vastdb/errors.py

@@ -0,0 +1,158 @@
+import logging
+import requests
+import xml.etree.ElementTree
+
+from enum import Enum
+from dataclasses import dataclass
+
+
+class HttpStatus(Enum):
+    SUCCESS = 200
+    BAD_REQUEST = 400
+    FOBIDDEN = 403
+    NOT_FOUND = 404
+    METHOD_NOT_ALLOWED = 405
+    REQUEST_TIMEOUT = 408
+    CONFLICT = 409
+    INTERNAL_SERVER_ERROR = 500
+    NOT_IMPLEMENTED = 501
+    SERVICE_UNAVAILABLE = 503
+
+
+log = logging.getLogger(__name__)
+
+
+@dataclass
+class HttpError(Exception):
+    code: str
+    message: str
+    url: str
+    status: int  # HTTP status
+    headers: requests.structures.CaseInsensitiveDict  # HTTP response headers
+
+    def __post_init__(self):
+        self.args = [vars(self)]
+
+
+class NotFound(HttpError):
+    pass
+
+
+class Forbidden(HttpError):
+    pass
+
+
+class BadRequest(HttpError):
+    pass
+
+
+class MethodNotAllowed(HttpError):
+    pass
+
+
+class RequestTimeout(HttpError):
+    pass
+
+
+class Conflict(HttpError):
+    pass
+
+
+class InternalServerError(HttpError):
+    pass
+
+
+class NotImplemented(HttpError):
+    pass
+
+
+class ServiceUnavailable(HttpError):
+    pass
+
+
+class UnexpectedError(HttpError):
+    pass
+
+
+@dataclass
+class ImportFilesError(Exception):
+    message: str
+    error_dict: dict
+
+
+class InvalidArgument(Exception):
+    pass
+
+
+class Missing(Exception):
+    pass
+
+
+@dataclass
+class MissingBucket(Missing):
+    bucket: str
+
+
+@dataclass
+class MissingSchema(Missing):
+    bucket: str
+    schema: str
+
+
+@dataclass
+class MissingTable(Missing):
+    bucket: str
+    schema: str
+    table: str
+
+
+@dataclass
+class MissingProjection(Missing):
+    bucket: str
+    schema: str
+    table: str
+    projection: str
+
+
+ERROR_TYPES_MAP = {
+    HttpStatus.BAD_REQUEST: BadRequest,
+    HttpStatus.FOBIDDEN: Forbidden,
+    HttpStatus.NOT_FOUND: NotFound,
+    HttpStatus.METHOD_NOT_ALLOWED: MethodNotAllowed,
+    HttpStatus.REQUEST_TIMEOUT: RequestTimeout,
+    HttpStatus.CONFLICT: Conflict,
+    HttpStatus.INTERNAL_SERVER_ERROR: InternalServerError,
+    HttpStatus.NOT_IMPLEMENTED: NotImplemented,
+    HttpStatus.SERVICE_UNAVAILABLE: ServiceUnavailable,
+}
+
+
+def from_response(res: requests.Response):
+    if res.status_code == HttpStatus.SUCCESS.value:
+        return None
+
+    log.debug("response: url='%s', code=%s, headers=%s, body='%s'", res.request.url, res.status_code, res.headers, res.text)
+    # try to parse S3 XML response for the error details:
+    code = None
+    message = None
+    if res.text:
+        try:
+            root = xml.etree.ElementTree.fromstring(res.text)
+            code = root.find('Code')
+            code = code.text if code is not None else None
+            message = root.find('Message')
+            message = message.text if message is not None else None
+        except xml.etree.ElementTree.ParseError:
+            log.debug("invalid XML: %r", res.text)
+
+    kwargs = dict(
+        code=code,
+        message=message,
+        url=res.request.url,
+        status=res.status_code,
+        headers=res.headers,
+    )
+    log.warning("RPC failed: %s", kwargs)
+    status = HttpStatus(res.status_code)
+    error_type = ERROR_TYPES_MAP.get(status, UnexpectedError)
+    raise error_type(**kwargs)