pyarrow-bigquery 0.1.0__tar.gz

pyarrow_bigquery-0.1.0/PKG-INFO

@@ -0,0 +1,262 @@
+Metadata-Version: 2.1
+Name: pyarrow-bigquery
+Version: 0.1.0
+Summary: A simple library to **write to** and **download from** BigQuery tables as PyArrow tables.
+Author-email: Sebastian Pawluś <sebastian.pawlus@gmail.com>
+License: MIT
+Keywords: pyarrow,bigquery
+Description-Content-Type: text/markdown
+Requires-Dist: google-cloud-bigquery<5,>=3
+Requires-Dist: google-cloud-bigquery-storage<3,>=2
+Requires-Dist: pyarrow<17,>=16
+Requires-Dist: tenacity
+
+# pyarrow-bigquery
+
+A simple library to **write to** and **download from** BigQuery tables as PyArrow tables.
+
+## Installation 
+
+```bash 
+pip install pyarrow-bigquery
+```
+
+## Quick Start
+
+This guide will help you quickly get started with `pyarrow-bigquery`, a library that allows you to **read** from and **write** to Google BigQuery using PyArrow.
+
+### Reading from BigQuery
+
+`pyarrow-bigquery` exposes two methods to read BigQuery tables as PyArrow tables. Depending on your use case or the size of the table, you might want to use one method over the other.
+
+#### Read the Whole Table
+
+When the table is small enough to fit in memory, you can read it directly using `bq.read_table`.
+
+```python
+import pyarrow.bigquery as bq
+
+table = bq.read_table("gcp_project.dataset.small_table")
+
+print(table.num_rows)
+```
+
+#### Read with Batches
+
+If the target table is larger than memory or you have other reasons not to fetch the whole table at once, you can use the `bq.reader` iterator method along with the `batch_size` parameter to limit how much data is fetched per iteration.
+
+```python
+import pyarrow.bigquery as bq
+
+for table in bq.reader("gcp_project.dataset.big_table", batch_size=100):
+    print(table.num_rows)
+```
+
+### Writing to BigQuery
+
+Similarly, the package exposes two methods to write to BigQuery. Depending on your use case or the size of the table, you might want to use one method over the other.
+
+#### Write the Whole Table
+
+When you want to write a complete table at once, you can use the `bq.write_table` method.
+
+```python
+import pyarrow as pa
+import pyarrow.bigquery as bq
+
+table = pa.Table.from_arrays([[1, 2, 3, 4]], names=['integers'])
+
+bq.write_table(table, 'gcp_project.dataset.table')
+```
+
+#### Write in Batches (Smaller Chunks)
+
+If you need to write data in smaller chunks, you can use the `bq.writer` method with the `schema` parameter to define the table structure.
+
+```python
+import pyarrow as pa
+import pyarrow.bigquery as bq
+
+schema = pa.schema([
+    ("integers", pa.int64())
+])
+
+with bq.writer("gcp_project.dataset.table", schema=schema) as w:
+    w.write_batch(record_batch)
+    w.write_table(table)
+```
+
+## API Reference
+
+### `pyarrow.bigquery.write_table`
+
+Write a PyArrow Table to a BigQuery Table. No return value.
+
+**Parameters:**
+
+- `table`: `pa.Table`  
+  PyArrow table.
+
+- `where`: `str`  
+  Destination location in BigQuery catalog.
+
+- `project`: `str`, *default* `None`  
+  BigQuery execution project, also the billing project. If not provided, it will be extracted from `where`.
+
+- `table_create`: `bool`, *default* `True`  
+  Specifies if the BigQuery table should be created.
+
+- `table_expire`: `None | int`, *default* `None`  
+  Amount of seconds after which the created table will expire. Used only if `table_create` is `True`. Set to `None` to disable expiration.
+
+- `table_overwrite`: `bool`, *default* `False`  
+  If the table already exists, destroy it and create a new one.
+
+- `worker_type`: `threading.Thread | multiprocessing.Process`, *default* `threading.Thread`  
+  Worker backend for fetching data.
+
+- `worker_count`: `int`, *default* `os.cpu_count()`  
+  Number of threads or processes to use for fetching data from BigQuery.
+
+- `batch_size`: `int`, *default* `100`  
+  Batch size for fetched rows.
+
+```python
+bq.write_table(table, 'gcp_project.dataset.table')
+```
+
+### `pyarrow.bigquery.writer`
+
+Context manager version of the write method. Useful when the PyArrow table is larger than memory size or the table is available in chunks.
+
+**Parameters:**
+
+- `schema`: `pa.Schema`  
+  PyArrow schema.
+
+- `where`: `str`  
+  Destination location in BigQuery catalog.
+
+- `project`: `str`, *default* `None`  
+  BigQuery execution project, also the billing project. If not provided, it will be extracted from `where`.
+
+- `table_create`: `bool`, *default* `True`  
+  Specifies if the BigQuery table should be created.
+
+- `table_expire`: `None | int`, *default* `None`  
+  Amount of seconds after which the created table will expire. Used only if `table_create` is `True`. Set to `None` to disable expiration.
+
+- `table_overwrite`: `bool`, *default* `False`  
+  If the table already exists, destroy it and create a new one.
+
+- `worker_type`: `threading.Thread | multiprocessing.Process`, *default* `threading.Thread`  
+  Worker backend for writing data.
+
+- `worker_count`: `int`, *default* `os.cpu_count()`  
+  Number of threads or processes to use for writing data to BigQuery.
+
+- `batch_size`: `int`, *default* `100`  
+  Batch size used for writes. Table will be automatically split to this value.
+
+Depending on the use case, you might want to use one of the methods below to write your data to a BigQuery table, using either `pa.Table` or `pa.RecordBatch`.
+
+#### `pyarrow.bigquery.writer.write_table`
+
+Context manager method to write a table.
+
+**Parameters:**
+
+- `table`: `pa.Table`  
+  PyArrow table.
+
+```python
+import pyarrow as pa
+import pyarrow.bigquery as bq
+
+schema = pa.schema([("value", pa.list_(pa.int64()))])
+
+with bq.writer("gcp_project.dataset.table", schema=schema) as w:
+    for a in range(1000):
+        w.write_table(pa.Table.from_pylist([{'value': [a] * 10}]))
+```
+
+#### `pyarrow.bigquery.writer.write_batch`
+
+Context manager method to write a record batch.
+
+**Parameters:**
+
+- `batch`: `pa.RecordBatch`  
+  PyArrow record batch.
+
+```python
+import pyarrow as pa
+import pyarrow.bigquery as bq
+
+schema = pa.schema([("value", pa.list_(pa.int64()))])
+
+with bq.writer("gcp_project.dataset.table", schema=schema) as w:
+    for a in range(1000):
+        w.write_batch(pa.RecordBatch.from_pylist([{'value': [1] * 10}]))
+```
+
+### `pyarrow.bigquery.read_table`
+
+**Parameters:**
+
+- `source`: `str`  
+  BigQuery table location.
+
+- `project`: `str`, *default* `None`  
+  BigQuery execution project, also the billing project. If not provided, it will be extracted from `source`.
+
+- `columns`: `str`, *default* `None`  
+  Columns to download. When not provided, all available columns will be downloaded.
+
+- `row_restrictions`: `str`, *default* `None`  
+  Row level filtering executed on the BigQuery side. More in [BigQuery documentation](https://cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1beta1).
+
+- `worker_type`: `threading.Thread | multiprocessing.Process`, *default* `threading.Thread`  
+  Worker backend for fetching data.
+
+- `worker_count`: `int`, *default* `os.cpu_count()`  
+  Number of threads or processes to use for fetching data from BigQuery.
+
+- `batch_size`: `int`, *default* `100`  
+  Batch size used for fetching. Table will be automatically split to this value.
+
+### `pyarrow.bigquery.reader`
+
+**Parameters:**
+
+- `source`: `str`  
+  BigQuery table location.
+
+- `project`: `str`, *default* `None`  
+  BigQuery execution project, also the billing project. If not provided, it will be extracted from `source`.
+
+- `columns`: `str`, *default* `None`  
+  Columns to download. When not provided, all available columns will be downloaded.
+
+- `row_restrictions`: `str`, *default* `None`  
+  Row level filtering executed on the BigQuery side. More in [BigQuery documentation](https://cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1beta1).
+
+- `worker_type`: `threading.Thread | multiprocessing.Process`, *default* `threading.Thread`  
+  Worker backend for fetching data.
+
+- `worker_count`: `int`, *default* `os.cpu_count()`  
+  Number of threads or processes to use for fetching data from BigQuery.
+
+- `batch_size`: `int`, *default* `100`  
+  Batch size used for fetching. Table will be automatically split to this value.
+
+```python
+import pyarrow as pa
+import pyarrow.bigquery as bq
+
+parts = []
+for part in bq.reader("gcp_project.dataset.table"):
+    parts.append(part)
+
+table = pa.concat_tables(parts)
+```

pyarrow_bigquery-0.1.0/README.md

@@ -0,0 +1,249 @@
+# pyarrow-bigquery
+
+A simple library to **write to** and **download from** BigQuery tables as PyArrow tables.
+
+## Installation 
+
+```bash 
+pip install pyarrow-bigquery
+```
+
+## Quick Start
+
+This guide will help you quickly get started with `pyarrow-bigquery`, a library that allows you to **read** from and **write** to Google BigQuery using PyArrow.
+
+### Reading from BigQuery
+
+`pyarrow-bigquery` exposes two methods to read BigQuery tables as PyArrow tables. Depending on your use case or the size of the table, you might want to use one method over the other.
+
+#### Read the Whole Table
+
+When the table is small enough to fit in memory, you can read it directly using `bq.read_table`.
+
+```python
+import pyarrow.bigquery as bq
+
+table = bq.read_table("gcp_project.dataset.small_table")
+
+print(table.num_rows)
+```
+
+#### Read with Batches
+
+If the target table is larger than memory or you have other reasons not to fetch the whole table at once, you can use the `bq.reader` iterator method along with the `batch_size` parameter to limit how much data is fetched per iteration.
+
+```python
+import pyarrow.bigquery as bq
+
+for table in bq.reader("gcp_project.dataset.big_table", batch_size=100):
+    print(table.num_rows)
+```
+
+### Writing to BigQuery
+
+Similarly, the package exposes two methods to write to BigQuery. Depending on your use case or the size of the table, you might want to use one method over the other.
+
+#### Write the Whole Table
+
+When you want to write a complete table at once, you can use the `bq.write_table` method.
+
+```python
+import pyarrow as pa
+import pyarrow.bigquery as bq
+
+table = pa.Table.from_arrays([[1, 2, 3, 4]], names=['integers'])
+
+bq.write_table(table, 'gcp_project.dataset.table')
+```
+
+#### Write in Batches (Smaller Chunks)
+
+If you need to write data in smaller chunks, you can use the `bq.writer` method with the `schema` parameter to define the table structure.
+
+```python
+import pyarrow as pa
+import pyarrow.bigquery as bq
+
+schema = pa.schema([
+    ("integers", pa.int64())
+])
+
+with bq.writer("gcp_project.dataset.table", schema=schema) as w:
+    w.write_batch(record_batch)
+    w.write_table(table)
+```
+
+## API Reference
+
+### `pyarrow.bigquery.write_table`
+
+Write a PyArrow Table to a BigQuery Table. No return value.
+
+**Parameters:**
+
+- `table`: `pa.Table`  
+  PyArrow table.
+
+- `where`: `str`  
+  Destination location in BigQuery catalog.
+
+- `project`: `str`, *default* `None`  
+  BigQuery execution project, also the billing project. If not provided, it will be extracted from `where`.
+
+- `table_create`: `bool`, *default* `True`  
+  Specifies if the BigQuery table should be created.
+
+- `table_expire`: `None | int`, *default* `None`  
+  Amount of seconds after which the created table will expire. Used only if `table_create` is `True`. Set to `None` to disable expiration.
+
+- `table_overwrite`: `bool`, *default* `False`  
+  If the table already exists, destroy it and create a new one.
+
+- `worker_type`: `threading.Thread | multiprocessing.Process`, *default* `threading.Thread`  
+  Worker backend for fetching data.
+
+- `worker_count`: `int`, *default* `os.cpu_count()`  
+  Number of threads or processes to use for fetching data from BigQuery.
+
+- `batch_size`: `int`, *default* `100`  
+  Batch size for fetched rows.
+
+```python
+bq.write_table(table, 'gcp_project.dataset.table')
+```
+
+### `pyarrow.bigquery.writer`
+
+Context manager version of the write method. Useful when the PyArrow table is larger than memory size or the table is available in chunks.
+
+**Parameters:**
+
+- `schema`: `pa.Schema`  
+  PyArrow schema.
+
+- `where`: `str`  
+  Destination location in BigQuery catalog.
+
+- `project`: `str`, *default* `None`  
+  BigQuery execution project, also the billing project. If not provided, it will be extracted from `where`.
+
+- `table_create`: `bool`, *default* `True`  
+  Specifies if the BigQuery table should be created.
+
+- `table_expire`: `None | int`, *default* `None`  
+  Amount of seconds after which the created table will expire. Used only if `table_create` is `True`. Set to `None` to disable expiration.
+
+- `table_overwrite`: `bool`, *default* `False`  
+  If the table already exists, destroy it and create a new one.
+
+- `worker_type`: `threading.Thread | multiprocessing.Process`, *default* `threading.Thread`  
+  Worker backend for writing data.
+
+- `worker_count`: `int`, *default* `os.cpu_count()`  
+  Number of threads or processes to use for writing data to BigQuery.
+
+- `batch_size`: `int`, *default* `100`  
+  Batch size used for writes. Table will be automatically split to this value.
+
+Depending on the use case, you might want to use one of the methods below to write your data to a BigQuery table, using either `pa.Table` or `pa.RecordBatch`.
+
+#### `pyarrow.bigquery.writer.write_table`
+
+Context manager method to write a table.
+
+**Parameters:**
+
+- `table`: `pa.Table`  
+  PyArrow table.
+
+```python
+import pyarrow as pa
+import pyarrow.bigquery as bq
+
+schema = pa.schema([("value", pa.list_(pa.int64()))])
+
+with bq.writer("gcp_project.dataset.table", schema=schema) as w:
+    for a in range(1000):
+        w.write_table(pa.Table.from_pylist([{'value': [a] * 10}]))
+```
+
+#### `pyarrow.bigquery.writer.write_batch`
+
+Context manager method to write a record batch.
+
+**Parameters:**
+
+- `batch`: `pa.RecordBatch`  
+  PyArrow record batch.
+
+```python
+import pyarrow as pa
+import pyarrow.bigquery as bq
+
+schema = pa.schema([("value", pa.list_(pa.int64()))])
+
+with bq.writer("gcp_project.dataset.table", schema=schema) as w:
+    for a in range(1000):
+        w.write_batch(pa.RecordBatch.from_pylist([{'value': [1] * 10}]))
+```
+
+### `pyarrow.bigquery.read_table`
+
+**Parameters:**
+
+- `source`: `str`  
+  BigQuery table location.
+
+- `project`: `str`, *default* `None`  
+  BigQuery execution project, also the billing project. If not provided, it will be extracted from `source`.
+
+- `columns`: `str`, *default* `None`  
+  Columns to download. When not provided, all available columns will be downloaded.
+
+- `row_restrictions`: `str`, *default* `None`  
+  Row level filtering executed on the BigQuery side. More in [BigQuery documentation](https://cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1beta1).
+
+- `worker_type`: `threading.Thread | multiprocessing.Process`, *default* `threading.Thread`  
+  Worker backend for fetching data.
+
+- `worker_count`: `int`, *default* `os.cpu_count()`  
+  Number of threads or processes to use for fetching data from BigQuery.
+
+- `batch_size`: `int`, *default* `100`  
+  Batch size used for fetching. Table will be automatically split to this value.
+
+### `pyarrow.bigquery.reader`
+
+**Parameters:**
+
+- `source`: `str`  
+  BigQuery table location.
+
+- `project`: `str`, *default* `None`  
+  BigQuery execution project, also the billing project. If not provided, it will be extracted from `source`.
+
+- `columns`: `str`, *default* `None`  
+  Columns to download. When not provided, all available columns will be downloaded.
+
+- `row_restrictions`: `str`, *default* `None`  
+  Row level filtering executed on the BigQuery side. More in [BigQuery documentation](https://cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1beta1).
+
+- `worker_type`: `threading.Thread | multiprocessing.Process`, *default* `threading.Thread`  
+  Worker backend for fetching data.
+
+- `worker_count`: `int`, *default* `os.cpu_count()`  
+  Number of threads or processes to use for fetching data from BigQuery.
+
+- `batch_size`: `int`, *default* `100`  
+  Batch size used for fetching. Table will be automatically split to this value.
+
+```python
+import pyarrow as pa
+import pyarrow.bigquery as bq
+
+parts = []
+for part in bq.reader("gcp_project.dataset.table"):
+    parts.append(part)
+
+table = pa.concat_tables(parts)
+```

pyarrow_bigquery-0.1.0/pyproject.toml

@@ -0,0 +1,36 @@
+[project]
+name = "pyarrow-bigquery"
+version = "0.1.0"
+description = "A simple library to **write to** and **download from** BigQuery tables as PyArrow tables."
+authors = [{ name = "Sebastian Pawluś", email = "sebastian.pawlus@gmail.com" }]
+readme = "README.md"
+keywords = ["pyarrow", "bigquery"]
+
+dependencies = [
+    "google-cloud-bigquery>=3,<5",
+    "google-cloud-bigquery-storage>=2,<3",
+    "pyarrow>=16,<17",
+    "tenacity"
+]
+
+[project.license]
+text = "MIT"
+
+[build-system]
+requires = ["setuptools>=40.6.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src/"]
+include = ["pyarrow.bigquery*"]
+
+[[tool.mypy.overrides]]
+module = ["pyarrow.*"]
+ignore_missing_imports = true
+
+
+[tool.ruff]
+exclude = [".git"]
+
+line-length = 120
+indent-width = 4

pyarrow_bigquery-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

pyarrow_bigquery-0.1.0/src/pyarrow/bigquery/__init__.py

@@ -0,0 +1,2 @@
+from .read import reader, read_table, reader_query, read_query  # noqa
+from .write import writer, write_table  # noqa