data-pebbles 0.0.1__tar.gz

data_pebbles-0.0.1/PKG-INFO

@@ -0,0 +1,143 @@
+Metadata-Version: 2.3
+Name: data-pebbles
+Version: 0.0.1
+Summary: A Python library for interacting with the open source data catalog Data Pebbles.
+Author: Leon David Zipp
+Author-email: Leon David Zipp <leondavidzipp@proton.me>
+Requires-Dist: attrs>=24.2.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: mlflow>=2.0.0
+Requires-Dist: polars>=1.0.0
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+
+# Data Pebbles SDK
+
+Python SDK for the [Data Pebbles](https://github.com/LeonDavidZipp/data-pebbles) platform — manage data across bronze, silver, and gold layers with built-in lineage tracking.
+
+## Installation
+
+```bash
+pip install data-pebbles
+```
+
+Requires Python 3.13+. Dependencies: `polars`, `httpx`, `mlflow`.
+
+## Quick Start
+
+```python
+from data_pebbles import DataPebbles
+
+dp = DataPebbles("http://localhost:8000", token="your-token")
+
+# Bronze: upload raw files (.csv, .parquet, .json, .xlsx)
+dp.bronze.create_source("raw_sales")
+dp.bronze.upload(1, file_path="sales.csv")
+raw = dp.bronze.download(1)
+
+# Silver: cleaned LazyFrames with lineage
+dp.silver.create_source("clean_sales")
+dp.silver.upload(2, lf, from_source_id=1)
+lf = dp.silver.download(2)
+
+# Gold: aggregated LazyFrames with multi-source lineage
+dp.gold.create_source("sales_summary")
+dp.gold.upload(3, lf, from_source_ids=[2])
+```
+
+The client can be used as a context manager:
+
+```python
+with DataPebbles("http://localhost:8000") as dp:
+    sources = dp.bronze.list_sources()
+```
+
+## Layers
+
+### Bronze
+
+Stores raw, unprocessed files. Only `.csv`, `.parquet`, `.json`, and `.xlsx` files are accepted.
+
+| Method | Description |
+| --- | --- |
+| `create_source(name)` | Create a new source |
+| `list_sources()` | List all sources |
+| `get_source(source_id)` | Get source metadata |
+| `update_source(source_id, name)` | Rename a source |
+| `delete_source(source_id)` | Delete a source |
+| `list_versions(source_id)` | List all versions |
+| `upload(source_id, *, file_path=None, data=None, file_name="upload")` | Upload a file by path or raw bytes |
+| `download(source_id, *, version=None)` | Download raw bytes (defaults to latest version) |
+| `activate_version(source_id, version)` | Set a version as active |
+| `delete_version(source_id, version)` | Delete a version |
+
+### Silver
+
+Stores cleaned, structured data as Parquet. Works with Polars DataFrames/LazyFrames and tracks lineage back to bronze.
+
+| Method | Description |
+| --- | --- |
+| `create_source(name)` | Create a new source |
+| `list_sources()` | List all sources |
+| `get_source(source_id)` | Get source metadata |
+| `update_source(source_id, name)` | Rename a source |
+| `delete_source(source_id)` | Delete a source |
+| `list_versions(source_id)` | List all versions with lineage |
+| `upload(source_id, data, *, from_source_id)` | Upload a DataFrame/LazyFrame with bronze lineage |
+| `download(source_id, *, version=None)` | Download as a Polars LazyFrame |
+
+### Gold
+
+Stores aggregated, business-ready data as Parquet. Supports multi-source lineage from silver.
+
+| Method | Description |
+| --- | --- |
+| `create_source(name)` | Create a new source |
+| `list_sources()` | List all sources |
+| `get_source(source_id)` | Get source metadata |
+| `update_source(source_id, name)` | Rename a source |
+| `delete_source(source_id)` | Delete a source |
+| `list_versions(source_id)` | List all versions with lineage |
+| `upload(source_id, data, *, from_source_ids)` | Upload a DataFrame/LazyFrame with silver lineage |
+| `download(source_id, *, version=None)` | Download as a Polars LazyFrame |
+
+## Transform Decorators
+
+Automate downloading, transforming, and uploading data between layers with lineage tracking.
+
+### silver_transform
+
+Transforms bronze → silver. The decorator auto-parses bronze data into a `LazyFrame` based on the original file extension. The decorated function receives a `LazyFrame` and returns a `LazyFrame`.
+
+```python
+@dp.silver_transform(target_id=2, from_bronze_id=1)
+def clean(lf: pl.LazyFrame) -> pl.LazyFrame:
+    return lf.filter(pl.col("amount") > 0)
+
+clean()            # uses latest bronze version
+clean(version=5)   # uses a specific bronze version
+```
+
+For CSV files with a non-standard delimiter, use `csv_separator`:
+
+```python
+@dp.silver_transform(target_id=2, from_bronze_id=1, csv_separator=";")
+def clean_eu(lf: pl.LazyFrame) -> pl.LazyFrame:
+    return lf.filter(pl.col("amount") > 0)
+```
+
+### gold_transform
+
+Transforms silver → gold. The decorated function receives a dict mapping silver source IDs to their LazyFrames and returns a `LazyFrame`.
+
+```python
+@dp.gold_transform(target_id=3, from_silver_ids=[1, 2])
+def aggregate(sources: dict[int, pl.LazyFrame]) -> pl.LazyFrame:
+    return (
+        pl.concat(sources.values())
+        .group_by("category")
+        .agg(pl.sum("amount"))
+    )
+
+aggregate()
+```

data_pebbles-0.0.1/README.md

@@ -0,0 +1,130 @@
+# Data Pebbles SDK
+
+Python SDK for the [Data Pebbles](https://github.com/LeonDavidZipp/data-pebbles) platform — manage data across bronze, silver, and gold layers with built-in lineage tracking.
+
+## Installation
+
+```bash
+pip install data-pebbles
+```
+
+Requires Python 3.13+. Dependencies: `polars`, `httpx`, `mlflow`.
+
+## Quick Start
+
+```python
+from data_pebbles import DataPebbles
+
+dp = DataPebbles("http://localhost:8000", token="your-token")
+
+# Bronze: upload raw files (.csv, .parquet, .json, .xlsx)
+dp.bronze.create_source("raw_sales")
+dp.bronze.upload(1, file_path="sales.csv")
+raw = dp.bronze.download(1)
+
+# Silver: cleaned LazyFrames with lineage
+dp.silver.create_source("clean_sales")
+dp.silver.upload(2, lf, from_source_id=1)
+lf = dp.silver.download(2)
+
+# Gold: aggregated LazyFrames with multi-source lineage
+dp.gold.create_source("sales_summary")
+dp.gold.upload(3, lf, from_source_ids=[2])
+```
+
+The client can be used as a context manager:
+
+```python
+with DataPebbles("http://localhost:8000") as dp:
+    sources = dp.bronze.list_sources()
+```
+
+## Layers
+
+### Bronze
+
+Stores raw, unprocessed files. Only `.csv`, `.parquet`, `.json`, and `.xlsx` files are accepted.
+
+| Method | Description |
+| --- | --- |
+| `create_source(name)` | Create a new source |
+| `list_sources()` | List all sources |
+| `get_source(source_id)` | Get source metadata |
+| `update_source(source_id, name)` | Rename a source |
+| `delete_source(source_id)` | Delete a source |
+| `list_versions(source_id)` | List all versions |
+| `upload(source_id, *, file_path=None, data=None, file_name="upload")` | Upload a file by path or raw bytes |
+| `download(source_id, *, version=None)` | Download raw bytes (defaults to latest version) |
+| `activate_version(source_id, version)` | Set a version as active |
+| `delete_version(source_id, version)` | Delete a version |
+
+### Silver
+
+Stores cleaned, structured data as Parquet. Works with Polars DataFrames/LazyFrames and tracks lineage back to bronze.
+
+| Method | Description |
+| --- | --- |
+| `create_source(name)` | Create a new source |
+| `list_sources()` | List all sources |
+| `get_source(source_id)` | Get source metadata |
+| `update_source(source_id, name)` | Rename a source |
+| `delete_source(source_id)` | Delete a source |
+| `list_versions(source_id)` | List all versions with lineage |
+| `upload(source_id, data, *, from_source_id)` | Upload a DataFrame/LazyFrame with bronze lineage |
+| `download(source_id, *, version=None)` | Download as a Polars LazyFrame |
+
+### Gold
+
+Stores aggregated, business-ready data as Parquet. Supports multi-source lineage from silver.
+
+| Method | Description |
+| --- | --- |
+| `create_source(name)` | Create a new source |
+| `list_sources()` | List all sources |
+| `get_source(source_id)` | Get source metadata |
+| `update_source(source_id, name)` | Rename a source |
+| `delete_source(source_id)` | Delete a source |
+| `list_versions(source_id)` | List all versions with lineage |
+| `upload(source_id, data, *, from_source_ids)` | Upload a DataFrame/LazyFrame with silver lineage |
+| `download(source_id, *, version=None)` | Download as a Polars LazyFrame |
+
+## Transform Decorators
+
+Automate downloading, transforming, and uploading data between layers with lineage tracking.
+
+### silver_transform
+
+Transforms bronze → silver. The decorator auto-parses bronze data into a `LazyFrame` based on the original file extension. The decorated function receives a `LazyFrame` and returns a `LazyFrame`.
+
+```python
+@dp.silver_transform(target_id=2, from_bronze_id=1)
+def clean(lf: pl.LazyFrame) -> pl.LazyFrame:
+    return lf.filter(pl.col("amount") > 0)
+
+clean()            # uses latest bronze version
+clean(version=5)   # uses a specific bronze version
+```
+
+For CSV files with a non-standard delimiter, use `csv_separator`:
+
+```python
+@dp.silver_transform(target_id=2, from_bronze_id=1, csv_separator=";")
+def clean_eu(lf: pl.LazyFrame) -> pl.LazyFrame:
+    return lf.filter(pl.col("amount") > 0)
+```
+
+### gold_transform
+
+Transforms silver → gold. The decorated function receives a dict mapping silver source IDs to their LazyFrames and returns a `LazyFrame`.
+
+```python
+@dp.gold_transform(target_id=3, from_silver_ids=[1, 2])
+def aggregate(sources: dict[int, pl.LazyFrame]) -> pl.LazyFrame:
+    return (
+        pl.concat(sources.values())
+        .group_by("category")
+        .agg(pl.sum("amount"))
+    )
+
+aggregate()
+```

data_pebbles-0.0.1/pyproject.toml

@@ -0,0 +1,34 @@
+[project]
+name = "data-pebbles"
+version = "0.0.1"
+description = "A Python library for interacting with the open source data catalog Data Pebbles."
+readme = "README.md"
+authors = [
+    { name = "Leon David Zipp", email = "leondavidzipp@proton.me" }
+]
+requires-python = ">=3.13"
+dependencies = [
+    "attrs>=24.2.0",
+    "httpx>=0.27.0",
+    "mlflow>=2.0.0",
+    "polars>=1.0.0",
+]
+
+[build-system]
+requires = ["uv_build>=0.9.26,<0.10.0"]
+build-backend = "uv_build"
+
+[tool.pyright]
+exclude = ["src/data_pebbles/client", ".venv", "**/__pycache__", "tests"]
+
+[tool.mypy]
+exclude = ["src/data_pebbles/client", ".venv", "**/__pycache__", "tests"]
+
+[tool.ruff]
+exclude = ["src/data_pebbles/client"]
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.2",
+    "ruff>=0.15.0",
+]

data_pebbles-0.0.1/src/data_pebbles/__init__.py

@@ -0,0 +1,10 @@
+import mlflow
+
+from data_pebbles.sdk import (
+	BronzeLayer,
+	DataPebbles,
+	GoldLayer,
+	SilverLayer,
+)
+
+__all__ = ["DataPebbles", "BronzeLayer", "SilverLayer", "GoldLayer", "mlflow"]

data_pebbles-0.0.1/src/data_pebbles/client/__init__.py

@@ -0,0 +1,8 @@
+"""A client library for accessing FastAPI"""
+
+from .client import AuthenticatedClient, Client
+
+__all__ = (
+	"AuthenticatedClient",
+	"Client",
+)

data_pebbles-0.0.1/src/data_pebbles/client/api/__init__.py

@@ -0,0 +1 @@
+"""Contains methods for accessing the API"""

data_pebbles-0.0.1/src/data_pebbles/client/api/api_endpoints_for_interacting_with_the_bronze_layer/__init__.py

@@ -0,0 +1 @@
+"""Contains endpoint functions for accessing the API"""

data_pebbles-0.0.1/src/data_pebbles/client/api/api_endpoints_for_interacting_with_the_bronze_layer/activate_version_bronze_source_id_versions_version_patch.py

@@ -0,0 +1,172 @@
+from http import HTTPStatus
+from typing import Any
+from urllib.parse import quote
+
+import httpx
+
+from ... import errors
+from ...client import AuthenticatedClient, Client
+from ...models.http_validation_error import HTTPValidationError
+from ...types import Response
+
+
+def _get_kwargs(
+	source_id: int,
+	version: int,
+) -> dict[str, Any]:
+	_kwargs: dict[str, Any] = {
+		"method": "patch",
+		"url": "/bronze/{source_id}/versions/{version}".format(
+			source_id=quote(str(source_id), safe=""),
+			version=quote(str(version), safe=""),
+		),
+	}
+
+	return _kwargs
+
+
+def _parse_response(
+	*, client: AuthenticatedClient | Client, response: httpx.Response
+) -> Any | HTTPValidationError | None:
+	if response.status_code == 200:
+		response_200 = response.json()
+		return response_200
+
+	if response.status_code == 422:
+		response_422 = HTTPValidationError.from_dict(response.json())
+
+		return response_422
+
+	if client.raise_on_unexpected_status:
+		raise errors.UnexpectedStatus(response.status_code, response.content)
+	else:
+		return None
+
+
+def _build_response(
+	*, client: AuthenticatedClient | Client, response: httpx.Response
+) -> Response[Any | HTTPValidationError]:
+	return Response(
+		status_code=HTTPStatus(response.status_code),
+		content=response.content,
+		headers=response.headers,
+		parsed=_parse_response(client=client, response=response),
+	)
+
+
+def sync_detailed(
+	source_id: int,
+	version: int,
+	*,
+	client: AuthenticatedClient | Client,
+) -> Response[Any | HTTPValidationError]:
+	"""Activate Version
+
+	Args:
+	    source_id (int):
+	    version (int):
+
+	Raises:
+	    errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
+	    httpx.TimeoutException: If the request takes longer than Client.timeout.
+
+	Returns:
+	    Response[Any | HTTPValidationError]
+	"""
+
+	kwargs = _get_kwargs(
+		source_id=source_id,
+		version=version,
+	)
+
+	response = client.get_httpx_client().request(
+		**kwargs,
+	)
+
+	return _build_response(client=client, response=response)
+
+
+def sync(
+	source_id: int,
+	version: int,
+	*,
+	client: AuthenticatedClient | Client,
+) -> Any | HTTPValidationError | None:
+	"""Activate Version
+
+	Args:
+	    source_id (int):
+	    version (int):
+
+	Raises:
+	    errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
+	    httpx.TimeoutException: If the request takes longer than Client.timeout.
+
+	Returns:
+	    Any | HTTPValidationError
+	"""
+
+	return sync_detailed(
+		source_id=source_id,
+		version=version,
+		client=client,
+	).parsed
+
+
+async def asyncio_detailed(
+	source_id: int,
+	version: int,
+	*,
+	client: AuthenticatedClient | Client,
+) -> Response[Any | HTTPValidationError]:
+	"""Activate Version
+
+	Args:
+	    source_id (int):
+	    version (int):
+
+	Raises:
+	    errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
+	    httpx.TimeoutException: If the request takes longer than Client.timeout.
+
+	Returns:
+	    Response[Any | HTTPValidationError]
+	"""
+
+	kwargs = _get_kwargs(
+		source_id=source_id,
+		version=version,
+	)
+
+	response = await client.get_async_httpx_client().request(**kwargs)
+
+	return _build_response(client=client, response=response)
+
+
+async def asyncio(
+	source_id: int,
+	version: int,
+	*,
+	client: AuthenticatedClient | Client,
+) -> Any | HTTPValidationError | None:
+	"""Activate Version
+
+	Args:
+	    source_id (int):
+	    version (int):
+
+	Raises:
+	    errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
+	    httpx.TimeoutException: If the request takes longer than Client.timeout.
+
+	Returns:
+	    Any | HTTPValidationError
+	"""
+
+	return (
+		await asyncio_detailed(
+			source_id=source_id,
+			version=version,
+			client=client,
+		)
+	).parsed

data_pebbles-0.0.1/src/data_pebbles/client/api/api_endpoints_for_interacting_with_the_bronze_layer/create_source_bronze_post.py

@@ -0,0 +1,164 @@
+from http import HTTPStatus
+from typing import Any
+
+import httpx
+
+from ... import errors
+from ...client import AuthenticatedClient, Client
+from ...models.create_source_request import CreateSourceRequest
+from ...models.http_validation_error import HTTPValidationError
+from ...types import Response
+
+
+def _get_kwargs(
+	*,
+	body: CreateSourceRequest,
+) -> dict[str, Any]:
+	headers: dict[str, Any] = {}
+
+	_kwargs: dict[str, Any] = {
+		"method": "post",
+		"url": "/bronze/",
+	}
+
+	_kwargs["json"] = body.to_dict()
+
+	headers["Content-Type"] = "application/json"
+
+	_kwargs["headers"] = headers
+	return _kwargs
+
+
+def _parse_response(
+	*, client: AuthenticatedClient | Client, response: httpx.Response
+) -> Any | HTTPValidationError | None:
+	if response.status_code == 200:
+		response_200 = response.json()
+		return response_200
+
+	if response.status_code == 422:
+		response_422 = HTTPValidationError.from_dict(response.json())
+
+		return response_422
+
+	if client.raise_on_unexpected_status:
+		raise errors.UnexpectedStatus(response.status_code, response.content)
+	else:
+		return None
+
+
+def _build_response(
+	*, client: AuthenticatedClient | Client, response: httpx.Response
+) -> Response[Any | HTTPValidationError]:
+	return Response(
+		status_code=HTTPStatus(response.status_code),
+		content=response.content,
+		headers=response.headers,
+		parsed=_parse_response(client=client, response=response),
+	)
+
+
+def sync_detailed(
+	*,
+	client: AuthenticatedClient | Client,
+	body: CreateSourceRequest,
+) -> Response[Any | HTTPValidationError]:
+	"""Create Source
+
+	Args:
+	    body (CreateSourceRequest):
+
+	Raises:
+	    errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
+	    httpx.TimeoutException: If the request takes longer than Client.timeout.
+
+	Returns:
+	    Response[Any | HTTPValidationError]
+	"""
+
+	kwargs = _get_kwargs(
+		body=body,
+	)
+
+	response = client.get_httpx_client().request(
+		**kwargs,
+	)
+
+	return _build_response(client=client, response=response)
+
+
+def sync(
+	*,
+	client: AuthenticatedClient | Client,
+	body: CreateSourceRequest,
+) -> Any | HTTPValidationError | None:
+	"""Create Source
+
+	Args:
+	    body (CreateSourceRequest):
+
+	Raises:
+	    errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
+	    httpx.TimeoutException: If the request takes longer than Client.timeout.
+
+	Returns:
+	    Any | HTTPValidationError
+	"""
+
+	return sync_detailed(
+		client=client,
+		body=body,
+	).parsed
+
+
+async def asyncio_detailed(
+	*,
+	client: AuthenticatedClient | Client,
+	body: CreateSourceRequest,
+) -> Response[Any | HTTPValidationError]:
+	"""Create Source
+
+	Args:
+	    body (CreateSourceRequest):
+
+	Raises:
+	    errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
+	    httpx.TimeoutException: If the request takes longer than Client.timeout.
+
+	Returns:
+	    Response[Any | HTTPValidationError]
+	"""
+
+	kwargs = _get_kwargs(
+		body=body,
+	)
+
+	response = await client.get_async_httpx_client().request(**kwargs)
+
+	return _build_response(client=client, response=response)
+
+
+async def asyncio(
+	*,
+	client: AuthenticatedClient | Client,
+	body: CreateSourceRequest,
+) -> Any | HTTPValidationError | None:
+	"""Create Source
+
+	Args:
+	    body (CreateSourceRequest):
+
+	Raises:
+	    errors.UnexpectedStatus: If the server returns an undocumented status code and Client.raise_on_unexpected_status is True.
+	    httpx.TimeoutException: If the request takes longer than Client.timeout.
+
+	Returns:
+	    Any | HTTPValidationError
+	"""
+
+	return (
+		await asyncio_detailed(
+			client=client,
+			body=body,
+		)
+	).parsed