boti-data 1.0.0__tar.gz → 1.0.2__tar.gz

{boti_data-1.0.0/src/boti_data.egg-info → boti_data-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boti-data
-Version: 1.0.0
+Version: 1.0.2
 Summary: Data infrastructure for the Boti ecosystem
 Author-email: Luis Valverde <lvalverdeb@gmail.com>
 License-Expression: MIT
@@ -11,22 +11,21 @@ Project-URL: Issues, https://github.com/lvalverdeb/boti-data/issues
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Requires-Python: >=3.13
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
-Requires-Dist: asyncmy>=0.2.11
-Requires-Dist: boti<2.0.0,>=0.1.0
-Requires-Dist: dask[dataframe,distributed]>=2026.3.0
-Requires-Dist: fsspec>=2026.3.0
-Requires-Dist: ipykernel>=7.2.0
-Requires-Dist: pandas>=3.0.2
-Requires-Dist: polars>=1.29.0
-Requires-Dist: pyarrow>=23.0.1
-Requires-Dist: pydantic>=2.12.5
-Requires-Dist: pymysql>=1.1.2
-Requires-Dist: s3fs>=2026.3.0
-Requires-Dist: sqlalchemy[asyncio]>=2.0.49
+Requires-Dist: asyncmy<0.3.0,>=0.2.11
+Requires-Dist: boti<2.0.0,>=1.0.2
+Requires-Dist: boti-dask<2.0.0,>=1.0.0
+Requires-Dist: dask[dataframe,distributed]<2027.0.0,>=2026.3.0
+Requires-Dist: pandas<4.0.0,>=3.0.2
+Requires-Dist: polars<2.0.0,>=1.29.0
+Requires-Dist: pymysql<2.0.0,>=1.1.2
+Requires-Dist: sqlalchemy[asyncio]<3.0.0,>=2.0.49
 
 # boti-data
 
@@ -123,31 +122,107 @@ pip install "boti[data]"
 
 ## Imports
 
-`boti-data` uses the top-level Python package `boti_data`:
+`boti-data` uses the top-level Python package `boti_data`. Dask runtime/session/resilience helpers live in `boti_dask` and are not re-exported here.
 
 ```python
 from boti_data import (
+    AsyncFrameEnricher,
+    AttachmentSpec,
+    CsvSink,
+    CsvSinkConfig,
     ConnectionCatalog,
+    DatacubeConfig,
+    DatacubeContract,
     DataGateway,
     DataHelper,
-    FieldMap,
+    FieldMap,    
+    HybridDataset,
+    JsonlSink,
+    JsonlSinkConfig,
     ParquetDataConfig,
     ParquetDataResource,
+    ParquetPipeline,
+    SinkRegistry,
+    SinkPipeline,
+    available_sinks,
+    create_sink,
+    register_sink,
     SqlAlchemyModelBuilder,
     SqlDatabaseConfig,
     SqlDatabaseResource,
 )
 ```
 
+For Dask runtime/session/resilience utilities, import from `boti_dask` directly:
+
+```python
+from boti_dask import (
+    DaskSession,
+    UniqueValuesExtractor,
+    apply_recommended_dask_config,
+    async_safe_compute,
+    async_safe_gather,
+    async_safe_head,
+    async_safe_persist,
+    async_safe_wait,
+    dask_is_empty,
+    dask_is_probably_empty,
+    dask_session,
+    inspect_graph,
+    safe_compute,
+    safe_gather,
+    safe_head,
+    safe_persist,
+    safe_wait,
+)
+```
+
 Lower-level modules are also available:
 
 ```python
 from boti_data.db import SqlDatabaseConfig, SqlDatabaseResource
+from boti_data.datacube import DatacubeConfig, DatacubeContract
 from boti_data.gateway import DataGateway
 from boti_data.parquet import ParquetDataConfig, ParquetDataResource
 from boti_data.schema import validate_schema
 ```
 
+## Datacube backend
+
+`DataGateway` also supports a callable-backed `datacube` backend for in-process cube loaders while keeping the same return-type API (`pandas`/`dask`/`arrow`/`polars`).
+
+```python
+import pandas as pd
+
+from boti_data import DatacubeConfig, DatacubeContract, DataGateway
+
+
+def loader(request):
+    frame = pd.DataFrame({"id": [1, 2], "status": ["active", "inactive"]})
+    if request.filters.get("status__exact") == "active":
+        frame = frame[frame["status"] == "active"]
+    return frame
+
+
+contract = DatacubeContract(
+    request_transformer=lambda request: request.model_copy(update={"cube": request.cube or "orders_v2"}),
+    frame_transformer=lambda frame, request: frame.assign(cube=request.cube),
+)
+
+with DataGateway(
+    DatacubeConfig(loader=loader, default_cube="orders", contract=contract),
+    table="orders",
+    sticky_filters={"status__exact": "active"},
+) as gateway:
+    df = gateway.load(return_type="pandas")
+```
+
+You can also configure this path with `DataGateway.from_config({"backend": "datacube", ...})` or `DataHelper({"backend": "datacube", ...})`.
+
+See `docs/DATACUBE_CONTRACT.md` for hook ordering, loader precedence, and validator rejection guidance.
+See `docs/SECURITY_HARDENING.md` for SQL/raw-sql policy and distributed credential hardening defaults.
+For a runnable rejection flow, see `examples/data_facade_datacube_contract_rejection.py`.
+
 ## DataHelper
 
 `DataHelper` is the primary entry point for most use cases. It is a thin facade over `DataGateway` that provides a clean, consistent interface for loading data whether you are working locally, in a notebook, or inside a distributed Dask pipeline.
@@ -196,6 +271,170 @@ async with DataHelper(config, table="orders") as helper:
     df = await helper.aload(status="confirmed")
 ```
 
+For synchronous scripts that still need to call async load paths, use the explicit sync bridge:
+
+```python
+with DataHelper(config, table="orders") as helper:
+    df = helper.aload_sync(status="confirmed")
+```
+
+If an event loop is already running (for example notebooks or ASGI handlers), use `await helper.aload(...)` directly.
+
+## ParquetPipeline
+
+`ParquetPipeline` is the first orchestration layer built on top of `DataHelper` and `HybridDataset`.
+It materializes a lazy source load into a parquet dataset directory and can optionally reload the
+written dataset through `ParquetReader`.
+
+```python
+from boti_data import DataHelper, ParquetPipeline
+
+helper = DataHelper(
+    backend="sqlalchemy",
+    connection_url=f"sqlite:///{db_path}",
+    poolclass="sqlalchemy.pool.NullPool",
+    query_only=False,
+    table="source_events",
+)
+
+pipeline = ParquetPipeline(
+    helper,
+    {
+        "backend": "parquet",
+        "storage_path": "/tmp/source_events_dataset",
+        "partition_on": ["partition_date"],
+    },
+    date_field="event_date",
+)
+
+result = pipeline.materialize(
+    filters={"status__exact": "active"},
+    reload=True,
+    reload_options={"filters": {"partition_date__exact": "2026-04-17"}},
+)
+
+assert result.reloaded is True
+frame = result.frame
+```
+
+Use `materialize()` / `amaterialize()` for the one-step workflow, or call `to_parquet()` and
+`from_parquet()` separately when you want explicit control over the write and reload phases.
+See `examples/data_parquet_pipeline.py` for a runnable end-to-end example.
+
+## SinkPipeline and Sinks
+
+Phase 3 adds a minimal sink/plugin layer next to `ParquetPipeline`.
+
+- `SinkPipeline` orchestrates `DataHelper` / `HybridDataset` loads into any sink implementing the pipeline sink contract
+- `CsvSink` and `JsonlSink` are write-only dataset sinks (`.csv` / `.jsonl` shards)
+- `ParquetPipeline` now uses the same sink abstraction internally via `ParquetSink`
+
+Use the sink registry for named sink creation:
+
+```python
+from boti_data import SinkPipeline, available_sinks
+
+assert "jsonl" in available_sinks()
+
+pipeline = SinkPipeline(
+    helper,
+    "jsonl",
+    sink_config={
+        "storage_path": "/tmp/source_events_jsonl",
+        "partition_on": ["partition_date"],
+    },
+    date_field="event_date",
+)
+result = pipeline.write(filters={"status__exact": "active"})
+```
+
+```python
+from boti_data import CsvSink, DataHelper, SinkPipeline
+
+helper = DataHelper(
+    backend="sqlalchemy",
+    connection_url=f"sqlite:///{db_path}",
+    poolclass="sqlalchemy.pool.NullPool",
+    query_only=False,
+    table="source_events",
+)
+
+sink = CsvSink(
+    {
+        "storage_path": "/tmp/source_events_csv",
+        "partition_on": ["partition_date"],
+    }
+)
+
+pipeline = SinkPipeline(helper, sink, date_field="event_date")
+result = pipeline.write(filters={"status__exact": "active"})
+
+assert result.files
+```
+
+`CsvSink`/`JsonlSink` are intentionally write-only in this release. Use `ParquetPipeline` when you need
+write + reload orchestration. See `examples/data_csv_sink_pipeline.py` and
+`examples/data_jsonl_sink_pipeline.py` for runnable examples.
+
+## Enrichment v1
+
+`AsyncFrameEnricher` adds declarative attachment-based enrichment before downstream writes.
+Select attachments by `AttachmentSpec.key` via `cols=[...]`, and enforce bounded unique-value
+extraction with `max_unique_values`.
+
+```python
+from boti_data import AsyncFrameEnricher, AttachmentSpec
+
+enricher = AsyncFrameEnricher(
+    [
+        AttachmentSpec(
+            key="customer_segment",
+            required_cols={"customer_id"},
+            attachment_fn=customer_segment_attachment,
+            col_to_kwarg={"customer_id": "ids"},
+            left_on=["customer_id"],
+            right_on=["id"],
+            drop_cols=["id"],
+            max_unique_values=5000,
+        )
+    ]
+)
+enriched = await enricher.aenrich(base_frame, cols=["customer_segment"])
+```
+
+`SinkPipeline` accepts `enricher=...` and optional `enrich_cols=[...]` to apply enrichment right
+before sink writes.
+
+### Raw `sql=` safety policy (eager SQL only)
+
+`DataGateway` supports convenience `sql="..."` reads for eager SQL paths, with explicit safety controls:
+
+- raw SQL is validated as read-only single-statement `SELECT`/`WITH`
+- mutating SQL (`INSERT`, `UPDATE`, `DELETE`, `DROP`, etc.) is rejected
+- multi-statement SQL is rejected
+- lazy SQL still requires `statement` + `model` (raw `sql=` is eager-only)
+
+Use constructor policy `raw_sql_policy` to control availability:
+
+- `raw_sql_policy="readonly_opt_in"` (default): call site must pass `allow_raw_sql=True`
+- `raw_sql_policy="disabled"`: raw `sql=` is blocked even if `allow_raw_sql=True`
+
+```python
+from boti_data import DataGateway, SqlDatabaseConfig
+
+config = SqlDatabaseConfig(connection_url="sqlite:///example.db", query_only=True)
+
+with DataGateway(config, raw_sql_policy="readonly_opt_in") as gateway:
+    df = gateway.load(
+        sql="SELECT id, status FROM users WHERE status = :status",
+        params={"status": "active"},
+        as_pandas=True,
+        allow_raw_sql=True,
+    )
+```
+
+Prefer `statement` + `model` for production query paths whenever possible, especially when you need lazy/partitioned execution.
+
 ---
 
 ## Output engines: pandas, polars, and dask
@@ -233,9 +472,14 @@ df = helper.load(status="confirmed", return_type="arrow")  # pyarrow.Table
 | `polars` | `polars.DataFrame` | CPU-intensive transforms, single-machine performance |
 | `arrow` | `pyarrow.Table` | Zero-copy interchange, serialisation, ML pipelines |
 | `dask` | `dask.dataframe.DataFrame` | Large datasets, distributed clusters, lazy evaluation |
-| `auto` | decided at runtime | Unknown result size; `boti-data` probes row count and chooses |
+| `auto` | decided at runtime | Unknown result size; `boti-data` uses backend-aware size heuristics |
 
-`return_type="auto"` uses pandas when the result is small (≤ 10,000 rows or ≤ 32 MB) and switches to Dask otherwise. Use it when you do not know the result size in advance and want sensible defaults.
+`return_type="auto"` uses pandas for small results and switches to Dask for larger scans, with backend-specific checks:
+
+- SQL: probes up to 10,000 rows (and respects `limit` / explicit `partitioned` controls).
+- Parquet: uses pandas when the selected scan is small enough to estimate eagerly (typically up to 4 files and <= 32 MB total), otherwise Dask.
+
+Use `auto` when result size is uncertain and you want sensible defaults without hardcoding an engine.
 
 ---
 
@@ -270,6 +514,98 @@ async def get_orders(status: str) -> pd.DataFrame:
         return await helper.pandas.aload(status=status)
 ```
 
+---
+
+## Dask resilience helpers
+
+`boti-data` integrates with `boti_dask` for opt-in Dask runtime and resilience workflows.
+
+Available helpers:
+
+- `inspect_graph(...)`
+- `safe_compute(...)`
+- `safe_head(...)`
+- `safe_gather(...)`
+- `safe_persist(...)`
+- `safe_wait(...)`
+- `async_safe_compute(...)`
+- `async_safe_head(...)`
+- `async_safe_gather(...)`
+- `async_safe_persist(...)`
+- `async_safe_wait(...)`
+- `dask_is_probably_empty(...)`
+- `dask_is_empty(...)`
+- `UniqueValuesExtractor`
+- `apply_recommended_dask_config(...)`
+
+### Core rules
+
+- Dask DataFrame-like graphs do **not** silently fall back to local threaded compute.
+- Retry behavior is limited to recoverable communication failures.
+- The helpers are opt-in and designed to complement explicit `DaskSession` ownership.
+
+### Minimal example
+
+```python
+import asyncio
+import dask
+import dask.dataframe as dd
+import pandas as pd
+
+from boti_dask import (
+    apply_recommended_dask_config,
+    async_safe_compute,
+    dask_session,
+    inspect_graph,
+    safe_compute,
+    safe_persist,
+    safe_wait,
+)
+
+frame = dd.from_pandas(pd.DataFrame({"id": [1, 2, 3]}), npartitions=2)
+
+with apply_recommended_dask_config():
+    with dask_session(cluster_kwargs={"n_workers": 1, "threads_per_worker": 1, "processes": False}) as client:
+        metrics = inspect_graph(frame)
+        persisted = safe_persist(frame, dask_client=client)
+        safe_wait(persisted, dask_client=client)
+        total = safe_compute(frame["id"].sum(), dask_client=client)
+        preview = safe_head(frame, n=2, dask_client=client)
+
+        async def run_async() -> int:
+            delayed_value = dask.delayed(lambda: 6 * 7)()
+            return await async_safe_compute(delayed_value, dask_client=client)
+
+        async_total = asyncio.run(run_async())
+```
+
+### Dry-run and preview on gateway loads
+
+Use `dry_run=True` to build and inspect a lazy Dask load graph without materializing it, and `preview(...)` / `apreview(...)` to safely sample rows inside the session:
+
+```python
+with DataHelper.session(scheduler_address="tcp://scheduler:8786", verify_connectivity=True) as client:
+    with DataHelper(config, table="transactions") as helper:
+        ddf = helper.dask.load(year=2024, dry_run=True, diagnostics=True)
+        preview = helper.dask.preview(year=2024, n=5)
+```
+
+### Resilient joins
+
+For large indexed distributed joins, opt into resilient persistence:
+
+```python
+joined = helper.left_join(
+    left,
+    right,
+    join_key="id",
+    join_schema_map={"id": "Int64"},
+    persist=True,
+    resilient=True,
+    diagnostics=True,
+)
+```
+
 ### Parquet sources
 
 ```python
@@ -335,8 +671,57 @@ with DataHelper.session(scheduler_address="tcp://scheduler:8786") as client:
         # Subsequent operations reuse the persisted graph
         monthly = ddf.groupby("month").agg({"amount": "sum"}).compute()
         by_region = ddf.groupby("region").size().compute()
+
+        # Preview persisted data before the managed session closes.
+        preview = helper.preview(year=2024, n=5, persist=True)
+```
+
+### Shared sessions for repeated notebook cells
+
+When repeated cells need to reuse the same scheduler connection, opt into shared-session reuse explicitly:
+
+```python
+with DataHelper.session(
+    scheduler_address="tcp://scheduler:8786",
+    shared=True,
+    shared_key="notebook-main-cluster",
+) as client:
+    with DataHelper(config, table="transactions") as helper:
+        ddf = helper.dask.load(year=2024)
+```
+
+### HybridDataset distributed pattern
+
+Use this pattern when you split reads across historical and live sources but want one
+distributed execution context. `HybridDataset` composes two helpers, and
+`DataHelper.session(...)` keeps scheduler ownership explicit.
+
+```python
+import asyncio
+
+from dask.distributed import LocalCluster
+from boti_data import DataHelper, HybridDataset
+
+historical = DataHelper(backend="sqlalchemy", connection_url="sqlite:///events.db", table="events_hist")
+live = DataHelper(backend="sqlalchemy", connection_url="sqlite:///events.db", table="events_live")
+dataset = HybridDataset(historical, live, date_field="event_date", split_date="2026-04-18")
+
+with LocalCluster(n_workers=1, threads_per_worker=1, processes=False, dashboard_address=":0") as cluster:
+    with DataHelper.session(
+        scheduler_address=cluster.scheduler_address,
+        verify_connectivity=True,
+        shared=True,
+        shared_key="hybrid-dataset-distributed",
+    ):
+        ddf = dataset.dask.load(start="2026-04-14", end="2026-04-19", diagnostics=True)
+        frame = ddf.compute()
+        eager_frame = asyncio.run(
+            dataset.aload(start="2026-04-16", end="2026-04-18", return_type="pandas")
+        )
 ```
 
+For a complete runnable script, see `examples/data_hybrid_dataset_distributed.py`.
+
 ### Semi-join across distributed frames
 
 ```python
@@ -428,7 +813,7 @@ Setting `worker_connection_env_var` prevents both problems and is the recommende
 
 ### Parquet in distributed settings
 
-Parquet resources use `fsspec` for filesystem access. The filesystem object is not pickled directly; instead, `ParquetDataResource` uses a `fs_factory` callable or a `filesystem_profile` name that workers can use to reconstruct the filesystem independently.
+Parquet resources use `fsspec` for filesystem access. The filesystem object is not pickled directly; instead, `ParquetDataResource` uses a `fs_factory` callable or a `filesystem_profile` name that workers can use to reconstruct the filesystem independently. When `filesystem_profile` is used, filesystem creation is routed through `ConnectionCatalog` adapters so retry/cache behavior stays consistent.
 
 ```python
 from boti_data import DataHelper, ParquetDataConfig, ConnectionCatalog
@@ -461,6 +846,7 @@ Use the following as a guide:
 | Data does not fit in one machine's RAM | `helper.dask.load()` + local or remote cluster |
 | Heavy transforms over millions of rows | `helper.dask.load()` + Dask cluster |
 | Async service (FastAPI, ASGI) | `await helper.pandas.aload()` or `await helper.dask.aload()` |
+| Sync script that needs async load path | `helper.aload_sync(...)` |
 | Joining two large tables on a cluster | `helper.semi_join(series, on="key")` |
 | Scheduled overnight batch job | Dask cluster + `persist=True` for multi-pass jobs |
 
@@ -566,6 +952,26 @@ with DataHelper(config, table="transactions") as helper:
     df = helper.pandas.load_period("created_at", "2024-01-01", "2024-06-30")
 ```
 
+### HybridDataset — historical + live composition
+
+```python
+historical = DataHelper(backend="parquet", storage_path="/data/historical", parquet_filename="orders")
+live = DataHelper(backend="sqlalchemy", connection_url="sqlite:///live.db", table="orders_live")
+
+dataset = HybridDataset(
+    historical,
+    live,
+    date_field="order_date",
+    split_date="2026-04-18",
+)
+
+# mixed window (auto -> dask for hybrid composition)
+ddf = dataset.load(start="2026-04-15", end="2026-04-20", return_type="auto")
+
+# explicit source override through engine-bound view
+pdf = dataset.pandas.load(start="2026-04-18", end="2026-04-20", source="live")
+```
+
 ### DataHelper — parquet source
 
 ```python