boti-data 1.3.6__tar.gz → 1.3.7__tar.gz

{boti_data-1.3.6/src/boti_data.egg-info → boti_data-1.3.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boti-data
-Version: 1.3.6
+Version: 1.3.7
 Summary: Data infrastructure for the Boti ecosystem
 Author: Luis Valverde
 Maintainer-email: Luis Valverde <lvalverdeb@gmail.com>
@@ -27,6 +27,10 @@ Requires-Dist: asyncpg<0.32.0,>=0.30.0; extra == "postgres"
 Requires-Dist: psycopg[binary]<4.0.0,>=3.2.0; extra == "postgres"
 Provides-Extra: pgvector
 Requires-Dist: pgvector<0.6.0,>=0.3.0; extra == "pgvector"
+Provides-Extra: clickhouse
+Requires-Dist: clickhouse-connect<2.0.0,>=1.6.0; extra == "clickhouse"
+Provides-Extra: duckdb
+Requires-Dist: duckdb-engine<0.18.0,>=0.17.0; extra == "duckdb"
 Provides-Extra: dataframes
 Requires-Dist: boti-dask<2.0.0,>=1.0.0; extra == "dataframes"
 Requires-Dist: dask[dataframe,distributed]<2027.0.0,>=2026.3.0; extra == "dataframes"
@@ -131,8 +135,25 @@ pip install "boti[data]"
 ```bash
 pip install "boti-data[mysql]"      # asyncmy + pymysql
 pip install "boti-data[postgres]"   # asyncpg + psycopg[binary]
+pip install "boti-data[clickhouse]" # clickhouse-connect (sync only)
+pip install "boti-data[duckdb]"     # duckdb-engine (sync only)
 ```
 
+ClickHouse support is sync-only: there is no maintained async SQLAlchemy driver for ClickHouse, so `AsyncSqlDatabaseResource` rejects a ClickHouse connection URL — use `SqlDatabaseResource` instead. `SqlUnitOfWork`/`AsyncSqlUnitOfWork` also reject ClickHouse outright, since ClickHouse's transaction support is experimental, single-node, and MergeTree-only, with no real cross-table ACID guarantees to back a multi-table unit of work — use `SqlRepository`/`AsyncSqlRepository` directly, one table at a time, instead.
+
+Verified against a real ClickHouse instance:
+
+- `SqlRepository`/`AsyncSqlRepository` are a get/insert/update/delete-**by-primary-key** API, but MergeTree's `ORDER BY` is a sort/index prefix, not a uniqueness constraint — many real ClickHouse tables have several rows per `ORDER BY` key. Because of that, `get()`/`update()`/`delete()` each **refuse to run against a ClickHouse table unless called with `assume_unique_key=True`** (e.g. `repo.get(pk, assume_unique_key=True)`). This flag is a per-call caller assertion, not something boti-data verifies or enforces: it does not check the data, and if the key genuinely isn't unique, `get(key, assume_unique_key=True)` will still return one arbitrary matching row and `delete(key, assume_unique_key=True)`/`update(key, ..., assume_unique_key=True)` will still affect *every* row sharing that key, not just one. Only pass it once you've confirmed the key is actually unique for that table. `insert()` has no such parameter — appending a row is never ambiguous.
+- `.update()` also fails outright on a table without `enable_block_number_column` enabled (`ALTER TABLE ... MODIFY SETTING enable_block_number_column = 1`, off by default) — ClickHouse raises `Lightweight updates are not supported` otherwise.
+- `query_only=True` is enforced server-side (ClickHouse's `readonly` setting, applied via `set_client_setting` on connect), not just by boti-data's own read-only session guard.
+
+DuckDB support is also sync-only (no maintained async driver, same as ClickHouse — `AsyncSqlDatabaseResource` rejects a DuckDB connection URL), but is otherwise a better fit than ClickHouse for `SqlRepository`'s by-primary-key model:
+
+- **`SqlUnitOfWork`/`AsyncSqlUnitOfWork` are fully supported** — DuckDB has real multi-table ACID transactions (unlike ClickHouse), verified with an actual mid-transaction constraint violation correctly rolling back writes across two tables.
+- **No `assume_unique_key`-style flag is needed** — DuckDB genuinely enforces `PRIMARY KEY`/`UNIQUE` constraints at the database level, and boti-data recovers the real key columns from DuckDB's catalog even though `duckdb-engine`'s own reflection can't see them (a known upstream limitation — it warns "doesn't yet support reflection on indices"). Without that recovery step, every DuckDB table would look keyless and `SqlRepository` would silently bind to an arbitrary first column instead of the real key — this was caught and fixed during implementation, not just assumed away.
+- **`query_only=True` is enforced via a connect-time flag**, not a runtime statement: DuckDB's read-only mode (`duckdb.connect(read_only=True)`) can only be set when a connection is opened, so boti-data injects `connect_args={"read_only": True}` itself before the engine is even constructed, rather than configuring it afterward the way Postgres/MySQL/ClickHouse do.
+- **DuckDB allows only one process to hold a database file open for read-write at a time** — opening a file in its default mode claims an exclusive lock on connect, even for a connection that only ever reads. `SqlPartitionedLoader` (which spawns one engine per worker process) requires `query_only=True` for a file-backed DuckDB DSN for exactly this reason — otherwise the first worker to connect would block every other worker, regardless of whether any of them write. Concurrent read-only access from multiple processes works fine. In-memory (`duckdb:///:memory:`) DSNs are rejected for distributed loading the same way SQLite's are, since each worker process would get its own independent, empty database.
+
 If a Postgres table has a [pgvector](https://github.com/pgvector/pgvector) `vector` column, install `boti-data[pgvector]` too — this registers pgvector's `Vector` type with SQLAlchemy's reflection so `SqlAlchemyModelBuilder`/`SqlRepository`/`DataGateway` map that column to a usable `list[float]` (with its dimension preserved) instead of an unreflectable `NullType`:
 
 ```bash

boti_data-1.3.6/PKG-INFO → boti_data-1.3.7/README.md

@@ -1,38 +1,3 @@
-Metadata-Version: 2.4
-Name: boti-data
-Version: 1.3.6
-Summary: Data infrastructure for the Boti ecosystem
-Author: Luis Valverde
-Maintainer-email: Luis Valverde <lvalverdeb@gmail.com>
-License-Expression: MIT
-Project-URL: Homepage, https://github.com/lvalverdeb/boti-data
-Project-URL: Repository, https://github.com/lvalverdeb/boti-data
-Project-URL: Documentation, https://github.com/lvalverdeb/boti-data#readme
-Project-URL: Issues, https://github.com/lvalverdeb/boti-data/issues
-Classifier: Development Status :: 3 - Alpha
-Classifier: Intended Audience :: Developers
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Requires-Python: >=3.13
-Description-Content-Type: text/markdown
-Requires-Dist: boti<2.0.0,>=1.0.2
-Requires-Dist: sqlalchemy[asyncio]<3.0.0,>=2.0.49
-Provides-Extra: mysql
-Requires-Dist: asyncmy<0.3.0,>=0.2.11; extra == "mysql"
-Requires-Dist: pymysql<2.0.0,>=1.1.2; extra == "mysql"
-Provides-Extra: postgres
-Requires-Dist: asyncpg<0.32.0,>=0.30.0; extra == "postgres"
-Requires-Dist: psycopg[binary]<4.0.0,>=3.2.0; extra == "postgres"
-Provides-Extra: pgvector
-Requires-Dist: pgvector<0.6.0,>=0.3.0; extra == "pgvector"
-Provides-Extra: dataframes
-Requires-Dist: boti-dask<2.0.0,>=1.0.0; extra == "dataframes"
-Requires-Dist: dask[dataframe,distributed]<2027.0.0,>=2026.3.0; extra == "dataframes"
-Requires-Dist: pandas<4.0.0,>=3.0.2; extra == "dataframes"
-Requires-Dist: polars<2.0.0,>=1.29.0; extra == "dataframes"
-
 # boti-data
 
 `boti-data` is the **data access and data transformation layer** of the Boti ecosystem.
@@ -131,8 +96,25 @@ pip install "boti[data]"
 ```bash
 pip install "boti-data[mysql]"      # asyncmy + pymysql
 pip install "boti-data[postgres]"   # asyncpg + psycopg[binary]
+pip install "boti-data[clickhouse]" # clickhouse-connect (sync only)
+pip install "boti-data[duckdb]"     # duckdb-engine (sync only)
 ```
 
+ClickHouse support is sync-only: there is no maintained async SQLAlchemy driver for ClickHouse, so `AsyncSqlDatabaseResource` rejects a ClickHouse connection URL — use `SqlDatabaseResource` instead. `SqlUnitOfWork`/`AsyncSqlUnitOfWork` also reject ClickHouse outright, since ClickHouse's transaction support is experimental, single-node, and MergeTree-only, with no real cross-table ACID guarantees to back a multi-table unit of work — use `SqlRepository`/`AsyncSqlRepository` directly, one table at a time, instead.
+
+Verified against a real ClickHouse instance:
+
+- `SqlRepository`/`AsyncSqlRepository` are a get/insert/update/delete-**by-primary-key** API, but MergeTree's `ORDER BY` is a sort/index prefix, not a uniqueness constraint — many real ClickHouse tables have several rows per `ORDER BY` key. Because of that, `get()`/`update()`/`delete()` each **refuse to run against a ClickHouse table unless called with `assume_unique_key=True`** (e.g. `repo.get(pk, assume_unique_key=True)`). This flag is a per-call caller assertion, not something boti-data verifies or enforces: it does not check the data, and if the key genuinely isn't unique, `get(key, assume_unique_key=True)` will still return one arbitrary matching row and `delete(key, assume_unique_key=True)`/`update(key, ..., assume_unique_key=True)` will still affect *every* row sharing that key, not just one. Only pass it once you've confirmed the key is actually unique for that table. `insert()` has no such parameter — appending a row is never ambiguous.
+- `.update()` also fails outright on a table without `enable_block_number_column` enabled (`ALTER TABLE ... MODIFY SETTING enable_block_number_column = 1`, off by default) — ClickHouse raises `Lightweight updates are not supported` otherwise.
+- `query_only=True` is enforced server-side (ClickHouse's `readonly` setting, applied via `set_client_setting` on connect), not just by boti-data's own read-only session guard.
+
+DuckDB support is also sync-only (no maintained async driver, same as ClickHouse — `AsyncSqlDatabaseResource` rejects a DuckDB connection URL), but is otherwise a better fit than ClickHouse for `SqlRepository`'s by-primary-key model:
+
+- **`SqlUnitOfWork`/`AsyncSqlUnitOfWork` are fully supported** — DuckDB has real multi-table ACID transactions (unlike ClickHouse), verified with an actual mid-transaction constraint violation correctly rolling back writes across two tables.
+- **No `assume_unique_key`-style flag is needed** — DuckDB genuinely enforces `PRIMARY KEY`/`UNIQUE` constraints at the database level, and boti-data recovers the real key columns from DuckDB's catalog even though `duckdb-engine`'s own reflection can't see them (a known upstream limitation — it warns "doesn't yet support reflection on indices"). Without that recovery step, every DuckDB table would look keyless and `SqlRepository` would silently bind to an arbitrary first column instead of the real key — this was caught and fixed during implementation, not just assumed away.
+- **`query_only=True` is enforced via a connect-time flag**, not a runtime statement: DuckDB's read-only mode (`duckdb.connect(read_only=True)`) can only be set when a connection is opened, so boti-data injects `connect_args={"read_only": True}` itself before the engine is even constructed, rather than configuring it afterward the way Postgres/MySQL/ClickHouse do.
+- **DuckDB allows only one process to hold a database file open for read-write at a time** — opening a file in its default mode claims an exclusive lock on connect, even for a connection that only ever reads. `SqlPartitionedLoader` (which spawns one engine per worker process) requires `query_only=True` for a file-backed DuckDB DSN for exactly this reason — otherwise the first worker to connect would block every other worker, regardless of whether any of them write. Concurrent read-only access from multiple processes works fine. In-memory (`duckdb:///:memory:`) DSNs are rejected for distributed loading the same way SQLite's are, since each worker process would get its own independent, empty database.
+
 If a Postgres table has a [pgvector](https://github.com/pgvector/pgvector) `vector` column, install `boti-data[pgvector]` too — this registers pgvector's `Vector` type with SQLAlchemy's reflection so `SqlAlchemyModelBuilder`/`SqlRepository`/`DataGateway` map that column to a usable `list[float]` (with its dimension preserved) instead of an unreflectable `NullType`:
 
 ```bash

{boti_data-1.3.6 → boti_data-1.3.7}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "boti-data"
-version = "1.3.6"
+version = "1.3.7"
 description = "Data infrastructure for the Boti ecosystem"
 readme = "README.md"
 requires-python = ">=3.13"
@@ -46,6 +46,25 @@ postgres = [
 pgvector = [
     "pgvector>=0.3.0,<0.6.0",
 ]
+# clickhouse-connect bundles both the DBAPI driver and its own SQLAlchemy
+# dialect ("clickhousedb://") in one package, so unlike mysql/postgres there's
+# no separate async driver to pair it with — ClickHouse support in boti-data is
+# sync-only (SqlDatabaseResource only; AsyncSqlDatabaseResource and
+# SqlUnitOfWork/AsyncSqlUnitOfWork reject a clickhouse backend outright, since
+# ClickHouse has no maintained async SQLAlchemy driver and no real cross-table
+# transactional guarantees to back a unit of work with).
+clickhouse = [
+    "clickhouse-connect>=1.6.0,<2.0.0",
+]
+# duckdb-engine bundles the SQLAlchemy dialect ("duckdb://"); the duckdb driver
+# itself comes along as its transitive dependency. Like clickhouse, there's no
+# separate async driver to pair it with, so DuckDB support in boti-data is
+# sync-only (AsyncSqlDatabaseResource rejects a duckdb backend outright). Unlike
+# ClickHouse, DuckDB has real PRIMARY KEY/UNIQUE enforcement and real multi-table
+# ACID transactions, so SqlUnitOfWork/AsyncSqlUnitOfWork are fully supported here.
+duckdb = [
+    "duckdb-engine>=0.17.0,<0.18.0",
+]
 # The dataframe-shaped, bulk-oriented side of the package (DataGateway/
 # DataHelper, ParquetPipeline, HybridDataset, the sink pipelines, etc.) — not
 # needed by the lightweight SqlRepository/AsyncSqlRepository single-row API,
@@ -93,6 +112,18 @@ dev = [
     # here so tests/data/test_sql_model_registry_pgvector.py can verify the
     # registration side effect actually happens when the package is installed.
     "pgvector>=0.3.0,<0.6.0",
+    # Not part of [project.optional-dependencies] clickhouse's runtime need on
+    # its own — it's here so the clickhouse cases in
+    # tests/data/test_sql_manager_registry.py and tests/data/test_sql_repository.py
+    # can construct real clickhousedb:// engines (DSN parsing / driver validation
+    # only, never dialed — same hermetic pattern as the mysql/postgres DSN tests).
+    "clickhouse-connect>=1.6.0,<2.0.0",
+    # Not part of [project.optional-dependencies] duckdb's runtime need on its
+    # own — it's here so the duckdb cases in tests/data/test_sql_manager_registry.py
+    # and tests/data/test_sql_repository.py can construct real duckdb:// engines,
+    # including genuine temp-file round-trips (DuckDB is embedded, so — unlike
+    # mysql/postgres/clickhouse — these tests need no external server at all).
+    "duckdb-engine>=0.17.0,<0.18.0",
     "mypy>=1.14.0",
     "pytest>=9.0.3",
     "pytest-asyncio>=1.3.0",

{boti_data-1.3.6 → boti_data-1.3.7}/src/boti_data/db/partitioned_loader.py

@@ -58,14 +58,27 @@ class SqlPartitionedLoader(PicklableLifecycleCoreMixin, LifecycleCore):
     def _validate_distributed_config(self) -> None:
         """Reject connection URLs that cannot be reopened safely by worker-local engines."""
         parsed = sqlalchemy_url.make_url(self.config.connection_url.get_secret_value())
-        if parsed.get_backend_name() != "sqlite":
+        backend = parsed.get_backend_name()
+        if backend not in {"sqlite", "duckdb"}:
             return
 
         database = parsed.database or ""
         if database == "" or ":memory:" in database:
             raise ValueError(
                 "SqlPartitionedLoader requires a database reachable by worker-local connections. "
-                "SQLite in-memory DSNs are not supported; use a file-backed DSN or request as_pandas=True."
+                f"{backend.capitalize()} in-memory DSNs are not supported; use a file-backed DSN "
+                "or request as_pandas=True."
+            )
+
+        if backend == "duckdb" and not self.config.query_only:
+            raise ValueError(
+                "SqlPartitionedLoader requires query_only=True for a file-backed DuckDB DSN. "
+                "Opening a DuckDB file in its default (read-write) mode claims an exclusive lock "
+                "on connect, even for a connection that only ever reads — so the first worker "
+                "process to connect would block every other worker with a file lock error, "
+                "regardless of whether any of them actually write. query_only=True opens each "
+                "worker's connection in DuckDB's native read-only mode instead, which multiple "
+                "processes can share concurrently without conflict."
             )
 
     def _cleanup(self) -> None:

{boti_data-1.3.6 → boti_data-1.3.7}/src/boti_data/db/sql_engine.py

@@ -70,6 +70,8 @@ def _build_driver_example(parsed_url: sqlalchemy_url.URL, *, async_mode: bool) -
         "mysql": "mysql+pymysql://",
         "postgresql": "postgresql+psycopg://",
         "sqlite": "sqlite://",
+        "clickhousedb": "clickhousedb://",
+        "duckdb": "duckdb://",
     }
     return sync_examples.get(backend, f"{backend}://")
 
@@ -104,6 +106,16 @@ def _validate_async_driver_url(
     consumer_name: str = "AsyncSqlDatabaseResource",
 ) -> sqlalchemy_url.URL:
     parsed = _parse_connection_url(connection_url, consumer_name=consumer_name)
+    if parsed.get_backend_name() == "clickhousedb":
+        raise SQLAlchemyError(
+            f"{consumer_name} does not support ClickHouse: there is no maintained "
+            "asynchronous SQLAlchemy driver for it. Use SqlDatabaseResource instead."
+        )
+    if parsed.get_backend_name() == "duckdb":
+        raise SQLAlchemyError(
+            f"{consumer_name} does not support DuckDB: there is no maintained "
+            "asynchronous SQLAlchemy driver for it. Use SqlDatabaseResource instead."
+        )
     if not parsed.get_dialect().is_async:
         suggested_driver = _build_driver_example(parsed, async_mode=True)
         raise SQLAlchemyError(
@@ -114,17 +126,42 @@ def _validate_async_driver_url(
     return parsed
 
 
+def _apply_duckdb_read_only_connect_arg(
+    connect_args: dict[str, Any], *, connection_url: str, query_only: bool
+) -> dict[str, Any]:
+    """Inject connect_args={"read_only": True} for DuckDB when query_only=True.
+
+    Unlike Postgres/MySQL/ClickHouse, DuckDB's read-only mode is a connect-time
+    flag (duckdb.connect(read_only=True)), not something settable after the
+    engine/connection already exists — so this has to happen before
+    create_engine() is called, not via _configure_query_only_engine(). Injection
+    wins on conflict, consistent with query_only=True always overriding
+    elsewhere in this codebase.
+    """
+    if not query_only:
+        return connect_args
+    backend = sqlalchemy_url.make_url(_normalize_connection_url(connection_url)).get_backend_name()
+    if backend != "duckdb":
+        return connect_args
+    return {**connect_args, "read_only": True}
+
+
 def _build_sync_engine_kwargs(
     config: SqlDatabaseConfig,
     *,
     poolclass_override: type[Pool] | None = None,
 ) -> dict[str, Any]:
     poolclass = poolclass_override or config.poolclass
+    connect_args = _apply_duckdb_read_only_connect_arg(
+        config.connect_args,
+        connection_url=config.connection_url.get_secret_value(),
+        query_only=config.query_only,
+    )
     kwargs: dict[str, Any] = {
         "pool_recycle": config.pool_recycle,
         "pool_pre_ping": config.pool_pre_ping,
         "poolclass": poolclass,
-        "connect_args": config.connect_args,
+        "connect_args": connect_args,
         "execution_options": config.execution_options,
     }
     if poolclass not in (NullPool, StaticPool):
@@ -249,11 +286,16 @@ def _resolve_worker_connection_url(config: WorkerSqlConfig) -> str:
 
 
 def _build_worker_engine_kwargs(config: WorkerSqlConfig) -> dict[str, Any]:
+    connect_args = _apply_duckdb_read_only_connect_arg(
+        config.connect_args,
+        connection_url=_resolve_worker_connection_url(config),
+        query_only=config.query_only,
+    )
     return {
         "pool_recycle": config.pool_recycle,
         "pool_pre_ping": config.pool_pre_ping,
         "poolclass": NullPool,
-        "connect_args": config.connect_args,
+        "connect_args": connect_args,
         "execution_options": config.execution_options,
     }
 
@@ -296,7 +338,7 @@ def _create_worker_sync_engine(config: SqlDatabaseConfig | WorkerSqlConfig) -> E
 def _validate_query_only_support(parsed_url: sqlalchemy_url.URL) -> None:
     backend = parsed_url.get_backend_name()
 
-    if backend in {"postgresql", "mysql"}:
+    if backend in {"postgresql", "mysql", "clickhousedb", "duckdb"}:
         return
 
     if backend == "sqlite":
@@ -349,5 +391,26 @@ def _configure_query_only_engine(engine: Engine, parsed_url: sqlalchemy_url.URL)
 
         return
 
+    if backend == "clickhousedb":
+
+        @event.listens_for(engine, "connect")
+        def _set_clickhouse_read_only(dbapi_connection: Any, _connection_record: Any) -> None:
+            # ClickHouse's HTTP interface is stateless per request, so a plain
+            # "SET readonly = 1" statement wouldn't persist across later
+            # queries on this connection the way a Postgres/MySQL SET does.
+            # set_client_setting() instead updates the driver Client's
+            # persistent default settings, which clickhouse-connect merges
+            # into every subsequent query on this connection.
+            dbapi_connection.client.set_client_setting("readonly", 1)
+
+        return
+
+    if backend == "duckdb":
+        # DuckDB's read-only mode is a connect-time flag (duckdb.connect(read_only=True)),
+        # not a runtime SQL statement, so there's nothing left to configure post-connect —
+        # it's already applied via connect_args by _apply_duckdb_read_only_connect_arg()
+        # before the engine was ever constructed.
+        return
+
     if backend == "sqlite":
         return

{boti_data-1.3.6 → boti_data-1.3.7}/src/boti_data/db/sql_model_registry.py

@@ -18,7 +18,7 @@ from typing import Any, Union
 
 from boti.core.logger import Logger
 from boti.core.security import is_valid_dotted_identifier
-from sqlalchemy import MetaData, Table
+from sqlalchemy import MetaData, PrimaryKeyConstraint, Table, text
 from sqlalchemy.engine import Engine
 from sqlalchemy.ext.asyncio import AsyncEngine
 
@@ -43,6 +43,42 @@ except ImportError as exc:
     _logger.debug("pgvector.sqlalchemy not available, vector type registration skipped: %s", exc)
 
 
+def _recover_duckdb_primary_key(
+    engine: Engine, tbl: Table, *, tname: str, schema: str | None
+) -> None:
+    """Recover a DuckDB table's real PRIMARY KEY from its catalog.
+
+    duckdb-engine's reflection doesn't report PRIMARY KEY/UNIQUE constraints at
+    all ("doesn't yet support reflection on indices"), even though DuckDB
+    enforces them at the database level — so without this, every DuckDB table
+    looks keyless to SQLAlchemy and _build_model_class's fallback binds
+    SqlRepository to an arbitrary first column instead of the real key,
+    silently corrupting get()/update()/delete() on any table whose real key
+    isn't a single first-position column. information_schema is queried
+    directly (not duckdb_constraints(), a DuckDB-specific function) since it's
+    the same standard introspection surface already assumed elsewhere in this
+    codebase for other backends.
+    """
+    if engine.dialect.name != "duckdb" or tbl.primary_key.columns:
+        return
+    with engine.connect() as conn:
+        rows = conn.execute(
+            text(
+                "SELECT kcu.column_name "
+                "FROM information_schema.table_constraints tc "
+                "JOIN information_schema.key_column_usage kcu "
+                "  ON tc.constraint_name = kcu.constraint_name "
+                "WHERE tc.constraint_type = 'PRIMARY KEY' AND tc.table_name = :tname "
+                "  AND (:schema IS NULL OR tc.table_schema = :schema) "
+                "ORDER BY kcu.ordinal_position"
+            ),
+            {"tname": tname, "schema": schema},
+        ).fetchall()
+    pk_columns = [tbl.c[row[0]] for row in rows if row[0] in tbl.c]
+    if pk_columns:
+        tbl.append_constraint(PrimaryKeyConstraint(*pk_columns))
+
+
 class SqlModelRegistry:
     """
     Thread-safe registry that reflects tables sequentially and
@@ -377,6 +413,8 @@ class SqlModelRegistry:
         if tbl is None:
             md.reflect(bind=engine, only=[tname], schema=schema)
             tbl = md.tables.get(qname)
+            if tbl is not None:
+                _recover_duckdb_primary_key(engine, tbl, tname=tname, schema=schema)
         return tbl
 
     @staticmethod

{boti_data-1.3.6 → boti_data-1.3.7}/src/boti_data/db/sql_repository.py

@@ -21,6 +21,8 @@ from typing import Any
 from boti.core.lifecycle import LifecycleCore
 from boti.core.logger import Logger
 from sqlalchemy import inspect as sa_inspect
+from sqlalchemy.engine import url as sqlalchemy_url
+from sqlalchemy.exc import SQLAlchemyError
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy.orm import Session
 
@@ -48,6 +50,34 @@ def _require_exactly_one_source(config: SqlDatabaseConfig | None, session: Any |
         raise ValueError("Provide exactly one of 'config' or 'session'.")
 
 
+def _require_unique_key_for_clickhouse(
+    engine: Any, *, assume_unique_key: bool, operation: str
+) -> None:
+    if assume_unique_key or engine.url.get_backend_name() != "clickhousedb":
+        return
+    raise SQLAlchemyError(
+        f"{operation} refuses to run against ClickHouse without assume_unique_key=True. "
+        "ClickHouse's ORDER BY/primary key is a sort prefix, not a uniqueness constraint "
+        "(even ReplacingMergeTree only dedups eventually), so several rows can share the "
+        "same key: get() could silently return an arbitrary match, and update()/delete() "
+        "would affect every matching row instead of one. Pass assume_unique_key=True only "
+        "once you've verified the key is actually unique for this table — this is a caller "
+        "assertion boti-data cannot verify or enforce itself."
+    )
+
+
+def _reject_clickhouse_backend(config: SqlDatabaseConfig, *, consumer_name: str) -> None:
+    backend = sqlalchemy_url.make_url(config.connection_url.get_secret_value()).get_backend_name()
+    if backend == "clickhousedb":
+        raise SQLAlchemyError(
+            f"{consumer_name} does not support ClickHouse. ClickHouse has no real "
+            "cross-table transactional guarantees (transactions are experimental, "
+            "single-node, MergeTree-only), so committing or rolling back several "
+            "repositories together can't be honestly backed. Use SqlRepository/"
+            "AsyncSqlRepository directly, one table at a time, instead."
+        )
+
+
 class SqlRepository(LifecycleCore):
     """Single-row get/insert/update/delete against one table, by primary key.
 
@@ -78,6 +108,14 @@ class SqlRepository(LifecycleCore):
 
     ``pk`` accepts a scalar or a tuple for composite keys, mirroring
     ``Session.get()``'s own signature.
+
+    ``get()``/``update()``/``delete()`` each accept ``assume_unique_key``,
+    which must be explicitly passed as ``True`` to run against a ClickHouse
+    table: ClickHouse's ORDER BY/primary key is a sort prefix, not a
+    uniqueness constraint, so without this assertion those calls could
+    silently touch the wrong row (``get()``) or every row sharing a key
+    (``update()``/``delete()``) instead of just one. ``insert()`` is
+    unaffected — appending a row is never ambiguous.
     """
 
     def __init__(
@@ -126,7 +164,10 @@ class SqlRepository(LifecycleCore):
         else:
             session.flush()
 
-    def get(self, pk: Any) -> dict[str, Any] | None:
+    def get(self, pk: Any, *, assume_unique_key: bool = False) -> dict[str, Any] | None:
+        _require_unique_key_for_clickhouse(
+            self.engine, assume_unique_key=assume_unique_key, operation="SqlRepository.get()"
+        )
         with self._session_scope() as session:
             instance = session.get(self._model, pk)
             return _row_to_dict(instance) if instance is not None else None
@@ -139,7 +180,14 @@ class SqlRepository(LifecycleCore):
             session.refresh(instance)
             return _row_to_dict(instance)
 
-    def update(self, pk: Any, values: dict[str, Any]) -> dict[str, Any] | None:
+    def update(
+        self, pk: Any, values: dict[str, Any], *, assume_unique_key: bool = False
+    ) -> dict[str, Any] | None:
+        _require_unique_key_for_clickhouse(
+            self.engine,
+            assume_unique_key=assume_unique_key,
+            operation="SqlRepository.update()",
+        )
         with self._session_scope() as session:
             instance = session.get(self._model, pk)
             if instance is None:
@@ -150,7 +198,12 @@ class SqlRepository(LifecycleCore):
             session.refresh(instance)
             return _row_to_dict(instance)
 
-    def delete(self, pk: Any) -> bool:
+    def delete(self, pk: Any, *, assume_unique_key: bool = False) -> bool:
+        _require_unique_key_for_clickhouse(
+            self.engine,
+            assume_unique_key=assume_unique_key,
+            operation="SqlRepository.delete()",
+        )
         with self._session_scope() as session:
             instance = session.get(self._model, pk)
             if instance is None:
@@ -169,6 +222,10 @@ class AsyncSqlRepository(LifecycleCore):
     ``__aenter__``, not ``__init__``, since resolving the engine requires
     awaiting ``EngineRegistry.get_or_create_async()`` (``config`` path) or
     ``SqlModelRegistry.get_model_async()`` (``session`` path either way).
+
+    ``get()``/``update()``/``delete()`` each accept ``assume_unique_key`` —
+    see :class:`SqlRepository`'s docstring for why this is required against
+    ClickHouse.
     """
 
     def __init__(
@@ -232,7 +289,12 @@ class AsyncSqlRepository(LifecycleCore):
         else:
             await session.flush()
 
-    async def get(self, pk: Any) -> dict[str, Any] | None:
+    async def get(self, pk: Any, *, assume_unique_key: bool = False) -> dict[str, Any] | None:
+        _require_unique_key_for_clickhouse(
+            self.engine,
+            assume_unique_key=assume_unique_key,
+            operation="AsyncSqlRepository.get()",
+        )
         async with self._session_scope() as session:
             instance = await session.get(self._model, pk)
             return _row_to_dict(instance) if instance is not None else None
@@ -245,7 +307,14 @@ class AsyncSqlRepository(LifecycleCore):
             await session.refresh(instance)
             return _row_to_dict(instance)
 
-    async def update(self, pk: Any, values: dict[str, Any]) -> dict[str, Any] | None:
+    async def update(
+        self, pk: Any, values: dict[str, Any], *, assume_unique_key: bool = False
+    ) -> dict[str, Any] | None:
+        _require_unique_key_for_clickhouse(
+            self.engine,
+            assume_unique_key=assume_unique_key,
+            operation="AsyncSqlRepository.update()",
+        )
         async with self._session_scope() as session:
             instance = await session.get(self._model, pk)
             if instance is None:
@@ -256,7 +325,12 @@ class AsyncSqlRepository(LifecycleCore):
             await session.refresh(instance)
             return _row_to_dict(instance)
 
-    async def delete(self, pk: Any) -> bool:
+    async def delete(self, pk: Any, *, assume_unique_key: bool = False) -> bool:
+        _require_unique_key_for_clickhouse(
+            self.engine,
+            assume_unique_key=assume_unique_key,
+            operation="AsyncSqlRepository.delete()",
+        )
         async with self._session_scope() as session:
             instance = await session.get(self._model, pk)
             if instance is None:
@@ -283,6 +357,7 @@ class SqlUnitOfWork:
     """
 
     def __init__(self, config: SqlDatabaseConfig, *, query_only: bool = True) -> None:
+        _reject_clickhouse_backend(config, consumer_name="SqlUnitOfWork")
         effective_config = config.model_copy(update={"query_only": query_only})
         self._resource = SqlDatabaseResource(effective_config)
         self._session: Session = self._resource.session()
@@ -326,6 +401,7 @@ class AsyncSqlUnitOfWork:
     """
 
     def __init__(self, config: SqlDatabaseConfig, *, query_only: bool = True) -> None:
+        _reject_clickhouse_backend(config, consumer_name="AsyncSqlUnitOfWork")
         self._config = config.model_copy(update={"query_only": query_only})
         self._resource: AsyncSqlDatabaseResource | None = None
         self._session: AsyncSession | None = None

boti_data-1.3.6/README.md → boti_data-1.3.7/src/boti_data.egg-info/PKG-INFO

@@ -1,3 +1,42 @@
+Metadata-Version: 2.4
+Name: boti-data
+Version: 1.3.7
+Summary: Data infrastructure for the Boti ecosystem
+Author: Luis Valverde
+Maintainer-email: Luis Valverde <lvalverdeb@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/lvalverdeb/boti-data
+Project-URL: Repository, https://github.com/lvalverdeb/boti-data
+Project-URL: Documentation, https://github.com/lvalverdeb/boti-data#readme
+Project-URL: Issues, https://github.com/lvalverdeb/boti-data/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: boti<2.0.0,>=1.0.2
+Requires-Dist: sqlalchemy[asyncio]<3.0.0,>=2.0.49
+Provides-Extra: mysql
+Requires-Dist: asyncmy<0.3.0,>=0.2.11; extra == "mysql"
+Requires-Dist: pymysql<2.0.0,>=1.1.2; extra == "mysql"
+Provides-Extra: postgres
+Requires-Dist: asyncpg<0.32.0,>=0.30.0; extra == "postgres"
+Requires-Dist: psycopg[binary]<4.0.0,>=3.2.0; extra == "postgres"
+Provides-Extra: pgvector
+Requires-Dist: pgvector<0.6.0,>=0.3.0; extra == "pgvector"
+Provides-Extra: clickhouse
+Requires-Dist: clickhouse-connect<2.0.0,>=1.6.0; extra == "clickhouse"
+Provides-Extra: duckdb
+Requires-Dist: duckdb-engine<0.18.0,>=0.17.0; extra == "duckdb"
+Provides-Extra: dataframes
+Requires-Dist: boti-dask<2.0.0,>=1.0.0; extra == "dataframes"
+Requires-Dist: dask[dataframe,distributed]<2027.0.0,>=2026.3.0; extra == "dataframes"
+Requires-Dist: pandas<4.0.0,>=3.0.2; extra == "dataframes"
+Requires-Dist: polars<2.0.0,>=1.29.0; extra == "dataframes"
+
 # boti-data
 
 `boti-data` is the **data access and data transformation layer** of the Boti ecosystem.
@@ -96,8 +135,25 @@ pip install "boti[data]"
 ```bash
 pip install "boti-data[mysql]"      # asyncmy + pymysql
 pip install "boti-data[postgres]"   # asyncpg + psycopg[binary]
+pip install "boti-data[clickhouse]" # clickhouse-connect (sync only)
+pip install "boti-data[duckdb]"     # duckdb-engine (sync only)
 ```
 
+ClickHouse support is sync-only: there is no maintained async SQLAlchemy driver for ClickHouse, so `AsyncSqlDatabaseResource` rejects a ClickHouse connection URL — use `SqlDatabaseResource` instead. `SqlUnitOfWork`/`AsyncSqlUnitOfWork` also reject ClickHouse outright, since ClickHouse's transaction support is experimental, single-node, and MergeTree-only, with no real cross-table ACID guarantees to back a multi-table unit of work — use `SqlRepository`/`AsyncSqlRepository` directly, one table at a time, instead.
+
+Verified against a real ClickHouse instance:
+
+- `SqlRepository`/`AsyncSqlRepository` are a get/insert/update/delete-**by-primary-key** API, but MergeTree's `ORDER BY` is a sort/index prefix, not a uniqueness constraint — many real ClickHouse tables have several rows per `ORDER BY` key. Because of that, `get()`/`update()`/`delete()` each **refuse to run against a ClickHouse table unless called with `assume_unique_key=True`** (e.g. `repo.get(pk, assume_unique_key=True)`). This flag is a per-call caller assertion, not something boti-data verifies or enforces: it does not check the data, and if the key genuinely isn't unique, `get(key, assume_unique_key=True)` will still return one arbitrary matching row and `delete(key, assume_unique_key=True)`/`update(key, ..., assume_unique_key=True)` will still affect *every* row sharing that key, not just one. Only pass it once you've confirmed the key is actually unique for that table. `insert()` has no such parameter — appending a row is never ambiguous.
+- `.update()` also fails outright on a table without `enable_block_number_column` enabled (`ALTER TABLE ... MODIFY SETTING enable_block_number_column = 1`, off by default) — ClickHouse raises `Lightweight updates are not supported` otherwise.
+- `query_only=True` is enforced server-side (ClickHouse's `readonly` setting, applied via `set_client_setting` on connect), not just by boti-data's own read-only session guard.
+
+DuckDB support is also sync-only (no maintained async driver, same as ClickHouse — `AsyncSqlDatabaseResource` rejects a DuckDB connection URL), but is otherwise a better fit than ClickHouse for `SqlRepository`'s by-primary-key model:
+
+- **`SqlUnitOfWork`/`AsyncSqlUnitOfWork` are fully supported** — DuckDB has real multi-table ACID transactions (unlike ClickHouse), verified with an actual mid-transaction constraint violation correctly rolling back writes across two tables.
+- **No `assume_unique_key`-style flag is needed** — DuckDB genuinely enforces `PRIMARY KEY`/`UNIQUE` constraints at the database level, and boti-data recovers the real key columns from DuckDB's catalog even though `duckdb-engine`'s own reflection can't see them (a known upstream limitation — it warns "doesn't yet support reflection on indices"). Without that recovery step, every DuckDB table would look keyless and `SqlRepository` would silently bind to an arbitrary first column instead of the real key — this was caught and fixed during implementation, not just assumed away.
+- **`query_only=True` is enforced via a connect-time flag**, not a runtime statement: DuckDB's read-only mode (`duckdb.connect(read_only=True)`) can only be set when a connection is opened, so boti-data injects `connect_args={"read_only": True}` itself before the engine is even constructed, rather than configuring it afterward the way Postgres/MySQL/ClickHouse do.
+- **DuckDB allows only one process to hold a database file open for read-write at a time** — opening a file in its default mode claims an exclusive lock on connect, even for a connection that only ever reads. `SqlPartitionedLoader` (which spawns one engine per worker process) requires `query_only=True` for a file-backed DuckDB DSN for exactly this reason — otherwise the first worker to connect would block every other worker, regardless of whether any of them write. Concurrent read-only access from multiple processes works fine. In-memory (`duckdb:///:memory:`) DSNs are rejected for distributed loading the same way SQLite's are, since each worker process would get its own independent, empty database.
+
 If a Postgres table has a [pgvector](https://github.com/pgvector/pgvector) `vector` column, install `boti-data[pgvector]` too — this registers pgvector's `Vector` type with SQLAlchemy's reflection so `SqlAlchemyModelBuilder`/`SqlRepository`/`DataGateway` map that column to a usable `list[float]` (with its dimension preserved) instead of an unreflectable `NullType`:
 
 ```bash

{boti_data-1.3.6 → boti_data-1.3.7}/src/boti_data.egg-info/requires.txt

@@ -1,12 +1,18 @@
 boti<2.0.0,>=1.0.2
 sqlalchemy[asyncio]<3.0.0,>=2.0.49
 
+[clickhouse]
+clickhouse-connect<2.0.0,>=1.6.0
+
 [dataframes]
 boti-dask<2.0.0,>=1.0.0
 dask[dataframe,distributed]<2027.0.0,>=2026.3.0
 pandas<4.0.0,>=3.0.2
 polars<2.0.0,>=1.29.0
 
+[duckdb]
+duckdb-engine<0.18.0,>=0.17.0
+
 [mysql]
 asyncmy<0.3.0,>=0.2.11
 pymysql<2.0.0,>=1.1.2