sqlmodel-object-helpers 0.0.9__tar.gz → 0.0.10__tar.gz

{sqlmodel_object_helpers-0.0.9 → sqlmodel_object_helpers-0.0.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: sqlmodel-object-helpers
-Version: 0.0.9
+Version: 0.0.10
 Summary: Generic async query helpers for SQLModel: filtering, eager loading, pagination
 Project-URL: Homepage, https://github.com/itstandart/sqlmodel-object-helpers
 Project-URL: Repository, https://github.com/itstandart/sqlmodel-object-helpers
@@ -43,6 +43,8 @@ Generic async query helpers for [SQLModel](https://sqlmodel.tiangolo.com/): filt
 - **Pagination** - Page/per\_page with total count and configurable max\_per\_page limit
 - **Projections** - Select specific columns across joins with dot-notation and SQL aliases
 - **CRUD Mutations** - `add_object`, `update_object`, `delete_object` plus bulk `update_objects`, `delete_objects` with flush-only semantics for composable transactions
+- **Idempotent Bulk Insert** - `insert_objects_ignore_conflicts`: one `INSERT ... ON CONFLICT DO NOTHING` per chunk (PostgreSQL, SQLite), safe to replay
+- **Bulk Upsert** - `upsert_objects`: one `INSERT ... ON CONFLICT DO UPDATE` per chunk (PostgreSQL, SQLite); update expressions run on every call
 - **Count & Exists** - `count_objects` (single `SELECT count(*)`) and `exists_object` (`SELECT EXISTS(...)`) without loading data
 - **Row Locking** - `for_update` parameter on `get_object` for `SELECT ... FOR UPDATE`
 - **Time Filtering** - `TimeFilter` for `created_at`/`updated_at` range filtering with half-open interval semantics
@@ -392,6 +394,14 @@ user = await soh.add_object(session, User(name="Alice"))
 users = await soh.add_objects(session, [User(name="Alice"), User(name="Bob")])
 ```
 
+Choose the insert helper by its contract:
+
+| Need | Helper | Result |
+|---|---|---|
+| Persist ORM objects and populate generated fields; duplicates are errors | `add_objects` | ORM objects |
+| Insert rows and skip duplicates by a unique key | [`insert_objects_ignore_conflicts`](#insert_objects_ignore_conflicts) | Inserted count |
+| Insert rows and update existing ones by a unique key | [`upsert_objects`](#upsert_objects) | Inserted + updated count |
+
 ### update_object
 
 ```python
@@ -440,6 +450,47 @@ count = await soh.delete_objects(
 
 Filters are **required** for safety -- empty filters are rejected. Dot-notation (relationship) filters are not supported in bulk operations.
 
+### insert_objects_ignore_conflicts
+
+Idempotent bulk insert -- `INSERT ... ON CONFLICT (...) DO NOTHING`, no objects loaded or refreshed:
+
+```python
+import sqlmodel_object_helpers as soh
+
+inserted = await soh.insert_objects_ignore_conflicts(
+    session, PaymentLog,
+    rows=[{"source_id": "run1:1", "base": "a-cso"}, {"source_id": "run1:2", "base": "a-cso"}],
+    conflict_columns=["source_id"],
+)
+# inserted -> rows actually written; duplicates by source_id are skipped and not counted
+```
+
+Replaying the same batch (or two concurrent identical batches) is safe, unlike `add_objects`, where one duplicate fails the whole batch. All rows must have the same keys. Rows go to the database as given (no Pydantic validation or coercion -- pass already-typed values); omitted columns get their column defaults: `Field(default=...)` / `default_factory` of a plain field and `server_default` (SQLModel does not copy `Field(default)` into an explicit `sa_column`). Large inputs are split using the actual compiled bind-parameter count, including omitted Python defaults and SQL expressions. One row exceeding the parameter budget raises `MutationError`. PostgreSQL and SQLite only (`MutationError` on other dialects).
+
+The caller owns the transaction. If a later chunk fails, roll back the transaction to discard earlier chunks too. The helper does not commit or perform its own rollback. Standalone wrappers manage commit/rollback automatically.
+
+### upsert_objects
+
+Bulk upsert -- `INSERT ... ON CONFLICT (...) DO UPDATE SET ...`: new rows are inserted, rows hitting the unique key are updated:
+
+```python
+import sqlmodel_object_helpers as soh
+from sqlalchemy import func
+
+written = await soh.upsert_objects(
+    session, Ast,
+    rows=[{"code": "a", "width": 10, "ordinal": 1}, {"code": "b", "width": 20, "ordinal": 2}],
+    conflict_columns=["code"],
+    update_columns=["width", "ordinal"],        # default: every row field except conflict_columns
+    set_extra={"updated_at": func.now()},       # applied only on update
+)
+# written -> distinct table rows inserted or updated, counted by primary key from RETURNING
+```
+
+If a key repeats inside `rows`, the last row wins and the row is counted once. Repeats are decided by the database for the column type (`NUMERIC` rounding, `citext`, `NULLS NOT DISTINCT`), not by Python equality: distinct values such as two instants of a DST fold stay distinct, and key values do not have to be hashable (`ARRAY`). On PostgreSQL a chunk of several rows runs inside a SAVEPOINT (`session.begin_nested()` flushes pending session changes first); a chunk the database rejects for a repeated key is retried row by row, which is slower. SQLite applies repeated keys in order within one statement. Every `conflict_columns` field must be present in each row. `update_columns` must be a subset of the row fields outside `conflict_columns`; `set_extra` takes model fields (value or SQL expression) that are in neither. When nothing is left to update, use `insert_objects_ignore_conflicts` instead (`MutationError`). `Column.onupdate` does not fire on conflict -- pass it in `set_extra`. Objects already loaded in the session are not refreshed. A conditional update (`WHERE`) or `RETURNING` is not covered -- build the statement with SQLAlchemy for that. Other rules as in `insert_objects_ignore_conflicts`: same keys in all rows, no Pydantic validation, actual parameter-based chunking, caller-managed transaction, PostgreSQL and SQLite only.
+
+Repeating an upsert does not create duplicate keys, but it repeats the update: expressions such as `hits + 1` and `func.now()` run again. NULL conflict-key values follow the index: an ordinary UNIQUE keeps them distinct, `NULLS NOT DISTINCT` treats them as one key.
+
 ### check_for_related_records
 
 Pre-deletion check that inspects all ONETOMANY relationships:
@@ -772,6 +823,8 @@ Endpoints can be migrated one at a time. Three strategies per endpoint:
 - `soh.update_objects(session, model, data, filters)` -- Bulk update via single `UPDATE ... WHERE`
 - `soh.delete_object(session, model, *, instance, pk)` -- Delete by reference or PK
 - `soh.delete_objects(session, model, filters)` -- Bulk delete via single `DELETE ... WHERE`
+- `soh.insert_objects_ignore_conflicts(session, model, rows, *, conflict_columns)` -- Bulk insert via `INSERT ... ON CONFLICT DO NOTHING`, returns inserted count
+- `soh.upsert_objects(session, model, rows, *, conflict_columns, update_columns=None, set_extra=None)` -- Bulk insert via `INSERT ... ON CONFLICT DO UPDATE`, returns inserted + updated count
 - `soh.check_for_related_records(session, model, pk)` -- Pre-deletion dependency check
 
 ### Dynamic Meta
@@ -850,6 +903,14 @@ uv sync
 uv run pytest tests/
 ```
 
+PostgreSQL integration tests use the optional `postgres` dependency group (asyncpg). Each test creates a randomly named schema and removes that schema afterwards; it does not recreate application tables. Without `SOH_TEST_POSTGRES_URL`, these tests are skipped.
+
+```bash
+uv sync --group postgres
+SOH_TEST_POSTGRES_URL=postgresql+asyncpg://user:password@localhost/test_db \
+  uv run --group postgres pytest -q tests/test_conflict_mutations_postgres.py
+```
+
 ### Syncing with Remote
 
 ```bash
@@ -891,6 +952,14 @@ This project follows [Semantic Versioning](https://semver.org/) (MAJOR.MINOR.PAT
 
 ## Changelog
 
+### 0.0.10 — unreleased
+
+- `insert_objects_ignore_conflicts` and `upsert_objects`: bulk insert with explicit unique-key conflict handling, inserted/updated counts, and standalone wrappers.
+- Chunk sizes account for actual SQL parameters, including omitted Python defaults, repeated positional parameters, expanding `IN` expressions, and update expressions. Tested with SQLAlchemy 2.0.46, 2.0.52 and 2.1.6.
+- Upsert reports a missing conflict-key field as `MutationError`. Repeated update expressions are explicitly documented as non-idempotent.
+- Upsert leaves repeated keys to the database instead of Python equality: no silent loss of distinct keys (DST fold), no `DatabaseError` for keys equal only after column type coercion (`NUMERIC(10,2)`), no `TypeError` for unhashable keys (`ARRAY`). The count is distinct rows by primary key.
+- PostgreSQL tests run through the optional `postgres` dependency group; tested on PostgreSQL 18.4 and 18.1, the versions of the SRO Agent and ExamCRM dev databases.
+
 ### 0.0.9 — 2026-10-09
 
 - Dependency lower bound is now `sqlmodel>=0.0.27`. SQLModel 0.0.22–0.0.26 cannot create models on Python 3.14 (`Field 'id' requires a type annotation`), so the previous `>=0.0.22` bound was never satisfiable. Test suite passes on every SQLModel from 0.0.27 to 0.0.48.

{sqlmodel_object_helpers-0.0.9 → sqlmodel_object_helpers-0.0.10}/README.md

@@ -15,6 +15,8 @@ Generic async query helpers for [SQLModel](https://sqlmodel.tiangolo.com/): filt
 - **Pagination** - Page/per\_page with total count and configurable max\_per\_page limit
 - **Projections** - Select specific columns across joins with dot-notation and SQL aliases
 - **CRUD Mutations** - `add_object`, `update_object`, `delete_object` plus bulk `update_objects`, `delete_objects` with flush-only semantics for composable transactions
+- **Idempotent Bulk Insert** - `insert_objects_ignore_conflicts`: one `INSERT ... ON CONFLICT DO NOTHING` per chunk (PostgreSQL, SQLite), safe to replay
+- **Bulk Upsert** - `upsert_objects`: one `INSERT ... ON CONFLICT DO UPDATE` per chunk (PostgreSQL, SQLite); update expressions run on every call
 - **Count & Exists** - `count_objects` (single `SELECT count(*)`) and `exists_object` (`SELECT EXISTS(...)`) without loading data
 - **Row Locking** - `for_update` parameter on `get_object` for `SELECT ... FOR UPDATE`
 - **Time Filtering** - `TimeFilter` for `created_at`/`updated_at` range filtering with half-open interval semantics
@@ -364,6 +366,14 @@ user = await soh.add_object(session, User(name="Alice"))
 users = await soh.add_objects(session, [User(name="Alice"), User(name="Bob")])
 ```
 
+Choose the insert helper by its contract:
+
+| Need | Helper | Result |
+|---|---|---|
+| Persist ORM objects and populate generated fields; duplicates are errors | `add_objects` | ORM objects |
+| Insert rows and skip duplicates by a unique key | [`insert_objects_ignore_conflicts`](#insert_objects_ignore_conflicts) | Inserted count |
+| Insert rows and update existing ones by a unique key | [`upsert_objects`](#upsert_objects) | Inserted + updated count |
+
 ### update_object
 
 ```python
@@ -412,6 +422,47 @@ count = await soh.delete_objects(
 
 Filters are **required** for safety -- empty filters are rejected. Dot-notation (relationship) filters are not supported in bulk operations.
 
+### insert_objects_ignore_conflicts
+
+Idempotent bulk insert -- `INSERT ... ON CONFLICT (...) DO NOTHING`, no objects loaded or refreshed:
+
+```python
+import sqlmodel_object_helpers as soh
+
+inserted = await soh.insert_objects_ignore_conflicts(
+    session, PaymentLog,
+    rows=[{"source_id": "run1:1", "base": "a-cso"}, {"source_id": "run1:2", "base": "a-cso"}],
+    conflict_columns=["source_id"],
+)
+# inserted -> rows actually written; duplicates by source_id are skipped and not counted
+```
+
+Replaying the same batch (or two concurrent identical batches) is safe, unlike `add_objects`, where one duplicate fails the whole batch. All rows must have the same keys. Rows go to the database as given (no Pydantic validation or coercion -- pass already-typed values); omitted columns get their column defaults: `Field(default=...)` / `default_factory` of a plain field and `server_default` (SQLModel does not copy `Field(default)` into an explicit `sa_column`). Large inputs are split using the actual compiled bind-parameter count, including omitted Python defaults and SQL expressions. One row exceeding the parameter budget raises `MutationError`. PostgreSQL and SQLite only (`MutationError` on other dialects).
+
+The caller owns the transaction. If a later chunk fails, roll back the transaction to discard earlier chunks too. The helper does not commit or perform its own rollback. Standalone wrappers manage commit/rollback automatically.
+
+### upsert_objects
+
+Bulk upsert -- `INSERT ... ON CONFLICT (...) DO UPDATE SET ...`: new rows are inserted, rows hitting the unique key are updated:
+
+```python
+import sqlmodel_object_helpers as soh
+from sqlalchemy import func
+
+written = await soh.upsert_objects(
+    session, Ast,
+    rows=[{"code": "a", "width": 10, "ordinal": 1}, {"code": "b", "width": 20, "ordinal": 2}],
+    conflict_columns=["code"],
+    update_columns=["width", "ordinal"],        # default: every row field except conflict_columns
+    set_extra={"updated_at": func.now()},       # applied only on update
+)
+# written -> distinct table rows inserted or updated, counted by primary key from RETURNING
+```
+
+If a key repeats inside `rows`, the last row wins and the row is counted once. Repeats are decided by the database for the column type (`NUMERIC` rounding, `citext`, `NULLS NOT DISTINCT`), not by Python equality: distinct values such as two instants of a DST fold stay distinct, and key values do not have to be hashable (`ARRAY`). On PostgreSQL a chunk of several rows runs inside a SAVEPOINT (`session.begin_nested()` flushes pending session changes first); a chunk the database rejects for a repeated key is retried row by row, which is slower. SQLite applies repeated keys in order within one statement. Every `conflict_columns` field must be present in each row. `update_columns` must be a subset of the row fields outside `conflict_columns`; `set_extra` takes model fields (value or SQL expression) that are in neither. When nothing is left to update, use `insert_objects_ignore_conflicts` instead (`MutationError`). `Column.onupdate` does not fire on conflict -- pass it in `set_extra`. Objects already loaded in the session are not refreshed. A conditional update (`WHERE`) or `RETURNING` is not covered -- build the statement with SQLAlchemy for that. Other rules as in `insert_objects_ignore_conflicts`: same keys in all rows, no Pydantic validation, actual parameter-based chunking, caller-managed transaction, PostgreSQL and SQLite only.
+
+Repeating an upsert does not create duplicate keys, but it repeats the update: expressions such as `hits + 1` and `func.now()` run again. NULL conflict-key values follow the index: an ordinary UNIQUE keeps them distinct, `NULLS NOT DISTINCT` treats them as one key.
+
 ### check_for_related_records
 
 Pre-deletion check that inspects all ONETOMANY relationships:
@@ -744,6 +795,8 @@ Endpoints can be migrated one at a time. Three strategies per endpoint:
 - `soh.update_objects(session, model, data, filters)` -- Bulk update via single `UPDATE ... WHERE`
 - `soh.delete_object(session, model, *, instance, pk)` -- Delete by reference or PK
 - `soh.delete_objects(session, model, filters)` -- Bulk delete via single `DELETE ... WHERE`
+- `soh.insert_objects_ignore_conflicts(session, model, rows, *, conflict_columns)` -- Bulk insert via `INSERT ... ON CONFLICT DO NOTHING`, returns inserted count
+- `soh.upsert_objects(session, model, rows, *, conflict_columns, update_columns=None, set_extra=None)` -- Bulk insert via `INSERT ... ON CONFLICT DO UPDATE`, returns inserted + updated count
 - `soh.check_for_related_records(session, model, pk)` -- Pre-deletion dependency check
 
 ### Dynamic Meta
@@ -822,6 +875,14 @@ uv sync
 uv run pytest tests/
 ```
 
+PostgreSQL integration tests use the optional `postgres` dependency group (asyncpg). Each test creates a randomly named schema and removes that schema afterwards; it does not recreate application tables. Without `SOH_TEST_POSTGRES_URL`, these tests are skipped.
+
+```bash
+uv sync --group postgres
+SOH_TEST_POSTGRES_URL=postgresql+asyncpg://user:password@localhost/test_db \
+  uv run --group postgres pytest -q tests/test_conflict_mutations_postgres.py
+```
+
 ### Syncing with Remote
 
 ```bash
@@ -863,6 +924,14 @@ This project follows [Semantic Versioning](https://semver.org/) (MAJOR.MINOR.PAT
 
 ## Changelog
 
+### 0.0.10 — unreleased
+
+- `insert_objects_ignore_conflicts` and `upsert_objects`: bulk insert with explicit unique-key conflict handling, inserted/updated counts, and standalone wrappers.
+- Chunk sizes account for actual SQL parameters, including omitted Python defaults, repeated positional parameters, expanding `IN` expressions, and update expressions. Tested with SQLAlchemy 2.0.46, 2.0.52 and 2.1.6.
+- Upsert reports a missing conflict-key field as `MutationError`. Repeated update expressions are explicitly documented as non-idempotent.
+- Upsert leaves repeated keys to the database instead of Python equality: no silent loss of distinct keys (DST fold), no `DatabaseError` for keys equal only after column type coercion (`NUMERIC(10,2)`), no `TypeError` for unhashable keys (`ARRAY`). The count is distinct rows by primary key.
+- PostgreSQL tests run through the optional `postgres` dependency group; tested on PostgreSQL 18.4 and 18.1, the versions of the SRO Agent and ExamCRM dev databases.
+
 ### 0.0.9 — 2026-10-09
 
 - Dependency lower bound is now `sqlmodel>=0.0.27`. SQLModel 0.0.22–0.0.26 cannot create models on Python 3.14 (`Field 'id' requires a type annotation`), so the previous `>=0.0.22` bound was never satisfiable. Test suite passes on every SQLModel from 0.0.27 to 0.0.48.

{sqlmodel_object_helpers-0.0.9 → sqlmodel_object_helpers-0.0.10}/pyproject.toml

@@ -45,10 +45,13 @@ packages = ["src/sqlmodel_object_helpers"]
 
 [dependency-groups]
 dev = [
-    "ruff==0.16.10",
+    "ruff==0.17.2",
     "pyright==1.1.414",
     "tzdata==2026.5",
     "pytest==9.1.1",
     "pytest-asyncio==1.4.0",
     "aiosqlite==0.22.1",
 ]
+postgres = [
+    "asyncpg==0.31.0",
+]

{sqlmodel_object_helpers-0.0.9 → sqlmodel_object_helpers-0.0.10}/src/sqlmodel_object_helpers/__init__.py

@@ -1,6 +1,6 @@
 """sqlmodel-object-helpers — reusable query helpers for SQLModel projects."""
 
-__version__ = "0.0.9"
+__version__ = "0.0.10"
 
 from .constants import QueryHelperSettings, settings
 from .dynamic_meta import (
@@ -26,8 +26,10 @@ from .mutations import (
     check_for_related_records,
     delete_object,
     delete_objects,
+    insert_objects_ignore_conflicts,
     update_object,
     update_objects,
+    upsert_objects,
 )
 from .operators import SPECIAL_OPERATORS, SUPPORTED_OPERATORS, LogicalOperator, Operator
 from .query import count_objects, exists_object, get_object, get_objects, get_projection
@@ -119,9 +121,11 @@ __all__ = [
     "get_object",
     "get_objects",
     "get_projection",
+    "insert_objects_ignore_conflicts",
     "invalidate_meta_cache",
     "load_pg_comments",
     "settings",
     "update_object",
     "update_objects",
+    "upsert_objects",
 ]

{sqlmodel_object_helpers-0.0.9 → sqlmodel_object_helpers-0.0.10}/src/sqlmodel_object_helpers/mutations.py

@@ -18,12 +18,15 @@ composed into a single atomic transaction at the call-site::
 
 import logging
 import time
+from collections.abc import Callable, Iterator, Mapping, Sequence
 from typing import Any, cast
 
 from sqlalchemy import CursorResult, inspect
 from sqlalchemy import delete as sa_delete
 from sqlalchemy import update as sa_update
-from sqlalchemy.exc import IntegrityError
+from sqlalchemy.dialects.postgresql import insert as pg_insert
+from sqlalchemy.dialects.sqlite import insert as sqlite_insert
+from sqlalchemy.exc import DBAPIError, IntegrityError
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy.orm import ONETOMANY
 from sqlmodel import SQLModel, and_, select
@@ -35,6 +38,15 @@ from .query import get_object
 
 logger = logging.getLogger("sqlmodel_object_helpers")
 
+# Dialects that support INSERT ... ON CONFLICT.
+_ON_CONFLICT_INSERT: dict[str, Callable[..., Any]] = {"postgresql": pg_insert, "sqlite": sqlite_insert}
+
+# Headroom below the bind-parameter limits of asyncpg (32767) and SQLite (32766).
+_MAX_BIND_PARAMS = 30_000
+
+# PostgreSQL SQLSTATE: ON CONFLICT DO UPDATE affected the same row twice in one statement.
+_CARDINALITY_VIOLATION = "21000"
+
 
 async def add_object[T: SQLModel](
     session: AsyncSession,
@@ -416,6 +428,302 @@ async def delete_objects[T: SQLModel](
         return row_count
 
 
+def _prepare_bulk_insert(
+    func_name: str,
+    session: AsyncSession,
+    model: type[SQLModel],
+    rows: Sequence[Mapping[str, Any]],
+    conflict_columns: Sequence[str],
+) -> tuple[Callable[..., Any], Any, set[str]]:
+    """Validate rows and the conflict key; return the INSERT constructor, model columns and row fields."""
+    model_name = model.__name__
+
+    if not conflict_columns:
+        msg = f"{func_name}({model_name}): conflict_columns are required"
+        raise MutationError(msg)
+
+    columns = inspect(model).columns
+    fields = set(rows[0])
+    if not fields:
+        msg = f"{func_name}({model_name}): rows cannot be empty mappings"
+        raise MutationError(msg)
+
+    for key in (*fields, *conflict_columns):
+        if key not in columns:
+            msg = f"Model {model_name} has no field {key}"
+            raise MutationError(msg)
+
+    if any(set(row) != fields for row in rows):
+        msg = f"{func_name}({model_name}): all rows must have the same keys"
+        raise MutationError(msg)
+
+    dialect = session.get_bind(model).dialect.name
+    insert_fn = _ON_CONFLICT_INSERT.get(dialect)
+    if insert_fn is None:
+        msg = f"{func_name}({model_name}): ON CONFLICT is not supported on {dialect}"
+        raise MutationError(msg)
+
+    return insert_fn, columns, fields
+
+
+def _bulk_insert_statements(
+    session: AsyncSession,
+    model: type[SQLModel],
+    rows: Sequence[Mapping[str, Any]],
+    fields: set[str],
+    make_stmt: Callable[[list[dict[str, Any]]], Any],
+) -> Iterator[tuple[list[dict[str, Any]], Any]]:
+    """Chunk rows by the compiled bind-parameter count, including defaults and update expressions."""
+    dialect = session.get_bind(model).dialect
+    chunk_size = max(1, _MAX_BIND_PARAMS // len(fields))
+    offset = 0
+    while offset < len(rows):
+        chunk = [dict(row) for row in rows[offset : offset + chunk_size]]
+        while True:
+            stmt = make_stmt(chunk)
+            compiled = stmt.compile(dialect=dialect)
+            expanded = compiled.construct_expanded_state()
+            # Positional dialects may repeat one named parameter several times.
+            count = len(expanded.positiontup) if expanded.positiontup is not None else len(expanded.parameters)
+            if count <= _MAX_BIND_PARAMS:
+                break
+            if len(chunk) == 1:
+                msg = f"A single row requires {count} bind parameters; limit is {_MAX_BIND_PARAMS}"
+                raise MutationError(msg)
+            chunk_size = max(1, min(len(chunk) - 1, len(chunk) * _MAX_BIND_PARAMS // count))
+            chunk = chunk[:chunk_size]
+        yield chunk, stmt
+        offset += len(chunk)
+
+
+def _hashable(value: Any) -> Any:
+    """Make a RETURNING value hashable: ARRAY values come back as lists."""
+    if isinstance(value, list | tuple):
+        return tuple(_hashable(item) for item in value)
+    return value
+
+
+def _row_keys(result: Any) -> set[Any]:
+    """Primary keys of the rows returned by RETURNING."""
+    return {_hashable(tuple(row)) for row in result}
+
+
+def _sqlstate(error: DBAPIError) -> str | None:
+    """Driver error SQLSTATE: ``sqlstate`` on asyncpg and psycopg, ``pgcode`` on psycopg2."""
+    return getattr(error.orig, "sqlstate", None) or getattr(error.orig, "pgcode", None)
+
+
+async def _upsert_chunk(
+    session: AsyncSession,
+    chunk: list[dict[str, Any]],
+    stmt: Any,
+    make_stmt: Callable[[list[dict[str, Any]]], Any],
+    *,
+    savepoint: bool,
+) -> set[Any]:
+    """Execute one upsert chunk and return the primary keys of the affected rows."""
+    if not savepoint or len(chunk) == 1:
+        return _row_keys(await session.execute(stmt))
+    try:
+        async with session.begin_nested():
+            return _row_keys(await session.execute(stmt))
+    except DBAPIError as e:
+        if _sqlstate(e) != _CARDINALITY_VIOLATION:
+            raise
+    # PostgreSQL rejects keys that are equal for the database (per column type) within one statement.
+    logger.debug("upsert chunk of %d rows has equal keys, retrying row by row", len(chunk))
+    keys: set[Any] = set()
+    for row in chunk:
+        keys |= _row_keys(await session.execute(make_stmt([row])))
+    return keys
+
+
+async def _execute_bulk_insert(
+    func_name: str,
+    session: AsyncSession,
+    model: type[SQLModel],
+    rows: Sequence[Mapping[str, Any]],
+    fields: set[str],
+    make_stmt: Callable[[list[dict[str, Any]]], Any],
+    *,
+    upsert: bool = False,
+) -> int:
+    """Execute chunks without committing: total rowcount, or distinct RETURNING PKs for upsert."""
+    start = time.perf_counter()
+    model_name = model.__name__
+    affected = 0
+    written: set[Any] = set()
+    # Only PostgreSQL rejects a repeated key within one statement; SQLite applies rows in order.
+    savepoint = session.get_bind(model).dialect.name == "postgresql"
+
+    try:
+        for chunk, stmt in _bulk_insert_statements(session, model, rows, fields, make_stmt):
+            if upsert:
+                written |= await _upsert_chunk(session, chunk, stmt, make_stmt, savepoint=savepoint)
+            else:
+                result = cast("CursorResult[Any]", await session.execute(stmt))
+                affected += result.rowcount
+        await session.flush()
+    except MutationError:
+        raise
+    except IntegrityError as e:
+        logger.warning("%s(%s) — integrity error: %s", func_name, model_name, e.orig)
+        msg = f"Integrity error inserting {model_name}: {e.orig}"
+        raise MutationError(msg) from e
+    except Exception as e:
+        logger.exception("%s(%s) — unexpected error", func_name, model_name)
+        msg = f"Database error inserting {model_name}: {e}"
+        raise DatabaseError(msg) from e
+
+    if upsert:
+        affected = len(written)
+    logger.debug(
+        "%s(%s) → %d of %d rows, %.3fs", func_name, model_name, affected, len(rows), time.perf_counter() - start
+    )
+    return affected
+
+
+async def insert_objects_ignore_conflicts[T: SQLModel](
+    session: AsyncSession,
+    model: type[T],
+    rows: Sequence[Mapping[str, Any]],
+    *,
+    conflict_columns: Sequence[str],
+) -> int:
+    """
+    Insert rows, skipping duplicates by the unique key ``conflict_columns``.
+
+    Executes ``INSERT ... ON CONFLICT (...) DO NOTHING``. Replayed and concurrent batches
+    are protected by the database unique constraint. No ORM objects are created or
+    refreshed. Large inputs are split by the actual compiled bind parameters, including
+    Python defaults of omitted columns and SQL expressions. Does not commit; the caller
+    rolls back the whole operation, including after a failure in a later chunk.
+
+    Values are passed without Pydantic validation and must already have the right types.
+    Omitted columns get their column defaults: ``Field(default=...)``/``default_factory``
+    of a plain field and ``server_default``. ``Field(default)`` of a field with
+    ``sa_column`` does not reach the column.
+
+    Args:
+        session: Async SQLAlchemy session (caller manages transaction).
+        model: SQLModel class to insert into.
+        rows: ``field → value`` mappings, all with the same keys.
+        conflict_columns: Columns of the unique index or constraint that defines a duplicate.
+
+    Returns:
+        Number of inserted rows, excluding skipped duplicates.
+
+    Raises:
+        MutationError: Invalid fields, conflict key or row keys, unsupported dialect,
+            a single row over the bind-parameter limit, or another constraint violation.
+        DatabaseError: On unexpected database errors.
+
+    """
+    if not rows:
+        return 0
+
+    func_name = "insert_objects_ignore_conflicts"
+    insert_fn, columns, fields = _prepare_bulk_insert(func_name, session, model, rows, conflict_columns)
+    index_elements = [columns[name] for name in conflict_columns]
+
+    def make_stmt(chunk: list[dict[str, Any]]) -> Any:
+        return insert_fn(model).values(chunk).on_conflict_do_nothing(index_elements=index_elements)
+
+    return await _execute_bulk_insert(func_name, session, model, rows, fields, make_stmt)
+
+
+async def upsert_objects[T: SQLModel](
+    session: AsyncSession,
+    model: type[T],
+    rows: Sequence[Mapping[str, Any]],
+    *,
+    conflict_columns: Sequence[str],
+    update_columns: Sequence[str] | None = None,
+    set_extra: Mapping[str, Any] | None = None,
+) -> int:
+    """
+    Insert new rows and update existing ones by the key ``conflict_columns``.
+
+    Executes ``INSERT ... ON CONFLICT (...) DO UPDATE SET ...`` without creating or
+    refreshing ORM objects. Chunks are split by the actual bind-parameter count, including
+    defaults and ``set_extra``. Does not commit; the caller owns the rollback of all chunks.
+
+    If a key repeats in the input, the last row wins. The database decides what is a repeat
+    by the column type (``NUMERIC`` rounding, ``citext``, ``NULLS NOT DISTINCT``), not Python
+    equality. On PostgreSQL a multi-row chunk runs in a SAVEPOINT (``begin_nested`` flushes
+    the session first); the database rejects a chunk with a repeated key, and it is re-run
+    row by row. All key columns must be present in rows. Replaying the operation creates no
+    duplicates, but ``set_extra`` expressions run again: ``hits + 1`` is not idempotent.
+
+    Values are passed without Pydantic validation. Defaults of omitted columns apply on
+    insert; on conflict those columns keep their values unless set in ``set_extra``.
+    ``Column.onupdate`` does not fire on conflict — pass it in ``set_extra``.
+    Objects already loaded into the session are not refreshed.
+
+    Args:
+        session: Async SQLAlchemy session (caller manages transaction).
+        model: SQLModel class to insert into or update.
+        rows: ``field → value`` mappings, all with the same keys.
+        conflict_columns: Columns of the unique index or constraint.
+        update_columns: Row columns updated on conflict. Defaults to all input fields
+            except the conflict key.
+        set_extra: Extra values or SQL expressions applied only on update,
+            e.g. ``{"updated_at": func.now()}``.
+
+    Returns:
+        Number of distinct table rows (by primary key from RETURNING) inserted or updated;
+        a key repeated in the input counts once.
+
+    Raises:
+        MutationError: Invalid input, a single row over the bind-parameter limit,
+            unsupported dialect, or a database constraint violation.
+        DatabaseError: On unexpected database errors.
+
+    """
+    if not rows:
+        return 0
+
+    func_name = "upsert_objects"
+    model_name = model.__name__
+    insert_fn, columns, fields = _prepare_bulk_insert(func_name, session, model, rows, conflict_columns)
+
+    if any(name not in fields for name in conflict_columns):
+        msg = f"{func_name}({model_name}): conflict_columns must be present in rows"
+        raise MutationError(msg)
+
+    to_update = (
+        list(update_columns) if update_columns is not None else [f for f in rows[0] if f not in conflict_columns]
+    )
+    extra = dict(set_extra or {})
+    for name in extra:
+        if name not in columns:
+            msg = f"Model {model_name} has no field {name}"
+            raise MutationError(msg)
+    for name in to_update:
+        if name not in fields or name in conflict_columns:
+            msg = f"{func_name}({model_name}): update column {name} must be a row field outside conflict_columns"
+            raise MutationError(msg)
+    for name in extra:
+        if name in conflict_columns or name in to_update:
+            msg = f"{func_name}({model_name}): set_extra column {name} is a conflict or update column"
+            raise MutationError(msg)
+    if not to_update and not extra:
+        msg = f"{func_name}({model_name}): nothing to update, use insert_objects_ignore_conflicts"
+        raise MutationError(msg)
+
+    index_elements = [columns[name] for name in conflict_columns]
+    extra_set = {columns[name]: value for name, value in extra.items()}
+    primary_key = inspect(model).primary_key
+
+    def make_stmt(chunk: list[dict[str, Any]]) -> Any:
+        stmt = insert_fn(model).values(chunk)
+        # Keys are columns, not field names: sa_column may use a different database name.
+        set_ = {columns[name]: stmt.excluded[columns[name].key] for name in to_update} | extra_set
+        return stmt.on_conflict_do_update(index_elements=index_elements, set_=set_).returning(*primary_key)
+
+    return await _execute_bulk_insert(func_name, session, model, rows, fields, make_stmt, upsert=True)
+
+
 async def check_for_related_records[T: SQLModel](
     session: AsyncSession,
     model: type[T],

{sqlmodel_object_helpers-0.0.9 → sqlmodel_object_helpers-0.0.10}/src/sqlmodel_object_helpers/standalone.py

@@ -26,6 +26,7 @@ Migration to DI sessions later::
 """
 
 import logging
+from collections.abc import Mapping, Sequence
 from typing import Any, Literal, overload
 
 from pydantic import BaseModel
@@ -387,6 +388,39 @@ async def delete_objects[T: SQLModel](
         return await _m.delete_objects(s, model, filters, logical_operator)
 
 
+async def insert_objects_ignore_conflicts[T: SQLModel](
+    model: type[T],
+    rows: Sequence[Mapping[str, Any]],
+    *,
+    conflict_columns: Sequence[str],
+) -> int:
+    """Insert rows, skipping duplicates. Auto-creates session, commits on success."""
+    logger.debug("standalone.insert_objects_ignore_conflicts(%s, rows=%d)", model.__name__, len(rows))
+    async with auto_session() as s:
+        return await _m.insert_objects_ignore_conflicts(s, model, rows, conflict_columns=conflict_columns)
+
+
+async def upsert_objects[T: SQLModel](
+    model: type[T],
+    rows: Sequence[Mapping[str, Any]],
+    *,
+    conflict_columns: Sequence[str],
+    update_columns: Sequence[str] | None = None,
+    set_extra: Mapping[str, Any] | None = None,
+) -> int:
+    """Insert or update rows. Auto-creates session, commits on success."""
+    logger.debug("standalone.upsert_objects(%s, rows=%d)", model.__name__, len(rows))
+    async with auto_session() as s:
+        return await _m.upsert_objects(
+            s,
+            model,
+            rows,
+            conflict_columns=conflict_columns,
+            update_columns=update_columns,
+            set_extra=set_extra,
+        )
+
+
 async def check_for_related_records[T: SQLModel](
     model: type[T],
     pk: dict[str, Any],