sqlmodel-object-helpers 0.0.10__tar.gz → 0.0.11__tar.gz

{sqlmodel_object_helpers-0.0.10 → sqlmodel_object_helpers-0.0.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: sqlmodel-object-helpers
-Version: 0.0.10
+Version: 0.0.11
 Summary: Generic async query helpers for SQLModel: filtering, eager loading, pagination
 Project-URL: Homepage, https://github.com/itstandart/sqlmodel-object-helpers
 Project-URL: Repository, https://github.com/itstandart/sqlmodel-object-helpers
@@ -43,8 +43,8 @@ Generic async query helpers for [SQLModel](https://sqlmodel.tiangolo.com/): filt
 - **Pagination** - Page/per\_page with total count and configurable max\_per\_page limit
 - **Projections** - Select specific columns across joins with dot-notation and SQL aliases
 - **CRUD Mutations** - `add_object`, `update_object`, `delete_object` plus bulk `update_objects`, `delete_objects` with flush-only semantics for composable transactions
-- **Idempotent Bulk Insert** - `insert_objects_ignore_conflicts`: one `INSERT ... ON CONFLICT DO NOTHING` per chunk (PostgreSQL, SQLite), safe to replay
-- **Bulk Upsert** - `upsert_objects`: one `INSERT ... ON CONFLICT DO UPDATE` per chunk (PostgreSQL, SQLite); update expressions run on every call
+- **Idempotent Bulk Insert** - `insert_objects_ignore_conflicts`: `INSERT ... ON CONFLICT DO NOTHING` as executemany of one single-row statement (PostgreSQL, SQLite), safe to replay
+- **Bulk Upsert** - `upsert_objects`: `INSERT ... ON CONFLICT DO UPDATE` as executemany of one single-row statement (PostgreSQL, SQLite); update expressions run on every call
 - **Count & Exists** - `count_objects` (single `SELECT count(*)`) and `exists_object` (`SELECT EXISTS(...)`) without loading data
 - **Row Locking** - `for_update` parameter on `get_object` for `SELECT ... FOR UPDATE`
 - **Time Filtering** - `TimeFilter` for `created_at`/`updated_at` range filtering with half-open interval semantics
@@ -465,9 +465,9 @@ inserted = await soh.insert_objects_ignore_conflicts(
 # inserted -> rows actually written; duplicates by source_id are skipped and not counted
 ```
 
-Replaying the same batch (or two concurrent identical batches) is safe, unlike `add_objects`, where one duplicate fails the whole batch. All rows must have the same keys. Rows go to the database as given (no Pydantic validation or coercion -- pass already-typed values); omitted columns get their column defaults: `Field(default=...)` / `default_factory` of a plain field and `server_default` (SQLModel does not copy `Field(default)` into an explicit `sa_column`). Large inputs are split using the actual compiled bind-parameter count, including omitted Python defaults and SQL expressions. One row exceeding the parameter budget raises `MutationError`. PostgreSQL and SQLite only (`MutationError` on other dialects).
+Replaying the same batch (or two concurrent identical batches) is safe, unlike `add_objects`, where one duplicate fails the whole batch. All rows must have the same keys. Rows go to the database as given (no Pydantic validation or coercion -- pass already-typed values); omitted columns get their column defaults: `Field(default=...)` / `default_factory` of a plain field and `server_default` (SQLModel does not copy `Field(default)` into an explicit `sa_column`). Rows go as executemany of one single-row statement: SQLAlchemy [insertmanyvalues](https://docs.sqlalchemy.org/en/20/core/connections.html#engine-insertmanyvalues) renders up to `insertmanyvalues_page_size` rows (1000 by default) per INSERT and keeps each statement within the dialect parameter limit, omitted Python defaults included. A row with an SQL expression value (`func.now()`, `literal(...)`) cannot be an executemany parameter set, so it is inserted by its own statement, in input order. An engine created with `use_insertmanyvalues=False` cannot return rows from an executemany, so there every row runs as its own statement. A Core INSERT does not autoflush, so pending session changes are flushed first. PostgreSQL and SQLite only (`MutationError` on other dialects).
 
-The caller owns the transaction. If a later chunk fails, roll back the transaction to discard earlier chunks too. The helper does not commit or perform its own rollback. Standalone wrappers manage commit/rollback automatically.
+The caller owns the transaction. If a later batch fails, roll back the transaction to discard earlier batches too. The helper does not commit or perform its own rollback. Standalone wrappers manage commit/rollback automatically.
 
 ### upsert_objects
 
@@ -487,10 +487,19 @@ written = await soh.upsert_objects(
 # written -> distinct table rows inserted or updated, counted by primary key from RETURNING
 ```
 
-If a key repeats inside `rows`, the last row wins and the row is counted once. Repeats are decided by the database for the column type (`NUMERIC` rounding, `citext`, `NULLS NOT DISTINCT`), not by Python equality: distinct values such as two instants of a DST fold stay distinct, and key values do not have to be hashable (`ARRAY`). On PostgreSQL a chunk of several rows runs inside a SAVEPOINT (`session.begin_nested()` flushes pending session changes first); a chunk the database rejects for a repeated key is retried row by row, which is slower. SQLite applies repeated keys in order within one statement. Every `conflict_columns` field must be present in each row. `update_columns` must be a subset of the row fields outside `conflict_columns`; `set_extra` takes model fields (value or SQL expression) that are in neither. When nothing is left to update, use `insert_objects_ignore_conflicts` instead (`MutationError`). `Column.onupdate` does not fire on conflict -- pass it in `set_extra`. Objects already loaded in the session are not refreshed. A conditional update (`WHERE`) or `RETURNING` is not covered -- build the statement with SQLAlchemy for that. Other rules as in `insert_objects_ignore_conflicts`: same keys in all rows, no Pydantic validation, actual parameter-based chunking, caller-managed transaction, PostgreSQL and SQLite only.
+If a key repeats inside `rows`, the last row wins and the row is counted once, with the result of row-by-row execution. PostgreSQL rejects a statement that hits one key twice, so on PostgreSQL:
+
+- a key value equal in Python to an earlier one starts the next statement, in input order (rows of other keys may depend on earlier rows through another unique constraint): one INSERT per page plus one per repeat. `ARRAY` and `JSONB` values are compared by content;
+- each page of up to `insertmanyvalues_page_size` rows of a multi-row call runs in a SAVEPOINT, also when its rows go one by one (a row with an SQL expression, an expanding `IN` in `set_extra`, `use_insertmanyvalues=False`), since only the database decides what is a repeat: the column type (`NUMERIC` rounding, `citext`, the `VARCHAR(n)` cut), the collation of the column or of the unique index, a `BEFORE INSERT` trigger that changes the key, `NULLS NOT DISTINCT`. A page the database rejects for a repeated key is rolled back to the SAVEPOINT and re-run row by row in a new SAVEPOINT, so an error during the retry also rolls back that entire page; distinct values such as two instants of a DST fold stay distinct. Each SAVEPOINT costs two round trips: one per page, plus one when a page is retried; a single-row call goes without it. A released SAVEPOINT keeps its subtransaction until commit: more than 64 pages in one transaction (over 64000 rows at the default page size) overflow the backend subtransaction cache, and until commit other sessions check this transaction's rows through `pg_subtrans`, which is slower under load. Split such a load into several transactions.
+
+The distinct primary-key count preserves JSON types: `true` and `1` (also `false` and `0`) are different keys, including inside objects and arrays. Numerically equal JSON numbers such as `1` and `1.0` remain one key, and object field order does not affect equality. Repeated keys across pages are counted once.
+
+SQLite applies repeated keys in order within one statement. Every `conflict_columns` field must be present in each row. `update_columns` must be a subset of the row fields outside `conflict_columns`; `set_extra` takes model fields (value or SQL expression) that are in neither; a `None` value becomes SQL `NULL`. An expanding `IN` or `literal_execute` parameter in `set_extra` cannot be used with executemany, so each row then runs as its own statement. When nothing is left to update, use `insert_objects_ignore_conflicts` instead (`MutationError`). `Column.onupdate` does not fire on conflict -- pass it in `set_extra`. Objects already loaded in the session are not refreshed. A conditional update (`WHERE`) or `RETURNING` is not covered -- build the statement with SQLAlchemy for that. Other rules as in `insert_objects_ignore_conflicts`: same keys in all rows, no Pydantic validation, executemany batching, caller-managed transaction, PostgreSQL and SQLite only.
 
 Repeating an upsert does not create duplicate keys, but it repeats the update: expressions such as `hits + 1` and `func.now()` run again. NULL conflict-key values follow the index: an ordinary UNIQUE keeps them distinct, `NULLS NOT DISTINCT` treats them as one key.
 
+Other database errors are not retried. On PostgreSQL an error in a multi-row upsert is rolled back to the SAVEPOINT of its page; an error in `insert_objects_ignore_conflicts`, in a single-row upsert or on SQLite aborts the transaction, as with a hand-written INSERT. Earlier pages stay written either way: roll back after `MutationError` / `DatabaseError`, or wrap the call in `session.begin_nested()` to keep the earlier work.
+
 ### check_for_related_records
 
 Pre-deletion check that inspects all ONETOMANY relationships:
@@ -952,7 +961,18 @@ This project follows [Semantic Versioning](https://semver.org/) (MAJOR.MINOR.PAT
 
 ## Changelog
 
-### 0.0.10 — unreleased
+### 0.0.11 — unreleased
+
+- **Performance** — `insert_objects_ignore_conflicts` and `upsert_objects` execute one single-row statement as executemany instead of compiling a multi-row `VALUES` statement per chunk; SQLAlchemy insertmanyvalues renders the rows into multi-row pages. No statement cache is involved: SQLAlchemy does not cache PostgreSQL `INSERT ... ON CONFLICT`. On PostgreSQL 18 with asyncpg, inserting 1000 rows of an 18-column table takes about 27 ms instead of 610 ms (20000 rows: 0.56 s instead of 13 s), on par with a hand-written bulk insert; upserting 1000 rows of a 6-column table takes about 12 ms instead of 165 ms (10000 rows: 0.11 s instead of 1.65 s), SAVEPOINTs included. Server-side execution time is unchanged.
+- `upsert_objects` on PostgreSQL: a key repeated in Python terms starts the next statement, in input order, instead of re-running the batch row by row. Each page of a multi-row call keeps the 0.0.10 SAVEPOINT, also when its rows go one by one: a repeat only the database sees (column type, collation of the column or of the unique index, a `BEFORE INSERT` trigger) is retried row by row in a new SAVEPOINT. Another error, including during that retry, rolls back the page without aborting the caller's transaction. A single-row call goes without a SAVEPOINT.
+- A `JSONB` primary key works in both operations: `insert_objects_ignore_conflicts` counts the returned rows, and the upsert count compares `JSONB` / `ARRAY` primary keys by content (the 0.0.10 upsert failed with `DatabaseError`). JSON booleans are distinct from numbers, both as scalar keys and inside objects or arrays; numerically equal JSON numbers remain one key.
+- An engine with `use_insertmanyvalues=False` runs every row as its own statement: an executemany cannot return rows there.
+- A row with SQL expression values is inserted by its own statement in input order; a `None` in `set_extra` is sent as `NULL`, which keeps the upsert batched; an expanding `IN` in `set_extra` makes the upsert run row by row.
+- Pending session changes are flushed before the insert on every dialect: a Core INSERT does not autoflush.
+- A single row no longer fails with `MutationError` for exceeding a parameter budget: SQLAlchemy keeps every statement within the dialect limit.
+- Tested on PostgreSQL 13.23, 14.24, 15.19, 16.15, 17.7, 18.1 and 18.4 (on 13 and 14 without the `NULLS NOT DISTINCT` test, a PostgreSQL 15 feature) and with SQLAlchemy 2.0.46, 2.0.52 and 2.1.6.
+
+### 0.0.10 — 2026-10-26
 
 - `insert_objects_ignore_conflicts` and `upsert_objects`: bulk insert with explicit unique-key conflict handling, inserted/updated counts, and standalone wrappers.
 - Chunk sizes account for actual SQL parameters, including omitted Python defaults, repeated positional parameters, expanding `IN` expressions, and update expressions. Tested with SQLAlchemy 2.0.46, 2.0.52 and 2.1.6.

{sqlmodel_object_helpers-0.0.10 → sqlmodel_object_helpers-0.0.11}/README.md

@@ -15,8 +15,8 @@ Generic async query helpers for [SQLModel](https://sqlmodel.tiangolo.com/): filt
 - **Pagination** - Page/per\_page with total count and configurable max\_per\_page limit
 - **Projections** - Select specific columns across joins with dot-notation and SQL aliases
 - **CRUD Mutations** - `add_object`, `update_object`, `delete_object` plus bulk `update_objects`, `delete_objects` with flush-only semantics for composable transactions
-- **Idempotent Bulk Insert** - `insert_objects_ignore_conflicts`: one `INSERT ... ON CONFLICT DO NOTHING` per chunk (PostgreSQL, SQLite), safe to replay
-- **Bulk Upsert** - `upsert_objects`: one `INSERT ... ON CONFLICT DO UPDATE` per chunk (PostgreSQL, SQLite); update expressions run on every call
+- **Idempotent Bulk Insert** - `insert_objects_ignore_conflicts`: `INSERT ... ON CONFLICT DO NOTHING` as executemany of one single-row statement (PostgreSQL, SQLite), safe to replay
+- **Bulk Upsert** - `upsert_objects`: `INSERT ... ON CONFLICT DO UPDATE` as executemany of one single-row statement (PostgreSQL, SQLite); update expressions run on every call
 - **Count & Exists** - `count_objects` (single `SELECT count(*)`) and `exists_object` (`SELECT EXISTS(...)`) without loading data
 - **Row Locking** - `for_update` parameter on `get_object` for `SELECT ... FOR UPDATE`
 - **Time Filtering** - `TimeFilter` for `created_at`/`updated_at` range filtering with half-open interval semantics
@@ -437,9 +437,9 @@ inserted = await soh.insert_objects_ignore_conflicts(
 # inserted -> rows actually written; duplicates by source_id are skipped and not counted
 ```
 
-Replaying the same batch (or two concurrent identical batches) is safe, unlike `add_objects`, where one duplicate fails the whole batch. All rows must have the same keys. Rows go to the database as given (no Pydantic validation or coercion -- pass already-typed values); omitted columns get their column defaults: `Field(default=...)` / `default_factory` of a plain field and `server_default` (SQLModel does not copy `Field(default)` into an explicit `sa_column`). Large inputs are split using the actual compiled bind-parameter count, including omitted Python defaults and SQL expressions. One row exceeding the parameter budget raises `MutationError`. PostgreSQL and SQLite only (`MutationError` on other dialects).
+Replaying the same batch (or two concurrent identical batches) is safe, unlike `add_objects`, where one duplicate fails the whole batch. All rows must have the same keys. Rows go to the database as given (no Pydantic validation or coercion -- pass already-typed values); omitted columns get their column defaults: `Field(default=...)` / `default_factory` of a plain field and `server_default` (SQLModel does not copy `Field(default)` into an explicit `sa_column`). Rows go as executemany of one single-row statement: SQLAlchemy [insertmanyvalues](https://docs.sqlalchemy.org/en/20/core/connections.html#engine-insertmanyvalues) renders up to `insertmanyvalues_page_size` rows (1000 by default) per INSERT and keeps each statement within the dialect parameter limit, omitted Python defaults included. A row with an SQL expression value (`func.now()`, `literal(...)`) cannot be an executemany parameter set, so it is inserted by its own statement, in input order. An engine created with `use_insertmanyvalues=False` cannot return rows from an executemany, so there every row runs as its own statement. A Core INSERT does not autoflush, so pending session changes are flushed first. PostgreSQL and SQLite only (`MutationError` on other dialects).
 
-The caller owns the transaction. If a later chunk fails, roll back the transaction to discard earlier chunks too. The helper does not commit or perform its own rollback. Standalone wrappers manage commit/rollback automatically.
+The caller owns the transaction. If a later batch fails, roll back the transaction to discard earlier batches too. The helper does not commit or perform its own rollback. Standalone wrappers manage commit/rollback automatically.
 
 ### upsert_objects
 
@@ -459,10 +459,19 @@ written = await soh.upsert_objects(
 # written -> distinct table rows inserted or updated, counted by primary key from RETURNING
 ```
 
-If a key repeats inside `rows`, the last row wins and the row is counted once. Repeats are decided by the database for the column type (`NUMERIC` rounding, `citext`, `NULLS NOT DISTINCT`), not by Python equality: distinct values such as two instants of a DST fold stay distinct, and key values do not have to be hashable (`ARRAY`). On PostgreSQL a chunk of several rows runs inside a SAVEPOINT (`session.begin_nested()` flushes pending session changes first); a chunk the database rejects for a repeated key is retried row by row, which is slower. SQLite applies repeated keys in order within one statement. Every `conflict_columns` field must be present in each row. `update_columns` must be a subset of the row fields outside `conflict_columns`; `set_extra` takes model fields (value or SQL expression) that are in neither. When nothing is left to update, use `insert_objects_ignore_conflicts` instead (`MutationError`). `Column.onupdate` does not fire on conflict -- pass it in `set_extra`. Objects already loaded in the session are not refreshed. A conditional update (`WHERE`) or `RETURNING` is not covered -- build the statement with SQLAlchemy for that. Other rules as in `insert_objects_ignore_conflicts`: same keys in all rows, no Pydantic validation, actual parameter-based chunking, caller-managed transaction, PostgreSQL and SQLite only.
+If a key repeats inside `rows`, the last row wins and the row is counted once, with the result of row-by-row execution. PostgreSQL rejects a statement that hits one key twice, so on PostgreSQL:
+
+- a key value equal in Python to an earlier one starts the next statement, in input order (rows of other keys may depend on earlier rows through another unique constraint): one INSERT per page plus one per repeat. `ARRAY` and `JSONB` values are compared by content;
+- each page of up to `insertmanyvalues_page_size` rows of a multi-row call runs in a SAVEPOINT, also when its rows go one by one (a row with an SQL expression, an expanding `IN` in `set_extra`, `use_insertmanyvalues=False`), since only the database decides what is a repeat: the column type (`NUMERIC` rounding, `citext`, the `VARCHAR(n)` cut), the collation of the column or of the unique index, a `BEFORE INSERT` trigger that changes the key, `NULLS NOT DISTINCT`. A page the database rejects for a repeated key is rolled back to the SAVEPOINT and re-run row by row in a new SAVEPOINT, so an error during the retry also rolls back that entire page; distinct values such as two instants of a DST fold stay distinct. Each SAVEPOINT costs two round trips: one per page, plus one when a page is retried; a single-row call goes without it. A released SAVEPOINT keeps its subtransaction until commit: more than 64 pages in one transaction (over 64000 rows at the default page size) overflow the backend subtransaction cache, and until commit other sessions check this transaction's rows through `pg_subtrans`, which is slower under load. Split such a load into several transactions.
+
+The distinct primary-key count preserves JSON types: `true` and `1` (also `false` and `0`) are different keys, including inside objects and arrays. Numerically equal JSON numbers such as `1` and `1.0` remain one key, and object field order does not affect equality. Repeated keys across pages are counted once.
+
+SQLite applies repeated keys in order within one statement. Every `conflict_columns` field must be present in each row. `update_columns` must be a subset of the row fields outside `conflict_columns`; `set_extra` takes model fields (value or SQL expression) that are in neither; a `None` value becomes SQL `NULL`. An expanding `IN` or `literal_execute` parameter in `set_extra` cannot be used with executemany, so each row then runs as its own statement. When nothing is left to update, use `insert_objects_ignore_conflicts` instead (`MutationError`). `Column.onupdate` does not fire on conflict -- pass it in `set_extra`. Objects already loaded in the session are not refreshed. A conditional update (`WHERE`) or `RETURNING` is not covered -- build the statement with SQLAlchemy for that. Other rules as in `insert_objects_ignore_conflicts`: same keys in all rows, no Pydantic validation, executemany batching, caller-managed transaction, PostgreSQL and SQLite only.
 
 Repeating an upsert does not create duplicate keys, but it repeats the update: expressions such as `hits + 1` and `func.now()` run again. NULL conflict-key values follow the index: an ordinary UNIQUE keeps them distinct, `NULLS NOT DISTINCT` treats them as one key.
 
+Other database errors are not retried. On PostgreSQL an error in a multi-row upsert is rolled back to the SAVEPOINT of its page; an error in `insert_objects_ignore_conflicts`, in a single-row upsert or on SQLite aborts the transaction, as with a hand-written INSERT. Earlier pages stay written either way: roll back after `MutationError` / `DatabaseError`, or wrap the call in `session.begin_nested()` to keep the earlier work.
+
 ### check_for_related_records
 
 Pre-deletion check that inspects all ONETOMANY relationships:
@@ -924,7 +933,18 @@ This project follows [Semantic Versioning](https://semver.org/) (MAJOR.MINOR.PAT
 
 ## Changelog
 
-### 0.0.10 — unreleased
+### 0.0.11 — unreleased
+
+- **Performance** — `insert_objects_ignore_conflicts` and `upsert_objects` execute one single-row statement as executemany instead of compiling a multi-row `VALUES` statement per chunk; SQLAlchemy insertmanyvalues renders the rows into multi-row pages. No statement cache is involved: SQLAlchemy does not cache PostgreSQL `INSERT ... ON CONFLICT`. On PostgreSQL 18 with asyncpg, inserting 1000 rows of an 18-column table takes about 27 ms instead of 610 ms (20000 rows: 0.56 s instead of 13 s), on par with a hand-written bulk insert; upserting 1000 rows of a 6-column table takes about 12 ms instead of 165 ms (10000 rows: 0.11 s instead of 1.65 s), SAVEPOINTs included. Server-side execution time is unchanged.
+- `upsert_objects` on PostgreSQL: a key repeated in Python terms starts the next statement, in input order, instead of re-running the batch row by row. Each page of a multi-row call keeps the 0.0.10 SAVEPOINT, also when its rows go one by one: a repeat only the database sees (column type, collation of the column or of the unique index, a `BEFORE INSERT` trigger) is retried row by row in a new SAVEPOINT. Another error, including during that retry, rolls back the page without aborting the caller's transaction. A single-row call goes without a SAVEPOINT.
+- A `JSONB` primary key works in both operations: `insert_objects_ignore_conflicts` counts the returned rows, and the upsert count compares `JSONB` / `ARRAY` primary keys by content (the 0.0.10 upsert failed with `DatabaseError`). JSON booleans are distinct from numbers, both as scalar keys and inside objects or arrays; numerically equal JSON numbers remain one key.
+- An engine with `use_insertmanyvalues=False` runs every row as its own statement: an executemany cannot return rows there.
+- A row with SQL expression values is inserted by its own statement in input order; a `None` in `set_extra` is sent as `NULL`, which keeps the upsert batched; an expanding `IN` in `set_extra` makes the upsert run row by row.
+- Pending session changes are flushed before the insert on every dialect: a Core INSERT does not autoflush.
+- A single row no longer fails with `MutationError` for exceeding a parameter budget: SQLAlchemy keeps every statement within the dialect limit.
+- Tested on PostgreSQL 13.23, 14.24, 15.19, 16.15, 17.7, 18.1 and 18.4 (on 13 and 14 without the `NULLS NOT DISTINCT` test, a PostgreSQL 15 feature) and with SQLAlchemy 2.0.46, 2.0.52 and 2.1.6.
+
+### 0.0.10 — 2026-10-26
 
 - `insert_objects_ignore_conflicts` and `upsert_objects`: bulk insert with explicit unique-key conflict handling, inserted/updated counts, and standalone wrappers.
 - Chunk sizes account for actual SQL parameters, including omitted Python defaults, repeated positional parameters, expanding `IN` expressions, and update expressions. Tested with SQLAlchemy 2.0.46, 2.0.52 and 2.1.6.

{sqlmodel_object_helpers-0.0.10 → sqlmodel_object_helpers-0.0.11}/pyproject.toml

@@ -53,5 +53,5 @@ dev = [
     "aiosqlite==0.22.1",
 ]
 postgres = [
-    "asyncpg==0.31.0",
+    "asyncpg==0.32.0",
 ]

{sqlmodel_object_helpers-0.0.10 → sqlmodel_object_helpers-0.0.11}/src/sqlmodel_object_helpers/__init__.py

@@ -1,6 +1,6 @@
 """sqlmodel-object-helpers — reusable query helpers for SQLModel projects."""
 
-__version__ = "0.0.10"
+__version__ = "0.0.11"
 
 from .constants import QueryHelperSettings, settings
 from .dynamic_meta import (

{sqlmodel_object_helpers-0.0.10 → sqlmodel_object_helpers-0.0.11}/src/sqlmodel_object_helpers/mutations.py

@@ -18,10 +18,10 @@ composed into a single atomic transaction at the call-site::
 
 import logging
 import time
-from collections.abc import Callable, Iterator, Mapping, Sequence
+from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
 from typing import Any, cast
 
-from sqlalchemy import CursorResult, inspect
+from sqlalchemy import BindParameter, ClauseElement, CursorResult, inspect, null
 from sqlalchemy import delete as sa_delete
 from sqlalchemy import update as sa_update
 from sqlalchemy.dialects.postgresql import insert as pg_insert
@@ -29,6 +29,7 @@ from sqlalchemy.dialects.sqlite import insert as sqlite_insert
 from sqlalchemy.exc import DBAPIError, IntegrityError
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy.orm import ONETOMANY
+from sqlalchemy.sql import visitors
 from sqlmodel import SQLModel, and_, select
 
 from .exceptions import DatabaseError, MutationError, ObjectNotFoundError
@@ -41,9 +42,6 @@ logger = logging.getLogger("sqlmodel_object_helpers")
 # Dialects that support INSERT ... ON CONFLICT.
 _ON_CONFLICT_INSERT: dict[str, Callable[..., Any]] = {"postgresql": pg_insert, "sqlite": sqlite_insert}
 
-# Headroom below the bind-parameter limits of asyncpg (32767) and SQLite (32766).
-_MAX_BIND_PARAMS = 30_000
-
 # PostgreSQL SQLSTATE: ON CONFLICT DO UPDATE affected the same row twice in one statement.
 _CARDINALITY_VIOLATION = "21000"
 
@@ -466,46 +464,99 @@ def _prepare_bulk_insert(
     return insert_fn, columns, fields
 
 
-def _bulk_insert_statements(
-    session: AsyncSession,
-    model: type[SQLModel],
-    rows: Sequence[Mapping[str, Any]],
-    fields: set[str],
-    make_stmt: Callable[[list[dict[str, Any]]], Any],
-) -> Iterator[tuple[list[dict[str, Any]], Any]]:
-    """Chunk rows by the compiled bind-parameter count, including defaults and update expressions."""
-    dialect = session.get_bind(model).dialect
-    chunk_size = max(1, _MAX_BIND_PARAMS // len(fields))
-    offset = 0
-    while offset < len(rows):
-        chunk = [dict(row) for row in rows[offset : offset + chunk_size]]
-        while True:
-            stmt = make_stmt(chunk)
-            compiled = stmt.compile(dialect=dialect)
-            expanded = compiled.construct_expanded_state()
-            # Positional dialects may repeat one named parameter several times.
-            count = len(expanded.positiontup) if expanded.positiontup is not None else len(expanded.parameters)
-            if count <= _MAX_BIND_PARAMS:
-                break
-            if len(chunk) == 1:
-                msg = f"A single row requires {count} bind parameters; limit is {_MAX_BIND_PARAMS}"
-                raise MutationError(msg)
-            chunk_size = max(1, min(len(chunk) - 1, len(chunk) * _MAX_BIND_PARAMS // count))
-            chunk = chunk[:chunk_size]
-        yield chunk, stmt
-        offset += len(chunk)
+def _bind_rows(columns: Any, fields: set[str], rows: Sequence[Mapping[str, Any]]) -> Sequence[Mapping[str, Any]]:
+    """Rename row fields to column keys: ``sa_column`` may give a column another name."""
+    keys = {name: columns[name].key for name in fields}
+    if all(name == key for name, key in keys.items()):
+        return rows
+    return [{keys[name]: value for name, value in row.items()} for row in rows]
+
+
+def _is_sql_type(value_type: type) -> bool:
+    """SQL expression or ORM attribute: such a value cannot be an executemany parameter."""
+    return issubclass(value_type, ClauseElement) or hasattr(value_type, "__clause_element__")
+
+
+def _batches(rows: Sequence[Mapping[str, Any]], size: int) -> Iterator[tuple[list[Mapping[str, Any]], bool]]:
+    """Split rows into executemany batches in input order; a row with SQL expressions goes alone (``True``)."""
+    if not any(_is_sql_type(value_type) for value_type in {type(value) for row in rows for value in row.values()}):
+        for offset in range(0, len(rows), size):
+            yield list(rows[offset : offset + size]), False
+        return
+    batch: list[Mapping[str, Any]] = []
+    for row in rows:
+        if any(_is_sql_type(type(value)) for value in row.values()):
+            if batch:
+                yield batch, False
+                batch = []
+            yield [row], True
+            continue
+        batch.append(row)
+        if len(batch) == size:
+            yield batch, False
+            batch = []
+    if batch:
+        yield batch, False
+
+
+def _has_post_compile_params(values: Iterable[Any]) -> bool:
+    """Expanding ``IN`` or ``literal_execute`` parameters, which executemany rejects."""
+    return any(
+        isinstance(node, BindParameter) and (node.expanding or node.literal_execute)
+        for value in values
+        if isinstance(value, ClauseElement)
+        for node in visitors.iterate(value)
+    )
 
 
 def _hashable(value: Any) -> Any:
-    """Make a RETURNING value hashable: ARRAY values come back as lists."""
+    """Make JSON/ARRAY keys hashable while keeping booleans distinct from numbers."""
+    if isinstance(value, bool):
+        return bool, value
+    if isinstance(value, dict):
+        return frozenset((key, _hashable(item)) for key, item in value.items())
     if isinstance(value, list | tuple):
         return tuple(_hashable(item) for item in value)
     return value
 
 
-def _row_keys(result: Any) -> set[Any]:
-    """Primary keys of the rows returned by RETURNING."""
-    return {_hashable(tuple(row)) for row in result}
+def _distinct_count(returned: list[Any]) -> int:
+    """Count distinct RETURNING primary keys, normalizing scalar JSON booleans and numbers too."""
+    hashable = [_hashable(tuple(row)) for row in returned]
+    try:
+        return len(set(hashable))
+    except TypeError:  # Compare unhashable custom types by equality.
+        distinct: list[Any] = []
+        for row in hashable:
+            if row not in distinct:
+                distinct.append(row)
+        return len(distinct)
+
+
+def _distinct_key_parts(key_columns: Sequence[Any], batch: list[Mapping[str, Any]]) -> list[list[Mapping[str, Any]]]:
+    """Cut a batch before each key equal in Python to an earlier one, so no statement repeats it."""
+    parts: list[list[Mapping[str, Any]]] = [[]]
+    seen: set[Any] = set()
+    names = [column.key for column in key_columns]
+    try:
+        # Parts keep the input order: rows of other keys may depend on earlier rows through another unique constraint.
+        for row in batch:
+            key = tuple([row[name] for name in names])
+            # NULL keys never conflict under the default NULLS DISTINCT; NULLS NOT DISTINCT goes to the retry.
+            if all(value is not None for value in key):
+                try:
+                    repeat = key in seen
+                except TypeError:  # ARRAY values come as lists, JSON as dicts
+                    key = _hashable(key)
+                    repeat = key in seen
+                if repeat:
+                    parts.append([])
+                    seen.clear()
+                seen.add(key)
+            parts[-1].append(row)
+    except TypeError:  # an unhashable key type: the database decides, see _upsert_page
+        return [batch]
+    return parts
 
 
 def _sqlstate(error: DBAPIError) -> str | None:
@@ -513,29 +564,40 @@ def _sqlstate(error: DBAPIError) -> str | None:
     return getattr(error.orig, "sqlstate", None) or getattr(error.orig, "pgcode", None)
 
 
-async def _upsert_chunk(
+async def _execute_rows(session: AsyncSession, stmt: Any, rows: Sequence[Mapping[str, Any]], size: int) -> list[Any]:
+    """Execute rows in executemany batches of ``size``; return the RETURNING rows."""
+    returned: list[Any] = []
+    for batch, sql_row in _batches(rows, size):
+        if sql_row:
+            returned += (await session.execute(stmt.values(batch[0]))).all()
+        else:
+            returned += (await session.execute(stmt, batch)).all()
+    return returned
+
+
+async def _upsert_page(
     session: AsyncSession,
-    chunk: list[dict[str, Any]],
     stmt: Any,
-    make_stmt: Callable[[list[dict[str, Any]]], Any],
+    key_columns: Sequence[Any],
+    page: list[Mapping[str, Any]],
     *,
-    savepoint: bool,
-) -> set[Any]:
-    """Execute one upsert chunk and return the primary keys of the affected rows."""
-    if not savepoint or len(chunk) == 1:
-        return _row_keys(await session.execute(stmt))
+    row_by_row: bool,
+) -> list[Any]:
+    """Execute an upsert page and its retry in SAVEPOINTs; return the RETURNING rows."""
+    returned: list[Any] = []
     try:
         async with session.begin_nested():
-            return _row_keys(await session.execute(stmt))
+            # A single-row statement cannot repeat a key, so row-by-row pages need no cuts.
+            for part in [page] if row_by_row else _distinct_key_parts(key_columns, page):
+                returned += await _execute_rows(session, stmt, part, 1 if row_by_row else len(part))
+            return returned
     except DBAPIError as e:
         if _sqlstate(e) != _CARDINALITY_VIOLATION:
             raise
-    # PostgreSQL rejects keys that are equal for the database (per column type) within one statement.
-    logger.debug("upsert chunk of %d rows has equal keys, retrying row by row", len(chunk))
-    keys: set[Any] = set()
-    for row in chunk:
-        keys |= _row_keys(await session.execute(make_stmt([row])))
-    return keys
+    # The database equated distinct Python keys through a column type, index collation or trigger.
+    logger.debug("upsert batch of %d rows has equal keys, retrying row by row", len(page))
+    async with session.begin_nested():
+        return await _execute_rows(session, stmt, page, 1)
 
 
 async def _execute_bulk_insert(
@@ -543,29 +605,30 @@ async def _execute_bulk_insert(
     session: AsyncSession,
     model: type[SQLModel],
     rows: Sequence[Mapping[str, Any]],
-    fields: set[str],
-    make_stmt: Callable[[list[dict[str, Any]]], Any],
+    stmt: Any,
     *,
-    upsert: bool = False,
+    upsert_keys: Sequence[Any] = (),
+    row_by_row: bool = False,
 ) -> int:
-    """Execute chunks without committing: total rowcount, or distinct RETURNING PKs for upsert."""
+    """Execute ``stmt`` in executemany batches without committing; return the number of written table rows."""
     start = time.perf_counter()
     model_name = model.__name__
-    affected = 0
-    written: set[Any] = set()
-    # Only PostgreSQL rejects a repeated key within one statement; SQLite applies rows in order.
-    savepoint = session.get_bind(model).dialect.name == "postgresql"
+    dialect = session.get_bind(model).dialect
+    # Without insertmanyvalues (use_insertmanyvalues=False) an executemany cannot return rows.
+    row_by_row = row_by_row or not dialect.insert_executemany_returning
+    page_size = dialect.insertmanyvalues_page_size
+    returned: list[Any] = []
 
     try:
-        for chunk, stmt in _bulk_insert_statements(session, model, rows, fields, make_stmt):
-            if upsert:
-                written |= await _upsert_chunk(session, chunk, stmt, make_stmt, savepoint=savepoint)
-            else:
-                result = cast("CursorResult[Any]", await session.execute(stmt))
-                affected += result.rowcount
+        # A Core statement does not autoflush: pending ORM changes (e.g. parent rows) go first.
         await session.flush()
-    except MutationError:
-        raise
+        # Only PostgreSQL rejects a repeated key within one statement; SQLite applies rows in order.
+        if upsert_keys and dialect.name == "postgresql" and len(rows) > 1:
+            for offset in range(0, len(rows), page_size):
+                page = list(rows[offset : offset + page_size])
+                returned += await _upsert_page(session, stmt, upsert_keys, page, row_by_row=row_by_row)
+        else:
+            returned += await _execute_rows(session, stmt, rows, 1 if row_by_row else page_size)
     except IntegrityError as e:
         logger.warning("%s(%s) — integrity error: %s", func_name, model_name, e.orig)
         msg = f"Integrity error inserting {model_name}: {e.orig}"
@@ -575,12 +638,12 @@ async def _execute_bulk_insert(
         msg = f"Database error inserting {model_name}: {e}"
         raise DatabaseError(msg) from e
 
-    if upsert:
-        affected = len(written)
+    # Each statement returns the rows it wrote: a repeated upsert key returns one row several times.
+    written = _distinct_count(returned) if upsert_keys else len(returned)
     logger.debug(
-        "%s(%s) → %d of %d rows, %.3fs", func_name, model_name, affected, len(rows), time.perf_counter() - start
+        "%s(%s) → %d of %d rows, %.3fs", func_name, model_name, written, len(rows), time.perf_counter() - start
     )
-    return affected
+    return written
 
 
 async def insert_objects_ignore_conflicts[T: SQLModel](
@@ -595,9 +658,12 @@ async def insert_objects_ignore_conflicts[T: SQLModel](
 
     Executes ``INSERT ... ON CONFLICT (...) DO NOTHING``. Replayed and concurrent batches
     are protected by the database unique constraint. No ORM objects are created or
-    refreshed. Large inputs are split by the actual compiled bind parameters, including
-    Python defaults of omitted columns and SQL expressions. Does not commit; the caller
-    rolls back the whole operation, including after a failure in a later chunk.
+    refreshed. Rows go as executemany of one single-row statement: SQLAlchemy renders up to
+    ``insertmanyvalues_page_size`` rows per INSERT within the dialect parameter limit.
+    A row with SQL expression values is inserted by its own statement, in input order, and
+    so is every row of an engine with ``use_insertmanyvalues=False``.
+    Pending session changes are flushed first. Does not commit; the caller rolls back
+    the whole operation, including after a failure in a later batch.
 
     Values are passed without Pydantic validation and must already have the right types.
     Omitted columns get their column defaults: ``Field(default=...)``/``default_factory``
@@ -615,7 +681,7 @@ async def insert_objects_ignore_conflicts[T: SQLModel](
 
     Raises:
         MutationError: Invalid fields, conflict key or row keys, unsupported dialect,
-            a single row over the bind-parameter limit, or another constraint violation.
+            or another constraint violation.
         DatabaseError: On unexpected database errors.
 
     """
@@ -624,12 +690,12 @@ async def insert_objects_ignore_conflicts[T: SQLModel](
 
     func_name = "insert_objects_ignore_conflicts"
     insert_fn, columns, fields = _prepare_bulk_insert(func_name, session, model, rows, conflict_columns)
-    index_elements = [columns[name] for name in conflict_columns]
-
-    def make_stmt(chunk: list[dict[str, Any]]) -> Any:
-        return insert_fn(model).values(chunk).on_conflict_do_nothing(index_elements=index_elements)
-
-    return await _execute_bulk_insert(func_name, session, model, rows, fields, make_stmt)
+    stmt = (
+        insert_fn(model.__table__)  # type: ignore[attr-defined]  # SQLAlchemy metaclass
+        .on_conflict_do_nothing(index_elements=[columns[name] for name in conflict_columns])
+        .returning(*inspect(model).primary_key)
+    )
+    return await _execute_bulk_insert(func_name, session, model, _bind_rows(columns, fields, rows), stmt)
 
 
 async def upsert_objects[T: SQLModel](
@@ -645,15 +711,21 @@ async def upsert_objects[T: SQLModel](
     Insert new rows and update existing ones by the key ``conflict_columns``.
 
     Executes ``INSERT ... ON CONFLICT (...) DO UPDATE SET ...`` without creating or
-    refreshing ORM objects. Chunks are split by the actual bind-parameter count, including
-    defaults and ``set_extra``. Does not commit; the caller owns the rollback of all chunks.
-
-    If a key repeats in the input, the last row wins. The database decides what is a repeat
-    by the column type (``NUMERIC`` rounding, ``citext``, ``NULLS NOT DISTINCT``), not Python
-    equality. On PostgreSQL a multi-row chunk runs in a SAVEPOINT (``begin_nested`` flushes
-    the session first); the database rejects a chunk with a repeated key, and it is re-run
-    row by row. All key columns must be present in rows. Replaying the operation creates no
-    duplicates, but ``set_extra`` expressions run again: ``hits + 1`` is not idempotent.
+    refreshing ORM objects. Rows are batched as in ``insert_objects_ignore_conflicts``;
+    an expanding ``IN`` or ``literal_execute`` parameter in ``set_extra`` cannot be used
+    with executemany, so each row then runs as its own statement. Does not commit; the
+    caller owns the rollback of all batches and must roll back after an error.
+
+    If a key repeats in the input, the last row wins. PostgreSQL rejects a statement that hits
+    one key twice, so a key value equal in Python to an earlier one starts the next statement;
+    statements keep the input order, with the result of row-by-row execution. On PostgreSQL
+    each page of ``insertmanyvalues_page_size`` rows of a multi-row call runs in a SAVEPOINT,
+    also when its rows go one by one: the database decides what is a repeat (column type,
+    ``NUMERIC`` rounding, index collation, a ``BEFORE INSERT`` trigger, ``NULLS NOT
+    DISTINCT``), and a page it rejects is rolled back to the SAVEPOINT and re-run row by row.
+    All key columns must be present in rows.
+    Replaying the operation creates no duplicates, but ``set_extra`` expressions run again:
+    ``hits + 1`` is not idempotent.
 
     Values are passed without Pydantic validation. Defaults of omitted columns apply on
     insert; on conflict those columns keep their values unless set in ``set_extra``.
@@ -675,8 +747,7 @@ async def upsert_objects[T: SQLModel](
         a key repeated in the input counts once.
 
     Raises:
-        MutationError: Invalid input, a single row over the bind-parameter limit,
-            unsupported dialect, or a database constraint violation.
+        MutationError: Invalid input, unsupported dialect, or a database constraint violation.
         DatabaseError: On unexpected database errors.
 
     """
@@ -711,17 +782,22 @@ async def upsert_objects[T: SQLModel](
         msg = f"{func_name}({model_name}): nothing to update, use insert_objects_ignore_conflicts"
         raise MutationError(msg)
 
+    # A literal None would become a SET bind parameter, and SQLAlchemy inserts such statements row by row.
+    extra_set = {columns[name]: null() if value is None else value for name, value in extra.items()}
+    stmt = insert_fn(model.__table__)  # type: ignore[attr-defined]  # SQLAlchemy metaclass
+    # Keys are columns, not field names: sa_column may use a different database name.
+    set_ = {columns[name]: stmt.excluded[columns[name].key] for name in to_update} | extra_set
     index_elements = [columns[name] for name in conflict_columns]
-    extra_set = {columns[name]: value for name, value in extra.items()}
-    primary_key = inspect(model).primary_key
-
-    def make_stmt(chunk: list[dict[str, Any]]) -> Any:
-        stmt = insert_fn(model).values(chunk)
-        # Keys are columns, not field names: sa_column may use a different database name.
-        set_ = {columns[name]: stmt.excluded[columns[name].key] for name in to_update} | extra_set
-        return stmt.on_conflict_do_update(index_elements=index_elements, set_=set_).returning(*primary_key)
-
-    return await _execute_bulk_insert(func_name, session, model, rows, fields, make_stmt, upsert=True)
+    stmt = stmt.on_conflict_do_update(index_elements=index_elements, set_=set_).returning(*inspect(model).primary_key)
+    return await _execute_bulk_insert(
+        func_name,
+        session,
+        model,
+        _bind_rows(columns, fields, rows),
+        stmt,
+        upsert_keys=index_elements,
+        row_by_row=_has_post_compile_params(extra_set.values()),
+    )
 
 
 async def check_for_related_records[T: SQLModel](