matrx-orm 3.1.200__tar.gz → 3.1.201__tar.gz

{matrx_orm-3.1.200 → matrx_orm-3.1.201}/CHANGELOG.md

@@ -2,6 +2,11 @@
 
 ## Unreleased
 
+- **Fix: the shared-lock sampler no longer leaves its timeouts on a pooled backend.**
+  `pytest_lock_guard` sent session-level `set lock_timeout` / `set statement_timeout`
+  through the transaction pooler, where they stay on the backend for the next client
+  (both found leaked on the clone's pool, 2026-10-01). It now uses `SET LOCAL` inside
+  each sample's own transaction.
 - **Fix: a driver timeout longer than the server's statement ceiling screams.**
   `call_function` / `call_function_as` / `execute_query` read the connection's
   effective `statement_timeout` and log one WARNING per call site when the

{matrx_orm-3.1.200 → matrx_orm-3.1.201}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: matrx-orm
-Version: 3.1.200
+Version: 3.1.201
 Summary: Async-first PostgreSQL ORM with bidirectional migrations, schema introspection, many-to-many relationships, and built-in state caching
 Project-URL: Homepage, https://github.com/AI-Matrix-Engine/aidream-current
 Project-URL: Repository, https://github.com/AI-Matrix-Engine/aidream-current

{matrx_orm-3.1.200 → matrx_orm-3.1.201}/matrx_orm/core/FEATURE.md

@@ -282,6 +282,34 @@ trip from a laptop): 435 → ~200 round trips. Pinned by
 `tests/level1/test_transaction_pool_release_sends_nothing.py` and
 `tests/level1/test_rls_session_scope.py`.
 
+**A backend another client left read-only never refuses our writes.** A transaction
+pooler shares server connections with every client of the same login and never resets
+one between clients, so a session-level `SET default_transaction_read_only = on` (or
+`SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY`) sent outside a transaction — the
+shape of an ad-hoc "read-only production" helper — leaves that backend read-only for
+everyone (2026-10-01 incident: dispatchers, workflow claims, request audit, chat writes;
+recurring since 2026-09-12). A transaction-mode pool's connections are
+`ReadWriteConnection` (`read_write.py`): every top-level BEGIN naming no access mode is
+sent as `... READ WRITE` (zero extra round trips), and an autocommit statement refused as
+read-only re-runs once inside `BEGIN READ WRITE` with a WARNING naming the cause. An
+explicit `READ ONLY` is kept; the session is never rewritten. Read-only work through the
+pooler belongs in `BEGIN READ ONLY`, never a session SET. Pinned by
+`tests/test_a_read_only_backend_never_refuses_our_writes.py` (loopback `ISOLATED_PG17_DSN`)
+and `tests/level1/test_read_write_declared.py`.
+
+**A backend handed over inside another client's transaction never takes our writes.** The
+second shape of the same incident: a probe sent `begin read only; …; rollback;` through the
+pooler, an error or timeout skipped the `rollback`, and the pooler handed the open READ ONLY
+transaction to the server. The pooler answers our handshake "idle", so asyncpg learns of it
+only from the reply. A statement that comes back inside a transaction this connection never
+opened is the stranger's: rolled back, logged at ERROR with the remedy, re-run once in our own
+`BEGIN READ WRITE … COMMIT` (the rollback undid it, so exactly once). A top-level BEGIN answered
+with 25001 is the same: rolled back and begun again; a second stranger raises
+`InheritedTransactionError` by name. Pinned by
+`tests/test_a_backend_handed_over_mid_transaction_never_takes_our_writes.py`, which runs an
+in-process transaction pooler that never resets its one server connection (loopback
+`ISOLATED_PG17_DSN`; 4 of 5 fail on the code before it).
+
 **The destructive live proof is loopback-only.**
 `tests/live_pooler_role.py::assert_isolated_pooler_target` runs before DSN
 construction or connection and refuses every shared or remote database. The proof
@@ -360,6 +388,8 @@ can never disagree about who the blocker was.
 Verified against the live database: 2026-08-21.
 
 ## Change log
+- 2026-10-01: `ReadWriteConnection` rolls back a stranger's open transaction it was handed (statement reply or 25001 on BEGIN), re-runs our statement in its own transaction, and names a repeat `InheritedTransactionError`.
+- 2026-10-01: `ReadWriteConnection` for transaction-mode pools — every BEGIN declares READ WRITE and a read-only-refused autocommit statement re-runs once in one; a pooled backend another client left read-only at session level no longer fails the platform's writes.
 - 2026-09-30: Fewer round trips per request (agent-launch latency, lane deep): one-statement proof of a clean empty read; ceilings + declared context + `rls_session` settings in ONE BEGIN-time `set_config` statement (`transaction(opening_gucs=)`, `set_local_gucs`, `emit_local_gucs`); transaction-mode pools release with no reset statement. 435 → ~200 round trips on a flashcard launch.
 - 2026-09-30: `statement_ceiling.py` — a driver timeout above the server's statement ceiling is logged once per call site; `SERVER_CEILING` sentinel. ~19 callers passed 60-600 s under the 30 s `postgres` ceiling (the nightly keyword classifier was cancelled at 30 s for eight days).
 - 2026-09-12 (later): ONE retry mechanism. Two independent lock-retry

{matrx_orm-3.1.200 → matrx_orm-3.1.201}/matrx_orm/core/async_db_manager.py

@@ -25,6 +25,7 @@ from matrx_orm.core.pooler_role import (
     reprove_empty_read,
 )
 from matrx_orm.core.statement_ceiling import resolve_driver_timeout
+from matrx_orm.core.read_write import ReadWriteConnection
 from matrx_orm.core.tls import resolve_asyncpg_ssl
 from matrx_orm.exceptions import (
     AdapterError,
@@ -693,6 +694,15 @@ class AsyncDatabaseManager:
                                 if effective_mode == "transaction"
                                 else None
                             ),
+                            # A transaction pooler hands us server connections other
+                            # clients may have left read-only at SESSION level; every
+                            # BEGIN declares READ WRITE and a refused autocommit
+                            # statement re-runs once in one (core/read_write.py).
+                            connection_class=(
+                                ReadWriteConnection
+                                if effective_mode == "transaction"
+                                else asyncpg.Connection
+                            ),
                         )
                         try:
                             async with asyncio.timeout(create_timeout):

matrx_orm-3.1.201/matrx_orm/core/read_write.py

@@ -0,0 +1,221 @@
+"""A pooled backend another client left read-only never refuses our writes.
+
+A transaction-mode pooler (Supavisor / PgBouncer, port 6543) shares its server
+connections between EVERY client that logs in as the same user, and never resets
+one between clients. A client that runs a session-level read-only setting outside a
+transaction — ``SET default_transaction_read_only = on``,
+``SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY`` (the shape of an ad-hoc
+"read-only production" helper) — leaves it on whichever server connection the pooler
+handed it. Every later transaction routed there starts read-only, and every write
+fails with ``cannot execute ... in a read-only transaction`` while the database is
+writable. (2026-10-01: SMS and notification dispatchers, workflow claims, the request
+audit and chat writes; recurring since at least 2026-09-12. Proven live: one of 21
+pooled backends carried ``default_transaction_read_only = on``, ``source = session``.)
+
+``ReadWriteConnection`` is the connection class of every transaction-mode pool:
+
+* every top-level ``BEGIN`` that names no access mode is sent as ``BEGIN ... READ
+  WRITE`` — the ORM's own ``transaction()`` and any raw ``conn.transaction()``
+  alike, at no extra round trip. An explicit ``READ ONLY`` is kept.
+* an AUTOCOMMIT statement the backend refuses as read-only (SQLSTATE 25006) is re-run
+  once inside one explicit ``BEGIN READ WRITE`` transaction, and the event is logged
+  with its remedy. The refusal proves the statement never ran: Postgres checks the
+  access mode before a statement modifies anything, and an implicit transaction
+  rolls back whole.
+
+THE SECOND SHAPE — a server connection handed over INSIDE another client's transaction
+(2026-10-01: live probes sent ``begin read only; …; rollback;`` as one string through the
+pooler; an error or timeout skipped the ``rollback`` and the open READ ONLY transaction went to
+the platform server). A transaction pooler answers our handshake with "idle" itself, so asyncpg
+only learns of the stranger's transaction from the reply to our next statement:
+
+* a statement that comes back inside a transaction this connection never opened (asyncpg's
+  ``_top_xact`` unset, no BEGIN of ours open) ran inside a STRANGER's transaction: that
+  transaction is rolled back, logged at ERROR with its remedy, and the statement re-runs once in
+  our own ``BEGIN READ WRITE … COMMIT``. The rollback undid whatever the statement did, so the
+  re-run is exactly once.
+* a top-level ``BEGIN`` the server answers with SQLSTATE 25001 ("there is already a transaction
+  in progress" / "read-write mode must be set before any query") is the same stranger: rolled
+  back, begun again once; a second stranger raises ``InheritedTransactionError`` by name —
+  never a bare 25006 three statements later.
+
+It never touches the session it was handed — the poison stays visible to whoever
+investigates (``SHOW default_transaction_read_only``) and the guard stays per
+transaction. Session-mode pools keep the plain asyncpg connection: their server
+connection is ours alone and asyncpg's release reset clears it.
+"""
+
+from __future__ import annotations
+
+import logging
+import re
+from collections.abc import Awaitable, Callable
+from typing import Any
+
+import asyncpg
+
+logger = logging.getLogger(__name__)
+
+# One statement that opens a top-level transaction: BEGIN / BEGIN WORK / BEGIN
+# TRANSACTION / START TRANSACTION, optionally with modes and one trailing ';'.
+_BEGIN_STATEMENT = re.compile(
+    r"^\s*(?:BEGIN|START\s+TRANSACTION)\b(?P<modes>[^;]*?)\s*;?\s*$",
+    re.IGNORECASE | re.DOTALL,
+)
+_ACCESS_MODE = re.compile(r"\bREAD\s+(?:ONLY|WRITE)\b", re.IGNORECASE)
+# A statement that ends the top-level transaction (not ROLLBACK TO / RELEASE a savepoint).
+_END_STATEMENT = re.compile(
+    r"^\s*(?:COMMIT|END|ABORT|ROLLBACK)\b(?!\s+(?:TO|PREPARED)\b)(?!\s+(?:WORK|TRANSACTION)\s+TO\b)",
+    re.IGNORECASE,
+)
+_ACTIVE_SQL_TRANSACTION = "25001"
+
+
+def declare_read_write(query: str) -> str:
+    """``query`` with ``READ WRITE`` declared when it is a BEGIN naming no access mode.
+
+    Anything else — a savepoint, a multi-statement script, a BEGIN that already says
+    ``READ ONLY`` or ``READ WRITE`` — is returned unchanged.
+    """
+    match = _BEGIN_STATEMENT.match(query)
+    if match is None or _ACCESS_MODE.search(match.group("modes")):
+        return query
+    return query.strip().rstrip(";").rstrip() + " READ WRITE"
+
+
+_REFUSED_REMEDY = (
+    "[matrx_orm] A pooled server connection refused an autocommit statement as "
+    "read-only: another client of the transaction pooler left a SESSION-level "
+    "read-only default on it (SET default_transaction_read_only / SET SESSION "
+    "CHARACTERISTICS AS TRANSACTION READ ONLY sent outside a transaction). The "
+    "statement never ran and was re-run inside BEGIN READ WRITE. Remedy: find the "
+    "client sending that SET through the pooler — read-only work belongs in "
+    "BEGIN READ ONLY, one transaction at a time. Statement: %s"
+)
+
+
+_STRANGER_REMEDY = (
+    "[matrx_orm] The pooler handed this connection a server connection still inside "
+    "another client's open transaction (a client of the transaction pooler left without "
+    "COMMIT/ROLLBACK, e.g. a 'begin read only; ...; rollback;' probe that errored or timed "
+    "out). That transaction was rolled back and our statement re-runs in its own "
+    "transaction. Remedy: find the client — probes belong on a DIRECT connection (:5432) "
+    "with ROLLBACK in a finally. Statement: %s"
+)
+
+
+class InheritedTransactionError(RuntimeError):
+    """Every server connection the pooler offered was inside another client's transaction."""
+
+
+class ReadWriteConnection(asyncpg.Connection):  # type: ignore[misc]
+    """The connection class of a transaction-mode pool (see module docstring)."""
+
+    _matrx_own_xact = False  # a top-level BEGIN of ours is open
+    _matrx_saw_25001 = False  # the server warned "already a transaction in progress"
+
+    def _process_log_message(self, fields: Any, last_query: Any) -> None:
+        if fields.get("C") == _ACTIVE_SQL_TRANSACTION:
+            self._matrx_saw_25001 = True
+        super()._process_log_message(fields, last_query)
+
+    async def _reset(self) -> None:
+        self._matrx_own_xact = False
+        await super()._reset()
+
+    def _in_strangers_transaction(self) -> bool:
+        return self.is_in_transaction() and self._top_xact is None and not self._matrx_own_xact
+
+    async def _drop_strangers_transaction(self, query: str) -> None:
+        logger.error(_STRANGER_REMEDY, " ".join(str(query).split())[:160])
+        self._matrx_own_xact = False
+        await asyncpg.Connection.execute(self, "ROLLBACK")
+
+    async def _begin(self, query: str) -> str:
+        """A top-level BEGIN that never silently joins a stranger's transaction."""
+        if self._in_strangers_transaction():
+            await self._drop_strangers_transaction(query)
+        for _ in range(2):
+            self._matrx_saw_25001 = False
+            try:
+                result = await asyncpg.Connection.execute(self, query)
+            except asyncpg.exceptions.ActiveSQLTransactionError:
+                pass
+            else:
+                if not self._matrx_saw_25001:
+                    self._matrx_own_xact = True
+                    return result
+            await self._drop_strangers_transaction(query)
+        raise InheritedTransactionError(
+            "Two server connections in a row were inside another client's open transaction; "
+            "nothing was written. A client of the transaction pooler is leaving transactions "
+            "open (see matrx_orm/core/read_write.py)."
+        )
+
+    async def _in_own_transaction(
+        self, call: Callable[..., Awaitable[Any]], query: str, args: tuple[Any, ...], kwargs: dict[str, Any]
+    ) -> Any:
+        await self._begin("BEGIN READ WRITE")
+        try:
+            result = await call(query, *args, **kwargs)
+        except BaseException:
+            self._matrx_own_xact = False
+            try:
+                await asyncpg.Connection.execute(self, "ROLLBACK")
+            except Exception:  # noqa: BLE001 — the pool's release rolls back what this cannot
+                pass
+            raise
+        self._matrx_own_xact = False
+        await asyncpg.Connection.execute(self, "COMMIT")
+        return result
+
+    async def _healed(
+        self, call: Callable[..., Awaitable[Any]], query: str, args: tuple[Any, ...], kwargs: dict[str, Any]
+    ) -> Any:
+        if self._in_strangers_transaction():
+            await self._drop_strangers_transaction(query)
+        if self.is_in_transaction():
+            result = await call(query, *args, **kwargs)
+            if self._top_xact is None and _END_STATEMENT.match(str(query)):
+                self._matrx_own_xact = False
+            return result
+        try:
+            result = await call(query, *args, **kwargs)
+        except asyncpg.exceptions.ReadOnlySQLTransactionError:
+            if self.is_in_transaction():
+                await self._drop_strangers_transaction(query)
+            else:
+                logger.warning(_REFUSED_REMEDY, " ".join(str(query).split())[:160])
+        except Exception:
+            if not self._in_strangers_transaction():
+                raise
+            await self._drop_strangers_transaction(query)
+        else:
+            if not self._in_strangers_transaction():
+                return result
+            # The reply came from inside a stranger's transaction; rolling it back undid it.
+            await self._drop_strangers_transaction(query)
+        return await self._in_own_transaction(call, query, args, kwargs)
+
+    async def execute(self, query: str, *args: Any, **kwargs: Any) -> str:
+        if not args:
+            declared = declare_read_write(query)
+            if _BEGIN_STATEMENT.match(declared) and not (self.is_in_transaction() and not self._in_strangers_transaction()):
+                return await self._begin(declared)
+            query = declared
+        return await self._healed(super().execute, query, args, kwargs)
+
+    async def fetch(self, query: str, *args: Any, **kwargs: Any) -> list[Any]:
+        return await self._healed(super().fetch, query, args, kwargs)
+
+    async def fetchrow(self, query: str, *args: Any, **kwargs: Any) -> Any:
+        return await self._healed(super().fetchrow, query, args, kwargs)
+
+    async def fetchval(self, query: str, *args: Any, **kwargs: Any) -> Any:
+        return await self._healed(super().fetchval, query, args, kwargs)
+
+    async def executemany(self, command: str, args: Any, **kwargs: Any) -> None:
+        return await self._healed(super().executemany, command, (args,), kwargs)
+
+
+__all__ = ["InheritedTransactionError", "ReadWriteConnection", "declare_read_write"]

{matrx_orm-3.1.200 → matrx_orm-3.1.201}/matrx_orm/pytest_lock_guard.py

@@ -413,13 +413,21 @@ def _run_sampler() -> None:  # pragma: no cover — exercised by the guard's own
         with psycopg.connect(
             _STATE.dsn, autocommit=True, prepare_threshold=None
         ) as conn, conn.cursor() as cur:
-            cur.execute("set lock_timeout = '2s'")
-            cur.execute("set statement_timeout = '10s'")
-            _STATE.baseline_oids = _baseline_oids(cur)
+            # The limits are SET LOCAL inside each statement's own transaction, never a session SET:
+            # through Supavisor's transaction pooler a session SET stays on the backend and the next
+            # client inherits it (lock_timeout=2s / statement_timeout=10s were found leaked on the
+            # clone's pool, 2026-10-01 — common-docs v5/PROGRESS-POOLER-LEAK-GUARD.md).
+            def limited(fn):
+                with conn.transaction():
+                    cur.execute("set local lock_timeout = '2s'")
+                    cur.execute("set local statement_timeout = '10s'")
+                    return fn()
+
+            _STATE.baseline_oids = limited(lambda: _baseline_oids(cur))
             _STATE.active = True
             while not _STATE.stop.wait(SAMPLE_INTERVAL_SECONDS):
                 try:
-                    found = _sample_once(cur, time.monotonic())
+                    found = limited(lambda: _sample_once(cur, time.monotonic()))
                 except Exception as exc:  # noqa: BLE001 — never kill the run over a sample
                     log.warning("[%s] sample failed: %s", _PLUGIN_NAME, exc)
                     continue

{matrx_orm-3.1.200 → matrx_orm-3.1.201}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "matrx-orm"
-version = "3.1.200"
+version = "3.1.201"
 description = "Async-first PostgreSQL ORM with bidirectional migrations, schema introspection, many-to-many relationships, and built-in state caching"
 readme = "README.md"
 license = { text = "MIT" }

matrx_orm-3.1.201/tests/level1/test_read_write_declared.py

@@ -0,0 +1,108 @@
+"""A transaction-mode pool declares READ WRITE on every BEGIN (no database needed).
+
+The live half — a backend another client left read-only at session level, written
+through anyway — is ``tests/test_a_read_only_backend_never_refuses_our_writes.py``.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import asyncpg
+import pytest
+
+from matrx_orm.core import async_db_manager as adm
+from matrx_orm.core.async_db_manager import AsyncDatabaseManager
+from matrx_orm.core.config import DatabaseProjectConfig, register_database, registry
+from matrx_orm.core.read_write import ReadWriteConnection, declare_read_write
+
+
+@pytest.mark.parametrize(
+    ("sent", "expected"),
+    [
+        ("BEGIN", "BEGIN READ WRITE"),
+        ("BEGIN;", "BEGIN READ WRITE"),
+        ("begin", "begin READ WRITE"),
+        ("BEGIN ISOLATION LEVEL REPEATABLE READ;", "BEGIN ISOLATION LEVEL REPEATABLE READ READ WRITE"),
+        ("BEGIN ISOLATION LEVEL READ COMMITTED;", "BEGIN ISOLATION LEVEL READ COMMITTED READ WRITE"),
+        ("START TRANSACTION", "START TRANSACTION READ WRITE"),
+        ("BEGIN WORK", "BEGIN WORK READ WRITE"),
+    ],
+)
+def test_a_begin_naming_no_access_mode_declares_read_write(sent: str, expected: str) -> None:
+    assert declare_read_write(sent) == expected
+
+
+@pytest.mark.parametrize(
+    "sent",
+    [
+        "BEGIN READ ONLY;",
+        "BEGIN ISOLATION LEVEL SERIALIZABLE READ ONLY DEFERRABLE;",
+        "begin read only",
+        "BEGIN READ WRITE",
+        "SAVEPOINT sp_1;",
+        "BEGIN; SET LOCAL statement_timeout = 1000",
+        "COMMIT;",
+        "SELECT 'BEGIN'",
+    ],
+)
+def test_everything_else_is_sent_unchanged(sent: str) -> None:
+    assert declare_read_write(sent) == sent
+
+
+class _FakePool:
+    def get_size(self) -> int:
+        return 1
+
+    def terminate(self) -> None:
+        pass
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    ("port", "expected"), [(6543, ReadWriteConnection), (5432, asyncpg.Connection)]
+)
+async def test_only_a_transaction_mode_pool_uses_the_read_write_connection(
+    monkeypatch, port: int, expected: type
+) -> None:
+    seen: dict[str, Any] = {}
+
+    async def _make(pool: _FakePool) -> _FakePool:
+        return pool
+
+    def fake_create_pool(**kwargs: Any):
+        seen.update(kwargs)
+        return _make(_FakePool())
+
+    monkeypatch.setattr(adm.asyncpg, "create_pool", fake_create_pool)
+    monkeypatch.setattr(AsyncDatabaseManager, "_start_keepalive", classmethod(lambda *_a: None))
+    name = f"read_write_class_{port}"
+    registry._configs.pop(name, None)
+    register_database(
+        DatabaseProjectConfig(
+            name=name,
+            alias=f"{name}_alias",
+            host="127.0.0.1",
+            port=str(port),
+            database_name="postgres",
+            user="postgres",
+            password="unused",
+            pool_min=1,
+            pool_max=2,
+            ssl=False,
+            write_queue_enabled=False,
+        )
+    )
+    try:
+        await AsyncDatabaseManager.get_pool(name)
+    finally:
+        AsyncDatabaseManager._pools.pop(name, None)
+        AsyncDatabaseManager._pool_loops.pop(name, None)
+        AsyncDatabaseManager._locks.pop(name, None)
+        AsyncDatabaseManager._growth_locks.pop(name, None)
+        task = AsyncDatabaseManager._keepalive_tasks.pop(name, None)
+        if task is not None:
+            task.cancel()
+        registry._configs.pop(name, None)
+
+    assert seen.get("connection_class") is expected