matrx-orm 3.1.34__tar.gz → 3.1.36__tar.gz

{matrx_orm-3.1.34 → matrx_orm-3.1.36}/CLAUDE.md

@@ -26,6 +26,7 @@ Every table that becomes a Model needs row identity: a single-column primary key
 - **Upsert writes only what the caller supplied.** `upsert` / `bulk_upsert` / `upsert_with_conflict` materialize every field's Python default so the **INSERT** half is complete — but `ON CONFLICT DO UPDATE SET` covers **only the columns present in the caller's `data`** (an explicit `update_fields` still wins). Letting defaults reach the UPDATE half silently RESET every defaulted column on each re-upsert of an existing row: it once reset scraped-and-analyzed `research.rs_source` rows to `scrape_status='pending', server_attempts=0` (a watchdog then finalized them `'failed'`) and undid users' manual `is_included=false`. A caller-supplied `None` is excluded too — it is dropped from the INSERT, and `EXCLUDED.<col>` on an omitted column resolves to that column's **DEFAULT**. An upsert of only conflict keys means "ensure this row exists" and reconciles to a no-op self-assignment, never a raise (a bare `DO NOTHING` would break `RETURNING *`). Pinned by `tests/level1/test_upsert_default_leak.py`.
 - **Read-only models** (`_read_only = True`): a model that can be queried but NEVER written through the ORM — every write path raises `ReadOnlyModelError` (enforced at `QueryExecutor.insert/bulk_insert/upsert/update/delete`, so it covers all of `create`/`save`/`update`/`delete`/`update_where`/`delete_where`/`upsert`/`bulk_*`). Used for DB-managed tables the app must not mutate. The canonical **`auth.users`** model is baked into the schema generator (`get_string_user_model`) as the full, curated Supabase auth schema (safe columns only — no password/token fields), `_read_only`, bound per-database. Read a user's profile/metadata there; mutate users only through Supabase Auth (GoTrue).
 - **Connection-level type codecs** (`async_db_manager._init_connection_default`): every pooled connection decodes **uuid → `str`**, **vector → `list[float]`**, and **json/jsonb → `dict`/`list`** at the driver boundary — the TYPE decides the decode, never the call path. So `call_function` scalar/rows, raw fetches, and field hydration all agree; a caller must **never `json.loads` a fetched json/jsonb value** (it's already decoded; the 2026-07-13 `build_agent_context` crash was a jsonb RPC fetched pre-codec as a raw string). Encoding: a pre-serialised JSON `str` passes through untouched; a native dict/list is dumped (`_encode_json`). Pinned by `tests/level1/test_connection_codecs.py`.
+- **Connection pools are event-loop-owned.** `AsyncDatabaseManager.get_pool()` recreates a pool when a caller arrives on a different loop. If the former owner loop is already closed, stale-pool termination and cleanup discard that pool without trying to schedule callbacks onto the dead loop; unrelated runtime errors still raise. Pinned by `tests/level1/test_pool_loop_recreation.py`.
 - **Query layer**: `QueryBuilder`, typed predicates/projections (`F`, `Q`,
   `Predicate`, `Projection`), window functions, QueryBuilder-backed CTEs, and
   subqueries. SQL-shaped strings fail closed: plain strings name validated

{matrx_orm-3.1.34 → matrx_orm-3.1.36}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: matrx-orm
-Version: 3.1.34
+Version: 3.1.36
 Summary: Async-first PostgreSQL ORM with bidirectional migrations, schema introspection, many-to-many relationships, and built-in state caching
 Project-URL: Homepage, https://github.com/AI-Matrix-Engine/aidream-current
 Project-URL: Repository, https://github.com/AI-Matrix-Engine/aidream-current

{matrx_orm-3.1.34 → matrx_orm-3.1.36}/matrx_orm/core/async_db_manager.py

@@ -39,6 +39,18 @@ from matrx_orm.exceptions import (
 logger = logging.getLogger("matrx_orm.async_db_manager")
 
 
+def _terminate_pool_after_loop_change(pool: Any, config_name: str) -> None:
+    try:
+        pool.terminate()
+    except RuntimeError as exc:
+        if "event loop is closed" not in str(exc).lower():
+            raise
+        logger.warning(
+            "[matrx_orm] Discarded stale pool for '%s' after its owner event loop closed",
+            config_name,
+        )
+
+
 # asyncpg SQLSTATE class 42 "undefined …" — the SQL named something the live DB
 # does not have. All siblings of SyntaxOrAccessError; PostgresSyntaxError is
 # caught separately (it's an ORM-bug shape, not a schema-drift shape). These map
@@ -323,15 +335,15 @@ class AsyncDatabaseManager:
 
                                 def _dispose_old_pool() -> None:
                                     old_task.cancel()
-                                    old_pool.terminate()
+                                    _terminate_pool_after_loop_change(old_pool, config_name)
 
                                 owner_loop.call_soon_threadsafe(_dispose_old_pool)
                             else:
-                                old_pool.terminate()
+                                _terminate_pool_after_loop_change(old_pool, config_name)
                         except RuntimeError:
-                            old_pool.terminate()
+                            _terminate_pool_after_loop_change(old_pool, config_name)
                     else:
-                        old_pool.terminate()
+                        _terminate_pool_after_loop_change(old_pool, config_name)
 
                     # Remove the old pool so we can create a new one
                     del cls._pools[config_name]
@@ -1116,6 +1128,21 @@ class AsyncDatabaseManager:
             for config_name, pool in list(cls._pools.items()):
                 try:
                     await pool.close()
+                except RuntimeError as e:
+                    if "event loop is closed" in str(e).lower():
+                        logger.warning(
+                            "[matrx_orm] Discarded stale pool for '%s' during cleanup "
+                            "after its owner event loop closed",
+                            config_name,
+                        )
+                        continue
+                    raise StateError(
+                        model=None,
+                        operation="cleanup_pools",
+                        reason=f"Failed to close pool for {config_name}",
+                        details={"config_name": config_name},
+                        original_error=e,
+                    ) from e
                 except Exception as e:
                     raise StateError(
                         model=None,

{matrx_orm-3.1.34 → matrx_orm-3.1.36}/matrx_orm/core/base.py

@@ -1155,24 +1155,30 @@ class Model(RuntimeMixin, metaclass=ModelMeta):
         return pks[0]
 
     @classmethod
-    async def load_by_id(cls, item_id: Any) -> Self:
+    async def load_by_id(cls, item_id: Any, use_cache: bool = True) -> Self:
         """Load a row by its primary key. Raises DoesNotExist if absent.
 
         Alias for ``get(**{pk: item_id})`` — mirrors ``BaseManager.load_by_id``.
         Composite-PK models accept a dict ``{pk: val}`` or a PK-ordered
         tuple/list (a bare scalar raises — see ``_build_pk_filter``).
         For a None-returning lookup use :meth:`load_by_id_or_none`.
+
+        ``use_cache=False`` is the AUTHORITATIVE read: pass it whenever the row
+        decides identity, ownership, or whether the request may run at all. It
+        also refreshes the cache with what it read, so a bypass never leaves a
+        stale entry behind.
         """
-        return await cls.get(**cls._build_pk_filter(item_id))
+        return await cls.get(use_cache=use_cache, **cls._build_pk_filter(item_id))
 
     @classmethod
-    async def load_by_id_or_none(cls, item_id: Any) -> Self | None:
+    async def load_by_id_or_none(cls, item_id: Any, use_cache: bool = True) -> Self | None:
         """Load a row by its primary key, or None if absent.
 
         Alias for ``get_or_none(**{pk: item_id})``. Composite-PK models accept
-        a dict / PK-ordered tuple, same as :meth:`load_by_id`.
+        a dict / PK-ordered tuple, same as :meth:`load_by_id`. See
+        :meth:`load_by_id` for when to pass ``use_cache=False``.
         """
-        return await cls.get_or_none(**cls._build_pk_filter(item_id))
+        return await cls.get_or_none(use_cache=use_cache, **cls._build_pk_filter(item_id))
 
     @classmethod
     async def filter_items(cls, *args: Any, **kwargs: Any) -> list[Self]:

{matrx_orm-3.1.34 → matrx_orm-3.1.36}/matrx_orm/session/README.md

@@ -134,6 +134,24 @@ Complex predicates (operator suffixes, JSON paths, joins) only match
 DB rows — the pending state is assumed safe (the DB still applies the
 predicate on its side).
 
+## 🚨 Flush stubs are PARTIAL — they never enter the identity cache
+
+`flush._execute_group` turns each queued UPDATE into a lightweight
+`model_cls(**{pk, **payload})` instance so `bulk_update` can build per-row SQL.
+That instance is **not a row snapshot**: every column outside the payload holds a
+model default (usually `None`). Each one is therefore stamped by
+`matrx_orm.state.mark_partial(...)`, and `ModelState.cache` **evicts** instead of
+storing any partial — the next read refetches the whole row.
+
+This is load-bearing, not hygiene. `bulk_update` write-through used to install
+those stubs as the row's cache entry, so for the whole cache TTL (SHORT_TERM = 10
+min) a freshly-updated row read back with NULLs in every untouched column. On
+2026-07-25 that made a saved conversation read back with no `organization_id`
+seconds after its own turn-1 title/`message_count` UPDATE flushed, and aidream's
+identity guard hard-stopped the next turn. Any new code path that builds a model
+instance from a column subset MUST call `mark_partial` — an undeclared downgrade
+is evicted anyway and SCREAMS (see `state._scream_downgrade`).
+
 ## Nested Sessions
 
 Sub-agents, parallel-execute branches, workflow super-steps — every

{matrx_orm-3.1.34 → matrx_orm-3.1.36}/matrx_orm/session/flush.py

@@ -29,6 +29,7 @@ from collections.abc import Sequence
 from contextvars import ContextVar
 
 from matrx_orm.session.op import SessionOp
+from matrx_orm.state import mark_partial
 
 logger = logging.getLogger("matrx_orm.session.flush")
 
@@ -122,7 +123,13 @@ async def _execute_group(
         for op in ops:
             kwargs = {pk_field: op.pk_value}
             kwargs.update(op.payload)
-            instances.append(model_cls(**kwargs))
+            # These are stubs, NOT row snapshots: every column outside `kwargs`
+            # is a model default (None), not the row's value. Declare that, or
+            # bulk_update's write-through would install one as the row's cache
+            # entry and blank every other column for the TTL (this is the bug
+            # that made a saved conversation read back with no organization_id
+            # on its next turn). See matrx_orm.state.mark_partial.
+            instances.append(mark_partial(model_cls(**kwargs), kwargs))
 
         await model_cls.bulk_update(instances, fields=all_fields)
         return len(instances)
@@ -132,7 +139,9 @@ async def _execute_group(
         pks = getattr(meta, "primary_keys", None) if meta else None
         pk_field = pks[0] if pks else ops[0].pk_field
 
-        instances = [model_cls(**{pk_field: op.pk_value}) for op in ops]
+        instances = [
+            mark_partial(model_cls(**{pk_field: op.pk_value}), (pk_field,)) for op in ops
+        ]
         await model_cls.bulk_delete(instances)
         return len(instances)
 

{matrx_orm-3.1.34 → matrx_orm-3.1.36}/matrx_orm/state.py

@@ -15,6 +15,48 @@ from matrx_orm.exceptions import (
 )
 
 
+# Attribute a write-through instance sets to declare "I am a {pk + payload}
+# stub, not a row snapshot". Set by matrx_orm.session.flush; honored by
+# ModelState.cache (evict, never store) and by mark_partial() below.
+PARTIAL_FIELDS_ATTR = "_matrx_partial_fields"
+
+# One scream per (model, blanked-field-set) — a downgrade is a code defect, so
+# it must be visible, but it must not drown a hot loop in identical banners.
+_SCREAMED_DOWNGRADES: set[tuple[str, tuple[str, ...]]] = set()
+
+
+def mark_partial(instance, fields):
+    """Declare ``instance`` a partial write-through stub of its row.
+
+    Every caller that builds a model instance from a PK plus a subset of
+    columns (session flush, targeted bulk_update) MUST call this. Undeclared
+    partials trip the no-downgrade scream in :meth:`ModelState.cache`.
+    """
+    setattr(instance, PARTIAL_FIELDS_ATTR, frozenset(fields))
+    return instance
+
+
+def _scream_downgrade(model_class, blanked: list[str]) -> None:
+    signature = (model_class.__name__, tuple(sorted(blanked)))
+    if signature in _SCREAMED_DOWNGRADES:
+        return
+    _SCREAMED_DOWNGRADES.add(signature)
+    vcprint(
+        "\n"
+        "══════════════════════════════════════════════════════════════════════\n"
+        "  MATRX ORM — CACHE DOWNGRADE BLOCKED (no-downgrade law)\n"
+        "══════════════════════════════════════════════════════════════════════\n"
+        f"  model:    {model_class.__name__}\n"
+        f"  would blank: {', '.join(sorted(blanked))}\n"
+        "  action:   cache entry EVICTED (next read refetches the full row).\n"
+        "            Nothing was lost, but a PARTIAL model instance reached a\n"
+        "            write-through cache path without calling\n"
+        "            matrx_orm.state.mark_partial() — fix the writer.\n"
+        "══════════════════════════════════════════════════════════════════════",
+        color="red",
+    )
+
+
 class CachePolicy(Enum):
     PERMANENT = "permanent"
     LONG_TERM = "long_term"
@@ -192,7 +234,26 @@ class ModelState:
         return True
 
     async def cache(self, record):
-        """Cache a record with proper error handling."""
+        """Cache a record with proper error handling.
+
+        THE NO-DOWNGRADE LAW: a cache entry may only ever be REPLACED by a
+        record that is at least as informed as the one already there. A
+        write-through instance built from ``{pk, **payload}`` is NOT a row
+        snapshot — every column outside the payload holds a model default
+        (usually ``None``), not the row's real value. Storing one blanks the
+        row for every cached reader until the TTL expires.
+
+        That is exactly how a conversation with a perfectly good
+        ``organization_id`` in Postgres read back as org-less for 10 minutes
+        after its own turn-1 title/message_count UPDATE flushed
+        (``session/flush.py`` builds partial instances; ``bulk_update`` cached
+        them), which then hard-failed the next turn's identity restore.
+
+        So: partial records EVICT instead of caching (the next read refetches
+        the whole row), and an unflagged record that would blank a populated
+        field evicts + SCREAMS — a downgrade means a partial instance reached
+        a write-through path without declaring itself.
+        """
         try:
             if not record:
                 raise ValidationError(model=self.model_class, reason="Cannot cache None record")
@@ -204,6 +265,20 @@ class ModelState:
                     reason="Could not generate cache key for record",
                 )
 
+            if getattr(record, PARTIAL_FIELDS_ATTR, None) is not None:
+                # Declared partial (session flush write-through). Designed
+                # path — evict silently so the next read hits the DB.
+                self.evict_by_id(cache_key)
+                return
+
+            existing = self._cache.get(cache_key)
+            if existing is not None and existing is not record:
+                blanked = self._downgraded_fields(existing, record)
+                if blanked:
+                    self.evict_by_id(cache_key)
+                    _scream_downgrade(self.model_class, blanked)
+                    return
+
             self._cache[cache_key] = record
             self._cache_times[cache_key] = datetime.now()
 
@@ -222,6 +297,14 @@ class ModelState:
                 original_error=e,
             ) from e
 
+    def _downgraded_fields(self, existing, incoming) -> list[str]:
+        """Column names the incoming record would blank on the cached one."""
+        blanked: list[str] = []
+        for name in getattr(self.model_class, "_fields", {}):
+            if getattr(existing, name, None) is not None and getattr(incoming, name, None) is None:
+                blanked.append(name)
+        return blanked
+
     async def remove(self, record):
         cache_key = self._get_record_cache_key(record)
         self._cache.pop(cache_key, None)

{matrx_orm-3.1.34 → matrx_orm-3.1.36}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "matrx-orm"
-version = "3.1.34"
+version = "3.1.36"
 description = "Async-first PostgreSQL ORM with bidirectional migrations, schema introspection, many-to-many relationships, and built-in state caching"
 readme = "README.md"
 license = { text = "MIT" }

{matrx_orm-3.1.34 → matrx_orm-3.1.36}/tests/level1/test_bulk_upsert_increment_set_fields.py

@@ -23,10 +23,10 @@ No DB needed — a stub ``connection``/execute_write captures the rendered SQL
 
 from __future__ import annotations
 
+from datetime import timedelta
 from typing import Any
 
 import pytest
-from matrx_orm import Now
 from matrx_orm.core.base import Model, ModelMeta
 from matrx_orm.core.expressions import Case, Compare, Excluded, F, Interval, Is, Now, When
 from matrx_orm.core.fields import DateTimeField, FloatField, UUIDField
@@ -138,7 +138,7 @@ async def test_window_roll_case_set_fields_shifts_params_past_values() -> None:
     assert f"({table}.cost_used_usd + $" in sql
     assert "updated_at = NOW()" in sql
     assert len(params) == 8
-    assert params[4:6] == ("24 hours", "24 hours")
+    assert params[4:6] == (timedelta(hours=24), timedelta(hours=24))
     assert params[6:] == (2.5, 2.5)
 
 

matrx_orm-3.1.36/tests/level1/test_pool_loop_recreation.py

@@ -0,0 +1,57 @@
+from __future__ import annotations
+
+from typing import Any
+
+import pytest
+
+from matrx_orm.core.async_db_manager import (
+    AsyncDatabaseManager,
+    _terminate_pool_after_loop_change,
+)
+
+
+class _ClosedLoopPool:
+    def __init__(self) -> None:
+        self.terminate_calls = 0
+        self.close_calls = 0
+
+    def terminate(self) -> None:
+        self.terminate_calls += 1
+        raise RuntimeError("Event loop is closed")
+
+    async def close(self) -> None:
+        self.close_calls += 1
+        raise RuntimeError("Event loop is closed")
+
+
+def test_stale_pool_termination_tolerates_closed_owner_loop() -> None:
+    pool = _ClosedLoopPool()
+
+    _terminate_pool_after_loop_change(pool, "main")
+
+    assert pool.terminate_calls == 1
+
+
+def test_stale_pool_termination_preserves_other_runtime_errors() -> None:
+    class _BrokenPool:
+        def terminate(self) -> None:
+            raise RuntimeError("unexpected pool corruption")
+
+    with pytest.raises(RuntimeError, match="unexpected pool corruption"):
+        _terminate_pool_after_loop_change(_BrokenPool(), "main")
+
+
+@pytest.mark.asyncio
+async def test_cleanup_discards_pool_owned_by_closed_loop(monkeypatch: Any) -> None:
+    pool = _ClosedLoopPool()
+    monkeypatch.setattr(AsyncDatabaseManager, "_pools", {"main": pool})
+    monkeypatch.setattr(AsyncDatabaseManager, "_pool_loops", {"main": 1})
+    monkeypatch.setattr(AsyncDatabaseManager, "_locks", {"main": object()})
+    monkeypatch.setattr(AsyncDatabaseManager, "_growth_locks", {"main": object()})
+    monkeypatch.setattr(AsyncDatabaseManager, "_keepalive_tasks", {})
+
+    await AsyncDatabaseManager.cleanup()
+
+    assert pool.close_calls == 1
+    assert AsyncDatabaseManager._pools == {}
+    assert AsyncDatabaseManager._pool_loops == {}

matrx_orm-3.1.36/tests/session/test_cache_no_downgrade.py

@@ -0,0 +1,97 @@
+"""THE NO-DOWNGRADE LAW — a cache entry may never lose information.
+
+Regression guard for the 2026-07-25 incident: a conversation with a correct
+``organization_id`` in Postgres read back org-less for the whole 10-minute
+cache TTL, because ``session/flush.py`` builds ``{pk + payload}`` stubs for
+queued UPDATEs and ``operations/update.py::bulk_update`` wrote them straight
+into the identity cache. The next turn's identity restore then STOPPED the
+request on data that did not exist.
+
+Two independent guarantees are locked in here:
+  1. flush declares its stubs partial (``mark_partial``).
+  2. ``ModelState.cache`` never stores a partial or downgrading record — it
+     evicts, so the next read refetches the whole row.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from matrx_orm.state import PARTIAL_FIELDS_ATTR, CachePolicy, ModelState, mark_partial
+
+
+class _Conversation:
+    """Minimal Model stand-in with the fields the incident turned on."""
+
+    _fields = {
+        "id": object(),
+        "organization_id": object(),
+        "title": object(),
+        "message_count": object(),
+    }
+
+    class _Meta:
+        primary_keys = ["id"]
+
+    _meta = _Meta()
+
+    def __init__(self, **kwargs):
+        for name in self._fields:
+            setattr(self, name, kwargs.get(name))
+
+    @classmethod
+    def get_database_name(cls):
+        return "test_db"
+
+
+def _state() -> ModelState:
+    state = ModelState(_Conversation)
+    state.policy = CachePolicy.SHORT_TERM
+    return state
+
+
+def _full_row() -> _Conversation:
+    return _Conversation(
+        id="c1", organization_id="org-1", title="Anthropic Model Sync", message_count=3
+    )
+
+
+@pytest.mark.asyncio
+async def test_flush_update_stub_is_declared_partial():
+    stub = mark_partial(_Conversation(id="c1", message_count=5), {"id", "message_count"})
+    assert getattr(stub, PARTIAL_FIELDS_ATTR) == frozenset({"id", "message_count"})
+
+
+@pytest.mark.asyncio
+async def test_partial_stub_evicts_instead_of_blanking_the_row():
+    state = _state()
+    await state.cache(_full_row())
+    assert (await state.get(id="c1")).organization_id == "org-1"
+
+    # Exactly what a queued conversation UPDATE flush produces on turn 1.
+    await state.cache(mark_partial(_Conversation(id="c1", message_count=5), {"id", "message_count"}))
+
+    # The org-less stub must NOT be servable. Cache miss → caller refetches.
+    assert await state.get(id="c1") is None
+
+
+@pytest.mark.asyncio
+async def test_undeclared_downgrade_is_refused_too():
+    state = _state()
+    await state.cache(_full_row())
+    await state.cache(_Conversation(id="c1", message_count=7))  # no mark_partial
+    assert await state.get(id="c1") is None
+
+
+@pytest.mark.asyncio
+async def test_equally_informed_record_still_replaces():
+    state = _state()
+    await state.cache(_full_row())
+    fresh = _Conversation(
+        id="c1", organization_id="org-1", title="Renamed", message_count=9
+    )
+    await state.cache(fresh)
+    cached = await state.get(id="c1")
+    assert cached is fresh
+    assert cached.title == "Renamed"
+    assert cached.organization_id == "org-1"