matrx-orm 3.1.97__tar.gz → 3.1.99__tar.gz

{matrx_orm-3.1.97 → matrx_orm-3.1.99}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: matrx-orm
-Version: 3.1.97
+Version: 3.1.99
 Summary: Async-first PostgreSQL ORM with bidirectional migrations, schema introspection, many-to-many relationships, and built-in state caching
 Project-URL: Homepage, https://github.com/AI-Matrix-Engine/aidream-current
 Project-URL: Repository, https://github.com/AI-Matrix-Engine/aidream-current

matrx_orm-3.1.99/matrx_orm/FEATURE.md

@@ -0,0 +1,55 @@
+# Matrx ORM runtime contracts
+
+## Exception diagnostics stay event-loop safe
+
+- **Caller capture is metadata-only.** `_capture_caller_frames()` walks live Python frames and reads only code filename, function name, and line number.
+- **Source-file reads are forbidden during exception construction.** Never use `traceback.extract_stack()` here: it enters `linecache.checkcache()` and synchronously stats source files, which can freeze every request in an async server.
+- **Diagnostic value is preserved.** `ORMException._caller_frames` retains up to three external application frames in oldest-to-newest order while excluding ORM, site-package, frozen-import, and runpy frames.
+- **The regression guard is direct.** `tests/level1/test_exceptions.py::TestORMExceptionBase::test_caller_capture_never_reads_source_files` makes traceback extraction fail and proves exception construction still captures the application call site.
+
+## The capture lane is never NULL-org
+
+`session/fallback.py` owns both failure-capture tables — `ops.system_error`
+(`record_error`) and `ops.system_write_failure` (`record_failures`,
+`drain_spilled_ops`, `create_or_capture`). Both carry `organization_id NOT NULL`.
+
+- **Every capture row is org-stamped by `_capture_org()`.** Three rungs: an org
+  the caller passed, else `None` — handing the row to the DB's
+  `public._stamp_org_default`, which resolves the actor's personal org — else
+  `SYSTEM_ORGANIZATION_ID`.
+- **The system-org rung is the point, not a shrug.** A boot step, a cron sweep,
+  an orphan task and an idle-txn stall have **no acting user by definition**.
+  NULL is not a scope for them (db-rules §2 NO NULL ORG); the system org is.
+- **Never let this raise.** Both writers swallow their own failures by contract,
+  so a `23502` here would make the error queue silently fail to record errors.
+  `_capture_org()` cannot return `None` when there is no actor, and DB trigger
+  `_120_stamp_capture_org` re-stamps anything that still arrives org-less.
+- **`SYSTEM_ORGANIZATION_ID` lives in this package deliberately.** matrx-orm may
+  never import aidream; the system org is part of the database structure this
+  package exists to enforce.
+- **Passing an explicit org always wins** — `record_error(..., organization_id=...)`
+  and `record_failures(..., organization_id=...)`.
+
+## Entry points
+
+- Runtime hierarchy and formatting: `exceptions.py`
+- Failure capture + org stamping: `session/fallback.py`
+- Focused unit coverage: `tests/level1/test_exceptions.py`
+
+## Change Log
+
+- 2026-08-21 — NO NULL ORG on the capture lane (db-rules §2). Both capture
+  writers now stamp `organization_id` (`_capture_org()`); `ops.system_error` and
+  `ops.system_write_failure` are `NOT NULL` with backstop trigger
+  `_120_stamp_capture_org` (aidream migration 0443). `ops.system_error` left the
+  `known_null_org_writers` allowlist in both ratchet baselines.
+- 2026-08-21 — Fixed the `ON CONFLICT` retry counter in `_bulk_insert_failures`:
+  it qualified the target as `public.system_write_failure`, which no longer
+  exists, so every retry-conflict capture raised `UndefinedTableError` and
+  spilled to disk instead of incrementing `retry_count`.
+- 2026-08-21 — Repointed the poison-proof error capture primitive to the live
+  `ops.system_error` table after the schema reorganization.
+- 2026-08-18 — Replaced traceback extraction with live-frame metadata walking so ordinary ORM exceptions cannot synchronously stat source files on the event loop.
+
+Verified against `exceptions.py` and `tests/level1/test_exceptions.py` on 2026-08-18;
+`session/fallback.py` and live `ops.*` triggers/constraints on 2026-08-21.

{matrx_orm-3.1.97 → matrx_orm-3.1.99}/matrx_orm/core/FEATURE.md

@@ -126,7 +126,52 @@ Pinned by `tests/level1/test_rls_session_scope.py` and
 
 Verified against code: 2026-08-17.
 
+## Idle-in-transaction ceiling and long-transaction naming
+
+**A pooled connection may not sit idle inside a transaction.** Every pool ships
+`idle_in_transaction_session_timeout = IDLE_IN_TRANSACTION_TIMEOUT_MS` (60s) in
+`server_settings`, so Postgres ends a session that opens a transaction and then
+stops talking. A long-*running* statement is `active` and is never affected —
+the timeout counts idle time only. A caller with a genuine need overrides the
+key through its own `server_settings`.
+
+Why it is not optional: an open transaction holds its locks (DDL queues behind
+it for the WHOLE database) and pins the cluster xmin horizon, which stops
+autovacuum from reclaiming dead tuples everywhere, not just on the tables that
+session touched. On 2026-08-21 one session sat idle in a transaction for 18+
+minutes; every `CREATE POLICY` on the live database blocked to `lock_timeout`
+while the migration applier told each agent to fix SQL that was correct.
+
+**Startup parameters do not survive a pooler.** Supavisor drops non-allowlisted
+startup parameters in both transaction and session mode (measured 2026-08-21:
+`SHOW idle_in_transaction_session_timeout` returns `0` on a connection that
+requested 2000ms, and `PGOPTIONS` is dropped the same way). The package default
+therefore governs direct-Postgres deployments; behind a pooler the same ceiling
+has to be a server-side role or database default. That split is the
+package-vs-implementation boundary, not a bug in either half.
+
+**The one session that legitimately needs an exemption is a backup.** A
+parallel `pg_dump` leader holds its snapshot `idle in transaction` for the whole
+dump by design, so it must be exempt — and because a pooler eats the exemption,
+a dump belongs on the DIRECT host, which is what it wants anyway (one long-lived
+session is the opposite of what a pooler is for). That is the standard shape
+everywhere: tight ceiling for application sessions, explicit exemption for the
+backup session, and the exemption VERIFIED before the dump starts rather than
+assumed (`db/mirror/refresh.py::resolve_dump_endpoint`).
+
+**`long_transactions()` names the blocker.** `matrx_orm.long_transactions`
+(async) and `matrx_orm.long_transactions_sql` (a parameter-free SQL string, so
+sync psycopg tooling runs the identical query) return every transaction open
+past a threshold, categorized `idle_in_transaction` or `long_active`, with
+`describe_transaction(s)` for the human line. One primitive serves the
+in-process guard and the migration applier, so an alarm and an error message
+can never disagree about who the blocker was.
+
+Verified against the live database: 2026-08-21.
+
 ## Change log
+- 2026-08-21: Added the idle-in-transaction ceiling on every pool and the
+  `long_transactions` blocker-naming primitive (feedback c72a32ac).
 
 - 2026-08-18: Indexed ModelMeta-derived table names so generated models remain
   discoverable by exact schema/table identity.

{matrx_orm-3.1.97 → matrx_orm-3.1.99}/matrx_orm/secrets_battery/model.py

@@ -158,6 +158,12 @@ class UserSecretAudit(Model):
 
     id = IntegerField(primary_key=True, null=False)
     user_secret_id = UUIDField()
+    # NOT NULL in Postgres (NO NULL ORG, db-rules §2) but NULLABLE here on
+    # purpose: this package has no tenancy model, so a caller that does not
+    # already hold the org passes None and the BEFORE-INSERT resolver
+    # `users._stamp_secret_audit_org` fills it (migration 0444). Flipping this
+    # to null=False would make Field.validate raise on that legitimate call
+    # BEFORE the row ever reaches the trigger. The column is still never NULL.
     organization_id = UUIDField()
     user_id = UUIDField()
     actor_id = UUIDField()

{matrx_orm-3.1.97 → matrx_orm-3.1.99}/matrx_orm/secrets_battery/org_service.py

@@ -160,7 +160,17 @@ async def record_secret_audit(
 
     Audit is a trail, not a transaction: a failed audit write is logged loudly
     but never propagates, because losing the trail must not also lose the
-    operation the user asked for."""
+    operation the user asked for.
+
+    ``organization_id`` is OPTIONAL here and stays that way on purpose: this
+    package has no tenancy model and must not grow one (see the module
+    docstring). Pass it when the caller already holds it; when it is ``None``
+    the DATABASE resolves it — ``users._stamp_secret_audit_org`` (BEFORE INSERT,
+    aidream/db/migrations/0444_org_null_ban_user_secret_audit.sql) walks the
+    audited secret, then the subject user, then the actor, then the session, and
+    lands on the system org for a genuinely system-context event. The column is
+    ``NOT NULL``: NO NULL ORG (db-rules §2, owner ruling 2026-08-21). Do not
+    reintroduce a ``None`` org as a "system"/"unknown" marker."""
     try:
         await UserSecretAudit.create(
             user_secret_id=user_secret_id,

{matrx_orm-3.1.97 → matrx_orm-3.1.99}/matrx_orm/session/fallback.py

@@ -107,6 +107,43 @@ def _safe_uuid(value: Any) -> str | None:
         return None
 
 
+# ── NO NULL ORG — the capture lane (db-rules §2, owner ruling 2026-08-21) ────
+# "If something belongs to the system, that CANNOT EVER be represented by a
+# NULL org!" A system-context failure — a boot step, a cron sweep, an orphan
+# task, an idle-txn stall — has no acting user BY DEFINITION, so NULL is not
+# merely untidy here, it is the wrong answer: those rows belong to the SYSTEM
+# org. The constant lives in this package on purpose. matrx-orm may never
+# import aidream (CLAUDE.md, the package paradox), and the system org's id is
+# part of OUR database structure — enforcing exactly that structure is what
+# this package exists to do, not a coupling to the host.
+SYSTEM_ORGANIZATION_ID = "39c38960-d30c-4840-b0c1-c9960de95582"
+
+
+def _capture_org(organization_id: str | None, user_id: str | None) -> str | None:
+    """The organization one capture row belongs to. Never a NULL that survives.
+
+    Three rungs, in the order db-rules §2 lays them out:
+
+    1. an org the caller resolved — always wins;
+    2. an acting user with no org — returns ``None`` DELIBERATELY, handing the
+       row to ``public._stamp_org_default``, the canonical backstop that
+       resolves that user's personal org. Re-resolving it here would mean a
+       second authority (and an extra round-trip on an error path);
+    3. no org and no user — the SYSTEM org.
+
+    Rung 3 is the whole point. ``organization_id`` on both capture tables is
+    NOT NULL, so without it an org-less capture would raise ``23502`` and the
+    error queue would fail to record errors — the worst possible place for a
+    write to start failing silently.
+    """
+    explicit = _safe_uuid(organization_id)
+    if explicit is not None:
+        return explicit
+    if _safe_uuid(user_id) is not None:
+        return None
+    return SYSTEM_ORGANIZATION_ID
+
+
 def _op_user_id(op: SessionOp, safe_user_id: str | None) -> str | None:
     """Best-effort actor for a dropped op: the request user if known, else the
     op's own owner column. Keeps the forensic row attributed (the DB's
@@ -127,6 +164,7 @@ def _op_failure_row(
     session_id: str | None,
     safe_user_id: str | None,
     safe_conversation_id: str | None,
+    organization_id: str | None,
     error_text: str,
     tb_text: str,
     failed_at: datetime,
@@ -134,11 +172,13 @@ def _op_failure_row(
     """The single source of truth for one system_write_failure row's column
     values. Used by BOTH the DB insert and the disk spill, so a spilled file is
     byte-identical in shape to a table row (drain re-inserts it verbatim)."""
+    row_user_id = _op_user_id(op, safe_user_id)
     return {
         "op_id": op.op_id,
         "request_id": request_id or session_id,
-        "user_id": _op_user_id(op, safe_user_id),
+        "user_id": row_user_id,
         "conversation_id": safe_conversation_id,
+        "organization_id": _capture_org(organization_id, row_user_id),
         "table_target": op.table,
         "op_type": op.op_type,
         "primary_key": _serialize_primary_key(op.pk_field, op.pk_value),
@@ -246,11 +286,12 @@ async def drain_spilled_ops(database: str | None = None) -> int:
                     """
                     INSERT INTO ops.system_write_failure (
                         op_id, request_id, user_id, conversation_id,
+                        organization_id,
                         table_target, op_type, primary_key, payload,
                         depends_on, error_text, traceback, failed_at, retry_count
                     ) VALUES (
-                        $1, $2, $3, $4, $5, $6, $7::jsonb, $8::jsonb,
-                        $9::jsonb, $10, $11, $12, 0
+                        $1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9::jsonb,
+                        $10::jsonb, $11, $12, $13, 0
                     )
                     ON CONFLICT (op_id) DO NOTHING
                     """,
@@ -258,6 +299,10 @@ async def drain_spilled_ops(database: str | None = None) -> int:
                     row.get("request_id"),
                     _safe_uuid(row.get("user_id")),
                     _safe_uuid(row.get("conversation_id")),
+                    # A file spilled before the org column existed has no
+                    # "organization_id" key — the resolver then re-derives it
+                    # from the row's user, so an OLD spill file still drains.
+                    _capture_org(row.get("organization_id"), row.get("user_id")),
                     row.get("table_target"),
                     row.get("op_type"),
                     row.get("primary_key"),
@@ -308,6 +353,7 @@ async def record_failures(
     request_id: str | None = None,
     user_id: str | None = None,
     conversation_id: str | None = None,
+    organization_id: str | None = None,
     database: str | None = None,
     error_prefix: str | None = None,
 ) -> list[str]:
@@ -316,6 +362,11 @@ async def record_failures(
     Never raises. If the fresh-connection write itself fails, dumps every
     op to stdout in red and to the structured logger.
 
+    ``organization_id`` is the org the captured rows belong to. Omit it and
+    the row still lands org-stamped, never NULL: the acting user's personal org
+    when the op has an owner, else the SYSTEM org (:func:`_capture_org`,
+    db-rules §2 NO NULL ORG).
+
     ``error_prefix`` stamps a recoverability marker onto ``error_text`` (the
     same mechanism ``drain_spilled_ops`` uses for ``DISK_SPILL_RECOVERED_MARKER``)
     so the auto-replay loop can classify this capture without pattern-matching
@@ -350,6 +401,7 @@ async def record_failures(
             session_id=session_id,
             safe_user_id=safe_user_id,
             safe_conversation_id=safe_conversation_id,
+            organization_id=organization_id,
             error_text=error_text,
             tb_text=tb_text,
             failed_at=failed_at,
@@ -427,14 +479,15 @@ async def _bulk_insert_failures(
                     """
                     INSERT INTO ops.system_write_failure (
                         op_id, request_id, user_id, conversation_id,
+                        organization_id,
                         table_target, op_type, primary_key, payload,
                         depends_on, error_text, traceback, failed_at, retry_count
                     ) VALUES (
-                        $1, $2, $3, $4, $5, $6, $7::jsonb, $8::jsonb,
-                        $9::jsonb, $10, $11, $12, 0
+                        $1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9::jsonb,
+                        $10::jsonb, $11, $12, $13, 0
                     )
                     ON CONFLICT (op_id) DO UPDATE SET
-                        retry_count = public.system_write_failure.retry_count + 1,
+                        retry_count = ops.system_write_failure.retry_count + 1,
                         error_text = EXCLUDED.error_text,
                         traceback = EXCLUDED.traceback,
                         failed_at = EXCLUDED.failed_at
@@ -443,6 +496,7 @@ async def _bulk_insert_failures(
                     row["request_id"],
                     row["user_id"],
                     row["conversation_id"],
+                    row["organization_id"],
                     row["table_target"],
                     row["op_type"],
                     row["primary_key"],
@@ -471,6 +525,7 @@ async def record_error(
     user_id: str | None = None,
     conversation_id: str | None = None,
     agent_id: str | None = None,
+    organization_id: str | None = None,
     source_app: str | None = None,
     route: str | None = None,
     error_type: str | None = None,
@@ -510,6 +565,10 @@ async def record_error(
             sampling the loop thread, or any external sampler) and ``exc`` is a
             synthetic, never-raised marker with no ``__traceback__`` of its own.
             Defaults to the formatted traceback of ``exc``.
+        organization_id: the org this failure belongs to, when the caller
+            knows it. Omitted, the row is STILL org-stamped rather than NULL:
+            the acting user's personal org when ``user_id`` is present, else
+            the SYSTEM org (:func:`_capture_org`, db-rules §2 NO NULL ORG).
         payload: request args/body to capture (jsonb). Serialized defensively.
         context: free-form context (debug_label, task_name, partial cost, …).
     """
@@ -534,6 +593,7 @@ async def record_error(
             user_id=user_id,
             conversation_id=conversation_id,
             agent_id=agent_id,
+            organization_id=organization_id,
             source_app=source_app,
             route=route,
             error_type=resolved_error_type,
@@ -581,6 +641,7 @@ async def _insert_error(
     user_id: str | None,
     conversation_id: str | None,
     agent_id: str | None,
+    organization_id: str | None,
     source_app: str | None,
     route: str | None,
     error_type: str | None,
@@ -623,11 +684,12 @@ async def _insert_error(
             """
             INSERT INTO ops.system_error (
                 kind, request_id, user_id, conversation_id, agent_id,
+                organization_id,
                 source_app, route, error_type, error_text, traceback,
                 payload, context, occurred_at
             ) VALUES (
-                $1, $2, $3, $4, $5, $6, $7, $8, $9, $10,
-                $11::jsonb, $12::jsonb, $13
+                $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11,
+                $12::jsonb, $13::jsonb, $14
             )
             """,
             kind,
@@ -635,6 +697,7 @@ async def _insert_error(
             safe_user_id,
             safe_conversation_id,
             safe_agent_id,
+            _capture_org(organization_id, safe_user_id),
             source_app,
             route,
             error_type,
@@ -747,6 +810,7 @@ async def create_or_capture(
     request_id: str | None = None,
     user_id: str | None = None,
     conversation_id: str | None = None,
+    organization_id: str | None = None,
     database: str | None = None,
 ) -> Any:
     """Create ONE row, and never lose its payload if the write fails.
@@ -817,6 +881,7 @@ async def create_or_capture(
             request_id=request_id,
             user_id=user_id,
             conversation_id=conversation_id,
+            organization_id=organization_id or payload.get("organization_id"),
             database=database,
             error_prefix=DIRECT_WRITE_PRESERVED_MARKER,
         )

{matrx_orm-3.1.97 → matrx_orm-3.1.99}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "matrx-orm"
-version = "3.1.97"
+version = "3.1.99"
 description = "Async-first PostgreSQL ORM with bidirectional migrations, schema introspection, many-to-many relationships, and built-in state caching"
 readme = "README.md"
 license = { text = "MIT" }

{matrx_orm-3.1.97 → matrx_orm-3.1.99}/tests/session/test_disk_spill_fallback.py

@@ -193,7 +193,7 @@ async def test_drain_relands_marks_recoverable_and_deletes(spill_dir, fake_db):
     # auto-replay loop picks it up.
     assert len(fake_db) == 1
     args = fake_db[0]
-    error_text = args[9]  # 10th positional ($10) = error_text
+    error_text = args[10]  # 11th positional ($11) = error_text
     assert DISK_SPILL_RECOVERED_MARKER in error_text
     # File deleted after successful re-land.
     assert not list(spill_dir.glob("*.json"))

matrx_orm-3.1.97/matrx_orm/FEATURE.md

@@ -1,21 +0,0 @@
-# Matrx ORM runtime contracts
-
-## Exception diagnostics stay event-loop safe
-
-- **Caller capture is metadata-only.** `_capture_caller_frames()` walks live Python frames and reads only code filename, function name, and line number.
-- **Source-file reads are forbidden during exception construction.** Never use `traceback.extract_stack()` here: it enters `linecache.checkcache()` and synchronously stats source files, which can freeze every request in an async server.
-- **Diagnostic value is preserved.** `ORMException._caller_frames` retains up to three external application frames in oldest-to-newest order while excluding ORM, site-package, frozen-import, and runpy frames.
-- **The regression guard is direct.** `tests/level1/test_exceptions.py::TestORMExceptionBase::test_caller_capture_never_reads_source_files` makes traceback extraction fail and proves exception construction still captures the application call site.
-
-## Entry points
-
-- Runtime hierarchy and formatting: `exceptions.py`
-- Focused unit coverage: `tests/level1/test_exceptions.py`
-
-## Change Log
-
-- 2026-08-21 — Repointed the poison-proof error capture primitive to the live
-  `ops.system_error` table after the schema reorganization.
-- 2026-08-18 — Replaced traceback extraction with live-frame metadata walking so ordinary ORM exceptions cannot synchronously stat source files on the event loop.
-
-Verified against `exceptions.py` and `tests/level1/test_exceptions.py` on 2026-08-18.