mainsequence-metatable 0.1.31.dev58__tar.gz → 0.1.31.dev60__tar.gz

{mainsequence_metatable-0.1.31.dev58 → mainsequence_metatable-0.1.31.dev60}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.31.dev58
+Version: 0.1.31.dev60
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT

{mainsequence_metatable-0.1.31.dev58 → mainsequence_metatable-0.1.31.dev60}/docs/adr/api/0007-database-enforced-table-access.md

@@ -51,6 +51,15 @@
 > are when grants changed outside MetaTables are repaired, not on every catalog
 > change. MySQL and SQL Server keep the full sweep on every change.
 
+> Amendment (2026-10-10): [ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md#conditional-system-bootstrap-2026-10-10-amendment)
+> replaces the unconditional deployment sweep on PostgreSQL/TimescaleDB with
+> fresh native-security and login validation under the security-state lock.
+> Every deployment still checks table policies. Unchanged security evidence uses
+> incremental grant application and one conditional write of the complete
+> manifest; drift, credential rotation, pending state or missing evidence retains
+> full repair. An explicit admin repair remains a full sweep. A routine catalog
+> update must not establish a new native baseline for unverified outside drift.
+
 > Amendment (2026-10-09): [ADR 0022](0022-concurrent-requests-and-connection-reuse.md)
 > reuses login-role connections within a pod, only by the login role that opened them
 > and never through `SET ROLE`; no session state carries over from one use to the next.

{mainsequence_metatable-0.1.31.dev58 → mainsequence_metatable-0.1.31.dev60}/docs/adr/api/0013-application-owned-migrations.md

@@ -124,6 +124,51 @@ HTTP request/response contract is unchanged. Client inventory improvements requi
 a package update and application rebuild. API batching benefits existing clients
 after the shared API is deployed, with larger gains for clients sending batches.
 
+### Conditional catalog reconciliation (2026-10-10 amendment)
+
+Reconciliation means making the catalog describe the freshly observed physical
+table while preserving declared semantic metadata. It is additional MetaTables
+behavior around Alembic, not an instruction to reapply migration scripts or rebuild
+identical catalog rows. An unchanged Alembic revision does not prove absence of
+physical drift or completion of an earlier catalog finalization.
+
+```mermaid
+flowchart TD
+    A[Run pending Alembic revisions] --> B[Fresh admission and physical inspection]
+    B --> C[Reconcile physical facts with existing semantic metadata]
+    C --> D{Do stored projections match?}
+    D -->|Yes| E[Keep existing projection rows and UUIDs]
+    D -->|No| F[Replace damaged or changed projections]
+    E --> G[Update only changed contract and lifecycle fields]
+    F --> G
+    G --> H[Fresh security-policy checks and commit]
+```
+
+- Finalization retains the catalog lock, per-table Writer/provider/source checks,
+  fresh physical inspection, owner reachability, foreign-key target validation,
+  missing-table/removal handling, and Timescale hypertable checks. No physical
+  snapshot or authorization result is cached across requests.
+- The API reads existing column, index and foreign-key projections in three batch
+  queries. Comparison includes their scalar fields, full contract fragments and
+  resolved FK target UIDs, and detects missing, extra or damaged rows. Projection
+  row UUIDs and reflection ordering of independent objects are not differences;
+  column ordinals and the order of columns inside keys/indexes remain meaningful.
+- Matching projections are kept, even if only the recorded Alembic revision or
+  lifecycle metadata changes. Changed projections are replaced inside the existing
+  per-table savepoint. Equal-valued contract, snapshot and lifecycle fields are not
+  assigned, avoiding unnecessary ORM dirtiness and metadata/search writes.
+- Hosted/local runtime security-policy checks are explicitly retained at commit
+  even when table metadata is unchanged. Existing policy reconciliation repairs
+  policy differences; equal policies and their JSON permission manifest do not
+  require metadata writes. A failed physical check or conversion still fails
+  rather than being hidden by equality.
+- A healthy unchanged finalization issues no table/projection metadata DML. A
+  matching contract with damaged projections still repairs them. A previously
+  committed migration with failed finalization can retry at the same revision;
+  physical drift is freshly observed and catalog reconciliation completes without
+  replaying already-applied DDL. The response contract and client calls stay the
+  same; no consuming-application opt-in is required.
+
 ## MetaTables system migrations
 
 Hosted system upgrades also run as a deployment prerequisite Job
@@ -167,3 +212,14 @@ fresh reflection after DDL, unchanged-revision reconciliation, authorization bef
 inspection, independent failures and dialect fallbacks. PostgreSQL query grouping is
 checked with a recording driver; SQLite executes real local DDL. Hosted performance
 and live MySQL/SQL Server/PostgreSQL verification remain pending.
+
+The conditional-reconciliation amendment adds SQL-recording checks for zero
+metadata writes and stable projection UUIDs on unchanged finalization, damaged
+scalar/fragment and missing/extra projection repair, revision-only updates,
+semantic changes at the same revision, drift plus failed-finalization retry, and
+fresh permission/missing/inaccessible/owner checks, including self-referencing key
+projection repair. A real local-runtime check verifies zero catalog DML on an
+unchanged upgrade while still refreshing physical security policies, and detects
+a newly added trigger that changes write safety without changing the contract.
+Existing local SQLite application-provider tests verify actual physical drift and recovery without
+reapplying DDL. These are focused non-container checks, not new required CI jobs.

mainsequence_metatable-0.1.31.dev60/docs/adr/api/0014-main-sequence-release-jobs-and-production-migrations.md

@@ -0,0 +1,183 @@
+# ADR 0014: System migrations in the deployment workflow
+
+Date: 2026-10-02
+
+Status: Accepted. Implemented on `development`; hosted verification pending.
+Amended 2026-10-03 ([MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13)): the Job reads the declared runtime database itself
+and initializes it on the first deployment.
+
+Owners: MetaTables API. Main Sequence owns Job execution, images, deployment and
+platform authentication.
+
+Related: [ADR 0001: Runtime storage and bootstrap](0001-unified-api-storage-and-local-sqlite.md),
+[ADR 0008: Database workflows and recovery](0008-mysql-mssql-table-workflows.md) and
+[ADR 0013: Application-owned migrations](0013-application-owned-migrations.md).
+
+## Context
+
+Main Sequence deploys the hosted API from `.mainsequence/workflows/metatables-api.yaml`.
+Settings runs system migrations inside the running API, and that API can only
+apply the revisions it ships. A release's new revisions must be applied before
+its API serves.
+
+## Decision
+
+### The Job prepares the declared database (amended 2026-10-03)
+
+The Job no longer reads the running API. `metatables runtime upgrade` loads
+`runtime_database` from the candidate image's `configuration.yaml` and the
+Environment Secret it names ([ADR 0001](0001-unified-api-storage-and-local-sqlite.md)).
+Every run executes the same steps:
+
+1. Check that the login can secure the database (ADR 0007 prerequisites).
+2. Apply the system migrations.
+3. Register the runtime DataSource, or update its connection to the declaration.
+4. Freshly validate access setup and repair changes. Per-User role passwords
+   derive from the login password, so a rotated password reaches them on the
+   next deployment. The conditional reconciliation amendment below replaces
+   unconditional rebuilding on every deployment.
+
+The first deployment therefore initializes the database. A missing declaration
+or Secret, an unreachable database or unmet prerequisites fail the Job and block
+the rollout. The deployment workflow is the authority, so no admin caller is
+required. A new runtime DataSource records the Job's SDK user as its creator.
+The Job prints `initialized`, `upgraded` or `up_to_date` with the revisions.
+
+Settings' **Run MetaTables migrations** and `metatables runtime initialize` remain
+for Local mode only. The original decision follows. The amendment supersedes
+steps 1–2, the skip rules and the Settings initialization paragraph.
+
+### Conditional system bootstrap (2026-10-10 amendment)
+
+An unchanged deployment remains a validation gate, but does not restart completed
+work or rewrite an identical catalog/security projection. This concerns the
+MetaTables **system** Job, separate from application-provider finalization in
+[ADR 0013](0013-application-owned-migrations.md).
+
+```text
+resolve declaration and Secret
+  -> bootstrap lock
+  -> fresh history, recovery and runtime-binding checks
+  -> current heads + completed journal?
+       yes: fresh physical system-schema validation; preserve journal
+       no: existing Alembic execution and durable recovery; validate schema
+  -> reconcile runtime registration if changed
+  -> security-state lock; fresh native security and login validation
+  -> unchanged evidence? fresh table-policy checks; apply only grant differences
+       otherwise: full security repair, including credential rotation
+  -> commit; fresh registration/reference verification; release bootstrap lock
+```
+
+Hosted configuration defers database inspection to the authoritative preflight
+inside the bootstrap lock. The local path retains its pre-lock check, so an old
+store cannot be replaced or a new file created before compatibility is checked.
+Registration/reference validation is read again after commit under the lock;
+it does not repeat schema reflection or the already verified migration history.
+Startup and explicit activation continue their existing fresh checks.
+
+Matching Alembic heads alone are insufficient. Only a **completed** progress
+journal allows skipping Alembic and journal writes, after fresh schema validation.
+A missing journal is established through the existing path. An incomplete
+journal retains plan/binding checks, acknowledged-step recovery and rejection
+of unknown outcomes. A completed journal records its original execution; a
+source-code change without new revisions does not reset that history. Schema
+drift still blocks deployment and is not silently repaired by replaying Alembic.
+
+On PostgreSQL/TimescaleDB, the existing private `SQLSecurityState.manifest` stores
+versioned native-security evidence and a domain-separated credential checksum,
+alongside roles, memberships, privileges and routine evidence. No schema revision
+is needed. Initialization compares fresh database evidence: role attributes and
+memberships; schema, relation, column, routine, database and default ACLs; owners,
+routine definitions and row-security policies. Table data and statistics are
+excluded. Declared schemas and grant-target tables are included even when the
+login has grant authority without owning them. Fresh per-login authentication
+uses the configured TLS material and a new connection, never a pooled session
+that could conceal a password change.
+The security-state row stays locked until commit to serialize with catalog
+changes and caller admission.
+
+All registered table policies are freshly checked even on the unchanged path.
+Unchanged table owners and `PUBLIC` grants cause no ownership/revoke writes.
+Policies and grant differences still apply when a trigger or catalog change
+affects access. Native drift, credential rotation, pending/unready state or
+missing evidence invokes full repair. Login authentication failures likewise
+require repair. Exhausted login connection capacity conservatively falls back
+to the existing repair. libpq can omit SQLSTATE on a connection/authentication
+failure; these ambiguous driver connection errors also require repair rather
+than allowing a skip. Other probe failures fail deployment. An old installation
+therefore performs one full repair to establish evidence, then converges.
+Explicit admin repair remains a full sweep; MySQL/SQL Server keep their existing
+security initialization behavior.
+
+The complete manifest is compared and persisted once, including backend fields;
+reconciliation must not remove and then restore the routine fingerprint. Ordinary
+incremental changes cannot checkpoint unverified native drift as success. Failed
+security work rolls back its catalog checkpoint and remains repairable on retry.
+
+Named timings cover declaration/Secret resolution, bootstrap/security lock waits,
+preflight, catalog/Alembic execution, system-schema validation, registration,
+native security and login authentication validation, routine repair, table
+policies, roles/grants, credential-store setup, commit and final registration
+verification. Each timing
+reports completion/failure without connection strings, passwords, checksums or
+driver exception text. Successful security setup logs the selected path and its
+reason, including first-run evidence establishment.
+
+Focused verification uses disposable SQLite catalog transactions and recording
+PostgreSQL adapters: unchanged deployments invoke no Alembic or catalog/journal
+DML; unchanged native security emits no role/grant DDL or manifest UPDATE;
+schema drift, interrupted/ambiguous recovery, credential rotation, permission
+drift and failed repair remain checked. Live PostgreSQL/TimescaleDB validation is
+separate from these non-container checks.
+
+### Original decision (2026-10-02)
+
+The deployment workflow upgrades the runtime from the candidate image before the
+rollout:
+
+```text
+candidate image -> migrate-system Job -> API deployment
+```
+
+A failed upgrade blocks the rollout.
+
+The Job, `jobs/migrate_system.py`, runs `metatables runtime upgrade`:
+
+1. Find the Environment's MetaTables API the same way the client does.
+2. Read `GET /runtime-context/` as an organization admin. For admins,
+   `bootstrap.candidate` is the Settings DataSource's public configuration with
+   its Secret references; the Secrets are resolved through the SDK.
+3. Apply this package's system migrations through the shared bootstrap operation:
+   backend lock, progress recovery, runtime registration, access setup and
+   verification. A runtime already at the current revision is left untouched.
+
+The command acts only on an active runtime. With no API deployment or no active
+runtime yet, it succeeds without DDL. The Environment's DataSource is set
+through the deployed API, so if it is set the command migrates when necessary;
+if it is not, it skips migrations and the API deploys. An API release that the
+workflow created but has never deployed counts as no API deployment, so the
+first run deploys the API. A non-admin identity or an unreadable
+context fails before DDL. When it registers the runtime DataSource, it records
+the Job identity's user UID as the creator.
+
+Settings' **Run MetaTables migrations** and `metatables runtime initialize` keep
+initializing a newly selected DataSource: the API serving them is the deployed
+revision and runs its own migrations, so there is no version skew. API startup
+stays inspection-only.
+
+Both targets declare `automatic_redeployment` with `tag_regex: null`, so every
+eligible commit runs the graph. Package publication stays in GitHub Actions.
+
+The upgrade owns only `metatables.api.backend.migrations` and
+`metatables_catalog_version`. Application providers keep their own histories
+under ADR 0013.
+
+## Consequences
+
+- The previous API keeps serving during the upgrade, so each schema change must
+  stay compatible with the previous release. Coordination with its writes is not
+  defined yet.
+- An upgrade followed by a failed deployment leaves the database upgraded. There
+  is no automatic downgrade.
+- The workflow runs only when the platform receives the branch's commit.
+  Verification in a hosted Environment is pending.

{mainsequence_metatable-0.1.31.dev58 → mainsequence_metatable-0.1.31.dev60}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mainsequence-metatable"
-version = "0.1.31.dev58"
+version = "0.1.31.dev60"
 description = "Main Sequence MetaTables Python client and standalone API"
 readme = "README.md"
 requires-python = ">=3.13"

{mainsequence_metatable-0.1.31.dev58 → mainsequence_metatable-0.1.31.dev60}/src/mainsequence_metatable.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.31.dev58
+Version: 0.1.31.dev60
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT

{mainsequence_metatable-0.1.31.dev58 → mainsequence_metatable-0.1.31.dev60}/src/mainsequence_metatable.egg-info/SOURCES.txt

@@ -332,6 +332,7 @@ src/metatables/api/backend/migrations/versions/0012_table_creator.py
 src/metatables/api/backend/migrations/versions/__init__.py
 src/metatables/api/backend/operations/__init__.py
 src/metatables/api/backend/operations/bootstrap_migrations.py
+src/metatables/api/backend/operations/bootstrap_timing.py
 src/metatables/api/backend/operations/catalog_schema.py
 src/metatables/api/backend/operations/create_table.py
 src/metatables/api/backend/operations/finalize_managed.py
@@ -517,6 +518,7 @@ tests/e2e/test_local_actor_client_api.py
 tests/e2e/test_local_cli_workflow.py
 tests/e2e/test_local_client_selection.py
 tests/e2e/test_managed_finalize_client_api.py
+tests/e2e/test_managed_finalize_unchanged.py
 tests/e2e/test_namespaces_admin_http.py
 tests/e2e/test_public_imports.py
 tests/e2e/test_remaining_client_routes.py
@@ -574,6 +576,7 @@ tests/packaging/test_distribution_sources.py
 tests/packaging/test_editable_build.py
 tests/packaging/test_installed_distribution.py
 tests/permissions/test_access_map.py
+tests/permissions/test_deployment_security.py
 tests/permissions/test_governed_sql_scope.py
 tests/permissions/test_incremental_security.py
 tests/permissions/test_local_actor_membership.py

{mainsequence_metatable-0.1.31.dev58 → mainsequence_metatable-0.1.31.dev60}/src/metatables/api/app/bootstrap.py

@@ -34,6 +34,7 @@ from metatables.api.backend.integrations.credential_store import (
 )
 from metatables.api.backend.integrations.local_workspace import _bind_file, _read_marker
 from metatables.api.backend.integrations.runtime_binding import RuntimeStorageBinding
+from metatables.api.backend.operations.bootstrap_timing import bootstrap_phase
 from metatables.api.backend.persistence.database import (
     create_catalog_engine,
     migration_config,
@@ -234,7 +235,7 @@ class RuntimeBootstrap:
         self.reader = None
         self.resources.close()
 
-    def configure(self, body, owner, *, validate_schema=True):
+    def configure(self, body, owner, *, validate_schema=True, inspect_database=True):
         # Validate public configuration before replacing the current candidate.
         values = RuntimeSourceInput.model_validate(body).model_dump(mode="json")
         kind = values["class_type"]
@@ -272,7 +273,8 @@ class RuntimeBootstrap:
         self.candidate, self.owner = values, owner
         self.active, self.uid = False, None
         self.status, self.error, self.current = "unconfigured", None, []
-        self.inspect(validate_schema=validate_schema)
+        if inspect_database:
+            self.inspect(validate_schema=validate_schema)
 
     def open_engine(self, *, create=False):
         from metatables.api.backend.backends.registry import backend_for_engine
@@ -332,34 +334,7 @@ class RuntimeBootstrap:
                         validate_system_schema,
                     )
                     validate_system_schema(connection, self.backend.catalog)
-            with self.factory() as session:
-                row = session.scalar(select(DataSource).where(DataSource.is_runtime.__eq__(True)))
-                if row is None:
-                    self.status = "registration_required"
-                    with engine.connect() as connection:
-                        self.require_prerequisites(connection)
-                elif not self.matches(row):
-                    if self.config.local_mode:
-                        self.status = "incompatible"
-                        self.error = "The stored runtime DataSource does not match this connection."
-                    else:
-                        # The declaration is authoritative; the deployment updates the record.
-                        self.status = "registration_required"
-                        self.error = "The runtime database declaration changed. The next deployment updates its registration."
-                else:
-                    binding = RuntimeStorageBinding(str(engine.url), self.workspace, row.uid, self.candidate["configuration"])
-                    for registered in session.scalars(select(DataSource)):
-                        binding.validate_source(registered)
-                    for model in (MetaTable, PhysicalOperation):
-                        query = select(model.uid).where(model.data_source_uid != row.uid)
-                        if model is MetaTable:
-                            query = query.where(MetaTable.management_mode != "external_registered")
-                        if session.scalar(query.limit(1)):
-                            raise BootstrapError("Managed runtime catalog references another DataSource.")
-                    if session.scalar(select(MetaTable.uid).where(~MetaTable.data_source_uid.in_(select(DataSource.uid))).limit(1)):
-                        raise BootstrapError("MetaTable references an unregistered DataSource.")
-                    self.uid = row.uid
-                    self.status = "ready"
+            self.inspect_registration()
         except (BootstrapError, CatalogConfigurationError) as exc:
             logger.warning("Runtime database is incompatible", exc_info=True)
             self.status, self.error = "incompatible", str(exc)
@@ -375,6 +350,39 @@ class RuntimeBootstrap:
             logger.info("Runtime database inspection took %.3fs (status %s, schema validated: %s)",
                         perf_counter() - started, self.status, validate_schema)
 
+    def inspect_registration(self):
+        """Fresh binding and reference checks, without repeating schema/history reads."""
+        self.error = None
+        engine = self.engine
+        with self.factory() as session:
+            row = session.scalar(select(DataSource).where(DataSource.is_runtime.__eq__(True)))
+            if row is None:
+                self.status = "registration_required"
+                with engine.connect() as connection:
+                    self.require_prerequisites(connection)
+            elif not self.matches(row):
+                if self.config.local_mode:
+                    self.status = "incompatible"
+                    self.error = "The stored runtime DataSource does not match this connection."
+                else:
+                    # The declaration is authoritative; the deployment updates the record.
+                    self.status = "registration_required"
+                    self.error = "The runtime database declaration changed. The next deployment updates its registration."
+            else:
+                binding = RuntimeStorageBinding(str(engine.url), self.workspace, row.uid, self.candidate["configuration"])
+                for registered in session.scalars(select(DataSource)):
+                    binding.validate_source(registered)
+                for model in (MetaTable, PhysicalOperation):
+                    query = select(model.uid).where(model.data_source_uid != row.uid)
+                    if model is MetaTable:
+                        query = query.where(MetaTable.management_mode != "external_registered")
+                    if session.scalar(query.limit(1)):
+                        raise BootstrapError("Managed runtime catalog references another DataSource.")
+                if session.scalar(select(MetaTable.uid).where(~MetaTable.data_source_uid.in_(select(DataSource.uid))).limit(1)):
+                    raise BootstrapError("MetaTable references an unregistered DataSource.")
+                self.uid = row.uid
+                self.status = "ready"
+
     def require_prerequisites(self, connection):
         # Before any setup DDL: a database the login cannot secure must never be
         # half-initialized. A registered runtime may hold external objects.
@@ -401,42 +409,55 @@ class RuntimeBootstrap:
         try:
             # Check before creating a file or scope marker: an earlier local
             # store must remain untouched even when this target is absent.
-            self.inspect(validate_schema=False)
-            if self.status not in {"migration_required", "registration_required", "ready"}:
-                raise BootstrapError(self.error or "The selected database cannot be migrated.")
-            with self.migration_lock():
-                self.inspect(validate_schema=False)
+            if self.config.local_mode:
+                with bootstrap_phase("local_preflight"):
+                    self.inspect(validate_schema=False)
+                if self.status not in {"migration_required", "registration_required", "ready"}:
+                    raise BootstrapError(self.error or "The selected database cannot be migrated.")
+            with ExitStack() as locks:
+                with bootstrap_phase("lock_wait"):
+                    locks.enter_context(self.migration_lock())
+                with bootstrap_phase("locked_preflight"):
+                    self.inspect(validate_schema=False)
                 if self.status not in {"migration_required", "registration_required", "ready"}:
                     raise BootstrapError(self.error or "The selected database cannot be migrated.")
+                previous = sorted(self.current)
                 self.status = "migrating"
-                upgrade_catalog(self.engine)
+                with bootstrap_phase("catalog_migrations"):
+                    upgrade_catalog(self.engine)
                 # The registry is used only after Alembic created/upgraded it.
                 with self.factory() as session:
-                    row = session.scalar(select(DataSource).where(DataSource.is_runtime.__eq__(True)))
-                    if row is None:
-                        if session.scalar(select(DataSource.uid).where(DataSource.is_default.__eq__(True))):
-                            raise BootstrapError("An existing default requires explicit reconciliation before runtime registration.")
-                        row = DataSource(uid=self.workspace.data_source.uid if self.workspace else uuid4(),
-                            created_by_user_uid=user_uid, **self.candidate, is_runtime=True,
-                            is_default=True, status="AVAILABLE", storage_access_mode="read_write")
-                        session.add(row)
-                    elif not self.matches(row):
-                        if self.config.local_mode:
-                            raise BootstrapError("The existing runtime registration cannot be replaced by this configuration or caller.")
-                        # Same database, new declared connection (host, login Secret, TLS).
-                        row.class_type, row.configuration = self.candidate["class_type"], self.candidate["configuration"]
-                        row.is_default, row.status, row.storage_access_mode = True, "AVAILABLE", "read_write"
-                    from metatables.api.backend.backends.registry import backend_for
-                    session.flush()
-                    backend_for(replace(self.backend.source, uid=row.uid)).access.initialize(session)
-                    credentials = bind_store(self.credential_store, session, user_uid)
-                    if hasattr(credentials, "initialize"):
-                        credentials.initialize()
-                    session.commit()
+                    with bootstrap_phase("runtime_registration"):
+                        row = session.scalar(select(DataSource).where(DataSource.is_runtime.__eq__(True)))
+                        if row is None:
+                            if session.scalar(select(DataSource.uid).where(DataSource.is_default.__eq__(True))):
+                                raise BootstrapError("An existing default requires explicit reconciliation before runtime registration.")
+                            row = DataSource(uid=self.workspace.data_source.uid if self.workspace else uuid4(),
+                                created_by_user_uid=user_uid, **self.candidate, is_runtime=True,
+                                is_default=True, status="AVAILABLE", storage_access_mode="read_write")
+                            session.add(row)
+                        elif not self.matches(row):
+                            if self.config.local_mode:
+                                raise BootstrapError("The existing runtime registration cannot be replaced by this configuration or caller.")
+                            # Same database, new declared connection (host, login Secret, TLS).
+                            row.class_type, row.configuration = self.candidate["class_type"], self.candidate["configuration"]
+                            row.is_default, row.status, row.storage_access_mode = True, "AVAILABLE", "read_write"
+                        from metatables.api.backend.backends.registry import backend_for
+                        session.flush()
+                    with bootstrap_phase("security_reconciliation"):
+                        backend_for(replace(self.backend.source, uid=row.uid)).access.initialize(session)
+                    with bootstrap_phase("credential_store"):
+                        credentials = bind_store(self.credential_store, session, user_uid)
+                        if hasattr(credentials, "initialize"):
+                            credentials.initialize()
+                    with bootstrap_phase("registration_commit"):
+                        session.commit()
                 # upgrade_catalog already verified the complete system schema.
                 # Recheck registration before releasing the lock, without
                 # repeating every table/column/index/constraint lookup.
-                self.inspect(validate_schema=False)
+                self.current = list(self.expected)
+                with bootstrap_phase("registration_verification"):
+                    self.inspect_registration()
                 if self.status != "ready":
                     raise BootstrapError(self.error or "Migration finished but the runtime is not ready.")
         except (BootstrapError, CatalogConfigurationError) as exc:
@@ -462,6 +483,7 @@ class RuntimeBootstrap:
             self.error = (f"MetaTables migration or registration failed ({cause}). The runtime was not activated; "
                           "inspect the database before retrying. The log has the full error.")
             raise BootstrapError(self.error) from None
+        return previous
 
     def persist_selection(self):
         from metatables.api.app.runtime_control import write_state
@@ -563,18 +585,19 @@ def deploy_runtime_database(user_uid):
 
     The workflow's migration Job runs this from the candidate image before the
     API rolls out, every time; any failure blocks the rollout. Access setup is
-    reapplied too, so a rotated login password reaches the per-User roles.
+    freshly checked too; drift and rotated credentials are repaired as needed.
     """
     from metatables.api.app.configuration import DEPLOYMENT_CONFIGURATION
     from metatables.api.app.runtime import RuntimeConfig
     target = RuntimeBootstrap(RuntimeConfig(local_mode=False, configuration_path=DEPLOYMENT_CONFIGURATION), None)
     try:
-        candidate = target.initial_candidate()
-        if candidate is None:
-            raise BootstrapError(target.error)
-        target.configure(candidate, user_uid, validate_schema=False)
-        previous = sorted(target.current)
-        target.migrate(user_uid)
+        with bootstrap_phase("candidate_resolution"):
+            candidate = target.initial_candidate()
+            if candidate is None:
+                raise BootstrapError(target.error)
+        target.configure(candidate, user_uid, validate_schema=False, inspect_database=False)
+        with bootstrap_phase("deployment_migration"):
+            previous = target.migrate(user_uid)
         revisions = sorted(target.current)
         status = "initialized" if not previous else "upgraded" if previous != revisions else "up_to_date"
         return dict(status=status, previous_revisions=previous, revisions=revisions, data_source_uid=str(target.uid))

{mainsequence_metatable-0.1.31.dev58 → mainsequence_metatable-0.1.31.dev60}/src/metatables/api/app/routes/managed_finalize.py

@@ -17,6 +17,7 @@ from sqlalchemy.orm import Session
 from metatables.api.app.auth.boundary import trusted_actor_facts
 from metatables.api.app.auth.local_actor import catalog_session
 from metatables.api.app.routes.table_introspection import data_source_provider
+from metatables.api.backend.backends.access import check_table_security_on_commit
 from metatables.api.backend.backends.operations import (
     PhysicalConnectionUnavailable,
     PhysicalIntrospectionError,
@@ -45,8 +46,11 @@ from metatables.api.backend.persistence.models import (
     TimeIndexTable,
 )
 from metatables.api.backend.persistence.projections import (
+    contract_projection_matches,
+    load_contract_projections,
     materialize_contract_projection,
     resolved_foreign_key_targets,
+    same_contract_state,
 )
 
 logger = logging.getLogger(__name__)
@@ -194,6 +198,7 @@ def finalize_managed(
         finally:
             logger.info("Managed finalization: physical inspection of %s tables took %.3fs",
                         len(eligible), perf_counter() - inspection_started)
+    projections = load_contract_projections(session, [uid for uid, *_ in eligible])
     for uid, table, previous, specialization in eligible:
         try:
             snapshot = snapshots[(table.physical_schema, table.physical_table_name)]
@@ -232,13 +237,24 @@ def finalize_managed(
                 )
                 targets = resolved_foreign_key_targets(
                     session, data_source_uid=table.data_source_uid, contract=contract,
+                    source_table_uid=uid,
                 )
-                session.execute(delete(MetaTableForeignKey).where(MetaTableForeignKey.source_table_uid == uid))
-                session.execute(delete(MetaTableIndex).where(MetaTableIndex.meta_table_uid == uid))
-                session.execute(delete(MetaTableColumn).where(MetaTableColumn.meta_table_uid == uid))
-                session.flush()
-                table.table_contract = contract
-                table.contract_version = contract["version"]
+                replace_projection = not contract_projection_matches(projections[uid], contract, targets)
+                if replace_projection:
+                    session.execute(delete(MetaTableForeignKey).where(MetaTableForeignKey.source_table_uid == uid))
+                    session.execute(delete(MetaTableIndex).where(MetaTableIndex.meta_table_uid == uid))
+                    session.execute(delete(MetaTableColumn).where(MetaTableColumn.meta_table_uid == uid))
+                    session.flush()
+                updates = {
+                    "contract_version": contract["version"],
+                    "physical_schema": contract["physical"]["schema"] or table.physical_schema,
+                    "provisioning_status": "active",
+                    "protect_from_deletion": True,
+                }
+                if not same_contract_state(table.table_contract, contract):
+                    updates["table_contract"] = contract
+                if not same_contract_state(table.introspection_snapshot, snapshot):
+                    updates["introspection_snapshot"] = snapshot
                 authoring = contract.get("authoring") or {}
                 has_time_profile = (
                     contract.get("table_kind") == "time_indexed"
@@ -254,19 +270,22 @@ def finalize_managed(
                         cadence=spec.cadence,
                     )
                     session.add(specialization)
-                    table.kind = "time_index"
-                table.physical_schema = contract["physical"]["schema"] or table.physical_schema
-                convert_time_index_table(source, table, specialization, convert=make_hypertable)
-                table.introspection_snapshot = snapshot
-                table.provisioning_status = "active"
-                table.protect_from_deletion = True
+                    updates["kind"] = "time_index"
                 if "alembic_revision" in payload.model_fields_set:
-                    table.alembic_revision = payload.alembic_revision
+                    updates["alembic_revision"] = payload.alembic_revision
                 if version_root is not None and uid != version_root:
-                    table.alembic_version_meta_table_uid = version_root
-                materialize_contract_projection(
-                    session, table=table, contract=contract, foreign_key_targets=targets
-                )
+                    updates["alembic_version_meta_table_uid"] = version_root
+                # Do not dirty equal-valued rows: pre-flush hooks see assignments
+                # before SQLAlchemy decides whether they need an UPDATE.
+                for field, value in updates.items():
+                    if getattr(table, field) != value:
+                        setattr(table, field, value)
+                convert_time_index_table(source, table, specialization, convert=make_hypertable)
+                if replace_projection:
+                    materialize_contract_projection(
+                        session, table=table, contract=contract, foreign_key_targets=targets
+                    )
+            check_table_security_on_commit(session, table)
             rows.append(_row(table, uid, previous=previous, specialization=specialization,
                              exists=True, finalized=previous != "active"))
         except (CatalogInvariantError, ContractValidationError, IntegrityError) as exc: