mainsequence-metatable 0.1.30.dev57__tar.gz → 0.1.31.dev59__tar.gz

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/.agents/skills/client/metatables-migrations/SKILL.md

@@ -190,12 +190,43 @@ API Writer checks govern connection admission and provider catalog operations.
 The client reserves catalog entries, runs Alembic, closes its physical connection,
 and finalizes contracts through the API. Treat connection material as private.
 
-Python code can call `metatables.upgrade_application("ledger.migrations:migration")`;
-`src/metatables/examples/scripts/setup_metatables.py` shows the pattern, and the
-deployment migration Job uses the same call. Remove the retired
+Applications and consuming libraries use the same public Python entry point in
+an explicit setup function or deployment migration Job:
+
+```python
+from metatables import upgrade_application
+
+result = upgrade_application("ledger.migrations:migration")
+```
+
+`src/metatables/examples/scripts/setup_metatables.py` shows the pattern. Keep
+migration execution out of a library's import-time side effects. Remove the retired
 `application_migration_providers` setting and use Python references instead of aliases.
 Application migration histories remain separate from API system migrations.
 
+## Repeated runs and migration performance
+
+Use the shared `upgrade_application` path for every provider. Table-inventory and
+physical-inspection batching are automatic in the client/API implementation; no
+separate batching import, private backend helper or opt-in flag is needed.
+An unchanged Alembic revision still reconciles physical contracts and refreshes
+table descriptions, labels and column metadata. Do not skip finalization based
+only on `migrated=False` or equal revision heads: missing/unreadable tables must
+still fail, and only explicit removals may delete a catalog binding.
+
+For a slow migration, read the "Migration timings" section in
+`docs/client/deploy-application-migrations.md`. Enable INFO logging for
+`metatables.migrations.runner` to see preparation, database setup, table checks,
+Alembic, finalization and total duration. The API logs inspection and total time
+per finalization batch. Measure these separately from platform job scheduling
+and workflow coordination before attributing the delay to schema validation.
+
+Check the installed client and deployed API versions. Client optimizations need
+a package update and application image rebuild; API batching needs a shared API
+deployment. After installing a release containing the changes, refresh copied
+skills with `metatables copy-metatables-skills --path /path/to/application`.
+This command copies guidance; Python imports alone do not refresh it.
+
 ## Lifecycle invariants
 
 - Provider tables and the Alembic registry are `platform_managed` +

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.30.dev57
+Version: 0.1.31.dev59
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/docs/adr/api/0013-application-owned-migrations.md

@@ -95,6 +95,80 @@ requires inspection and application-owned recovery. No server executor journal i
 claimed for client DDL. Downgrade to base clears the recorded revision and reconciles
 dropped tables. Revision files already applied remain immutable.
 
+### Batched inspection (2026-10-09 amendment)
+
+This is an implementation refinement of the lifecycle above, not a change to
+execution ownership, authorization or the migration API.
+
+- The client reads one table inventory per schema before Alembic and a fresh one
+  afterwards. Missing inventory entries still use `has_table` so views and temporary
+  tables retain their previous behavior. The first pass also rejects unversioned
+  application storage; it is not repeated for that check.
+- The API authorizes every requested binding before physical inspection. It resolves
+  the admitted DataSource once per finalization request and shares a read connection
+  across its tables. PostgreSQL/TimescaleDB use four metadata queries per schema
+  (plus a diagnostic query for missing tables). SQLite/MySQL/SQL Server use SQLAlchemy
+  bulk reflection with dialect fallbacks and preserve their existing projections.
+- Reflection caches last only for that inspection. No snapshot is reused across DDL
+  or requests. An unchanged Alembic revision still finalizes physical contracts and
+  refreshes semantic metadata; it is not evidence that the database has no drift.
+- Missing, inaccessible and foreign-owned tables keep their existing failure
+  semantics. A bulk database error falls back to independent table inspections after
+  closing the failed transaction, so an unrelated table can still finalize. Only
+  explicitly removed, physically absent application tables lose their binding.
+- Client phase timings and API inspection/total timings report where time is spent.
+  They do not include platform job scheduling or workflow coordination.
+
+The client still sends bounded finalization requests (ten tables per batch); the
+HTTP request/response contract is unchanged. Client inventory improvements require
+a package update and application rebuild. API batching benefits existing clients
+after the shared API is deployed, with larger gains for clients sending batches.
+
+### Conditional catalog reconciliation (2026-10-10 amendment)
+
+Reconciliation means making the catalog describe the freshly observed physical
+table while preserving declared semantic metadata. It is additional MetaTables
+behavior around Alembic, not an instruction to reapply migration scripts or rebuild
+identical catalog rows. An unchanged Alembic revision does not prove absence of
+physical drift or completion of an earlier catalog finalization.
+
+```mermaid
+flowchart TD
+    A[Run pending Alembic revisions] --> B[Fresh admission and physical inspection]
+    B --> C[Reconcile physical facts with existing semantic metadata]
+    C --> D{Do stored projections match?}
+    D -->|Yes| E[Keep existing projection rows and UUIDs]
+    D -->|No| F[Replace damaged or changed projections]
+    E --> G[Update only changed contract and lifecycle fields]
+    F --> G
+    G --> H[Fresh security-policy checks and commit]
+```
+
+- Finalization retains the catalog lock, per-table Writer/provider/source checks,
+  fresh physical inspection, owner reachability, foreign-key target validation,
+  missing-table/removal handling, and Timescale hypertable checks. No physical
+  snapshot or authorization result is cached across requests.
+- The API reads existing column, index and foreign-key projections in three batch
+  queries. Comparison includes their scalar fields, full contract fragments and
+  resolved FK target UIDs, and detects missing, extra or damaged rows. Projection
+  row UUIDs and reflection ordering of independent objects are not differences;
+  column ordinals and the order of columns inside keys/indexes remain meaningful.
+- Matching projections are kept, even if only the recorded Alembic revision or
+  lifecycle metadata changes. Changed projections are replaced inside the existing
+  per-table savepoint. Equal-valued contract, snapshot and lifecycle fields are not
+  assigned, avoiding unnecessary ORM dirtiness and metadata/search writes.
+- Hosted/local runtime security-policy checks are explicitly retained at commit
+  even when table metadata is unchanged. Existing policy reconciliation repairs
+  policy differences; equal policies and their JSON permission manifest do not
+  require metadata writes. A failed physical check or conversion still fails
+  rather than being hidden by equality.
+- A healthy unchanged finalization issues no table/projection metadata DML. A
+  matching contract with damaged projections still repairs them. A previously
+  committed migration with failed finalization can retry at the same revision;
+  physical drift is freshly observed and catalog reconciliation completes without
+  replaying already-applied DDL. The response contract and client calls stay the
+  same; no consuming-application opt-in is required.
+
 ## MetaTables system migrations
 
 Hosted system upgrades also run as a deployment prerequisite Job
@@ -132,3 +206,20 @@ connection dialect/TLS options, catalog authorization and runtime hold release.
 SQLite integration tests use disposable files. Hosted driver configuration is
 checked without starting databases. Container/database-matrix tests are on demand;
 backend/runtime tests are not added to required CI checks.
+
+The batching amendment adds non-container checks for query/connection counts,
+fresh reflection after DDL, unchanged-revision reconciliation, authorization before
+inspection, independent failures and dialect fallbacks. PostgreSQL query grouping is
+checked with a recording driver; SQLite executes real local DDL. Hosted performance
+and live MySQL/SQL Server/PostgreSQL verification remain pending.
+
+The conditional-reconciliation amendment adds SQL-recording checks for zero
+metadata writes and stable projection UUIDs on unchanged finalization, damaged
+scalar/fragment and missing/extra projection repair, revision-only updates,
+semantic changes at the same revision, drift plus failed-finalization retry, and
+fresh permission/missing/inaccessible/owner checks, including self-referencing key
+projection repair. A real local-runtime check verifies zero catalog DML on an
+unchanged upgrade while still refreshing physical security policies, and detects
+a newly added trigger that changes write safety without changing the contract.
+Existing local SQLite application-provider tests verify actual physical drift and recovery without
+reapplying DDL. These are focused non-container checks, not new required CI jobs.

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/docs/client/define-and-migrate-tables.md

@@ -89,7 +89,21 @@ metatables --local migrations upgrade --provider ledger.migrations:migration
 The local API must already be running; `--local` selects its connection, so inspect
 runtime status to verify the database actually selected in Settings.
 
-Python setup code can call `metatables.upgrade_application("ledger.migrations:migration")`.
+Applications and libraries use the same public import in their migration Job or
+explicit setup function:
+
+```python
+from metatables import upgrade_application
+
+result = upgrade_application("ledger.migrations:migration")
+```
+
+The shared runner automatically uses the installed client's table inventories and
+the API's batched finalization. No separate batching import or option is needed.
+Keep this call in the explicit migration lifecycle, rather than executing it when
+a library is imported. See [migration timings](deploy-application-migrations.md#migration-timings)
+for diagnostics and the client/API updates needed to receive these optimizations.
+
 For current and downgrade, use `metatables.migrations.runner.run_migration` with
 `operation="current"` or `operation="downgrade", revision="0001"`.
 

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/docs/client/deploy-application-migrations.md

@@ -26,6 +26,28 @@ upgrade` or `downgrade` against a hosted API from a developer machine or agent
 session. Do not migrate at application startup either: several pods would race,
 and a failure would not stop the rollout.
 
+## Migration timings
+
+At INFO level, `metatables.migrations.runner` logs preparation, database setup,
+table-presence checks before and after Alembic, Alembic execution, catalog
+finalization and total duration. The API logs physical inspection and total time
+for each finalization batch. Use these to distinguish database work from platform
+scheduling and workflow coordination.
+
+An already-current database still reconciles the provider's catalog contracts.
+Checks use fresh table inventories and batched physical reflection; an unchanged
+revision does not skip missing-table checks, permissions or metadata refresh.
+PostgreSQL batches metadata queries by schema; other dialects use SQLAlchemy's
+bulk reflection and its supported fallbacks. Caches do not survive the request.
+Client improvements require updating the application's installed MetaTables package
+and rebuilding its image; API improvements require deploying the shared API.
+Applications and libraries continue to use `from metatables import upgrade_application`;
+batching is automatic in this shared path, with no new import or opt-in flag.
+After updating the installed package, refresh its agent guidance with
+`metatables copy-metatables-skills --path /path/to/application`. This copies the
+matching migration skill and its guide snapshots; importing Python code does not
+update a project's copied skills.
+
 ## Declare the migration Job
 
 Add one Job script that applies every provider the application owns, in
@@ -117,6 +139,10 @@ its branch to discover the API; declare it on the Job:
           level: view
 ```
 
+`upgrade_application()` waits up to five minutes for the API to wake or finish
+replacing its pods before it fails with `ApiResolutionError`; other client calls
+wait 30 seconds.
+
 Team membership takes effect on the workload's next request, in the catalog and in
 the database. When a workload User changes, add the new one to the Team; nothing
 in MetaTables changes. A table registered in a namespace the caller already

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/docs/client/installation-and-connection.md

@@ -74,7 +74,7 @@ application migrations, persistence and credentials.
 
 ## Copy client skills into an application
 
-After installing MetaTables, explicitly copy its five client usage skills into a
+After installing MetaTables, explicitly copy its client usage skills into a
 consuming project:
 
 ```bash
@@ -88,12 +88,17 @@ metatables --json copy-metatables-skills --path /path/to/application
 `metatables` does not copy skills.
 
 The destination is `<application>/.agents/skills/metatables/`. It contains the
-local development, table, application migration, time-index updater and legacy
+local development, table, application migration, table discovery, time-index updater and legacy
 application upgrade skills, with matching
 guide/example snapshots in each skill's `references/` folder. The installed client
 version is recorded in `PINNED_FROM.txt`. Skills for API implementation or
 MetaTables project development are outside this bundle.
 
+After upgrading the installed package, run the copy command again to refresh the
+matching guidance. The migration skill includes the public `upgrade_application`
+import, repeated-run behavior and migration timing diagnostics. Python imports
+use the installed library independently of these copied agent instructions.
+
 Running the command again replaces the whole `metatables` namespace: folders the
 installed client does not ship, including any added there, are removed. Keep the
 application's own skills in another `.agents/skills/` namespace; the copy never

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/docs/contributing/documentation.md

@@ -56,7 +56,7 @@ belong to their component documentation and project instructions.
 | --- | --- |
 | `.agents/skills/client/metatables-local-development/SKILL.md` | Local-first application development, runtime/DataSource checks, isolated verification and explicit return to the environment. |
 | `.agents/skills/client/metatables-meta-tables/SKILL.md` | SQLAlchemy contracts, external registration, governed queries and client access operations. |
-| `.agents/skills/client/metatables-migrations/SKILL.md` | Alembic providers, offline revisions, approved API-side execution and migration recovery for managed application tables. |
+| `.agents/skills/client/metatables-migrations/SKILL.md` | Application-owned Alembic providers, offline revisions, shared migration execution, timing diagnostics and recovery for managed application tables. |
 | `.agents/skills/client/metatables-table-discovery/SKILL.md` | Table search, and the authoritative rules for table descriptions, labels and column metadata (ADR 0018). |
 | `.agents/skills/client/metatables-time-index-table-updates/SKILL.md` | Producers, dependencies, incremental frames and existing-table readers. |
 

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/docs/operations/catalog-migrations.md

@@ -10,6 +10,13 @@ new image, and a failure blocks the rollout. Settings is read-only in Hosted mod
 See [the deployment gate](hosted-runtime.md#deployment-gate) and
 [ADR 0014](../adr/api/0014-main-sequence-release-jobs-and-production-migrations.md).
 
+The deployment gate checks revisions and registration during preflight, then
+validates the full system schema once inside the migration runner, including when
+the database is already current. Registration and permission setup finish under
+the same bootstrap lock. Schema reflection is batched across the required tables;
+the job logs the number of tables validated and elapsed time. Startup and explicit
+activation still perform their own fresh schema validation.
+
 Local mode initializes its SQLite file explicitly in Settings:
 
 1. Open **Settings**. Local mode prefills the single local SQLite runtime file,

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mainsequence-metatable"
-version = "0.1.30.dev57"
+version = "0.1.31.dev59"
 description = "Main Sequence MetaTables Python client and standalone API"
 readme = "README.md"
 requires-python = ">=3.13"

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/src/mainsequence_metatable.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.30.dev57
+Version: 0.1.31.dev59
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/src/mainsequence_metatable.egg-info/SOURCES.txt

@@ -249,6 +249,7 @@ src/metatables/api/app/routes/update_statistics.py
 src/metatables/api/backend/__init__.py
 src/metatables/api/backend/backends/__init__.py
 src/metatables/api/backend/backends/access.py
+src/metatables/api/backend/backends/batch_inspection.py
 src/metatables/api/backend/backends/catalog.py
 src/metatables/api/backend/backends/catalog_ddl.py
 src/metatables/api/backend/backends/catalog_types.py
@@ -468,6 +469,7 @@ tests/catalog/test_contract_projection.py
 tests/catalog/test_database_config.py
 tests/catalog/test_index_statistics.py
 tests/catalog/test_migration.py
+tests/catalog/test_schema_validation.py
 tests/contracts/capabilities.json
 tests/contracts/client_signatures.json
 tests/contracts/database_operations.json
@@ -515,6 +517,7 @@ tests/e2e/test_local_actor_client_api.py
 tests/e2e/test_local_cli_workflow.py
 tests/e2e/test_local_client_selection.py
 tests/e2e/test_managed_finalize_client_api.py
+tests/e2e/test_managed_finalize_unchanged.py
 tests/e2e/test_namespaces_admin_http.py
 tests/e2e/test_public_imports.py
 tests/e2e/test_remaining_client_routes.py
@@ -545,6 +548,7 @@ tests/examples/test_documented_examples.py
 tests/examples/test_postgresql_examples.py
 tests/examples/test_setup_launch.py
 tests/examples/test_tutorial.py
+tests/integrations/test_batch_introspection.py
 tests/integrations/test_connection_reuse.py
 tests/integrations/test_credential_store.py
 tests/integrations/test_data_source_rules.py

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/src/metatables/api/app/bootstrap.py

@@ -7,6 +7,7 @@ from contextlib import ExitStack, contextmanager
 from dataclasses import replace
 from pathlib import Path
 from threading import Lock
+from time import perf_counter
 from types import SimpleNamespace
 from typing import Literal
 from uuid import UUID, uuid4
@@ -233,7 +234,7 @@ class RuntimeBootstrap:
         self.reader = None
         self.resources.close()
 
-    def configure(self, body, owner):
+    def configure(self, body, owner, *, validate_schema=True):
         # Validate public configuration before replacing the current candidate.
         values = RuntimeSourceInput.model_validate(body).model_dump(mode="json")
         kind = values["class_type"]
@@ -271,7 +272,7 @@ class RuntimeBootstrap:
         self.candidate, self.owner = values, owner
         self.active, self.uid = False, None
         self.status, self.error, self.current = "unconfigured", None, []
-        self.inspect()
+        self.inspect(validate_schema=validate_schema)
 
     def open_engine(self, *, create=False):
         from metatables.api.backend.backends.registry import backend_for_engine
@@ -288,8 +289,13 @@ class RuntimeBootstrap:
             self.factory = sessionmaker(self.engine, expire_on_commit=False)
         return self.engine
 
-    def inspect(self):
+    def inspect(self, *, validate_schema=True):
+        # Migration preflight defers the full schema check to upgrade_catalog.
+        # Its result can also be reused for registration verification while the
+        # same bootstrap lock is held. Startup and activation always check it;
+        # startup's activation reuses the check startup just ran (MetaTables #55).
         self.error = None
+        started = perf_counter()
         try:
             script = ScriptDirectory.from_config(migration_config())
             self.expected = list(script.get_heads())
@@ -321,8 +327,11 @@ class RuntimeBootstrap:
                     if not self.current:
                         self.require_prerequisites(connection)
                     return
-                from metatables.api.backend.operations.catalog_schema import validate_system_schema
-                validate_system_schema(connection, self.backend.catalog)
+                if validate_schema:
+                    from metatables.api.backend.operations.catalog_schema import (
+                        validate_system_schema,
+                    )
+                    validate_system_schema(connection, self.backend.catalog)
             with self.factory() as session:
                 row = session.scalar(select(DataSource).where(DataSource.is_runtime.__eq__(True)))
                 if row is None:
@@ -362,6 +371,9 @@ class RuntimeBootstrap:
             self.status = "unavailable"
             self.error = (f"Cannot verify the selected database or its Alembic history ({cause}). Check configuration, "
                           "Secret access and the API's migration version. The log has the full error.")
+        finally:
+            logger.info("Runtime database inspection took %.3fs (status %s, schema validated: %s)",
+                        perf_counter() - started, self.status, validate_schema)
 
     def require_prerequisites(self, connection):
         # Before any setup DDL: a database the login cannot secure must never be
@@ -389,11 +401,11 @@ class RuntimeBootstrap:
         try:
             # Check before creating a file or scope marker: an earlier local
             # store must remain untouched even when this target is absent.
-            self.inspect()
+            self.inspect(validate_schema=False)
             if self.status not in {"migration_required", "registration_required", "ready"}:
                 raise BootstrapError(self.error or "The selected database cannot be migrated.")
             with self.migration_lock():
-                self.inspect()
+                self.inspect(validate_schema=False)
                 if self.status not in {"migration_required", "registration_required", "ready"}:
                     raise BootstrapError(self.error or "The selected database cannot be migrated.")
                 self.status = "migrating"
@@ -421,11 +433,17 @@ class RuntimeBootstrap:
                     if hasattr(credentials, "initialize"):
                         credentials.initialize()
                     session.commit()
+                # upgrade_catalog already verified the complete system schema.
+                # Recheck registration before releasing the lock, without
+                # repeating every table/column/index/constraint lookup.
+                self.inspect(validate_schema=False)
+                if self.status != "ready":
+                    raise BootstrapError(self.error or "Migration finished but the runtime is not ready.")
+        except (BootstrapError, CatalogConfigurationError) as exc:
             self.inspect()
-            if self.status != "ready":
-                raise BootstrapError(self.error or "Migration finished but the runtime is not ready.")
-        except BootstrapError:
-            self.inspect()
+            if isinstance(exc, CatalogConfigurationError):
+                self.status, self.error = "incompatible", str(exc)
+                raise BootstrapError(self.error) from exc
             raise
         except Exception as exc:
             self.inspect()
@@ -502,8 +520,11 @@ class RuntimeBootstrap:
                 self.error = message
             raise BootstrapError(message) from None
 
-    def activate(self, request, user_uid=None, *, persist=True):
-        self.inspect()
+    def activate(self, request, user_uid=None, *, persist=True, inspected=False):
+        # ``inspected``: the caller fully inspected this candidate just before,
+        # as startup does right after configure(). Every other activation checks.
+        if not inspected:
+            self.inspect()
         if self.status != "ready":
             raise BootstrapError(self.error or "Run MetaTables migrations before selecting this DataSource.")
         with self.factory() as session:
@@ -551,7 +572,7 @@ def deploy_runtime_database(user_uid):
         candidate = target.initial_candidate()
         if candidate is None:
             raise BootstrapError(target.error)
-        target.configure(candidate, user_uid)
+        target.configure(candidate, user_uid, validate_schema=False)
         previous = sorted(target.current)
         target.migrate(user_uid)
         revisions = sorted(target.current)

{mainsequence_metatable-0.1.30.dev57 → mainsequence_metatable-0.1.31.dev59}/src/metatables/api/app/main.py

@@ -152,7 +152,8 @@ def create_app(config: RuntimeConfig | None = None) -> FastAPI:
         bootstrap = _app.state.runtime.bootstrap
         if bootstrap and bootstrap.status == "ready":
             from types import SimpleNamespace
-            bootstrap.activate(SimpleNamespace(app=_app), persist=False)
+            # initialize_runtime() just validated this candidate (MetaTables #55).
+            bootstrap.activate(SimpleNamespace(app=_app), persist=False, inspected=True)
         control = getattr(_app.state, "runtime_control", None)
         if control is not None:
             control.started()