mainsequence-metatable 0.1.7.dev9__tar.gz → 0.1.8.dev11__tar.gz

mainsequence_metatable-0.1.8.dev11/.mainsequence/workflows/metatables-api.yaml

@@ -0,0 +1,40 @@
+api_version: "2.3.0"
+name: metatables-api
+resources:
+  - key: metatables-api
+    kind: fastapi
+    spec:
+      source_path: api/metatables/main.py
+      automatic_deployment: true
+      automatic_redeployment:
+        enabled: true
+        tag_regex: null
+      min_scale: 1
+      revision_retention_count: 3
+
+  - key: migrate-system
+    kind: job
+    spec:
+      name: MetaTables system migrations
+      execution_path: jobs/migrate_system.py
+      cpu_request: "1"
+      memory_request: "1"
+      max_runtime_seconds: 1800
+      automatic_redeployment:
+        enabled: true
+        tag_regex: null
+
+execution:
+  steps:
+    image:
+      prepare_image: metatables-api
+
+    migrate:
+      run_job: migrate-system
+      image_from: image
+      needs: [image]
+
+    deploy_api:
+      deploy: metatables-api
+      image_from: image
+      needs: [migrate]

{mainsequence_metatable-0.1.7.dev9/src/mainsequence_metatable.egg-info → mainsequence_metatable-0.1.8.dev11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.7.dev9
+Version: 0.1.8.dev11
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/docs/adr/api/0001-unified-api-storage-and-local-sqlite.md

@@ -99,6 +99,11 @@ by startup or migration. This supersedes the previous branch isolation policy.
 
 ### Explicit runtime bootstrap in the selected DataSource (amended 2026-09-29)
 
+Hosted upgrades: before each API rollout, the deployment workflow runs
+`metatables runtime upgrade` from the new image against the active runtime
+([ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md)). Initializing a selected DataSource and startup
+inspection follow the lifecycle below.
+
 This amendment supersedes the separate catalog connection, the two-file SQLite
 binding, automatic local catalog upgrades, and the requirement that switching to
 Hosted already have a migrated catalog. It corrects this ADR's runtime lifecycle;

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/docs/adr/api/0008-mysql-mssql-table-workflows.md

@@ -253,6 +253,10 @@ this work must not require recreation or transfer them to another DataSource.
 
 ### First-bootstrap and migration recovery
 
+The deployment workflow's `metatables runtime upgrade` ([ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md))
+runs this same protocol, with its backend locks, progress reconciliation,
+registration and schema verification, before each hosted API rollout.
+
 The shared bootstrap service operates before a DataSource row or the full operation
 journal exists. Startup and Settings inspection remain read-only. Only the explicit
 **Run MetaTables migrations** action may initialize or upgrade the selected database.

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/docs/adr/api/0013-application-owned-migrations.md

@@ -80,6 +80,10 @@ dropped tables. Revision files already applied remain immutable.
 
 ## MetaTables system migrations
 
+Hosted system upgrades also run as a deployment prerequisite Job
+([ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md)). Application providers keep their own histories and are not
+part of that Job.
+
 The API continues to own `metatables.api.backend.migrations`, `metatables_catalog_version`
 and the explicit admin bootstrap/upgrade operation in Settings. Application
 migration commands do not invoke that operation. Applications must keep their

mainsequence_metatable-0.1.8.dev11/docs/adr/api/0014-main-sequence-release-jobs-and-production-migrations.md

@@ -0,0 +1,67 @@
+# ADR 0014: System migrations in the deployment workflow
+
+Date: 2026-10-02
+
+Status: Accepted. Implemented on `development`; hosted verification pending.
+
+Owners: MetaTables API. Main Sequence owns Job execution, images, deployment and
+platform authentication.
+
+Related: [ADR 0001: Runtime storage and bootstrap](0001-unified-api-storage-and-local-sqlite.md),
+[ADR 0008: Database workflows and recovery](0008-mysql-mssql-table-workflows.md) and
+[ADR 0013: Application-owned migrations](0013-application-owned-migrations.md).
+
+## Context
+
+Main Sequence deploys the hosted API from `.mainsequence/workflows/metatables-api.yaml`.
+Settings runs system migrations inside the running API, and that API can only
+apply the revisions it ships. A release's new revisions must be applied before
+its API serves.
+
+## Decision
+
+The deployment workflow upgrades the runtime from the candidate image before the
+rollout:
+
+```text
+candidate image -> migrate-system Job -> API deployment
+```
+
+A failed upgrade blocks the rollout.
+
+The Job, `jobs/migrate_system.py`, runs `metatables runtime upgrade`:
+
+1. Find the Environment's MetaTables API the same way the client does.
+2. Read `GET /runtime-context/` as an organization admin. For admins,
+   `bootstrap.candidate` is the Settings DataSource's public configuration with
+   its Secret references; the Secrets are resolved through the SDK.
+3. Apply this package's system migrations through the shared bootstrap operation:
+   backend lock, progress recovery, runtime registration, access setup and
+   verification. A runtime already at the current revision is left untouched.
+
+The command acts only on an active runtime. With no API deployment or no active
+runtime yet, it succeeds without DDL. A non-admin identity or an unreadable
+context fails before DDL. When it registers the runtime DataSource, it records
+the Job identity's user UID as the creator.
+
+Settings' **Run MetaTables migrations** and `metatables runtime initialize` keep
+initializing a newly selected DataSource: the API serving them is the deployed
+revision and runs its own migrations, so there is no version skew. API startup
+stays inspection-only.
+
+Both targets declare `automatic_redeployment` with `tag_regex: null`, so every
+eligible commit runs the graph. Package publication stays in GitHub Actions.
+
+The upgrade owns only `metatables.api.backend.migrations` and
+`metatables_catalog_version`. Application providers keep their own histories
+under ADR 0013.
+
+## Consequences
+
+- The previous API keeps serving during the upgrade, so each schema change must
+  stay compatible with the previous release. Coordination with its writes is not
+  defined yet.
+- An upgrade followed by a failed deployment leaves the database upgraded. There
+  is no automatic downgrade.
+- The workflow runs only when the platform receives the branch's commit.
+  Verification in a hosted Environment is pending.

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/docs/adr/index.md

@@ -34,3 +34,4 @@ decisions belong in the SDK repository and are linked as dependencies.
 | [0010: Import external tables and views](api/0010-import-external-tables-and-views.md) | Accepted; implemented (verification boundaries in ADR) | Shared backend discovery, metadata import/refresh, and bounded reads of external tables and views, preserving one runtime catalog and existing grants. |
 | [0012: Bounded data transfer, concurrent execution, and safe retries](api/0012-bounded-data-transfer-and-safe-retries.md) | Accepted; implemented; hosted engine verification pending | API/client row and byte budgets, incremental fetching, concurrent admission, durable chunk receipts, safe recovery, and required user documentation; independent of Artifacts and temporary URLs. |
 | [0013: Application-owned migrations](api/0013-application-owned-migrations.md) | Accepted and implemented | Application-side Alembic, environment database authority, API catalog integration and separate system migrations. |
+| [0014: System migrations in the deployment workflow](api/0014-main-sequence-release-jobs-and-production-migrations.md) | Accepted; implemented on development; hosted verification pending | A Main Sequence Job upgrades the active runtime from the candidate image before each API rollout. |

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/docs/operations/catalog-migrations.md

@@ -1,5 +1,10 @@
 # Initialize and upgrade the runtime database
 
+Hosted upgrades also run automatically: before each API rollout, the deployment
+workflow runs `metatables runtime upgrade` from the new image against the active
+runtime ([ADR 0014](../adr/api/0014-main-sequence-release-jobs-and-production-migrations.md)).
+The operation below initializes a newly selected DataSource.
+
 Settings selects the runtime mode and a DataSource. That database stores MetaTables'
 system tables, including `data_source`, alongside user tables. There is no separate
 catalog connection to configure. Local and hosted use the same initialization flow.

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/docs/operations/hosted-runtime.md

@@ -1,5 +1,11 @@
 # Operate a hosted API
 
+The deployment workflow upgrades the active runtime before each API rollout: its
+migration Job runs `metatables runtime upgrade` from the new image, and a failure
+stops the rollout ([ADR 0014](../adr/api/0014-main-sequence-release-jobs-and-production-migrations.md)).
+Settings and `metatables runtime initialize` still initialize a newly selected
+DataSource.
+
 The repository's `.mainsequence/workflows/metatables-api.yaml` declares the
 automatic FastAPI deployment at `api/metatables/main.py`. This file imports
 `metatables.api.metatables.main:app`; the server and its deployment-specific

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/docs/reference/capabilities.md

@@ -304,4 +304,5 @@ This inventory records the support status of public resource methods and CLI com
 | `init` | tested | visible | Enable local development while preserving settings; no database initialization. |
 | `runtime status` | tested | visible | Read bootstrap and effective runtime state before or after initialization. |
 | `runtime initialize` | tested | visible | Explicit API system migrations under current admin admission. |
+| `runtime upgrade` | tested | visible | Deploy-time upgrade of the active Settings runtime from this package's migrations; skips with no API or no active runtime. |
 | `admin repair` | tested | visible | Explicitly stage pinned source/dependencies; refuse replacement while in use. |

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/docs/reference/cli.md

@@ -27,6 +27,7 @@ The list below contains visible leaf commands. Hidden aliases and unavailable co
 | `metatables migrations upgrade` | target_revision, --provider, --timeout, --ttl-seconds, --json | Run the application's Alembic upgrade and finalize its catalog tables. |
 | `metatables runtime initialize` |  | Explicitly apply the selected API's system migrations. |
 | `metatables runtime status` |  | Read bootstrap state and effective runtime, including before initialization. |
+| `metatables runtime upgrade` |  | Apply this package's system migrations to the active hosted runtime before a rollout. |
 | `metatables serve` | --local, --port, --configuration, --admin, --admin-path, --admin-port, --open-browser, --sdk-session-source | Launch the installed API and optionally the reusable local Admin. |
 | `metatables time-index-table delete` | uid, --full-delete-selected, --full-delete-downstream-tables, --delete-with-no-table, --override-protection, --timeout |  |
 | `metatables time-index-table detail` | uid, --timeout |  |

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/mkdocs.yml

@@ -79,3 +79,4 @@ nav:
           - "0010: Import external tables and views": adr/api/0010-import-external-tables-and-views.md
           - "0012: Bounded data transfer and safe retries": adr/api/0012-bounded-data-transfer-and-safe-retries.md
           - "0013: Application-owned migrations": adr/api/0013-application-owned-migrations.md
+          - "0014: System migrations in the deployment workflow": adr/api/0014-main-sequence-release-jobs-and-production-migrations.md

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mainsequence-metatable"
-version = "0.1.7.dev9"
+version = "0.1.8.dev11"
 description = "Main Sequence MetaTables Python client and standalone API"
 readme = "README.md"
 requires-python = ">=3.13"

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11/src/mainsequence_metatable.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.7.dev9
+Version: 0.1.8.dev11
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/src/mainsequence_metatable.egg-info/SOURCES.txt

@@ -28,6 +28,7 @@ docs/adr/api/0009-shared-credential-store.md
 docs/adr/api/0010-import-external-tables-and-views.md
 docs/adr/api/0012-bounded-data-transfer-and-safe-retries.md
 docs/adr/api/0013-application-owned-migrations.md
+docs/adr/api/0014-main-sequence-release-jobs-and-production-migrations.md
 docs/adr/client/0003-api-endpoint-resolution.md
 docs/adr/client/0011-cli-local-development-with-managed-admin.md
 docs/api/authentication-and-errors.md
@@ -467,6 +468,7 @@ tests/e2e/test_remaining_client_routes.py
 tests/e2e/test_run_graphs.py
 tests/e2e/test_run_logs.py
 tests/e2e/test_runtime_transition.py
+tests/e2e/test_runtime_upgrade.py
 tests/e2e/test_sdk_local_startup.py
 tests/e2e/test_sqlite_runtime.py
 tests/e2e/test_table_delete_client_api.py

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/src/metatables/api/app/bootstrap.py

@@ -455,6 +455,26 @@ class RuntimeBootstrap:
         request.app.state.runtime_control.instance_id = str(uuid4())
 
 
+def upgrade_selected_runtime(candidate, selected_source_uid, user_uid):
+    """Apply this package's system migrations to a hosted runtime selected in Settings.
+
+    A deployed API can only run the revisions it ships, so a new release upgrades
+    the active runtime from its own image before that API rolls out.
+    """
+    from metatables.api.app.runtime import RuntimeConfig
+    target = RuntimeBootstrap(RuntimeConfig(local_mode=False), None)
+    target.selected_source_uid = selected_source_uid
+    try:
+        target.configure(candidate, user_uid)
+        previous = sorted(target.current)
+        if target.status == "ready":
+            return dict(status="up_to_date", revisions=previous)
+        target.migrate(user_uid)
+        return dict(status="upgraded", previous_revisions=previous, revisions=sorted(target.current))
+    finally:
+        target.close()
+
+
 router = APIRouter(prefix="/runtime-bootstrap", tags=["Runtime bootstrap"])
 
 

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/src/metatables/cli/app.py

@@ -231,6 +231,35 @@ def runtime_initialize() -> None:
     _execute(initialize_runtime)
 
 
+@runtime.command("upgrade")
+def runtime_upgrade() -> None:
+    """Apply this package's system migrations to the active hosted runtime before a rollout."""
+    _execute(_upgrade_runtime)
+
+
+def _upgrade_runtime() -> dict:
+    from uuid import UUID
+
+    from metatables.endpoint import ApiNotDeployedError
+    from metatables.runtime import get_runtime_status
+    try:
+        context = get_runtime_status()
+    except ApiNotDeployedError as exc:
+        return {"status": "skipped", "reason": str(exc)}
+    if context.get("local_mode"):
+        raise RuntimeError("Upgrade applies to a hosted runtime; use Settings or 'metatables --local runtime initialize'.")
+    if not context.get("is_admin"):
+        raise RuntimeError("Upgrading the runtime requires an admin caller.")
+    bootstrap = context.get("bootstrap") or {}
+    if not bootstrap.get("active"):
+        return {"status": "skipped", "reason": "The API has no active runtime; initialize it from Settings."}
+    if not bootstrap.get("candidate"):
+        raise RuntimeError("The API did not return its runtime DataSource configuration.")
+    from metatables.api.app.bootstrap import upgrade_selected_runtime
+    return upgrade_selected_runtime(bootstrap["candidate"], bootstrap.get("selected_source_uid"),
+                                    UUID(context["user_uid"]))
+
+
 @contextmanager
 def _client_transport():
     """Configure only MetaTables models; do not redirect SDK platform calls."""
@@ -242,6 +271,11 @@ def _client_transport():
         with api_endpoint_scope():
             yield
         return
+    if os.environ.get("MAINSEQUENCE_AUTH_MODE") in {"runtime_credential", "session_jwt"}:
+        # Deployed runtimes and Main Sequence Jobs keep their platform-issued auth.
+        with api_endpoint_scope():
+            yield
+        return
     from mainsequence.cli.config import get_tokens
     from mainsequence.client.utils import loaders
 

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/src/metatables/endpoint.py

@@ -23,6 +23,10 @@ class ApiResolutionError(RuntimeError):
     """The configured MetaTables API could not be selected unambiguously."""
 
 
+class ApiNotDeployedError(ApiResolutionError):
+    """No MetaTables API deployment exists yet in the caller's Environment."""
+
+
 @dataclass(frozen=True)
 class ApiEndpoint:
     url: str
@@ -181,7 +185,7 @@ def _discover(generation: int, environment_uid: str) -> ApiEndpoint:
     matches = [release for release in candidates
                if environments[release.code_repository_branch_uid] == environment_uid]
     if not matches:
-        raise ApiResolutionError(f"No visible FastAPI deployment named {name!r} in Environment {environment_uid!r}. Apply the API workflow in that Environment.")
+        raise ApiNotDeployedError(f"No visible FastAPI deployment named {name!r} in Environment {environment_uid!r}. Apply the API workflow in that Environment.")
     if len(matches) != 1:
         raise ApiResolutionError(f"More than one visible FastAPI deployment named {name!r} in Environment {environment_uid!r}; remove duplicate deployments in that Environment.")
     release = matches[0]

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/tests/contracts/capabilities.json

@@ -3335,6 +3335,18 @@
         "tests/e2e/test_local_cli_workflow.py::test_local_cli_bootstrap_migrate_write_read_restart"
       ]
     },
+    {
+      "command": "runtime upgrade",
+      "hidden": false,
+      "status": "tested",
+      "note": "Deploy-time upgrade of the active Settings runtime from this package's migrations; skips with no API or no active runtime.",
+      "tests": [
+        "tests/e2e/test_runtime_upgrade.py::test_upgrade_targets_the_active_settings_data_source",
+        "tests/e2e/test_runtime_upgrade.py::test_first_deployment_has_nothing_to_upgrade",
+        "tests/e2e/test_runtime_upgrade.py::test_runtime_without_activation_is_left_to_settings",
+        "tests/e2e/test_runtime_upgrade.py::test_refuses_before_any_ddl"
+      ]
+    },
     {
       "command": "admin repair",
       "hidden": false,

mainsequence_metatable-0.1.8.dev11/tests/e2e/test_runtime_upgrade.py

@@ -0,0 +1,55 @@
+"""`metatables runtime upgrade`: the deploy-time upgrade of the Settings runtime."""
+
+from uuid import UUID
+
+import pytest
+
+import metatables.api.app.bootstrap as bootstrap
+import metatables.runtime as runtime
+from metatables.cli import app as cli
+from metatables.endpoint import ApiNotDeployedError
+
+USER = "6f0c6c4e-8a52-4a8e-9f62-4f2b5a7f0e11"
+CANDIDATE = {"display_name": "Runtime", "class_type": "postgresql", "configuration": {"host": "db"}}
+
+
+def context(**changes):
+    value = {"local_mode": False, "is_admin": True, "user_uid": USER,
+             "bootstrap": {"active": True, "candidate": CANDIDATE, "selected_source_uid": "source"}}
+    return value | changes
+
+
+def no_ddl(*_):
+    pytest.fail("the upgrade must not touch the database")
+
+
+def test_upgrade_targets_the_active_settings_data_source(monkeypatch):
+    calls = []
+    monkeypatch.setattr(runtime, "get_runtime_status", context)
+    monkeypatch.setattr(bootstrap, "upgrade_selected_runtime", lambda *args: calls.append(args) or {"status": "upgraded"})
+    assert cli._upgrade_runtime() == {"status": "upgraded"}
+    assert calls == [(CANDIDATE, "source", UUID(USER))]
+
+
+def test_first_deployment_has_nothing_to_upgrade(monkeypatch):
+    def not_deployed():
+        raise ApiNotDeployedError("No visible FastAPI deployment named 'metatables'.")
+
+    monkeypatch.setattr(runtime, "get_runtime_status", not_deployed)
+    monkeypatch.setattr(bootstrap, "upgrade_selected_runtime", no_ddl)
+    assert cli._upgrade_runtime()["status"] == "skipped"
+
+
+def test_runtime_without_activation_is_left_to_settings(monkeypatch):
+    monkeypatch.setattr(runtime, "get_runtime_status", lambda: context(bootstrap={"active": False, "candidate": CANDIDATE}))
+    monkeypatch.setattr(bootstrap, "upgrade_selected_runtime", no_ddl)
+    assert cli._upgrade_runtime()["status"] == "skipped"
+
+
+@pytest.mark.parametrize(("changes", "message"), [({"is_admin": False}, "admin"), ({"local_mode": True}, "hosted"),
+                                                  ({"bootstrap": {"active": True}}, "configuration")])
+def test_refuses_before_any_ddl(monkeypatch, changes, message):
+    monkeypatch.setattr(runtime, "get_runtime_status", lambda: context(**changes))
+    monkeypatch.setattr(bootstrap, "upgrade_selected_runtime", no_ddl)
+    with pytest.raises(RuntimeError, match=message):
+        cli._upgrade_runtime()

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8.dev11}/uv.lock

@@ -632,7 +632,7 @@ server = [
 
 [[package]]
 name = "mainsequence-metatable"
-version = "0.1.7"
+version = "0.1.8"
 source = { editable = "." }
 dependencies = [
     { name = "alembic" },

mainsequence_metatable-0.1.7.dev9/.mainsequence/workflows/metatables-api.yaml

@@ -1,10 +0,0 @@
-api_version: "2.3.0"
-name: metatables-api
-resources:
-  - key: metatables-api
-    kind: fastapi
-    spec:
-      source_path: api/metatables/main.py
-      automatic_deployment: true
-      min_scale: 1
-      revision_retention_count: 3