mainsequence-metatable 0.1.7.dev9__tar.gz → 0.1.8__tar.gz

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8}/.agents/skills/client/metatables-local-development/SKILL.md

@@ -140,12 +140,16 @@ local rows, catalog UIDs, credentials and fixture data are not promoted by a swi
 When returning to the environment is part of the requested workflow:
 
 1. Finish local tests and stop active writers. In Admin **Settings → Runtime mode**,
-   inspect the displayed hosted environment and choose the intended registered
-   DataSource. Use **Switch to Hosted**. This changes the API worker and selected
-   database; it does not copy or merge the local database. Leave local capability
-   enabled if the developer needs to switch back later.
+   inspect the displayed hosted environment and use **Switch to Hosted**. Hosted
+   opens the runtime database the API's deployment declares, through the same
+   Environment Secret; there is no DataSource to choose and the launcher never
+   migrates it. This changes the API worker and database; it does not copy or merge
+   the local database. Leave local capability enabled if the developer needs to
+   switch back later.
 2. Re-read runtime status and verify `local_mode: false`, the intended verified
-   hosted environment, the expected DataSource and readiness. The same local
+   hosted environment, the expected DataSource and readiness. `migration_required`
+   means the branch has system migrations the deployed API does not have yet; it
+   clears once that code is deployed. The same local
    connection command still reaches this supervised API; its `--local` flag alone
    does not prove SQLite. If the switch fails and the old worker is restored,
    report the actual mode instead of claiming the switch succeeded.

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8}/.agents/skills/client/metatables-meta-tables/SKILL.md

@@ -35,8 +35,9 @@ identity and Git source facts use SDK imports. Table operations use the client's
 with SQLite behind it; ordinary clients never open local table files. Local
 workspaces use one persistent SQLite database for system and user tables per
 checkout; Git branches share it.
-Settings explicitly initializes or selects the runtime DataSource in both modes;
-startup never runs Alembic. See `docs/operations/catalog-migrations.md`.
+A hosted API's deployment declares and initializes its runtime DataSource; Local
+Settings explicitly initializes or selects the SQLite file. Startup never runs
+Alembic. See `docs/operations/catalog-migrations.md`.
 
 ## Authoring decisions
 
@@ -118,6 +119,12 @@ A null cutoff requires a scope. Do not replace that workflow with ad hoc SQL.
 Deletion/cascade requires edit access, relevant protection checks, and explicit
 cascade intent; it is not an administrator-role bypass or migration repair.
 
+On a TimescaleDB DataSource, time-index tables are hypertables. Their compression
+and retention policies are set only through `TimeIndexMetaTable.set_timescale_policies`
+(Writer) and read with `get_timescale_policies`. Do not add policies to table
+contracts or Alembic revisions; to keep them in code, call the client from a
+bootstrap Job. Retention permanently drops older chunks, including backfills.
+
 ## Validation
 
 Check authored models, normalization, physical naming, FK targets, and time grain

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8}/.agents/skills/client/metatables-migrations/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: metatables-migrations
-description: "Create and evolve managed application tables through Alembic migration providers with the installed MetaTables Python client. Covers provider scope, scaffolding, revision authoring, client-side execution, and reservation or finalization failures. Excludes API catalog migrations, client-library implementation, API/server changes and repository tooling."
+description: "Create and evolve managed application tables through Alembic migration providers with the installed MetaTables Python client. Covers provider scope and placement, scaffolding, revision authoring, client-side execution, and reservation or finalization failures. Excludes API catalog migrations, client-library implementation, API/server changes and repository tooling."
 ---
 
 # MetaTables client application migrations
@@ -34,9 +34,10 @@ finalize catalog bindings through the API. Use the
 [table skill](../metatables-meta-tables/SKILL.md) for contract design.
 
 Check `docs/reference/capabilities.md` and `metatables migrations --help` before
-promising a command. The API's own catalog migrations are a separate history that
-an admin runs from Settings; see `docs/operations/catalog-migrations.md`. They
-are not application-provider revisions.
+promising a command. The API's own catalog migrations are a separate history: a
+hosted API's deployment applies them before it rolls out, and Local mode runs them
+from Settings; see `docs/operations/catalog-migrations.md`. They are not
+application-provider revisions.
 
 ## Source and runtime context
 
@@ -55,9 +56,11 @@ are not application-provider revisions.
 1. Define the SQLAlchemy models with stable, application-prefixed physical names.
 2. Identify one provider module, migration namespace, target `MetaData`, model
    registry, and prefixed Alembic version-table binding.
-3. Keep provider scope explicit. Do not scan all imported models or installed
+3. Choose a provider module name that no other installed package uses (see
+   [Provider placement](#provider-placement)).
+4. Keep provider scope explicit. Do not scan all imported models or installed
    packages.
-4. Scaffold only when the application has no provider yet:
+5. Scaffold only when the application has no provider yet:
 
    ```bash
    metatables migrations scaffold \
@@ -65,13 +68,17 @@ are not application-provider revisions.
      --module ledger.migrations \
      --namespace ledger \
      --base ledger.tables:Base \
-     --metadata ledger.tables:Base.metadata
+     --metadata ledger.tables:Base.metadata \
+     --alembic-version-table-name ledger__alembic_version
    ```
 
-5. Edit the generated `registry.py` so it returns exactly the models owned by
+   Always pass `--module` and `--alembic-version-table-name`. Without them the
+   scaffold creates a top-level `migrations` module and the shared
+   `public.alembic_version` table.
+6. Edit the generated `registry.py` so it returns exactly the models owned by
    that migration stream. Scaffolding alone selects no models and creates no
    tables.
-6. Create and review an Alembic revision (defaults to autogeneration):
+7. Create and review an Alembic revision (defaults to autogeneration):
 
    ```bash
    metatables migrations revision \
@@ -83,6 +90,27 @@ Use `--source-root` and `--code-repository-root` when the application does not
 use the default `src/` layout. Keep applied revisions immutable; add a new
 revision for every later schema change.
 
+## Provider placement
+
+The application decides where its provider lives. The module name, however,
+must not collide with any other package installed in the same environment. A
+library's provider ships in its wheel and runs inside its consumers'
+environments, next to providers from other libraries. When two packages install
+the same top-level module, such as `migrations`, the later install overwrites
+the earlier one's files. Imports then load the wrong provider, and Alembic reads
+the wrong revisions.
+
+- Place the provider under an import package the application owns, for example
+  `ledger.migrations`, or give it a name specific to the application, for example
+  `ledger_migrations`.
+- Point `script_location` and `version_location_prefix` at that module. The
+  scaffold derives both from `--module`. A hand-written
+  `build_metatable_migration_provider` call must pass both, because the builder
+  defaults to `migrations:`.
+- Before releasing a package that others install, list the built wheel with
+  `python -m zipfile -l <wheel>`. Its top-level entries must be packages the
+  application owns.
+
 ## Execute with the environment connection
 
 After verifying the intended runtime, use the application's provider reference:
@@ -132,6 +160,13 @@ Read `docs/operations/recovery-and-observability.md`.
 
 - If provider loading fails, verify the import path, model registry, metadata,
   version-table binding, and installed `metatables` version.
+- If the provider loads another package's code or revisions, find the file that
+  its module resolves to with
+  `python -c "import importlib.util as u; print(u.find_spec('migrations').origin)"`,
+  replacing `migrations` with the provider module. A shared module name means
+  another install overwrote it. Move the provider as the
+  [legacy upgrade skill](../metatables-upgrade-legacy-app/SKILL.md) describes;
+  do not edit files in `site-packages`.
 - If reservation fails, compare provider scope and physical identities before
   changing code. Do not create a second catalog row for the same table.
 - If Alembic succeeds but finalization fails, physical DDL may already be
@@ -145,6 +180,6 @@ Read `docs/operations/recovery-and-observability.md`.
 
 ## Validation
 
-Verify provider scope, revision content, the selected environment connection,
-repeat execution, and final active bindings. Report separately what was checked
+Verify provider scope and placement, revision content, the selected environment
+connection, repeat execution, and final active bindings. Report separately what was checked
 offline and what was exercised against a configured API.

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8}/.agents/skills/client/metatables-upgrade-legacy-app/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: metatables-upgrade-legacy-app
-description: "Upgrade consuming applications from MainSequence 8.x MetaTables or older extracted clients to the installed metatables package. Use for legacy imports, removed SQL scopes, repository compilers and application migration-provider integration. Excludes MetaTables client/API implementation and ordinary schema revision authoring."
+description: "Upgrade consuming applications from MainSequence 8.x MetaTables or older extracted clients to the installed metatables package. Use for legacy imports, removed SQL scopes, repository compilers, application migration-provider integration and provider modules that collide with other installed packages. Excludes MetaTables client/API implementation and ordinary schema revision authoring."
 ---
 
 # Upgrade a legacy MetaTables application
@@ -58,11 +58,33 @@ Read `docs/client/query-and-mutate.md` for current query behavior and
 
 ## Upgrade application migration integration
 
-Keep the existing provider package, namespace, model registry, version-table
-binding and applied revision history. Update bootstrap/import wiring to the
-installed client's public migration interfaces. Do not scaffold a replacement
+Keep the existing provider `package=` value, namespace, model registry,
+version-table binding and applied revision history. Update bootstrap/import wiring
+to the installed client's public migration interfaces. Do not scaffold a replacement
 provider, rewrite applied history or stamp the database to bypass an upgrade issue.
 
+### Move a provider whose module name collides
+
+A provider installed under a generic top-level module, such as `migrations`,
+shares files and imports with every other installed package that chose the same
+name. Move it under a module the application owns, for example `ledger.migrations`
+or `ledger_migrations`:
+
+- Move the provider package with its `env.py`, `script.py.mako` and `versions/`
+  tree.
+- Update `script_location`, `version_location_prefix`, the `env.py` import,
+  imports inside the provider, packaging configuration, and every provider
+  reference in code, scripts and jobs.
+- Keep `package=`, `migration_namespace`, the version-table binding and the
+  revision files unchanged. The catalog binds provider tables to
+  `package:namespace` and rejects a reservation from a different key. Alembic
+  stores only revision IDs, so the module path can change.
+- Do not ship a module under the old name. A forwarding module would keep the
+  file collision.
+
+Then confirm that `migrations current` reports the same revision as before the
+move, and that the built wheel no longer installs the old top-level name.
+
 Providers load in the application process from Python references such as
 `ledger.migrations:migration`. The client executes Alembic with the environment
 connection; the API handles authorization, reservations and finalization. The

mainsequence_metatable-0.1.8/.mainsequence/workflows/metatables-api.yaml

@@ -0,0 +1,40 @@
+api_version: "2.3.0"
+name: metatables-api
+resources:
+  - key: metatables-api
+    kind: fastapi
+    spec:
+      source_path: api/metatables/main.py
+      automatic_deployment: true
+      automatic_redeployment:
+        enabled: true
+        tag_regex: null
+      min_scale: 1
+      revision_retention_count: 3
+
+  - key: migrate-system
+    kind: job
+    spec:
+      name: MetaTables system migrations
+      execution_path: jobs/migrate_system.py
+      cpu_request: "1"
+      memory_request: "1"
+      max_runtime_seconds: 1800
+      automatic_redeployment:
+        enabled: true
+        tag_regex: null
+
+execution:
+  steps:
+    image:
+      prepare_image: metatables-api
+
+    migrate:
+      run_job: migrate-system
+      image_from: image
+      needs: [image]
+
+    deploy_api:
+      deploy: metatables-api
+      image_from: image
+      needs: [migrate]

{mainsequence_metatable-0.1.7.dev9/src/mainsequence_metatable.egg-info → mainsequence_metatable-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.7.dev9
+Version: 0.1.8
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT
@@ -14,7 +14,7 @@ Classifier: Operating System :: OS Independent
 Requires-Python: >=3.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: mainsequence[server]<10,>=9.0.1
+Requires-Dist: mainsequence[server]<10,>=9.0.4
 Requires-Dist: alembic<2,>=1.18
 Requires-Dist: cachetools
 Requires-Dist: cryptography<51,>=46

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8}/docs/adr/api/0001-unified-api-storage-and-local-sqlite.md

@@ -28,6 +28,10 @@ does not cache this API's runtime context or select its runtime mode or DataSour
 Updated 2026-09-28: application Environment policy and relationships are removed.
 Amended 2026-09-29: one enforced runtime binding and explicit bootstrap select one
 database for system and user tables.
+Amended 2026-10-02: each runtime mode lists and selects only its own DataSources,
+and inspection checks the login's prerequisites before setup.
+Amended 2026-10-03: the deployment declares the hosted runtime database; Settings
+is read-only in Hosted mode.
 
 Companion decision: [Main Sequence SDK ADR 0035: Independent Git source and platform execution context](https://github.com/mainsequence-sdk/mainsequence-sdk/blob/metatables_removal/docs/adr/0035-independent-git-source-and-platform-context.md).
 The SDK prerequisite is implemented on `metatables_removal`; the current local
@@ -81,6 +85,114 @@ API requirement. The guides and examples describe the implemented shared path.
 
 ## Decision
 
+### The deployment declares the hosted runtime database (amended 2026-10-03)
+
+Implemented 2026-10-03 for [MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13) in the API, the migration Job and the Admin.
+The PostgreSQL transition test and the container suite were updated but not run;
+hosted verification is pending.
+
+Settings used to save the hosted selection in the database it pointed to, with a
+pointer file on the pod. A restarted or additional pod lost that pointer. The
+hosted runtime database is now declared by the deployment:
+
+- **Declaration.** The API's `configuration.yaml`
+  (`metatables/api/metatables/configuration.yaml`) has a `runtime_database`
+  section with the engine, the name of the Environment Secret holding the
+  connection URI, the default schema, the TLS mode and optional certificate
+  Secret names. It holds names and settings, never secret values. Each branch
+  keeps its own file, so each Environment's deployment reads its own file and
+  Secret.
+- **Secret.** The database administrator creates it in each Environment, for
+  example `postgresql://login:password@host:5432/database`. Host, port, database,
+  login and password come only from the URI, certificates only from the named
+  Secrets. The runtime DataSource's `password_secret_uid` is the URI Secret's UID
+  ([ADR 0009](0009-shared-credential-store.md)).
+- **Deployment.** On every deployment the `migrate-system` Job reads the
+  declaration and the Secret. It checks the prerequisites, applies the system
+  migrations, registers the runtime DataSource and reapplies access setup
+  ([ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md)).
+  Pointing the Secret at another database follows the same procedure: that
+  database is initialized and becomes the runtime. Each database keeps its own
+  catalog, and nothing is transferred. When only the connection changes (host,
+  login, TLS), the deployment updates the registration.
+- **API.** Every pod reads the same declaration and Secret at startup and takes
+  the runtime row from the catalog. It never runs DDL and writes no pointer, so
+  restarts and additional pods agree and report the same instance identity
+  ([MetaTables #14](https://github.com/mainsequence-projects/MetaTables/issues/14)).
+  A pod that starts on an unprepared database reports `migration_required` or
+  `registration_required` and serves no application operations. A Secret change
+  therefore takes effect on deployment.
+- **Settings, Hosted mode.** Settings is read-only: it shows the declaration, the
+  resolved connection, the status and the revisions.
+  - Hosted `configure`, `migrate` and `activate` answer 409.
+  - `select` and the pre-activation Hosted registry
+    (`runtime-source-candidates.json`, `hosted_source_uid`) are removed.
+  - Hosted DataSources are registered only in the active runtime catalog.
+  - Settings does not show the migration Job's result; the deployment run
+    reports a failed Job.
+- **Developer launcher, Hosted mode.** The launcher reads the packaged deployment
+  file and the same Secret, so it uses the same database; its own
+  `configuration.yaml` only grants the developer capability. It never migrates. A
+  newer local migration reports `migration_required` until it is deployed.
+
+Local mode is unchanged. For Hosted mode, this supersedes three earlier
+decisions:
+- the Hosted registry and selection in the 2026-10-02 amendment;
+- Settings-initiated hosted initialization in the 2026-09-29 bootstrap amendment;
+- the persisted hosted selection in the 2026-09-29 bootstrap amendment.
+
+### Each runtime mode selects only its own DataSources (amended 2026-10-02)
+
+Hosted rows are superseded by the 2026-10-03 amendment above: Hosted has no
+registry before activation and no selection.
+
+Implemented 2026-10-02 in the API and Admin; the PostgreSQL runtime-transition test
+was updated but not run.
+
+A DataSource belongs to the runtime mode that registered it. Data Sources and
+Settings show only the current mode's registrations:
+
+| Mode | Registrations | Credential references |
+| --- | --- | --- |
+| Local | The workspace SQLite catalog | Local CredentialStore |
+| Hosted | The Hosted registry until activation, then the hosted catalog | SDK Secrets |
+
+Neither mode lists, selects or copies the other mode's registrations. This applies
+ADR 0009's rule against automatic credential transfers to selection. A Hosted
+DataSource is registered and selected after switching to Hosted. Local mode offers
+**Switch to Hosted** without choosing a source; Hosted starts with its saved
+selection or uninitialized, as the supervised-selection amendment below requires.
+
+Gap observed 2026-10-02: in Local mode, the Settings Hosted picker listed the local
+catalog (`GET /data-sources/`). Selecting a TimescaleDB registered there made
+`POST /runtime-bootstrap/hosted/select/` copy the row, with its local password
+reference, into `.local/runtime-source-candidates.json`. SDK Secrets could not
+resolve that reference ("SDK Secret access failed"). Local mode also offered no
+way to register a Hosted source: **Add DataSource** writes the local catalog, and
+the Admin disables **Switch to Hosted** until a Hosted source is selected. The
+operations guide documents this cross-mode flow.
+
+Fix:
+
+1. API: remove Local-mode Hosted preselection: `POST /runtime-bootstrap/hosted/select/`,
+   the Local `hosted_bootstrap` and its runtime-context field. Remove the
+   local-catalog lookup and `remember()` from `registered_hosted_source`. Hosted
+   `POST /runtime-bootstrap/select/` resolves only the Hosted registry or the
+   active hosted catalog.
+2. Admin: in Local mode, Settings enables **Switch to Hosted** with no source
+   picker. The Hosted picker and its **Add DataSource** render only in Hosted
+   mode, where Data Sources is the Hosted registry.
+3. Docs: `docs/operations/local-runtime.md` describes the order Switch to Hosted,
+   register in Data Sources, select in Settings, then run MetaTables migrations
+   or use the existing catalog.
+4. Tests: the Local-mode selection route no longer exists; Hosted selection
+   rejects a local-catalog UID; a fresh Hosted runtime registers a source with an
+   SDK Secret reference and selects it; Settings in Local mode switches without a
+   selection.
+
+Hosted registry entries already copied from a local catalog carry unusable
+references; developers remove them. No data migration is provided.
+
 ### One local runtime across branches (amended 2026-09-30)
 
 MetaTables owns one persistent local SQLite runtime for a checkout. Git branches,
@@ -99,11 +211,19 @@ by startup or migration. This supersedes the previous branch isolation policy.
 
 ### Explicit runtime bootstrap in the selected DataSource (amended 2026-09-29)
 
+Hosted upgrades: before each API rollout, the deployment workflow runs
+`metatables runtime upgrade` from the new image against the active runtime
+([ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md)). Initializing a selected DataSource and startup
+inspection follow the lifecycle below.
+
 This amendment supersedes the separate catalog connection, the two-file SQLite
 binding, automatic local catalog upgrades, and the requirement that switching to
 Hosted already have a migrated catalog. It corrects this ADR's runtime lifecycle;
 it does not introduce a second execution architecture.
 
+For Hosted mode, the 2026-10-03 amendment replaces Settings selection and
+initialization with the deployment's declaration and migration Job.
+
 Settings selects the runtime mode and its DataSource. That database contains both
 MetaTables' system tables and user tables. Local selects one persistent SQLite
 file; Hosted selects PostgreSQL/TimescaleDB. MySQL and MSSQL remain connection-only
@@ -117,6 +237,14 @@ configuration inserted as the runtime DataSource in that database. Failed work
 never activates a partially initialized runtime. A database already initialized
 and compatible can be selected without running migrations.
 
+Amended 2026-10-02: before any runtime is registered, inspection also checks,
+read-only, the login's prerequisites under
+[ADR 0007](0007-database-enforced-table-access.md): `CREATEROLE`, and permission to
+create the `metatables` schema and tables in the default schema. Unmet
+prerequisites make the target incompatible, with each reason shown in Settings,
+so setup never leaves a database half-initialized. A registered runtime is not
+rechecked.
+
 Settings, runtime context, mode selection and bootstrap actions do not depend on
 catalog sessions or catalog memberships. They retain the existing request admission.
 The caller who configures a pending target controls its initialization; an existing
@@ -210,10 +338,18 @@ The supervisor passes mode and transport parameters explicitly to the child;
 `METATABLES_LOCAL_RUNTIME` has no authority. Database credentials remain private
 deployment inputs and are never stored in YAML or sent to Vite.
 
-The runtime descriptor reports capability, switching availability and a fresh
-instance identity. Admin requests carry that identity so stale requests are
-rejected after a restart. During transition the UI blocks work, reloads runtime
-context and remounts cached views. A mode change affects the entire developer API
+The runtime descriptor reports capability, switching availability and an
+instance identity. Admin requests carry that identity so requests made against a
+previous runtime are rejected. During transition the UI blocks work, reloads runtime
+context and remounts cached views.
+
+Amended 2026-10-03 for [MetaTables #14](https://github.com/mainsequence-projects/MetaTables/issues/14):
+the instance identity names the runtime binding, not the process. It is derived
+from the mode, the runtime database's public configuration, the runtime
+DataSource UID, its activation and the catalog revision. Every process and pod
+serving one binding reports the same identity, so an API deploy or restart keeps
+open pages valid; selecting another DataSource, activating it, applying
+migrations or switching mode changes it. A mode change affects the entire developer API
 instance, never an individual user's view of a shared service.
 
 ### Enforced runtime binding (amended 2026-09-29)

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8}/docs/adr/api/0007-database-enforced-table-access.md

@@ -6,6 +6,14 @@
 > conflicting application-migration execution and credential restrictions;
 > governed API operations and MetaTables system migrations remain separate.
 
+> Amendment (2026-10-02): MetaTables secures only what it owns: its `metatables`
+> schema, the tables it manages and the roles it creates. Database-wide defaults
+> and `PUBLIC` access to other owners' objects, including extensions and other
+> applications sharing the database, are the database administrator's
+> responsibility. The login needs `CREATEROLE` and permission to create the
+> `metatables` schema and tables in the DataSource's default schema; it need not
+> own the database.
+
 
 Date: 2026-09-29
 
@@ -160,12 +168,14 @@ client.
 - **Ownership.** Tables are owned by `mt_owner`, a role that cannot log in, because
   an owner holds every privilege. The API's own role manages grants and schema
   changes and never runs caller SQL.
-- **Defaults.** PostgreSQL's default `PUBLIC` privileges are revoked: connecting,
-  creating objects and temporary tables, and executing new functions.
+- **Defaults.** On objects MetaTables owns, PostgreSQL's default `PUBLIC`
+  privileges are revoked, including for future tables and routines. Database-wide
+  defaults (connecting, creating objects and temporary tables) and other owners'
+  objects belong to the database administrator.
 - **Privileged routines.** Restricted caller roles must not be able to execute
   application routines with elevated owner privileges, including
   `SECURITY DEFINER` routines. Initialization establishes this restriction for
-  existing routines as well as future defaults, covering direct grants, inherited
+  the routines MetaTables owns, existing and future, covering direct grants, inherited
   roles and `PUBLIC`. Revoking a direct User grant alone is insufficient if
   another grant still permits execution. Schema changes and extension installation
   preserve the restriction before exposing new or changed routines to callers.
@@ -261,7 +271,8 @@ and [MySQL implicit commits](https://dev.mysql.com/doc/refman/8.4/en/implicit-co
 
 - The explorer can run any PostgreSQL query, and the API has no SQL parser left
   to get wrong.
-- The DataSource login needs `CREATEROLE`.
+- The DataSource login needs `CREATEROLE` and the right to create the `metatables`
+  schema; it does not need to own the database.
 - Physical table names become visible to explorer users.
 - Cascading foreign keys are unavailable where the explorer is enabled.
 - Team membership reaches the database at the pace of the existing one-hour

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8}/docs/adr/api/0008-mysql-mssql-table-workflows.md

@@ -253,6 +253,10 @@ this work must not require recreation or transfer them to another DataSource.
 
 ### First-bootstrap and migration recovery
 
+The deployment workflow's `metatables runtime upgrade` ([ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md))
+runs this same protocol, with its backend locks, progress reconciliation,
+registration and schema verification, before each hosted API rollout.
+
 The shared bootstrap service operates before a DataSource row or the full operation
 journal exists. Startup and Settings inspection remain read-only. Only the explicit
 **Run MetaTables migrations** action may initialize or upgrade the selected database.

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8}/docs/adr/api/0009-shared-credential-store.md

@@ -9,6 +9,10 @@
 
 Date: 2026-09-30
 
+Amended 2026-10-03 ([MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13)): the hosted runtime login's password comes from the
+URI Secret that the deployment declares. See
+[Bootstrap, restart and lifecycle](#bootstrap-restart-and-lifecycle).
+
 Status: Accepted. Implemented by the API CredentialStore providers and catalog
 revision `0006_local_credentials`.
 
@@ -131,17 +135,21 @@ coverage must additionally run on its corresponding host.
 ### Bootstrap, restart and lifecycle
 
 Local SQLite opens using the existing API-owned file binding, before credential
-lookup. Hosted bootstrap resolves the selected connection's UUID through SDK
-Secrets before opening the catalog. Its credential never depends on opening the
-database that needs that password. Saved selection JSON holds only public
-settings and UUID references.
+lookup. Hosted bootstrap reads the Environment Secret named by
+`runtime_database.uri_secret` (amended 2026-10-03). It reads the Secret by name
+through the SDK before opening the catalog. The login and password come from that
+URI, and the runtime DataSource references the Secret's UID as
+`password_secret_uid`. The hosted store serves that UID only as the database
+password and refuses to delete it. Every other reference still resolves through
+SDK Secrets by UID. The credential never depends on opening the database that
+needs it. Local selection JSON holds only public settings and UUID references.
 
 A fresh local catalog must receive explicit system migrations before saving
 sources/credentials. Local saves never substitute JSON for encrypted SQL storage.
 Previously saved local candidate metadata can be materialized by explicit setup,
-retaining DataSource UIDs. Hosted pre-catalog registration continues to use SDK
-Secrets and its existing metadata registry; it becomes the runtime row during
-explicit setup. Startup never silently applies migrations.
+retaining DataSource UIDs. Hosted has no pre-catalog registration: the
+deployment's migration Job registers the declared runtime, and other hosted
+sources are registered in its catalog. Startup never silently applies migrations.
 
 `metatables credentials` provides admin-only local operations:
 

{mainsequence_metatable-0.1.7.dev9 → mainsequence_metatable-0.1.8}/docs/adr/api/0013-application-owned-migrations.md

@@ -80,6 +80,10 @@ dropped tables. Revision files already applied remain immutable.
 
 ## MetaTables system migrations
 
+Hosted system upgrades also run as a deployment prerequisite Job
+([ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md)). Application providers keep their own histories and are not
+part of that Job.
+
 The API continues to own `metatables.api.backend.migrations`, `metatables_catalog_version`
 and the explicit admin bootstrap/upgrade operation in Settings. Application
 migration commands do not invoke that operation. Applications must keep their

mainsequence_metatable-0.1.8/docs/adr/api/0014-main-sequence-release-jobs-and-production-migrations.md

@@ -0,0 +1,98 @@
+# ADR 0014: System migrations in the deployment workflow
+
+Date: 2026-10-02
+
+Status: Accepted. Implemented on `development`; hosted verification pending.
+Amended 2026-10-03 ([MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13)): the Job reads the declared runtime database itself
+and initializes it on the first deployment.
+
+Owners: MetaTables API. Main Sequence owns Job execution, images, deployment and
+platform authentication.
+
+Related: [ADR 0001: Runtime storage and bootstrap](0001-unified-api-storage-and-local-sqlite.md),
+[ADR 0008: Database workflows and recovery](0008-mysql-mssql-table-workflows.md) and
+[ADR 0013: Application-owned migrations](0013-application-owned-migrations.md).
+
+## Context
+
+Main Sequence deploys the hosted API from `.mainsequence/workflows/metatables-api.yaml`.
+Settings runs system migrations inside the running API, and that API can only
+apply the revisions it ships. A release's new revisions must be applied before
+its API serves.
+
+## Decision
+
+### The Job prepares the declared database (amended 2026-10-03)
+
+The Job no longer reads the running API. `metatables runtime upgrade` loads
+`runtime_database` from the candidate image's `configuration.yaml` and the
+Environment Secret it names ([ADR 0001](0001-unified-api-storage-and-local-sqlite.md)).
+Every run executes the same steps:
+
+1. Check that the login can secure the database (ADR 0007 prerequisites).
+2. Apply the system migrations.
+3. Register the runtime DataSource, or update its connection to the declaration.
+4. Reapply access setup. Per-User role passwords derive from the login password,
+   so a rotated password reaches them on the next deployment.
+
+The first deployment therefore initializes the database. A missing declaration
+or Secret, an unreachable database or unmet prerequisites fail the Job and block
+the rollout. The deployment workflow is the authority, so no admin caller is
+required. A new runtime DataSource records the Job's SDK user as its creator.
+The Job prints `initialized`, `upgraded` or `up_to_date` with the revisions.
+
+Settings' **Run MetaTables migrations** and `metatables runtime initialize` remain
+for Local mode only. The original decision follows. The amendment supersedes
+steps 1–2, the skip rules and the Settings initialization paragraph.
+
+### Original decision (2026-10-02)
+
+The deployment workflow upgrades the runtime from the candidate image before the
+rollout:
+
+```text
+candidate image -> migrate-system Job -> API deployment
+```
+
+A failed upgrade blocks the rollout.
+
+The Job, `jobs/migrate_system.py`, runs `metatables runtime upgrade`:
+
+1. Find the Environment's MetaTables API the same way the client does.
+2. Read `GET /runtime-context/` as an organization admin. For admins,
+   `bootstrap.candidate` is the Settings DataSource's public configuration with
+   its Secret references; the Secrets are resolved through the SDK.
+3. Apply this package's system migrations through the shared bootstrap operation:
+   backend lock, progress recovery, runtime registration, access setup and
+   verification. A runtime already at the current revision is left untouched.
+
+The command acts only on an active runtime. With no API deployment or no active
+runtime yet, it succeeds without DDL. The Environment's DataSource is set
+through the deployed API, so if it is set the command migrates when necessary;
+if it is not, it skips migrations and the API deploys. An API release that the
+workflow created but has never deployed counts as no API deployment, so the
+first run deploys the API. A non-admin identity or an unreadable
+context fails before DDL. When it registers the runtime DataSource, it records
+the Job identity's user UID as the creator.
+
+Settings' **Run MetaTables migrations** and `metatables runtime initialize` keep
+initializing a newly selected DataSource: the API serving them is the deployed
+revision and runs its own migrations, so there is no version skew. API startup
+stays inspection-only.
+
+Both targets declare `automatic_redeployment` with `tag_regex: null`, so every
+eligible commit runs the graph. Package publication stays in GitHub Actions.
+
+The upgrade owns only `metatables.api.backend.migrations` and
+`metatables_catalog_version`. Application providers keep their own histories
+under ADR 0013.
+
+## Consequences
+
+- The previous API keeps serving during the upgrade, so each schema change must
+  stay compatible with the previous release. Coordination with its writes is not
+  defined yet.
+- An upgrade followed by a failed deployment leaves the database upgraded. There
+  is no automatic downgrade.
+- The workflow runs only when the platform receives the branch's commit.
+  Verification in a hosted Environment is pending.