mainsequence-metatable 0.1.8.dev17__tar.gz → 0.1.8.dev18__tar.gz

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.8.dev18}/.agents/skills/client/metatables-local-development/SKILL.md

@@ -140,12 +140,16 @@ local rows, catalog UIDs, credentials and fixture data are not promoted by a swi
 When returning to the environment is part of the requested workflow:
 
 1. Finish local tests and stop active writers. In Admin **Settings → Runtime mode**,
-   inspect the displayed hosted environment and choose the intended registered
-   DataSource. Use **Switch to Hosted**. This changes the API worker and selected
-   database; it does not copy or merge the local database. Leave local capability
-   enabled if the developer needs to switch back later.
+   inspect the displayed hosted environment and use **Switch to Hosted**. Hosted
+   opens the runtime database the API's deployment declares, through the same
+   Environment Secret; there is no DataSource to choose and the launcher never
+   migrates it. This changes the API worker and database; it does not copy or merge
+   the local database. Leave local capability enabled if the developer needs to
+   switch back later.
 2. Re-read runtime status and verify `local_mode: false`, the intended verified
-   hosted environment, the expected DataSource and readiness. The same local
+   hosted environment, the expected DataSource and readiness. `migration_required`
+   means the branch has system migrations the deployed API does not have yet; it
+   clears once that code is deployed. The same local
    connection command still reaches this supervised API; its `--local` flag alone
    does not prove SQLite. If the switch fails and the old worker is restored,
    report the actual mode instead of claiming the switch succeeded.

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.8.dev18}/.agents/skills/client/metatables-meta-tables/SKILL.md

@@ -35,8 +35,9 @@ identity and Git source facts use SDK imports. Table operations use the client's
 with SQLite behind it; ordinary clients never open local table files. Local
 workspaces use one persistent SQLite database for system and user tables per
 checkout; Git branches share it.
-Settings explicitly initializes or selects the runtime DataSource in both modes;
-startup never runs Alembic. See `docs/operations/catalog-migrations.md`.
+A hosted API's deployment declares and initializes its runtime DataSource; Local
+Settings explicitly initializes or selects the SQLite file. Startup never runs
+Alembic. See `docs/operations/catalog-migrations.md`.
 
 ## Authoring decisions
 
@@ -118,6 +119,12 @@ A null cutoff requires a scope. Do not replace that workflow with ad hoc SQL.
 Deletion/cascade requires edit access, relevant protection checks, and explicit
 cascade intent; it is not an administrator-role bypass or migration repair.
 
+On a TimescaleDB DataSource, time-index tables are hypertables. Their compression
+and retention policies are set only through `TimeIndexMetaTable.set_timescale_policies`
+(Writer) and read with `get_timescale_policies`. Do not add policies to table
+contracts or Alembic revisions; to keep them in code, call the client from a
+bootstrap Job. Retention permanently drops older chunks, including backfills.
+
 ## Validation
 
 Check authored models, normalization, physical naming, FK targets, and time grain

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.8.dev18}/.agents/skills/client/metatables-migrations/SKILL.md

@@ -34,9 +34,10 @@ finalize catalog bindings through the API. Use the
 [table skill](../metatables-meta-tables/SKILL.md) for contract design.
 
 Check `docs/reference/capabilities.md` and `metatables migrations --help` before
-promising a command. The API's own catalog migrations are a separate history that
-an admin runs from Settings; see `docs/operations/catalog-migrations.md`. They
-are not application-provider revisions.
+promising a command. The API's own catalog migrations are a separate history: a
+hosted API's deployment applies them before it rolls out, and Local mode runs them
+from Settings; see `docs/operations/catalog-migrations.md`. They are not
+application-provider revisions.
 
 ## Source and runtime context
 

{mainsequence_metatable-0.1.8.dev17/src/mainsequence_metatable.egg-info → mainsequence_metatable-0.1.8.dev18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.8.dev17
+Version: 0.1.8.dev18
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.8.dev18}/docs/adr/api/0001-unified-api-storage-and-local-sqlite.md

@@ -30,6 +30,8 @@ Amended 2026-09-29: one enforced runtime binding and explicit bootstrap select o
 database for system and user tables.
 Amended 2026-10-02: each runtime mode lists and selects only its own DataSources,
 and inspection checks the login's prerequisites before setup.
+Amended 2026-10-03: the deployment declares the hosted runtime database; Settings
+is read-only in Hosted mode.
 
 Companion decision: [Main Sequence SDK ADR 0035: Independent Git source and platform execution context](https://github.com/mainsequence-sdk/mainsequence-sdk/blob/metatables_removal/docs/adr/0035-independent-git-source-and-platform-context.md).
 The SDK prerequisite is implemented on `metatables_removal`; the current local
@@ -83,8 +85,67 @@ API requirement. The guides and examples describe the implemented shared path.
 
 ## Decision
 
+### The deployment declares the hosted runtime database (amended 2026-10-03)
+
+Implemented 2026-10-03 for [MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13) in the API, the migration Job and the Admin.
+The PostgreSQL transition test and the container suite were updated but not run;
+hosted verification is pending.
+
+Settings used to save the hosted selection in the database it pointed to, with a
+pointer file on the pod. A restarted or additional pod lost that pointer. The
+hosted runtime database is now declared by the deployment:
+
+- **Declaration.** The API's `configuration.yaml`
+  (`metatables/api/metatables/configuration.yaml`) has a `runtime_database`
+  section with the engine, the name of the Environment Secret holding the
+  connection URI, the default schema, the TLS mode and optional certificate
+  Secret names. It holds names and settings, never secret values. Each branch
+  keeps its own file, so each Environment's deployment reads its own file and
+  Secret.
+- **Secret.** The database administrator creates it in each Environment, for
+  example `postgresql://login:password@host:5432/database`. Host, port, database,
+  login and password come only from the URI, certificates only from the named
+  Secrets. The runtime DataSource's `password_secret_uid` is the URI Secret's UID
+  ([ADR 0009](0009-shared-credential-store.md)).
+- **Deployment.** On every deployment the `migrate-system` Job reads the
+  declaration and the Secret. It checks the prerequisites, applies the system
+  migrations, registers the runtime DataSource and reapplies access setup
+  ([ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md)).
+  Pointing the Secret at another database follows the same procedure: that
+  database is initialized and becomes the runtime. Each database keeps its own
+  catalog, and nothing is transferred. When only the connection changes (host,
+  login, TLS), the deployment updates the registration.
+- **API.** Every pod reads the same declaration and Secret at startup and takes
+  the runtime row from the catalog. It never runs DDL and writes no pointer, so
+  restarts and additional pods agree and report the same instance identity
+  ([MetaTables #14](https://github.com/mainsequence-projects/MetaTables/issues/14)).
+  A pod that starts on an unprepared database reports `migration_required` or
+  `registration_required` and serves no application operations. A Secret change
+  therefore takes effect on deployment.
+- **Settings, Hosted mode.** Settings is read-only: it shows the declaration, the
+  resolved connection, the status and the revisions.
+  - Hosted `configure`, `migrate` and `activate` answer 409.
+  - `select` and the pre-activation Hosted registry
+    (`runtime-source-candidates.json`, `hosted_source_uid`) are removed.
+  - Hosted DataSources are registered only in the active runtime catalog.
+  - Settings does not show the migration Job's result; the deployment run
+    reports a failed Job.
+- **Developer launcher, Hosted mode.** The launcher reads the packaged deployment
+  file and the same Secret, so it uses the same database; its own
+  `configuration.yaml` only grants the developer capability. It never migrates. A
+  newer local migration reports `migration_required` until it is deployed.
+
+Local mode is unchanged. For Hosted mode, this supersedes three earlier
+decisions:
+- the Hosted registry and selection in the 2026-10-02 amendment;
+- Settings-initiated hosted initialization in the 2026-09-29 bootstrap amendment;
+- the persisted hosted selection in the 2026-09-29 bootstrap amendment.
+
 ### Each runtime mode selects only its own DataSources (amended 2026-10-02)
 
+Hosted rows are superseded by the 2026-10-03 amendment above: Hosted has no
+registry before activation and no selection.
+
 Implemented 2026-10-02 in the API and Admin; the PostgreSQL runtime-transition test
 was updated but not run.
 
@@ -160,6 +221,9 @@ binding, automatic local catalog upgrades, and the requirement that switching to
 Hosted already have a migrated catalog. It corrects this ADR's runtime lifecycle;
 it does not introduce a second execution architecture.
 
+For Hosted mode, the 2026-10-03 amendment replaces Settings selection and
+initialization with the deployment's declaration and migration Job.
+
 Settings selects the runtime mode and its DataSource. That database contains both
 MetaTables' system tables and user tables. Local selects one persistent SQLite
 file; Hosted selects PostgreSQL/TimescaleDB. MySQL and MSSQL remain connection-only
@@ -274,10 +338,18 @@ The supervisor passes mode and transport parameters explicitly to the child;
 `METATABLES_LOCAL_RUNTIME` has no authority. Database credentials remain private
 deployment inputs and are never stored in YAML or sent to Vite.
 
-The runtime descriptor reports capability, switching availability and a fresh
-instance identity. Admin requests carry that identity so stale requests are
-rejected after a restart. During transition the UI blocks work, reloads runtime
-context and remounts cached views. A mode change affects the entire developer API
+The runtime descriptor reports capability, switching availability and an
+instance identity. Admin requests carry that identity so requests made against a
+previous runtime are rejected. During transition the UI blocks work, reloads runtime
+context and remounts cached views.
+
+Amended 2026-10-03 for [MetaTables #14](https://github.com/mainsequence-projects/MetaTables/issues/14):
+the instance identity names the runtime binding, not the process. It is derived
+from the mode, the runtime database's public configuration, the runtime
+DataSource UID, its activation and the catalog revision. Every process and pod
+serving one binding reports the same identity, so an API deploy or restart keeps
+open pages valid; selecting another DataSource, activating it, applying
+migrations or switching mode changes it. A mode change affects the entire developer API
 instance, never an individual user's view of a shared service.
 
 ### Enforced runtime binding (amended 2026-09-29)

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.8.dev18}/docs/adr/api/0009-shared-credential-store.md

@@ -9,6 +9,10 @@
 
 Date: 2026-09-30
 
+Amended 2026-10-03 ([MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13)): the hosted runtime login's password comes from the
+URI Secret that the deployment declares. See
+[Bootstrap, restart and lifecycle](#bootstrap-restart-and-lifecycle).
+
 Status: Accepted. Implemented by the API CredentialStore providers and catalog
 revision `0006_local_credentials`.
 
@@ -131,17 +135,21 @@ coverage must additionally run on its corresponding host.
 ### Bootstrap, restart and lifecycle
 
 Local SQLite opens using the existing API-owned file binding, before credential
-lookup. Hosted bootstrap resolves the selected connection's UUID through SDK
-Secrets before opening the catalog. Its credential never depends on opening the
-database that needs that password. Saved selection JSON holds only public
-settings and UUID references.
+lookup. Hosted bootstrap reads the Environment Secret named by
+`runtime_database.uri_secret` (amended 2026-10-03). It reads the Secret by name
+through the SDK before opening the catalog. The login and password come from that
+URI, and the runtime DataSource references the Secret's UID as
+`password_secret_uid`. The hosted store serves that UID only as the database
+password and refuses to delete it. Every other reference still resolves through
+SDK Secrets by UID. The credential never depends on opening the database that
+needs it. Local selection JSON holds only public settings and UUID references.
 
 A fresh local catalog must receive explicit system migrations before saving
 sources/credentials. Local saves never substitute JSON for encrypted SQL storage.
 Previously saved local candidate metadata can be materialized by explicit setup,
-retaining DataSource UIDs. Hosted pre-catalog registration continues to use SDK
-Secrets and its existing metadata registry; it becomes the runtime row during
-explicit setup. Startup never silently applies migrations.
+retaining DataSource UIDs. Hosted has no pre-catalog registration: the
+deployment's migration Job registers the declared runtime, and other hosted
+sources are registered in its catalog. Startup never silently applies migrations.
 
 `metatables credentials` provides admin-only local operations:
 

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.8.dev18}/docs/adr/api/0014-main-sequence-release-jobs-and-production-migrations.md

@@ -3,6 +3,8 @@
 Date: 2026-10-02
 
 Status: Accepted. Implemented on `development`; hosted verification pending.
+Amended 2026-10-03 ([MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13)): the Job reads the declared runtime database itself
+and initializes it on the first deployment.
 
 Owners: MetaTables API. Main Sequence owns Job execution, images, deployment and
 platform authentication.
@@ -20,6 +22,31 @@ its API serves.
 
 ## Decision
 
+### The Job prepares the declared database (amended 2026-10-03)
+
+The Job no longer reads the running API. `metatables runtime upgrade` loads
+`runtime_database` from the candidate image's `configuration.yaml` and the
+Environment Secret it names ([ADR 0001](0001-unified-api-storage-and-local-sqlite.md)).
+Every run executes the same steps:
+
+1. Check that the login can secure the database (ADR 0007 prerequisites).
+2. Apply the system migrations.
+3. Register the runtime DataSource, or update its connection to the declaration.
+4. Reapply access setup. Per-User role passwords derive from the login password,
+   so a rotated password reaches them on the next deployment.
+
+The first deployment therefore initializes the database. A missing declaration
+or Secret, an unreachable database or unmet prerequisites fail the Job and block
+the rollout. The deployment workflow is the authority, so no admin caller is
+required. A new runtime DataSource records the Job's SDK user as its creator.
+The Job prints `initialized`, `upgraded` or `up_to_date` with the revisions.
+
+Settings' **Run MetaTables migrations** and `metatables runtime initialize` remain
+for Local mode only. The original decision follows. The amendment supersedes
+steps 1–2, the skip rules and the Settings initialization paragraph.
+
+### Original decision (2026-10-02)
+
 The deployment workflow upgrades the runtime from the candidate image before the
 rollout:
 

mainsequence_metatable-0.1.8.dev18/docs/adr/api/0015-timescale-compression-and-retention-policies.md

@@ -0,0 +1,192 @@
+# ADR 0015: TimescaleDB compression and retention policies
+
+Date: 2026-10-02
+
+Status: Accepted.
+
+Implementation status: Implemented on the `timescale-policies` branches of the API
+and the Admin, with focused tests. TimescaleDB container verification is pending.
+
+Owner: MetaTables API. The Admin owns the screens described here.
+
+Related decisions: [ADR 0002: Table ownership](0002-application-administration-and-table-ownership.md),
+[ADR 0007: Database-enforced access](0007-database-enforced-table-access.md) (amended
+below), [ADR 0012: Safe retries](0012-bounded-data-transfer-and-safe-retries.md) and
+[ADR 0013: Application-owned migrations](0013-application-owned-migrations.md).
+
+## Context
+
+Time-index tables on `timescale_db` DataSources grow without bound. Users need to
+compress older data and drop expired data on a schedule, without database access.
+
+Source inspection on 2026-10-02 found:
+
+- The Admin's Timescale Policies tab calls `GET/PATCH meta-tables/{uid}/policies`
+  and appears only when a table reports `capabilities.timescale_policies`. The API
+  provides neither, so the tab never appears.
+- No current path creates hypertables. `storage_layout.partitioning.strategy`
+  accepts `timescale_hypertable`, but only the legacy
+  `create_multi_index_table_for_metadata` calls `create_hypertable`. Timescale
+  policies require hypertables.
+- The legacy policy helpers in `adapters/maintenance/timescaledb.py` have no route
+  callers. They sleep between steps, remove and re-add policies in separate
+  commits, and some build SQL with f-strings.
+- TimescaleDB creates a policy job owned by the hypertable owner and refuses to
+  create it when that role lacks `LOGIN`. ADR 0007 makes `mt_owner`, a `NOLOGIN`
+  role, the owner of every managed table.
+
+## Decision
+
+### Core model
+
+**TimescaleDB stores the policy; MetaTables keeps no copy.** A policy is a
+Timescale background job. Timescale already schedules and runs it, so MetaTables
+adds no scheduler and no policy table. The API authorizes in the catalog, reads
+jobs from `timescaledb_information`, and writes with Timescale's policy functions.
+
+Policies are set only through the API and the Admin. An application that wants
+its policies in code runs a Job at bootstrap that calls the client method. Table
+contracts and application migrations carry no policy fields.
+
+| Policy | User sets | MetaTables derives |
+| --- | --- | --- |
+| Compression | `after`; optional `schedule_interval`, `initial_start`, `timezone` | `segmentby` = identity dimensions; `orderby` = time index `DESC` |
+| Retention | `after`; optional `schedule_interval`, `initial_start`, `timezone` | — |
+
+- `after: null` removes that policy. Removing compression leaves already
+  compressed chunks compressed.
+- If both are set, retention `after` must be longer than compression `after`.
+- Each change runs in one transaction with parameterized SQL and no sleeps.
+- Dropping a table removes its jobs.
+- TimescaleDB 2.11 or later is required, since it accepts inserts, updates,
+  deletes and upserts on compressed chunks. Uploads and tail deletes therefore
+  need no decompression step.
+
+### Eligible tables
+
+A table can have policies when:
+
+1. its DataSource is `timescale_db`;
+2. it is a time-index table and not `external_registered`;
+3. its physical table is a hypertable; and
+4. the installed TimescaleDB version is supported.
+
+Conditions 1–2 are catalog facts. They set `capabilities.timescale_policies` on
+the table response without opening a database connection. Conditions 3–4 are
+checked live when policies are read or written. Saving additionally requires a
+`read_write` DataSource. Tables on any other engine never show policies.
+
+### Hypertables
+
+On a `timescale_db` DataSource, managed finalization converts each new time-index
+table into a hypertable on its time index (`create_hypertable`, `if_not_exists`,
+`migrate_data`). Strategy `backend_default` means hypertable there; `none` opts
+out. `partitioning.strategy` is the only switch, so the duplicate
+`backend_hints.timescale.partitioning` hint and `time_index_table.partition_strategy`
+are removed. Chunk interval stays at Timescale's default. Every unique constraint
+must include the time index; the storage layout's `uniqueness` already does.
+
+### Authorization and ownership
+
+Writer on the table is the only permission check: a Writer can finalize the
+hypertable and set or remove both policies. Readers can view policies and job
+status. Organization admins also see every job on the DataSource.
+
+**Amendment to ADR 0007:** on `timescale_db` DataSources, hypertables are owned
+by the API's database login instead of `mt_owner`, so their policy jobs run as
+that login. Access setup leaves hypertable ownership unchanged instead of handing
+the tables to `mt_owner`. The API's login already owns the database and is a member of
+`mt_owner`, so this grants it nothing new. Reader/Writer grants, caller SQL roles,
+application migrations and other engines are unchanged. If the DataSource later
+uses a different login, existing jobs keep running as the old one until the
+tables are reassigned.
+
+Each policy change is recorded in the `physical_operation` journal as
+`timescale_policy`, with the requesting User. System migration
+`0009_timescale_policies` adds that kind, drops `time_index_table.partition_strategy`
+and removes the Timescale hint from stored layouts; existing runtimes upgrade
+through Settings.
+
+### API
+
+```text
+TimescalePolicy        { after, schedule_interval?, initial_start?, timezone? }
+TimescaleTablePolicies { compression, retention,
+                         eligibility { eligible, reason? },               read-only
+                         jobs [TimescaleJob],                             read-only
+                         compression_stats { before_bytes, after_bytes,
+                                             compressed_chunks, total_chunks } }  read-only
+TimescaleJob           { job_id, kind, table_uid?, hypertable, scheduled, status,
+                         last_run_status, last_successful_finish, next_start,
+                         total_failures, last_error? }
+```
+
+Intervals use PostgreSQL interval text such as `30 days`; the database validates them.
+
+| Endpoint | Access | Behavior |
+| --- | --- | --- |
+| `GET /meta-tables/{uid}/timescale-policies/` | Reader | Policies, eligibility, jobs and compression stats. |
+| `PUT /meta-tables/{uid}/timescale-policies/` | Writer | Replaces both policies; idempotent, safe to retry. Returns the GET body. |
+| `GET /data-sources/{uid}/timescale-jobs/` | Any User who can see the source | Jobs for tables the caller can read; admins see all. `timescale_db` only. |
+
+Errors: `404 timescale_policies_unavailable` for other engines,
+`403 write_access_required`, `409 timescale_not_hypertable`,
+`409 timescale_version_unsupported`, `409 timescale_extension_missing`, the existing
+DataSource access codes for read-only or disabled sources, and `422` for an invalid
+interval or timezone or `timescale_retention_not_after_compression`.
+
+The Python client exposes `get_timescale_policies()` and `set_timescale_policies()`
+on time-index tables.
+
+### Admin
+
+- **Table → Timescale Policies** (the existing tab, reworked): Compression and
+  Retention cards, each with an on/off switch, an `after` number with a unit
+  (days, weeks, months), and an Advanced section for schedule, start and timezone.
+  Each card shows its job's status (Scheduled, Running, Failed, Paused), last
+  success, next run and failures. Compression shows size before and after.
+  Saving retention asks for confirmation, naming the cutoff date and stating that
+  older chunks are permanently dropped on every run and that older backfills are
+  dropped too. Readers see the tab read-only. An ineligible table shows the reason.
+- **DataSource → Jobs** (`timescale_db` only): a table of jobs — table link,
+  policy, status, last run, next run, failures — with a failed-only filter and
+  Refresh, under Policies and Failed jobs counts.
+
+Screens use Command Center SDK components and theme tokens only.
+
+### Monitoring
+
+Monitoring is read on demand from `jobs`, `job_stats`, `job_errors` and
+`hypertable_compression_stats`. MetaTables stores no job history, polls nothing,
+and sends no alerts.
+
+## Consequences
+
+- One source of truth: a policy changed directly in the database appears in the
+  Admin as it is.
+- The legacy Timescale policy, decompression and hypertable helpers and their
+  runtime registrations are deleted.
+- Retention is destructive and available to every Writer; the confirmation is
+  the safeguard.
+- Policy jobs belong to the API's login. Rotating its password does not affect
+  them; no role gains `LOGIN`.
+
+## Alternatives considered
+
+- **Policies in table contracts or migrations.** Two authorities would overwrite
+  each other. A bootstrap Job calling the API covers code-defined policies.
+- **A catalog copy with a reconciler.** It duplicates Timescale's job store and
+  adds drift handling.
+- **A MetaTables scheduler.** Timescale already schedules policy jobs.
+- **`mt_owner` with `LOGIN` and no password.** It changes a login setting and
+  depends on the server never accepting passwordless authentication for that role.
+
+## Verification
+
+Focused tests cover eligibility, interval validation, authorization, the request
+contract and the derived compression settings, using a fake backend. The
+TimescaleDB container suite, run only on explicit request, verifies hypertable
+conversion, policy add/replace/remove, job ownership by the API's login, Reader and
+Writer grants reaching chunks and compressed chunks, and upserts into compressed
+chunks. Admin checks cover tab visibility, read-only mode and the retention
+confirmation.

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.8.dev18}/docs/adr/index.md

@@ -23,7 +23,7 @@ decisions belong in the SDK repository and are linked as dependencies.
 
 | ADR | Status | Scope |
 | --- | --- | --- |
-| [0001: One API execution path with SQLite for local development](api/0001-unified-api-storage-and-local-sqlite.md) | Accepted and implemented | Shared execution, API-owned local storage, SDK context dependency, and DuckDB removal; each runtime mode lists and selects only its own DataSources. |
+| [0001: One API execution path with SQLite for local development](api/0001-unified-api-storage-and-local-sqlite.md) | Accepted and implemented; hosted declaration verification pending | Shared execution, API-owned local storage, SDK context dependency, and DuckDB removal; the deployment declares the hosted runtime database in `configuration.yaml` and an Environment Secret, and Settings is read-only in Hosted mode. |
 | [0002: Application administration and table ownership](api/0002-application-administration-and-table-ownership.md) | Accepted and implemented | Platform admin facts, table Reader/Writer ownership, User/Team grants, live namespace inheritance, and application-owned access management. |
 | [0004: Execute approved application migrations inside the API](api/0004-approved-application-migrations.md) | Superseded by ADR 0013 | Historical API-installed provider execution model. |
 | [0005: Shared run logs with local file capture](api/0005-run-logs-and-local-file-capture.md) | Accepted and implemented; live hosted verification pending | Shared run correlation and log API, runtime-bound local JSONL files, and hosted retrieval through the SDK. |
@@ -34,4 +34,5 @@ decisions belong in the SDK repository and are linked as dependencies.
 | [0010: Import external tables and views](api/0010-import-external-tables-and-views.md) | Accepted; implemented (verification boundaries in ADR) | Shared backend discovery, metadata import/refresh, and bounded reads of external tables and views, preserving one runtime catalog and existing grants. |
 | [0012: Bounded data transfer, concurrent execution, and safe retries](api/0012-bounded-data-transfer-and-safe-retries.md) | Accepted; implemented; hosted engine verification pending | API/client row and byte budgets, incremental fetching, concurrent admission, durable chunk receipts, safe recovery, and required user documentation; independent of Artifacts and temporary URLs. |
 | [0013: Application-owned migrations](api/0013-application-owned-migrations.md) | Accepted and implemented | Application-side Alembic, environment database authority, API catalog integration and separate system migrations. |
-| [0014: System migrations in the deployment workflow](api/0014-main-sequence-release-jobs-and-production-migrations.md) | Accepted; implemented on development; hosted verification pending | A Main Sequence Job upgrades the active runtime from the candidate image before each API rollout. |
+| [0014: System migrations in the deployment workflow](api/0014-main-sequence-release-jobs-and-production-migrations.md) | Accepted; implemented on development; hosted verification pending | A Main Sequence Job verifies, migrates and registers the declared runtime database from the candidate image before each API rollout, including the first. |
+| [0015: TimescaleDB compression and retention policies](api/0015-timescale-compression-and-retention-policies.md) | Accepted; implemented on branch; TimescaleDB verification pending | Time-index tables on TimescaleDB become hypertables; Writers set compression and retention through the API/Admin; Timescale keeps the jobs, and the Admin shows their status. |

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.8.dev18}/docs/api/data-sources.md

@@ -18,11 +18,12 @@ selected driver and runs `SELECT 1`. For `timescale_db`, that probe does not
 verify that the TimescaleDB extension is installed in the target database.
 
 The **Data Sources** section and `/data-sources/` expose one registration workflow.
-Hosted database DataSources can be registered before hosted runtime initialization.
-In Local mode, run system migrations in Settings first; remote registrations then
-use the same catalog table as the SQLite runtime row. The initialized
-workspace SQLite DataSource appears in the same list and remains runtime-managed
-with `can_manage: false`. Its storage and access cannot be changed through this API.
+Registrations live only in the active runtime's catalog. Before the runtime is
+active, the list is empty and registration is refused: Hosted answers 409 until the
+deployment has initialized its runtime database, and Local requires system migrations
+in Settings first. The runtime DataSource appears in the same list and remains
+runtime-managed with `can_manage: false`. Its storage and access cannot be changed
+through this API.
 
 External registration, editing, validation and removal require Organization admin
 access but do not depend on the runtime database's SQL permission initialization.
@@ -30,15 +31,14 @@ These operations do not reconcile runtime table grants. If database permissions
 need repair in Security, caller queries remain blocked while external DataSource
 management remains available.
 
-Each runtime mode lists only its own registrations: Local the workspace catalog,
-Hosted its registry. In Hosted mode, Settings lists eligible PostgreSQL, TimescaleDB,
-MySQL and MSSQL DataSources and selects one by UID through `/runtime-bootstrap/select/`.
-Local mode cannot select a Hosted DataSource; switch to Hosted first. Registration preserves the UID when its catalog is
-initialized. The API's private pre-initialization storage is an implementation detail.
+Each runtime mode lists only its own catalog's registrations. The hosted runtime
+DataSource is declared by the API's deployment, not chosen from this list; see
+[hosted runtime](../operations/hosted-runtime.md).
 
 The active runtime DataSource hosts the system catalog and default application
-tables. It exposes `can_manage: false`; configuration, inspection and selection
-belong to Settings. Other registrations cannot replace it by setting `is_default`.
+tables. It exposes `can_manage: false`; its configuration belongs to the deployment
+declaration in Hosted and to Settings in Local. Other registrations cannot replace
+it by setting `is_default`.
 Hosted mode rejects injected SQLite catalog records with
 `hosted_runtime_rejects_local_storage`. Execution remains bound to the selected
 runtime DataSource; registering another database does not enable its table operations.
@@ -99,9 +99,9 @@ check: explicitly validate after registration or a configuration change.
 
 All remote engines require `host`, `database_name`, and `database_user`.
 `password_secret_uid` is an opaque CredentialStore UUID: a local encrypted record
-or a hosted SDK Secret. It is required
-for the selected hosted runtime, including databases that otherwise permit
-passwordless administrator connections: caller credentials are derived from it.
+or a hosted SDK Secret. For the hosted runtime it is the UID of the URI Secret,
+whose login password is required even where the database permits passwordless
+administrator connections: caller credentials are derived from it.
 The write-only password is a top-level request field, separate from persistent
 `configuration`. Plaintext passwords inside configuration, connection strings,
 arbitrary driver options, and unknown fields are rejected. Ports must be between
@@ -140,7 +140,7 @@ JSON bodies and a small client script. Those same bodies can be submitted to
 In MetaTables Admin, open **Data Sources → Register source**, choose an engine,
 fill in connection settings and Secret UIDs, and save. The form supplies the
 engine's defaults and only its supported TLS options. Open the saved source to
-validate, edit, disable, or remove it. Select the runtime DataSource in Settings. All hosted engines use the same bootstrap and table APIs.
+validate, edit, disable, or remove it. The hosted runtime DataSource is declared by the deployment, not registered here. All hosted engines use the same bootstrap and table APIs.
 
 For a configuration edit, submit the complete configuration rather than a nested
 partial patch. The registration's engine cannot change; create another registration
@@ -151,8 +151,9 @@ for a different engine. The default schema of a referenced source cannot change.
 - `403`: another caller tried to manage the registration or replace its default.
 - `404`: the registration does not exist.
 - `409`: a source is disabled during validation, referenced/default during deletion,
-  has a protected schema, or the
-  operation conflicts with the runtime's fixed storage binding.
+  has a protected schema, the
+  operation conflicts with the runtime's fixed storage binding, or a Hosted
+  registration arrives before the runtime is active.
 - `422`: invalid engine, configuration, field type, or Secret UID.
 - `503`: connection or credential storage failed, including unavailable drivers,
   keys, catalog initialization or hosted Secrets. Runtime-context reports the
@@ -182,10 +183,12 @@ other caller sessions. SQL Server requires login/user/role administration,
 `VIEW DEFINITION` and grant management. Managed database services must provide
 these capabilities; missing permissions leave caller SQL disabled.
 
-Settings checks these PostgreSQL requirements as soon as you select a database
-without a MetaTables runtime, before any migration runs. A missing one shows
-**Needs attention** with the reason, for example `metatables_dev lacks CREATEROLE`;
-**Run MetaTables migrations** stays unavailable and nothing is written.
+Preparing the database, its login and these privileges is the database
+administrator's responsibility. On every deployment, the hosted migration Job checks
+the PostgreSQL requirements before any migration runs: `CREATEROLE`, `CREATE` on the
+database or ownership of the `metatables` schema, and `CREATE` on the default schema.
+A missing one fails the Job with the reason, for example `metatables_dev lacks
+CREATEROLE`; nothing is written and the API does not roll out.
 
 The database may be shared with other applications. Setup revokes `PUBLIC` access
 only on objects MetaTables owns: its `metatables` schema, the tables it manages and
@@ -207,4 +210,6 @@ Timescale extension operations check the installed extension in the selected
 PostgreSQL database at invocation. Registering `timescale_db` does not establish
 that the extension exists. User roles and grants authorize operations; database
 features are resolved by the bound backend. Responses contain no static
-`supports_*` list and the Admin detail view has no capability table.
+`supports_*` list and the Admin detail view has no capability table. Time-index
+tables on `timescale_db` are hypertables owned by the API's login and can carry
+[compression and retention policies](time-index-and-updates.md#timescaledb-policies).