mainsequence-metatable 0.1.8.dev16__tar.gz → 0.1.8.dev18__tar.gz

{mainsequence_metatable-0.1.8.dev16 → mainsequence_metatable-0.1.8.dev18}/.agents/skills/client/metatables-local-development/SKILL.md

@@ -140,12 +140,16 @@ local rows, catalog UIDs, credentials and fixture data are not promoted by a swi
 When returning to the environment is part of the requested workflow:
 
 1. Finish local tests and stop active writers. In Admin **Settings → Runtime mode**,
-   inspect the displayed hosted environment and choose the intended registered
-   DataSource. Use **Switch to Hosted**. This changes the API worker and selected
-   database; it does not copy or merge the local database. Leave local capability
-   enabled if the developer needs to switch back later.
+   inspect the displayed hosted environment and use **Switch to Hosted**. Hosted
+   opens the runtime database the API's deployment declares, through the same
+   Environment Secret; there is no DataSource to choose and the launcher never
+   migrates it. This changes the API worker and database; it does not copy or merge
+   the local database. Leave local capability enabled if the developer needs to
+   switch back later.
 2. Re-read runtime status and verify `local_mode: false`, the intended verified
-   hosted environment, the expected DataSource and readiness. The same local
+   hosted environment, the expected DataSource and readiness. `migration_required`
+   means the branch has system migrations the deployed API does not have yet; it
+   clears once that code is deployed. The same local
    connection command still reaches this supervised API; its `--local` flag alone
    does not prove SQLite. If the switch fails and the old worker is restored,
    report the actual mode instead of claiming the switch succeeded.

{mainsequence_metatable-0.1.8.dev16 → mainsequence_metatable-0.1.8.dev18}/.agents/skills/client/metatables-meta-tables/SKILL.md

@@ -35,8 +35,9 @@ identity and Git source facts use SDK imports. Table operations use the client's
 with SQLite behind it; ordinary clients never open local table files. Local
 workspaces use one persistent SQLite database for system and user tables per
 checkout; Git branches share it.
-Settings explicitly initializes or selects the runtime DataSource in both modes;
-startup never runs Alembic. See `docs/operations/catalog-migrations.md`.
+A hosted API's deployment declares and initializes its runtime DataSource; Local
+Settings explicitly initializes or selects the SQLite file. Startup never runs
+Alembic. See `docs/operations/catalog-migrations.md`.
 
 ## Authoring decisions
 
@@ -118,6 +119,12 @@ A null cutoff requires a scope. Do not replace that workflow with ad hoc SQL.
 Deletion/cascade requires edit access, relevant protection checks, and explicit
 cascade intent; it is not an administrator-role bypass or migration repair.
 
+On a TimescaleDB DataSource, time-index tables are hypertables. Their compression
+and retention policies are set only through `TimeIndexMetaTable.set_timescale_policies`
+(Writer) and read with `get_timescale_policies`. Do not add policies to table
+contracts or Alembic revisions; to keep them in code, call the client from a
+bootstrap Job. Retention permanently drops older chunks, including backfills.
+
 ## Validation
 
 Check authored models, normalization, physical naming, FK targets, and time grain

{mainsequence_metatable-0.1.8.dev16 → mainsequence_metatable-0.1.8.dev18}/.agents/skills/client/metatables-migrations/SKILL.md

@@ -34,9 +34,10 @@ finalize catalog bindings through the API. Use the
 [table skill](../metatables-meta-tables/SKILL.md) for contract design.
 
 Check `docs/reference/capabilities.md` and `metatables migrations --help` before
-promising a command. The API's own catalog migrations are a separate history that
-an admin runs from Settings; see `docs/operations/catalog-migrations.md`. They
-are not application-provider revisions.
+promising a command. The API's own catalog migrations are a separate history: a
+hosted API's deployment applies them before it rolls out, and Local mode runs them
+from Settings; see `docs/operations/catalog-migrations.md`. They are not
+application-provider revisions.
 
 ## Source and runtime context
 

{mainsequence_metatable-0.1.8.dev16/src/mainsequence_metatable.egg-info → mainsequence_metatable-0.1.8.dev18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.8.dev16
+Version: 0.1.8.dev18
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT

{mainsequence_metatable-0.1.8.dev16 → mainsequence_metatable-0.1.8.dev18}/docs/adr/api/0001-unified-api-storage-and-local-sqlite.md

@@ -30,6 +30,8 @@ Amended 2026-09-29: one enforced runtime binding and explicit bootstrap select o
 database for system and user tables.
 Amended 2026-10-02: each runtime mode lists and selects only its own DataSources,
 and inspection checks the login's prerequisites before setup.
+Amended 2026-10-03: the deployment declares the hosted runtime database; Settings
+is read-only in Hosted mode.
 
 Companion decision: [Main Sequence SDK ADR 0035: Independent Git source and platform execution context](https://github.com/mainsequence-sdk/mainsequence-sdk/blob/metatables_removal/docs/adr/0035-independent-git-source-and-platform-context.md).
 The SDK prerequisite is implemented on `metatables_removal`; the current local
@@ -83,8 +85,67 @@ API requirement. The guides and examples describe the implemented shared path.
 
 ## Decision
 
+### The deployment declares the hosted runtime database (amended 2026-10-03)
+
+Implemented 2026-10-03 for [MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13) in the API, the migration Job and the Admin.
+The PostgreSQL transition test and the container suite were updated but not run;
+hosted verification is pending.
+
+Settings used to save the hosted selection in the database it pointed to, with a
+pointer file on the pod. A restarted or additional pod lost that pointer. The
+hosted runtime database is now declared by the deployment:
+
+- **Declaration.** The API's `configuration.yaml`
+  (`metatables/api/metatables/configuration.yaml`) has a `runtime_database`
+  section with the engine, the name of the Environment Secret holding the
+  connection URI, the default schema, the TLS mode and optional certificate
+  Secret names. It holds names and settings, never secret values. Each branch
+  keeps its own file, so each Environment's deployment reads its own file and
+  Secret.
+- **Secret.** The database administrator creates it in each Environment, for
+  example `postgresql://login:password@host:5432/database`. Host, port, database,
+  login and password come only from the URI, certificates only from the named
+  Secrets. The runtime DataSource's `password_secret_uid` is the URI Secret's UID
+  ([ADR 0009](0009-shared-credential-store.md)).
+- **Deployment.** On every deployment the `migrate-system` Job reads the
+  declaration and the Secret. It checks the prerequisites, applies the system
+  migrations, registers the runtime DataSource and reapplies access setup
+  ([ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md)).
+  Pointing the Secret at another database follows the same procedure: that
+  database is initialized and becomes the runtime. Each database keeps its own
+  catalog, and nothing is transferred. When only the connection changes (host,
+  login, TLS), the deployment updates the registration.
+- **API.** Every pod reads the same declaration and Secret at startup and takes
+  the runtime row from the catalog. It never runs DDL and writes no pointer, so
+  restarts and additional pods agree and report the same instance identity
+  ([MetaTables #14](https://github.com/mainsequence-projects/MetaTables/issues/14)).
+  A pod that starts on an unprepared database reports `migration_required` or
+  `registration_required` and serves no application operations. A Secret change
+  therefore takes effect on deployment.
+- **Settings, Hosted mode.** Settings is read-only: it shows the declaration, the
+  resolved connection, the status and the revisions.
+  - Hosted `configure`, `migrate` and `activate` answer 409.
+  - `select` and the pre-activation Hosted registry
+    (`runtime-source-candidates.json`, `hosted_source_uid`) are removed.
+  - Hosted DataSources are registered only in the active runtime catalog.
+  - Settings does not show the migration Job's result; the deployment run
+    reports a failed Job.
+- **Developer launcher, Hosted mode.** The launcher reads the packaged deployment
+  file and the same Secret, so it uses the same database; its own
+  `configuration.yaml` only grants the developer capability. It never migrates. A
+  newer local migration reports `migration_required` until it is deployed.
+
+Local mode is unchanged. For Hosted mode, this supersedes three earlier
+decisions:
+- the Hosted registry and selection in the 2026-10-02 amendment;
+- Settings-initiated hosted initialization in the 2026-09-29 bootstrap amendment;
+- the persisted hosted selection in the 2026-09-29 bootstrap amendment.
+
 ### Each runtime mode selects only its own DataSources (amended 2026-10-02)
 
+Hosted rows are superseded by the 2026-10-03 amendment above: Hosted has no
+registry before activation and no selection.
+
 Implemented 2026-10-02 in the API and Admin; the PostgreSQL runtime-transition test
 was updated but not run.
 
@@ -160,6 +221,9 @@ binding, automatic local catalog upgrades, and the requirement that switching to
 Hosted already have a migrated catalog. It corrects this ADR's runtime lifecycle;
 it does not introduce a second execution architecture.
 
+For Hosted mode, the 2026-10-03 amendment replaces Settings selection and
+initialization with the deployment's declaration and migration Job.
+
 Settings selects the runtime mode and its DataSource. That database contains both
 MetaTables' system tables and user tables. Local selects one persistent SQLite
 file; Hosted selects PostgreSQL/TimescaleDB. MySQL and MSSQL remain connection-only
@@ -174,12 +238,12 @@ never activates a partially initialized runtime. A database already initialized
 and compatible can be selected without running migrations.
 
 Amended 2026-10-02: before any runtime is registered, inspection also checks,
-read-only, that the login can secure the database under
-[ADR 0007](0007-database-enforced-table-access.md): it owns the database, has
-`CREATEROLE`, and owns every non-system object. Unmet prerequisites make the
-target incompatible, with each reason shown in Settings, so setup never leaves a
-database half-initialized. A registered runtime is not rechecked, because it may
-hold imported external objects.
+read-only, the login's prerequisites under
+[ADR 0007](0007-database-enforced-table-access.md): `CREATEROLE`, and permission to
+create the `metatables` schema and tables in the default schema. Unmet
+prerequisites make the target incompatible, with each reason shown in Settings,
+so setup never leaves a database half-initialized. A registered runtime is not
+rechecked.
 
 Settings, runtime context, mode selection and bootstrap actions do not depend on
 catalog sessions or catalog memberships. They retain the existing request admission.
@@ -274,10 +338,18 @@ The supervisor passes mode and transport parameters explicitly to the child;
 `METATABLES_LOCAL_RUNTIME` has no authority. Database credentials remain private
 deployment inputs and are never stored in YAML or sent to Vite.
 
-The runtime descriptor reports capability, switching availability and a fresh
-instance identity. Admin requests carry that identity so stale requests are
-rejected after a restart. During transition the UI blocks work, reloads runtime
-context and remounts cached views. A mode change affects the entire developer API
+The runtime descriptor reports capability, switching availability and an
+instance identity. Admin requests carry that identity so requests made against a
+previous runtime are rejected. During transition the UI blocks work, reloads runtime
+context and remounts cached views.
+
+Amended 2026-10-03 for [MetaTables #14](https://github.com/mainsequence-projects/MetaTables/issues/14):
+the instance identity names the runtime binding, not the process. It is derived
+from the mode, the runtime database's public configuration, the runtime
+DataSource UID, its activation and the catalog revision. Every process and pod
+serving one binding reports the same identity, so an API deploy or restart keeps
+open pages valid; selecting another DataSource, activating it, applying
+migrations or switching mode changes it. A mode change affects the entire developer API
 instance, never an individual user's view of a shared service.
 
 ### Enforced runtime binding (amended 2026-09-29)

{mainsequence_metatable-0.1.8.dev16 → mainsequence_metatable-0.1.8.dev18}/docs/adr/api/0007-database-enforced-table-access.md

@@ -6,6 +6,14 @@
 > conflicting application-migration execution and credential restrictions;
 > governed API operations and MetaTables system migrations remain separate.
 
+> Amendment (2026-10-02): MetaTables secures only what it owns: its `metatables`
+> schema, the tables it manages and the roles it creates. Database-wide defaults
+> and `PUBLIC` access to other owners' objects, including extensions and other
+> applications sharing the database, are the database administrator's
+> responsibility. The login needs `CREATEROLE` and permission to create the
+> `metatables` schema and tables in the DataSource's default schema; it need not
+> own the database.
+
 
 Date: 2026-09-29
 
@@ -160,12 +168,14 @@ client.
 - **Ownership.** Tables are owned by `mt_owner`, a role that cannot log in, because
   an owner holds every privilege. The API's own role manages grants and schema
   changes and never runs caller SQL.
-- **Defaults.** PostgreSQL's default `PUBLIC` privileges are revoked: connecting,
-  creating objects and temporary tables, and executing new functions.
+- **Defaults.** On objects MetaTables owns, PostgreSQL's default `PUBLIC`
+  privileges are revoked, including for future tables and routines. Database-wide
+  defaults (connecting, creating objects and temporary tables) and other owners'
+  objects belong to the database administrator.
 - **Privileged routines.** Restricted caller roles must not be able to execute
   application routines with elevated owner privileges, including
   `SECURITY DEFINER` routines. Initialization establishes this restriction for
-  existing routines as well as future defaults, covering direct grants, inherited
+  the routines MetaTables owns, existing and future, covering direct grants, inherited
   roles and `PUBLIC`. Revoking a direct User grant alone is insufficient if
   another grant still permits execution. Schema changes and extension installation
   preserve the restriction before exposing new or changed routines to callers.
@@ -261,7 +271,8 @@ and [MySQL implicit commits](https://dev.mysql.com/doc/refman/8.4/en/implicit-co
 
 - The explorer can run any PostgreSQL query, and the API has no SQL parser left
   to get wrong.
-- The DataSource login needs `CREATEROLE`.
+- The DataSource login needs `CREATEROLE` and the right to create the `metatables`
+  schema; it does not need to own the database.
 - Physical table names become visible to explorer users.
 - Cascading foreign keys are unavailable where the explorer is enabled.
 - Team membership reaches the database at the pace of the existing one-hour

{mainsequence_metatable-0.1.8.dev16 → mainsequence_metatable-0.1.8.dev18}/docs/adr/api/0009-shared-credential-store.md

@@ -9,6 +9,10 @@
 
 Date: 2026-09-30
 
+Amended 2026-10-03 ([MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13)): the hosted runtime login's password comes from the
+URI Secret that the deployment declares. See
+[Bootstrap, restart and lifecycle](#bootstrap-restart-and-lifecycle).
+
 Status: Accepted. Implemented by the API CredentialStore providers and catalog
 revision `0006_local_credentials`.
 
@@ -131,17 +135,21 @@ coverage must additionally run on its corresponding host.
 ### Bootstrap, restart and lifecycle
 
 Local SQLite opens using the existing API-owned file binding, before credential
-lookup. Hosted bootstrap resolves the selected connection's UUID through SDK
-Secrets before opening the catalog. Its credential never depends on opening the
-database that needs that password. Saved selection JSON holds only public
-settings and UUID references.
+lookup. Hosted bootstrap reads the Environment Secret named by
+`runtime_database.uri_secret` (amended 2026-10-03). It reads the Secret by name
+through the SDK before opening the catalog. The login and password come from that
+URI, and the runtime DataSource references the Secret's UID as
+`password_secret_uid`. The hosted store serves that UID only as the database
+password and refuses to delete it. Every other reference still resolves through
+SDK Secrets by UID. The credential never depends on opening the database that
+needs it. Local selection JSON holds only public settings and UUID references.
 
 A fresh local catalog must receive explicit system migrations before saving
 sources/credentials. Local saves never substitute JSON for encrypted SQL storage.
 Previously saved local candidate metadata can be materialized by explicit setup,
-retaining DataSource UIDs. Hosted pre-catalog registration continues to use SDK
-Secrets and its existing metadata registry; it becomes the runtime row during
-explicit setup. Startup never silently applies migrations.
+retaining DataSource UIDs. Hosted has no pre-catalog registration: the
+deployment's migration Job registers the declared runtime, and other hosted
+sources are registered in its catalog. Startup never silently applies migrations.
 
 `metatables credentials` provides admin-only local operations:
 

{mainsequence_metatable-0.1.8.dev16 → mainsequence_metatable-0.1.8.dev18}/docs/adr/api/0014-main-sequence-release-jobs-and-production-migrations.md

@@ -3,6 +3,8 @@
 Date: 2026-10-02
 
 Status: Accepted. Implemented on `development`; hosted verification pending.
+Amended 2026-10-03 ([MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13)): the Job reads the declared runtime database itself
+and initializes it on the first deployment.
 
 Owners: MetaTables API. Main Sequence owns Job execution, images, deployment and
 platform authentication.
@@ -20,6 +22,31 @@ its API serves.
 
 ## Decision
 
+### The Job prepares the declared database (amended 2026-10-03)
+
+The Job no longer reads the running API. `metatables runtime upgrade` loads
+`runtime_database` from the candidate image's `configuration.yaml` and the
+Environment Secret it names ([ADR 0001](0001-unified-api-storage-and-local-sqlite.md)).
+Every run executes the same steps:
+
+1. Check that the login can secure the database (ADR 0007 prerequisites).
+2. Apply the system migrations.
+3. Register the runtime DataSource, or update its connection to the declaration.
+4. Reapply access setup. Per-User role passwords derive from the login password,
+   so a rotated password reaches them on the next deployment.
+
+The first deployment therefore initializes the database. A missing declaration
+or Secret, an unreachable database or unmet prerequisites fail the Job and block
+the rollout. The deployment workflow is the authority, so no admin caller is
+required. A new runtime DataSource records the Job's SDK user as its creator.
+The Job prints `initialized`, `upgraded` or `up_to_date` with the revisions.
+
+Settings' **Run MetaTables migrations** and `metatables runtime initialize` remain
+for Local mode only. The original decision follows. The amendment supersedes
+steps 1–2, the skip rules and the Settings initialization paragraph.
+
+### Original decision (2026-10-02)
+
 The deployment workflow upgrades the runtime from the candidate image before the
 rollout:
 

mainsequence_metatable-0.1.8.dev18/docs/adr/api/0015-timescale-compression-and-retention-policies.md

@@ -0,0 +1,192 @@
+# ADR 0015: TimescaleDB compression and retention policies
+
+Date: 2026-10-02
+
+Status: Accepted.
+
+Implementation status: Implemented on the `timescale-policies` branches of the API
+and the Admin, with focused tests. TimescaleDB container verification is pending.
+
+Owner: MetaTables API. The Admin owns the screens described here.
+
+Related decisions: [ADR 0002: Table ownership](0002-application-administration-and-table-ownership.md),
+[ADR 0007: Database-enforced access](0007-database-enforced-table-access.md) (amended
+below), [ADR 0012: Safe retries](0012-bounded-data-transfer-and-safe-retries.md) and
+[ADR 0013: Application-owned migrations](0013-application-owned-migrations.md).
+
+## Context
+
+Time-index tables on `timescale_db` DataSources grow without bound. Users need to
+compress older data and drop expired data on a schedule, without database access.
+
+Source inspection on 2026-10-02 found:
+
+- The Admin's Timescale Policies tab calls `GET/PATCH meta-tables/{uid}/policies`
+  and appears only when a table reports `capabilities.timescale_policies`. The API
+  provides neither, so the tab never appears.
+- No current path creates hypertables. `storage_layout.partitioning.strategy`
+  accepts `timescale_hypertable`, but only the legacy
+  `create_multi_index_table_for_metadata` calls `create_hypertable`. Timescale
+  policies require hypertables.
+- The legacy policy helpers in `adapters/maintenance/timescaledb.py` have no route
+  callers. They sleep between steps, remove and re-add policies in separate
+  commits, and some build SQL with f-strings.
+- TimescaleDB creates a policy job owned by the hypertable owner and refuses to
+  create it when that role lacks `LOGIN`. ADR 0007 makes `mt_owner`, a `NOLOGIN`
+  role, the owner of every managed table.
+
+## Decision
+
+### Core model
+
+**TimescaleDB stores the policy; MetaTables keeps no copy.** A policy is a
+Timescale background job. Timescale already schedules and runs it, so MetaTables
+adds no scheduler and no policy table. The API authorizes in the catalog, reads
+jobs from `timescaledb_information`, and writes with Timescale's policy functions.
+
+Policies are set only through the API and the Admin. An application that wants
+its policies in code runs a Job at bootstrap that calls the client method. Table
+contracts and application migrations carry no policy fields.
+
+| Policy | User sets | MetaTables derives |
+| --- | --- | --- |
+| Compression | `after`; optional `schedule_interval`, `initial_start`, `timezone` | `segmentby` = identity dimensions; `orderby` = time index `DESC` |
+| Retention | `after`; optional `schedule_interval`, `initial_start`, `timezone` | — |
+
+- `after: null` removes that policy. Removing compression leaves already
+  compressed chunks compressed.
+- If both are set, retention `after` must be longer than compression `after`.
+- Each change runs in one transaction with parameterized SQL and no sleeps.
+- Dropping a table removes its jobs.
+- TimescaleDB 2.11 or later is required, since it accepts inserts, updates,
+  deletes and upserts on compressed chunks. Uploads and tail deletes therefore
+  need no decompression step.
+
+### Eligible tables
+
+A table can have policies when:
+
+1. its DataSource is `timescale_db`;
+2. it is a time-index table and not `external_registered`;
+3. its physical table is a hypertable; and
+4. the installed TimescaleDB version is supported.
+
+Conditions 1–2 are catalog facts. They set `capabilities.timescale_policies` on
+the table response without opening a database connection. Conditions 3–4 are
+checked live when policies are read or written. Saving additionally requires a
+`read_write` DataSource. Tables on any other engine never show policies.
+
+### Hypertables
+
+On a `timescale_db` DataSource, managed finalization converts each new time-index
+table into a hypertable on its time index (`create_hypertable`, `if_not_exists`,
+`migrate_data`). Strategy `backend_default` means hypertable there; `none` opts
+out. `partitioning.strategy` is the only switch, so the duplicate
+`backend_hints.timescale.partitioning` hint and `time_index_table.partition_strategy`
+are removed. Chunk interval stays at Timescale's default. Every unique constraint
+must include the time index; the storage layout's `uniqueness` already does.
+
+### Authorization and ownership
+
+Writer on the table is the only permission check: a Writer can finalize the
+hypertable and set or remove both policies. Readers can view policies and job
+status. Organization admins also see every job on the DataSource.
+
+**Amendment to ADR 0007:** on `timescale_db` DataSources, hypertables are owned
+by the API's database login instead of `mt_owner`, so their policy jobs run as
+that login. Access setup leaves hypertable ownership unchanged instead of handing
+the tables to `mt_owner`. The API's login already owns the database and is a member of
+`mt_owner`, so this grants it nothing new. Reader/Writer grants, caller SQL roles,
+application migrations and other engines are unchanged. If the DataSource later
+uses a different login, existing jobs keep running as the old one until the
+tables are reassigned.
+
+Each policy change is recorded in the `physical_operation` journal as
+`timescale_policy`, with the requesting User. System migration
+`0009_timescale_policies` adds that kind, drops `time_index_table.partition_strategy`
+and removes the Timescale hint from stored layouts; existing runtimes upgrade
+through Settings.
+
+### API
+
+```text
+TimescalePolicy        { after, schedule_interval?, initial_start?, timezone? }
+TimescaleTablePolicies { compression, retention,
+                         eligibility { eligible, reason? },               read-only
+                         jobs [TimescaleJob],                             read-only
+                         compression_stats { before_bytes, after_bytes,
+                                             compressed_chunks, total_chunks } }  read-only
+TimescaleJob           { job_id, kind, table_uid?, hypertable, scheduled, status,
+                         last_run_status, last_successful_finish, next_start,
+                         total_failures, last_error? }
+```
+
+Intervals use PostgreSQL interval text such as `30 days`; the database validates them.
+
+| Endpoint | Access | Behavior |
+| --- | --- | --- |
+| `GET /meta-tables/{uid}/timescale-policies/` | Reader | Policies, eligibility, jobs and compression stats. |
+| `PUT /meta-tables/{uid}/timescale-policies/` | Writer | Replaces both policies; idempotent, safe to retry. Returns the GET body. |
+| `GET /data-sources/{uid}/timescale-jobs/` | Any User who can see the source | Jobs for tables the caller can read; admins see all. `timescale_db` only. |
+
+Errors: `404 timescale_policies_unavailable` for other engines,
+`403 write_access_required`, `409 timescale_not_hypertable`,
+`409 timescale_version_unsupported`, `409 timescale_extension_missing`, the existing
+DataSource access codes for read-only or disabled sources, and `422` for an invalid
+interval or timezone or `timescale_retention_not_after_compression`.
+
+The Python client exposes `get_timescale_policies()` and `set_timescale_policies()`
+on time-index tables.
+
+### Admin
+
+- **Table → Timescale Policies** (the existing tab, reworked): Compression and
+  Retention cards, each with an on/off switch, an `after` number with a unit
+  (days, weeks, months), and an Advanced section for schedule, start and timezone.
+  Each card shows its job's status (Scheduled, Running, Failed, Paused), last
+  success, next run and failures. Compression shows size before and after.
+  Saving retention asks for confirmation, naming the cutoff date and stating that
+  older chunks are permanently dropped on every run and that older backfills are
+  dropped too. Readers see the tab read-only. An ineligible table shows the reason.
+- **DataSource → Jobs** (`timescale_db` only): a table of jobs — table link,
+  policy, status, last run, next run, failures — with a failed-only filter and
+  Refresh, under Policies and Failed jobs counts.
+
+Screens use Command Center SDK components and theme tokens only.
+
+### Monitoring
+
+Monitoring is read on demand from `jobs`, `job_stats`, `job_errors` and
+`hypertable_compression_stats`. MetaTables stores no job history, polls nothing,
+and sends no alerts.
+
+## Consequences
+
+- One source of truth: a policy changed directly in the database appears in the
+  Admin as it is.
+- The legacy Timescale policy, decompression and hypertable helpers and their
+  runtime registrations are deleted.
+- Retention is destructive and available to every Writer; the confirmation is
+  the safeguard.
+- Policy jobs belong to the API's login. Rotating its password does not affect
+  them; no role gains `LOGIN`.
+
+## Alternatives considered
+
+- **Policies in table contracts or migrations.** Two authorities would overwrite
+  each other. A bootstrap Job calling the API covers code-defined policies.
+- **A catalog copy with a reconciler.** It duplicates Timescale's job store and
+  adds drift handling.
+- **A MetaTables scheduler.** Timescale already schedules policy jobs.
+- **`mt_owner` with `LOGIN` and no password.** It changes a login setting and
+  depends on the server never accepting passwordless authentication for that role.
+
+## Verification
+
+Focused tests cover eligibility, interval validation, authorization, the request
+contract and the derived compression settings, using a fake backend. The
+TimescaleDB container suite, run only on explicit request, verifies hypertable
+conversion, policy add/replace/remove, job ownership by the API's login, Reader and
+Writer grants reaching chunks and compressed chunks, and upserts into compressed
+chunks. Admin checks cover tab visibility, read-only mode and the retention
+confirmation.

{mainsequence_metatable-0.1.8.dev16 → mainsequence_metatable-0.1.8.dev18}/docs/adr/index.md

@@ -23,7 +23,7 @@ decisions belong in the SDK repository and are linked as dependencies.
 
 | ADR | Status | Scope |
 | --- | --- | --- |
-| [0001: One API execution path with SQLite for local development](api/0001-unified-api-storage-and-local-sqlite.md) | Accepted and implemented | Shared execution, API-owned local storage, SDK context dependency, and DuckDB removal; each runtime mode lists and selects only its own DataSources. |
+| [0001: One API execution path with SQLite for local development](api/0001-unified-api-storage-and-local-sqlite.md) | Accepted and implemented; hosted declaration verification pending | Shared execution, API-owned local storage, SDK context dependency, and DuckDB removal; the deployment declares the hosted runtime database in `configuration.yaml` and an Environment Secret, and Settings is read-only in Hosted mode. |
 | [0002: Application administration and table ownership](api/0002-application-administration-and-table-ownership.md) | Accepted and implemented | Platform admin facts, table Reader/Writer ownership, User/Team grants, live namespace inheritance, and application-owned access management. |
 | [0004: Execute approved application migrations inside the API](api/0004-approved-application-migrations.md) | Superseded by ADR 0013 | Historical API-installed provider execution model. |
 | [0005: Shared run logs with local file capture](api/0005-run-logs-and-local-file-capture.md) | Accepted and implemented; live hosted verification pending | Shared run correlation and log API, runtime-bound local JSONL files, and hosted retrieval through the SDK. |
@@ -34,4 +34,5 @@ decisions belong in the SDK repository and are linked as dependencies.
 | [0010: Import external tables and views](api/0010-import-external-tables-and-views.md) | Accepted; implemented (verification boundaries in ADR) | Shared backend discovery, metadata import/refresh, and bounded reads of external tables and views, preserving one runtime catalog and existing grants. |
 | [0012: Bounded data transfer, concurrent execution, and safe retries](api/0012-bounded-data-transfer-and-safe-retries.md) | Accepted; implemented; hosted engine verification pending | API/client row and byte budgets, incremental fetching, concurrent admission, durable chunk receipts, safe recovery, and required user documentation; independent of Artifacts and temporary URLs. |
 | [0013: Application-owned migrations](api/0013-application-owned-migrations.md) | Accepted and implemented | Application-side Alembic, environment database authority, API catalog integration and separate system migrations. |
-| [0014: System migrations in the deployment workflow](api/0014-main-sequence-release-jobs-and-production-migrations.md) | Accepted; implemented on development; hosted verification pending | A Main Sequence Job upgrades the active runtime from the candidate image before each API rollout. |
+| [0014: System migrations in the deployment workflow](api/0014-main-sequence-release-jobs-and-production-migrations.md) | Accepted; implemented on development; hosted verification pending | A Main Sequence Job verifies, migrates and registers the declared runtime database from the candidate image before each API rollout, including the first. |
+| [0015: TimescaleDB compression and retention policies](api/0015-timescale-compression-and-retention-policies.md) | Accepted; implemented on branch; TimescaleDB verification pending | Time-index tables on TimescaleDB become hypertables; Writers set compression and retention through the API/Admin; Timescale keeps the jobs, and the Admin shows their status. |