PyPI - limitsdb - Versions diffs - 0.6.0__tar.gz → 0.7.0__tar.gz - Mend

limitsdb 0.6.0tar.gz → 0.7.0tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (33) hide show

{limitsdb-0.6.0 → limitsdb-0.7.0}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: limitsdb
-Version: 0.6.0
+Version: 0.7.0
 Summary: Configuration-driven ILM (Information Lifecycle Management) for production and history databases
 License: Apache License
          Version 2.0, January 2004
@@ -420,15 +420,23 @@ The script:
 - stops at the first error (`whenever sqlerror exit failure`) and ends with a
   `LimitsDb: OK` banner on success;
-- contains no passwords: it prompts for them (hidden input);
+- contains no passwords: it prompts for all of them (hidden input) at its start,
+  before it changes anything;
 - can be rerun: each object is created in a PL/SQL block that ignores only its
   own "already exists" error, and an existing database link is skipped;
-- creates the database link after reconnecting as the runtime user, who owns it
-  (it asks for that user's password).
+- creates the database link after reconnecting as the runtime user, who owns it.
+  It asks for that user's password once and reuses it to reconnect. An existing user keeps its password, so the one entered must
+  match it; the script says so before it reconnects.
 Run it in SQL*Plus connected as a user with administrative privileges:
 `sqlplus admin@<dsn> @bootstrap_<schema>_<env>_<timestamp>.sql`.
+When `ldb-impl` has admin credentials and the runtime user already existed with
+a different password, the reconnection as that user is rejected (ORA-01017).
+`ldb-impl` then logs that the user rejected the configured password, naming the
+user and the database, and exits 1; it never changes the password of an existing
+user.
 ## Run ILM
 ```bash
@@ -441,7 +449,7 @@ ldb-run --schema <SCHEMA> [--profile <PROFILE>] --action SOURCE_ILM|HISTORY_ILM
 Common overrides:
 ```
---mode VALIDATE|PLAN|PREVIEW|SCRIPT|EXECUTE   # VALIDATE setup, PLAN DAG, PREVIEW run without moving data, SCRIPT emit SQL, EXECUTE run
+--mode VALIDATE|PREPARE|PLAN|PREVIEW|SCRIPT|EXECUTE   # VALIDATE setup and privileges (read-only), PREPARE grants and objects, PLAN DAG, PREVIEW run without moving data, SCRIPT emit SQL, EXECUTE run
 --chunk-size <int>                            # Rows per chunk when processing large tables
 --parallel-max <int>                          # Maximum number of tables processed in parallel
 --ddl-pending-dir <path>                      # Where failed history DDL is written for a DBA (default: ldb_ddl_pending)
@@ -471,9 +479,25 @@ application role, and creating or altering history tables before a `PREVIEW` or
 `EXECUTE` run (`PREVIEW` needs the objects in place to validate and time the
 queries; it skips only the data movement). Both operations are idempotent.
-If admin credentials are absent from `secrets[.<PROFILE>].json`, the privilege
-grants are skipped silently — the assumption is that grants are already in
-place.
+A `PREVIEW` writes its own statuses in `ldb_ctl`, the `EXECUTE` ones prefixed with `P` (`PTSTART`, `PCSTART`,
+`PCEND`, `PLSTART`, `PLEND`, `PTEND`, `PERROR`), and finishes each table with `PTEND`, not `TEND`, so it never marks a
+table as processed for the day: an `EXECUTE` run afterwards (same day) still moves the data. A `PREVIEW` is refused
+(`ORA-20006`) over a table that has an `EXECUTE` status for the same process date. An `EXECUTE` continues an incomplete
+`EXECUTE` of the same process date (finished tables stay finished and the rows already moved are kept); in any other case
+the table starts over and the abandoned run is closed in `ldb_log` as "Incomplete process closed by a later process".
+Only one `ldb-run` can work on a table at a time: the runner takes a lock per table in `LDB_RUN_LOCK` and a second runner
+stops with exit code 1 before starting anything. After upgrading, rerun `ldb-impl` so the new table and the supporting
+procedures (`check_save_status`, `check_referencing_tables`) are installed.
+The generated `SCRIPT` takes the mode from SQL*Plus and runs without the runner lock:
+`sqlplus user/password@dsn @script EXECUTE` (or `PREVIEW`). Its process date is the `ldb_process_date` variable.
+If source admin credentials are absent from `secrets[.<PROFILE>].json`, the
+source privilege grants are not applied. `ldb-run` checks with the runtime
+connection which privileges are missing through the role and logs a warning
+naming them; a table the runtime user cannot see is skipped with a warning.
+Run `--mode PREPARE` to get the missing grants as a file for a DBA.
 History table management follows the same principle:
@@ -486,7 +510,7 @@ History table management follows the same principle:
   DBA pre-creates all history objects and admin credentials are not stored.
 - **Admin credentials absent, history schema cannot be inspected** (for example
   the runtime connection or dictionary query fails): `ldb-run` exits with code 1
-  and logs the cause. It does not run ILM against a history schema whose state is
+  and tells the operator to run `ldb-impl` so a DBA completes the configuration. It does not run ILM against a history schema whose state is
   unknown.
 - **Admin credentials absent, DDL changes are required**: `ldb-run` applies the
   DDL directly over the runtime history connection, with no prior privilege
@@ -504,9 +528,46 @@ History table management follows the same principle:
   If the file cannot be written (for example the directory is not writable),
   `ldb-run` logs the error with the failed statements and exits with code 1.
+### Preparing grants and objects with `--mode PREPARE`
+`--mode PREPARE` brings permissions and objects into place without running any
+ILM work, so the problems show up before `SCRIPT`, `PREVIEW` or `EXECUTE`. It
+is safe to rerun and meant to run first (`VALIDATE`, `PREPARE`, then the rest):
+- With source admin credentials (`SOURCE_ILM`) it grants the source tables to
+  the source role. Without them it writes `source_grants_<schema>_<ts>.sql` under
+  `ddl_pending_dir` for a DBA, with only the privileges the runtime user does not
+  receive yet through the role, and warns. Once the grants are in place no file
+  is written.
+- It checks, with the runtime connections alone, that the source and history
+  roles are enabled for the runtime users and give them `SELECT`, `INSERT`,
+  `UPDATE` and `DELETE` on every configured table. A role that is granted but
+  not a default role gives nothing.
+  The role must stay a default role of the runtime user (`ldb-run` warns
+  when it is not); `GRANT role` makes it default unless the user was altered.
+- The default role names are `LDB_SOURCE_ROLE` and `LDB_HISTORY_ROLE`. When
+  several schemas (configuration sets) share the same SOURCE or HISTORY
+  database, those names collide; set `source_role_name` / `history_role_name`
+  per schema to keep the roles apart. Reusing one role across schemas also
+  works and is rarely a problem.
+- It creates or alters the history tables and grants them to the history role,
+  with the same admin or runtime-connection rules as `EXECUTE`: statements that
+  fail go to the pending file.
+- It exits with code 1 if the runtime user still cannot see a configured table
+  (it does not exist or `SELECT` is not granted). Other modes only skip such a
+  table with a warning.
+`VALIDATE` runs the same privilege check without changing anything: a gap on
+the action's own environment fails it, a gap on the other environment is a
+warning, since `PREPARE` creates the history objects. `SCRIPT`, `PREVIEW` and
+`EXECUTE` warn about the gaps they find.
+`PREVIEW` and `EXECUTE` keep applying DDL and grants themselves, so a change in
+source reaches history without failing the run.
 ### History DDL in `--mode SCRIPT`
-`--mode SCRIPT` applies nothing to the databases. The ILM SQL blocks, which
+`--mode SCRIPT` applies no DDL and no history grants to the databases. The ILM SQL blocks, which
 run many times, go to stdout. The one-time statements go to files under
 `ddl_pending_dir`, so they stay out of the script that is rerun:
@@ -516,7 +577,14 @@ run many times, go to stdout. The one-time statements go to files under
 - `source_grants_<schema>_<YYYYMMDD_HHMMSS>.sql` (`SOURCE_ILM` only): the grants
   on the source tables for the source role. Run it against the source database.
-Both are plain SQL for SQL*Plus that stop at the first failing statement; the
+The runtime user reads the column metadata of the source tables through their
+grants, so when source admin credentials exist `SCRIPT` applies the source
+grants itself and writes no `source_grants` file. Without them the file is
+written and `ldb-run` logs a warning. In every mode, a table the runtime user
+cannot see (it does not exist, or `SELECT` is not granted) is skipped with a
+warning instead of failing the run, and so is a table that references it.
+Both files are plain SQL for SQL*Plus that stop at the first failing statement; the
 grants are unconditional and safe to run again. The history DDL is always
 emitted, computed like this:
@@ -538,7 +606,7 @@ the same pending file as the failed DDL, so a DBA can run it.
 | Parameter                | Purpose                                                                                                                                                         | Default / values                                                                     |
 | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
 | `action`                 | ILM target (`SOURCE_ILM` moves/purges source, `HISTORY_ILM` cleans downstream history).                                                                         | Default `SOURCE_ILM`; choices `SOURCE_ILM`, `HISTORY_ILM`.                           |
-| `mode`                   | Execution mode: `VALIDATE` config/credentials, `PLAN` dependency order, `PREVIEW` run the real queries and apply DDL and grants but do not insert or delete data (a.k.a. legacy `DRY_RUN`), `SCRIPT` emit the history DDL and the ILM SQL without applying anything, `EXECUTE` run changes. | Default `PREVIEW`; choices `VALIDATE`, `PLAN`, `PREVIEW`, `SCRIPT`, `EXECUTE`.       |
+| `mode`                   | Execution mode: `VALIDATE` config, credentials and role privileges (read-only), `PREPARE` apply grants and history objects without running ILM, `PLAN` dependency order, `PREVIEW` run the real queries and apply DDL and grants but do not insert or delete data (a.k.a. legacy `DRY_RUN`), `SCRIPT` emit the history DDL and the ILM SQL without applying anything, `EXECUTE` run changes. | Default `PREVIEW`; choices `VALIDATE`, `PLAN`, `PREVIEW`, `SCRIPT`, `EXECUTE`.       |
 | `chunk_size`             | Rows per chunk when processing large tables.                                                                                                                    | Default `100000`.                                                                    |
 | `use_added_columns`      | Populate derived columns in history tables.                                                                                                                     | Default `True` (boolean toggle).                                                     |
 | `add_ldb_columns`        | Add LimitsDb execution-date columns in history tables.                                                                                                        | Default `True` (boolean toggle).                                                     |
@@ -551,10 +619,10 @@ the same pending file as the failed DDL, so a DBA can run it.
 | `profile`                | Environment name (e.g., `dev`, `prod`). Differentiates configuration files for different environments within the same configuration set. | Optional; default `null`. |
 | `ilm_config_file`        | YAML file with tables (bypass DB discovery).                                                                                                                    | Optional; default `null`.                                                            |
 | `source_dsn`             | Source DSN / connection descriptor.                                                                                                                             | Default empty string.                                                                |
-| `source_username`        | Source username.                                                                                                                                                | Default empty string.                                                                |
+| `source_username`        | Source username, as named in the data dictionary: lowercase (`ldb`) is a quoted user created as `"ldb"`, uppercase (`LDB`) a regular one.                    |  | Default empty string.                                                                |
 | `source_password`        | Source password stored in secrets.                                                                                                                              | Default empty string; encrypted in `secrets.<PROFILE>.json`.                         |
 | `history_dsn`            | History DSN / connection descriptor.                                                                                                                            | Default empty string.                                                                |
-| `history_username`       | History username.                                                                                                                                               | Default empty string.                                                                |
+| `history_username`       | History username, with the same case rules as `source_username`.                                                                                                || Default empty string.                                                                |
 | `history_password`       | History password stored in secrets.                                                                                                                             | Default empty string; encrypted in `secrets.<PROFILE>.json`.                         |
 | `admin_source_username`  | Admin source username.                                                                                                                                          | Default empty string.                                                                |
 | `admin_source_password`  | Admin source password stored in secrets.                                                                                                                        | Default empty string; encrypted in `secrets.<PROFILE>.json`.                         |
@@ -599,7 +667,9 @@ Each entry in `ilm[.<PROFILE>].yml` merges table-level attributes with one or mo
 - **Retention windows:** `retain_months_source` is required whenever `purge_date_expr` is present. `retain_months_history` extends the history window but also depends on defining the source retention. During `HISTORY_ILM` runs, the engine sums source and history months to determine the cut-off date.
 - **Date expressions:** `purge_date_expr` identifies the date column (or expression) that anchors retention. Prefix column references with `@` (for example, `"@DSP_DATE"`); the runner swaps the prefix for the proper table alias at execution time.
 - **Additional filters:** Add optional filters through `additional_filter_expr` (source runs) and `history_addtl_filter_expr` (history runs). Both accept the same `@column` syntax, and the history expression falls back to the source expression when omitted.
-- **Referencing tables:** Use `referencing_tables` to pull parent conditions into child tables. List entries as `<TABLE> <ALIAS>` (optionally `<OWNER>.<TABLE> <ALIAS>`) and supply matching `join_expr` fragments. The runner inherits active conditions from each referenced table and rewrites the join fragments—`@` becomes `inner` joins by default or `left outer` joins when `source_orphan_purge: true` to find orphans.
+- **Referencing tables:** Use `referencing_tables` to pull parent conditions into child tables. List entries as `<TABLE> <ALIAS>` (optionally `<OWNER>.<TABLE> <ALIAS>`) and supply matching `join_expr` fragments. The runner inherits the active conditions of each referenced table: the conditions of one table are alternatives (`or`), and a child row is purged only when its own conditions and those of every referenced table are met (`and` between tables). It rewrites the join fragments—`@` becomes `inner` joins by default or `left outer` joins when `source_orphan_purge: true` to find orphans.
+- **Join owners in history:** `join_expr` names each referenced table by its source owner. In `HISTORY_ILM` runs with `use_added_columns: false` the runner joins the history copy instead, replacing `<source_owner>.<table>` with `<history_owner>.<table>` for every table listed in `referencing_tables`. With `use_added_columns: true` history runs do not join.
+- **Extra tables:** Filters may contain subqueries over tables that ILM does not manage. List them in `extra_select_tables` (read by `additional_filter_expr`, so source runs) and `history_extra_select_tables` (read by `history_addtl_filter_expr`) as comma-separated `[OWNER.]TABLE` entries; an entry without owner belongs to the environment's own owner. The environment's role gets `SELECT` on them wherever it gets the other table grants (`ldb-run`, `--mode PREPARE`, `--mode SCRIPT`) and the privilege checks include them. In history, `history_extra_select_tables` falls back to `extra_select_tables`, just as `history_addtl_filter_expr` falls back to `additional_filter_expr`. A table also managed by ILM keeps its full privileges. For tables that are not in the history database write the access in `history_addtl_filter_expr` yourself (for example `TABLE@SRC`, or `TABLE UNION ALL TABLE@SRC` in a subquery); LimitsDb does not rewrite subqueries. `SRC` is only the default name of the history-to-source database link; it can be overridden with `history_to_source_dblink_name` (and `source_to_history_dblink_name`, default `HIST`, for the other direction). The link connects as the source runtime user and uses the grants of its source role, so a table read as `TABLE@SRC` must also be listed in `extra_select_tables`.
 - **Historical predicate snapshot:** With `use_added_columns` enabled, every archived row preserves the values needed to reevaluate all its predicates in history, including values sourced from related tables. Distinct cut-off dates are stored in helper columns such as `ldb_date_<suffix>`. Predicates that share a date reuse the same helper column while retaining their own filters and retention windows. Other values sourced from related tables are materialized as `<column>_<table_alias>`.
 - **Orphan handling:** Enable `source_orphan_purge` with `orphan_check_column` to archive source rows whose referenced parent is absent. The source query uses `left outer` joins and treats the row as an orphan when any comma-separated check column is `NULL`. `referencing_tables`, `join_expr`, `orphan_check_column`, `use_added_columns: true`, and a positive historical retention are required. Orphans follow the table's normal source-to-history flow rather than being deleted directly. LimitsDb snapshots this state in `LDB_IS_ORPHAN` and stores a fallback date so `HISTORY_ILM` can apply the configured retention without accessing the missing parent.
@@ -625,7 +695,7 @@ Every key is commented; the default value and short help appear inline. Example:
 ```yaml
 # action: "SOURCE_ILM"  # ILM target: months_keep_history_max (SOURCE_ILM) or history (HISTORY_ILM)
-# mode: "PREVIEW"       # VALIDATE | PLAN | PREVIEW | SCRIPT | EXECUTE (PREVIEW mirrors legacy DRY_RUN)
+# mode: "PREVIEW"       # VALIDATE | PREPARE | PLAN | PREVIEW | SCRIPT | EXECUTE
 # chunk_size: 100000    # Rows per chunk when processing large tables
 # use_added_columns: True  # Populate derived columns in history tables
 # add_ldb_columns: True  # Add LimitsDb execution-date columns in history tables

{limitsdb-0.6.0 → limitsdb-0.7.0}/README.md RENAMED Viewed

@@ -274,15 +274,23 @@ The script:
 - stops at the first error (`whenever sqlerror exit failure`) and ends with a
   `LimitsDb: OK` banner on success;
-- contains no passwords: it prompts for them (hidden input);
+- contains no passwords: it prompts for all of them (hidden input) at its start,
+  before it changes anything;
 - can be rerun: each object is created in a PL/SQL block that ignores only its
   own "already exists" error, and an existing database link is skipped;
-- creates the database link after reconnecting as the runtime user, who owns it
-  (it asks for that user's password).
+- creates the database link after reconnecting as the runtime user, who owns it.
+  It asks for that user's password once and reuses it to reconnect. An existing user keeps its password, so the one entered must
+  match it; the script says so before it reconnects.
 Run it in SQL*Plus connected as a user with administrative privileges:
 `sqlplus admin@<dsn> @bootstrap_<schema>_<env>_<timestamp>.sql`.
+When `ldb-impl` has admin credentials and the runtime user already existed with
+a different password, the reconnection as that user is rejected (ORA-01017).
+`ldb-impl` then logs that the user rejected the configured password, naming the
+user and the database, and exits 1; it never changes the password of an existing
+user.
 ## Run ILM
 ```bash
@@ -295,7 +303,7 @@ ldb-run --schema <SCHEMA> [--profile <PROFILE>] --action SOURCE_ILM|HISTORY_ILM
 Common overrides:
 ```
---mode VALIDATE|PLAN|PREVIEW|SCRIPT|EXECUTE   # VALIDATE setup, PLAN DAG, PREVIEW run without moving data, SCRIPT emit SQL, EXECUTE run
+--mode VALIDATE|PREPARE|PLAN|PREVIEW|SCRIPT|EXECUTE   # VALIDATE setup and privileges (read-only), PREPARE grants and objects, PLAN DAG, PREVIEW run without moving data, SCRIPT emit SQL, EXECUTE run
 --chunk-size <int>                            # Rows per chunk when processing large tables
 --parallel-max <int>                          # Maximum number of tables processed in parallel
 --ddl-pending-dir <path>                      # Where failed history DDL is written for a DBA (default: ldb_ddl_pending)
@@ -325,9 +333,25 @@ application role, and creating or altering history tables before a `PREVIEW` or
 `EXECUTE` run (`PREVIEW` needs the objects in place to validate and time the
 queries; it skips only the data movement). Both operations are idempotent.
-If admin credentials are absent from `secrets[.<PROFILE>].json`, the privilege
-grants are skipped silently — the assumption is that grants are already in
-place.
+A `PREVIEW` writes its own statuses in `ldb_ctl`, the `EXECUTE` ones prefixed with `P` (`PTSTART`, `PCSTART`,
+`PCEND`, `PLSTART`, `PLEND`, `PTEND`, `PERROR`), and finishes each table with `PTEND`, not `TEND`, so it never marks a
+table as processed for the day: an `EXECUTE` run afterwards (same day) still moves the data. A `PREVIEW` is refused
+(`ORA-20006`) over a table that has an `EXECUTE` status for the same process date. An `EXECUTE` continues an incomplete
+`EXECUTE` of the same process date (finished tables stay finished and the rows already moved are kept); in any other case
+the table starts over and the abandoned run is closed in `ldb_log` as "Incomplete process closed by a later process".
+Only one `ldb-run` can work on a table at a time: the runner takes a lock per table in `LDB_RUN_LOCK` and a second runner
+stops with exit code 1 before starting anything. After upgrading, rerun `ldb-impl` so the new table and the supporting
+procedures (`check_save_status`, `check_referencing_tables`) are installed.
+The generated `SCRIPT` takes the mode from SQL*Plus and runs without the runner lock:
+`sqlplus user/password@dsn @script EXECUTE` (or `PREVIEW`). Its process date is the `ldb_process_date` variable.
+If source admin credentials are absent from `secrets[.<PROFILE>].json`, the
+source privilege grants are not applied. `ldb-run` checks with the runtime
+connection which privileges are missing through the role and logs a warning
+naming them; a table the runtime user cannot see is skipped with a warning.
+Run `--mode PREPARE` to get the missing grants as a file for a DBA.
 History table management follows the same principle:
@@ -340,7 +364,7 @@ History table management follows the same principle:
   DBA pre-creates all history objects and admin credentials are not stored.
 - **Admin credentials absent, history schema cannot be inspected** (for example
   the runtime connection or dictionary query fails): `ldb-run` exits with code 1
-  and logs the cause. It does not run ILM against a history schema whose state is
+  and tells the operator to run `ldb-impl` so a DBA completes the configuration. It does not run ILM against a history schema whose state is
   unknown.
 - **Admin credentials absent, DDL changes are required**: `ldb-run` applies the
   DDL directly over the runtime history connection, with no prior privilege
@@ -358,9 +382,46 @@ History table management follows the same principle:
   If the file cannot be written (for example the directory is not writable),
   `ldb-run` logs the error with the failed statements and exits with code 1.
+### Preparing grants and objects with `--mode PREPARE`
+`--mode PREPARE` brings permissions and objects into place without running any
+ILM work, so the problems show up before `SCRIPT`, `PREVIEW` or `EXECUTE`. It
+is safe to rerun and meant to run first (`VALIDATE`, `PREPARE`, then the rest):
+- With source admin credentials (`SOURCE_ILM`) it grants the source tables to
+  the source role. Without them it writes `source_grants_<schema>_<ts>.sql` under
+  `ddl_pending_dir` for a DBA, with only the privileges the runtime user does not
+  receive yet through the role, and warns. Once the grants are in place no file
+  is written.
+- It checks, with the runtime connections alone, that the source and history
+  roles are enabled for the runtime users and give them `SELECT`, `INSERT`,
+  `UPDATE` and `DELETE` on every configured table. A role that is granted but
+  not a default role gives nothing.
+  The role must stay a default role of the runtime user (`ldb-run` warns
+  when it is not); `GRANT role` makes it default unless the user was altered.
+- The default role names are `LDB_SOURCE_ROLE` and `LDB_HISTORY_ROLE`. When
+  several schemas (configuration sets) share the same SOURCE or HISTORY
+  database, those names collide; set `source_role_name` / `history_role_name`
+  per schema to keep the roles apart. Reusing one role across schemas also
+  works and is rarely a problem.
+- It creates or alters the history tables and grants them to the history role,
+  with the same admin or runtime-connection rules as `EXECUTE`: statements that
+  fail go to the pending file.
+- It exits with code 1 if the runtime user still cannot see a configured table
+  (it does not exist or `SELECT` is not granted). Other modes only skip such a
+  table with a warning.
+`VALIDATE` runs the same privilege check without changing anything: a gap on
+the action's own environment fails it, a gap on the other environment is a
+warning, since `PREPARE` creates the history objects. `SCRIPT`, `PREVIEW` and
+`EXECUTE` warn about the gaps they find.
+`PREVIEW` and `EXECUTE` keep applying DDL and grants themselves, so a change in
+source reaches history without failing the run.
 ### History DDL in `--mode SCRIPT`
-`--mode SCRIPT` applies nothing to the databases. The ILM SQL blocks, which
+`--mode SCRIPT` applies no DDL and no history grants to the databases. The ILM SQL blocks, which
 run many times, go to stdout. The one-time statements go to files under
 `ddl_pending_dir`, so they stay out of the script that is rerun:
@@ -370,7 +431,14 @@ run many times, go to stdout. The one-time statements go to files under
 - `source_grants_<schema>_<YYYYMMDD_HHMMSS>.sql` (`SOURCE_ILM` only): the grants
   on the source tables for the source role. Run it against the source database.
-Both are plain SQL for SQL*Plus that stop at the first failing statement; the
+The runtime user reads the column metadata of the source tables through their
+grants, so when source admin credentials exist `SCRIPT` applies the source
+grants itself and writes no `source_grants` file. Without them the file is
+written and `ldb-run` logs a warning. In every mode, a table the runtime user
+cannot see (it does not exist, or `SELECT` is not granted) is skipped with a
+warning instead of failing the run, and so is a table that references it.
+Both files are plain SQL for SQL*Plus that stop at the first failing statement; the
 grants are unconditional and safe to run again. The history DDL is always
 emitted, computed like this:
@@ -392,7 +460,7 @@ the same pending file as the failed DDL, so a DBA can run it.
 | Parameter                | Purpose                                                                                                                                                         | Default / values                                                                     |
 | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
 | `action`                 | ILM target (`SOURCE_ILM` moves/purges source, `HISTORY_ILM` cleans downstream history).                                                                         | Default `SOURCE_ILM`; choices `SOURCE_ILM`, `HISTORY_ILM`.                           |
-| `mode`                   | Execution mode: `VALIDATE` config/credentials, `PLAN` dependency order, `PREVIEW` run the real queries and apply DDL and grants but do not insert or delete data (a.k.a. legacy `DRY_RUN`), `SCRIPT` emit the history DDL and the ILM SQL without applying anything, `EXECUTE` run changes. | Default `PREVIEW`; choices `VALIDATE`, `PLAN`, `PREVIEW`, `SCRIPT`, `EXECUTE`.       |
+| `mode`                   | Execution mode: `VALIDATE` config, credentials and role privileges (read-only), `PREPARE` apply grants and history objects without running ILM, `PLAN` dependency order, `PREVIEW` run the real queries and apply DDL and grants but do not insert or delete data (a.k.a. legacy `DRY_RUN`), `SCRIPT` emit the history DDL and the ILM SQL without applying anything, `EXECUTE` run changes. | Default `PREVIEW`; choices `VALIDATE`, `PLAN`, `PREVIEW`, `SCRIPT`, `EXECUTE`.       |
 | `chunk_size`             | Rows per chunk when processing large tables.                                                                                                                    | Default `100000`.                                                                    |
 | `use_added_columns`      | Populate derived columns in history tables.                                                                                                                     | Default `True` (boolean toggle).                                                     |
 | `add_ldb_columns`        | Add LimitsDb execution-date columns in history tables.                                                                                                        | Default `True` (boolean toggle).                                                     |
@@ -405,10 +473,10 @@ the same pending file as the failed DDL, so a DBA can run it.
 | `profile`                | Environment name (e.g., `dev`, `prod`). Differentiates configuration files for different environments within the same configuration set. | Optional; default `null`. |
 | `ilm_config_file`        | YAML file with tables (bypass DB discovery).                                                                                                                    | Optional; default `null`.                                                            |
 | `source_dsn`             | Source DSN / connection descriptor.                                                                                                                             | Default empty string.                                                                |
-| `source_username`        | Source username.                                                                                                                                                | Default empty string.                                                                |
+| `source_username`        | Source username, as named in the data dictionary: lowercase (`ldb`) is a quoted user created as `"ldb"`, uppercase (`LDB`) a regular one.                    |  | Default empty string.                                                                |
 | `source_password`        | Source password stored in secrets.                                                                                                                              | Default empty string; encrypted in `secrets.<PROFILE>.json`.                         |
 | `history_dsn`            | History DSN / connection descriptor.                                                                                                                            | Default empty string.                                                                |
-| `history_username`       | History username.                                                                                                                                               | Default empty string.                                                                |
+| `history_username`       | History username, with the same case rules as `source_username`.                                                                                                || Default empty string.                                                                |
 | `history_password`       | History password stored in secrets.                                                                                                                             | Default empty string; encrypted in `secrets.<PROFILE>.json`.                         |
 | `admin_source_username`  | Admin source username.                                                                                                                                          | Default empty string.                                                                |
 | `admin_source_password`  | Admin source password stored in secrets.                                                                                                                        | Default empty string; encrypted in `secrets.<PROFILE>.json`.                         |
@@ -453,7 +521,9 @@ Each entry in `ilm[.<PROFILE>].yml` merges table-level attributes with one or mo
 - **Retention windows:** `retain_months_source` is required whenever `purge_date_expr` is present. `retain_months_history` extends the history window but also depends on defining the source retention. During `HISTORY_ILM` runs, the engine sums source and history months to determine the cut-off date.
 - **Date expressions:** `purge_date_expr` identifies the date column (or expression) that anchors retention. Prefix column references with `@` (for example, `"@DSP_DATE"`); the runner swaps the prefix for the proper table alias at execution time.
 - **Additional filters:** Add optional filters through `additional_filter_expr` (source runs) and `history_addtl_filter_expr` (history runs). Both accept the same `@column` syntax, and the history expression falls back to the source expression when omitted.
-- **Referencing tables:** Use `referencing_tables` to pull parent conditions into child tables. List entries as `<TABLE> <ALIAS>` (optionally `<OWNER>.<TABLE> <ALIAS>`) and supply matching `join_expr` fragments. The runner inherits active conditions from each referenced table and rewrites the join fragments—`@` becomes `inner` joins by default or `left outer` joins when `source_orphan_purge: true` to find orphans.
+- **Referencing tables:** Use `referencing_tables` to pull parent conditions into child tables. List entries as `<TABLE> <ALIAS>` (optionally `<OWNER>.<TABLE> <ALIAS>`) and supply matching `join_expr` fragments. The runner inherits the active conditions of each referenced table: the conditions of one table are alternatives (`or`), and a child row is purged only when its own conditions and those of every referenced table are met (`and` between tables). It rewrites the join fragments—`@` becomes `inner` joins by default or `left outer` joins when `source_orphan_purge: true` to find orphans.
+- **Join owners in history:** `join_expr` names each referenced table by its source owner. In `HISTORY_ILM` runs with `use_added_columns: false` the runner joins the history copy instead, replacing `<source_owner>.<table>` with `<history_owner>.<table>` for every table listed in `referencing_tables`. With `use_added_columns: true` history runs do not join.
+- **Extra tables:** Filters may contain subqueries over tables that ILM does not manage. List them in `extra_select_tables` (read by `additional_filter_expr`, so source runs) and `history_extra_select_tables` (read by `history_addtl_filter_expr`) as comma-separated `[OWNER.]TABLE` entries; an entry without owner belongs to the environment's own owner. The environment's role gets `SELECT` on them wherever it gets the other table grants (`ldb-run`, `--mode PREPARE`, `--mode SCRIPT`) and the privilege checks include them. In history, `history_extra_select_tables` falls back to `extra_select_tables`, just as `history_addtl_filter_expr` falls back to `additional_filter_expr`. A table also managed by ILM keeps its full privileges. For tables that are not in the history database write the access in `history_addtl_filter_expr` yourself (for example `TABLE@SRC`, or `TABLE UNION ALL TABLE@SRC` in a subquery); LimitsDb does not rewrite subqueries. `SRC` is only the default name of the history-to-source database link; it can be overridden with `history_to_source_dblink_name` (and `source_to_history_dblink_name`, default `HIST`, for the other direction). The link connects as the source runtime user and uses the grants of its source role, so a table read as `TABLE@SRC` must also be listed in `extra_select_tables`.
 - **Historical predicate snapshot:** With `use_added_columns` enabled, every archived row preserves the values needed to reevaluate all its predicates in history, including values sourced from related tables. Distinct cut-off dates are stored in helper columns such as `ldb_date_<suffix>`. Predicates that share a date reuse the same helper column while retaining their own filters and retention windows. Other values sourced from related tables are materialized as `<column>_<table_alias>`.
 - **Orphan handling:** Enable `source_orphan_purge` with `orphan_check_column` to archive source rows whose referenced parent is absent. The source query uses `left outer` joins and treats the row as an orphan when any comma-separated check column is `NULL`. `referencing_tables`, `join_expr`, `orphan_check_column`, `use_added_columns: true`, and a positive historical retention are required. Orphans follow the table's normal source-to-history flow rather than being deleted directly. LimitsDb snapshots this state in `LDB_IS_ORPHAN` and stores a fallback date so `HISTORY_ILM` can apply the configured retention without accessing the missing parent.
@@ -479,7 +549,7 @@ Every key is commented; the default value and short help appear inline. Example:
 ```yaml
 # action: "SOURCE_ILM"  # ILM target: months_keep_history_max (SOURCE_ILM) or history (HISTORY_ILM)
-# mode: "PREVIEW"       # VALIDATE | PLAN | PREVIEW | SCRIPT | EXECUTE (PREVIEW mirrors legacy DRY_RUN)
+# mode: "PREVIEW"       # VALIDATE | PREPARE | PLAN | PREVIEW | SCRIPT | EXECUTE
 # chunk_size: 100000    # Rows per chunk when processing large tables
 # use_added_columns: True  # Populate derived columns in history tables
 # add_ldb_columns: True  # Add LimitsDb execution-date columns in history tables

{limitsdb-0.6.0 → limitsdb-0.7.0}/limitsdb/cli/ldb_impl.py RENAMED Viewed

@@ -3,12 +3,14 @@
 from __future__ import annotations
 import argparse
+from collections.abc import Callable
 from datetime import datetime
 from pathlib import Path
-from typing import Any
+from typing import Any, Literal
 from limitsdb.core.ldb_crypto import load_or_create_key
-from limitsdb.core.ldb_errors import LimitsDbError
+from limitsdb.core.ldb_ddl import render_script
+from limitsdb.core.ldb_errors import InvalidCredentialsError, LimitsDbError
 from limitsdb.core.ldb_logger import configure_logger, get_logger, reconfigure_logger
 from limitsdb.core.ldb_params_config import Config, build_config
 from limitsdb.core.ldb_utils import encrypt_secrets_in_place
@@ -16,6 +18,7 @@ from limitsdb.db.ldb_engine_loader import get_db_engine
 from limitsdb.db.ldb_engines import (
     ColumnDefinition,
     DatabaseLinkDefinition,
+    DdlSink,
     RoleDefinition,
     SequenceDefinition,
     TableDefinition,
@@ -60,6 +63,7 @@ def _build_control_tables(owner: str) -> list[TableDefinition]:
                 ColumnDefinition("CTL_PROCESS_END", "date"),
                 ColumnDefinition("CTL_ROWS_PROCESSED", "number", precision=18, scale=0),
                 ColumnDefinition("CTL_PLSQL", "clob"),
+                ColumnDefinition("CTL_RUN_ID", "string", length=36),
             ),
             primary_key=("CTL_OWNER", "CTL_TABLE_NAME"),
         ),
@@ -81,6 +85,22 @@ def _build_control_tables(owner: str) -> list[TableDefinition]:
             ),
             primary_key=("LOG_ID",),
         ),
+        # Runner mutex: the 'L' row is locked by the live runner; the 'K' row publishes its run_id
+        TableDefinition(
+            owner=owner,
+            name="LDB_RUN_LOCK",
+            columns=(
+                ColumnDefinition("RL_OWNER", "string", length=50, nullable=False),
+                ColumnDefinition("RL_TABLE_NAME", "string", length=50, nullable=False),
+                ColumnDefinition("RL_KIND", "string", length=1, nullable=False),
+                ColumnDefinition("RL_RUN_ID", "string", length=36),
+                ColumnDefinition("RL_MODE", "string", length=10),
+                ColumnDefinition("RL_HOST", "string", length=255),
+                ColumnDefinition("RL_PID", "number", precision=18, scale=0),
+                ColumnDefinition("RL_START", "date"),
+            ),
+            primary_key=("RL_OWNER", "RL_TABLE_NAME", "RL_KIND"),
+        ),
     ]
@@ -164,13 +184,15 @@ def run_cli() -> None:
     source_has_admin = bool(administration.source.username and administration.source.password)
     history_has_admin = bool(administration.history.username and administration.history.password)
-    def write_script(env: str, **definitions: Any) -> None:
-        content = engine.generate_bootstrap_script(title=f"{env} database of schema {config.schema}", **definitions)
+    def write_script(env: str, steps: Callable[[DdlSink], None]) -> None:
+        fragments: list[str] = []
+        steps(fragments.append)
+        title = f"{env} database of schema {config.schema}"
         stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
         path = Path(config.execution.ddl_pending_dir) / f"bootstrap_{config.schema}_{env.lower()}_{stamp}.sql"
         try:
             path.parent.mkdir(parents=True, exist_ok=True)
-            path.write_text(content, encoding="utf-8")
+            path.write_text(render_script(title, fragments), encoding="utf-8")
         except OSError as exc:
             logger.error(
                 "Admin credentials for %s are absent and the script could not be written to %s: %s", env, path, exc
@@ -178,29 +200,48 @@ def run_cli() -> None:
             raise SystemExit(1) from exc
         script_files.append(path)
+    def bootstrap_steps(
+        roles: list[RoleDefinition],
+        user: UserDefinition,
+        tables: list[TableDefinition],
+        sequences: list[SequenceDefinition],
+        link: DatabaseLinkDefinition,
+        runtime: tuple[str, str],
+    ) -> Callable[[DdlSink], None]:
+        def steps(sink: DdlSink) -> None:
+            engine.script_password_prompts([user], [link], ddl_sink=sink)
+            engine.ensure_roles(None, roles, ddl_sink=sink)
+            engine.ensure_users(None, [user], ddl_sink=sink)
+            engine.ensure_tables(None, tables, ddl_sink=sink)
+            engine.ensure_sequences(None, sequences, ddl_sink=sink)
+            engine.ensure_supporting_objects(None, runtime[0], ddl_sink=sink)
+            engine.ensure_database_links(None, [link], ddl_sink=sink, connect_as=runtime)
+        return steps
     if not source_has_admin:
         write_script(
             "SOURCE",
-            roles=source_roles,
-            users=[source_user],
-            tables=source_tables,
-            sequences=source_sequences,
-            supporting_owner=connections.source.username,
-            links=[source_db_link],
-            runtime_username=connections.source.username,
-            runtime_dsn=connections.source.dsn,
+            bootstrap_steps(
+                source_roles,
+                source_user,
+                source_tables,
+                source_sequences,
+                source_db_link,
+                (connections.source.username, connections.source.dsn),
+            ),
         )
     if not history_has_admin:
         write_script(
             "HISTORY",
-            roles=history_roles,
-            users=[history_user],
-            tables=history_tables,
-            sequences=history_sequences,
-            supporting_owner=connections.history.username,
-            links=[history_db_link],
-            runtime_username=connections.history.username,
-            runtime_dsn=connections.history.dsn,
+            bootstrap_steps(
+                history_roles,
+                history_user,
+                history_tables,
+                history_sequences,
+                history_db_link,
+                (connections.history.username, connections.history.dsn),
+            ),
         )
     if source_has_admin:
@@ -229,15 +270,29 @@ def run_cli() -> None:
         finally:
             engine.close_connection(history_admin_conn)
+    def connect_as_runtime(env: Literal["SOURCE", "HISTORY"], username: str) -> Any:
+        # The user exists here (ensure_users ran and never changes an existing password), so a rejected
+        # login can only be a configured password that differs from the one the user already has.
+        try:
+            return engine.get_connection(config, admin=False, env=env)
+        except InvalidCredentialsError as exc:
+            logger.error(
+                "The runtime user %s exists in %s but rejected the configured password; ldb-impl does not "
+                "change the password of an existing user. Set the configured password to the user's current one.",
+                username,
+                env,
+            )
+            raise SystemExit(1) from exc
     if source_has_admin:
-        source_user_conn = engine.get_connection(config, admin=False, env="SOURCE")
+        source_user_conn = connect_as_runtime("SOURCE", connections.source.username)
         try:
             summary["source_database_links"] = engine.ensure_database_links(source_user_conn, [source_db_link])
         finally:
             engine.close_connection(source_user_conn)
     if history_has_admin:
-        history_user_conn = engine.get_connection(config, admin=False, env="HISTORY")
+        history_user_conn = connect_as_runtime("HISTORY", connections.history.username)
         try:
             summary["history_database_links"] = engine.ensure_database_links(history_user_conn, [history_db_link])
         finally:

limitsdb 0.6.0__tar.gz → 0.7.0__tar.gz

limitsdb 0.6.0tar.gz → 0.7.0tar.gz