limitsdb 0.5.0__tar.gz → 0.7.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (34) hide show
  1. {limitsdb-0.5.0 → limitsdb-0.7.0}/PKG-INFO +223 -18
  2. {limitsdb-0.5.0 → limitsdb-0.7.0}/README.md +222 -17
  3. limitsdb-0.7.0/limitsdb/cli/ldb_impl.py +316 -0
  4. limitsdb-0.7.0/limitsdb/core/ldb_ddl.py +155 -0
  5. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_errors.py +8 -0
  6. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_ilm_config.py +6 -0
  7. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_logger.py +5 -5
  8. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_params_config.py +33 -8
  9. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_runner.py +641 -58
  10. limitsdb-0.7.0/limitsdb/core/ldb_status.py +31 -0
  11. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/ldb_engines.py +186 -19
  12. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/oracle/ldb_engine_impl.py +957 -256
  13. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/resources/ilm.example.yml +4 -2
  14. {limitsdb-0.5.0 → limitsdb-0.7.0}/pyproject.toml +1 -1
  15. limitsdb-0.5.0/limitsdb/cli/ldb_impl.py +0 -207
  16. limitsdb-0.5.0/limitsdb/core/ldb_status.py +0 -8
  17. {limitsdb-0.5.0 → limitsdb-0.7.0}/LICENSE +0 -0
  18. {limitsdb-0.5.0 → limitsdb-0.7.0}/LICENSE-DUAL.md +0 -0
  19. {limitsdb-0.5.0 → limitsdb-0.7.0}/NOTICE +0 -0
  20. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/__init__.py +0 -0
  21. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/cli/__init__.py +0 -0
  22. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/cli/ldb_crypt.py +0 -0
  23. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/cli/ldb_init.py +0 -0
  24. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/cli/ldb_run.py +0 -0
  25. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/__init__.py +0 -0
  26. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_config_loader.py +0 -0
  27. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_crypto.py +0 -0
  28. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_meta.py +0 -0
  29. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_utils.py +0 -0
  30. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/__init__.py +0 -0
  31. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/ldb_engine_loader.py +0 -0
  32. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/oracle/__init__.py +0 -0
  33. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/oracle/ldb_conf.sql +0 -0
  34. {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/resources/__init__.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: limitsdb
3
- Version: 0.5.0
3
+ Version: 0.7.0
4
4
  Summary: Configuration-driven ILM (Information Lifecycle Management) for production and history databases
5
5
  License: Apache License
6
6
  Version 2.0, January 2004
@@ -207,6 +207,52 @@ source .venv/bin/activate
207
207
  pip install limitsdb
208
208
  ```
209
209
 
210
+ ### Air-gapped or internet-restricted servers
211
+
212
+ For production database servers without internet access (common in enterprise Oracle
213
+ environments on Linux, AIX, or Solaris), download the packages on a machine that
214
+ does have access and transfer them to the target server.
215
+
216
+ **Step 1 — Download on a connected machine**
217
+
218
+ If the target server is Linux x86_64:
219
+
220
+ ```bash
221
+ pip download limitsdb \
222
+ --dest ./limitsdb-bundle \
223
+ --platform linux_x86_64 \
224
+ --python-version 312 \
225
+ --only-binary=:all:
226
+ ```
227
+
228
+ For the same platform as the current machine (Windows or macOS build host
229
+ copying to an identical Linux target, omit `--platform` and `--python-version`):
230
+
231
+ ```bash
232
+ pip download limitsdb --dest ./limitsdb-bundle
233
+ ```
234
+
235
+ **Step 2 — Transfer to the target server**
236
+
237
+ ```bash
238
+ scp -r ./limitsdb-bundle user@server:/opt/limitsdb-bundle
239
+ ```
240
+
241
+ **Step 3 — Install on the target server (no internet required)**
242
+
243
+ ```bash
244
+ python3 -m venv /opt/limitsdb-venv
245
+ source /opt/limitsdb-venv/bin/activate
246
+ pip install --no-index --find-links /opt/limitsdb-bundle limitsdb
247
+ ldb-run --help
248
+ ```
249
+
250
+ > **Note for AIX and Solaris**: `oracledb` may not have a prebuilt wheel for your
251
+ > platform. If `pip download --only-binary=:all:` finds nothing, the package must
252
+ > be compiled from source on the target, which requires Python development headers
253
+ > and a C compiler (`gcc` or `xlc`). Contact your system administrator before
254
+ > proceeding.
255
+
210
256
  For development setup (cloning the repo, running tests, CI) see [CONTRIBUTING.md](CONTRIBUTING.md).
211
257
 
212
258
  ## Quickstart
@@ -227,7 +273,7 @@ Spin up a minimal end-to-end run with the built-in scaffolding tools:
227
273
  ldb-crypt --schema billing --profile dev
228
274
  ```
229
275
 
230
- 4. **Dry-run the ILM plan** (no data changes):
276
+ 4. **Preview the ILM run** (applies the required DDL and grants, moves no data):
231
277
 
232
278
  ```bash
233
279
  ldb-run --schema billing --profile dev --action SOURCE_ILM --mode PREVIEW
@@ -356,9 +402,40 @@ What it creates in both source and history environments:
356
402
  - **Database link** — bidirectional links between source and history so the ILM
357
403
  engine can reach both environments from either side.
358
404
 
359
- `ldb-impl` requires all four credential pairs to be present and encrypted in
360
- `secrets[.<PROFILE>].json`: `source_password`, `history_password`,
361
- `admin_source_password`, and `admin_history_password`. Run `ldb-crypt` first.
405
+ `ldb-impl` needs the runtime credentials (`source_password`, `history_password`)
406
+ encrypted in `secrets[.<PROFILE>].json`. Run `ldb-crypt` first.
407
+
408
+ ### Without admin credentials
409
+
410
+ Admin credentials (`admin_source_*`, `admin_history_*`) are optional per
411
+ database. When they are absent for a database, `ldb-impl` creates nothing in it
412
+ and writes a SQL*Plus script for a DBA instead, one file per database:
413
+ `bootstrap_<schema>_<source|history>_<YYYYMMDD_HHMMSS>.sql` under
414
+ `ddl_pending_dir` (the same parameter as `ldb-run`: `--ddl-pending-dir`,
415
+ `LDB_DDL_PENDING_DIR`, default `ldb_ddl_pending`). A database whose admin credentials are present is still
416
+ provisioned directly. Generating scripts is not an error (exit code 0, with a
417
+ warning naming each file); if a script cannot be written, `ldb-impl` exits 1.
418
+
419
+ The script:
420
+
421
+ - stops at the first error (`whenever sqlerror exit failure`) and ends with a
422
+ `LimitsDb: OK` banner on success;
423
+ - contains no passwords: it prompts for all of them (hidden input) at its start,
424
+ before it changes anything;
425
+ - can be rerun: each object is created in a PL/SQL block that ignores only its
426
+ own "already exists" error, and an existing database link is skipped;
427
+ - creates the database link after reconnecting as the runtime user, who owns it.
428
+ It asks for that user's password once and reuses it to reconnect. An existing user keeps its password, so the one entered must
429
+ match it; the script says so before it reconnects.
430
+
431
+ Run it in SQL*Plus connected as a user with administrative privileges:
432
+ `sqlplus admin@<dsn> @bootstrap_<schema>_<env>_<timestamp>.sql`.
433
+
434
+ When `ldb-impl` has admin credentials and the runtime user already existed with
435
+ a different password, the reconnection as that user is rejected (ORA-01017).
436
+ `ldb-impl` then logs that the user rejected the configured password, naming the
437
+ user and the database, and exits 1; it never changes the password of an existing
438
+ user.
362
439
 
363
440
  ## Run ILM
364
441
 
@@ -372,9 +449,11 @@ ldb-run --schema <SCHEMA> [--profile <PROFILE>] --action SOURCE_ILM|HISTORY_ILM
372
449
  Common overrides:
373
450
 
374
451
  ```
375
- --mode VALIDATE|PLAN|PREVIEW|SCRIPT|EXECUTE # VALIDATE setup, PLAN DAG, PREVIEW simulate, SCRIPT emit SQL, EXECUTE run
452
+ --mode VALIDATE|PREPARE|PLAN|PREVIEW|SCRIPT|EXECUTE # VALIDATE setup and privileges (read-only), PREPARE grants and objects, PLAN DAG, PREVIEW run without moving data, SCRIPT emit SQL, EXECUTE run
376
453
  --chunk-size <int> # Rows per chunk when processing large tables
377
454
  --parallel-max <int> # Maximum number of tables processed in parallel
455
+ --ddl-pending-dir <path> # Where failed history DDL is written for a DBA (default: ldb_ddl_pending)
456
+ --history-baseline EMPTY|SOURCE # SCRIPT without a readable history: assume no history objects (default) or the same objects as source
378
457
  --log-level DEBUG|INFO|WARNING|ERROR|CRITICAL # Adjust logger verbosity
379
458
  --ilm-config-file <path> # Point to a specific ILM YAML file
380
459
  --config-dir <root> # Override config root discovery
@@ -396,34 +475,154 @@ ldb-run --schema billing --profile prod --action SOURCE_ILM --mode EXECUTE --set
396
475
 
397
476
  `ldb-run` uses admin credentials (`admin_source_password`,
398
477
  `admin_history_password`) for two operations: granting table privileges to the
399
- application role, and creating or altering history tables before an `EXECUTE`
400
- run. Both operations are idempotent.
401
-
402
- If admin credentials are absent from `secrets[.<PROFILE>].json`, the privilege
403
- grants are skipped silently — the assumption is that grants are already in
404
- place. History table management in `EXECUTE` mode still requires admin
405
- credentials in the current version.
478
+ application role, and creating or altering history tables before a `PREVIEW` or
479
+ `EXECUTE` run (`PREVIEW` needs the objects in place to validate and time the
480
+ queries; it skips only the data movement). Both operations are idempotent.
481
+
482
+ A `PREVIEW` writes its own statuses in `ldb_ctl`, the `EXECUTE` ones prefixed with `P` (`PTSTART`, `PCSTART`,
483
+ `PCEND`, `PLSTART`, `PLEND`, `PTEND`, `PERROR`), and finishes each table with `PTEND`, not `TEND`, so it never marks a
484
+ table as processed for the day: an `EXECUTE` run afterwards (same day) still moves the data. A `PREVIEW` is refused
485
+ (`ORA-20006`) over a table that has an `EXECUTE` status for the same process date. An `EXECUTE` continues an incomplete
486
+ `EXECUTE` of the same process date (finished tables stay finished and the rows already moved are kept); in any other case
487
+ the table starts over and the abandoned run is closed in `ldb_log` as "Incomplete process closed by a later process".
488
+
489
+ Only one `ldb-run` can work on a table at a time: the runner takes a lock per table in `LDB_RUN_LOCK` and a second runner
490
+ stops with exit code 1 before starting anything. After upgrading, rerun `ldb-impl` so the new table and the supporting
491
+ procedures (`check_save_status`, `check_referencing_tables`) are installed.
492
+
493
+ The generated `SCRIPT` takes the mode from SQL*Plus and runs without the runner lock:
494
+ `sqlplus user/password@dsn @script EXECUTE` (or `PREVIEW`). Its process date is the `ldb_process_date` variable.
495
+
496
+ If source admin credentials are absent from `secrets[.<PROFILE>].json`, the
497
+ source privilege grants are not applied. `ldb-run` checks with the runtime
498
+ connection which privileges are missing through the role and logs a warning
499
+ naming them; a table the runtime user cannot see is skipped with a warning.
500
+ Run `--mode PREPARE` to get the missing grants as a file for a DBA.
501
+
502
+ History table management follows the same principle:
503
+
504
+ - **Admin credentials present, tables already up to date**: the admin connection
505
+ is not opened. No DDL is executed.
506
+ - **Admin credentials present, DDL changes are required**: the admin connection
507
+ is opened and the changes are applied as before.
508
+ - **Admin credentials absent, tables already up to date**: `ldb-run` proceeds
509
+ without an admin connection. This is the normal case for environments where a
510
+ DBA pre-creates all history objects and admin credentials are not stored.
511
+ - **Admin credentials absent, history schema cannot be inspected** (for example
512
+ the runtime connection or dictionary query fails): `ldb-run` exits with code 1
513
+ and tells the operator to run `ldb-impl` so a DBA completes the configuration. It does not run ILM against a history schema whose state is
514
+ unknown.
515
+ - **Admin credentials absent, DDL changes are required**: `ldb-run` applies the
516
+ DDL directly over the runtime history connection, with no prior privilege
517
+ check. It succeeds when the runtime user owns the objects or holds the needed
518
+ privileges (for example `ALTER`). Any DDL statement that fails, for whatever
519
+ reason, is recorded and the remaining statements are still attempted. If at least one
520
+ statement failed, `ldb-run` writes the failed statements, each preceded by its
521
+ error, to `history_ddl_<schema>_<YYYYMMDD_HHMMSS>.sql` under `ddl_pending_dir`
522
+ (default `ldb_ddl_pending`), logs an error naming the file, and exits with
523
+ code 1. The file is plain SQL for SQL*Plus (no PL/SQL): it stops at the first
524
+ failing statement with a non-zero exit code, and only a run that reaches the
525
+ final `LimitsDb: OK` banner applied everything. The recorded errors are
526
+ comments. A DBA reviews and runs the file, then `ldb-run` is rerun. Nothing is
527
+ written to stdout, and the directory is created only when a file is written.
528
+ If the file cannot be written (for example the directory is not writable),
529
+ `ldb-run` logs the error with the failed statements and exits with code 1.
530
+
531
+ ### Preparing grants and objects with `--mode PREPARE`
532
+
533
+ `--mode PREPARE` brings permissions and objects into place without running any
534
+ ILM work, so the problems show up before `SCRIPT`, `PREVIEW` or `EXECUTE`. It
535
+ is safe to rerun and meant to run first (`VALIDATE`, `PREPARE`, then the rest):
536
+
537
+ - With source admin credentials (`SOURCE_ILM`) it grants the source tables to
538
+ the source role. Without them it writes `source_grants_<schema>_<ts>.sql` under
539
+ `ddl_pending_dir` for a DBA, with only the privileges the runtime user does not
540
+ receive yet through the role, and warns. Once the grants are in place no file
541
+ is written.
542
+ - It checks, with the runtime connections alone, that the source and history
543
+ roles are enabled for the runtime users and give them `SELECT`, `INSERT`,
544
+ `UPDATE` and `DELETE` on every configured table. A role that is granted but
545
+ not a default role gives nothing.
546
+ The role must stay a default role of the runtime user (`ldb-run` warns
547
+ when it is not); `GRANT role` makes it default unless the user was altered.
548
+ - The default role names are `LDB_SOURCE_ROLE` and `LDB_HISTORY_ROLE`. When
549
+ several schemas (configuration sets) share the same SOURCE or HISTORY
550
+ database, those names collide; set `source_role_name` / `history_role_name`
551
+ per schema to keep the roles apart. Reusing one role across schemas also
552
+ works and is rarely a problem.
553
+ - It creates or alters the history tables and grants them to the history role,
554
+ with the same admin or runtime-connection rules as `EXECUTE`: statements that
555
+ fail go to the pending file.
556
+ - It exits with code 1 if the runtime user still cannot see a configured table
557
+ (it does not exist or `SELECT` is not granted). Other modes only skip such a
558
+ table with a warning.
559
+
560
+ `VALIDATE` runs the same privilege check without changing anything: a gap on
561
+ the action's own environment fails it, a gap on the other environment is a
562
+ warning, since `PREPARE` creates the history objects. `SCRIPT`, `PREVIEW` and
563
+ `EXECUTE` warn about the gaps they find.
564
+
565
+ `PREVIEW` and `EXECUTE` keep applying DDL and grants themselves, so a change in
566
+ source reaches history without failing the run.
567
+
568
+ ### History DDL in `--mode SCRIPT`
569
+
570
+ `--mode SCRIPT` applies no DDL and no history grants to the databases. The ILM SQL blocks, which
571
+ run many times, go to stdout. The one-time statements go to files under
572
+ `ddl_pending_dir`, so they stay out of the script that is rerun:
573
+
574
+ - `history_ddl_<schema>_<YYYYMMDD_HHMMSS>.sql`: the history table DDL followed by
575
+ the grants on the history tables for the history role. Run it against the
576
+ history database.
577
+ - `source_grants_<schema>_<YYYYMMDD_HHMMSS>.sql` (`SOURCE_ILM` only): the grants
578
+ on the source tables for the source role. Run it against the source database.
579
+
580
+ The runtime user reads the column metadata of the source tables through their
581
+ grants, so when source admin credentials exist `SCRIPT` applies the source
582
+ grants itself and writes no `source_grants` file. Without them the file is
583
+ written and `ldb-run` logs a warning. In every mode, a table the runtime user
584
+ cannot see (it does not exist, or `SELECT` is not granted) is skipped with a
585
+ warning instead of failing the run, and so is a table that references it.
586
+
587
+ Both files are plain SQL for SQL*Plus that stop at the first failing statement; the
588
+ grants are unconditional and safe to run again. The history DDL is always
589
+ emitted, computed like this:
590
+
591
+ - **A history connection works**: the admin connection is used when admin
592
+ credentials exist, the runtime one otherwise. The current history structure is
593
+ read, and the file holds only the missing changes.
594
+ - **Neither works or there are no credentials**: `history_baseline` states what to
595
+ assume. `EMPTY` (default) assumes history has no objects, so every table, key
596
+ and index is created. `SOURCE` assumes history holds the same objects as
597
+ source, so only the history-specific differences are emitted (for example the
598
+ `LDB_PROCESS_DATE` column and the matching primary key).
599
+
600
+ Without admin credentials, `ldb-run` applies the history grants right after the
601
+ DDL over the runtime connection. Any grant the runtime user cannot apply goes to
602
+ the same pending file as the failed DDL, so a DBA can run it.
406
603
 
407
604
  ### Configuration parameters reference
408
605
 
409
606
  | Parameter | Purpose | Default / values |
410
607
  | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
411
608
  | `action` | ILM target (`SOURCE_ILM` moves/purges source, `HISTORY_ILM` cleans downstream history). | Default `SOURCE_ILM`; choices `SOURCE_ILM`, `HISTORY_ILM`. |
412
- | `mode` | Execution mode: `VALIDATE` config/credentials, `PLAN` dependency order, `PREVIEW` simulate (a.k.a. legacy `DRY_RUN`), `SCRIPT` emit SQL, `EXECUTE` run changes. | Default `PREVIEW`; choices `VALIDATE`, `PLAN`, `PREVIEW`, `SCRIPT`, `EXECUTE`. |
609
+ | `mode` | Execution mode: `VALIDATE` config, credentials and role privileges (read-only), `PREPARE` apply grants and history objects without running ILM, `PLAN` dependency order, `PREVIEW` run the real queries and apply DDL and grants but do not insert or delete data (a.k.a. legacy `DRY_RUN`), `SCRIPT` emit the history DDL and the ILM SQL without applying anything, `EXECUTE` run changes. | Default `PREVIEW`; choices `VALIDATE`, `PLAN`, `PREVIEW`, `SCRIPT`, `EXECUTE`. |
413
610
  | `chunk_size` | Rows per chunk when processing large tables. | Default `100000`. |
414
611
  | `use_added_columns` | Populate derived columns in history tables. | Default `True` (boolean toggle). |
415
612
  | `add_ldb_columns` | Add LimitsDb execution-date columns in history tables. | Default `True` (boolean toggle). |
416
613
  | `parallel_max` | Maximum number of parallel processes. | Default `10`. |
614
+ | `ddl_pending_dir` | Directory where `ldb-run` writes history DDL it could not apply without admin credentials, and the DDL and grants files of `--mode SCRIPT`. | Default `ldb_ddl_pending`. |
615
+ | `history_baseline` | State assumed for the history tables by `--mode SCRIPT` when it cannot read them over the runtime history connection: `EMPTY` (no objects) or `SOURCE` (same objects as source). | Default `EMPTY`; choices `EMPTY`, `SOURCE`. |
417
616
  | `db_engine` | Database engine. | Default and only current choice: `"oracle"`. |
418
617
  | `log_level` | Logging level. | Default `"INFO"`; choices `"DEBUG"`, `"INFO"`, `"WARNING"`, `"ERROR"`, `"CRITICAL"`. |
419
618
  | `schema` | Configuration set name: identifies a system or application (e.g., `billing`). Not a database schema — it is the folder that groups all configuration files for one target. | Required on CLI; no persisted default. |
420
619
  | `profile` | Environment name (e.g., `dev`, `prod`). Differentiates configuration files for different environments within the same configuration set. | Optional; default `null`. |
421
620
  | `ilm_config_file` | YAML file with tables (bypass DB discovery). | Optional; default `null`. |
422
621
  | `source_dsn` | Source DSN / connection descriptor. | Default empty string. |
423
- | `source_username` | Source username. | Default empty string. |
622
+ | `source_username` | Source username, as named in the data dictionary: lowercase (`ldb`) is a quoted user created as `"ldb"`, uppercase (`LDB`) a regular one. | | Default empty string. |
424
623
  | `source_password` | Source password stored in secrets. | Default empty string; encrypted in `secrets.<PROFILE>.json`. |
425
624
  | `history_dsn` | History DSN / connection descriptor. | Default empty string. |
426
- | `history_username` | History username. | Default empty string. |
625
+ | `history_username` | History username, with the same case rules as `source_username`. || Default empty string. |
427
626
  | `history_password` | History password stored in secrets. | Default empty string; encrypted in `secrets.<PROFILE>.json`. |
428
627
  | `admin_source_username` | Admin source username. | Default empty string. |
429
628
  | `admin_source_password` | Admin source password stored in secrets. | Default empty string; encrypted in `secrets.<PROFILE>.json`. |
@@ -440,6 +639,8 @@ credentials in the current version.
440
639
  | `use_added_columns` | — | — | `config[.<PROFILE>].yml: use_added_columns` |
441
640
  | `add_ldb_columns` | — | — | `config[.<PROFILE>].yml: add_ldb_columns` |
442
641
  | `parallel_max` | `--parallel-max` | `LDB_PARALLEL_MAX` | `config[.<PROFILE>].yml: parallel_max` |
642
+ | `ddl_pending_dir` | `--ddl-pending-dir` | `LDB_DDL_PENDING_DIR` | `config[.<PROFILE>].yml: ddl_pending_dir` |
643
+ | `history_baseline` | `--history-baseline` | `LDB_HISTORY_BASELINE` | `config[.<PROFILE>].yml: history_baseline` |
443
644
  | `db_engine` | `--db-engine` | `LDB_DB_ENGINE` | `config[.<PROFILE>].yml: db_engine` |
444
645
  | `log_level` | `--log-level` | `LDB_LOG_LEVEL` | `config[.<PROFILE>].yml: log_level` |
445
646
  | `schema` | `--schema` | `LDB_SCHEMA` | CLI only (not stored). |
@@ -466,7 +667,9 @@ Each entry in `ilm[.<PROFILE>].yml` merges table-level attributes with one or mo
466
667
  - **Retention windows:** `retain_months_source` is required whenever `purge_date_expr` is present. `retain_months_history` extends the history window but also depends on defining the source retention. During `HISTORY_ILM` runs, the engine sums source and history months to determine the cut-off date.
467
668
  - **Date expressions:** `purge_date_expr` identifies the date column (or expression) that anchors retention. Prefix column references with `@` (for example, `"@DSP_DATE"`); the runner swaps the prefix for the proper table alias at execution time.
468
669
  - **Additional filters:** Add optional filters through `additional_filter_expr` (source runs) and `history_addtl_filter_expr` (history runs). Both accept the same `@column` syntax, and the history expression falls back to the source expression when omitted.
469
- - **Referencing tables:** Use `referencing_tables` to pull parent conditions into child tables. List entries as `<TABLE> <ALIAS>` (optionally `<OWNER>.<TABLE> <ALIAS>`) and supply matching `join_expr` fragments. The runner inherits active conditions from each referenced table and rewrites the join fragments—`@` becomes `inner` joins by default or `left outer` joins when `source_orphan_purge: true` to find orphans.
670
+ - **Referencing tables:** Use `referencing_tables` to pull parent conditions into child tables. List entries as `<TABLE> <ALIAS>` (optionally `<OWNER>.<TABLE> <ALIAS>`) and supply matching `join_expr` fragments. The runner inherits the active conditions of each referenced table: the conditions of one table are alternatives (`or`), and a child row is purged only when its own conditions and those of every referenced table are met (`and` between tables). It rewrites the join fragments—`@` becomes `inner` joins by default or `left outer` joins when `source_orphan_purge: true` to find orphans.
671
+ - **Join owners in history:** `join_expr` names each referenced table by its source owner. In `HISTORY_ILM` runs with `use_added_columns: false` the runner joins the history copy instead, replacing `<source_owner>.<table>` with `<history_owner>.<table>` for every table listed in `referencing_tables`. With `use_added_columns: true` history runs do not join.
672
+ - **Extra tables:** Filters may contain subqueries over tables that ILM does not manage. List them in `extra_select_tables` (read by `additional_filter_expr`, so source runs) and `history_extra_select_tables` (read by `history_addtl_filter_expr`) as comma-separated `[OWNER.]TABLE` entries; an entry without owner belongs to the environment's own owner. The environment's role gets `SELECT` on them wherever it gets the other table grants (`ldb-run`, `--mode PREPARE`, `--mode SCRIPT`) and the privilege checks include them. In history, `history_extra_select_tables` falls back to `extra_select_tables`, just as `history_addtl_filter_expr` falls back to `additional_filter_expr`. A table also managed by ILM keeps its full privileges. For tables that are not in the history database write the access in `history_addtl_filter_expr` yourself (for example `TABLE@SRC`, or `TABLE UNION ALL TABLE@SRC` in a subquery); LimitsDb does not rewrite subqueries. `SRC` is only the default name of the history-to-source database link; it can be overridden with `history_to_source_dblink_name` (and `source_to_history_dblink_name`, default `HIST`, for the other direction). The link connects as the source runtime user and uses the grants of its source role, so a table read as `TABLE@SRC` must also be listed in `extra_select_tables`.
470
673
  - **Historical predicate snapshot:** With `use_added_columns` enabled, every archived row preserves the values needed to reevaluate all its predicates in history, including values sourced from related tables. Distinct cut-off dates are stored in helper columns such as `ldb_date_<suffix>`. Predicates that share a date reuse the same helper column while retaining their own filters and retention windows. Other values sourced from related tables are materialized as `<column>_<table_alias>`.
471
674
  - **Orphan handling:** Enable `source_orphan_purge` with `orphan_check_column` to archive source rows whose referenced parent is absent. The source query uses `left outer` joins and treats the row as an orphan when any comma-separated check column is `NULL`. `referencing_tables`, `join_expr`, `orphan_check_column`, `use_added_columns: true`, and a positive historical retention are required. Orphans follow the table's normal source-to-history flow rather than being deleted directly. LimitsDb snapshots this state in `LDB_IS_ORPHAN` and stores a fallback date so `HISTORY_ILM` can apply the configured retention without accessing the missing parent.
472
675
 
@@ -492,11 +695,13 @@ Every key is commented; the default value and short help appear inline. Example:
492
695
 
493
696
  ```yaml
494
697
  # action: "SOURCE_ILM" # ILM target: months_keep_history_max (SOURCE_ILM) or history (HISTORY_ILM)
495
- # mode: "PREVIEW" # VALIDATE | PLAN | PREVIEW | SCRIPT | EXECUTE (PREVIEW mirrors legacy DRY_RUN)
698
+ # mode: "PREVIEW" # VALIDATE | PREPARE | PLAN | PREVIEW | SCRIPT | EXECUTE
496
699
  # chunk_size: 100000 # Rows per chunk when processing large tables
497
700
  # use_added_columns: True # Populate derived columns in history tables
498
701
  # add_ldb_columns: True # Add LimitsDb execution-date columns in history tables
499
702
  # parallel_max: 10 # Maximum number of parallel processes
703
+ # ddl_pending_dir: "ldb_ddl_pending" # Where failed history DDL is written for a DBA
704
+ # history_baseline: "EMPTY" # SCRIPT without readable history: EMPTY | SOURCE
500
705
  # db_engine: "oracle" # Database engine
501
706
  # log_level: "INFO" # Logging level
502
707
  ```