limitsdb 0.5.0__tar.gz → 0.7.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {limitsdb-0.5.0 → limitsdb-0.7.0}/PKG-INFO +223 -18
- {limitsdb-0.5.0 → limitsdb-0.7.0}/README.md +222 -17
- limitsdb-0.7.0/limitsdb/cli/ldb_impl.py +316 -0
- limitsdb-0.7.0/limitsdb/core/ldb_ddl.py +155 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_errors.py +8 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_ilm_config.py +6 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_logger.py +5 -5
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_params_config.py +33 -8
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_runner.py +641 -58
- limitsdb-0.7.0/limitsdb/core/ldb_status.py +31 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/ldb_engines.py +186 -19
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/oracle/ldb_engine_impl.py +957 -256
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/resources/ilm.example.yml +4 -2
- {limitsdb-0.5.0 → limitsdb-0.7.0}/pyproject.toml +1 -1
- limitsdb-0.5.0/limitsdb/cli/ldb_impl.py +0 -207
- limitsdb-0.5.0/limitsdb/core/ldb_status.py +0 -8
- {limitsdb-0.5.0 → limitsdb-0.7.0}/LICENSE +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/LICENSE-DUAL.md +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/NOTICE +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/__init__.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/cli/__init__.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/cli/ldb_crypt.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/cli/ldb_init.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/cli/ldb_run.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/__init__.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_config_loader.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_crypto.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_meta.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/core/ldb_utils.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/__init__.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/ldb_engine_loader.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/oracle/__init__.py +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/db/oracle/ldb_conf.sql +0 -0
- {limitsdb-0.5.0 → limitsdb-0.7.0}/limitsdb/resources/__init__.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: limitsdb
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.7.0
|
|
4
4
|
Summary: Configuration-driven ILM (Information Lifecycle Management) for production and history databases
|
|
5
5
|
License: Apache License
|
|
6
6
|
Version 2.0, January 2004
|
|
@@ -207,6 +207,52 @@ source .venv/bin/activate
|
|
|
207
207
|
pip install limitsdb
|
|
208
208
|
```
|
|
209
209
|
|
|
210
|
+
### Air-gapped or internet-restricted servers
|
|
211
|
+
|
|
212
|
+
For production database servers without internet access (common in enterprise Oracle
|
|
213
|
+
environments on Linux, AIX, or Solaris), download the packages on a machine that
|
|
214
|
+
does have access and transfer them to the target server.
|
|
215
|
+
|
|
216
|
+
**Step 1 — Download on a connected machine**
|
|
217
|
+
|
|
218
|
+
If the target server is Linux x86_64:
|
|
219
|
+
|
|
220
|
+
```bash
|
|
221
|
+
pip download limitsdb \
|
|
222
|
+
--dest ./limitsdb-bundle \
|
|
223
|
+
--platform linux_x86_64 \
|
|
224
|
+
--python-version 312 \
|
|
225
|
+
--only-binary=:all:
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
For the same platform as the current machine (Windows or macOS build host
|
|
229
|
+
copying to an identical Linux target, omit `--platform` and `--python-version`):
|
|
230
|
+
|
|
231
|
+
```bash
|
|
232
|
+
pip download limitsdb --dest ./limitsdb-bundle
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
**Step 2 — Transfer to the target server**
|
|
236
|
+
|
|
237
|
+
```bash
|
|
238
|
+
scp -r ./limitsdb-bundle user@server:/opt/limitsdb-bundle
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
**Step 3 — Install on the target server (no internet required)**
|
|
242
|
+
|
|
243
|
+
```bash
|
|
244
|
+
python3 -m venv /opt/limitsdb-venv
|
|
245
|
+
source /opt/limitsdb-venv/bin/activate
|
|
246
|
+
pip install --no-index --find-links /opt/limitsdb-bundle limitsdb
|
|
247
|
+
ldb-run --help
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
> **Note for AIX and Solaris**: `oracledb` may not have a prebuilt wheel for your
|
|
251
|
+
> platform. If `pip download --only-binary=:all:` finds nothing, the package must
|
|
252
|
+
> be compiled from source on the target, which requires Python development headers
|
|
253
|
+
> and a C compiler (`gcc` or `xlc`). Contact your system administrator before
|
|
254
|
+
> proceeding.
|
|
255
|
+
|
|
210
256
|
For development setup (cloning the repo, running tests, CI) see [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
211
257
|
|
|
212
258
|
## Quickstart
|
|
@@ -227,7 +273,7 @@ Spin up a minimal end-to-end run with the built-in scaffolding tools:
|
|
|
227
273
|
ldb-crypt --schema billing --profile dev
|
|
228
274
|
```
|
|
229
275
|
|
|
230
|
-
4. **
|
|
276
|
+
4. **Preview the ILM run** (applies the required DDL and grants, moves no data):
|
|
231
277
|
|
|
232
278
|
```bash
|
|
233
279
|
ldb-run --schema billing --profile dev --action SOURCE_ILM --mode PREVIEW
|
|
@@ -356,9 +402,40 @@ What it creates in both source and history environments:
|
|
|
356
402
|
- **Database link** — bidirectional links between source and history so the ILM
|
|
357
403
|
engine can reach both environments from either side.
|
|
358
404
|
|
|
359
|
-
`ldb-impl`
|
|
360
|
-
`secrets[.<PROFILE>].json
|
|
361
|
-
|
|
405
|
+
`ldb-impl` needs the runtime credentials (`source_password`, `history_password`)
|
|
406
|
+
encrypted in `secrets[.<PROFILE>].json`. Run `ldb-crypt` first.
|
|
407
|
+
|
|
408
|
+
### Without admin credentials
|
|
409
|
+
|
|
410
|
+
Admin credentials (`admin_source_*`, `admin_history_*`) are optional per
|
|
411
|
+
database. When they are absent for a database, `ldb-impl` creates nothing in it
|
|
412
|
+
and writes a SQL*Plus script for a DBA instead, one file per database:
|
|
413
|
+
`bootstrap_<schema>_<source|history>_<YYYYMMDD_HHMMSS>.sql` under
|
|
414
|
+
`ddl_pending_dir` (the same parameter as `ldb-run`: `--ddl-pending-dir`,
|
|
415
|
+
`LDB_DDL_PENDING_DIR`, default `ldb_ddl_pending`). A database whose admin credentials are present is still
|
|
416
|
+
provisioned directly. Generating scripts is not an error (exit code 0, with a
|
|
417
|
+
warning naming each file); if a script cannot be written, `ldb-impl` exits 1.
|
|
418
|
+
|
|
419
|
+
The script:
|
|
420
|
+
|
|
421
|
+
- stops at the first error (`whenever sqlerror exit failure`) and ends with a
|
|
422
|
+
`LimitsDb: OK` banner on success;
|
|
423
|
+
- contains no passwords: it prompts for all of them (hidden input) at its start,
|
|
424
|
+
before it changes anything;
|
|
425
|
+
- can be rerun: each object is created in a PL/SQL block that ignores only its
|
|
426
|
+
own "already exists" error, and an existing database link is skipped;
|
|
427
|
+
- creates the database link after reconnecting as the runtime user, who owns it.
|
|
428
|
+
It asks for that user's password once and reuses it to reconnect. An existing user keeps its password, so the one entered must
|
|
429
|
+
match it; the script says so before it reconnects.
|
|
430
|
+
|
|
431
|
+
Run it in SQL*Plus connected as a user with administrative privileges:
|
|
432
|
+
`sqlplus admin@<dsn> @bootstrap_<schema>_<env>_<timestamp>.sql`.
|
|
433
|
+
|
|
434
|
+
When `ldb-impl` has admin credentials and the runtime user already existed with
|
|
435
|
+
a different password, the reconnection as that user is rejected (ORA-01017).
|
|
436
|
+
`ldb-impl` then logs that the user rejected the configured password, naming the
|
|
437
|
+
user and the database, and exits 1; it never changes the password of an existing
|
|
438
|
+
user.
|
|
362
439
|
|
|
363
440
|
## Run ILM
|
|
364
441
|
|
|
@@ -372,9 +449,11 @@ ldb-run --schema <SCHEMA> [--profile <PROFILE>] --action SOURCE_ILM|HISTORY_ILM
|
|
|
372
449
|
Common overrides:
|
|
373
450
|
|
|
374
451
|
```
|
|
375
|
-
--mode VALIDATE|PLAN|PREVIEW|SCRIPT|EXECUTE # VALIDATE setup, PLAN DAG, PREVIEW
|
|
452
|
+
--mode VALIDATE|PREPARE|PLAN|PREVIEW|SCRIPT|EXECUTE # VALIDATE setup and privileges (read-only), PREPARE grants and objects, PLAN DAG, PREVIEW run without moving data, SCRIPT emit SQL, EXECUTE run
|
|
376
453
|
--chunk-size <int> # Rows per chunk when processing large tables
|
|
377
454
|
--parallel-max <int> # Maximum number of tables processed in parallel
|
|
455
|
+
--ddl-pending-dir <path> # Where failed history DDL is written for a DBA (default: ldb_ddl_pending)
|
|
456
|
+
--history-baseline EMPTY|SOURCE # SCRIPT without a readable history: assume no history objects (default) or the same objects as source
|
|
378
457
|
--log-level DEBUG|INFO|WARNING|ERROR|CRITICAL # Adjust logger verbosity
|
|
379
458
|
--ilm-config-file <path> # Point to a specific ILM YAML file
|
|
380
459
|
--config-dir <root> # Override config root discovery
|
|
@@ -396,34 +475,154 @@ ldb-run --schema billing --profile prod --action SOURCE_ILM --mode EXECUTE --set
|
|
|
396
475
|
|
|
397
476
|
`ldb-run` uses admin credentials (`admin_source_password`,
|
|
398
477
|
`admin_history_password`) for two operations: granting table privileges to the
|
|
399
|
-
application role, and creating or altering history tables before
|
|
400
|
-
run
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
478
|
+
application role, and creating or altering history tables before a `PREVIEW` or
|
|
479
|
+
`EXECUTE` run (`PREVIEW` needs the objects in place to validate and time the
|
|
480
|
+
queries; it skips only the data movement). Both operations are idempotent.
|
|
481
|
+
|
|
482
|
+
A `PREVIEW` writes its own statuses in `ldb_ctl`, the `EXECUTE` ones prefixed with `P` (`PTSTART`, `PCSTART`,
|
|
483
|
+
`PCEND`, `PLSTART`, `PLEND`, `PTEND`, `PERROR`), and finishes each table with `PTEND`, not `TEND`, so it never marks a
|
|
484
|
+
table as processed for the day: an `EXECUTE` run afterwards (same day) still moves the data. A `PREVIEW` is refused
|
|
485
|
+
(`ORA-20006`) over a table that has an `EXECUTE` status for the same process date. An `EXECUTE` continues an incomplete
|
|
486
|
+
`EXECUTE` of the same process date (finished tables stay finished and the rows already moved are kept); in any other case
|
|
487
|
+
the table starts over and the abandoned run is closed in `ldb_log` as "Incomplete process closed by a later process".
|
|
488
|
+
|
|
489
|
+
Only one `ldb-run` can work on a table at a time: the runner takes a lock per table in `LDB_RUN_LOCK` and a second runner
|
|
490
|
+
stops with exit code 1 before starting anything. After upgrading, rerun `ldb-impl` so the new table and the supporting
|
|
491
|
+
procedures (`check_save_status`, `check_referencing_tables`) are installed.
|
|
492
|
+
|
|
493
|
+
The generated `SCRIPT` takes the mode from SQL*Plus and runs without the runner lock:
|
|
494
|
+
`sqlplus user/password@dsn @script EXECUTE` (or `PREVIEW`). Its process date is the `ldb_process_date` variable.
|
|
495
|
+
|
|
496
|
+
If source admin credentials are absent from `secrets[.<PROFILE>].json`, the
|
|
497
|
+
source privilege grants are not applied. `ldb-run` checks with the runtime
|
|
498
|
+
connection which privileges are missing through the role and logs a warning
|
|
499
|
+
naming them; a table the runtime user cannot see is skipped with a warning.
|
|
500
|
+
Run `--mode PREPARE` to get the missing grants as a file for a DBA.
|
|
501
|
+
|
|
502
|
+
History table management follows the same principle:
|
|
503
|
+
|
|
504
|
+
- **Admin credentials present, tables already up to date**: the admin connection
|
|
505
|
+
is not opened. No DDL is executed.
|
|
506
|
+
- **Admin credentials present, DDL changes are required**: the admin connection
|
|
507
|
+
is opened and the changes are applied as before.
|
|
508
|
+
- **Admin credentials absent, tables already up to date**: `ldb-run` proceeds
|
|
509
|
+
without an admin connection. This is the normal case for environments where a
|
|
510
|
+
DBA pre-creates all history objects and admin credentials are not stored.
|
|
511
|
+
- **Admin credentials absent, history schema cannot be inspected** (for example
|
|
512
|
+
the runtime connection or dictionary query fails): `ldb-run` exits with code 1
|
|
513
|
+
and tells the operator to run `ldb-impl` so a DBA completes the configuration. It does not run ILM against a history schema whose state is
|
|
514
|
+
unknown.
|
|
515
|
+
- **Admin credentials absent, DDL changes are required**: `ldb-run` applies the
|
|
516
|
+
DDL directly over the runtime history connection, with no prior privilege
|
|
517
|
+
check. It succeeds when the runtime user owns the objects or holds the needed
|
|
518
|
+
privileges (for example `ALTER`). Any DDL statement that fails, for whatever
|
|
519
|
+
reason, is recorded and the remaining statements are still attempted. If at least one
|
|
520
|
+
statement failed, `ldb-run` writes the failed statements, each preceded by its
|
|
521
|
+
error, to `history_ddl_<schema>_<YYYYMMDD_HHMMSS>.sql` under `ddl_pending_dir`
|
|
522
|
+
(default `ldb_ddl_pending`), logs an error naming the file, and exits with
|
|
523
|
+
code 1. The file is plain SQL for SQL*Plus (no PL/SQL): it stops at the first
|
|
524
|
+
failing statement with a non-zero exit code, and only a run that reaches the
|
|
525
|
+
final `LimitsDb: OK` banner applied everything. The recorded errors are
|
|
526
|
+
comments. A DBA reviews and runs the file, then `ldb-run` is rerun. Nothing is
|
|
527
|
+
written to stdout, and the directory is created only when a file is written.
|
|
528
|
+
If the file cannot be written (for example the directory is not writable),
|
|
529
|
+
`ldb-run` logs the error with the failed statements and exits with code 1.
|
|
530
|
+
|
|
531
|
+
### Preparing grants and objects with `--mode PREPARE`
|
|
532
|
+
|
|
533
|
+
`--mode PREPARE` brings permissions and objects into place without running any
|
|
534
|
+
ILM work, so the problems show up before `SCRIPT`, `PREVIEW` or `EXECUTE`. It
|
|
535
|
+
is safe to rerun and meant to run first (`VALIDATE`, `PREPARE`, then the rest):
|
|
536
|
+
|
|
537
|
+
- With source admin credentials (`SOURCE_ILM`) it grants the source tables to
|
|
538
|
+
the source role. Without them it writes `source_grants_<schema>_<ts>.sql` under
|
|
539
|
+
`ddl_pending_dir` for a DBA, with only the privileges the runtime user does not
|
|
540
|
+
receive yet through the role, and warns. Once the grants are in place no file
|
|
541
|
+
is written.
|
|
542
|
+
- It checks, with the runtime connections alone, that the source and history
|
|
543
|
+
roles are enabled for the runtime users and give them `SELECT`, `INSERT`,
|
|
544
|
+
`UPDATE` and `DELETE` on every configured table. A role that is granted but
|
|
545
|
+
not a default role gives nothing.
|
|
546
|
+
The role must stay a default role of the runtime user (`ldb-run` warns
|
|
547
|
+
when it is not); `GRANT role` makes it default unless the user was altered.
|
|
548
|
+
- The default role names are `LDB_SOURCE_ROLE` and `LDB_HISTORY_ROLE`. When
|
|
549
|
+
several schemas (configuration sets) share the same SOURCE or HISTORY
|
|
550
|
+
database, those names collide; set `source_role_name` / `history_role_name`
|
|
551
|
+
per schema to keep the roles apart. Reusing one role across schemas also
|
|
552
|
+
works and is rarely a problem.
|
|
553
|
+
- It creates or alters the history tables and grants them to the history role,
|
|
554
|
+
with the same admin or runtime-connection rules as `EXECUTE`: statements that
|
|
555
|
+
fail go to the pending file.
|
|
556
|
+
- It exits with code 1 if the runtime user still cannot see a configured table
|
|
557
|
+
(it does not exist or `SELECT` is not granted). Other modes only skip such a
|
|
558
|
+
table with a warning.
|
|
559
|
+
|
|
560
|
+
`VALIDATE` runs the same privilege check without changing anything: a gap on
|
|
561
|
+
the action's own environment fails it, a gap on the other environment is a
|
|
562
|
+
warning, since `PREPARE` creates the history objects. `SCRIPT`, `PREVIEW` and
|
|
563
|
+
`EXECUTE` warn about the gaps they find.
|
|
564
|
+
|
|
565
|
+
`PREVIEW` and `EXECUTE` keep applying DDL and grants themselves, so a change in
|
|
566
|
+
source reaches history without failing the run.
|
|
567
|
+
|
|
568
|
+
### History DDL in `--mode SCRIPT`
|
|
569
|
+
|
|
570
|
+
`--mode SCRIPT` applies no DDL and no history grants to the databases. The ILM SQL blocks, which
|
|
571
|
+
run many times, go to stdout. The one-time statements go to files under
|
|
572
|
+
`ddl_pending_dir`, so they stay out of the script that is rerun:
|
|
573
|
+
|
|
574
|
+
- `history_ddl_<schema>_<YYYYMMDD_HHMMSS>.sql`: the history table DDL followed by
|
|
575
|
+
the grants on the history tables for the history role. Run it against the
|
|
576
|
+
history database.
|
|
577
|
+
- `source_grants_<schema>_<YYYYMMDD_HHMMSS>.sql` (`SOURCE_ILM` only): the grants
|
|
578
|
+
on the source tables for the source role. Run it against the source database.
|
|
579
|
+
|
|
580
|
+
The runtime user reads the column metadata of the source tables through their
|
|
581
|
+
grants, so when source admin credentials exist `SCRIPT` applies the source
|
|
582
|
+
grants itself and writes no `source_grants` file. Without them the file is
|
|
583
|
+
written and `ldb-run` logs a warning. In every mode, a table the runtime user
|
|
584
|
+
cannot see (it does not exist, or `SELECT` is not granted) is skipped with a
|
|
585
|
+
warning instead of failing the run, and so is a table that references it.
|
|
586
|
+
|
|
587
|
+
Both files are plain SQL for SQL*Plus that stop at the first failing statement; the
|
|
588
|
+
grants are unconditional and safe to run again. The history DDL is always
|
|
589
|
+
emitted, computed like this:
|
|
590
|
+
|
|
591
|
+
- **A history connection works**: the admin connection is used when admin
|
|
592
|
+
credentials exist, the runtime one otherwise. The current history structure is
|
|
593
|
+
read, and the file holds only the missing changes.
|
|
594
|
+
- **Neither works or there are no credentials**: `history_baseline` states what to
|
|
595
|
+
assume. `EMPTY` (default) assumes history has no objects, so every table, key
|
|
596
|
+
and index is created. `SOURCE` assumes history holds the same objects as
|
|
597
|
+
source, so only the history-specific differences are emitted (for example the
|
|
598
|
+
`LDB_PROCESS_DATE` column and the matching primary key).
|
|
599
|
+
|
|
600
|
+
Without admin credentials, `ldb-run` applies the history grants right after the
|
|
601
|
+
DDL over the runtime connection. Any grant the runtime user cannot apply goes to
|
|
602
|
+
the same pending file as the failed DDL, so a DBA can run it.
|
|
406
603
|
|
|
407
604
|
### Configuration parameters reference
|
|
408
605
|
|
|
409
606
|
| Parameter | Purpose | Default / values |
|
|
410
607
|
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
|
|
411
608
|
| `action` | ILM target (`SOURCE_ILM` moves/purges source, `HISTORY_ILM` cleans downstream history). | Default `SOURCE_ILM`; choices `SOURCE_ILM`, `HISTORY_ILM`. |
|
|
412
|
-
| `mode` | Execution mode: `VALIDATE` config
|
|
609
|
+
| `mode` | Execution mode: `VALIDATE` config, credentials and role privileges (read-only), `PREPARE` apply grants and history objects without running ILM, `PLAN` dependency order, `PREVIEW` run the real queries and apply DDL and grants but do not insert or delete data (a.k.a. legacy `DRY_RUN`), `SCRIPT` emit the history DDL and the ILM SQL without applying anything, `EXECUTE` run changes. | Default `PREVIEW`; choices `VALIDATE`, `PLAN`, `PREVIEW`, `SCRIPT`, `EXECUTE`. |
|
|
413
610
|
| `chunk_size` | Rows per chunk when processing large tables. | Default `100000`. |
|
|
414
611
|
| `use_added_columns` | Populate derived columns in history tables. | Default `True` (boolean toggle). |
|
|
415
612
|
| `add_ldb_columns` | Add LimitsDb execution-date columns in history tables. | Default `True` (boolean toggle). |
|
|
416
613
|
| `parallel_max` | Maximum number of parallel processes. | Default `10`. |
|
|
614
|
+
| `ddl_pending_dir` | Directory where `ldb-run` writes history DDL it could not apply without admin credentials, and the DDL and grants files of `--mode SCRIPT`. | Default `ldb_ddl_pending`. |
|
|
615
|
+
| `history_baseline` | State assumed for the history tables by `--mode SCRIPT` when it cannot read them over the runtime history connection: `EMPTY` (no objects) or `SOURCE` (same objects as source). | Default `EMPTY`; choices `EMPTY`, `SOURCE`. |
|
|
417
616
|
| `db_engine` | Database engine. | Default and only current choice: `"oracle"`. |
|
|
418
617
|
| `log_level` | Logging level. | Default `"INFO"`; choices `"DEBUG"`, `"INFO"`, `"WARNING"`, `"ERROR"`, `"CRITICAL"`. |
|
|
419
618
|
| `schema` | Configuration set name: identifies a system or application (e.g., `billing`). Not a database schema — it is the folder that groups all configuration files for one target. | Required on CLI; no persisted default. |
|
|
420
619
|
| `profile` | Environment name (e.g., `dev`, `prod`). Differentiates configuration files for different environments within the same configuration set. | Optional; default `null`. |
|
|
421
620
|
| `ilm_config_file` | YAML file with tables (bypass DB discovery). | Optional; default `null`. |
|
|
422
621
|
| `source_dsn` | Source DSN / connection descriptor. | Default empty string. |
|
|
423
|
-
| `source_username` | Source username.
|
|
622
|
+
| `source_username` | Source username, as named in the data dictionary: lowercase (`ldb`) is a quoted user created as `"ldb"`, uppercase (`LDB`) a regular one. | | Default empty string. |
|
|
424
623
|
| `source_password` | Source password stored in secrets. | Default empty string; encrypted in `secrets.<PROFILE>.json`. |
|
|
425
624
|
| `history_dsn` | History DSN / connection descriptor. | Default empty string. |
|
|
426
|
-
| `history_username` | History username
|
|
625
|
+
| `history_username` | History username, with the same case rules as `source_username`. || Default empty string. |
|
|
427
626
|
| `history_password` | History password stored in secrets. | Default empty string; encrypted in `secrets.<PROFILE>.json`. |
|
|
428
627
|
| `admin_source_username` | Admin source username. | Default empty string. |
|
|
429
628
|
| `admin_source_password` | Admin source password stored in secrets. | Default empty string; encrypted in `secrets.<PROFILE>.json`. |
|
|
@@ -440,6 +639,8 @@ credentials in the current version.
|
|
|
440
639
|
| `use_added_columns` | — | — | `config[.<PROFILE>].yml: use_added_columns` |
|
|
441
640
|
| `add_ldb_columns` | — | — | `config[.<PROFILE>].yml: add_ldb_columns` |
|
|
442
641
|
| `parallel_max` | `--parallel-max` | `LDB_PARALLEL_MAX` | `config[.<PROFILE>].yml: parallel_max` |
|
|
642
|
+
| `ddl_pending_dir` | `--ddl-pending-dir` | `LDB_DDL_PENDING_DIR` | `config[.<PROFILE>].yml: ddl_pending_dir` |
|
|
643
|
+
| `history_baseline` | `--history-baseline` | `LDB_HISTORY_BASELINE` | `config[.<PROFILE>].yml: history_baseline` |
|
|
443
644
|
| `db_engine` | `--db-engine` | `LDB_DB_ENGINE` | `config[.<PROFILE>].yml: db_engine` |
|
|
444
645
|
| `log_level` | `--log-level` | `LDB_LOG_LEVEL` | `config[.<PROFILE>].yml: log_level` |
|
|
445
646
|
| `schema` | `--schema` | `LDB_SCHEMA` | CLI only (not stored). |
|
|
@@ -466,7 +667,9 @@ Each entry in `ilm[.<PROFILE>].yml` merges table-level attributes with one or mo
|
|
|
466
667
|
- **Retention windows:** `retain_months_source` is required whenever `purge_date_expr` is present. `retain_months_history` extends the history window but also depends on defining the source retention. During `HISTORY_ILM` runs, the engine sums source and history months to determine the cut-off date.
|
|
467
668
|
- **Date expressions:** `purge_date_expr` identifies the date column (or expression) that anchors retention. Prefix column references with `@` (for example, `"@DSP_DATE"`); the runner swaps the prefix for the proper table alias at execution time.
|
|
468
669
|
- **Additional filters:** Add optional filters through `additional_filter_expr` (source runs) and `history_addtl_filter_expr` (history runs). Both accept the same `@column` syntax, and the history expression falls back to the source expression when omitted.
|
|
469
|
-
- **Referencing tables:** Use `referencing_tables` to pull parent conditions into child tables. List entries as `<TABLE> <ALIAS>` (optionally `<OWNER>.<TABLE> <ALIAS>`) and supply matching `join_expr` fragments. The runner inherits active conditions
|
|
670
|
+
- **Referencing tables:** Use `referencing_tables` to pull parent conditions into child tables. List entries as `<TABLE> <ALIAS>` (optionally `<OWNER>.<TABLE> <ALIAS>`) and supply matching `join_expr` fragments. The runner inherits the active conditions of each referenced table: the conditions of one table are alternatives (`or`), and a child row is purged only when its own conditions and those of every referenced table are met (`and` between tables). It rewrites the join fragments—`@` becomes `inner` joins by default or `left outer` joins when `source_orphan_purge: true` to find orphans.
|
|
671
|
+
- **Join owners in history:** `join_expr` names each referenced table by its source owner. In `HISTORY_ILM` runs with `use_added_columns: false` the runner joins the history copy instead, replacing `<source_owner>.<table>` with `<history_owner>.<table>` for every table listed in `referencing_tables`. With `use_added_columns: true` history runs do not join.
|
|
672
|
+
- **Extra tables:** Filters may contain subqueries over tables that ILM does not manage. List them in `extra_select_tables` (read by `additional_filter_expr`, so source runs) and `history_extra_select_tables` (read by `history_addtl_filter_expr`) as comma-separated `[OWNER.]TABLE` entries; an entry without owner belongs to the environment's own owner. The environment's role gets `SELECT` on them wherever it gets the other table grants (`ldb-run`, `--mode PREPARE`, `--mode SCRIPT`) and the privilege checks include them. In history, `history_extra_select_tables` falls back to `extra_select_tables`, just as `history_addtl_filter_expr` falls back to `additional_filter_expr`. A table also managed by ILM keeps its full privileges. For tables that are not in the history database write the access in `history_addtl_filter_expr` yourself (for example `TABLE@SRC`, or `TABLE UNION ALL TABLE@SRC` in a subquery); LimitsDb does not rewrite subqueries. `SRC` is only the default name of the history-to-source database link; it can be overridden with `history_to_source_dblink_name` (and `source_to_history_dblink_name`, default `HIST`, for the other direction). The link connects as the source runtime user and uses the grants of its source role, so a table read as `TABLE@SRC` must also be listed in `extra_select_tables`.
|
|
470
673
|
- **Historical predicate snapshot:** With `use_added_columns` enabled, every archived row preserves the values needed to reevaluate all its predicates in history, including values sourced from related tables. Distinct cut-off dates are stored in helper columns such as `ldb_date_<suffix>`. Predicates that share a date reuse the same helper column while retaining their own filters and retention windows. Other values sourced from related tables are materialized as `<column>_<table_alias>`.
|
|
471
674
|
- **Orphan handling:** Enable `source_orphan_purge` with `orphan_check_column` to archive source rows whose referenced parent is absent. The source query uses `left outer` joins and treats the row as an orphan when any comma-separated check column is `NULL`. `referencing_tables`, `join_expr`, `orphan_check_column`, `use_added_columns: true`, and a positive historical retention are required. Orphans follow the table's normal source-to-history flow rather than being deleted directly. LimitsDb snapshots this state in `LDB_IS_ORPHAN` and stores a fallback date so `HISTORY_ILM` can apply the configured retention without accessing the missing parent.
|
|
472
675
|
|
|
@@ -492,11 +695,13 @@ Every key is commented; the default value and short help appear inline. Example:
|
|
|
492
695
|
|
|
493
696
|
```yaml
|
|
494
697
|
# action: "SOURCE_ILM" # ILM target: months_keep_history_max (SOURCE_ILM) or history (HISTORY_ILM)
|
|
495
|
-
# mode: "PREVIEW" # VALIDATE | PLAN | PREVIEW | SCRIPT | EXECUTE
|
|
698
|
+
# mode: "PREVIEW" # VALIDATE | PREPARE | PLAN | PREVIEW | SCRIPT | EXECUTE
|
|
496
699
|
# chunk_size: 100000 # Rows per chunk when processing large tables
|
|
497
700
|
# use_added_columns: True # Populate derived columns in history tables
|
|
498
701
|
# add_ldb_columns: True # Add LimitsDb execution-date columns in history tables
|
|
499
702
|
# parallel_max: 10 # Maximum number of parallel processes
|
|
703
|
+
# ddl_pending_dir: "ldb_ddl_pending" # Where failed history DDL is written for a DBA
|
|
704
|
+
# history_baseline: "EMPTY" # SCRIPT without readable history: EMPTY | SOURCE
|
|
500
705
|
# db_engine: "oracle" # Database engine
|
|
501
706
|
# log_level: "INFO" # Logging level
|
|
502
707
|
```
|